Infinite Horizon LQ Zero-Sum Stochastic Differential Games with Markovian Jumps ()
1. Introduction
The stochastic control problems governed by Itô’s differential equation have become a popular research topic in past decades. Recently, stochastic H∞ control problem with state and control—dependent noise was considered [1,2]. It has attracted much attention and has been widely applied to various fields. Particularly, the stochastic H2/H∞ control with state-dependent noise has been addressed [3,4]. Recently, linear quadratic differential games and their applications have been widely investigated in many literatures, and examples of differential games in economics and management science can be found e.g. in [5-9]. These results are mainly based on the deterministic systems. However, to the best of our knowledge, few results have been obtained for stochastic differential games with Markovian jumps.
In this paper, the stochastic zero-sum games for linear quadratic systems governed by Itô’s differential equations with state-dependent noise and Markovian jumps are addressed, Such class of systems has important applications in engineering practice since they can be used to represent random failure processes in manufacturing systems, electric power systems and so on, see [10-18]. In particular, stability and robust stabilization for such perturbed systems were investigated extensively in [13, 15,17]. A bounded real lemma for Markovian jump stochastic systems was derived in [13]. [11] studied the optimal filtering problem for such systems, while [12,14,16] addressed the issue of linear quadratic regulator. The goal of this paper is to develop the differential game theory for stochastic Itô systems with Markovian jumps, a necessary and sufficient condition is developed for the existence of optimal control strategies in terms of a coupled algebraic Riccati equations (AREs), which can be viewed as an extension of the existing results of [19]. In the end, stochastic H∞ control problem with Markovian jumps is given as our theoretical applications and an illustrative example is presented.
For convenience, we will make use of the following notations in this paper:
AT: transpose of a matrix or vector A; A−1: inverse of a matrix or vector A; A > 0 (A ≥ 0): positive definite (positive semidefinite) symmetric matrix A; χA: indicator function of a set A;
: space of all
with A (i) being n × m matrix,
;
;
: space of all n × n symmetric matrices;
: space of all
with A(i) being n × n symmetric matrix,
;
means
for
;
: space of all n-dimensional real vectors with usual 2-norm | • |.
2. Definitions and Preliminaries
Throughout this paper, let
be a given filtered probability space where there exists a standard one dimensional Wiener process
, and a right continuous homogeneous Markov chain
with state space
. We assume that
is independent of
and has the following transition probability:
(1)
where
for
and
. Ft stands for the smallest σ-algebra generated by process
, rs,
, i.e.
.
By
denote the space of all measureable functions
, which is Ft -measurable for every
, and
,
. Obviously,
is a Hilbert space with the inner product
![](https://www.scirp.org/html/12-7400912\7e4b3abf-842f-46ec-8934-167826cc9bb0.jpg)
Consider the following linear stochastic controlled system with Markovian jumps
(2)
where
and
are the state and control input, respectively. The coefficients
and
with A(i), B(i), C(i),
, being constant matrices.
It is well known that for any
and
, there exists a unique solution
of (1) with initial condition
,
. Next, we first introduce the definition of stochastic stabilizability which is an essential assumption in this paper.
Definition 1. System (2) or (A, B, C) is called stochastic stabilizable (in mean-square sense), if there exists a feedback control
with
being constant matrices, such that for any initial state
,
, the closed-loop system
![](https://www.scirp.org/html/12-7400912\796c537e-0a1a-44d2-b0ba-6deab920f654.jpg)
is asymptotically mean-square stable, i.e.
![](https://www.scirp.org/html/12-7400912\708598e3-33ac-48df-a639-46d96e2016bf.jpg)
Now we give two lemmas which are important in our subsequent analysis. For system (2), by applying Itô’s formula to
, we immediately obtain the following result.
Lemma 1. Suppose
is given, then for system (2) with initial condition
, we have (see Equation (3) below)
Lemma 2 [4]. For system (2), (A, B, C) is stochastic stabilizable if and only if (iff) the following Lyapunov-type equation:
(4)
has a unique positive semidefinite solution
.
3. Problem Formulation
Fix
. Let
be the set of the
-valued, square integrable processes adapted with the σ-field generated by
, rt,
, respectively. Associated with each
is a quadratic cost functional
:
(5)
where
,
,
,
,
,
represents the expectation of the enclosed random variable,
is the solution to the following linear stochastic differential equation with statedependent noise and Markovian parameter jumps
(6)
In (5) and (6),
, etc. whenever
. Now we consider the following zero-sum differential game problem.
Problem 1. Given a system described by (6), find
, such that
![](https://www.scirp.org/html/12-7400912\c14a264e-96d4-439c-aa1a-b8cf86a9c7f3.jpg)
(3)
or equivalently,
![](https://www.scirp.org/html/12-7400912\3a572a4f-9985-46fe-a576-393a75773dcd.jpg)
![](https://www.scirp.org/html/12-7400912\c8e7b0c6-8a17-4582-809d-990956b194ad.jpg)
That is, there are two players for the differential game. Player 1 chooses control
to minimize the objective J, while Player 2 chooses control
to maximize J. Now we introduce a new type of coupled algebraic Riccati equations associated with the problem 1.
Definition 2. The following system of algebraic equations
(7)
with
![](https://www.scirp.org/html/12-7400912\66b862ee-701c-4b6a-bacb-77d6c4c22872.jpg)
is called a system of coupled algebraic Riccati equations (AREs).
In the next section, we will give our main results of this paper.
4. Main Results
In this section, we will show that the solvability of the AREs (7) is sufficient and necessary for the existence of the optimal control strategies of problem 1.
Theorem 1. Suppose
is stochastic stabilizable, problem 1 has a pair of solutions
with respect to the initial
, where
and
are the following feedback strategies
![](https://www.scirp.org/html/12-7400912\cbbcb133-37fe-497d-b94f-9dc0f1e4952d.jpg)
respectively, iff the AREs (7) admits a solution
. In this case
1) ![](https://www.scirp.org/html/12-7400912\f1fdf620-4f9a-4133-ad88-8868b68d8937.jpg)
![](https://www.scirp.org/html/12-7400912\13d33898-e46d-432c-888c-eca18e7273c1.jpg)
2) ![](https://www.scirp.org/html/12-7400912\097488cb-7d78-4286-a9d3-27726c1934fc.jpg)
Proof. Sufficiency: Let
be a solution of the AREs (7). According to lemma 1, we have
![](https://www.scirp.org/html/12-7400912\ac1d0b71-4a1e-4b48-9b4e-25eb8075b20c.jpg)
By a series of simple computation together with (7), the cost function
can be expressed as following
![](https://www.scirp.org/html/12-7400912\66188cf0-5de4-4d94-bdf3-049355a74f54.jpg)
Thus,
is minimized by the control strategies
and
with the optimal value being
.
Necessity: Let
![](https://www.scirp.org/html/12-7400912\aac0057f-92af-439b-83fd-c6a492067ad6.jpg)
![](https://www.scirp.org/html/12-7400912\88628c46-94b3-4ce7-b12f-0e621b3a0c4b.jpg)
be the optimal control strategies to problem 1. Implement
and
in (6), then
![](https://www.scirp.org/html/12-7400912\9848559c-c81e-48f7-9162-e36ad6cef833.jpg)
where
![](https://www.scirp.org/html/12-7400912\8b9419e0-4a37-45f0-8308-3bd9dcb4b507.jpg)
According to lemma 2 and the stochastic optimal control theory, we can easily obtain the conclusion that the AREs (7) admit a solution
.
So this completes the proof of Theorem 1.
Remark 1. It is interesting to see the specialization of our results in the deterministic case (i.e.
for
). The corresponding AREs are
(8)
which can be viewed as an extended results of [12].
Remark 2. From Theorem 1 we can see that the derivation of the optimal control strategies for this type of differential games is transformed into deriving the solutions to coupled algebraic Riccati Equations (7), this conclusion be coincident with the results presented in [4], etc.
Remark 3. For the coupled algebraic Riccati Equations (7) may be solved by a standard numerical integration such as LMI method [20], or iterative algorithm similar with the algorithm presented in [21].
5. Application to Stochastic H∞ Control
Now, we apply the above developed theory to solve some problems related to stochastic H∞ control. Firstly, we statement the stochastic H∞ control problem with Markovian jumps, then, we demonstrate the usefulness of the above developed theory in the study of stochastic H∞ control.
Consider the following controlled system:
(9)
with the cost functional
(10)
where
is a right continuous Markov process on a given probability space
and the state space
and the transition probability described by (1); here
is a standard one dimensional Wiener processes. In (9) and (10),
is the state vector,
is the input control and
is the vector of the exogenous disturbances.
The following definition is parallel with the definition 2 presented in [22].
Definition 3. Given disturbance attenuation level γ > 0, the state feedback strategy
is said to be an H∞ control for system (9), if for
,
,
, we have
1)
stabilizes system (9) internally, i.e. when
,
, the state trajectory of (9) with any initial value
satisfies
![](https://www.scirp.org/html/12-7400912\811c2f09-01ff-4dbf-ae7f-5cf474464538.jpg)
2)
with
![](https://www.scirp.org/html/12-7400912\1d4e2538-86c9-482e-bf0f-17d7bac44ef1.jpg)
where
for all
.
Generally speaking, the H∞ control problem described by (9) and (10) is to find a control
such that
for arbitrary exogenous disturbances
. As stated in [23], if we view
and
in the stochastic H∞ control problem as two control strategies of players P1 and P2 from the viewpoint of game theory, the H∞ control problem can be converted into solving a stochastic game problem, while
is in fact the saddle point of this game, e.g.
![](https://www.scirp.org/html/12-7400912\cd944cf9-2e89-43a3-a002-5fd060db3c21.jpg)
According to Theorem 1 discussed in Section 4, the following results can be obtained straightly:
Theorem 2. For system described by (9), the stochastic H∞ control admits a pair of solutions
with
,
, iff the following AREs
(11)
with
![](https://www.scirp.org/html/12-7400912\93f81454-50c2-4256-ab88-a6bc4268348e.jpg)
has a solution
, where
.
In this case,
is an H∞ control for system (9), and
is the corresponding worst case disturbance.
Illustrative example: Consider system (9) with the coefficients as follows:
![](https://www.scirp.org/html/12-7400912\4468bdad-ea14-45e1-823b-35fd8456109e.jpg)
![](https://www.scirp.org/html/12-7400912\e21e56f1-7cb7-42db-a704-599809a40208.jpg)
![](https://www.scirp.org/html/12-7400912\ddce7402-feb2-444b-aa05-166ec43fd2ad.jpg)
![](https://www.scirp.org/html/12-7400912\695abe0b-ef5d-40e7-9d82-3be61b8bf885.jpg)
![](https://www.scirp.org/html/12-7400912\0e58d548-9c55-490f-82e3-c3788e9973bb.jpg)
Set
, solving (11) via using the algorithm proposed in [21], we have
![](https://www.scirp.org/html/12-7400912\afc9c9dc-c8a3-496f-96ef-75399a2793ac.jpg)
Therefore, the H∞ control is given by
while
;
while
.
Remark 4. Although we restrict ourselves to single noise stochastic systems throughout the paper, our main theorem still hold for multiple multiplicative noise case. For example, if we replace (6) with
(12)
where
,
, being independent, one dimensional Wiener processes, then Theorem 1 still holds with AREs (7) replaced by
(13)
with
![](https://www.scirp.org/html/12-7400912\4fc9dda0-c96f-4e79-8c81-11b6cb4854cf.jpg)
6. Conclusion
This paper has investigated the linear quadratic zero-sum stochastic differential games with state-dependent noise and Markovian jump parameters in infinite-time horizon, sufficient and necessary conditions for the existence of the optimal control strategies have been obtained, which are expressed in a system of coupled algebraic Riccati equations. The results obtained in this paper extend the existing results of [19]. Throughout this paper, we only have focused on the zero-sum LQ differential games for stochastic systems, while we believe that nonzero-sum LQ differential games still have essential applications, and further studies on such kind of case should be continued.
7. Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grant No. 71171061, the Natural Science Foundation of Guangdong Province under Grant No. S2011010004970.