On Global Minimization for the Value Function in Affine Optimal Control Problems

Jinghao Zhu

School of Mathematical Sciences, Tongji University, Shanghai, China.

DOI: 10.4236/oalib.1110408                

Abstract

In this paper we provide a computational approach to a minimization problem for the value function associated with an affine optimal control problem subject to terminal-constraint with quadratic cost plus a potential, for a fixed final time and initial point. We study the global minimization problem of the value function over the attainable set and the regularity properties of the value function at a global minimizer point. On the other hand, in the global minimization of the value function, by an example in this computational approach, we also focus on a numerical method by Riccati matrix differential equations.

Keywords

Global Minimization, Value Function, Affine Optimal Control, Regularity, Extremal Flow, Riccati Differential Equation

1. Introduction and Preliminary

The regularity properties of a value function associated with an optimal control problem have been deeply studied in the last decades, extensively using tools from geometric control theory and nonsmooth analysis. It is well known that the value function associated with an optimal control problem fails to be everywhere differentiable. Actually, it is not even continuous.

In [1] the authors define a value function on the target state set associated with an affine optimal control problem and study its regularity properties. Under certain assumptions, in the attainable set the authors find out some subsets on which the value function is continuous and differentiable. In this paper, by a point of view in optimization methods, we first solve a global minimization problem for the value function defined by authors of the paper in [1] , and then we show that the value function is continuous and differentiable at a global minimizer point.

In deed, the value function defined in [1] takes its value by an optimal control with a terminal constraint. In this paper we solve a global minimization problem by the optimal control without terminal constraints.

The optimal control problem we deal with in this paper is the following:

$\left({\mathcal{P}}_y\right)\mathrm{<mo>\; :\; </mo>}\min C_y\left(u\right)\mathrm{<mo>\; :\; </mo>}={\displaystyle {\int }_0^T}\left[u^{{\rm T}}\left(t\right)u\left(t\right)-Q\left(x\left(t\right)\right)\right]\text{d}t\mathrm{,}$ (1.1)

such that

$\begin{array}{l}\stackrel{\dot{}}{x}\left(t\right)=f\left(x\left(t\right)\right)+g\left(x\left(t\right)\right)u\left(t\right),x\left(0\right)=a\in R^n,\\ x\left(T\right)=y\in R^n,u\left(t\right)\in R^m,x\left(t\right)\in R^n,t\in \left[0,T\right].\end{array}$

Here the function $Q\left(x\right)\mathrm{<mo>\; :\; </mo>}{R}^n\to R^1$ is smooth and bounded from above, the vector function $f\left(x\right)\mathrm{<mo>\; :\; </mo>}{R}^n\to R^n$ and matrix function $g\left(x\right)\mathrm{<mo>\; :\; </mo>}{R}^n\to R^{n\times m}$ are smooth on $R^n$ . $a\in R^n$ is a given initial condition and $y\in R^n$ is a given terminal condition. An admissible control $u(\cdot )$ is a vector function in $L^2\left(\left[\mathrm{0,}T\right]\mathrm{,}{R}^m\right)$ , such that the solution $x_u(\cdot )$ for the equation $\stackrel{\dot{}}{x}\left(t\right)=f\left(x\left(t\right)\right)+g\left(x\left(t\right)\right)u\left(t\right)$ satisfying $x\left(0\right)=a\in R^n,x\left(T\right)=y$ is well defined on the interval $\left[\mathrm{0,}T\right]$ . The set of admissible controls is denoted by ${\Omega }_a$ . For the initial state $a\in R^n$ , we define value function $S_a(\cdot )\mathrm{<mo>\; :\; </mo>}{R}^n\to R^1$ associated with $\left({\mathcal{P}}_y\right)$ to be a function of terminal states as follows: for $y\in R^n$ ,

$S_a\left(y\right)=\inf \left\{C_y\left(u\right)|u\in L^2\left(\left[\mathrm{0,}T\right]\mathrm{,}{R}^m\right)\mathrm{,}{x}_u\left(0\right)=a\mathrm{,}{x}_u\left(T\right)=y\right\}\mathrm{,}$ (1.2)

with the understanding that $S_a\left(y\right)=+\infty$ if y cannot be attained by admissible trajectories in time T. It is also clear that $S_a\left(y\right)$ is bounded below noting that $C_y\left(u\right)$ is bounded below due to the assumption that $Q\left(x\right)$ is bounded from above. Define the attainable set $A_a$ as the set of points on $R^n$ that can be reached from a by admissible trajectories in time interval $\left[\mathrm{0,}T\right]$ . We always assume $A_a\ne \varnothing$ . In [1] the authors have studied the problem concerning the regularity properties of the value function $S_a\left(y\right)$ on a dense subset of the attainable set.

In this paper we provide a computational approach to the following minimization problem:

$\underset{y\in A_a}{\min }{S}_a\left(y\right)\mathrm{.}$ (1.3)

Meanwhile, we study the regularity properties of the value function $S_a\left(y\right)$ at a global minimizer.

Remark 1.1. In practice, when $Q\left(x\right)\equiv 0$ , because the cost functional is a square of the $L^2$ norm of control, by the minimization of the value function one can find an optimal target for the system to work at minimal cost.

To deal with minimizing $S_a\left(y\right)$ over $A_a$ , we need to solve the following optimal control problem without terminal state restriction.

$\left(\mathcal{B}\right)\mathrm{<mo>\; :\; </mo>}\min J_a\left(u\right)\mathrm{<mo>\; :\; </mo>}={\displaystyle {\int }_0^T}\left[u^{{\rm T}}\left(t\right)u\left(t\right)-Q\left(x\left(t\right)\right)\right]\text{d}t\mathrm{,}$ (1.4)

such that

$\stackrel{\dot{}}{x}\left(t\right)=f\left(x\left(t\right)\right)+g\left(x\left(t\right)\right)u\left(t\right),x\left(0\right)=a\in R^n,u\left(t\right)\in R^m,x\left(t\right)\in R^n,t\in \left[0,T\right].$

Remark 1.2. We see that, for $y\in A_a$ and u which is an admissible control steering the affine system from a to y,

$C_y\left(u\right)=J_a\left(u\right)\mathrm{.}$ (1.5)

Remark 1.3. Suppose that $\left(\mathcal{B}\right)$ is solvable (i.e. there exists an optimal control of the problem $\left(\mathcal{B}\right)$ ). Let $\hat{u}$ be an optimal control of the problem $\left(\mathcal{B}\right)$ . It implies that $J_a\left(\hat{u}\right)<\infty$ . Let the optimal control $\hat{u}$ steer the primal affine system from a to a point $\hat{y}$ . By (1.2), (1.5), we have

$\begin{array}{c}{S}_a\left(\hat{y}\right)=\inf \left\{C_{\hat{y}}\left(u\right):u\in L^2\left(\left[0,T\right],R^m\right),x_u\left(0\right)=a,x_u\left(T\right)=\hat{y}\right\}\\ =\inf \left\{J_a\left(u\right):u\in L^2\left(\left[0,T\right],R^m\right),x_u\left(0\right)=a,x_u\left(T\right)=\hat{y}\right\}=J_a\left(\hat{u}\right)<\infty .\end{array}$ (1.6)

Since $S_a\left(y\right)$ is bounded below, we conclude that $S_a\left(y\right)\mathrm{,}y\in A_a$ is not constantly infinity. Thus if $\left(\mathcal{B}\right)$ is solvable, then by (1.6) we see that ${\inf }_{y\in A_a}{S}_a\left(y\right)$ is finite. Thus the minimization (1.3) is meaningful. Moreover, we will show that $\left(\mathcal{B}\right)$ is solvable only if the minimization problem ${\min }_{y\in A_a}{S}_a\left(y\right)$ is solvable.

In this paper we focus on the Hamilton-Jacobi-Bellman equation [2] [3] with respect to the problem $\left(\mathcal{B}\right)$ . We present parameterized convection-diffusion equations for a viscosity approximation [4] [5] [6] to the Hamilton-Jacobi-Bellman equation. Then a parameterized convection-diffusion equation yields a piecewise differentiable flow for an approximation of the optimal objective value of the problem $\left(\mathcal{B}\right)$ .

The rest of the paper is organized as follows. In section 2, we study the global minimization problem of the value function $S_a\left(y\right)$ over the attainable set. In section 3, two results are given on continuity and differentiability of the value function $S_a\left(y\right)$ at a global minimizer. The section 4 is devoted to present a computational approach to the minimization problem of the value function $S_a\left(y\right)$ . Two examples are presented to illustrate the computational approach for a linear quadratic optimal control problem under terminal constraint in section 5. In section 6, we derive an iteration of difference equations for implementing the computational approach to ${\min }_{y\in A_a}{S}_a\left(y\right)$ given in section 4. A conclusion is in the last section.

2. Minimizing the Value Function $S_a\left(y\right)$ over the Attainable Set

For the problem $\left(\mathcal{P}\right)$ , to minimize the value function $S_a\left(y\right)$ over the attainable set $A_a$ , we consider the optimal control problem $\left(\mathcal{B}\right)$ in (1.4). For the problem $\left(\mathcal{B}\right)$ , we define its value function as follows:

$V\left(x\right)=\inf \left\{J_x\left(u\right)|u\in L^2\left(\left[0,T\right],R^m\right),x\left(0\right)=x\right\}.$ (2.1)

Theorem 2.1. If $\left(\mathcal{B}\right)$ is solvable (i.e. there exists an optimal control of the problem $\left(\mathcal{B}\right)$ ), then

$V\left(a\right)=\underset{y\in A_a}{\min }{S}_a\left(y\right).$ (2.2)

Proof. Let $\hat{u}$ be an optimal control of the problem $\left(\mathcal{B}\right)$ , which steers the control affine system (in (1.4)) from a to $\hat{y}$ . By the definition of $S_a\left(\hat{y}\right)$ ((1.2)), noting that by (1.5) for all controls u steering the affine control system from a to $\hat{y}$ satisfying $C_{\hat{y}}\left(u\right)=J_a\left(u\right)\ge J_a\left(\hat{u}\right)=C_{\hat{y}}\left(\hat{u}\right)$ , we have

$\begin{array}{l}V\left(a\right)=J_a\left(\hat{u}\right)=C_{\hat{y}}\left(\hat{u}\right)\\ =\inf \left\{C_{\hat{y}}\left(u\right)|u\in L^2\left(\left[0,T\right],R^m\right),x_u\left(0\right)=a,x_u\left(T\right)=\hat{y}\right\}=S_a\left(\hat{y}\right).\end{array}$ (2.3)

Thus we have

$V\left(a\right)=S_a\left(\hat{y}\right),$ (2.4)

On the other hand, if there is a $y\in A_a$ such that $S_a\left(y\right)<S_a\left(\hat{y}\right)$ , then, noting (1.5) and (1.2), there is a control u which steers the affine system from a to y such that $J_a\left(u\right)=C_y\left(u\right)<S_a\left(y\right)+\left(S_a\left(\hat{y}\right)-S_a\left(y\right)\right)=S_a\left(\hat{y}\right)=V\left(a\right)$ , which leads to a contradiction to the fact that $\hat{u}$ is an optimal control of the problem $\left(\mathcal{B}\right)$ . Thus we have $S_a\left(\hat{y}\right)={\min }_{y\in A_a}{S}_a\left(y\right)$ . It follows from (2.4) that we have $V\left(a\right)={\min }_{y\in A_a}{S}_a\left(y\right)$ . The proof of Theorem 2.1 is completed.

Theorem 2.2. If a vector $\hat{y}\in R^n$ satisfies $S_a\left(\hat{y}\right)={\min }_{y\in A_a}{S}_a\left(y\right)$ and an admissible control $\hat{u}$ satisfies $C_{\hat{y}}\left(\hat{u}\right)=S_a\left(\hat{y}\right)$ , then $\left(\mathcal{B}\right)$ is solvable.

Proof. Since the minimization problem ${\min }_{y\in A_a}{S}_a\left(y\right)$ is solvable, we have a point $\hat{y}\in A_a$ such that

$S_a\left(\hat{y}\right)=\underset{y\in A_a}{\min }{S}_a\left(y\right).$ (2.5)

We need to show that an admissible control $\hat{u}$ steering the system from a to $\hat{y}$ such that $C_{\hat{y}}\left(\hat{u}\right)=S_a\left(\hat{y}\right)$ is an optimal control of $\left(\mathcal{B}\right)$ . Let $\bar{u}$ be an arbitrary admissible control steering the system from a to $\bar{y}$ . We will show that $J_a\left(\hat{u}\right)\le J_a\left(\bar{u}\right)$ . By (1.5), (2.5) and the definition of $S_a\left(\bar{y}\right)$ , also noting the assumption $C_{\hat{y}}\left(\hat{u}\right)=S_a\left(\hat{y}\right)$ , we have

$J_a\left(\hat{u}\right)=C_{\hat{y}}\left(\hat{u}\right)=S_a\left(\hat{y}\right)=\underset{y\in A_a}{\min }{S}_a\left(y\right)\le S_a\left(\bar{y}\right)\le C_{\bar{y}}\left(\bar{u}\right)=J_a\left(\bar{u}\right).$ (2.6)

Since $\bar{u}$ is an arbitrary admissible control, by (2.6) we see that $\hat{u}$ is an optimal control of $\left(\mathcal{B}\right)$ . The proof of Theorem 2.2 is completed.

Remark 2.1. In the proof of Theorem 2.1 and Theorem 2.2, a basic fact is the both of them have the same control system and the same cost for the same admissible control. By these two theorems, we know that the optimal control of the problem $\left(\mathcal{B}\right)$ steers the control system from the initial state point to the minimizer of the value function $S_a\left(y\right)$ of the problem $\left(\mathcal{P}\right)$ over the attainable set. On the other hand, if $\hat{y}$ is a minimizer of $S_a\left(y\right)$ over $A_a$ and a control $\hat{u}$ , steering the system from the initial point a to $\hat{y}$ , is an optimal control of $\left({\mathcal{P}}_{\hat{y}}\right)$ , then $\hat{u}$ is an optimal control of $\left(\mathcal{B}\right)$ .

3. The Regularity Properties of Value Function $S_a\left(y\right)$ at a Minimizer Point

In this section we assume that $\left(\mathcal{B}\right)$ is solvable. Let $\hat{u}$ be an optimal control of the problem $\left(\mathcal{B}\right)$ , which steers the primal affine system from the initial point a to $\hat{y}$ . By (2.4) we see that $\hat{y}$ is a minimizer point of $S_a\left(y\right)$ over $A_a$ .

We know that the end-point map $E_a^T$ is smooth [1] [7] [8] [9] [10] . Further we assume that the end-point map $E_a^T$ is an open map at the optimal control $\hat{u}$ in the following theorems.

Theorem 3.1. If the minimizer point $\hat{y}\in \mathrm{int}{A}_a$ and the end-point map is an open map at $\hat{u}$ , then the value function $S_a\left(y\right)$ of $\left(\mathcal{P}\right)$ is continuous at the minimizer $\hat{y}$ of $S_a\left(y\right)$ .

Proof. Since $\hat{u}$ is an optimal control of the problem $\left(\mathcal{B}\right)$ , which steers the primal affine system from a to $\hat{y}$ , we have $V\left(a\right)=S_a\left(\hat{y}\right)$ by (2.4). Since the end-point map is open at $\hat{u}$ , there are positive numbers $\eta$ and $\delta$ such that the image of $\left\{u:{\Vert u-\hat{u}\Vert }_{L_2}<\eta \right\}$ under the end-point map covers $\left\{y:{\Vert y-\hat{y}\Vert }_{R^n}<\delta \right\}$ . Given $\epsilon >0$ . We can also choose positive number $\eta$ so small that, when ${\Vert u-\hat{u}\Vert }_{L_2}<\eta$ , the inequality $\left|J_a\left(u\right)-J_a\left(\hat{u}\right)\right|<\epsilon$ holds(see Proposition 32 in [3] ). Since $\hat{y}\in \mathrm{int}{A}_a$ , we can choose positive number $\delta$ above so small that, for all $\Delta y\in \left\{\xi \in R^m:{\Vert \xi \Vert }_{R^n}\le \delta \right\}$ , $\hat{y}+\Delta y\in A_a$ and $V\left(a\right)=S_a\left(\hat{y}\right)\le S_a\left(\hat{y}+\Delta y\right)$ noting that $\hat{y}$ is the minimizer of $S_a\left(y\right)$ . In other words, now, for given $\epsilon >0$ , we have two positive numbers $\eta$ and $\delta$ such that:

1) the image of $\left\{u\mathrm{<mo>\; :\; </mo>}{\Vert u-\hat{u}\Vert }_{L_2}<\eta \right\}$ under the end-point map covers $\left\{y:{\Vert y-\hat{y}\Vert }_{R^n}<\delta \right\}$ ;

2) when ${\Vert u-\hat{u}\Vert }_{L_2}<\eta$ , the inequality $\left|J_a\left(u\right)-J_a\left(\hat{u}\right)\right|<\epsilon$ holds;

2) when ${\Vert \Delta y\Vert }_{R^n}\le \delta$ , we have $\hat{y}+\Delta y\in A_a$ and $V\left(a\right)=S_a\left(\hat{y}\right)\le S_a\left(\hat{y}+\Delta y\right)$ .

Now for ${\Vert \Delta y\Vert }_{R^n}\le \delta$ , in $\left\{u\mathrm{<mo>\; :\; </mo>}{\Vert u-\hat{u}\Vert }_{L_2}<\eta \right\}$ , we have an admissible control v steering the primal affine system from a to $\hat{y}+\Delta y\in A_a$ . By the definition of $S_a\left(y\right)$ , also noting that the relationship $C_y\left(u\right)=J_a\left(u\right)$ and $J_a\left(\hat{u}\right)=V\left(a\right)=S_a\left(\hat{y}\right)$ , we have the following inequalities:

$\begin{array}{c}V\left(a\right)=S_a\left(\hat{y}\right)\le S_a\left(\hat{y}+\Delta y\right)=\underset{u}{\inf }{C}_{\hat{y}+\Delta y}\left(u\right)\le C_{\hat{y}+\Delta y}\left(v\right)=J_a\left(v\right)\\ \le J_a\left(\hat{u}\right)+\epsilon =S_a\left(\hat{y}\right)+\epsilon .\end{array}$ (3.1)

Consequently,

$0\le S_a\left(\hat{y}+\Delta y\right)-S_a\left(\hat{y}\right)\le J_a\left(v\right)-J_a\left(\hat{u}\right)<\epsilon .$ (3.2)

The proof of Theorem 3.1 is completed.

Next we study the differentiability of the value function $S_a\left(y\right)$ at a minimizer point.

Theorem 3.2. If the minimizer point $\hat{y}\in \mathrm{int}{A}_a$ and the end-point map is an one-to-one open map at $\hat{u}$ , then the value function $S_a\left(y\right)$ of $\left(\mathcal{P}\right)$ is differentiable at the minimizer $\hat{y}$ of $S_a\left(y\right)$ .

Proof. Since all functions $Q\left(x\right)\mathrm{,}f\left(x\right)\mathrm{,}g\left(x\right)$ are smooth, we see that the functional $J_a\left(u\right)$ is F-differentiable at $u\in L^2\left(\left[\mathrm{0,}T\right]\mathrm{,}{R}^m\right)$ . Noting that the end-point map is an open map at $\hat{u}$ and $J_a\left(u\right)\ge J_a\left(\hat{u}\right)$ for all admissible control near $\hat{u}$ in $L^2\left(\left[\mathrm{0,}T\right]\mathrm{,}{R}^m\right)$ , we have

$\underset{{\Vert \Delta u\Vert }_{L_2}\to 0}{lim}\frac{J_a\left(\hat{u}+\Delta u\right)-J_a\left(\hat{u}\right)}{{\Vert \Delta u\Vert }_{L_2}}=0$ (3.3)

which also implies that, for a small positive r, there exists $C_r>0$ such that, as $0<{\Vert \Delta u\Vert }_{L_2}<r$ ,

$\Vert J_a\left(\hat{u}+\Delta u\right)-J_a\left(\hat{u}\right)\Vert \le C_r{\Vert \Delta u\Vert }_{L_2}\mathrm{.}$ (3.4)

Since the end-point map $E_a^T$ is an open map at $\hat{u}$ , there are positive numbers $\eta$ and $\delta$ such that

$E_a^T:\left\{u:{\Vert u-\hat{u}\Vert }_{L_2}<\eta \right\}\to \left\{y:{\Vert y-\hat{y}\Vert }_{R^n}<\delta \right\},$

is surjective. There exists a smooth right inverse $\Phi \mathrm{<mo>\; :\; </mo>}\left\{y:{\Vert y-\hat{y}\Vert }_{R^n}<\delta \right\}\to \left\{u\mathrm{<mo>\; :\; </mo>}{\Vert u-\hat{u}\Vert }_{L_2}<\eta \right\}$ such that $E_a^T\left(\Phi \left(y\right)\right)=y$ for every y in $\left\{y:{\Vert y-\hat{y}\Vert }_{R^n}<\delta \right\}$ . Fix local coordinates around the initial state a, and let $B_a\left(r\right)\subset \left\{y:{\Vert y-\hat{y}\Vert }_{R^n}<r\right\}$ and $B_{\hat{u}}\left(r\right)\subset \left\{u\mathrm{<mo>\; :\; </mo>}{\Vert u-\hat{u}\Vert }_{L_2}<r\right\}$ denote some balls of radius $r>0$ centered at a and $\hat{u}$ , respectively. Due to $\Phi$ being smooth, there exists positive numbers $R_{\Phi }<\eta$ and $C_{\Phi }>0$ such that, for every $0\le r\le R_{\Phi }$ ,

$B_a\left(C_{\Phi }r\right)\subset E_a^T\left(B_{\hat{u}}\left(r\right)\right)\mathrm{,}$ (3.5)

Pick any point $y\in \mathrm{int}{A}_a$ such that $\Vert \Delta y\Vert =C_{\Phi }r$ , with $0\le r\le R_{\Phi }$ . Then by (3.5) there exists $v\in B_{\hat{u}}\left(r\right)$ satisfying $v=\hat{u}+\Delta u$ and ${\Vert \Delta u\Vert }_{L_2}\le r$ such that $E_a^T\left(v\right)=y$ . Noting that

$\Vert \Delta y\Vert =C_{\Phi }r\ge C_{\Phi }{\Vert \Delta u\Vert }_{L_2},$ (3.6)

When ${\Vert \Delta u\Vert }_{L_2}\le r$ ( $\Delta u=v-\hat{u}$ ), by (3.1), (3.2), (3.4), (3.6), also noting that $J_a\left(\hat{u}\right)=V\left(a\right)=S_a\left(\hat{y}\right)$ , we have,

$0\le S_a\left(\hat{y}+\Delta y\right)-S_a\left(\hat{y}\right)\le J_a\left(\hat{u}+\Delta u\right)-J_a\left(\hat{u}\right)\le C_r{\Vert \Delta u\Vert }_{L_2}\le \frac{C_r}{C_{\Phi }}\Vert \Delta y\Vert$

and

$0\le \frac{S_a\left(\hat{y}+\Delta y\right)-S_a\left(\hat{y}\right)}{\Vert \Delta y\Vert }\le \frac{J_a\left(\hat{u}+\Delta u\right)-J_a\left(\hat{u}\right)}{\Vert \Delta y\Vert }\mathrm{<mo>\; =\; </mo>}\frac{J_a\left(\hat{u}+\Delta u\right)-J_a\left(\hat{u}\right)}{\Vert \Delta u\Vert }\frac{\Vert \Delta u\Vert }{\Vert \Delta y\Vert }\mathrm{.}$ (3.7)

By (3.5) we see that when $\Vert \Delta y\Vert \to 0$ , i.e. $C_{\Phi }r\to 0$ ( $C_{\Phi }>0$ is picked above), we have ${\Vert \Delta u\Vert }_{L_2}\le r\to 0$ . Thus, by (3.7), (3.6), (3.3) we have

$\begin{array}{c}0\le \frac{S_a\left(\hat{y}+\Delta y\right)-S_a\left(\hat{y}\right)}{\Vert \Delta y\Vert }\le \frac{J_a\left(\hat{u}+\Delta u\right)-J_a\left(\hat{u}\right)}{\Vert \Delta y\Vert }\\ =\frac{J_a\left(\hat{u}+\Delta u\right)-J_a\left(\hat{u}\right)}{\Vert \Delta u\Vert }\frac{\Vert \Delta u\Vert }{\Vert \Delta y\Vert }\le \frac{J_a\left(\hat{u}+\Delta u\right)-J_a\left(\hat{u}\right)}{\Vert \Delta u\Vert }{C}_{\Phi }^{-1}\to 0.\end{array}$

Consequently, the value function $S_a\left(y\right)$ of $\left(\mathcal{P}\right)$ is differentiable at the minimizer $\hat{y}$ of $S_a\left(y\right)$ and the corresponding derivative is zero. The proof of Theorem 3.1 is completed.

Remark 3.1. For both theorems above we assume $\hat{y}\in \mathrm{int}{A}_a$ noting that the candidate as a minimizer of the value function should be in the interior of the attainable set. Since $A_a$ is the image of ${\Omega }_a$ under the end-point map, it is reasonable to assume that the end-point map is an open map at the optimal control $\hat{u}$ . In [1] the authors assume the end-point map to be open and a submersion at an optimal control and consider the regularity properties of the value function on a subset ( $\subset \mathrm{int}{A}_a$ ). In this paper we only need to assume that $\hat{y}$ is an image of the end-point map which is open at the optimal control $\hat{u}$ . But we do not assume the end-point map to be a submersion.

4. An Extremal Flow for Minimizing the Value Function $S_a\left(y\right)$ of Affine Optimal Control Problems under Terminal Constraint

In this section, for the problem $\left(\mathcal{P}\right)$ , to minimize the value function $S_a\left(y\right)$ over the attainable set, we create a so called extremal flow for computing optimal value $V\left(a\right)$ of the problem $\left(\mathcal{B}\right)$ numerically. We focus on the following HJB equation for $\left(t\mathrm{,}x\right)\in R\times R^n$ ,

$v_t\left(t\mathrm{,}x\right)+v_x^{{\rm T}}\left(t\mathrm{,}x\right)f\left(x\right)-Q\left(x\right)+\underset{u\in R^m}{\inf }\left\{v_x^{{\rm T}}\left(t\mathrm{,}x\right)g\left(x\right)u+u^{{\rm T}}u\right\}=\mathrm{0,}$ (4.1)

with boundary condition $v\left(T\mathrm{,}x\right)=0$ .

By elementary optimization, for given $\left(t\mathrm{,}x\right)$ , we see that $u=\frac{-1}{2}{g}^{{\rm T}}\left(x\right)v_x\left(t\mathrm{,}x\right)$ is the unique minimizer of $v_x^{{\rm T}}\left(t\mathrm{,}x\right)g\left(x\right)u+u^{{\rm T}}u$ over $R^m$ . Then we have

$\underset{u\in R^m}{\inf }\left\{v_x^{{\rm T}}\left(t\mathrm{,}x\right)g\left(x\right)u+u^{\text{T}}u\right\}=-\frac{1}{4}{v}_x^{{\rm T}}\left(t\mathrm{,}x\right)g\left(x\right)g^{\text{T}}\left(x\right)v_x\left(t\mathrm{,}x\right)\mathrm{.}$ (4.2)

Thus we can rewrite the equation in (4.1) as the following PDE:

$v_t\left(t\mathrm{,}x\right)+v_x^{{\rm T}}\left(t\mathrm{,}x\right)f\left(x\right)-Q\left(x\right)-\frac{1}{4}{v}_x^{{\rm T}}\left(t\mathrm{,}x\right)g\left(x\right)g^{{\rm T}}\left(x\right)v_x\left(t\mathrm{,}x\right)=\mathrm{0,}v\left(T\mathrm{,}x\right)=0.$ (4.3)

Remark 4.1. By classical PDE theory [4] [5] [6] , we know that a viscosity solution of the PDE in (4.3) can be obtained from smooth solutions $v_{\epsilon }$ to the family of convection-diffusion equations

$\begin{array}{l}\epsilon {\Delta }_{x}v\left(t\mathrm{,}x\right)=v_t\left(t\mathrm{,}x\right)+v_x^{{\rm T}}\left(t\mathrm{,}x\right)f\left(x\right)-Q\left(x\right)\\ -\frac{1}{4}{v}_x^{{\rm T}}\left(t\mathrm{,}x\right)g\left(x\right)g^{{\rm T}}\left(x\right)v_x\left(t\mathrm{,}x\right)\mathrm{,}v\left(T\mathrm{,}x\right)=0.\end{array}$ (4.4)

(parameterized by $\epsilon >0$ ) in the limits as $\epsilon \to 0^{+}$ , where ${\Delta }_{x}v\left(t,x\right)={\displaystyle {\sum }_{k=1}^n}\frac{{\partial }^{2}v\left(t,x\right)}{\partial x_k^2}$ . The convergence of (4.4) to (4.3) as $\epsilon \to 0^{+}$ has been established by the classical works of PDE on the viscosity approximation (see [2] ). Thus in the equation (2.4) the diffusion term $\epsilon {\Delta }_{x}v\left(t\mathrm{,}x\right)$ converges to zero locally uniformly as $\epsilon \to 0^{+}$ .

For computing $V\left(a\right)$ numerically, we define extremal flows as follows.

Definition 4.1. Given $\epsilon >0$ , for a solution $v\left(t\mathrm{,}x\right)$ of the PDE problem (4.4), we call $x_{\epsilon }(\cdot )$ an extremal flow, if it is a solution of Cauchy initial value problem

$\{\begin{array}{l}{\stackrel{\dot{}}{x}}_{\epsilon }\left(t\right)=f\left(x_{\epsilon }\left(t\right)\right)-\frac{1}{2}g\left(x_{\epsilon }\left(t\right)\right)g^{\text{T}}\left(x_{\epsilon }\left(t\right)\right)v_x\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\mathrm{,}\\ t\in \left[\mathrm{0,1}\right]\mathrm{,}{x}_{\epsilon }\left(0\right)=a\mathrm{,}\end{array}$ (4.5)

with a feedback control

$u_{\epsilon }\left(t\right)\mathrm{<mo>\; :\; </mo>}=\frac{-1}{2}{g}^{\text{T}}\left(x_{\epsilon }\left(t\right)\right)v_x\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\mathrm{.}$ (4.6)

The following theorem claims that optimal value $V\left(a\right)$ of the problem $\left(\mathcal{B}\right)$ can be approximated by solving the equation (4.4) and (4.5).

Theorem 4.1. Let $v\left(t\mathrm{,}x\right)$ be denoted as a solution of PDE in (4.4) corresponding to a positive real number $\epsilon$ . Given $y\in R^n$ , if $x_{\epsilon }(\cdot )$ is an extremal flow related to $v\left(t\mathrm{,}x\right)$ and $u_{\epsilon }(\cdot )$ is the corresponding feedback control (see (4.5), (4.6)), then we have

$\underset{\epsilon \to 0^{+}}{lim}{J}_a\left(u_{\epsilon }\right)=V\left(a\right)=\underset{y\in A_a}{\min }{S}_a\left(y\right)\mathrm{.}$ (4.7)

Proof. By (4.5), (4.6), (4.7), we have

$\begin{array}{c}\epsilon {\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)=v_t\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)+v_x^{{\rm T}}\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)f\left(x_{\epsilon }\left(t\right)\right)-Q\left(x_{\epsilon }\left(t\right)\right)\\ -\frac{1}{4}{v}_x^{{\rm T}}\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)g\left(x_{\epsilon }\left(t\right)\right)g^{{\rm T}}\left(x_{\epsilon }\left(t\right)\right)v_x\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\\ =v_t\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)+v_x^{{\rm T}}\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)f\left(x_{\epsilon }\left(t\right)\right)-Q\left(x_{\epsilon }\left(t\right)\right)\\ -\frac{1}{2}{v}_x^{{\rm T}}\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)g\left(x_{\epsilon }\left(t\right)\right)g^{{\rm T}}\left(x_{\epsilon }\left(t\right)\right)v_x\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\\ +\frac{1}{4}{v}_x^{{\rm T}}\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)g\left(x_{\epsilon }\left(t\right)\right)g^{{\rm T}}\left(x_{\epsilon }\left(t\right)\right)v_x\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\\ =\frac{\text{d}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)}{\text{d}t}-Q\left(x_{\epsilon }\left(t\right)\right)+u_{\epsilon }^{{\rm T}}\left(t\right)u_{\epsilon }\left(t\right)a\mathrm{.}e\mathrm{.}t\in \left[\mathrm{0,}T\right]\mathrm{.}\end{array}$ (4.8)

Integrating the equality in (4.8) with respect to t from 0 to T, noting that $v\left(T\mathrm{,}{x}_{\epsilon }\left(T\right)\right)=0$ , we have

${\displaystyle {\int }_0^T}\epsilon {\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\text{d}t=J_a\left(u_{\epsilon }\right)-v\left(\mathrm{0,}a\right)\mathrm{.}$ (4.9)

On the other hand, if $x(\cdot )$ is another trajectory corresponding to an admissible control $u(\cdot )$ with respect to $\left(\mathcal{B}\right)$ , we have

$\underset{u\in R^m}{\inf }\left\{v_x^{{\rm T}}\left(t\mathrm{,}x\left(t\right)\right)g\left(x\left(t\right)\right)u+u^{{\rm T}}u\right\}\le v_x^{{\rm T}}\left(t\mathrm{,}x\left(t\right)\right)g\left(x\left(t\right)\right)u\left(t\right)+u{\left(t\right)}^{{\rm T}}u\left(t\right)\mathrm{,}$ (4.10)

then for each $t\in \left[\mathrm{0,}T\right]$ , we have

$\begin{array}{c}\epsilon {\Delta }_{x}v\left(t\mathrm{,}x\left(t\right)\right)=v_t\left(t\mathrm{,}x\left(t\right)\right)+v_x^{{\rm T}}\left(t\mathrm{,}x\left(t\right)\right)f\left(x\left(t\right)\right)-Q\left(x\left(t\right)\right)\\ -\frac{1}{4}{v}_x^{{\rm T}}\left(t\mathrm{,}x\left(t\right)\right)g\left(x\left(t\right)\right)g^{{\rm T}}\left(x\left(t\right)\right)v_x\left(t\mathrm{,}x\left(t\right)\right)\\ =v_t\left(t\mathrm{,}x\left(t\right)\right)+v_x^{{\rm T}}\left(t\mathrm{,}x\left(t\right)\right)f\left(x\left(t\right)\right)-Q\left(x\left(t\right)\right)\\ +\underset{u\in R^m}{\inf }\left\{v_x^{{\rm T}}\left(t\mathrm{,}x\left(t\right)\right)g\left(x\left(t\right)\right)u+u^{{\rm T}}u\right\}\end{array}$

$\begin{array}{c}\le v_t\left(t\mathrm{,}x\left(t\right)\right)+v_x^{{\rm T}}\left(t\mathrm{,}x\left(t\right)\right)f\left(x\left(t\right)\right)-Q\left(x\left(t\right)\right)\\ +v_x^{{\rm T}}\left(t\mathrm{,}x\left(t\right)\right)g\left(x\left(t\right)\right)u\left(t\right)+u^{{\rm T}}\left(t\right)u\left(t\right)\\ =v_t\left(t\mathrm{,}x\left(t\right)\right)+v_x^{{\rm T}}\left(t\mathrm{,}x\left(t\right)\right)\frac{\text{d}x\left(t\right)}{\text{d}t}-Q\left(x\left(t\right)\right)+u^{{\rm T}}\left(t\right)u\left(t\right)\mathrm{.}\end{array}$ (4.11)

Integrating the above inequality over $\left[\mathrm{0,}T\right]$ , noting $v\left(T\mathrm{,}x\left(T\right)\right)=0$ , we obtain

${\displaystyle {\int }_0^T}\epsilon {\Delta }_{x}v\left(t\mathrm{,}x\left(t\right)\right)\text{d}t\le {\displaystyle {\int }_0^T}\frac{\text{d}}{\text{d}t}v\left(t\mathrm{,}x\left(t\right)\right)\text{d}t+J_a\left(u\right)=-v\left(\mathrm{0,}a\right)+J_a\left(u\right)\mathrm{.}$ (4.12)

By (4.9) and (4.12), we have

$\begin{array}{c}{J}_a\left(u_{\epsilon }\right)={\displaystyle {\int }_0^T}\epsilon {\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\text{d}t+v\left(\mathrm{0,}a\right)\\ \le J_a\left(u\right)+{\displaystyle {\int }_0^T}\epsilon {\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\text{d}t-{\displaystyle {\int }_0^T}\epsilon {\Delta }_{x}v\left(t\mathrm{,}x\left(t\right)\right)\text{d}t\mathrm{.}\end{array}$ (4.13)

Let $\hat{u}(\cdot )$ be an optimal control and $\hat{x}(\cdot )$ be the corresponding optimal trajectory with respect to $\left(\mathcal{B}\right)$ . Using (4.13) for the optimal pair $\left(\hat{x}(\cdot ),\hat{u}(\cdot )\right)$ , noting the fact $V\left(a\right)=J_a\left(\hat{u}\right)$ , we have

$\begin{array}{c}{J}_a\left(u_{\epsilon }\right)={\displaystyle {\int }_0^T}\epsilon {\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\text{d}t+v\left(\mathrm{0,}a\right)\\ \le V\left(a\right)+{\displaystyle {\int }_0^T}\epsilon {\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\text{d}t-{\displaystyle {\int }_0^T}\epsilon {\Delta }_{x}v\left(t\mathrm{,}\hat{x}\left(t\right)\right)\text{d}t\mathrm{.}\end{array}$ (4.14)

By (4.14), noting that $u_{\epsilon }\left(\mathrm{.}\right)$ is an admissible feedback control, we have

$V\left(a\right)\le J_a\left(u_{\epsilon }\right)\le V\left(a\right)+{\displaystyle {\int }_0^T}\left[\epsilon {\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)-\epsilon {\Delta }_{x}v\left(t\mathrm{,}\hat{x}\left(t\right)\right)\right]\text{d}t$

which yields

$0\le J_a\left(u_{\epsilon }\right)-V\left(a\right)\le {\displaystyle {\int }_0^T}\left[\epsilon {\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)-\epsilon {\Delta }_{x}v\left(t\mathrm{,}\hat{x}\left(t\right)\right)\right]\text{d}t\mathrm{.}$ (4.15)

Noting that, in the Equation (4.4), the diffusion term $\epsilon {\Delta }_{x}v\left(t\mathrm{,}x\right)$ converges to zero locally uniformly as $\epsilon \to 0^{+}$ (see Remark 4.1), we can show that, on a compact set $\Omega \left(\subset R\left(a\right)\right)$ which contains the optimal trajectory $\left\{\hat{x}(\cdot )\right\}$ and the flow $\left\{x_{\epsilon }(\cdot )\right\}$ ,

$\underset{\epsilon \to 0^{+}}{lim}\underset{\left(t\mathrm{,}x\right)\in \left[\mathrm{0,1}\right]\times \Omega }{\sup }\left|\epsilon {\Delta }_{x}v\left(t\mathrm{,}x\right)\right|=0.$ (4.16)

Thus, by Lebesgue Convergence Theorem, we have

$\underset{\epsilon \to 0^{+}}{lim}{\displaystyle {\int }_0^T}\left[\epsilon {\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\right]\text{d}t=\mathrm{0,}$ (4.17)

and

$\underset{\epsilon \to 0^{+}}{lim}{\displaystyle {\int }_0^T}\left[\epsilon {\Delta }_{x}v\left(t\mathrm{,}\hat{x}\left(t\right)\right)\right]\text{d}t=0.$ (4.18)

Thus by (4.15), (4.16), (4.17), (4.18) we have, as $\epsilon \to 0^{+}$ ,

$0\le J_a\left(u_{\epsilon }\right)-V\left(a\right)\le {\displaystyle {\int }_0^T}\left[\epsilon {\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)-\epsilon {\Delta }_{x}v\left(t\mathrm{,}\hat{x}\left(t\right)\right)\right]\text{d}t\to 0.$ (4.19)

It follows from Theorem 2.1 that

$\underset{\epsilon \to 0^{+}}{\lim }{J}_a\left(u_{\epsilon }\right)=V\left(a\right)=\underset{y\in A_a}{\min }{S}_a\left(y\right).$

The theorem has been proved.

In the proof of Theorem 4.1, replacing ${\Delta }_{x}v\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)$ with zero, the extremal flow will not depend on $\epsilon$ . By the same way we can prove the following result.

Theorem 4.2. If $v\left(t\mathrm{,}x\right)$ satisfies the PDE

$v_t\left(t\mathrm{,}x\right)+v_x^{{\rm T}}\left(t\mathrm{,}x\right)f\left(x\right)-Q\left(x\right)-\frac{1}{4}{v}_x^{{\rm T}}\left(t\mathrm{,}x\right)g\left(x\right)g^{{\rm T}}\left(x\right)v_x\left(t\mathrm{,}x\right)=\mathrm{0,}v\left(T\mathrm{,}x\right)=\mathrm{0,}$ (4.20)

and $\hat{x}(\cdot )$ is the solution to Cauchy initial value problem

$\{\begin{array}{l}\stackrel{\dot{}}{x}\left(t\right)=f\left(x\left(t\right)\right)-\frac{1}{2}g\left(x\left(t\right)\right)g^{\text{T}}\left(x\left(t\right)\right)v_x\left(t,x\left(t\right)\right),\\ t\in \left[0,1\right],x\left(0\right)=a,\end{array}$ (4.21)

with a feedback control

$\hat{u}\left(t\right)=\frac{-1}{2}{g}^{\text{T}}\left(\hat{x}\left(t\right)\right)v_x\left(t\mathrm{,}\hat{x}\left(t\right)\right)\mathrm{,}$ (4.22)

then we have

$J_a\left(\hat{u}\right)=V\left(a\right)=\underset{y\in A_a}{\min }{S}_a\left(y\right)\mathrm{.}$ (4.23)

5. Examples on Linear-Quadratic Optimal Control Problem under Terminal State Constraint for Illustrating Theorem 4.2

Example 5.1. We consider the following linear-quadratic optimal control problem with terminal state constraint:

$\left({\mathcal{P}}_{\mathcal{L}}\right)\mathrm{<mo>\; :\; </mo>}\{\begin{array}{l}\inf C_y\left(u\right)\mathrm{<mo>\; :\; </mo>}={\displaystyle {\int }_0^T}\left[u^{\text{T}}\left(t\right)u\left(t\right)+x^{\text{T}}\left(t\right)x\left(t\right)\right]\text{d}t\\ \stackrel{\dot{}}{x}\left(t\right)=Ax\left(t\right)+Bu\left(t\right)\mathrm{,}\\ x\left(0\right)=a\in R^n\mathrm{,}x\left(T\right)=y\in R^n\mathrm{,}\\ u\left(t\right)\in R^m\mathrm{,}x\left(t\right)\in R^n\mathrm{,}t\in \left[\mathrm{0,}T\right]\mathrm{,}\end{array}$ (5.1)

and the corresponding linear-quadratic optimal control problem:

$\left({\mathcal{B}}_{\mathcal{L}}\right):\{\begin{array}{l}\inf J_a\left(u\right):={\displaystyle {\int }_0^T}\left[u^{\text{T}}\left(t\right)u\left(t\right)+x^{\text{T}}\left(t\right)x\left(t\right)\right]\text{d}t\\ \stackrel{\dot{}}{x}\left(t\right)=Ax\left(t\right)+Bu\left(t\right),\\ x\left(0\right)=a\in R^n,\\ u\left(t\right)\in R^m,x\left(t\right)\in R^n,t\in \left[0,T\right].\end{array}$ (5.2)

where in (5.1) and (5.2), $A\in R^{n\times n}$ and $B\in R^{n\times m}$ .

By classical LQ optimal control theory [3] , there exists an absolutely continuous symmetric matrix function $S\left(t\right)$ , defined for $t\in \left[\mathrm{0,}T\right]$ , which satisfies the matrix Riccati Differential Equation on $\left[\mathrm{0,}T\right]$ :

$\stackrel{\dot{}}{S}+SA+A^{{\rm T}}S+I-SB{B}^{{\rm T}}S=\mathrm{0,}S\left(T\right)=0.$ (5.3)

Moreover the LQ optimal control problem $\left({\mathcal{B}}_{\mathcal{L}}\right)$ is solvable.

To use Theorem 4.2, we see that the function $v\left(t\mathrm{,}x\right)=x^{\text{T}}S\left(t\right)x$ satisfies the following HJB equation

$v_t\left(t\mathrm{,}x\right)+v_x^{{\rm T}}\left(t\mathrm{,}x\right)Ax+x^{\text{T}}x-\frac{1}{4}{v}_x^{{\rm T}}\left(t\mathrm{,}x\right)B{B}^{{\rm T}}{v}_x\left(t\mathrm{,}x\right)\mathrm{,}v\left(T\mathrm{,}x\right)=0.$ (5.4)

For $v\left(t\mathrm{,}x\right)=x^{\text{T}}S\left(t\right)x$ , we have

$v_x\left(t\mathrm{,}x\right)=2S\left(t\right)x\mathrm{.}$

For

$u=\frac{-1}{2}{B}^{{\rm T}}\left(x\right)v_x\left(t\mathrm{,}x\right)=-B^{{\rm T}}S\left(t\right)x\mathrm{,}$

to find an extremal flow $\hat{x}(\cdot )$ , we solve the Cauchy initial value problem

$\{\begin{array}{l}\stackrel{\dot{}}{x}=Ax+Bu=\left(A-B{B}^{\text{T}}S\left(t\right)\right)x\mathrm{,}\\ t\in \left[\mathrm{0,}T\right]\mathrm{,}x\left(0\right)=a\mathrm{,}\end{array}$ (5.5)

Let $\Phi \left(t\mathrm{,0}\right)$ be the solution of the matrix differential equation

$\stackrel{\dot{}}{X}\left(t\right)=\left(A-B{B}^{\text{T}}S\left(t\right)\right)X\left(t\right)\mathrm{,}X\left(0\right)=I\mathrm{.}$

By classical ordinary differential equation theory, $\Phi \left(t\mathrm{,0}\right)$ is the fundamental solution associated to $A-B{B}^{\text{T}}S(\cdot )$ and the solution of (5.5) is given by

$\hat{x}\left(t\right)=\Phi \left(t\mathrm{,0}\right)a\mathrm{.}$

Then we have a feedback control

$\hat{u}\left(t\right)\mathrm{<mo>\; :\; </mo>}=-B^{\text{T}}S\left(t\right)\hat{x}\left(t\right)\mathrm{.}$ (5.6)

By Theorem 4.2 we have

$\underset{y\in A_a}{\min }{S}_a\left(y\right)=J_a\left(\hat{u}\right)={\displaystyle {\int }_0^T}\left[{\hat{u}}^{\text{T}}\left(t\right)\hat{u}\left(t\right)+{\hat{x}}^{\text{T}}\left(t\right)\hat{x}\left(t\right)\right]\text{d}t\mathrm{.}$ (5.7)

Remark 5.1. We will provide an approximation approach to compute ${\min }_{y\in A_a}{S}_a\left(y\right)$ in the example as follows.

Example 5.2. We consider following linear-quadratic optimal control problem with terminal state constraint:

$\{\begin{array}{l}\inf C_y\left(u\right):={\displaystyle {\int }_0^1}\left[u^2\left(t\right)+x^2\left(t\right)\right]\text{d}t\\ \stackrel{\dot{}}{x}\left(t\right)=u\left(t\right),\\ x\left(0\right)=a\in R^1,x\left(T\right)=y\in R^1,\\ u\left(t\right)\in R^1,x\left(t\right)\in R^1,t\in \left[0,1\right],\end{array}$ (5.8)

and the corresponding linear-quadratic optimal control problem:

$\{\begin{array}{l}\inf J_a(u):={\displaystyle {\int }_0^1}\left[u^2\left(t\right)+x^2\left(t\right)\right]\text{d}t\\ \stackrel{\dot{}}{x}\left(t\right)=u\left(t\right),\\ x\left(0\right)=a\in R^1,\\ u\left(t\right)\in R^1,x\left(t\right)\in R^1,t\in \left[0,1\right].\end{array}$ (5.9)

Similar as the PDE in (5.4), the HJB equation for this example is

$v_t\left(t,x\right)+x^2-\frac{1}{4}{v}_x^2\left(t,x\right)=0,v\left(1,x\right)=0,t\in \left[0,1\right],x\in R^1.$ (5.10)

Similar as Example 5.1 we have

$v\left(t,x\right)=S\left(t\right)x^2,v_x\left(t,x\right)=2S\left(t\right)x,$ (5.11)

where $S\left(t\right)$ satisfies Riccati Differential Equation:

$\stackrel{\dot{}}{S}+1-S^2=0,S\left(1\right)=0,t\in \left[0,1\right],S\left(t\right)\in R^1.$ (5.12)

We solve the Cauchy initial value problem

$\{\begin{array}{l}\stackrel{\dot{}}{x}=-S\left(t\right)x,\\ t\in \left[0,1\right],x\left(0\right)=a.\end{array}$ (5.13)

to find an extremal flow $\hat{x}(\cdot )$ and the feedback control

$\hat{u}=\frac{-1}{2}{v}_x\left(t,\hat{x}\right)=-S\left(t\right)\hat{x},$

By Theorem 4.2, for this example we have

$\underset{y\in A_a}{\min }{S}_a\left(y\right)=J_a\left(\hat{u}\right)={\displaystyle {\int }_0^1}\left(S^2\left(t\right)+1\right){\hat{x}}^2\left(t\right)\text{d}t.$ (5.14)

For a numerical approach to compute ${\min }_{y\in A_a}{S}_a\left(y\right)$ , in the following we present a sequence of flows to converge the extremal flow and the corresponding feedback control for an approximation of ${\min }_{y\in A_a}{S}_a\left(y\right)$ .

By the iteration method given in [11] , we have a sequence of differentiable functions $\left\{S_i(\cdot )\mathrm{,}i=\mathrm{1,2},\cdots \right\}$ , satisfying

${\stackrel{\dot{}}{S}}_i+1-{\left(S_{i-1}\right)}^2=0,S_i\left(1\right)=0,t\in \left[0,1\right],$ (5.15)

$S_0\left(t\right)=1-t\mathrm{,}t\in \left[\mathrm{0,1}\right]\mathrm{,}$ (5.16)

such that $\left\{S_i(\cdot )\right\}$ converges uniformly to the solution $S\left(t\right)$ of the equation in (5.12). Then we have a sequence of $\left\{x_i(\cdot )\mathrm{,}i=\mathrm{1,2},\cdots \right\}$ . Such that for $i=1,2,\cdots$ ,

${\stackrel{\dot{}}{x}}_i\left(t\right)=u_i\mathrm{<mo>\; :\; </mo>}=-S_i\left(t\right)x_i\left(t\right)\mathrm{,}t\in \left[\mathrm{0,1}\right]\mathrm{,}{x}_i\left(0\right)=a\mathrm{.}$ (5.17)

Noting that $S\left(t\right)$ is bounded and $\left\{S_i(\cdot )\right\}$ converges uniformly to the solution $S\left(t\right)$ , we see that $\left\{S_i(\cdot )\right\}$ is uniformly bounded. Therefore by Bellman-Gronwall inequality we can show that $\left\{x_i(\cdot )\right\}$ is uniformly bounded. Further, we show that $\left\{x_i(\cdot )\right\}$ converges uniformly to $\hat{x}(\cdot )$ which is the solution of the equation in (5.13) as follows. We have, for $t\in \left[\mathrm{0,1}\right]$ and $i=\mathrm{1,2},\cdots$ ,

$\begin{array}{l}{x}_i\left(t\right)-\hat{x}\left(t\right)={\displaystyle {\int }_0^t}\left[S_i\left(s\right)x_i\left(s\right)-S\left(s\right)\hat{x}\left(s\right)\right]\text{d}s\\ ={\displaystyle {\int }_0^t}\left[\left(S_i\left(s\right)x_i\left(s\right)-S_i\left(s\right)\hat{x}\left(s\right)\right)+\left(S_i\left(s\right)\hat{x}\left(s\right)-S\left(s\right)\hat{x}\left(s\right)\right)\right]\text{d}s\mathrm{.}\end{array}$

Then for $t\in \left[\mathrm{0,1}\right]$ and $i=\mathrm{1,2},\cdots$ ,, by general integral estimation we have,

$\Vert x_i\left(t\right)-\hat{x}\left(t\right)\Vert \le {\displaystyle {\int }_0^1}\Vert S_i\left(s\right)-S\left(s\right)\Vert \Vert \hat{x}\left(s\right)\Vert \text{d}s+{\displaystyle {\int }_0^t}\Vert S_i\left(s\right)\Vert \Vert x_i\left(s\right)-\hat{x}\left(s\right)\Vert \text{d}s\mathrm{.}$

By Bellman-Gronwall inequality, we have,

$\Vert x_i\left(t\right)-\hat{x}\left(t\right)\Vert \le \left({\displaystyle {\int }_0^1}\Vert S_i\left(s\right)-S\left(s\right)\Vert \Vert \hat{x}\left(s\right)\Vert \text{d}s\right){\text{e}}^{{\displaystyle {\int }_0^t}\Vert S_i\left(s\right)\Vert \Vert x_i\left(s\right)-\hat{x}\left(s\right)\Vert \text{d}s}$ (5.18)

Noting that $\left\{\Vert x_i(\cdot )\Vert \right\}\mathrm{,}\Vert S_i(\cdot )\Vert \mathrm{,}i=\mathrm{0,1,2,}\cdots$ are uniformly bounded $\left[\mathrm{0,1}\right]$ , $\hat{x}(\cdot )$ is bounded on $\left[\mathrm{0,1}\right]$ and $\Vert S_i(\cdot )-S(\cdot )\Vert$ converges uniformly to zero, by (5.18) we have shown that $x_i\left(t\right)\mathrm{,}i=\mathrm{1,2,}\cdots$ converges uniformly to $\hat{x}\left(t\right)$ on $\left\{t\in \left[\mathrm{0,1}\right]\right\}$ . Meanwhile, if define $u_i\left(t\right)=-S_i\left(t\right)x_i\left(t\right)$ , then $u_i(\cdot )$ converges uniformly to the feedback control $\hat{u}\left(t\right)=-S\left(t\right)\hat{x}\left(t\right)$ on $\left[\mathrm{0,1}\right]$ . Noting that

$J_a\left(u_i\right)={\displaystyle {\int }_0^1}\left(S_i^2\left(t\right)+1\right)x_i^2\left(t\right)\text{d}t\mathrm{,}$

we have

$\underset{i\to \infty }{lim}{J}_a\left(u_i\right)=J_a\left(\hat{u}\right)=\underset{y\in A_a}{\min }{S}_a\left(y\right)\mathrm{.}$ (5.19)

6. A Numerical Approach to Compute $mi{n}_{y\in A_a}{S}_a\left(y\right)$ for the Sake of General Affine Optimal Control Problem $(\; P\; )$

In this section, we present an iteration of difference equations to illustrate the approximation of ${\min }_{y\in A_a}{S}_a\left(y\right)$ given by Theorem 4.1 concerning the affine optimal control problem $\left(\mathcal{P}\right)$ .

Given $\epsilon >0$ . Let $x_{\epsilon }\left(\mathrm{.}\right)$ satisfy

$\{\begin{array}{l}{\stackrel{\dot{}}{x}}_{\epsilon }\left(t\right)=f\left(x_{\epsilon }\left(t\right)\right)-\frac{1}{2}g\left(x_{\epsilon }\left(t\right)\right)g^{\text{T}}\left(x_{\epsilon }\left(t\right)\right)v_x\left(t,x_{\epsilon }\left(t\right)\right),\\ t\in \left[0,1\right],x_{\epsilon }\left(0\right)=a.\end{array}$ (6.1)

with a feedback control

$u_{\epsilon }\left(t\right)\mathrm{<mo>\; :\; </mo>}=\frac{-1}{2}{g}^{\text{T}}\left(x_{\epsilon }\left(t\right)\right)v_x\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)\mathrm{.}$ (6.2)

By the result in Theorem 4.1, we need to compute $J_a\left(u_{\epsilon }\right)$ . Consider the function

$H_{\epsilon }\left(t\right)\mathrm{<mo>\; :\; </mo>}=\frac{1}{4}{v}_x^{{\rm T}}\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)g\left(x_{\epsilon }\left(t\right)\right)g^{{\rm T}}\left(x_{\epsilon }\left(t\right)\right)v_x\left(t\mathrm{,}{x}_{\epsilon }\left(t\right)\right)-Q\left(x_{\epsilon }\left(t\right)\right)\mathrm{.}$ (6.3)

Noting the expression of the cost functional of $\left(\mathcal{B}\right)$ in (1.4), we will estimate

$J_a\left(u_{\epsilon }\right)={\displaystyle {\int }_0^T}{H}_{\epsilon }\left(t\right)\text{d}t\mathrm{.}$ (6.4)

Let $L=\left[\frac{1}{{\epsilon }^2}\right]+1$ . Dividing the time interval $\left[\mathrm{0,}T\right]$ evenly into L small intervals $\left[t_i\mathrm{,}{t}_{i+1}\right]\mathrm{,}i=\mathrm{0,1,}\cdots \mathrm{,}L-1$ , with $t_0=0$ , $t_L=T$ . Let $\tau :=\frac{T}{L}=t_{i+1}-t_i,i=0,1,\cdots ,L-1$ . Define

$F_L\left(h_{\epsilon }\right)=\tau \left[\frac{1}{2}{H}_{\epsilon }\left(0\right)+{\displaystyle \underset{i=1}{\overset{L-1}{\sum }}}{H}_{\epsilon }\left(i\tau \right)+\frac{1}{2}{H}_{\epsilon }\left(T\right)\right].$ (6.5)

By classical numerical mathematics [12] , we have

$\underset{L\to \infty }{lim}{F}_L\left(h_{\epsilon }\right)=J_a\left(u_{\epsilon }\right)\mathrm{.}$ (6.6)

By (6.3), for computing $F_L\left(h_{\epsilon }\right)$ , we need to estimate $x_{\epsilon }\left(t_i\right)\mathrm{,}i=\mathrm{0,1,}\cdots \mathrm{,}L-1$ by following difference equation

$\{\begin{array}{l}{x}_{\epsilon }\left(t_{i+1}\right)-x_{\epsilon }\left(t_i\right)=\tau \left[f\left(x_{\epsilon }\left(t_i\right)\right)-\frac{1}{2}g\left(x_{\epsilon }\left(t_i\right)\right)g^{\text{T}}\left(x_{\epsilon }\left(t_i\right)\right)v_x\left(t_i,x_{\epsilon }\left(t_i\right)\right)\right],\\ t_i\in \left[0,1\right],x_{\epsilon }\left(t_0\right)=a.\end{array}$ (6.7)

with appearing $v_x\left(t_i\mathrm{,}{x}_{\epsilon }\left(t_i\right)\right)\mathrm{,}i=\mathrm{0,1,}\cdots \mathrm{,}L-1$ . Therefore, next for a given $\left(t\mathrm{,}x\right)\in \left[\mathrm{0,}T\right]\times R^2$ we need to estimate $v_x\left(t\mathrm{,}x\right)$ . We present an iteration of difference equations as follows to compute $v_x\left(t\mathrm{,}x\right)$ numerically.

On $\left[\mathrm{0,}T\right]\times R^2$ we define

$h\left(t\mathrm{,}x\right)\mathrm{<mo>\; :\; </mo>}=\frac{-1}{2}{g}^{\text{T}}\left(x\right)v_x\left(t\mathrm{,}x\right)\mathrm{.}$ (6.8)

The equation in (4.4) can be rewritten as

$\begin{array}{c}\epsilon {\Delta }_{x}v\left(t\mathrm{,}x\right)=v_t\left(t\mathrm{,}x\right)+v_x^{\text{T}}\left(t\mathrm{,}x\right)f\left(x\right)+v_x^{\text{T}}\left(t\mathrm{,}x\right)g\left(x\right)h\left(t\mathrm{,}x\right)\\ -Q\left(x\right)+h{\left(t\mathrm{,}x\right)}^{\text{T}}h\left(t\mathrm{,}x\right)\mathrm{,}\end{array}$ (6.9)

with the boundary condition $v\left(\mathrm{1,}x\right)=0$ .

For simplicity, we restrict our discussion to the state in $R^2$ as follows. For a given $\left(t\mathrm{,}x\right)\in \left[\mathrm{0,}T\right]\times R^2$ , along the direction $h\left(t\mathrm{,}x\right)$ , we have a linear function $\bar{x}\left(s\right)=x+\left[f\left(x\right)+g\left(x\right)h\left(t\mathrm{,}x\right)\right]s\mathrm{,}s\ge \mathrm{0,}\bar{x}\left(0\right)=x$ such that

$\begin{array}{l}\epsilon {\Delta }_{x}v\left(t\mathrm{,}x\right)\\ =\underset{s\to 0^{+}}{\lim }\left[v_t\left(t+s\mathrm{,}\bar{x}\left(s\right)\right)+v_x^{\text{T}}\left(t+s\mathrm{,}\bar{x}\left(s\right)\right)\frac{\text{d}\bar{x}\left(s\right)}{\text{d}s}-Q\left(\bar{x}\left(s\right)\right)+h{\left(s\mathrm{,}\bar{x}\left(s\right)\right)}^{\text{T}}h\left(s\mathrm{,}\bar{x}\left(s\right)\right)\right]\\ ={\frac{\text{d}v\left(t+s\mathrm{,}\bar{x}\left(s\right)\right)}{\text{d}s}|}_{s=0}-Q\left(x\right)+h{\left(t\mathrm{,}x\right)}^{\text{T}}h\left(t\mathrm{,}x\right)\mathrm{,}\end{array}$ (6.10)

Noting that $\frac{\text{d}\bar{x}\left(s\right)}{\text{d}s}\equiv f\left(x\right)+g\left(x\right)h\left(t\mathrm{,}x\right)$ . Then when $0<\Delta t\ll 1$ , at $\left(t\mathrm{,}x\right)$ , we have the following difference equation:

$\epsilon {\Delta }_{x}v\left(t+\Delta t\mathrm{,}x\right)-\frac{v\left(t+\Delta t\mathrm{,}\bar{x}\left(\Delta t\right)\right)-v\left(t\mathrm{,}x\right)}{\Delta t}\approx -Q\left(x\right)+h{\left(t\mathrm{,}x\right)}^{\text{T}}h\left(t\mathrm{,}x\right)\mathrm{,}$ (6.11)

$\bar{x}\left(\Delta t\right)=x+\left[f\left(x\right)+g\left(x\right)h\left(t\mathrm{,}x\right)\right]\Delta t\mathrm{.}$ (6.12)

Write the state $x=\left(x_1\mathrm{,}{x}_2\right)$ in $R^2$ . Denote $\left[x_1-\mathrm{1,}{x}_1+1\right]\times \left[x_2-\mathrm{1,}{x}_2+1\right]$ by $\Omega$ . We focus on difference equations on $\left[t\mathrm{,1}\right]\times \Omega$ . For positive integers: $N\mathrm{,}L$ , let $i\mathrm{,}j\in \left\{-N\mathrm{,}\cdots \mathrm{,}-\mathrm{1,0,1,}\cdots \mathrm{,}N\right\}$ and $k\in \left\{\mathrm{0,1,}\cdots \mathrm{,}L\right\}$ . For $\zeta =\frac{2}{N}\mathrm{,}\tau =\frac{1}{L}$ , denote $x_{i\mathrm{,}j}=\left(x_1+i\zeta \mathrm{,}{x}_2+j\zeta \right)$ and $s_k=1-k\tau \left(1-t\right)$ . Let $w_{i\mathrm{,}j}^k$ denote the approximate grid value of the solution $v\left(s_k\mathrm{,}{x}_{i\mathrm{,}j}\right)$ and

$\delta w_{i\mathrm{,}j}^k\mathrm{<mo>\; :\; </mo>}={\left(2\zeta \right)}^{-1}\left(w_{i+\mathrm{1,}j}^k-w_{i-\mathrm{1,}j}^k\mathrm{,}{w}_{i\mathrm{,}j+1}^k-w_{i\mathrm{,}j-1}^k\right)\approx v_x\left(s_k\mathrm{,}{x}_{i\mathrm{,}j}\right)\mathrm{,}$ (6.13)

${\delta }^2{w}_{i\mathrm{,}j}^k\mathrm{<mo>\; :\; </mo>}={\zeta }^{-2}\left(w_{i+\mathrm{1,}j}^k+w_{i-\mathrm{1,}j}^k-4w_{i\mathrm{,}j}^k+w_{i\mathrm{,}j+1}^k+w_{i\mathrm{,}j-1}^k\right)\approx {\Delta }_{x}v\left(s_k\mathrm{,}{x}_{i\mathrm{,}j}\right)\mathrm{.}$ (6.14)

We use $w^k\left(x\right)\mathrm{,}k\in \left\{\mathrm{0,1,}\cdots \mathrm{,}L\right\}$ to denote the piecewise bi-linear interpolant of $w_{i\mathrm{,}j}^k$ , $P\left(x_1\pm 1\pm \alpha \mathrm{,}{x}_2+j\zeta \right)$ , $P\left(x_1+i\zeta \mathrm{,}{x}_2\pm 1\pm \alpha \right)$ , $P\left(x_1-1-\alpha \mathrm{,}{x}_2\pm 1\pm \alpha \right)$ and $P\left(x_1+1+\alpha \mathrm{,}{x}_2\pm 1\pm \alpha \right)$ , for all $i\mathrm{,}j$ .

By difference iteration method with (6.11)-(6.14) and $w^k\left(x\right)$ (a piecewise bi-linear interpolant of $w_{i\mathrm{,}j}^k$ ), for a given $\left(t\mathrm{,}x\right)\in \left[\mathrm{0,}T\right]\times R^2$ , we write algorithm for approximating $v_x\left(t\mathrm{,}x\right)$ as follows.

Algorithm 4.1.

1) Set $w_{i\mathrm{,}j}^0=P\left(x_{i\mathrm{,}j}\right)$ , $i\mathrm{,}j\in \left\{-N\mathrm{,}\cdots \mathrm{,}-\mathrm{1,0,1,}\cdots \mathrm{,}N\right\}$ ;

2) Compute

$u_{i\mathrm{,}j}^0=h\left(x_{i\mathrm{,}j}\mathrm{,}{g}^{\text{T}}\left(x_{i\mathrm{,}j}\right)\delta w_{i\mathrm{,}j}^0\right)\mathrm{,}$

${\bar{x}}_{i\mathrm{,}j}^0=x_{i\mathrm{,}j}+\left[f\left(x_{i\mathrm{,}j}\right)+g\left(x_{i\mathrm{,}j}\right)u_{i\mathrm{,}j}^0\right]\tau \mathrm{,}$

${\bar{w}}_{i\mathrm{,}j}^0=w^0\left({\bar{x}}_{i\mathrm{,}j}^0\right)\mathrm{<mo>\; ;\; </mo>}$

3) For $k=\mathrm{1,2,}\cdots \mathrm{,}L$ ,

$\epsilon {\delta }^2{w}_{i\mathrm{,}j}^{k-1}-\frac{{\bar{w}}_{i\mathrm{,}j}^{k-1}-w_{i\mathrm{,}j}^k}{\tau }=-Q\left(x_{i\mathrm{,}j}\right)+h{\left(t\mathrm{,}{x}_{i\mathrm{,}j}\right)}^{\text{T}}h\left(t\mathrm{,}{x}_{i\mathrm{,}j}\right)\mathrm{.}$ (6.15)

$u_{i\mathrm{,}j}^k=h\left(x_{i\mathrm{,}j}\mathrm{,}{g}^{\text{T}}\left(x_{i\mathrm{,}j}\right)\delta w_{i\mathrm{,}j}^k\right)$ (6.16)

${\bar{x}}_{i\mathrm{,}j}^k=x_{i\mathrm{,}j}+\left[f\left(x_{i\mathrm{,}j}\right)+g\left(x_{i\mathrm{,}j}\right)u_{i\mathrm{,}j}^k\right]\tau \mathrm{.}$ (6.17)

${\bar{w}}_{i\mathrm{,}j}^k=w^k\left({\bar{x}}_{i\mathrm{,}j}^k\right)\mathrm{<mo>\; ;\; </mo>}$ (6.18)

4) Compute

$\delta w_{\mathrm{0,0}}^L={\left(2h\right)}^{-1}\left(w_{\mathrm{1,0}}^L-w_{-\mathrm{1,0}}^L\mathrm{,}{w}_{\mathrm{0,1}}^L-w_{\mathrm{0,}-1}^L\right)\approx v_x\left(s_L\mathrm{,}{x}_{\mathrm{0,0}}\right)=v_x\left(t\mathrm{,}x\right)\mathrm{.}$ (6.19)

Remark 6.1. The above discretization scheme is essentially an Euler’s method in the characteristic direction to be stable in t [12] . The system matrix associated with the third step of the algorithm above is symmetric and positive definite if $\epsilon$ is sufficiently smaller than $\frac{h^2}{\tau }$ . We can benefit from these special properties to use an efficient iterative solver which takes advantage of the symmetry and positive-definiteness of the matrix.

Remark 6.2. In Algorithm 4.1, for computing $w_{i\mathrm{,}j}^k\left(k=\mathrm{1,2,}\cdots \mathrm{,}L\right)$ , by (6.15) we only need to have linear computations:

$w_{i\mathrm{,}j}^k={\bar{w}}_{i\mathrm{,}j}^{k-1}-\tau \left[\epsilon {\delta }^2{w}_{i\mathrm{,}j}^{k-1}+Q\left(x_{i\mathrm{,}j}\right)-h{\left(t\mathrm{,}{x}_{i\mathrm{,}j}\right)}^{\text{T}}h\left(t\mathrm{,}{x}_{i\mathrm{,}j}\right)\right]\mathrm{,}$ (6.20)

noting that $w_{i\mathrm{,}j}^{k-1}\mathrm{,}{\bar{x}}_{i\mathrm{,}j}^{k-1}\mathrm{,}{\bar{w}}_{i\mathrm{,}j}^{k-1}$ have been got by previous steps.

7. Conclusion

It is well-known that in general a value function of optimal control problem is non-smooth. It is hard to study the regularity properties of a non-smooth function. In this paper, we study the regularity properties of the value function in an affine optimal control problem by solving the global minimization problem for the value function over the attainable set. We also provide a computational approach to this global minimization by a convection-diffusion equation. By more works in future we may consider some global optimization for non-smooth function with the help of optimal control methods.

Conflicts of Interest

The author declares no conflicts of interest.

References

[1]	Barilari, D. and Boarotto, F. (2018) On the Set of Points of Smoothness for the Value Function of Affine Optimal Control Problems. SIAM Journal on Control and Optimization, 56, 649-671. https://doi.org/10.1137/17M1123948
[2]	Pontryagin, L.S. (1964) The Mathematical Theory of Optimal Processes. Pergamon Press, Oxford.
[3]	Sontag, E.D.P. (1998) Mathematical Control Theory: Deterministic Finite Dimensional Systems. 2nd Edition, Springer, New York.
[4]	Bardi, M. and Capuzzo-Dolcetta, I. (1997) Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations. Birkhauser, Boston. https://doi.org/10.1007/978-0-8176-4755-1
[5]	Crandall, M.G. and Lions, P.L. (1983) Viscosity Solution of Hamilton-Jacobi Equations. Transactions of the American Mathematical Society, 277, 1-42. https://doi.org/10.1090/S0002-9947-1983-0690039-8
[6]	Fleming, W.H. (1969) The Cauchy Problem for a Nonlinear First Order Partial Differential Equation. Journal of Differential Equations, 5, 515-530. https://doi.org/10.1016/0022-0396(69)90091-6
[7]	Agrachev, A.A., Barilari, D. and Boscain, U. (2012) Introduction to Riemannian and Sub-Riemannian Geometry. https://people.sissa.it/~agrachev/agrachev_files/ABB-final-SRnotes.pdf
[8]	Rifford, L. (2014) Sub-Riemannian Geometry and Optimal Transport. Springer, Cham. https://doi.org/10.1007/978-3-319-04804-8
[9]	Trelat, E. (2000) Some Properties of the Value Function and Its Level Sets for Affine Control Systems with Quadratic Cost. Journal of Dynamical and Control Systems, 61, 511-541.
[10]	Sussmann, H.J. and Jurdjevic, V. (1972) Controllability of Nonlinear Systems. Journal of Differential Equations, 12, 95-116. https://doi.org/10.1016/0022-0396(72)90007-1
[11]	Zhu, J.H. (2005) On Stochastic Riccati Equations for the Stochastic LQR Problem. System and Control Letters, 54, 119-124. https://doi.org/10.1016/j.sysconle.2004.07.003
[12]	Burden, R.L. and Faires, J.D. (1989) Numerical Analysis. 4th edition, Weber and Schmidt, Boston.