Offline Robustness of Distributional Actor-Critic Ensemble Reinforcement Learning

Zhongcui Ma¹, Dandan Lai¹, Jianxiang Zhu¹, Yaxin Peng^2*

¹Department of Mathematics, College of Sciences, Shanghai University, Shanghai, China.
²School of Future Technology, and Department of Mathematics, College of Sciences, Shanghai University, Shanghai, China.

DOI: 10.4236/apm.2025.154013                

Abstract

Offline reinforcement learning (RL) focuses on learning policies using static datasets without further exploration. With the introduction of distributional reinforcement learning into offline RL, current methods excel at quantifying the risk and ensuring the security of learned policies. However, these algorithms cannot effectively balance the distribution shift and robustness, and even a minor perturbation in observations can significantly impair policy performance. In this paper, we propose the algorithm of Offline Robustness of Distributional actor-critic Ensemble Reinforcement Learning (ORDER) to improve the robustness of policies. In ORDER, we introduce two approaches to enhance the robustness: 1) introduce the smoothing technique to policies and distribution functions for states near the dataset; 2) strengthen the quantile network. In addition to improving the robustness, we also theoretically prove that ORDER converges to a conservative lower bound, which can alleviate the distribution shift. In our experiments, we validate the effectiveness of ORDER in the D4RL benchmark through comparative experiments and ablation studies.

Keywords

Offline Reinforcement Learning, Distributional Reinforcement Learning, Robustness

1. Introduction

Offline reinforcement learning (RL) [1]-[3] concerns the problem of learning a policy from a fixed dataset without further interactions. Offline RL can reduce risk and costs since it eliminates the need for online interaction. In this way, offline RL can be well used in real-world applications such as autonomous driving [4], healthcare [5] and robot control [6].

Applying the standard policy improvement approaches to an offline dataset typically results in the distribution shift problem, making offline RL a challenging task [7] [8]. Some prior works have relieved this issue by penalizing the action-value of the out-of-distribution (OOD) actions [9]-[11]. Nevertheless, simply learning the expectation of action-value is unable to quantify risk and ensure that the learned policy acts safely. To overcome this problem, some efforts have been made to import distributional RL [12]-[14] into offline RL to learn the full distribution over future returns, which is used to make plans to avoid risky and unsafe actions. In addition, with the establishment of risk-sensitive objectives [14], distributional offline RL [15] [16] learns state representations better since they can acquire richer distributed signals, making them superior to traditional reinforcement learning algorithms even on risk-seeking and risk-averse objectives.

Unfortunately, research on distributional offline RL is less complete. CODAC [15] brings distributional RL into the offline setting by penalizing the predicted return quantiles for OOD actions. Meanwhile, MQN-CQR [16] learns a worst-case policy by optimizing the conditional value-at-risk of the distributional value function. However, existing distributional offline RL methods only focus on the safety of the learned policy. These methods leverage a conservative return distribution to impair the robustness, and will make policies highly sensitive, even a minor perturbation in observations [17]. As a result, merely possessing safety can not make a fine balance between conservatism and robustness, which does not pay enough attention to robustness.

In this paper, we propose Offline Robustness of Distributional actor-critic Ensemble Reinforcement Learning (ORDER) by introducing a smoothing technique to quantile networks. Firstly, we consider the dynamic entropy regularizer of the quantile function instead of an unchangeable constant to ensure sufficient exploration. Secondly, the increasing number of quantile networks is also beneficial in obtaining a more robust distribution value function. Thirdly, smooth regularization is brought into the distribution and policies of states near the dataset. In theory, we prove that ORDER obtains a uniform lower bound on all integrations of the quantiles with the distribution soft Bellman operator, which controls the distribution shift. Such bound also achieves the same effect for both expected returns and risk-sensitive objectives. Overall, ORDER can mitigate the OOD problem and simultaneously balance conservatism and robustness.

In our experiments, ORDER outperforms the existing distributional offline RL methods in the D4RL benchmark [18]. Meanwhile, our algorithm is also competitive against the current advanced algorithms. Our ablation experiments demonstrate that strengthening the quantile network is critical to the performance of ORDER. In addition, choosing different risk measure functions does not have a great impact on the performance of ORDER, which also shows the robustness of our method.

2. Preliminaries

2.1. Markov Decision Process and Offline Reinforcement Learning

Consider an episodic Markov decision process (MDP) $M=\left(\mathcal{S},\mathcal{A},\mathbb{P},T,R,\gamma \right)$ , where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathbb{P}\left(s^{\prime }|s,a\right)$ is the transition distribution, $T$ is the length of the episode, $R\left(s,a\right)$ is the reward function, and $\gamma$ is the discount factor. For a stochastic policy $\pi \left(a|s\right):\mathcal{S}\times \mathcal{A}\to ℝ$ , action-value function $Q^{\pi }$ is defined as

$Q^{\pi }\left(s,a\right):={\mathbb{E}}_{\pi }\left[{\displaystyle \sum }_{t=0}^{\infty }{\gamma }^{t}R\left(s_t,a_t\right)\right]|a_t~\pi \left(\cdot |s_t\right),s_{t+1}~\mathbb{P}\left(\cdot |s_t,a_t\right),s_0=s,a_0=a.$

Standard RL aims at learning the optimal policy ${\pi }^{\star }$ such that $Q^{{\pi }^{\star }}\left(s,a\right)\ge Q^{\pi }\left(s,a\right)$ for all $s\in \mathcal{S},a\in \mathcal{A}$ and all $\pi$ . The corresponding $Q$ -function of the policy satisfies the Bellman operator:

${\mathcal{T}}^{\pi }Q\left(s,a\right)=\mathbb{E}\left[R\left(s,a\right)\right]+\gamma \cdot {\mathbb{E}}_{\mathbb{P},\pi }\left[Q\left(s^{\prime },a^{\prime }\right)\right].$

In the offline setting, the agent is not allowed to interact with the environment [18]. The objective of agents is to learn an optimal policy only from a fixed dataset $\mathcal{D}:=\left\{\left(s,a,r,s^{\prime }\right)\right\}$ . For all states $s\in \mathcal{D}$ , let $\widehat{\pi }\left(a|s\right):=\frac{{{\displaystyle \sum }}_{s^{\prime },a^{\prime }\in \mathcal{D}}{1}_{\left(s^{\prime }=s,a^{\prime }=a\right)}}{{{\displaystyle \sum }}_{s^{\prime }\in \mathcal{D}}{1}_{s^{\prime }=s}}$ be the empirical behavior policy. In order to avoid the situation where the denominator of the fraction is 0 in the theoretical analysis, we assume that $\widehat{\pi }\left(a|s\right)>0$ . Broadly, actions that are not drawn from $\widehat{\pi }$ (i.e., those with low probability density) are the out-of-distribution (OOD).

2.2. Distributional Reinforcement Learning

Distributional RL directly models the full distribution of returns $Z^{\pi }\left(s,a\right)={\displaystyle {\sum }_{t=0}^{\infty }{\gamma }^{t}R\left(s_t,a_t\right)}$ , instead of merely learning its expected value [19]. The distributional Bellman operator for policy evaluation is ${\mathcal{T}}^{\pi }Z\left(s,a\right):\stackrel{D}{=}R\left(s,a\right)+\gamma Z\left(s^{\prime },a^{\prime }\right)$ , where $\stackrel{D}{=}$ indicates equality in distribution. Define the quantile function be the inverse of the cumulative density function $F_Z\left(z\right)=Pr\left(Z<z\right)$ as $F_Z^{-1}\left(\tau \right):=\text{inf}\left\{z\in ℝ:\tau \le F_Z\left(z\right)\right\}$ [20], where $\tau$ denotes the quantile fraction. For random variables $U$ and $V$ with quantile functions $F_U^{-1}$ and $F_V^{-1}$ , the $p$ -Wasserstein distance $W_p\left(U,V\right)={\left({\displaystyle {\int }_0^1{\left|F_U^{-1}\left(\tau \right)-F_V^{-1}\left(\tau \right)\right|}^p\text{d}\tau }\right)}^{1/p}$ is the $L_p$ metric on quantile functions. [19] gave that the distributional Bellman operator ${\mathcal{T}}^{\pi }$ is a $\gamma$ -contraction in the $W_p$ , i.e., let ${\overline{d}}_p\left(Z_1,Z_2\right):={\sup }_{s,a}{W}_p\left(Z_1\left(s,a\right),Z_2\left(s,a\right)\right)$ be the largest Wasserstein

distance over $\left(s,a\right)$ , and be the space of distributions over $ℝ$ with bounded $p$ -th moment, then

${\overline{d}}_p\left({\mathcal{T}}^{\pi }{Z}_1,{\mathcal{T}}^{\pi }{Z}_2\right)\le \gamma \cdot {\overline{d}}_p\left(Z_1,Z_2\right)\forall Z_1,Z_2\in \mathcal{Z}.$

Fitted distributional evaluation (FDE) [15] approximates ${\mathcal{T}}^{\pi }$ by ${\widehat{\mathcal{T}}}^{\pi }$ using $\mathcal{D}$ , then $Z^{\pi }$ can be estimated by starting from an arbitrary ${\widehat{Z}}^0$ and iteratively computing

$\begin{array}{l}{\widehat{Z}}^{k+1}=\underset{Z}{\arg \min }{ℒ}_p\left(Z,{\widehat{\mathcal{T}}}^{\pi }{\widehat{Z}}^k\right),\\ \text{where}{ℒ}_p\left(Z,Z^{\prime }\right)={\mathbb{E}}_{\mathcal{D}\left(s,a\right)}\left[W_p{\left(Z\left(s,a\right),Z^{\prime }\left(s,a\right)\right)}^p\right].\end{array}$

In distributional RL, let risk measure function $\Phi :\mathcal{Z}\to ℝ$ be a map from the value distribution space to real numbers. Given a distorted function $g\left(\tau \right)$ over $\left[0,1\right]$ , the distorted expectation of $Z$ is

${\Phi }_g\left(Z\left(s,a\right)\right)={\displaystyle {\int }_0^1{F}_{Z\left(s,a\right)}^{-1}}\left(\tau \right)g\left(\tau \right)\text{d}\tau ,$

and the corresponding policy is ${\pi }_g\left(s\right):=\arg {\max }_a{\Phi }_g\left(Z\left(s,a\right)\right)$ [13]. Specially, if $g=\text{Uniform}\left(\left[0,1\right]\right)$ , then $Q^{\pi }\left(s,a\right)={\Phi }_g\left(Z\left(s,a\right)\right)$ . For other choices of $g\left(\tau \right)$ , please refer to Section 5.2.

2.3. Robust Reinforcement Learning

Robust RL learns the policy by introducing worst-case adversarial noise to the system dynamics and formulating the noise distribution as the solution of a zero-sum minimax game. In order to learn robust policy $\pi$ , SR²L [21] obtains $\widehat{s}=\arg {\max }_{\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{D}_{\text{J}}\left(\pi \left(\cdot |s\right)\left|\right|\pi \left(\cdot |\widehat{s}\right)\right)$ by adding a perturbation to the state $s$ , where ${\mathbb{B}}_d\left(s,ϵ\right)=\left\{\widehat{s}:d\left(s,\widehat{s}\right)\le ϵ\right\}$ and the Jeffrey’s divergence $D_{\text{J}}$ for two distributions P and $Q$ is defined by $D_{\text{J}}\left(P\left|\right|Q\right)=\frac{1}{2}\left[D_{\text{KL}}\left(P\left|\right|Q\right)+D_{\text{KL}}\left(Q\left|\right|P\right)\right]$ , and then define a smoothness regularizer for policy as ${ℛ}_s^{\pi }={\mathbb{E}}_{s\sim {\rho }^{\pi }}{\max }_{\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{D}_{\text{J}}\left(\pi \left(\cdot |s\right)\left|\right|\pi \left(\cdot |\widehat{s}\right)\right)$. Analogously, ${ℛ}_s^Q={\mathbb{E}}_{s~{\rho }^{\pi }}{\max }_{\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{\left(Q\left(s,a\right)-Q\left(\widehat{s},a\right)\right)}^2$ is the smoothness regularizer for Q-function, where ${\rho }^{\pi }$ is state distribution.

3. Offline Robustness of Distributional Actor-Critic Ensemble

In ORDER, we first obtain a state with adversarial perturbations, and then introduce the smoothness regularization to both the policy and the distribution action-value function for states with adversarial noises. The smooth regularization can be used to learn a smooth Z-function and generate a smooth policy, which makes the algorithm robust. However, the introduction of smoothness could potentially result in an overestimation of values at the boundaries of the supported dataset. To overcome this problem, we incorporate a penalty factor for OOD actions to reduce the quantile values of these actions. In addition, we strengthen the quantile network by increasing the number of quantile networks, which is also beneficial to the robustness of our algorithm. The overall architecture of ORDER is shown in Figure 1.

Figure 1. Architecture diagram for ORDER.

3.1. Robust Distributional Action-Value Function

In this part, we sample three sets of state-action pairs and form three different loss functions to obtain a conservative smooth policy. First of all, we construct a perturbation set ${\mathbb{B}}_d\left(s,ϵ\right)$ to obtain $\left(\widehat{s},a\right)$ pairs, where ${\mathbb{B}}_d\left(s,ϵ\right)$ is an $ϵ$ -radius ball measured in metric $d\left(\cdot ,\cdot \right)$ and $\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)$ . Then we sample $\left(s,\widehat{a}\right)$ pairs from the current policy ${\pi }_{\varphi }$ , where $\widehat{a}~{\pi }_{\varphi }\left(s\right)$ . ORDER contains $M$ Z-function and denotes the parameters of the $m$ -th Z-function and the target Z-function as ${\theta }_m$ and ${{\theta }^{\prime }}_m$ , respectively. With the help of the constructions, we will give the different learning targets for $\left(s,a\right)$ , $\left(\widehat{s},a\right)$ and $\left(s,\widehat{a}\right)$ pairs, respectively.

SAC [7] incorporates an entropy term in its objective function to optimize both cumulative rewards and policy stochasticity, enhancing policy robustness and generalization. We also introduce an entropy term, for a $\left(s,a\right)$ pair sampled from $\mathcal{D}$ , we obtain the target as

${\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{Z}_{{\theta }_m}\left(s,a\right):=R\left(s,a\right)+\gamma \left[\underset{j=1,\cdots ,M}{\text{min}}{Z}_{{{\theta }^{\prime }}_j}\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right],$

where the next Z-function takes the minimum value among the target Z-functions. The loss function is defined as follows,

${ℒ}_Z\left({\theta }_m\right)={ℒ}_p\left(Z_{{\theta }_m},{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{Z}_{{\theta }_m}\right).$

Next, we introduce the smoothness regularizer term to the distribution action-value function that is designed to enhance the smoothness of the Z-function. Specifically, we minimize the difference between $Z_{{\theta }_m}\left(\widehat{s},a\right)$ and $Z_{{\theta }_m}\left(s,a\right)$ , where $\left(\widehat{s},a\right)$ is a state-action pair with a perturbed state. Then we take an adversarial $\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)$ which maximizes ${ℒ}_p\left(Z_{{\theta }_m}\left(\widehat{s},a\right),Z_{{\theta }_m}\left(s,a\right)\right)$ . The final smooth term we introduced is shown below:

${ℒ}_{\text{smooth}}\left(s,a;{\theta }_m\right)=\underset{\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{\max }\varrho \cdot {ℒ}_p\left(Z_{{\theta }_m}\left(\widehat{s},a\right),Z_{{\theta }_m}\left(s,a\right)\right),$(1)

where $\varrho \in \left[0,1\right]$ is a factor that balances the learning of in-distribution and out-of-distribution values. Thus, for the selected $\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)$ , we minimize Equation (1) to get a smooth Z-function. Since the actions should be near the offline data and close to the behavior actions in the dataset, we do not consider OOD action for smoothing in this part.

Finally, we consider the following loss function to prevent overestimation of OOD actions.

${ℒ}_{\text{OOD}}\left({\theta }_m,\beta \right)=\beta \cdot {\mathbb{E}}_{U\left(\tau \right),\mathcal{D}\left(s,a\right)}\left[c_0\left(s,a\right)\cdot F_{Z_{{\theta }_m}\left(s,a\right)}^{-1}\left(\tau \right)\right],$

for some state-action dependent scale factor $c_0$ and $U=\text{Uniform}\left(\left[0,1\right]\right)$ .

Incorporating both the in-distribution target and OOD target, we conclude the loss function in ORDER as follows,

$\underset{{\theta }_m}{\min }{\mathbb{E}}_{s,a,r,s^{\prime }~\mathcal{D}}\left[{ℒ}_Z\left({\theta }_m\right)+\alpha {ℒ}_{\text{smooth}}\left(s,a;{\theta }_m\right)+{ℒ}_{\text{OOD}}\left({\theta }_m,\beta \right)\right].$ (2)

3.2. Robust Policy

With the above smooth limits, we can learn a robust policy with fewer policy changes under perturbations. We choose a state $\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)$ as mentioned above which is maximizing $D_{\text{J}}\left({\pi }_{\varphi }\left(\cdot |s\right)\left|\right|{\pi }_{\varphi }\left(\cdot |\widehat{s}\right)\right)$. Consequently, our loss function for policy is as follows:

$\underset{\varphi }{\text{min}}\left[{\mathbb{E}}_{s,a,r,s^{\prime }~\mathcal{D}}\left[-\underset{j=1,\cdots ,M}{\text{min}}{\Phi }_g^j\left(s,a\right)+{\alpha }_1\underset{\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{\text{max}}{D}_{\text{J}}\left({\pi }_{\varphi }\left(\cdot |s\right)\left|\right|{\pi }_{\varphi }\left(\cdot |\widehat{s}\right)\right)+\log {\pi }_{\varphi }\left(a|s\right)\right]\right],$ (3)

where the first term is designed to get a conservative policy by maximizing the minimum of the distributional functions ensemble and the last term is a regularization term.

3.3. Implementation Details

In this subsection, we integrate the distributional evaluation and policy improvement algorithms introduced in Section 3.1 and Section 3.2 into the actor-critic framework. With the loss function introduced in Equation (2), it is natural to get the following iterative formula for $Z^{\pi }$ by starting from an arbitrary ${\tilde{Z}}^0$ ,

$\begin{array}{c}{\tilde{Z}}^{k+1}=\underset{Z}{\arg \min }\{{ℒ}_p\left(Z\left(s,a\right),{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)\right)+\underset{\widehat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\widehat{s},a\right),Z\left(s,a\right)\right)\\ +\beta \cdot {\mathbb{E}}_{U\left(\tau \right),\mathcal{D}\left(s,a\right)}\left[c_0\left(s,a\right)\cdot F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]\}.\end{array}$(4)

Following [17], we suggest employing a min-max objective in which the inner loop selects the current policy to maximize the objective, while the outer loop is responsible for minimizing the objective with respect to this policy:

$\begin{array}{c}{\tilde{Z}}^{k+1}=\underset{Z}{\arg \min }\underset{\mu }{\text{max}}\{\beta \cdot {\mathbb{E}}_{U\left(\tau \right)}\left[{\mathbb{E}}_{\mathcal{D}\left(s\right),\mu \left(\tilde{a}|s\right)}{F}_{Z\left(s,\tilde{a}\right)}^{-1}\left(\tau \right)-{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{F}_{Z\left(s,a\right)}^{-1}\right]\\ +{ℒ}_p\left(Z\left(s,a\right),{\widehat{\mathcal{T}}}_{\text{DS}}^{{\pi }^k}{\widehat{Z}}^k\left(s,a\right)\right)+\underset{\widehat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\widehat{s},a\right),Z\left(s,a\right)\right)\},\end{array}$

where $\mu$ is an actor policy. To establish a well-posed optimization problem, we introduce a regularization term in the original objective. Detailed analysis procedures are provided in Appendix A.1. The final optimization objective becomes

$\begin{array}{c}{\tilde{Z}}^{k+1}=\underset{Z}{\arg \min }\{\beta \cdot {\mathbb{E}}_{U\left(\tau \right)}\left[{\mathbb{E}}_{\mathcal{D}\left(s\right)}\log {\displaystyle \sum }_a\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)-{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{F}_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]\\ \stackrel{}{}+{ℒ}_p\left(Z\left(s,a\right),{\widehat{\mathcal{T}}}_{\text{DS}}^{{\pi }^k}{\widehat{Z}}^k\left(s,a\right)\right)+\underset{\widehat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\widehat{s},a\right),Z\left(s,a\right)\right)\},\end{array}$

where $U=\text{Uniform}\left(\left[0,1\right]\right)$ . To perform optimization with respect to the distribution $Z$ , we express the quantile function using a neural network $G_{\theta }\left(\tau ;s,a\right)\approx F_{Z\left(s,a\right)}^{-1}\left(\tau \right)$ . In order to calculate ${ℒ}_p\left(Z,Z^{\prime }\right)=W_p{\left(Z,Z^{\prime }\right)}^p$ [22], we minimize the weighted pairwise Huber regression loss of various quantile fractions. $\tau$ -Huber quantile regression loss [23] with threshold $\kappa$ is represented as

${ℒ}_{\kappa }\left(\delta ;\tau \right)=\{\begin{array}{ll}\left|\tau -1\left\{\delta <0\right\}\right|\cdot {\delta }^2/\left(2\kappa \right),\hfill & \text{if}\left|\delta \right|\le \kappa ,\hfill \\ \left|\tau -1\left\{\delta <0\right\}\right|\cdot \left(\left|\delta \right|-\kappa /2\right),\hfill & \text{otherwise},\hfill \end{array}$

where $\tau ,{\tau }^{\prime }~U$ are random quantiles, $\delta =r+\gamma G_{{\theta }^{\prime }}\left({\tau }^{\prime };s^{\prime },a^{\prime }\right)-G_{\theta }\left(\tau ;s,a\right)$ , $\left(s,a,r,s^{\prime }\right)~\mathcal{D}$ , $a^{\prime }~\pi \left(\cdot |s^{\prime }\right)$ . More details for the algorithm ORDER are presented in Algorithm 1.

3.4. Theoretical Analysis

Before presenting our theorems, similar to [15], we first give some assumptions about the MDP and dataset. Next, we assume that the search space in Equation (4) includes all possible functions.

Assumption 1. For all $s\in \mathcal{S}$ and $a\in \mathcal{A}$ , $F_{Z^{\pi }\left(s,a\right)}$ is smooth. Furthermore, the search space of the minimum over Z in Equation (4) is overall smooth functions $F_{Z\left(s,a\right)}$ .

The assumption is given to guarantee the boundness of the $p$ -th moments of $Z^{\pi }\left(s,a\right)$ as well as $Z^{\pi }\in \mathcal{Z}$ .

Assumption 2. For all $s\in \mathcal{S}$ and $a\in \mathcal{A}$ , there exists $\zeta \in {ℝ}_{>0}$ , such that $F_{Z^{\pi }\left(s,a\right)}$ is $\zeta$ -strongly monotone, i.e., ${F^{\prime }}_{Z^{\pi }\left(s,a\right)}\left(x\right)\ge \zeta$ .

This assumption is only designed to ensure the convergence of $F_{Z^{\pi }\left(s,a\right)}^{-1}\left(x\right)$ in our theoretical analysis. In algorithm implementation, we introduce a policy smoothing term and choose the minimum value of the Z-function in the ensemble to restrict the policy update amplitude, thus implicitly satisfying monotonicity. Next, we assume that the infinite norm of two quantiles with perturbation is restricted by a fixed constant.

Assumption 3. For all $s\in \mathcal{S}$ , $a\in \mathcal{A}$ and the selected $\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)$ , we assume that ${\Vert F_{Z\left(\widehat{s},a\right)}^{-1}-F_{Z\left(s,a\right)}^{-1}\Vert }_{\infty }\le \sigma$ , $\sigma$ is constant.

In the above assumption, ${\Vert \cdot \Vert }_{\infty }$ represents the infinite norm. Since perturbed observation $\widehat{s}$ is randomly sampling from an ${\ell }_{\infty }$ ball of norm $ϵ$ , the assumption is reasonable.

Finally, we assume $p>1$ . Therefore, we derive the following lemma which characterizes the conservative distribution soft evaluation iterates $Z\left(s,a\right)$ with the distributional soft Bellman operator.

Lemma 1. Suppose Assumptions 1 - 3 hold. For all $s\in \mathcal{D}$ , $a\in \mathcal{A}$ , $k\in \mathbb{N}$ , and $\tau \in \left[0,1\right]$ , we have $F_{Z\left(s,a\right)}^{-1}\left(\tau \right)=F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)-c\left(s,a\right)$, where $c\left(s,a\right)={\left|\beta p^{-1}{c}_0\left(s,a\right)\pm {\sigma }^{p-1}\right|}^{\frac{1}{p-1}}\cdot \text{sign}\left(c_0\left(s,a\right)\right)$ .

For detailed proof, please refer to Appendix B.2. Briefly, it is according to the result of a simple variational skill to handle that F is a function, and setting the derivative of Equation (4) equal to zero.

Next, we define the conservative soft distributional evaluation operator ${\tilde{\mathcal{T}}}^{\pi }={\mathcal{O}}_c{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }$ by compositing ${\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }$ and the shift operator ${\mathcal{O}}_c:\mathcal{Z}\to \mathcal{Z}$ , which is defined by $F_{{\mathcal{O}}_{c}Z\left(s,a\right)}^{-1}\left(\tau \right)=F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-c\left(s,a\right)$ . Consequently, it is following that ${\tilde{Z}}^{k+1}={\tilde{\mathcal{T}}}^{\pi }{\tilde{Z}}^k$ .

Now, the main theorem we came up with shows that the conservative distributional soft evaluation obtains a conservative quantile estimate of the true quantile at all quantiles $\tau$ .

Theorem 1. For any $\delta \in {ℝ}_{>0},c_0\left(s,a\right)>0$ , with probability at least $1-\delta$ , we have

$F_{Z^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)\ge F_{{\tilde{Z}}^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)+{\left(1-\gamma \right)}^{-1}\underset{s^{\prime },a^{\prime }}{\text{min}}\left\{c\left(s^{\prime },a^{\prime }\right)-\frac{1}{\zeta }\sqrt{\frac{5\left|\mathcal{S}\right|}{n\left(s^{\prime },a^{\prime }\right)}\text{log}\frac{4\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{\delta }}\right\},$

for all $s\in \mathcal{D},a\in \mathcal{A}$ , and $\tau \in \left[0,1\right]$ . Furthermore, for $\beta \ge {\text{max}}_{s,a}\left\{\frac{p\left(\text{Δ}{\left(s,a\right)}^{p-1}+{\sigma }^{p-1}\right)}{c_0\left(s,a\right)}\right\}$ , we have $F_{Z^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)\ge F_{{\tilde{Z}}^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)$ .

For detailed proof, please refer to Appendix B.2. As the theorem shows, the above inequality indicates that the quantile estimates obtained by ${\tilde{\mathcal{T}}}^{\pi }$ are a lower bound of the true quantiles. Furthermore, we give a sufficient condition to show that the result given in Theorem 1 is not a vacuous conclusion. Therefore, Theorem 1 theoretically illustrates that ORDER does not exacerbate the distribution shift problem, and the mitigation of the distribution shift problem will be demonstrated in the experimental section.

The performance of many RL algorithms will exhibit different behaviors under different distorted expectations. Consequently, we can acquire the same conservative estimates of these objectives, which is a kind of generalization of Theorem 1.

Corollary 1 For any $\delta \in {ℝ}_{>0}$ , $c_0\left(s,a\right)>0$ , sufficiently large $\beta$ and $g\left(\tau \right)$ , with probability at least $1-\delta$ , for all $s\in \mathcal{D},a\in \mathcal{A}$ , we have ${\Phi }_g\left(Z^{\pi }\left(s,a\right)\right)\ge {\Phi }_g\left({\tilde{Z}}^{\pi }\left(s,a\right)\right)$ .

In particular, $Q^{\pi }\left(s,a\right)\ge {\tilde{Q}}^{\pi }\left(s,a\right)$ is obtained if we take $g=\text{Uniform}\left(\left[0,1\right]\right)$ . By choosing different risk measure functions, we can apply this conclusion to any risk-sensitive offline RL.

4. Related Works

Offline RL [1] [24]-[26] learns a policy from previously collected static datasets. As a subfield of RL [27] [28], it has achieved significant accomplishments in practice [4]-[6] [29]. However, the main two challenges of offline RL are the distribution shift problem and robustness [26] [30] [31], which require various techniques to improve the stability and performance of learned policies.

4.1. Distribution Shift

The cause of the distribution shift problem is that the distribution of collected offline training data differs from the distribution of data in practice. BCQ [9] addresses this problem through policy regularization techniques, which formulates the policy as an adaptable deviation constrained by maximum values [32]. One solution to alleviate the distribution shift problem as mentioned in BEAR [33] is incorporating a weighted behavior-cloning loss achieved by minimizing maximum mean discrepancy (MMD) into the policy improvement step. Though learning a conservative Q-function caused by distribution shift, CQL [17] solves the overestimation of value functions, which theoretically proves that a lower bound of the true value is obtained. With the introduction of distributional reinforcement learning into offline RL, CODAC [15] learns a conservative return distribution by punishing the predicted quantiles returned for the OOD actions. From another perspective, continuous quantiles are used in MQN-CQR [16] to learn the quantile of return distribution with non-crossing guarantees. ORDER builds on these approaches, but considers the entropy regularizer of quantile function instead of an unchangeable constant for ensuring sufficient exploration and may relieve the training imbalance.

4.2. Robustness Issues

Owning to the distribution shift issues, current offline RL algorithms tend to prioritize caution in their approach to value estimation and action selection. Nevertheless, this selection can compromise the robustness of learned policies, making them highly sensitive to even minor perturbations in observations. As a groundbreaking piece of work, SR²L [21] achieves a more robust policy by introducing a smoothness regularizer into both the value function and the policy. A robust Q-learning algorithm proposed in REM [30] is presented that integrates multiple Q-value networks in a random convex combination of multiple Q-value estimates, ensuring that the final Q-value estimate remains robust. With arbitrarily large state spaces, RFQI [34] learns the optimal policy by employing function approximation using only an offline dataset and addresses robust offline reinforcement learning problems. In ORDER, we import the smoothing regularizer to the distribution functions and policies instead of simple action-value functions.

Our method is related to the previous offline RL algorithms based on constraining the learned value function [17] [35]. What sets our method apart is that it can better capture uncertain information about OOD actions and learn more robust policies with the introduction of the smoothing technique into distributional reinforcement learning. In addition, we enhance the network by using the entropy regularizer of quantile networks and increasing the number of network ensembles, which also improves robustness.

5. Experiments

In the sequel, we first compare ORDER against some offline RL algorithms. Then how different risk measure functions impact our proposed algorithm is investigated in Section 5.2. Besides, Section 5.3 shows the ensemble size of the quantile network in ORDER. And our approach significantly exceeds the baseline algorithm in most tasks.

We evaluate our experiments on the D4RL benchmark [18] with various continuous control tasks and datasets. Specifically, we employ three environments (HalfCheetah, Hopper and Walker2d) and four dataset types (random, medium, medium-replay, and medium-expert). The random or medium dataset is generated by a single random or medium policy. The medium-replay dataset contains experiences collected in training a medium-level policy, and the medium-expert dataset is a mixture of medium and expert datasets.

5.1. Comparison with Offline RL Algorithms

In all the aforementioned datasets, we compare our method against several previous popular offline RL algorithms, including 1) bootstrapping error accumulation reduction (BEAR) [33], 2) conservative q-learning (CQL) [17], 3) CODAC [15], 4) robust offline reinforcement learning (RORL) [36], 5) monotonic quantile network with conservative quantile regression (MQN-CQR) [16]. The results of BEAR and CQL are directly taken from [18]. For CODAC and MQN-CQR, their results are taken from the original paper. Since the RORL paper does not report scores with five random seeds, we run the RORL using the official code base. Implementation details are given in Appendix D. Table A1 and Table A2 list the hyperparameters of ORDER in different datasets. Without loss of generality, we employ the neutral risk measure in this subsection.

Table 1. Normalized average returns on D4RL benchmark, averaged over five random seeds. “r”, “m”, “m-r” and “m-e” indicate the abbreviations of random, medium, medium-replay and medium-expert, respectively. All methods are run for 1M gradient steps.

Datasets	BEAR	CQL	CODAC	RORL (Reproduced)	MQN-CQR	ORDER
hopper-r	3.9 ± 2.3	7.9 ± 0.4	11.0 ± 0.4	22.7 ± 8.4	13.2 ± 0.6	24.8 ± 7.8
hopper-m	51.8 ± 4.0	53.0 ± 28.5	70.8 ± 11.4	104.8 ± 0.3	94.7 ± 13.2	101.5 ± 0.2
hopper-m-r	52.2 ± 19.3	88.7 ± 12.9	100.2 ± 1.0	102.3 ± 0.5	95.6 ± 18.5	106.4 ± 0.1
hopper-m-e	50.6 ± 25.3	105.6 ± 12.9	112.0 ± 1.7	112.8 ± 0.2	113.0 ± 0.5	114.6 ± 3.3
walker2d-r	12.8 ± 10.2	5.1 ± 1.3	18.7 ± 4.5	21.5 ± 0.2	22.6 ± 6.1	28.4 ± 6.2
walker2d-m	-0.2 ± 0.1	73.3 ± 17.7	82.0 ± 0.5	103.2 ± 1.7	80.0 ± 0.5	86.0 ± 0.2
walker2d-m-r	7.0 ± 7.8	81.8 ± 2.7	33.2 ± 17.6	90.1 ± 0.6	52.3 ± 16.7	87.9 ± 4.8
walker2d-m-e	22.1 ± 44.9	107.9 ± 1.6	106.0 ± 4.6	120.3 ± 1.8	112.1 ± 8.9	115.1 ± 1.2
halfCheetah-r	2.3 ± 0.0	17.5 ± 1.5	34.6 ± 1.3	28.2 ± 0.7	32.6 ± 2.9	31.5 ± 1.0
halfCheetah-m	43.0 ± 0.2	47.0 ± 0.5	46.3 ± 1.0	64.7 ± 1.1	45.1 ± 1.5	63.7 ± 0.4
halfCheetah-m-r	36.3 ± 3.1	45.5 ± 0.7	44.0 ± 0.8	61.1 ± 0.7	45.3 ± 7.9	57.4 ± 1.7
halfCheetah-m-e	46.0 ± 4.7	75.6 ± 25.7	70.4 ± 19.4	108.2 ± 0.8	71.1 ± 4.9	93.2 ± 1.1

The performance of all these algorithms is exhibited in Table 1, which reports the average normalized scores along with their corresponding standard deviations. We observe that ORDER outperforms BEAR in all tasks and surpasses the performance of the CQL algorithm. Significantly, our algorithm surpasses the performance of current distributed offline RL methods (see CODAC and MQN-CQR in Table 1), which is attributed to the assurance of robustness in ORDER. Meanwhile, ORDER competes favorably with the current state-of-the-art algorithms, owing to the safety guaranteed by distributional RL.

5.2. Policy Training under Risk Measures Function

In this subsection, we investigate how the risk measure functions affect the performance of ORDER. We compare three risk-averse learned policies [14] in distribution RL with the risk-neutral measure function. Specifically, for different $g\left(\tau \right)$ , three ways of distorted expectation are considered,

• CPW: $g\left(\tau \right)={\tau }^{\lambda }/{\left({\tau }^{\lambda }+{\left(1-\tau \right)}^{\lambda }\right)}^{1/\lambda }$ , and $\lambda$ is set as 0.71.

• Wang: $g\left(\tau \right)=F_{\mathcal{N}}\left(F_{\mathcal{N}}^{-1}\left(\tau \right)+\lambda \right)$ , where $\lambda$ is set as 0.25 and $F_{\mathcal{N}}$ is the standard Gaussian CDF.

• CVaR: $g\left(\tau \right)=\text{min}\left\{\tau /\lambda ,1\right\}$ , and $\lambda$ is set as 0.25.

Besides, we evaluate three risk-seeking learned policies.

• The first risk-seeking method is mean-variance and $\lambda$ is set as −0.1.

• The second risk-seeking method is Var, and $\lambda$ is set as 0.75.

• The third risk-seeking method is Wang, and $\lambda$ is set as −0.75.

The results are shown in Table 2 and Table 3, indicating that there is little difference between risk-averse methods and risk-seeking learned policies. This suggests that risk measure functions within the ORDER framework are not highly sensitive. At this point, we also empirically demonstrate the robustness of our approach.

Table 2. Performance of ORDER under various risk-averse methods in hopper-medium-replay-v2. Each method is run with five random seeds.

Risk measure	Neutral	CPW (0.71)	CVaR (0.25)	Wang (0.75)
Performance of ORDER	106.4 ± 0.1	105.3 ± 0.4	106.2 ± 0.4	106.0 ± 1.5

Table 3. Performance of ORDER under various risk-seeking methods in hopper-medium-replay-v2. Each method is run with five random seeds.

Risk measure	Neutral	Mean-Std (−0.1)	VaR (0.75)	Wang (−0.75)
Performance of ORDER	106.4 ± 0.1	107.5 ± 0.3	106.5 ± 0.6	106.4 ± 0.2

5.3. Ablations on Benchmark Results

Without loss of generality, we choose the hopper-medium-replay-v2 dataset as an example to conduct the ablation study in this subsection. The performance of our ORDER algorithm under different Ms is visualized in Figure 2. We observe with the increase of M, the effect has a significant improvement in both computation efficiency and stability; as shown in the yellow and purple lines. However, M should not be too large, which is presumably attributed to the overfitting problem (see the blue line, the normalized score value fluctuates significantly around the training epoch of 700). In conclusion, M is set as four to balance between robustness enhancement and computational efficiency improvement.

Figure 2. The normalized score under different ensemble sizes. Each method is run with five random seeds.

6. Conclusion

In this work, we introduce Offline Robustness of Distributional actor-critic Ensemble Reinforcement Learning (ORDER) to balance the conservatism and robustness in the offline setting. To achieve robustness, we first take into account the entropy regularizer that helps fully explore the dataset and alleviates training imbalance issues. Moreover, we consider the ensemble of multiple quantile networks to enhance robustness. Furthermore, a smoothing technique is introduced to the policies and the distributional functions for the perturbed states. In addition, we theoretically prove that ORDER converges to a conservative lower bound, which also shows that we improve the robustness without exacerbating the OOD problem. Finally, ORDER shows its advantage against the existing distributional offline RL methods in the D4RL benchmark. We also validate the effectiveness of ORDER through ablation studies.

Appendix

A. Algorithm and Implementation Details

In this section, we provide a comprehensive account of our practical implementation of ORDER, offering a detailed explanation of the process.

A.1. ORDER Objective

To establish a well-defined optimization problem, we introduce a regularization term denoted as $ℛ\left(\mu \right)$ in the original objective:

$\begin{array}{c}{\widehat{Z}}^{k+1}=\underset{Z}{\arg \min }\underset{\mu }{\text{max}}\{\beta \cdot {\mathbb{E}}_{U\left(\tau \right)}\left[{\mathbb{E}}_{\mathcal{D}\left(s\right),\mu \left(\tilde{a}|s\right)}{F}_{Z\left(s,\tilde{a}\right)}^{-1}\left(\tau \right)-{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{F}_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]\\ +{ℒ}_p\left(Z,{\widehat{\mathcal{T}}}^{{\pi }^k}{\widehat{Z}}^k\right)+\underset{\widehat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\widehat{s},a\right),Z\left(s,a\right)\right)\}+ℛ\left(\mu \right).\end{array}$

Let $c_0\left(s,a\right)=\frac{\mu \left(a|s\right)-\widehat{\pi }\left(a|s\right)}{\widehat{\pi }\left(a|s\right)}$ and $ℛ\left(\mu \right)$ to be the entropy $\mathscr{H}\left(\mu \right)$ , then $\mu \left(a|s\right)\propto \text{exp}\left(Q\left(s,a\right)\right)=\text{exp}\left({\displaystyle {\int }_0^1{F}_{Z\left(s,a\right)}^{-1}}\left(\tau \right)\text{d}\tau \right)$ is the solution to the inner-maximization. Substituting this selection into the previously mentioned regularized objective function gives

$\begin{array}{c}{\widehat{Z}}^{k+1}=\underset{Z}{\arg \min }\{\beta \cdot {\mathbb{E}}_{U\left(\tau \right)}\left[{\mathbb{E}}_{\mathcal{D}\left(s\right)}\log {\displaystyle \sum }_a\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)-{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{F}_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]\\ \stackrel{}{}+{ℒ}_p\left(Z,{\widehat{\mathcal{T}}}^{{\pi }^k}{\widehat{Z}}^k\right)+\underset{\widehat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\widehat{s},a\right),Z\left(s,a\right)\right)\}.\end{array}$

As demonstrated in [15], we also introduce a parameter $\zeta \in {ℝ}_{>0}$ to threshold the quantile value difference between $\mu$ and $\widehat{\pi }$ , and give this difference a weight $\xi \in {ℝ}_{>0}$ . Then we get a trainable expression of $\beta$ through the process of dual gradient descent:

$\begin{array}{l}\underset{Z}{\min }\underset{\beta \ge 0}{\max }\{\beta \cdot {\mathbb{E}}_{U\left(\tau \right)}\left[\xi \cdot \left[{\mathbb{E}}_{\mathcal{D}\left(s\right)}\log {\displaystyle \sum }_a\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)-{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{F}_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]-\zeta \right]\\ \begin{array}{c}\\ \end{array}+{ℒ}_p\left(Z,{\widehat{\mathcal{T}}}^{{\pi }^k}{\widehat{Z}}^k\right)+\underset{\widehat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\widehat{s},a\right),Z\left(s,a\right)\right)\}.\end{array}$

Since all our experiments take place in continuous-control domains, it is not feasible to list all possible actions as and directly compute $\log {\displaystyle \sum }_a\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)$ . In our implementation, we employ the importance sampling approximation method described in [17], and obtain

$\begin{array}{l}\log {\displaystyle \sum }_a\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)\\ \approx \log \left(\frac{1}{2C}{\displaystyle \sum }_{a_i\sim U\left(\mathcal{A}\right)}^N\left[\frac{\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)}{U\left(\mathcal{A}\right)}\right]+\frac{1}{2C}{\displaystyle \sum }_{a_i\sim \pi \left(a|s\right)}^N\left[\frac{\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)}{\pi \left(a_i|s\right)}\right]\right),\end{array}$ (5)

where $U\left(\mathcal{A}\right)=\text{Uniform}\left(\mathcal{A}\right)$ represents a uniform distribution of actions. Algorithm 7.1 summarizes a single step of the actor and critic updates used by ORDER.

Algorithm 1. ORDER
1:	Hyperparameters: Number of generated quantiles N, number of quantile networks for value functions M, Huber loss threshold $\kappa$ , discount rate $\gamma$ , learning rates ${\eta }_{\text{a}ctor},{\eta }_Z$ , ${\eta }_{\beta }$ , tuning parameter $\alpha ,{\alpha }_1$ , OOD penalty scale $\xi$ , OOD penalty threshold $\zeta$
2:	Parameters: Critic parameters $\theta$ , Actor parameters $\varphi$ , Penalty $\beta$
3:	# Compute distributional TD loss
4:	Get the next action using current policy $a\text{'}\sim \pi \left(\cdot |s\text{'};\varphi \right)$
5:	for $i$ = 1 to $N$ do (For the $m$ -th quantile network)
6:	for $j$ = 1 to $N$ do
7:	${\delta }_{{\tau }_i,{\tau }_j\text{'}}^m=r+\gamma F_{Z\left(s\text{'},a\text{'}\right),\theta {\text{'}}_m}^{-1}\left({\tau }_j\text{'}\right)-F_{Z\left(s,a\right),{\theta }_m}^{-1}\left({\tau }_i\right)$
8:	end for
9:	end for
10:	Computer
11:	# Compute OOD penalty
12:	Sample $i\sim U\left(\left\{1,\dots ,N\right\}\right)$ and use quantile ${\tau }_i$
13:	Estimate according to Equation (5)
14:	Compute
15:	# Update quantile network
16:	Use Equation (1) to add perturbations to the state to obtain $\widehat{s}$ .
17:	Train $\theta$ using Equation (2) by SGD
18:	Update $\theta \leftarrow \theta -{\eta }_Z\nabla \left({ℒ}_Z\left({\theta }_m\right)+\alpha {ℒ}_{\text{s}mooth}\left(s,a;{\theta }_m\right)+{ℒ}_{\text{O}OD}\left({\theta }_m,\beta \right)\right)$
19:	# Update policy network with ${\text{Φ}}_j$ objective
20:	Get new actions with re-parameterized samples $\tilde{a}\sim \pi \left(\cdot |s;\varphi \right)$
21:	Computer ${\text{Φ}}_g\left(s,\tilde{a}\right)$ using $F_{Z\left(s,\tilde{a}\right),\theta }^{-1}\left({\tau }_i\right),i=1,\cdots ,N$
22:	${ℒ}_{\text{actor}}\left(\varphi \right)=\text{log}\left(\pi \left(\tilde{a}|s;\varphi \right)\right)+{\alpha }_1{\text{max}}_{\widehat{s}}{D}_{\text{J}}\left({\pi }_{\varphi }\left(\cdot |s\right)\left|\right|{\pi }_{\varphi }\left(\cdot |\widehat{s}\right)\right)-{\text{min}}_{j=1,\cdots ,M}{\text{Φ}}_g^j\left(s,\tilde{a}\right)$
23:	Update $\varphi \leftarrow \varphi +{\eta }_{\text{a}ctor}\nabla {ℒ}_{\text{a}ctor}\left(\varphi \right)$

B. Proofs

B.1. Proof of Lemma 1

Proof. By the definition of $p$ -Wasserstein distance, we can re-write Equation (4) as

$\begin{array}{l}{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{\displaystyle {\int }_0^1{\left|F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p\text{d}\tau }+\beta \cdot {\mathbb{E}}_{U\left(\tau \right),\mathcal{D}\left(s,a\right)}\left[c_0\left(s,a\right)\cdot F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]\\ +{\mathbb{E}}_{\mathcal{D}\left(s,a\right),\widehat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{\displaystyle {\int }_0^1{\left|F_{Z\left(\widehat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p\text{d}\tau }\\ ={\displaystyle {\int }_0^1{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}}[{\left|F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p+\beta \cdot c_0\left(s,a\right)\cdot F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\\ +{\left|F_{Z\left(\widehat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p]\text{d}\tau .\end{array}$

For arbitrary smooth functions ${\varphi }_{s,a}$ with compact support $\left[V_{\text{min}},V_{\text{max}}\right]$ , we consider a perturbation $G_{s,a}^{\epsilon }\left(\tau \right)=F_{Z\left(s,a\right)}^{-1}\left(\tau \right)+\epsilon \cdot {\varphi }_{s,a}\left(\tau \right)$ of $F_{Z\left(s,a\right)}^{-1}\left(\tau \right)$ , then the above formula can be written as

${\displaystyle {\int }_0^1{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}}\left[{\left|G_{s,a}^{\epsilon }\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p+\beta \cdot c_0\left(s,a\right)\cdot G_{s,a}^{\epsilon }\left(\tau \right)+{\left|G_{\widehat{s},a}^{\epsilon }\left(\tau \right)-G_{s,a}^{\epsilon }\left(\tau \right)\right|}^p\right]\text{d}\tau .$

Then the following equation is obtained considering the derivative of $\epsilon$ at $\epsilon =0$ :

$\begin{array}{l}{\frac{\text{d}}{\text{d}\epsilon }{\displaystyle {\int }_0^1{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}}\left[{\left|G_{s,a}^{\epsilon }\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p+\beta c_0\left(s,a\right)\cdot G_{s,a}^{\epsilon }\left(\tau \right)+{\left|G_{\widehat{s},a}^{\epsilon }\left(\tau \right)-G_{s,a}^{\epsilon }\left(\tau \right)\right|}^p\right]\text{d}\tau |}_{\epsilon =0}\\ ={\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{\displaystyle {\int }_0^1[p{\left|F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right)}\\ +\beta c_0\left(s,a\right)+p{\left|F_{Z\left(\widehat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(\widehat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)]{\varphi }_{s,a}\left(\tau \right)\text{d}\tau .\end{array}$

Owning to some perturbation $G_{s,a}^{\epsilon }$ will cause the objective value to decrease, so the above equation must be equal to 0 for $F_{Z\left(s,a\right)}^{-1}$ . If $\varphi \left(s,a\right)$ does not equal zero for each $s,a$ , since $\varphi \left(s,a\right)$ is arbitrary, it will also cause the above equation to be not equal to 0. Therefore, we obtain

$\begin{array}{l}{\displaystyle {\int }_0^1[p{\left|F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right)}+\beta c_0\left(s,a\right)\\ +p{\left|F_{Z\left(\widehat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(\widehat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)]{\varphi }_{s,a}\left(\tau \right)\text{d}\tau =0.\end{array}$

for all $s,a$ . According to the above term is zero for all $\varphi \left(s,a\right)$ , we have

$\begin{array}{l}p{\left|F_{Z\left(s,a\right)}^{-1}\left(s,a\right)\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right)\\ +\beta c_0\left(s,a\right)+p{\left|F_{Z\left(\widehat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(\widehat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)=0.\end{array}$

According to Assumption 3, the above equation can be converted to

$p{\left|F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right)=\beta c_0\left(s,a\right)\pm {\sigma }^{p-1},$

which holds if and only if

$F_{Z\left(s,a\right)}^{-1}\left(\tau \right)=F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{\widehat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)-c\left(s,a\right),$

where $c\left(s,a\right)={\left|\beta p^{-1}{c}_0\left(s,a\right)\pm {\sigma }^{p-1}\right|}^{\frac{1}{p-1}}\cdot \text{sign}\left(c_0\left(s,a\right)\right)$ . ≤

B.2. Proof of Theorem 1

Lemma 2. For all $\delta \in {ℝ}_{>0}$ , with probability at least $1-\delta$ , for any $Z\in \mathcal{Z}$ and $\left(s,a\right)\in \mathcal{D}$ , we have

${\Vert F_{{\widehat{\mathcal{T}}}^{\pi }Z\left(s,a\right)}-F_{{\mathcal{T}}^{\pi }Z\left(s,a\right)}\Vert }_{\infty }\le \sqrt{\frac{5\left|\mathcal{S}\right|}{n\left(s,a\right)}\text{log}\frac{4\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{\delta }},$(6)

where $n\left(s,a\right)$ represents the number of occurrences of $\left(s,a\right)$ in $\mathcal{D}$ .

Proof. Applying the definition of distributional soft Bellman operator to the cumulative density function, we obtain that

$\begin{array}{l}{F}_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }Z\left(s,a\right)}\left(x\right)-F_{{\mathcal{T}}_{\text{DS}}^{\pi }Z\left(s,a\right)}\left(x\right)\\ ={\displaystyle \sum }_{s^{\prime },a^{\prime }}\widehat{P}\left(s^{\prime }|s,a\right)\pi \left(a^{\prime }|s^{\prime }\right)F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+\widehat{R}\left(s,a\right)}\left(x\right)\\ -{\displaystyle \sum }_{s^{\prime },a^{\prime }}P\left(s^{\prime }|s,a\right)\pi \left(a^{\prime }|s^{\prime }\right)F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+R\left(s,a\right)}\left(x\right).\end{array}$

Adding and subtracting ${\displaystyle {\sum }_{s^{\prime },a^{\prime }}\widehat{P}}\left(s^{\prime }|s,a\right)\pi \left(a^{\prime }|s^{\prime }\right)F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+R\left(s,a\right)}\left(x\right)$ from this expression gives

$\begin{array}{l}{\displaystyle \sum }_{s^{\prime },a^{\prime }}\widehat{P}\left(s^{\prime }|s,a\right)\pi \left(a^{\prime }|s^{\prime }\right)\left(F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+\widehat{R}\left(s,a\right)}\left(x\right)-F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+R\left(s,a\right)}\left(x\right)\right)\\ +{\displaystyle \sum }_{s^{\prime },a^{\prime }}\left(\widehat{P}\left(s^{\prime }|s,a\right)-P\left(s^{\prime }|s,a\right)\right)\pi \left(a^{\prime }|s^{\prime }\right)F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+R\left(s,a\right)}\left(x\right).\end{array}$

We proceed by bounding the two terms in the summation. For the first term,

$\begin{array}{l}{F}_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+\widehat{R}\left(s,a\right)}\left(x\right)-F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+R\left(s,a\right)}\left(x\right)\\ ={\displaystyle \int \left[F_{\widehat{R}\left(s,a\right)}\left(r\right)-F_{R\left(s,a\right)}\left(r\right)\right]\text{d}{F}_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]}\left(x-r\right)}\\ \le {\displaystyle \int \left|F_{\widehat{R}\left(s,a\right)}\left(r\right)-F_{R\left(s,a\right)}\left(r\right)\right|\text{d}{F}_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]}\left(x-r\right)}\\ \le \underset{r}{\text{sup}}\left|F_{\widehat{R}\left(s,a\right)}\left(r\right)-F_{R\left(s,a\right)}\left(r\right)\right|{\displaystyle \int \text{d}{F}_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]}\left(x-r\right)}\\ ={\Vert F_{\widehat{R}\left(s,a\right)}\left(r\right)-F_{R\left(s,a\right)}\left(r\right)\Vert }_{\infty }.\end{array}$

Therefore,

$\begin{array}{l}{\displaystyle \sum }_{s^{\prime },a^{\prime }}\widehat{P}\left(s^{\prime }|s,a\right)\pi \left(a^{\prime }|s^{\prime }\right)\left(F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+\widehat{R}\left(s,a\right)}\left(x\right)-F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+R\left(s,a\right)}\left(x\right)\right)\\ \le {\displaystyle \sum }_{s^{\prime },a^{\prime }}\widehat{P}\left(s^{\prime }|s,a\right)\pi \left(a^{\prime }|s^{\prime }\right){\Vert F_{\widehat{R}\left(s,a\right)}\left(r\right)-F_{R\left(s,a\right)}\left(r\right)\Vert }_{\infty }\\ ={\Vert F_{\widehat{R}\left(s,a\right)}\left(r\right)-F_{R\left(s,a\right)}\left(r\right)\Vert }_{\infty }.\end{array}$

The following derivation process is similar to CODAC [15], so we can finally obtain the above conclusion.

It has been proved in CODAC that if ${\Vert F-G\Vert }_{\infty }\le ϵ$ , then ${\Vert F^{-1}-G^{-1}\Vert }_{\infty }\le ϵ/\zeta$ , where F and $G$ are two cumulative distribution functions (CDFs) with support $\chi$ , and F is $\zeta$ -strongly monotone. Thus, according to Lemma 2, we have

Lemma 3. For any return distributional Z with $\zeta$ -strongly monotone CDF $F_{Z\left(s,a\right)}$ and any $\delta \in {ℝ}_{>0}$ , with probability at least $1-\delta$ , for all $s\in \mathcal{D}$ and $a\in \mathcal{A}$ , we have

${\Vert F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }Z\left(s,a\right)}^{-1}-F_{{\mathcal{T}}^{\pi }Z\left(s,a\right)}^{-1}\Vert }_{\infty }\le \frac{1}{\zeta }\sqrt{\frac{5\left|\mathcal{S}\right|}{n\left(s,a\right)}\text{log}\frac{4\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{\delta }}.$

Let $\text{Δ}\left(s,a\right)=\frac{1}{\zeta }\sqrt{\frac{5\left|\mathcal{S}\right|}{n\left(s,a\right)}\text{log}\frac{4\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{\delta }}$ and followed by Lemma 1

$\begin{array}{c}{F}_{{\tilde{\mathcal{T}}}^{\pi }{Z}^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)=F_{{\widehat{\mathcal{T}}}_{\text{DS}}^{\pi }{Z}^{\pi }\left(s,a\right)}\left(\tau \right)-c\left(s,a\right)\\ \le F_{{\mathcal{T}}^{\pi }{Z}^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)-c\left(s,a\right)+\text{Δ}\left(s,a\right)\\ =F_{Z^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)-c\left(s,a\right)+\text{Δ}\left(s,a\right),\end{array}$

the second step holds by Lemma 2 with probability at least $1-\delta$ . For any $h\in ℝ$ , if $Z$ satisfies $F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\ge F_{{\mathcal{T}}^{\pi }Z\left(s,a\right)}^{-1}\left(\tau \right)+h$ for all $s\in \mathcal{S}$ and $a\in \mathcal{A}$ , then $F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\ge F_{{\mathcal{T}}^{\pi }Z\left(s,a\right)}^{-1}\left(\tau \right)+{\left(1-\gamma \right)}^{-1}h$ $\left(\forall \tau \in \left[0,1\right]\right)$ . Then,

$\begin{array}{c}{F}_{Z^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)\ge F_{{\tilde{\mathcal{T}}}^{\pi }{Z}^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)+c\left(s,a\right)-\Delta \left(s,a\right)\\ \ge F_{{\tilde{\mathcal{T}}}^{\pi }{Z}^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)+\underset{s,a}{\min }\left\{c\left(s,a\right)-\Delta \left(s,a\right)\right\}\\ \ge F_{{\tilde{Z}}^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)+{\left(1-\gamma \right)}^{-1}\underset{s,a}{\min }\left\{c\left(s,a\right)-\Delta \left(s,a\right)\right\}.\end{array}$ (7)

Notice that since for the last term in Equation (7) to be positive, we need

$\beta p^{-1}{c}_0\left(s,a\right)+{\sigma }^{p-1}\ge \text{Δ}{\left(s,a\right)}^{p-1}\left(\forall s,a\right).$

Owning to we have assumed that $c_0\left(s,a\right)>0$ , then it can be equivalent to

$\beta \ge \underset{s,a}{\text{max}}\left\{\frac{p\left(\text{Δ}{\left(s,a\right)}^{p-1}+{\sigma }^{p-1}\right)}{c_0\left(s,a\right)}\right\},$

we thus prove the conclusion of Theorem 1.

C. Experimental Settings

Our experimental procedure largely adheres to [18], and the results of non-distributional methods are directly taken from [18]. For all experiments, we run algorithms for 1000 epochs (1000 training steps each epoch, i.e., 1M gradient steps in total). Then we evaluate them using 10 test episodes in the original environment, which all last 1000 steps long. All benchmark results are averaged over 5 random seeds. The reported results are normalized to D4RL scores that measure how the performance compared with expert score and random score:

$\text{normalized}\text{score}=100\times \frac{\text{score}-\text{random score}}{\text{expert score}-\text{random score}}$ .

D. Implementation Details

We implement ORDER based on DSAC and keep the DSAC-specific hyperparameters the same. These hyperparameters are detailed in Table A1. As with CODAC, we introduce hyperparameters $\beta ,\zeta ,\xi$ (See Appendix A.1). In most cases, $\beta$ is a learnable parameter initialized to 1 with learning rate ${\eta }_{\beta }=3\times {10}^{-4}$ . In a few cases, setting $\beta =1$ throughout the entirety of training, which we indicate by setting $\zeta =-1$ .

Table A1. ORDER backbone hyperparameters.

Hyper-parameter	Value
Discount factor $\gamma$	0.99
Batch size	256
Replay buffer size	1e6
Optimizer	Adam
Minimum steps before training	1e4
Policy network learning rate ${\eta }_{\text{actor}}$	3e−4
Quantile network learning rate ${\eta }_Z$	3e−5
Huber regression threshold $\kappa$	1
Number of quantile fractions N	32
Quantile fraction embedding size	64

Table A2. Hyperparameters of ORDER for the benchmark results.

ss	$\xi$	$\zeta$	${\eta }_{\text{critic}}$	entropy tuning	$\alpha$	${\alpha }_1$
hopper-random	1	10	3e−5	yes	0.0001	0.1
hopper-medium	10	10	3e−4	yes	0.0	0.0
hopper-med-rep	1	10	3e−5	yes	0.0	0.0
hopper-med-exp	10	10	3e−5	no	0.0001	0.1
walker2d-random	1	10	3e−5	yes	0.0001	1.0
walker2d-medium	10	10	3e−5	no	0.0001	1.0
walker2d-med-rep	1	10	3e−5	yes	0.0	0.0
walker2d-med-exp	10	10	3e−5	no	0.0001	1.0
halfCheetah-random	1	10	3e−5	yes	0.0001	0.1
halfCheetah-medium	10	10	3e−5	no	0.0001	0.1
halfCheetah-med-rep	1	10	3e−5	yes	0.0001	0.1
halfCheetah-med-exp	0.1	−1	3e−4	no	0.0	0.0

Since we introduce smoothing techniques to the policy and quantile networks, we also add other hyperparameters. In Equation (2), the weight $\alpha$ for the quantile network smoothing loss ${ℒ}_{\text{smooth}}$ is searched in $\left\{0.0,0.0001\right\}$ . And beyond that, the weight ${\alpha }_1$ of the policy smoothing loss in Equation (3) is searched in $\left\{0.0,0.1,1.0\right\}$ . When training the policy and distribution action-value functions, we randomly sample $n=10$ perturbed observations from a ${\ell }_{\infty }$ ball of norm $ϵ$ and select the one to maximize $D_J\left({\pi }_{\theta }\left(\cdot |s\right)\left|\right|{\pi }_{\theta }\left(\cdot |\widehat{s}\right)\right)$ and ${ℒ}_{\text{smooth}}$ , respectively. For $Z$ smoothing loss in Equation (1), set parameter $\varrho$ to 0.2 for conservative value estimation. All the hyperparameters used in ORDER for the benchmark are listed in Table A2.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1]	Lange, S., Gabel, T. and Riedmiller, M. (2012) Batch Reinforcement Learning. In: Wiering, M. and van Otterlo, M., Eds., Reinforcement Learning, Springer, 45-73.[CrossRef]
[2]	Wang, R., Wu, Y., Salakhutdinov, R. and Kakade, S. (2021) Instabilities of Offline RL with Pretrained Neural Representation. Proceedings of the 38th International Conference on Machine Learning, Virtual, 18-24 July 2021, 10948-10960.
[3]	Nguyen-Tang, T., Yin, M., Gupta, S., Venkatesh, S. and Arora, R. (2023) On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation. Proceedings of the AAAI Conference on Artificial Intelligence, 37, 9310-9318.[CrossRef]
[4]	Diehl, C., Sievernich, T.S., Kruger, M., Hoffmann, F. and Bertram, T. (2023) Uncertainty-Aware Model-Based Offline Reinforcement Learning for Automated Driving. IEEE Robotics and Automation Letters, 8, 1167-1174.[CrossRef]
[5]	Zhang, Z., Mei, H. and Xu, Y. (2023) Continuous-Time Decision Transformer for Healthcare Applications. Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, Valencia, 25-27 April 2023, 6245-6262.
[6]	Singh, B., Kumar, R. and Singh, V.P. (2022) Reinforcement Learning in Robotic Applications: A Comprehensive Survey. Artificial Intelligence Review, 55, 945-990.[CrossRef]
[7]	Haarnoja, T., Zhou, A., Abbeel, P. and Levine, S. (2018) Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the 35th International Conference on Machine Learning, Stockholm, 10-15 July 2018, 1861-1870.
[8]	Ashvin, N., Murtaza, D., Abhishek, G. and Sergey, L. (2020) AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. arXiv: 2006.09359.[CrossRef]
[9]	Fujimoto, S., Meger, D. and Precup, D. (2019) Off-Policy Deep Reinforcement Learning without Exploration. Proceedings of the 36th International Conference on Machine Learning, Long Beach, 9-15 June 2019, 2052-2062.
[10]	Fujimoto, S. and Gu, S.S. (2021) A Minimalist Approach to Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 34, 20132-20145.
[11]	Lyu, J., Ma, X., Li, X. and Lu, Z. (2022) Mildly Conservative Q-Learning for Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 35, 1711-1724.
[12]	Dabney, W., Ostrovski, G., Silver, D. and Munos, R. (2018) Implicit Quantile Networks for Distributional Reinforcement Learning. Proceedings of the 35th International Conference on Machine Learning, Stockholm, 10-15 July 2018, 1096-1105.
[13]	Dabney, W., Rowland, M., Bellemare, M. and Munos, R. (2018) Distributional Reinforcement Learning with Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence, 32, 2892-2901.[CrossRef]
[14]	Ma, X., Xia, L., Zhou, Z., Yang, J. and Zhao, Q. (2020) DSAC: Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning. arXiv: 2004.14547.[CrossRef]
[15]	Ma, Y., Jayaraman, D. and Bastani, O. (2021) Conservative Offline Distributional Reinforcement Learning. Advances in Neural Information Processing Systems, 34, 19235-19247.
[16]	Bai, C., Xiao, T., Zhu, Z., Wang, L., Zhou, F., Garg, A., et al. (2022) Monotonic Quantile Network for Worst-Case Offline Reinforcement Learning. IEEE Transactions on Neural Networks and Learning Systems, 35, 8954-8968.[CrossRef]
[17]	Kumar, A., Zhou, A., Tucker, G. and Levine, S. (2020) Conservative Q-Learning for Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 33, 1179-1191.
[18]	Fu, J., Kumar, A., Nachum, O., Tucker, G. and Levine, S. (2020) D4RL: Datasets for Deep Data-Driven Reinforcement Learning. arXiv: 2004.07219.[CrossRef]
[19]	Bellemare, M.G., Dabney, W. and Munos, R. (2017) A Distributional Perspective on Reinforcement Learning. Proceedings of the 34th International Conference on Machine Learning, Sydney, 6-11 August 2017, 449-458.
[20]	Müller, A. (1997) Integral Probability Metrics and Their Generating Classes of Functions. Advances in Applied Probability, 29, 429-443.[CrossRef]
[21]	Shen, Q., Li, Y., Jiang, H., Wang, Z. and Zhao, T. (2020) Deep Reinforcement Learning with Robust and Smooth Policy. Proceedings of the 37th International Conference on Machine Learning, Virtual, 13-18 July 2020, 8707-8718.
[22]	Koenker, R. and Hallock, K.F. (2001) Quantile Regression. Journal of Economic Perspectives, 15, 143-156.[CrossRef]
[23]	Huber, P.J. (1992) Robust Estimation of a Location Parameter. In: Kotz, S. and Johnson, N.L., Eds., Breakthroughs in Statistics, Springer, 492-518.[CrossRef]
[24]	Wu, Y., Tucker, G. and Nachum, O. (2019) Behavior Regularized Offline Reinforcement Learning. arXiv: 1911.11361.[CrossRef]
[25]	Kidambi, R., Rajeswaran, A., Netrapalli, P. and Joachims. T. (2020) MOReL: Model-Based Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 33, 21810-21823.
[26]	Figueiredo Prudencio, R., Maximo, M.R.O.A. and Colombini, E.L. (2024) A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems. IEEE Transactions on Neural Networks and Learning Systems, 35, 10237-10257.[CrossRef] [PubMed]
[27]	Sutton, R.S. and Barto, A.G. (2018) Reinforcement Learning: An Introduction. MIT Press.
[28]	Puterman, M.L. (2014) Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons.
[29]	Singla, A., Rafferty, A.N., Radanovic, G. and Heffernan, N.T. (2021) Reinforcement Learning for Education: Opportunities and Challenges. arXiv:2107.08828.[CrossRef]
[30]	Agarwal, R., Schuurmans, D. and Norouzi, M. (2020) An Optimistic Perspective on Offline Reinforcement Learning. Proceedings of the 37th International Conference on Machine Learning, Vienna, 13-18 July 2020, 104-114.
[31]	Bai, C., Wang, L., Yang, Z., Deng, Z., Garg, A., Liu, P. and Wang, Z. (2022) Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning. arXiv: 2202.11566.[CrossRef]
[32]	Sohn, K., Lee, H. and Yan, X. (2015) Learning Structured Output Representation Using Deep Conditional Generative Models. Advances in Neural Information Processing Systems, 28, 3483-3491.
[33]	Kumar, A., Fu, J., Soh, M., Tucker, G. and Levine, S. (2019) Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. Advances in Neural Information Processing Systems, 32, 11761-11771.
[34]	Panaganti, K., Xu, Z., Kalathil, D. and Ghavamzadeh, M. (2022) Robust Reinforcement Learning Using Offline Data. Advances in Neural Information Processing Systems, 35, 32211-32224.
[35]	Kostrikov, I., Nair, A. and Levine, S. (2021) Offline Reinforcement Learning with Implicit Q-Learning. arXiv: 2110.06169.[CrossRef]
[36]	Yang, R., Bai, C., Ma, X., Wang, Z., Zhang, C. and Han, L. (2022) RORL: Robust Offline Reinforcement Learning via Conservative Smoothing. Advances in Neural Information Processing Systems, 35, 23851-23866.