Conservative Q-Learning for Offline Reinforcement Learning

Aviral Kumar    Aurick Zhou    George Tucker Affiliation:  UC Berkeley, Google Research, Brain Team aviralk@berkeley.edu    Sergey Levine Affiliation:  UC Berkeley, Google Research, Brain Team aviralk@berkeley.edu

Abstract

摘要

Effectively leveraging large, previously collected datasets in reinforcement learning (RL) is a key challenge for large-scale real-world applications. Offline RL algorithms promise to learn effective policies from previously-collected, static datasets without further interaction. However, in practice, offline RL presents a major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions. In this paper, we propose conservative Q-learning (CQL), which aims to address these limitations by learning a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees. In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward to implement on top of existing deep Q-learning and actor-critic implementations. On both discrete and continuous control domains, we show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return, especially when learning from complex and multi-modal data distributions.

在强化学习（RL）中有效利用大规模、已采集的数据集，是大规模真实世界应用的关键挑战。离线 RL 算法承诺：无需再与环境交互，即可从先前采集的静态数据集中学到有效策略。然而实践中离线 RL 仍很困难——标准异策略 RL 方法常因数据集与所学策略之间的分布偏移而高估价值，尤其是在复杂、多模态数据分布上训练时。本文提出保守 Q 学习（conservative Q-learning，CQL）：学习保守的 Q 函数，使策略在该 Q 函数下的期望价值成为其真实价值的下界。我们从理论上证明，CQL 给出当前策略价值的下界，并可纳入带理论改进保证的策略学习流程。实践上，CQL 在标准 Bellman 误差目标上增加一个简单的 Q 值正则项，可直接叠在现有深度 Q 学习与 actor-critic 实现之上。在离散与连续控制任务上，CQL 大幅优于已有离线 RL 方法，最终回报常可达其 2–5 倍，在复杂、多模态数据分布上尤为明显。

1 Introduction

1 引言

Recent advances in reinforcement learning (RL), especially when combined with expressive deep network function approximators, have produced promising results in domains ranging from robotics [31] to strategy games [4] and recommendation systems [37]. However, applying RL to real-world problems consistently poses practical challenges: in contrast to the kinds of data-driven methods that have been successful in supervised learning [24, 11], RL is classically regarded as an active learning process, where each training run requires active interaction with the environment. Interaction with the real world can be costly and dangerous, and the quantities of data that can be gathered online are substantially lower than the offline datasets that are used in supervised learning [10], which only need to be collected once. Offline RL, also known as batch RL, offers an appealing alternative [12, 17, 32, 3, 29, 57, 36]. Offline RL algorithms learn from large, previously collected datasets, without interaction. This in principle can make it possible to leverage large datasets, but in practice fully offline RL methods pose major technical difficulties, stemming from the distributional shift between the policy that collected the data and the learned policy. This has made current results fall short of the full promise of such methods.

强化学习（RL）近年进展显著，尤其是与表达力强的深度网络函数逼近器结合后，在机器人 [31]、策略游戏 [4] 与推荐系统 [37] 等领域取得了有希望的结果。但把 RL 用到真实世界仍常遇现实障碍：与监督学习中已成功的数据驱动方法 [24, 11] 不同，经典 RL 被看作主动学习过程，每次训练都需要与环境持续交互。与真实世界交互可能昂贵且危险，能在线采集的数据量也远少于监督学习中一次性采集、可反复使用的离线数据集 [10]。离线 RL（亦称 batch RL）提供了有吸引力的替代路径 [12, 17, 32, 3, 29, 57, 36]：从大规模已采集数据集学习、无需交互。原则上这能利用大数据，但实践中完全离线方法面临重大技术困难，根源在于采集数据的策略与所学策略之间的分布偏移，使得现有结果仍未兑现该方法的全部承诺。

Directly utilizing existing value-based off-policy RL algorithms in an offline setting generally results in poor performance, due to issues with bootstrapping from out-of-distribution actions [32, 17] and overfitting [15, 32, 3]. This typically manifests as erroneously optimistic value function estimates. If we can instead learn a conservative estimate of the value function, which provides a lower bound on the true values, this overestimation problem could be addressed. In fact, because policy evaluation and improvement typically only use the value of the policy, we can learn a less conservative lower bound Q-function, such that only the expected value of Q-function under the policy is lower-bounded, as opposed to a point-wise lower bound. We propose a novel method for learning such conservative Q-functions via a simple modification to standard value-based RL algorithms. The key idea behind our method is to minimize values under an appropriately chosen distribution over state-action tuples, and then further tighten this bound by also incorporating a maximization term over the data distribution.

直接把现有基于价值的异策略 RL 算法用在离线设定，通常表现很差，原因包括从分布外动作自举 [32, 17] 以及过拟合 [15, 32, 3]，典型表现是价值函数被错误地乐观估计。若能学到保守的价值估计、对真实价值提供下界，就能缓解这种高估。事实上，策略评估与改进通常只用到策略的价值，因此可学一个不那么保守的下界 Q 函数：只需策略下 Q 函数的期望被下界约束，而不必点态下界。我们提出一种通过对标准价值型 RL 算法做简单修改来学习此类保守 Q 函数的新方法。核心想法是：在恰当选取的状态—动作分布上最小化价值，并再引入数据分布上的最大化项以收紧该下界。

Our primary contribution is an algorithmic framework, which we call conservative Q-learning (CQL), for learning conservative, lower-bound estimates of the value function, by regularizing the Q-values during training. Our theoretical analysis of CQL shows that only the expected value of this Q-function under the policy lower-bounds the true policy value, preventing extra under-estimation that can arise with point-wise lower-bounded Q-functions, that have typically been explored in the opposite context in exploration literature [48, 28]. We also empirically demonstrate the robustness of our approach to Q-function estimation error. Our practical algorithm uses these conservative estimates for policy evaluation and offline RL. CQL can be implemented with less than 20 lines of code on top of a number of standard, online RL algorithms [21, 9], simply by adding the CQL regularization terms to the Q-function update. In our experiments, we demonstrate the efficacy of CQL for offline RL, in domains with complex dataset compositions, where prior methods are typically known to perform poorly [14] and domains with high-dimensional visual inputs [5, 3]. CQL outperforms prior methods by as much as 2-5x on many benchmark tasks, and is the only method that can outperform simple behavioral cloning on a number of realistic datasets collected from human interaction.

本文主要贡献是算法框架保守 Q 学习（CQL）：通过训练时对 Q 值正则，学到价值函数的保守下界估计。对 CQL 的理论分析表明，仅该 Q 函数在策略下的期望价值下界真实策略价值，从而避免点态下界 Q 函数可能带来的额外低估——后者在探索文献中常以相反动机出现 [48, 28]。我们也经验地展示了方法对 Q 函数估计误差的稳健性。实用算法用这些保守估计做策略评估与离线 RL。CQL 可在多种标准在线 RL 算法 [21, 9] 上用不到 20 行代码实现，只需在 Q 函数更新中加入 CQL 正则项。实验中，我们在复杂数据集构成（先验方法通常较差 [14]）以及高维视觉输入 [5, 3] 等领域验证了 CQL 的有效性。CQL 在许多基准任务上最多可领先先验方法 2–5 倍，并且是在若干真实人类交互数据集上唯一能超过简单行为克隆的方法。

2 Preliminaries

2 预备知识

The goal in reinforcement learning is to learn a policy that maximizes the expected cumulative discounted reward in a Markov decision process (MDP), which is defined by a tuple $(\mathcal{S},\mathcal{A},T,r,\gamma)$. $\mathcal{S},\mathcal{A}$ represent state and action spaces, $T(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})$ and $r(\mathbf{s},\mathbf{a})$ represent the dynamics and reward function, and $\gamma\in(0,1)$ represents the discount factor. ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$ represents the behavior policy, $\mathcal{D}$ is the dataset, and $d^{\pi_{\beta}}(\mathbf{s})$ is the discounted marginal state-distribution of ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$. The dataset $\mathcal{D}$ is sampled from $d^{\pi_{\beta}}(\mathbf{s}){\pi_{\beta}}(\mathbf{a}|\mathbf{s})$. On all states $\mathbf{s}\in\mathcal{D}$, let $\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s}_{)}:=\frac{\sum_{\mathbf{s},\mathbf{a}\in\mathcal{D}}\mathbf{1}[\mathbf{s}=\mathbf{s},\mathbf{a}=\mathbf{a}]}{\sum_{\mathbf{s}\in\mathcal{D}}\mathbf{1}[\mathbf{s}=\mathbf{s}]}$ denote the empirical behavior policy, at that state. We assume that the rewards $r$ satisfy: $|r(\mathbf{s},\mathbf{a})|\leq R_{\max}$.

强化学习的目标是在马尔可夫决策过程（MDP）中学习最大化期望累计折扣回报的策略；MDP 由元组 $(\mathcal{S},\mathcal{A},T,r,\gamma)$ 定义。$\mathcal{S},\mathcal{A}$ 表示状态与动作空间，$T(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})$ 与 $r(\mathbf{s},\mathbf{a})$ 表示动力学与奖励函数，$\gamma\in(0,1)$ 为折扣因子。${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$ 表示行为策略，$\mathcal{D}$ 为数据集，$d^{\pi_{\beta}}(\mathbf{s})$ 为 ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$ 的折扣边际状态分布。数据集 $\mathcal{D}$ 从 $d^{\pi_{\beta}}(\mathbf{s}){\pi_{\beta}}(\mathbf{a}|\mathbf{s})$ 采样。对所有 $\mathbf{s}\in\mathcal{D}$，令 $\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s}_{)}:=\frac{\sum_{\mathbf{s},\mathbf{a}\in\mathcal{D}}\mathbf{1}[\mathbf{s}=\mathbf{s},\mathbf{a}=\mathbf{a}]}{\sum_{\mathbf{s}\in\mathcal{D}}\mathbf{1}[\mathbf{s}=\mathbf{s}]}$ 表示该状态下的经验行为策略。我们假设奖励满足 $|r(\mathbf{s},\mathbf{a})|\leq R_{\max}$。

Off-policy RL algorithms based on dynamic programming maintain a parametric Q-function $Q_{\theta}(s,a)$ and, optionally, a parametric policy, $\pi_{\phi}(a|s)$. Q-learning methods train the Q-function by iteratively applying the Bellman optimality operator $\mathcal{B}^{*}Q(\mathbf{s},\mathbf{a})=r(\mathbf{s},\mathbf{a})+\gamma\mathbb{E}_{\mathbf{s}^{\prime}\sim P(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})}[\max_{\mathbf{a}^{\prime}}Q(\mathbf{s}^{\prime},\mathbf{a}^{\prime})]$, and use exact or an approximate maximization scheme, such as CEM [31] to recover the greedy policy. In an actor-critic algorithm, a separate policy is trained to maximize the Q-value. Actor-critic methods alternate between computing $Q^{\pi}$ via (partial) policy evaluation, by iterating the Bellman operator, $\mathcal{B}^{\pi}Q=r+\gamma P^{\pi}Q$, where $P^{\pi}$ is the transition matrix coupled with the policy: $P^{\pi}Q(\mathbf{s},\mathbf{a})=\mathbb{E}_{\mathbf{s}^{\prime}\sim T(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a}),\mathbf{a}^{\prime}\sim\pi(\mathbf{a}^{\prime}|\mathbf{s}^{\prime})}\left[Q(\mathbf{s}^{\prime},\mathbf{a}^{\prime})\right],$ and improving the policy $\pi(\mathbf{a}|\mathbf{s})$ by updating it towards actions that maximize the expected Q-value. Since $\mathcal{D}$ typically does not contain all possible transitions $(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})$, the policy evaluation step actually uses an empirical Bellman operator that only backs up a single sample. We denote this operator $\hat{\mathcal{B}}^{\pi}$. Given a dataset $\mathcal{D}=\{(\mathbf{s},\mathbf{a},r\mathbf{s}^{\prime})\}$ of tuples from trajectories collected under a behavior policy ${\pi_{\beta}}$:

基于动态规划的异策略 RL 算法维护参数化 Q 函数 $Q_{\theta}(s,a)$，并可选择维护参数化策略 $\pi_{\phi}(a|s)$。Q 学习方法通过迭代应用 Bellman 最优算子 $\mathcal{B}^{*}Q(\mathbf{s},\mathbf{a})=r(\mathbf{s},\mathbf{a})+\gamma\mathbb{E}_{\mathbf{s}^{\prime}\sim P(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})}[\max_{\mathbf{a}^{\prime}}Q(\mathbf{s}^{\prime},\mathbf{a}^{\prime})]$ 训练 Q 函数，并用精确或近似最大化（如 CEM [31]）恢复贪心策略。在 actor-critic 算法中，单独训练策略以最大化 Q 值。Actor-critic 方法在以下两者间交替：通过（部分）策略评估计算 $Q^{\pi}$，即迭代 Bellman 算子 $\mathcal{B}^{\pi}Q=r+\gamma P^{\pi}Q$，其中 $P^{\pi}$ 是与策略耦合的转移矩阵：$P^{\pi}Q(\mathbf{s},\mathbf{a})=\mathbb{E}_{\mathbf{s}^{\prime}\sim T(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a}),\mathbf{a}^{\prime}\sim\pi(\mathbf{a}^{\prime}|\mathbf{s}^{\prime})}\left[Q(\mathbf{s}^{\prime},\mathbf{a}^{\prime})\right]$；以及朝最大化期望 Q 值的动作更新策略 $\pi(\mathbf{a}|\mathbf{s})$。由于 $\mathcal{D}$ 通常不含全部可能转移 $(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})$，策略评估实际使用只回传单一样本的经验 Bellman 算子，记为 $\hat{\mathcal{B}}^{\pi}$。给定由行为策略 ${\pi_{\beta}}$ 采集轨迹得到的数据集 $\mathcal{D}=\{(\mathbf{s},\mathbf{a},r\mathbf{s}^{\prime})\}$：

$$\displaystyle\hat{Q}^{k+1}\leftarrow \\ \displaystyle\arg\min_{Q}\mathbb{E}_{\mathbf{s},\mathbf{a},\mathbf{s}^{\prime}\sim\mathcal{D}}\left[\left((r(\mathbf{s},\mathbf{a})+\gamma\mathbb{E}_{\mathbf{a}^{\prime}\sim\hat{\pi}^{k}(\mathbf{a}^{\prime}|\mathbf{s}^{\prime})}[\hat{Q}^{k}(\mathbf{s}^{\prime},\mathbf{a}^{\prime})])-Q(\mathbf{s},\mathbf{a})\right)^{2}\right]~\text{(policy evaluation)} \\ \displaystyle\hat{\pi}^{k+1}\leftarrow \\ \displaystyle\arg\max_{\pi}\mathbb{E}_{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim\pi^{k}(\mathbf{a}|\mathbf{s})}\left[\hat{Q}^{k+1}(\mathbf{s},\mathbf{a})\right]~~~\text{(policy improvement)}$$

Offline RL algorithms based on this basic recipe suffer from action distribution shift [32, 62, 29, 36] during training, because the target values for Bellman backups in policy evaluation use actions sampled from the learned policy, $\pi^{k}$, but the Q-function is trained only on actions sampled from the behavior policy that produced the dataset $\mathcal{D}$, ${\pi_{\beta}}$. Since $\pi$ is trained to maximize Q-values, it may be biased towards out-of-distribution (OOD) actions with erroneously high Q-values. In standard RL, such errors can be corrected by attempting an action in the environment and observing its actual value. However, the inability to interact with the environment makes it challenging to deal with Q-values for OOD actions in offline RL. Typical offline RL methods [32, 29, 62, 57] mitigate this problem by constraining the learned policy [36] away from OOD actions. Note that Q-function training in offline RL does not suffer from state distribution shift, as the Bellman backup never queries the Q-function on out-of-distribution states. However, the policy may suffer from state distribution shift at test time.

基于这一基本配方的离线 RL 算法在训练时会遭遇动作分布偏移 [32, 62, 29, 36]：策略评估中 Bellman 回传的目标值使用来自所学策略 $\pi^{k}$ 的动作，而 Q 函数只在产生数据集 $\mathcal{D}$ 的行为策略 ${\pi_{\beta}}$ 的动作上训练。由于 $\pi$ 被训练来最大化 Q 值，它可能偏向带有错误偏高 Q 值的分布外（OOD）动作。标准 RL 中，这类误差可通过在环境中尝试动作并观察真实价值来纠正；但无法与环境交互时，处理 OOD 动作的 Q 值就很困难。典型离线 RL 方法 [32, 29, 62, 57] 通过约束所学策略 [36] 远离 OOD 动作来缓解。注意离线 RL 中 Q 函数训练本身不受状态分布偏移影响，因为 Bellman 回传从不在分布外状态上查询 Q 函数；但策略在测试时仍可能遭遇状态分布偏移。

3 The Conservative Q-Learning (CQL) Framework

3 保守 Q 学习（CQL）框架

In this section, we develop a conservative Q-learning (CQL) algorithm, such that the expected value of a policy under the learned Q-function lower-bounds its true value. A lower bound on the Q-value prevents the over-estimation that is common in offline RL settings due to OOD actions and function approximation error [36, 32]. We use the term CQL to refer broadly to both Q-learning methods and actor-critic methods, though the latter also use an explicit policy. We start by focusing on the policy evaluation step in CQL, which can be used by itself as an off-policy evaluation procedure, or integrated into a complete offline RL algorithm, as we will discuss in Section 3.2.

本节给出保守 Q 学习（CQL）算法，使策略在所学 Q 函数下的期望价值下界其真实价值。Q 值下界可抑制离线 RL 中因 OOD 动作与函数逼近误差而导致的常见高估 [36, 32]。我们用 CQL 统称 Q 学习方法与 actor-critic 方法（后者还使用显式策略）。先聚焦 CQL 中的策略评估步骤：它可单独用作异策略评估，也可并入完整离线 RL 算法（见第 3.2 节）。

3.1 Conservative Off-Policy Evaluation

We aim to estimate the value $V^{\pi}(\mathbf{s})$ of a target policy $\pi$ given access to a dataset, $\mathcal{D}$, generated by following a behavior policy ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$. Because we are interested in preventing overestimation of the policy value, we learn a conservative, lower-bound Q-function by additionally minimizing Q-values alongside a standard Bellman error objective. Our choice of penalty is to minimize the expected Q-value under a particular distribution of state-action pairs, $\mu(\mathbf{s},\mathbf{a})$. Since standard Q-function training does not query the Q-function value at unobserved states, but queries the Q-function at unseen actions, we restrict $\mu$ to match the state-marginal in the dataset, such that $\mu(\mathbf{s},\mathbf{a})=d^{\pi_{\beta}}(\mathbf{s})\mu(\mathbf{a}|\mathbf{s})$. This gives rise to the iterative update for training the Q-function, as a function of a tradeoff factor $\alpha$:

3.1 保守异策略评估

给定由行为策略 ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$ 生成的数据集 $\mathcal{D}$，我们希望估计目标策略 $\pi$ 的价值 $V^{\pi}(\mathbf{s})$。为防止策略价值被高估，我们在标准 Bellman 误差目标之外额外最小化 Q 值，从而学到保守的下界 Q 函数。惩罚选取为：在特定状态—动作分布 $\mu(\mathbf{s},\mathbf{a})$ 下最小化期望 Q 值。标准 Q 训练不会在未观测状态上查询 Q，但会在未见动作上查询，因此将 $\mu$ 限制为与数据集状态边际一致：$\mu(\mathbf{s},\mathbf{a})=d^{\pi_{\beta}}(\mathbf{s})\mu(\mathbf{a}|\mathbf{s})$。由此得到依赖折中因子 $\alpha$ 的 Q 函数迭代更新：

$$\small{\hat{Q}^{k+1}\leftarrow\arg\min_{Q}~\alpha~\mathbb{E}_{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim\mu(\mathbf{a}|\mathbf{s})}\left[Q(\mathbf{s},\mathbf{a})\right]+\frac{1}{2}~\mathbb{E}_{\mathbf{s},\mathbf{a}\sim\mathcal{D}}\left[\left(Q(\mathbf{s},\mathbf{a})-\hat{\mathcal{B}}^{\pi}\hat{Q}^{k}(\mathbf{s},\mathbf{a})\right)^{2}\right],} \tag{1}$$

In Theorem 3.1, we show that the resulting Q-function, $\hat{Q}^{\pi}\coloneqq\lim_{k\rightarrow\infty}\hat{Q}^{k}$, lower-bounds $Q^{\pi}$ at all $(\mathbf{s},\mathbf{a})$. However, we can substantially tighten this bound if we are only interested in estimating $V^{\pi}(\mathbf{s})$. If we only require that the expected value of the $\hat{Q}^{\pi}$ under $\pi(\mathbf{a}|\mathbf{s})$ lower-bound $V^{\pi}$, we can improve the bound by introducing an additional Q-value maximization term under the data distribution, ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$, resulting in the iterative update (changes from Equation 1 in red):

定理 3.1 表明，所得 Q 函数 $\hat{Q}^{\pi}\coloneqq\lim_{k\rightarrow\infty}\hat{Q}^{k}$ 在所有 $(\mathbf{s},\mathbf{a})$ 上对 $Q^{\pi}$ 构成下界。若我们只关心估计 $V^{\pi}(\mathbf{s})$，则可显著收紧该界：只需 $\hat{Q}^{\pi}$ 在 $\pi(\mathbf{a}|\mathbf{s})$ 下的期望下界 $V^{\pi}$，便可在数据分布 ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$ 上再引入 Q 值最大化项，得到如下迭代更新（相对式 1 的改动标红）：

$$\small{\hat{Q}^{k+1}\leftarrow\arg\min_{Q}~~\alpha\cdot\left(\mathbb{E}{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim\mu(\mathbf{a}|\mathbf{s})}\left[Q(\mathbf{s},\mathbf{a})\right]-{\color[rgb]{1,0,0}\mathbb{E}{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim\hat{\pi}{\beta}(\mathbf{a}|\mathbf{s})}\left[Q(\mathbf{s},\mathbf{a})\right]}\right)}\
\small{+\frac{1}{2}~\mathbb{E}{\mathbf{s},\mathbf{a},\mathbf{s}^{\prime}\sim\mathcal{D}}\left[\left(Q(\mathbf{s},\mathbf{a})-\hat{\mathcal{B}}^{\pi}\hat{Q}^{k}(\mathbf{s},\mathbf{a})\right)^{2}\right]}. \tag{2}$$

In Theorem 3.2, we show that, while the resulting Q-value $\hat{Q}^{\pi}$ may not be a point-wise lower-bound, we have $\mathbb{E}_{\pi(\mathbf{a}|\mathbf{s})}[\hat{Q}^{\pi}(\mathbf{s},\mathbf{a})]\leq V^{\pi}(\mathbf{s})$ when $\mu(\mathbf{a}|\mathbf{s})=\pi(\mathbf{a}|\mathbf{s})$. Intuitively, since Equation 2 maximizes Q-values under the behavior policy $\hat{\pi}_{\beta}$, Q-values for actions that are likely under $\hat{\pi}_{\beta}$ might be overestimated, and hence $\hat{Q}^{\pi}$ may not lower-bound $Q^{\pi}$ pointwise. While in principle the maximization term can utilize other distributions besides $\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})$, we prove in Appendix D.2 that the resulting value is not guaranteed to be a lower bound for other distribution besides $\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})$.

定理 3.2 表明：所得 $\hat{Q}^{\pi}$ 未必点态下界，但当 $\mu(\mathbf{a}|\mathbf{s})=\pi(\mathbf{a}|\mathbf{s})$ 时有 $\mathbb{E}_{\pi(\mathbf{a}|\mathbf{s})}[\hat{Q}^{\pi}(\mathbf{s},\mathbf{a})]\leq V^{\pi}(\mathbf{s})$。直观上，式 2 在行为策略 $\hat{\pi}_{\beta}$ 下最大化 Q 值，因此在 $\hat{\pi}_{\beta}$ 下较可能的动作上 Q 值可能被高估，故 $\hat{Q}^{\pi}$ 未必点态下界 $Q^{\pi}$。最大化项原则上可用 $\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})$ 以外的分布，但附录 D.2 证明：除 $\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})$ 外，其他分布不能保证所得值为下界。

Theoretical analysis. We first note that Equations 1 and 2 use the empirical Bellman operator, $\hat{\mathcal{B}}^{\pi}$, instead of the actual Bellman operator, $\mathcal{B}^{\pi}$. Following [49, 28, 47], we use concentration properties of $\hat{\mathcal{B}}^{\pi}$ to control this error. Formally, for all $\mathbf{s},\mathbf{a}\in\mathcal{D}$, with probability $\geq 1-\delta$, $|\hat{\mathcal{B}}^{\pi}-\mathcal{B}^{\pi}|(\mathbf{s},\mathbf{a})\leq\frac{C_{r,T,\delta}}{\sqrt{|\mathcal{D}(\mathbf{s},\mathbf{a})|}}$, where $C_{r,T,\delta}$ is a constant dependent on the concentration properties (variance) of $r(\mathbf{s},\mathbf{a})$ and $T(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})$, and $\delta\in(0,1)$ (see Appendix D.3 for more details). For simplicity in the derivation, we assume that $\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})>0,\forall\mathbf{a}\in\mathcal{A}~,\forall\mathbf{s}\in\mathcal{D}$. $\frac{1}{\sqrt{|\mathcal{D}|}}$ denotes a vector of size $|\mathcal{S}||\mathcal{A}|$ containing square root inverse counts for each state-action pair, except when $\mathcal{D}(\mathbf{s},\mathbf{a})=0$, in which case the corresponding entry is a very large but finite value $\delta\geq\frac{2R_{\max}}{1-\gamma}$. Now, we show that the conservative Q-function learned by iterating Equation 1 lower-bounds the true Q-function. Proofs can be found in Appendix C.

理论分析。 首先，式 1 与 2 使用经验 Bellman 算子 $\hat{\mathcal{B}}^{\pi}$，而非真实算子 $\mathcal{B}^{\pi}$。沿用 [49, 28, 47]，我们用 $\hat{\mathcal{B}}^{\pi}$ 的集中性控制该误差。形式上，对所有 $\mathbf{s},\mathbf{a}\in\mathcal{D}$，以概率 $\geq 1-\delta$ 有 $|\hat{\mathcal{B}}^{\pi}-\mathcal{B}^{\pi}|(\mathbf{s},\mathbf{a})\leq\frac{C_{r,T,\delta}}{\sqrt{|\mathcal{D}(\mathbf{s},\mathbf{a})|}}$，其中 $C_{r,T,\delta}$ 依赖于 $r(\mathbf{s},\mathbf{a})$ 与 $T(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})$ 的集中性（方差），$\delta\in(0,1)$（细节见附录 D.3）。推导中为简洁起见，假设 $\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})>0,\forall\mathbf{a}\in\mathcal{A}~,\forall\mathbf{s}\in\mathcal{D}$。$\frac{1}{\sqrt{|\mathcal{D}|}}$ 表示长度为 $|\mathcal{S}||\mathcal{A}|$、各状态—动作对计数平方根倒数的向量；当 $\mathcal{D}(\mathbf{s},\mathbf{a})=0$ 时，对应分量取很大但有限的值 $\delta\geq\frac{2R_{\max}}{1-\gamma}$。下面证明迭代式 1 学到的保守 Q 函数下界真实 Q 函数。证明见附录 C。

Theorem 3.1.

For any $\mu(\mathbf{a}|\mathbf{s})$ with $\supp\mu\subset\supp\hat{\pi}_{\beta}$, with probability $\geq 1-\delta$, $\hat{Q}^{\pi}$ (the Q-function obtained by iterating Equation 1) satisifies:

定理 3.1。

对任意满足 $\supp\mu\subset\supp\hat{\pi}_{\beta}$ 的 $\mu(\mathbf{a}|\mathbf{s})$，以概率 $\geq 1-\delta$，由迭代式 1 得到的 $\hat{Q}^{\pi}$ 满足：

$$\forall\mathbf{s}\in\mathcal{D},\mathbf{a},~~\hat{Q}^{\pi}(s,a)\leq Q^{\pi}(\mathbf{s},\mathbf{a})-\alpha\left[\left(I-\gamma P^{\pi}\right)^{-1}\frac{\mu}{\hat{\pi}_{\beta}}\right](\mathbf{s},\mathbf{a})+\left[(I-\gamma P^{\pi})^{-1}\frac{C_{r,T,\delta}R_{\max}}{(1-\gamma)\sqrt{|\mathcal{D}|}}\right](\mathbf{s},\mathbf{a}).$$

Thus, if $\alpha$ is sufficiently large, then $\hat{Q}^{\pi}(\mathbf{s},\mathbf{a})\leq Q^{\pi}(\mathbf{s},\mathbf{a}),\forall\mathbf{s}\in\mathcal{D},\mathbf{a}$. When $\hat{\mathcal{B}}^{\pi}=\mathcal{B}^{\pi}$, any $\alpha>0$ guarantees $\hat{Q}^{\pi}(\mathbf{s},\mathbf{a})\leq Q^{\pi}(\mathbf{s},\mathbf{a}),\forall\mathbf{s}\in\mathcal{D},\mathbf{a}\in\mathcal{A}$.

因此，若 $\alpha$ 足够大，则 $\hat{Q}^{\pi}(\mathbf{s},\mathbf{a})\leq Q^{\pi}(\mathbf{s},\mathbf{a}),\forall\mathbf{s}\in\mathcal{D},\mathbf{a}$。当 $\hat{\mathcal{B}}^{\pi}=\mathcal{B}^{\pi}$ 时，任意 $\alpha>0$ 都保证 $\hat{Q}^{\pi}(\mathbf{s},\mathbf{a})\leq Q^{\pi}(\mathbf{s},\mathbf{a}),\forall\mathbf{s}\in\mathcal{D},\mathbf{a}\in\mathcal{A}$。

Next, we show that Equation 2 lower-bounds the expected value under the policy $\pi$, when $\mu=\pi$. We also show that Equation 2 does not lower-bound the Q-value estimates pointwise. For this result, we abuse notation and assume that $\frac{1}{\sqrt{|\mathcal{D}|}}$ refers to a vector of inverse square root of only state counts, with a similar correction as before used to handle the entries of this vector at states with zero counts.

接下来证明：当 $\mu=\pi$ 时，式 2 下界策略 $\pi$ 下的期望价值；我们也说明式 2 并不点态下界 Q 值估计。对此结果，我们滥用记号，令 $\frac{1}{\sqrt{|\mathcal{D}|}}$ 仅表示状态计数平方根倒数的向量，并对零计数状态沿用前述修正。

Theorem 3.2(Equation 2 results in a tighter lower bound).

The value of the policy under the Q-function from Equation 2, $\hat{V}^{\pi}(\mathbf{s})=\mathbb{E}_{\pi(\mathbf{a}|\mathbf{s})}[\hat{Q}^{\pi}(\mathbf{s},\mathbf{a})]$, lower-bounds the true value of the policy obtained via exact policy evaluation, $V^{\pi}(\mathbf{s})=\mathbb{E}_{\pi(\mathbf{a}|\mathbf{s})}[Q^{\pi}(\mathbf{s},\mathbf{a})]$, when $\mu=\pi$, according to:

定理 3.2（式 2 给出更紧的下界）。

当 $\mu=\pi$ 时，式 2 所得 Q 函数下的策略价值 $\hat{V}^{\pi}(\mathbf{s})=\mathbb{E}_{\pi(\mathbf{a}|\mathbf{s})}[\hat{Q}^{\pi}(\mathbf{s},\mathbf{a})]$，下界经由精确策略评估得到的真实策略价值 $V^{\pi}(\mathbf{s})=\mathbb{E}_{\pi(\mathbf{a}|\mathbf{s})}[Q^{\pi}(\mathbf{s},\mathbf{a})]$，具体为：

$$\forall\mathbf{s}\in\mathcal{D},~~\hat{V}^{\pi}(\mathbf{s})\leq V^{\pi}(\mathbf{s})-\alpha\left[\left(I-\gamma P^{\pi}\right)^{-1}\mathbb{E}_{\pi}\left[\frac{\pi}{\hat{\pi}_{\beta}}-1\right]\right](\mathbf{s})+\left[(I-\gamma P^{\pi})^{-1}\frac{C_{r,T,\delta}R_{\max}}{(1-\gamma)\sqrt{|\mathcal{D}|}}\right](\mathbf{s}).$$

Thus, if $\alpha>\frac{C_{r,T}R_{\max}}{1-\gamma}\cdot\max_{\mathbf{s}\in\mathcal{D}}\frac{1}{|\sqrt{|\mathcal{D}(\mathbf{s})|}}\cdot\left[\sum_{\mathbf{a}}\pi(\mathbf{a}|\mathbf{s})(\frac{\pi(\mathbf{a}|\mathbf{s})}{\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s}))}-1)\right]^{-1}$ ,  $\forall\mathbf{s}\in\mathcal{D},~\hat{V}^{\pi}(\mathbf{s})\leq{V}^{\pi}(\mathbf{s})$, with probability $\geq 1-\delta$. When $\hat{\mathcal{B}}^{\pi}={\mathcal{B}}^{\pi}$, then any $\alpha>0$ guarantees $\hat{V}^{\pi}(\mathbf{s})\leq V^{\pi}(\mathbf{s}),\forall\mathbf{s}\in\mathcal{D}$.

因此，若 $\alpha>\frac{C_{r,T}R_{\max}}{1-\gamma}\cdot\max_{\mathbf{s}\in\mathcal{D}}\frac{1}{|\sqrt{|\mathcal{D}(\mathbf{s})|}}\cdot\left[\sum_{\mathbf{a}}\pi(\mathbf{a}|\mathbf{s})(\frac{\pi(\mathbf{a}|\mathbf{s})}{\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s}))}-1)\right]^{-1}$ ，则以概率 $\geq 1-\delta$ 对 $\forall\mathbf{s}\in\mathcal{D}$ 有 $\hat{V}^{\pi}(\mathbf{s})\leq{V}^{\pi}(\mathbf{s})$。当 $\hat{\mathcal{B}}^{\pi}={\mathcal{B}}^{\pi}$ 时，任意 $\alpha>0$ 保证 $\hat{V}^{\pi}(\mathbf{s})\leq V^{\pi}(\mathbf{s}),\forall\mathbf{s}\in\mathcal{D}$。

The analysis presented above assumes that no function approximation is used in the Q-function, meaning that each iterate can be represented exactly. We can further generalize the result in Theorem 3.2 to the case of both linear function approximators and non-linear neural network function approximators, where the latter builds on the neural tangent kernel (NTK) framework [27]. Due to space constraints, we present these results in Theorem D.1 and Theorem D.2 in Appendix D.1.

以上分析假定 Q 函数未使用函数逼近，即每次迭代可精确表示。我们可将定理 3.2 推广到线性函数逼近器与非线性神经网络函数逼近器；后者建立在神经正切核（NTK）框架 [27] 上。因篇幅所限，结果见附录 D.1 的定理 D.1 与定理 D.2。

In summary, we showed that the basic CQL evaluation in Equation 1 learns a Q-function that lower-bounds the true Q-function $Q^{\pi}$, and the evaluation in Equation 2 provides a tighter lower bound on the expected Q-value of the policy $\pi$. For suitable $\alpha$, both bounds hold under sampling error and function approximation. We also note that as more data becomes available and $|\mathcal{D}(\mathbf{s},\mathbf{a})|$ increases, the theoretical value of $\alpha$ that is needed to guarantee a lower bound decreases, which indicates that in the limit of infinite data, a lower bound can be obtained by using extremely small values of $\alpha$. Next, we will extend on this result into a complete RL algorithm.

小结：我们证明式 1 的基本 CQL 评估学到下界真实 $Q^{\pi}$ 的 Q 函数，而式 2 的评估对策略 $\pi$ 的期望 Q 值给出更紧的下界。对合适的 $\alpha$，两界在采样误差与函数逼近下均成立。还注意到：随着数据增多、$|\mathcal{D}(\mathbf{s},\mathbf{a})|$ 增大，保证下界所需的理论 $\alpha$ 减小；在无限数据极限下，可用极小的 $\alpha$ 得到下界。下面把该结果扩展为完整 RL 算法。

3.2 Conservative Q-Learning for Offline RL

We now present a general approach for offline policy learning, which we refer to as conservative Q-learning (CQL). As discussed in Section 3.1, we can obtain Q-values that lower-bound the value of a policy $\pi$ by solving Equation 2 with $\mu=\pi$. How should we utilize this for policy optimization? We could alternate between performing full off-policy evaluation for each policy iterate, $\hat{\pi}^{k}$, and one step of policy improvement. However, this can be computationally expensive. Alternatively, since the policy $\hat{\pi}^{k}$ is typically derived from the Q-function, we could instead choose $\mu(\mathbf{a}|\mathbf{s})$ to approximate the policy that would maximize the current Q-function iterate, thus giving rise to an online algorithm.

3.2 面向离线 RL 的保守 Q 学习

我们给出离线策略学习的一般方法，称为保守 Q 学习（CQL）。如第 3.1 节所述，取 $\mu=\pi$ 求解式 2，可得到下界策略 $\pi$ 价值的 Q 值。如何用于策略优化？可以在每个策略迭代 $\hat{\pi}^{k}$ 上做完整异策略评估，再做一步策略改进，但计算可能昂贵。或者，由于 $\hat{\pi}^{k}$ 通常由 Q 函数导出，可令 $\mu(\mathbf{a}|\mathbf{s})$ 近似会最大化当前 Q 迭代的策略，从而得到在线式算法。

We can formally capture such online algorithms by defining a family of optimization problems over $\mu(\mathbf{a}|\mathbf{s})$, presented below, with modifications from Equation 2 marked in red. An instance of this family is denoted by CQL($\mathcal{R}$) and is characterized by a particular choice of regularizer $\mathcal{R}(\mu)$:

我们可形式化这类在线算法：在 $\mu(\mathbf{a}|\mathbf{s})$ 上定义一族优化问题（相对式 2 的改动标红）。该族的一个实例记为 CQL($\mathcal{R}$)，由正则子 $\mathcal{R}(\mu)$ 的具体选择刻画：

$$\small{\min_{Q}{\color[rgb]{1,0,0}\max_{\mu}}~~\alpha\left(\mathbb{E}{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim{\color[rgb]{1,0,0}\mu(\mathbf{a}|\mathbf{s})}}\left[Q(\mathbf{s},\mathbf{a})\right]-\mathbb{E}{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim\hat{\pi}{\beta}(\mathbf{a}|\mathbf{s})}\left[Q(\mathbf{s},\mathbf{a})\right]\right)}\
\small{+\frac{1}{2}~\mathbb{E}{\mathbf{s},\mathbf{a},\mathbf{s}^{\prime}\sim\mathcal{D}}\left[\left(Q(\mathbf{s},\mathbf{a})-\hat{\mathcal{B}}^{\pi_{k}}\hat{Q}^{k}(\mathbf{s},\mathbf{a})\right)^{2}\right]+{\color[rgb]{1,0,0}\mathcal{R}(\mu)}~~~\left(\text{CQL}(\mathcal{R})\right).} \tag{3}$$

Variants of CQL. To demonstrate the generality of the CQL family of optimization problems, we discuss two specific instances within this family that are of special interest, and we evaluate them empirically in Section 6. If we choose $\mathcal{R}(\mu)$ to be the KL-divergence against a prior distribution, $\rho(\mathbf{a}|\mathbf{s})$, i.e., $\mathcal{R}(\mu)=-D_{\mathrm{KL}}(\mu,\rho)$, then we get $\mu(\mathbf{a}|\mathbf{s})\propto\rho(\mathbf{a}|\mathbf{s})\cdot\exp(Q(\mathbf{s},\mathbf{a}))$ (for a derivation, see Appendix A). Frist, if $\rho=\text{Unif}(\mathbf{a})$, then the first term in Equation 3 corresponds to a soft-maximum of the Q-values at any state $\mathbf{s}$ and gives rise to the following variant of Equation 3, called CQL($\mathcal{H}$):

CQL 变体。 为体现 CQL 优化族的一般性，我们讨论其中两个特别关注的实例，并在第 6 节经验评估。若取 $\mathcal{R}(\mu)$ 为相对先验 $\rho(\mathbf{a}|\mathbf{s})$ 的 KL 散度，即 $\mathcal{R}(\mu)=-D_{\mathrm{KL}}(\mu,\rho)$，则有 $\mu(\mathbf{a}|\mathbf{s})\propto\rho(\mathbf{a}|\mathbf{s})\cdot\exp(Q(\mathbf{s},\mathbf{a}))$（推导见附录 A）。首先，若 $\rho=\text{Unif}(\mathbf{a})$，则式 3 第一项对应任意状态 $\mathbf{s}$ 上 Q 值的 soft-maximum，并得到式 3 的如下变体，称为 CQL($\mathcal{H}$)：

$$\small{\min_{Q}~\alpha\mathbb{E}_{\mathbf{s}\sim\mathcal{D}}\left[\log\sum_{\mathbf{a}}\exp(Q(\mathbf{s},\mathbf{a}))\!-\!\mathbb{E}_{\mathbf{a}\sim\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})}\left[Q(\mathbf{s},\mathbf{a})\right]\right]\!+\!\frac{1}{2}\mathbb{E}_{\mathbf{s},\mathbf{a},\mathbf{s}^{\prime}\sim\mathcal{D}}\left[\left(Q-\hat{\mathcal{B}}^{\pi_{k}}\hat{Q}^{k}\right)^{2}\right]\!.} \tag{4}$$

Second, if $\rho(\mathbf{a}|\mathbf{s})$ is chosen to be the previous policy $\hat{\pi}^{k-1}$, the first term in Equation 4 is replaced by an exponential weighted average of Q-values of actions from the chosen $\hat{\pi}^{k-1}(\mathbf{a}|\mathbf{s})$. Empirically, we find that this variant can be more stable with high-dimensional action spaces (e.g., Table 2) where it is challenging to estimate $\log\sum_{\mathbf{a}}\exp$ via sampling due to high variance. In Appendix A, we discuss an additional variant of CQL, drawing connections to distributionally robust optimization [45]. We will discuss a practical instantiation of a CQL deep RL algorithm in Section 4. CQL can be instantiated as either a Q-learning algorithm (with $\mathcal{B}^{*}$ instead of $\mathcal{B}^{\pi}$ in Equations 3, 4) or as an actor-critic algorithm.

其次，若取 $\rho(\mathbf{a}|\mathbf{s})$ 为上一策略 $\hat{\pi}^{k-1}$，则式 4 第一项换成对来自 $\hat{\pi}^{k-1}(\mathbf{a}|\mathbf{s})$ 的动作 Q 值做指数加权平均。经验上，该变体在高维动作空间（如表 2）更稳：此时用采样估计 $\log\sum_{\mathbf{a}}\exp$ 方差大、很难。附录 A 还讨论与分布鲁棒优化 [45] 相联系的另一 CQL 变体。第 4 节给出 CQL 深度 RL 算法的实用实例。CQL 既可实例化为 Q 学习（式 3、4 中用 $\mathcal{B}^{*}$ 代替 $\mathcal{B}^{\pi}$），也可实例化为 actor-critic。

Theoretical analysis of CQL. Next, we will theoretically analyze CQL to show that the policy updates derived in this way are indeed “conservative”, in the sense that each successive policy iterate is optimized against a lower bound on its value. For clarity, we state the results in the absence of finite-sample error, in this section, but sampling error can be incorporated in the same way as Theorems 3.1 and 3.2, and we discuss this in Appendix C. Theorem 3.3 shows that any variant of the CQL family learns Q-value estimates that lower-bound the actual Q-function under the action-distribution defined by the policy, $\pi^{k}$, under mild regularity conditions (slow updates on the policy).

CQL 的理论分析。 接下来从理论上分析 CQL，说明由此导出的策略更新确实「保守」：每一后继策略迭代都相对其价值的下界做优化。为清晰起见，本节先在无有限样本误差下陈述结果；采样误差可按定理 3.1、3.2 同样方式并入，见附录 C。定理 3.3 表明：在温和正则条件（策略缓变）下，CQL 族任一变体学到的 Q 值估计，在策略 $\pi^{k}$ 定义的动作分布下下界真实 Q 函数。

Theorem 3.3(CQL learns lower-bounded Q-values).

Let $\pi_{\hat{Q}^{k}}(\mathbf{a}|\mathbf{s})\propto\exp(\hat{Q}^{k}(\mathbf{s},\mathbf{a}))$ and assume that $D_{\mathrm{TV}}(\hat{\pi}^{k+1},\pi_{\hat{Q}^{k}})\leq\varepsilon$ (i.e., $\hat{\pi}^{k+1}$ changes slowly w.r.t to $\hat{Q}^{k}$). Then, the policy value under $\hat{Q}^{k}$, lower-bounds the actual policy value, $\hat{V}^{k+1}(\mathbf{s})\leq V^{k+1}(\mathbf{s})\forall\mathbf{s}$ if

定理 3.3（CQL 学到下界 Q 值）。

令 $\pi_{\hat{Q}^{k}}(\mathbf{a}|\mathbf{s})\propto\exp(\hat{Q}^{k}(\mathbf{s},\mathbf{a}))$，并假设 $D_{\mathrm{TV}}(\hat{\pi}^{k+1},\pi_{\hat{Q}^{k}})\leq\varepsilon$（即 $\hat{\pi}^{k+1}$ 相对 $\hat{Q}^{k}$ 变化缓慢）。则 $\hat{Q}^{k}$ 下的策略价值下界真实策略价值，$\hat{V}^{k+1}(\mathbf{s})\leq V^{k+1}(\mathbf{s})\forall\mathbf{s}$，若

$$\small{\mathbb{E}_{\pi_{\hat{Q}^{k}}(\mathbf{a}|\mathbf{s})}\left[\frac{\pi_{\hat{Q}^{k}}(\mathbf{a}|\mathbf{s})}{\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})}-1\right]\geq\max_{\mathbf{a}\text{~s.t.~}\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})>0}\left(\frac{\pi_{\hat{Q}^{k}}(\mathbf{a}|\mathbf{s})}{\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})}\right)\cdot\varepsilon}.$$

The LHS of this inequality is equal to the amount of conservatism induced in the value, $\hat{V}^{k+1}$ in iteration $k+1$ of the CQL update, if the learned policy were equal to soft-optimal policy for $\hat{Q}^{k}$, i.e., when $\hat{\pi}^{k+1}=\pi_{\hat{Q}^{k}}$. However, as the actual policy, $\hat{\pi}^{k+1}$, may be different, the RHS is the maximal amount of potential overestimation due to this difference. To get a lower bound, we require the amount of underestimation to be higher, which is obtained if $\varepsilon$ is small, i.e. the policy changes slowly.

不等式左边等于：若所学策略等于 $\hat{Q}^{k}$ 的 soft-最优策略（即 $\hat{\pi}^{k+1}=\pi_{\hat{Q}^{k}}$）时，CQL 第 $k+1$ 次迭代在价值 $\hat{V}^{k+1}$ 上引入的保守量。但实际策略 $\hat{\pi}^{k+1}$ 可能不同，右边则是由此差异导致的潜在高估上界。要得到下界，需要低估量更大，这在 $\varepsilon$ 较小、即策略变化缓慢时成立。

Our final result shows that CQL Q-function update is “gap-expanding”, by which we mean that the difference in Q-values at in-distribution actions and over-optimistically erroneous out-of-distribution actions is higher than the corresponding difference under the actual Q-function. This implies that the policy $\pi^{k}(\mathbf{a}|\mathbf{s})\propto\exp(\hat{Q}^{k}(\mathbf{s},\mathbf{a}))$, is constrained to be closer to the dataset distribution, $\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})$, thus the CQL update implicitly prevents the detrimental effects of OOD action and distribution shift, which has been a major concern in offline RL settings [32, 36, 17].

最后结果说明 CQL 的 Q 函数更新具有「间隙扩张（gap-expanding）」性质：分布内动作与被过度乐观估计的分布外动作之间的 Q 值差，大于真实 Q 函数下的相应差。这意味着策略 $\pi^{k}(\mathbf{a}|\mathbf{s})\propto\exp(\hat{Q}^{k}(\mathbf{s},\mathbf{a}))$ 被约束得更靠近数据集分布 $\hat{\pi}_{\beta}(\mathbf{a}|\mathbf{s})$，从而 CQL 更新隐式抑制 OOD 动作与分布偏移的有害影响——这正是离线 RL 中的主要关切 [32, 36, 17]。

Theorem 3.4(CQL is gap-expanding).

At any iteration $k$, CQL expands the difference in expected Q-values under the behavior policy ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$ and $\mu_{k}$, such that for large enough values of $\alpha_{k}$, we have that $\forall\mathbf{s},~\mathbb{E}_{{\pi_{\beta}}(\mathbf{a}|\mathbf{s})}[\hat{Q}^{k}(\mathbf{s},\mathbf{a})]-\mathbb{E}_{\mu_{k}(\mathbf{a}|\mathbf{s})}[\hat{Q}^{k}(\mathbf{s},\mathbf{a})]>\mathbb{E}_{{\pi_{\beta}}(\mathbf{a}|\mathbf{s})}[{Q}^{k}(\mathbf{s},\mathbf{a})]-\mathbb{E}_{\mu_{k}(\mathbf{a}|\mathbf{s})}[{Q}^{k}(\mathbf{s},\mathbf{a})]$.

定理 3.4（CQL 具有间隙扩张性）。

在任意迭代 $k$，CQL 扩大行为策略 ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$ 与 $\mu_{k}$ 下期望 Q 值之差，使得当 $\alpha_{k}$ 足够大时，对 $\forall\mathbf{s}$ 有 $\mathbb{E}_{{\pi_{\beta}}(\mathbf{a}|\mathbf{s})}[\hat{Q}^{k}(\mathbf{s},\mathbf{a})]-\mathbb{E}_{\mu_{k}(\mathbf{a}|\mathbf{s})}[\hat{Q}^{k}(\mathbf{s},\mathbf{a})]>\mathbb{E}_{{\pi_{\beta}}(\mathbf{a}|\mathbf{s})}[{Q}^{k}(\mathbf{s},\mathbf{a})]-\mathbb{E}_{\mu_{k}(\mathbf{a}|\mathbf{s})}[{Q}^{k}(\mathbf{s},\mathbf{a})]$。

When function approximation or sampling error makes OOD actions have higher learned Q-values, CQL backups are expected to be more robust, in that the policy is updated using Q-values that prefer in-distribution actions. As we will empirically show in Appendix B, prior offline RL methods that do not explicitly constrain or regularize the Q-function may not enjoy such robustness properties.

当函数逼近或采样误差使 OOD 动作学得更高的 Q 值时，CQL 回传预期更稳健：策略用偏好分布内动作的 Q 值来更新。附录 B 将经验表明：不显式约束或正则 Q 函数的先验离线 RL 方法，未必具备这种稳健性。

To summarize, we showed that the CQL RL algorithm learns lower-bound Q-values with large enough $\alpha$, meaning that the final policy attains at least the estimated value. We also showed that the Q-function is gap-expanding, meaning that it should only ever over-estimate the gap between in-distribution and out-of-distribution actions, preventing OOD actions.

小结：我们证明在 $\alpha$ 足够大时，CQL RL 算法学到下界 Q 值，因而最终策略至少达到所估价值。我们也证明 Q 函数具有间隙扩张性，即只会高估分布内与分布外动作之间的间隙，从而抑制 OOD 动作。

3.3 Safe Policy Improvement Guarantees

In Section 3.1 we proposed novel objectives for Q-function training such that the expected value of a policy under the resulting Q-function lower bounds the actual performance of the policy. In Section 3.2, we used the learned conservative Q-function for policy improvement. In this section, we show that this procedure actually optimizes a well-defined objective and provide a safe policy improvement result for CQL, along the lines of Theorems 1 and 2 in Laroche et al. 2017.

3.3 安全策略改进保证

第 3.1 节提出使策略在所得 Q 函数下期望价值下界真实表现的新型 Q 训练目标；第 3.2 节用所学保守 Q 做策略改进。本节说明该流程实际优化明确目标，并给出沿 Laroche 等 2017 定理 1、2 思路的 CQL 安全策略改进结果。

To begin with, we define empirical return of any policy $\pi$, ${J}(\pi,\hat{M})$, which is equal to the discounted return of a policy $\pi$ in the empirical MDP, $\hat{M}$, that is induced by the transitions observed in the dataset $\mathcal{D}$, i.e. $\hat{M}=\{s,a,r,s^{\prime}\in\mathcal{D}\}$. $J(\pi,M)$ refers to the expected discounted return attained by a policy $\pi$ in the actual underlying MDP, $M$. In Theorem 3.5, we first show that CQL (Equation 2) optimizes a well-defined penalized RL empirical objective. All proofs are found in Appendix D.4.

首先定义任意策略 $\pi$ 的经验回报 ${J}(\pi,\hat{M})$：即策略 $\pi$ 在由数据集 $\mathcal{D}$ 中观测转移诱导的经验 MDP $\hat{M}=\{s,a,r,s^{\prime}\in\mathcal{D}\}$ 中的折扣回报。$J(\pi,M)$ 表示策略 $\pi$ 在真实底层 MDP $M$ 中的期望折扣回报。定理 3.5 首先证明 CQL（式 2）优化一个明确的带惩罚经验 RL 目标。证明均见附录 D.4。

Theorem 3.5.

Let $\hat{Q}^{\pi}$ be the fixed point of Equation 2, then $\pi^{}(\mathbf{a}|\mathbf{s}):=\arg\max_{\pi}\mathbb{E}_{\mathbf{s}\sim\rho(\mathbf{s})}[\hat{V}^{\pi}(\mathbf{s})]$ is equivalently obtained by solving:*

定理 3.5。

令 $\hat{Q}^{\pi}$ 为式 2 的不动点，则 $\pi^{}(\mathbf{a}|\mathbf{s}):=\arg\max_{\pi}\mathbb{E}_{\mathbf{s}\sim\rho(\mathbf{s})}[\hat{V}^{\pi}(\mathbf{s})]$ 等价于求解：*

$$\small{\pi^{*}(\mathbf{a}|\mathbf{s})\leftarrow\arg\max_{\pi}~~J(\pi,\hat{M})-\alpha\frac{1}{1-\gamma}\mathbb{E}_{\mathbf{s}\sim d^{\pi}_{\hat{M}}(\mathbf{s})}\left[D_{\text{CQL}}(\pi,\hat{\pi}_{\beta})(\mathbf{s})\right]}, \tag{5}$$

where $D_{\text{CQL}}(\pi,{\pi_{\beta}})(\mathbf{s}):=\sum_{\mathbf{a}}\pi(\mathbf{a}|\mathbf{s})\cdot\left(\frac{\pi(\mathbf{a}|\mathbf{s})}{{\pi_{\beta}}(\mathbf{a}|\mathbf{s})}-1\right)$.

其中 $D_{\text{CQL}}(\pi,{\pi_{\beta}})(\mathbf{s}):=\sum_{\mathbf{a}}\pi(\mathbf{a}|\mathbf{s})\cdot\left(\frac{\pi(\mathbf{a}|\mathbf{s})}{{\pi_{\beta}}(\mathbf{a}|\mathbf{s})}-1\right)$。

Intuitively, Theorem 3.5 says that CQL optimizes the return of a policy in the empirical MDP, $\hat{M}$, while also ensuring that the learned policy $\pi$ is not too different from the behavior policy, $\hat{\pi}_{\beta}$ via a penalty that depends on $D_{\text{CQL}}$. Note that this penalty is implicitly introduced by virtue by the gap-expanding (Theorem 3.4) behavior of CQL. Next, building upon Theorem 3.5 and the analysis of CPO [1], we show that CQL provides a $\zeta$-safe policy improvement over $\hat{\pi}_{\beta}$.

直观上，定理 3.5 说明：CQL 优化经验 MDP $\hat{M}$ 中的策略回报，同时通过依赖 $D_{\text{CQL}}$ 的惩罚确保所学策略 $\pi$ 不太偏离行为策略 $\hat{\pi}_{\beta}$。注意该惩罚由 CQL 的间隙扩张行为（定理 3.4）隐式引入。接下来，基于定理 3.5 与 CPO [1] 的分析，我们证明 CQL 相对 $\hat{\pi}_{\beta}$ 提供 $\zeta$-安全策略改进。

Theorem 3.6.

Let $\pi^{}(\mathbf{a}|\mathbf{s})$be the policy obtained by optimizing Equation 5. Then, the policy$\pi^{}(\mathbf{a}|\mathbf{s})$is a$\zeta$-safe policy improvement over$\hat{\pi}_{\beta}$in the actual MDP$M$, i.e.,$J(\pi^{},M)\geq J(\hat{\pi}_{\beta},M)-\zeta$with high probability$1-\delta$, where$\zeta$ is given by,*

定理 3.6。

令 $\pi^{}(\mathbf{a}|\mathbf{s})$为优化式 5 所得策略。则在真实 MDP$M$中，$\pi^{}(\mathbf{a}|\mathbf{s})$是相对$\hat{\pi}_{\beta}$的$\zeta$-安全策略改进，即以高概率$1-\delta$有$J(\pi^{},M)\geq J(\hat{\pi}_{\beta},M)-\zeta$，其中$\zeta$ 为*

$$\small{\zeta=2\left({\frac{C_{r,\delta}}{1-\gamma}+\frac{\gamma R_{\max}C_{T,\delta}}{(1-\gamma)^{2}}}\right)\mathbb{E}{\mathbf{s}\sim d^{\pi^{*}}{\hat{M}}(\mathbf{s})}\left[\frac{\sqrt{|\mathcal{A}|}}{\sqrt{|\mathcal{D}(\mathbf{s})|}}\sqrt{D_{\text{CQL}}(\pi^{},\hat{\pi}_{\beta})(\mathbf{s})+1}\right]}-\
\small{\underbrace{\left(J(\pi^{},\hat{M})-J(\hat{\pi}{\beta},\hat{M})\right)}{\geq\alpha\frac{1}{1-\gamma}\mathbb{E}{\mathbf{s}\sim d^{\pi^{*}}{\hat{M}}(\mathbf{s})}\left[D_{\text{CQL}}(\pi^{*},\hat{\pi}_{\beta})(\mathbf{s})\right]}.} \tag{6}$$

The expression of $\zeta$ in Theorem 3.6 consists of two terms: the first term captures the decrease in policy performance in $M$, that occurs due to the mismatch between $\hat{M}$ and $M$, also referred to as sampling error. The second term captures the increase in policy performance due to CQL in empirical MDP, $\hat{M}$. The policy $\pi^{*}$ obtained by optimizing $\pi$ against the CQL Q-function improves upon the behavior policy, $\hat{\pi}_{\beta}$ for suitably chosen values of $\alpha$. When sampling error is small, i.e., $|\mathcal{D}(\mathbf{s})|$ is large, then smaller values of $\alpha$ are enough to provide an improvement over the behavior policy.

定理 3.6 中 $\zeta$ 的表达式含两项：第一项刻画因 $\hat{M}$ 与 $M$ 失配（亦称采样误差）导致的策略在 $M$ 中表现下降；第二项刻画 CQL 在经验 MDP $\hat{M}$ 中带来的策略表现提升。相对 CQL Q 函数优化得到的 $\pi^{*}$，在合适选取的 $\alpha$ 下改进行为策略 $\hat{\pi}_{\beta}$。当采样误差小（即 $|\mathcal{D}(\mathbf{s})|$ 大）时，较小的 $\alpha$ 即可保证相对行为策略的改进。

To summarize, CQL optimizes a well-defined, penalized empirical RL objective, and performs high-confidence safe policy improvement over the behavior policy. The extent of improvement is negatively influenced by higher sampling error, which decays as more samples are observed.

小结： CQL 优化明确的带惩罚经验 RL 目标，并以高置信度相对行为策略做安全策略改进。改进幅度受采样误差负向影响；随样本量增加，该影响衰减。

4 Practical Algorithm and Implementation Details

4 实用算法与实现细节

1:  Initialize Q-function, $Q_{\theta}$, and optionally a policy, $\pi_{\phi}$.
2:  for step $t$ in {1, …, N} do
3:    Train the Q-function using $G_{Q}$ gradient steps on objective from Equation 4 \theta_{t}:=\theta_{t-1}-\eta_{Q}\nabla_{\theta}{\color[rgb]{1,0,0}\text{CQL}(\mathcal{R})(\theta)}(Use $\mathcal{B}^{*}$ for Q-learning, $\mathcal{B}^{\pi_{\phi_{t}}}$ for actor-critic)
4:    (only with actor-critic) Improve policy $\pi_{\phi}$ via $G_{\pi}$ gradient steps on $\phi$ with SAC-style entropy regularization:$\phi_{t}:=\phi_{t-1}+\eta_{\pi}\mathbb{E}_{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim\pi_{\phi}(\cdot|\mathbf{s})}[Q_{\theta}(\mathbf{s},\mathbf{a})\!-\!\log\pi_{\phi}(\mathbf{a}|\mathbf{s})]$
5:  end for

Algorithm 1 Conservative Q-Learning (both variants)

1:  初始化 Q 函数 $Q_{\theta}$，并可选初始化策略 $\pi_{\phi}$。
2:  for 步 $t$ in {1, …, N} do
3:    用式 4 目标对 Q 函数做 $G_{Q}$ 步梯度更新 \theta_{t}:=\theta_{t-1}-\eta_{Q}\nabla_{\theta}{\color[rgb]{1,0,0}\text{CQL}(\mathcal{R})(\theta)}（Q 学习用 $\mathcal{B}^{*}$，actor-critic 用 $\mathcal{B}^{\pi_{\phi_{t}}}$）
4:    （仅 actor-critic）以 SAC 式熵正则对策略 $\pi_{\phi}$ 做 $G_{\pi}$ 步改进：$\phi_{t}:=\phi_{t-1}+\eta_{\pi}\mathbb{E}_{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim\pi_{\phi}(\cdot|\mathbf{s})}[Q_{\theta}(\mathbf{s},\mathbf{a})\!-\!\log\pi_{\phi}(\mathbf{a}|\mathbf{s})]$
5:  end for

算法 1 保守 Q 学习（两种变体）

We now describe two practical offline deep reinforcement learning methods based on CQL: an actor-critic variant and a Q-learning variant. Pseudocode is shown in Algorithm 1, with differences from conventional actor-critic algorithms (e.g., SAC [21]) and deep Q-learning algorithms (e.g., DQN [41]) in red. Our algorithm uses the CQL($\mathcal{H}$) (or CQL($\mathcal{R}$) in general) objective from the CQL framework for training the Q-function $Q_{\theta}$, which is parameterized by a neural network with parameters $\theta$. For the actor-critic algorithm, a policy $\pi_{\phi}$ is trained as well. Our algorithm modifies the objective for the Q-function (swaps out Bellman error with CQL($\mathcal{H}$)) or CQL($\rho$) in a standard actor-critic or Q-learning setting, as shown in Line 3. As discussed in Section 3.2, due to the explicit penalty on the Q-function, CQL methods do not use a policy constraint, unlike prior offline RL methods [32, 62, 57, 36]. Hence, we do not require fitting an additional behavior policy estimator, simplifying our method.

下面给出基于 CQL 的两种实用离线深度强化学习方法：actor-critic 变体与 Q 学习变体。伪代码见算法 1，相对常规 actor-critic（如 SAC [21]）与深度 Q 学习（如 DQN [41]）的差异标红。算法用 CQL 框架中的 CQL($\mathcal{H}$)（或一般的 CQL($\mathcal{R}$)）目标训练由参数 $\theta$ 的神经网络参数化的 Q 函数 $Q_{\theta}$。对 actor-critic，还训练策略 $\pi_{\phi}$。如第 3 行所示，我们在标准 actor-critic 或 Q 学习设定中把 Q 目标换成 CQL($\mathcal{H}$) 或 CQL($\rho$)。如第 3.2 节所述，由于对 Q 函数有显式惩罚，CQL 方法不像先验离线 RL [32, 62, 57, 36] 那样使用策略约束，因而无需再拟合额外的行为策略估计器，方法更简单。

Implementation details. Our algorithm requires an addition of only 20 lines of code on top of standard implementations of soft actor-critic (SAC) [21] for continuous control experiments and on top of QR-DQN [9] for the discrete control experiments. The tradeoff factor, $\alpha$ is automatically tuned via Lagrangian dual gradient descent for continuous control, and is fixed at constant values described in Appendix F for discrete control. We use default hyperparameters from SAC, except that the learning rate for the policy is chosen to be 3e-5 (vs 3e-4 or 1e-4 for the Q-function), as dictated by Theorem 3.3. Elaborate details are provided in Appendix F.

实现细节。 相对连续控制实验中的 soft actor-critic（SAC） [21] 与离散控制实验中的 QR-DQN [9] 标准实现，我们的算法只需增加约 20 行代码。折中因子 $\alpha$ 在连续控制中用拉格朗日对偶梯度下降自动调节，在离散控制中固定为附录 F 所述常数。除策略学习率取 3e-5（Q 函数为 3e-4 或 1e-4，由定理 3.3 要求）外，其余超参沿用 SAC 默认。细节见附录 F。

5 Related Work

5 相关工作

We now briefly discuss prior work in offline RL and off-policy evaluation, comparing and contrasting these works with our approach. More technical discussion of related work is provided in Appendix E.

下面简要讨论离线 RL 与异策略评估的先验工作，并与本文方法对照。更技术性的相关讨论见附录 E。

Off-policy evaluation (OPE). Several different paradigms have been used to perform off-policy evaluation. Earlier works [53, 51, 54] used per-action importance sampling on Monte-Carlo returns to obtain an OPE return estimator. Recent approaches [38, 19, 42, 64] use marginalized importance sampling by directly estimating the state-distribution importance ratios via some form of dynamic programming [36] and typically exhibit less variance than per-action importance sampling at the cost of bias. Because these methods use dynamic programming, they can suffer from OOD actions  [36, 19, 22, 42]. In contrast, the regularizer in CQL explicitly addresses the impact of OOD actions due to its gap-expanding behavior, and obtains conservative value estimates.

异策略评估（OPE）。 多种范式被用于异策略评估。早期工作 [53, 51, 54] 对蒙特卡洛回报做逐动作重要性采样以得到 OPE 回报估计。近期方法 [38, 19, 42, 64] 用边缘化重要性采样，经某种动态规划直接估计状态分布重要性比 [36]，通常比逐动作重要性采样方差更小、但有偏。由于使用动态规划，它们仍可能受 OOD 动作影响 [36, 19, 22, 42]。相比之下，CQL 的正则子因其间隙扩张行为显式应对 OOD 动作影响，并得到保守价值估计。

Offline RL. As discussed in Section 2, offline Q-learning methods suffer from issues pertaining to OOD actions. Prior works have attempted to solve this problem by constraining the learned policy to be “close” to the behavior policy, for example as measured by KL-divergence [29, 62, 50, 57], Wasserstein distance [62], or MMD [32], and then only using actions sampled from this constrained policy in the Bellman backup or applying a value penalty. SPIBB [35, 43] methods bootstrap using the behavior policy in a Q-learning algorithm for unseen actions. Most of these methods require a separately estimated model to the behavior policy, ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$ [17, 32, 62, 29, 57, 58], and are thus limited by their ability to accurately estimate the unknown behavior policy [44], which might be especially complex in settings where the data is collected from multiple sources [36]. In contrast, CQL does not require estimating the behavior policy. Prior work has explored some forms of Q-function penalties [25, 61], but only in the standard online RL setting with demonstrations. Luo et al. 2019 learn a conservatively-extrapolated value function by enforcing a linear extrapolation property over the state-space, and a learned dynamics model to obtain policies for goal-reaching tasks. Kakade and Langford 2002 proposed the CPI algorithm, that improves a policy conservatively in online RL.

离线 RL。 如第 2 节所述，离线 Q 学习受 OOD 动作问题困扰。先验工作试图通过约束所学策略「接近」行为策略来解决，度量包括 KL 散度 [29, 62, 50, 57]、Wasserstein 距离 [62] 或 MMD [32]，再仅用该约束策略采样的动作做 Bellman 回传或施加价值惩罚。SPIBB [35, 43] 方法在 Q 学习中对未见动作用行为策略自举。这些方法大多需要单独估计行为策略 ${\pi_{\beta}}(\mathbf{a}|\mathbf{s})$ [17, 32, 62, 29, 57, 58]，因而受限于准确估计未知行为策略的能力 [44]——当数据来自多源时行为策略可能尤其复杂 [36]。相比之下，CQL 无需估计行为策略。先前也有若干 Q 函数惩罚形式 [25, 61]，但仅在带示范的标准在线 RL 设定中。Luo 等 2019 通过对状态空间施加线性外推性质，学习保守外推的价值函数，并结合所学动力学模型为目标到达任务得到策略。Kakade 与 Langford 2002 提出 CPI 算法，在在线 RL 中保守地改进策略。

Alternate prior approaches to offline RL estimate some sort of uncertainty to determine the trustworthiness of a Q-value prediction [32, 3, 36], typically using uncertainty estimation techniques from exploration in online RL [49, 28, 48, 7]. These methods have not been generally performant in offline RL [17, 32, 36] due to the high-fidelity requirements of uncertainty estimates in offline RL [36]. Robust MDPs [26, 52, 59, 46] have been a popular theoretical abstraction for offline RL, but tend to be highly conservative in policy improvement. We expect CQL to be less conservative since CQL does not underestimate Q-values for all state-action tuples. Works on high confidence policy improvement [60] provides safety guarantees for improvement but tend to be conservative. The gap-expanding property of CQL backups, shown in Theorem 3.4, is related to how gap-increasing Bellman backup operators [6, 39] are more robust to estimation error in online RL.

另一类先验离线 RL 方法估计某种不确定性以判断 Q 值预测是否可信 [32, 3, 36]，通常借用在线 RL 探索中的不确定性估计技术 [49, 28, 48, 7]。由于离线 RL 对不确定性估计保真度要求很高 [36]，这些方法在离线 RL 中整体表现并不出色 [17, 32, 36]。鲁棒 MDP [26, 52, 59, 46] 是离线 RL 的常见理论抽象，但在策略改进上往往过于保守。我们预期 CQL 不那么保守，因为它并不对所有状态—动作对都低估 Q 值。高置信策略改进 [60] 提供改进的安全保证，但往往保守。定理 3.4 所示 CQL 回传的间隙扩张性质，与间隙增大 Bellman 回传算子 [6, 39] 在在线 RL 中对估计误差更稳健的思路相关。

Theoretical results. Our theoretical results (Theorems 3.5, 3.6) are related to prior work on safe policy improvement [35, 52], and a direct comparison to Theorems 1 and 2 in Laroche et al. 2017 suggests similar quadratic dependence on the horizon and an inverse square-root dependence on the counts. Our bounds improve over the $\infty$-norm bounds in Petrik et al. 2016. Prior analyses have also focused on error propagation in approximate dynamic programming [13, 8, 63, 56, 32], and generally obtain bounds in terms of concentrability coefficients that capture the effect of distribution shift.

理论结果。 我们的理论结果（定理 3.5、3.6）与安全策略改进的先验工作 [35, 52] 相关；直接对照 Laroche 等 2017 的定理 1、2，可见对视界的二次依赖与对计数的平方根倒数依赖类似。我们的界改进了 Petrik 等 2016 的 $\infty$-范数界。先验分析也关注近似动态规划中的误差传播 [13, 8, 63, 56, 32]，通常以刻画分布偏移影响的 concentrability 系数给出界。

6 Experimental Evaluation

6 实验评估

We compare CQL to prior offline RL methods on a range of domains and dataset compositions, including continuous and discrete action spaces, state observations of varying dimensionality, and high-dimensional image inputs. We first evaluate actor-critic CQL, using CQL($\mathcal{H}$) from Algorithm 1, on continuous control datasets from the D4RL benchmark [14]. We compare to: prior offline RL methods that use a policy constraint – BEAR [32] and BRAC [62]; SAC [21], an off-policy actor-critic method that we adapt to offline setting; and behavioral cloning (BC).

我们在多种领域与数据集构成上，将 CQL 与先验离线 RL 方法对照，涵盖连续与离散动作空间、不同维状态观测，以及高维图像输入。首先用算法 1 中的 CQL($\mathcal{H}$) 评估 actor-critic CQL，数据来自 D4RL 基准 [14] 的连续控制数据集。对照包括：使用策略约束的先验离线 RL——BEAR [32] 与 BRAC [62]；改编到离线设定的异策略 actor-critic SAC [21]；以及行为克隆（BC）。

Task Name	SAC	BC	BEAR	BRAC-p	BRAC-v	CQL($\mathcal{H}$)
halfcheetah-random	30.5	2.1	25.5	23.5	28.1	35.4
hopper-random	11.3	9.8	9.5	11.1	12.0	10.8
walker2d-random	4.1	1.6	6.7	0.8	0.5	7.0
halfcheetah-medium	-4.3	36.1	38.6	44.0	45.5	44.4
walker2d-medium	0.9	6.6	33.2	72.7	81.3	79.2
hopper-medium	0.8	29.0	47.6	31.2	32.3	58.0
halfcheetah-expert	-1.9	107.0	108.2	3.8	-1.1	104.8
hopper-expert	0.7	109.0	110.3	6.6	3.7	109.9
walker2d-expert	-0.3	125.7	106.1	-0.2	-0.0	153.9
halfcheetah-medium-expert	1.8	35.8	51.7	43.8	45.3	62.4
walker2d-medium-expert	1.9	11.3	10.8	-0.3	0.9	98.7
hopper-medium-expert	1.6	111.9	4.0	1.1	0.8	111.0
halfcheetah-random-expert	53.0	1.3	24.6	30.2	2.2	92.5
walker2d-random-expert	0.8	0.7	1.9	0.2	2.7	91.1
hopper-random-expert	5.6	10.1	10.1	5.8	11.1	110.5
halfcheetah-mixed	-2.4	38.4	36.2	45.6	45.9	46.2
hopper-mixed	3.5	11.8	25.3	0.7	0.8	48.6
walker2d-mixed	1.9	11.3	10.8	-0.3	0.9	26.7

Table 1: Performance of CQL($\mathcal{H}$) and prior methods on gym domains from D4RL, on the normalized return metric, averaged over 4 seeds. Note that CQL performs similarly or better than the best prior method with simple datasets, and greatly outperforms prior methods with complex distributions (“–mixed”, “–random-expert”, “–medium-expert”).

任务名	SAC	BC	BEAR	BRAC-p	BRAC-v	CQL($\mathcal{H}$)
halfcheetah-random	30.5	2.1	25.5	23.5	28.1	35.4
hopper-random	11.3	9.8	9.5	11.1	12.0	10.8
walker2d-random	4.1	1.6	6.7	0.8	0.5	7.0
halfcheetah-medium	-4.3	36.1	38.6	44.0	45.5	44.4
walker2d-medium	0.9	6.6	33.2	72.7	81.3	79.2
hopper-medium	0.8	29.0	47.6	31.2	32.3	58.0
halfcheetah-expert	-1.9	107.0	108.2	3.8	-1.1	104.8
hopper-expert	0.7	109.0	110.3	6.6	3.7	109.9
walker2d-expert	-0.3	125.7	106.1	-0.2	-0.0	153.9
halfcheetah-medium-expert	1.8	35.8	51.7	43.8	45.3	62.4
walker2d-medium-expert	1.9	11.3	10.8	-0.3	0.9	98.7
hopper-medium-expert	1.6	111.9	4.0	1.1	0.8	111.0
halfcheetah-random-expert	53.0	1.3	24.6	30.2	2.2	92.5
walker2d-random-expert	0.8	0.7	1.9	0.2	2.7	91.1
hopper-random-expert	5.6	10.1	10.1	5.8	11.1	110.5
halfcheetah-mixed	-2.4	38.4	36.2	45.6	45.9	46.2
hopper-mixed	3.5	11.8	25.3	0.7	0.8	48.6
walker2d-mixed	1.9	11.3	10.8	-0.3	0.9	26.7

表 1：CQL($\mathcal{H}$) 与先验方法在 D4RL gym 领域上的表现（归一化回报，4 个种子平均）。在简单数据集上 CQL 与最优先验方法相当或更好；在复杂分布（「–mixed」「–random-expert」「–medium-expert」）上大幅领先。

Gym domains. Results for the gym domains are shown in Table 1. The results for BEAR, BRAC, SAC, and BC are based on numbers reported by Fu et al. 2020a. On the datasets generated from a single policy, marked as “-random”, “-expert” and “-medium”, CQL roughly matches or exceeds the best prior methods, but by a small margin. However, on datasets that combine multiple policies (“-mixed”, “-medium-expert” and “-random-expert”), that are more likely to be common in practical datasets, CQL outperforms prior methods by large margins, sometimes as much as 2-3x.

Gym 领域。 Gym 结果见表 1。BEAR、BRAC、SAC 与 BC 的数字取自 Fu 等 2020a 的报告。在由单一策略生成的数据集（标为「-random」「-expert」「-medium」）上，CQL 大致匹配或略超最优先验方法。而在更可能出现于实际数据的多策略混合数据集（「-mixed」「-medium-expert」「-random-expert」）上，CQL 大幅领先，有时可达 2–3 倍。

Adroit tasks. The more complex Adroit [55] tasks (shown on the right) in D4RL require controlling a 24-DoF robotic hand, using limited data from human demonstrations. These tasks are substantially more difficult than the gym tasks in terms of both the dataset composition and high dimensionality. Prior offline RL methods generally struggle to learn meaningful behaviors on

Adroit 任务。 D4RL 中更复杂的 Adroit [55] 任务（右图）需用有限的人类示范数据控制 24 自由度机械手。无论数据集构成还是维度，都明显难于 gym 任务。先验离线 RL 方法通常难以学到有意义的行为。

these tasks, and the strongest baseline is BC. As shown in Table 2, CQL variants are the only methods that improve over BC, attaining scores that are 2-9x those of the next best offline RL method. CQL($\rho$) with $\rho=\hat{\pi}^{k-1}$ (the previous policy) outperforms CQL($\mathcal{H}$) on a number of these tasks, due to the higher action dimensionality resulting in higher variance for the CQL($\mathcal{H}$) importance weights. Both variants outperform prior methods.

在这些任务上最强基线是 BC。如表 2 所示，CQL 变体是唯一超过 BC 的方法，得分可达次优离线 RL 方法的 2–9 倍。因动作维更高导致 CQL($\mathcal{H}$) 重要性权重方差更大，取 $\rho=\hat{\pi}^{k-1}$（上一策略）的 CQL($\rho$) 在若干任务上优于 CQL($\mathcal{H}$)。两变体均优于先验方法。

Domain	Task Name	BC	SAC	BEAR	BRAC-p	BRAC-v	CQL($\mathcal{H}$)	CQL($\rho$)
AntMaze	antmaze-umaze	65.0	0.0	73.0	50.0	70.0	74.0	73.5
antmaze-umaze-diverse	55.0	0.0	61.0	40.0	70.0	84.0	61.0
antmaze-medium-play	0.0	0.0	0.0	0.0	0.0	61.2	4.6
antmaze-medium-diverse	0.0	0.0	8.0	0.0	0.0	53.7	5.1
antmaze-large-play	0.0	0.0	0.0	0.0	0.0	15.8	3.2
antmaze-large-diverse	0.0	0.0	0.0	0.0	0.0	14.9	2.3
Adroit	pen-human	34.4	6.3	-1.0	8.1	0.6	37.5	55.8
hammer-human	1.5	0.5	0.3	0.3	0.2	4.4	2.1
door-human	0.5	3.9	-0.3	-0.3	-0.3	9.9	9.1
relocate-human	0.0	0.0	-0.3	-0.3	-0.3	0.20	0.35
pen-cloned	56.9	23.5	26.5	1.6	-2.5	39.2	40.3
hammer-cloned	0.8	0.2	0.3	0.3	0.3	2.1	5.7
door-cloned	-0.1	0.0	-0.1	-0.1	-0.1	0.4	3.5
relocate-cloned	-0.1	-0.2	-0.3	-0.3	-0.3	-0.1	-0.1
Kitchen	kitchen-complete	33.8	15.0	0.0	0.0	0.0	43.8	31.3
kitchen-partial	33.8	0.0	13.1	0.0	0.0	49.8	50.1
kitchen-undirected	47.5	2.5	47.2	0.0	0.0	51.0	52.4

Table 2: Normalized scores of all methods on AntMaze, Adroit, and kitchen domains from D4RL, averaged across 4 seeds. On the harder mazes, CQL is the only method that attains non-zero returns, and is the only method to outperform simple behavioral cloning on Adroit tasks with human demonstrations. We observed that the CQL($\rho$) variant, which avoids importance weights, trains more stably, with no sudden fluctuations in policy performance over the course of training, on the higher-dimensional Adroit tasks.

领域	任务名	BC	SAC	BEAR	BRAC-p	BRAC-v	CQL($\mathcal{H}$)	CQL($\rho$)
AntMaze	antmaze-umaze	65.0	0.0	73.0	50.0	70.0	74.0	73.5
antmaze-umaze-diverse	55.0	0.0	61.0	40.0	70.0	84.0	61.0
antmaze-medium-play	0.0	0.0	0.0	0.0	0.0	61.2	4.6
antmaze-medium-diverse	0.0	0.0	8.0	0.0	0.0	53.7	5.1
antmaze-large-play	0.0	0.0	0.0	0.0	0.0	15.8	3.2
antmaze-large-diverse	0.0	0.0	0.0	0.0	0.0	14.9	2.3
Adroit	pen-human	34.4	6.3	-1.0	8.1	0.6	37.5	55.8
hammer-human	1.5	0.5	0.3	0.3	0.2	4.4	2.1
door-human	0.5	3.9	-0.3	-0.3	-0.3	9.9	9.1
relocate-human	0.0	0.0	-0.3	-0.3	-0.3	0.20	0.35
pen-cloned	56.9	23.5	26.5	1.6	-2.5	39.2	40.3
hammer-cloned	0.8	0.2	0.3	0.3	0.3	2.1	5.7
door-cloned	-0.1	0.0	-0.1	-0.1	-0.1	0.4	3.5
relocate-cloned	-0.1	-0.2	-0.3	-0.3	-0.3	-0.1	-0.1
Kitchen	kitchen-complete	33.8	15.0	0.0	0.0	0.0	43.8	31.3
kitchen-partial	33.8	0.0	13.1	0.0	0.0	49.8	50.1
kitchen-undirected	47.5	2.5	47.2	0.0	0.0	51.0	52.4

表 2：各方法在 D4RL 的 AntMaze、Adroit 与 kitchen 领域上的归一化分数（4 种子平均）。在更难迷宫上，CQL 是唯一取得非零回报的方法，也是在人类示范 Adroit 任务上唯一超过简单行为克隆的方法。我们观察到：避免重要性权重的 CQL($\rho$) 变体在更高维 Adroit 任务上训练更稳，训练过程中策略表现没有突然波动。