Off-Policy Deep Reinforcement Learning without Exploration

无需探索的离策略深度强化学习

Scott Fujimoto Affiliation:  Department of Computer Science, McGill University, Montreal, Canada Affiliation:  Mila Québec AI Institute Correspondence to: scott.fujimoto@mail.mcgill.ca    David Meger Affiliation:  Department of Computer Science, McGill University, Montreal, Canada Affiliation:  Mila Québec AI Institute    Doina Precup Affiliation:  Department of Computer Science, McGill University, Montreal, Canada Affiliation:  Mila Québec AI Institute

Off-Policy Deep Reinforcement Learning without Exploration: Supplementary Material

无需探索的离策略深度强化学习：补充材料

Scott Fujimoto Affiliation:  Department of Computer Science, McGill University, Montreal, Canada Affiliation:  Mila Québec AI Institute Correspondence to: scott.fujimoto@mail.mcgill.ca    David Meger Affiliation:  Department of Computer Science, McGill University, Montreal, Canada Affiliation:  Mila Québec AI Institute    Doina Precup Affiliation:  Department of Computer Science, McGill University, Montreal, Canada Affiliation:  Mila Québec AI Institute

Abstract

摘要

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning without data correlated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

许多实际强化学习应用要求智能体只能从已经采集好的固定批次数据中学习，无法再继续采集。本文表明：由于外推引入的误差，标准离策略深度强化学习算法（如 DQN 与 DDPG）若缺少与当前策略分布相关的数据，就无法有效学习，因而在这种固定批次设定下失效。我们提出一类新的离策略算法——批次约束强化学习：通过限制动作空间，迫使智能体相对给定数据的某个子集表现得接近同策略。我们给出首个能从任意固定批次数据有效学习的连续控制深度强化学习算法，并在多项任务上实证展示其行为质量。

Keywords:

marginparsep has been altered. topmargin has been altered. marginparwidth has been altered. marginparpush has been altered.

关键词：

marginparsep has been altered. topmargin has been altered. marginparwidth has been altered. marginparpush has been altered.

The page layout violates the ICML style.

页面布局违反了 ICML 样式。

Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you.

请勿更改页面布局，也不要引入 geometry、savetrees 或 fullpage 等会替你改布局的宏包。

We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again.

我们无法可靠地撤销对样式的任意改动。请移除相关宏包或改布局的命令后再试。

1 Introduction

1 引言

Batch reinforcement learning, the task of learning from a fixed dataset without further interactions with the environment, is a crucial requirement for scaling reinforcement learning to tasks where the data collection procedure is costly, risky, or time-consuming. Off-policy batch reinforcement learning has important implications for many practical applications. It is often preferable for data collection to be performed by some secondary controlled process, such as a human operator or a carefully monitored program. If assumptions on the quality of the behavioral policy can be made, imitation learning can be used to produce strong policies. However, most imitation learning algorithms are known to fail when exposed to suboptimal trajectories, or require further interactions with the environment to compensate (Hester et al. 2017; Sun et al. 2018; Cheng et al. 2018). On the other hand, batch reinforcement learning offers a mechanism for learning from a fixed dataset without restrictions on the quality of the data.

批次强化学习——在固定数据集上学习、不再与环境交互——是把强化学习扩展到数据采集昂贵、危险或耗时任务上的关键需求。离策略批次强化学习对许多实际应用有重要含义。数据采集往往更适合由某种受控的次级过程完成，例如人类操作员或受仔细监控的程序。若能对行为策略质量做假设，可用模仿学习得到强策略。但多数模仿学习算法在接触次优轨迹时会失败，或需要继续与环境交互来补偿（Hester et al. 2017；Sun et al. 2018；Cheng et al. 2018）。另一方面，批次强化学习提供了从固定数据集学习、且不对数据质量设限的机制。

Most modern off-policy deep reinforcement learning algorithms fall into the category of growing batch learning (Lange et al. 2012), in which data is collected and stored into an experience replay dataset (Lin 1992), which is used to train the agent before further data collection occurs. However, we find that these “off-policy” algorithms can fail in the batch setting, becoming unsuccessful if the dataset is uncorrelated to the true distribution under the current policy. Our most surprising result shows that off-policy agents perform dramatically worse than the behavioral agent when trained with the same algorithm on the same dataset.

多数现代离策略深度强化学习算法属于增长批次学习（Lange et al. 2012）：数据被采集并存入经验回放数据集（Lin 1992），用于训练智能体后再继续采集。但我们发现这些「离策略」算法在批次设定下会失败——若数据集与当前策略下的真实分布不相关，就学不好。最令人惊讶的结果是：用同一算法、同一数据集训练时，离策略智能体的表现会大幅差于行为智能体。

This inability to learn truly off-policy is due to a fundamental problem with off-policy reinforcement learning we denote extrapolation**error, a phenomenon in which unseen state-action pairs are erroneously estimated to have unrealistic values. Extrapolation error can be attributed to a mismatch in the distribution of data induced by the policy and the distribution of data contained in the batch. As a result, it may be impossible to learn a value function for a policy which selects actions not contained in the batch.

这种无法真正离策略学习的原因，在于我们称为外推误差的离策略强化学习基本问题：未见过的状态–动作对被错误估计为不切实际的价值。外推误差可归因于策略诱导的数据分布与批次中数据分布的失配。因此，若策略选择的动作不在批次中，可能根本无法为其学到价值函数。

To overcome extrapolation error in off-policy learning, we introduce batch-constrained reinforcement learning, where agents are trained to maximize reward while minimizing the mismatch between the state-action visitation of the policy and the state-action pairs contained in the batch. Our deep reinforcement learning algorithm, Batch-Constrained deep Q-learning (BCQ), uses a state-conditioned generative model to produce only previously seen actions. This generative model is combined with a Q-network, to select the highest valued action which is similar to the data in the batch. Under mild assumptions, we prove this batch-constrained paradigm is necessary for unbiased value estimation from incomplete datasets for finite deterministic MDPs.

为克服离策略学习中的外推误差，我们引入批次约束强化学习：训练智能体最大化回报的同时，最小化策略的状态–动作访问与批次中状态–动作对之间的失配。我们的深度强化学习算法——批次约束深度 Q 学习（BCQ）——用状态条件生成模型只产生先前见过的动作。该生成模型与 Q 网络结合，选出与批次数据相似且价值最高的动作。在温和假设下，我们证明：对有限确定性 MDP，要从不完整数据集得到无偏价值估计，这种批次约束范式是必要的。

Unlike any previous continuous control deep reinforcement learning algorithms, BCQ is able to learn successfully without interacting with the environment by considering extrapolation error. Our algorithm is evaluated on a series of batch reinforcement learning tasks in MuJoCo environments (Todorov et al. 2012; Brockman et al. 2016), where extrapolation error is particularly problematic due to the high-dimensional continuous action space, which is impossible to sample exhaustively. Our algorithm offers a unified view on imitation and off-policy learning, and is capable of learning from purely expert demonstrations, as well as from finite batches of suboptimal data, without further exploration. We remark that BCQ is only one way to approach batch-constrained reinforcement learning in a deep setting, and we hope that it will be serve as a foundation for future algorithms. To ensure reproducibility, we provide precise experimental and implementation details, and our code is made available (https://github.com/sfujim/BCQ).

与以往任何连续控制深度强化学习算法不同，BCQ 通过考虑外推误差，无需与环境交互即可成功学习。算法在 MuJoCo 环境（Todorov et al. 2012；Brockman et al. 2016）的一系列批次强化学习任务上评测；高维连续动作空间无法穷尽采样，外推误差在此尤为严重。我们的算法统一了模仿学习与离策略学习的视角，既能从纯专家示范学习，也能从有限批次的次优数据学习，而无需继续探索。需要指出，BCQ 只是深度设定下实现批次约束强化学习的一种方式，我们希望它能成为未来算法的基础。为确保可复现，我们提供精确实验与实现细节，并公开代码（https://github.com/sfujim/BCQ）。

2 Background

2 背景

In reinforcement learning, an agent interacts with its environment, typically assumed to be a Markov decision process (MDP) $(\mathcal{S},\mathcal{A},p_{M},r,\gamma)$, with state space $\mathcal{S}$, action space $\mathcal{A}$, and transition dynamics $p_{M}(s^{\prime}|s,a)$. At each discrete time step, the agent receives a reward $r(s,a,s^{\prime})\in\mathbb{R}$ for performing action $a$ in state $s$ and arriving at the state $s^{\prime}$. The goal of the agent is to maximize the expectation of the sum of discounted rewards, known as the return $R_{t}=\sum_{i={t+1}}^{\infty}\gamma^{i}r(s_{i},a_{i},s_{i+1})$, which weighs future rewards with respect to the discount factor $\gamma\in[0,1)$.

强化学习中，智能体与环境交互；环境通常假设为马尔可夫决策过程（MDP）$(\mathcal{S},\mathcal{A},p_{M},r,\gamma)$，含状态空间 $\mathcal{S}$、动作空间 $\mathcal{A}$ 与转移动力学 $p_{M}(s^{\prime}|s,a)$。在每个离散时间步，智能体在状态 $s$ 执行动作 $a$ 并到达 $s^{\prime}$ 时获得奖励 $r(s,a,s^{\prime})\in\mathbb{R}$。目标是最大化折扣奖励之和的期望，即回报 $R_{t}=\sum_{i={t+1}}^{\infty}\gamma^{i}r(s_{i},a_{i},s_{i+1})$，其中未来奖励由折扣因子 $\gamma\in[0,1)$ 加权。

The agent selects actions with respect to a policy $\pi:\mathcal{S}\rightarrow\mathcal{A}$, which exhibits a distribution $\mu^{\pi}(s)$ over the states $s\in\mathcal{S}$ visited by the policy. Each policy $\pi$ has a corresponding value function $Q^{\pi}(s,a)=\mathbb{E}_{\pi}[R_{t}|s,a]$, the expected return when following the policy after taking action $a$ in state $s$. For a given policy $\pi$, the value function can be computed through the Bellman operator $\mathcal{T}^{\pi}$:

智能体按策略 $\pi:\mathcal{S}\rightarrow\mathcal{A}$ 选动作；该策略在其访问的状态 $s\in\mathcal{S}$ 上诱导分布 $\mu^{\pi}(s)$。每个策略 $\pi$ 对应价值函数 $Q^{\pi}(s,a)=\mathbb{E}_{\pi}[R_{t}|s,a]$，即在状态 $s$ 采取动作 $a$ 后再遵循该策略的期望回报。对给定策略 $\pi$，价值函数可通过 Bellman 算子 $\mathcal{T}^{\pi}$ 计算：

$$\mathcal{T}^{\pi}Q(s,a)=\mathbb{E}_{s^{\prime}}[r+\gamma Q(s^{\prime},\pi(s^{\prime}))]. \tag{1}$$

The Bellman operator is a contraction for $\gamma\in[0,1)$ with unique fixed point $Q^{\pi}(s,a)$ (Bertsekas & Tsitsiklis 1996). $Q^{*}(s,a)=\max_{\pi}Q^{\pi}(s,a)$ is known as the optimal value function, which has a corresponding optimal policy obtained through greedy action choices. For large or continuous state and action spaces, the value can be approximated with neural networks, e.g. using the Deep Q-Network algorithm (DQN) (Mnih et al. 2015). In DQN, the value function $Q_{\theta}$ is updated using the target:

对 $\gamma\in[0,1)$，Bellman 算子是压缩映射，有唯一不动点 $Q^{\pi}(s,a)$（Bertsekas & Tsitsiklis 1996）。$Q^{*}(s,a)=\max_{\pi}Q^{\pi}(s,a)$ 称为最优价值函数，对应通过贪婪选动作得到的最优策略。对大或连续的状态与动作空间，可用神经网络近似价值，例如 Deep Q-Network 算法（DQN）（Mnih et al. 2015）。在 DQN 中，价值函数 $Q_{\theta}$ 用如下目标更新：

$$r+\gamma Q_{\theta^{\prime}}(s^{\prime},\pi(s^{\prime})),\quad\pi(s^{\prime})=\oneargmax_{a}Q_{\theta^{\prime}}(s^{\prime},a), \tag{2}$$

Q-learning is an off-policy algorithm (Sutton & Barto 1998), meaning the target can be computed without consideration of how the experience was generated. In principle, off-policy reinforcement learning algorithms are able to learn from data collected by any behavioral policy. Typically, the loss is minimized over mini-batches of tuples of the agent’s past data, $(s,a,r,s^{\prime})\in\mathcal{B}$, sampled from an experience replay dataset $\mathcal{B}$ (Lin 1992). For shorthand, we often write $s\in\mathcal{B}$ if there exists a transition tuple containing $s$ in the batch $\mathcal{B}$, and similarly for $(s,a)$ or $(s,a,s^{\prime})\in\mathcal{B}$. In batch reinforcement learning, we assume $\mathcal{B}$ is fixed and no further interaction with the environment occurs. To further stabilize learning, a target network with frozen parameters $Q_{\theta^{\prime}}$, is used in the learning target. The parameters of the target network $\theta^{\prime}$ are updated to the current network parameters $\theta$ after a fixed number of time steps, or by averaging $\theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta^{\prime}$ for some small $\tau$ (Lillicrap et al. 2015).

Q-learning 是离策略算法（Sutton & Barto 1998），意味着计算目标时无需考虑经验如何产生。原则上，离策略强化学习算法可从任意行为策略采集的数据中学习。通常，损失在智能体历史数据的小批次元组 $(s,a,r,s^{\prime})\in\mathcal{B}$ 上最小化，这些元组从经验回放数据集 $\mathcal{B}$（Lin 1992）中采样。为简写，若批次 $\mathcal{B}$ 中存在含 $s$ 的转移元组，则写 $s\in\mathcal{B}$；对 $(s,a)$ 或 $(s,a,s^{\prime})\in\mathcal{B}$ 同理。在批次强化学习中，我们假设 $\mathcal{B}$ 固定且不再与环境交互。为进一步稳定学习，学习目标中使用参数冻结的目标网络 $Q_{\theta^{\prime}}$。目标网络参数 $\theta^{\prime}$ 在固定步数后更新为当前网络参数 $heta$，或以小 $\tau$ 做平均 $\theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta^{\prime}$（Lillicrap et al. 2015）。

In a continuous action space, the analytic maximum of Equation (2) is intractable. In this case, actor-critic methods are commonly used, where action selection is performed through a separate policy network $\pi_{\phi}$, known as the actor, and updated with respect to a value estimate, known as the critic (Sutton & Barto 1998; Konda & Tsitsiklis 2003). This policy can be updated following the deterministic policy gradient theorem (Silver et al. 2014):

在连续动作空间中，公式 (2) 的解析最大值不可求。此时常用 actor-critic：由独立的策略网络 $\pi_{\phi}$（actor）选动作，并相对价值估计（critic）更新（Sutton & Barto 1998；Konda & Tsitsiklis 2003）。该策略可按确定性策略梯度定理（Silver et al. 2014）更新：

$$\phi\leftarrow\oneargmax_{\phi}\mathbb{E}_{s\in\mathcal{B}}[Q_{\theta}(s,\pi_{\phi}(s))], \tag{3}$$

which corresponds to learning an approximation to the maximum of $Q_{\theta}$, by propagating the gradient through both $\pi_{\phi}$ and $Q_{\theta}$. When combined with off-policy deep Q-learning to learn $Q_{\theta}$, this algorithm is referred to as Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al. 2015).

这对应于通过把梯度同时经 $\pi_{\phi}$ 与 $Q_{\theta}$ 回传，学习对 $Q_{\theta}$ 最大值的近似。与用离策略深度 Q 学习来学 $Q_{\theta}$ 相结合时，该算法称为深度确定性策略梯度（DDPG）（Lillicrap et al. 2015）。

3 Extrapolation Error

3 外推误差

Extrapolation error is an error in off-policy value learning which is introduced by the mismatch between the dataset and true state-action visitation of the current policy. The value estimate $Q(s,a)$ is affected by extrapolation error during a value update where the target policy selects an unfamiliar action $a^{\prime}$ at the next state $s^{\prime}$ in the backed-up value estimate, such that $(s^{\prime},a^{\prime})$ is unlikely, or not contained, in the dataset. The cause of extrapolation error can be attributed to several related factors:

外推误差是离策略价值学习中的误差，由数据集与当前策略真实状态–动作访问之间的失配引入。当价值更新时，目标策略在下一状态 $s^{\prime}$ 选出不熟悉的动作 $a^{\prime}$，使备份价值估计中的 $(s^{\prime},a^{\prime})$ 在数据集中不太可能出现或不存在，则 $Q(s,a)$ 会受外推误差影响。外推误差的成因可归为若干相关因素：

Absent Data. If any state-action pair $(s,a)$ is unavailable, then error is introduced as some function of the amount of similar data and approximation error. This means that the estimate of $Q_{\theta}(s^{\prime},\pi(s^{\prime}))$ may be arbitrarily bad without sufficient data near $(s^{\prime},\pi(s^{\prime}))$.

数据缺失。 若任一状态–动作对 $(s,a)$ 不可用，则误差是相似数据量与逼近误差的某种函数。这意味着若 $(s^{\prime},\pi(s^{\prime}))$ 附近数据不足，$Q_{\theta}(s^{\prime},\pi(s^{\prime}))$ 的估计可能任意差。

Model Bias. When performing off-policy Q-learning with a batch $\mathcal{B}$, the Bellman operator $\mathcal{T}^{\pi}$ is approximated by sampling transitions tuples $(s,a,r,s^{\prime})$ from $\mathcal{B}$ to estimate the expectation over $s^{\prime}$. However, for a stochastic MDP, without infinite state-action visitation, this produces a biased estimate of the transition dynamics:

模型偏差。 用批次 $\mathcal{B}$ 做离策略 Q 学习时，Bellman 算子 $\mathcal{T}^{\pi}$ 通过从 $\mathcal{B}$ 采样转移元组 $(s,a,r,s^{\prime})$ 来近似对 $s^{\prime}$ 的期望。但对随机 MDP，若没有无穷的状态–动作访问，这会对转移动力学产生有偏估计：

$$\mathcal{T}^{\pi}Q(s,a)\approx\mathbb{E}_{s^{\prime}\sim\mathcal{B}}[r+\gamma Q(s^{\prime},\pi(s^{\prime}))], \tag{4}$$

where the expectation is with respect to transitions in the batch $\mathcal{B}$, rather than the true MDP.

其中期望相对批次 $\mathcal{B}$ 中的转移，而非真实 MDP。

Training Mismatch. Even with sufficient data, in deep Q-learning systems, transitions are sampled uniformly from the dataset, giving a loss weighted with respect to the likelihood of data in the batch:

训练失配。 即便数据充足，在深度 Q 学习系统中，转移从数据集均匀采样，损失按批次中数据的似然加权：

$$\approx\frac{1}{|\mathcal{B}|}\sum_{(s,a,r,s^{\prime})\in\mathcal{B}}||r+\gamma Q_{\theta^{\prime}}(s^{\prime},\pi(s^{\prime}))-Q_{\theta}(s,a)||^{2}. \tag{5}$$

If the distribution of data in the batch does not correspond with the distribution under the current policy, the value function may be a poor estimate of actions selected by the current policy, due to the mismatch in training.

若批次中数据分布与当前策略下的分布不一致，由于训练失配，价值函数可能对当前策略所选动作给出很差的估计。

We remark that re-weighting the loss in Equation (5) with respect to the likelihood under the current policy can still result in poor estimates if state-action pairs with high likelihood under the current policy are not found in the batch. This means only a subset of possible policies can be evaluated accurately. As a result, learning a value estimate with off-policy data can result in large amounts of extrapolation error if the policy selects actions which are not similar to the data found in the batch. In the following section, we discuss how state of the art off-policy deep reinforcement learning algorithms fail to address the concern of extrapolation error, and demonstrate the implications in practical examples.

需要指出：即便把公式 (5) 的损失按当前策略下的似然重加权，若当前策略下高似然的状态–动作对不在批次中，估计仍可能很差。这意味着只有一部分可能策略能被准确评估。因此，用离策略数据学习价值估计时，若策略选择与批次数据不相似的动作，可能产生大量外推误差。下一节讨论当前最先进的离策略深度强化学习算法如何未能解决外推误差，并在实际例子中展示其影响。

3.1 Extrapolation Error in Deep Reinforcement Learning

3.1 深度强化学习中的外推误差

Deep Q-learning algorithms (Mnih et al. 2015) have been labeled as off-policy due to their connection to off-policy Q-learning (Watkins 1989). However, these algorithms tend to use near-on-policy exploratory policies, such as $\epsilon$-greedy, in conjunction with a replay buffer (Lin 1992). As a result, the generated dataset tends to be heavily correlated to the current policy. In this section, we examine how these off-policy algorithms perform when learning with uncorrelated datasets. Our results demonstrate that the performance of a state of the art deep actor-critic algorithm, DDPG (Lillicrap et al. 2015), deteriorates rapidly when the data is uncorrelated and the value estimate produced by the deep Q-network diverges. These results suggest that off-policy deep reinforcement learning algorithms are ineffective when learning truly off-policy.

深度 Q 学习算法（Mnih et al. 2015）因其与离策略 Q-learning（Watkins 1989）的联系而被标为离策略。但这些算法往往配合回放缓冲使用近同策略的探索策略，如 $\epsilon$-greedy（Lin 1992）。因此生成的数据集往往与当前策略高度相关。本节考察这些离策略算法在不相关数据集上学习时的表现。结果表明：当前先进的深度 actor-critic 算法 DDPG（Lillicrap et al. 2015）在数据不相关时表现迅速恶化，深度 Q 网络产生的价值估计发散。这些结果说明，离策略深度强化学习算法在真正离策略学习时是无效的。

Our practical experiments examine three different batch settings in OpenAI gym’s Hopper-v1 environment (Todorov et al. 2012; Brockman et al. 2016), which we use to train an off-policy DDPG agent with no interaction with the environment. Experiments with additional environments and specific details can be found in the Supplementary Material.

实际实验在 OpenAI gym 的 Hopper-v1 环境（Todorov et al. 2012；Brockman et al. 2016）上考察三种批次设定，用其训练离策略 DDPG 智能体且不与环境交互。更多环境与具体细节见补充材料。

(a) Final buffer performance

Batch 1 (Final buffer). We train a DDPG agent for 1 million time steps, adding $\mathcal{N}(0,0.5)$ Gaussian noise to actions for high exploration, and store all experienced transitions. This collection procedure creates a dataset with a diverse set of states and actions, with the aim of sufficient coverage.

批次 1（最终缓冲）。 训练一个 DDPG 智能体 100 万步，对动作加 $\mathcal{N}(0,0.5)$ 高斯噪声以增强探索，并存储全部经历的转移。该采集过程旨在得到状态与动作多样、覆盖充分的数据集。

Batch 2 (Concurrent). We concurrently train the off-policy and behavioral DDPG agents, for 1 million time steps. To ensure sufficient exploration, a standard $\mathcal{N}(0,0.1)$ Gaussian noise is added to actions taken by the behavioral policy. Each transition experienced by the behavioral policy is stored in a buffer replay, which both agents learn from. As a result, both agents are trained with the identical dataset.

批次 2（并发）。 同时训练离策略与行为 DDPG 智能体共 100 万步。为保证充分探索，行为策略动作加标准 $\mathcal{N}(0,0.1)$ 高斯噪声。行为策略经历的每次转移存入回放缓冲，两个智能体都从中学习。因此两者用同一数据集训练。

Batch 3 (Imitation). A trained DDPG agent acts as an expert, and is used to collect a dataset of 1 million transitions.

批次 3（模仿）。 训练好的 DDPG 智能体充当专家，采集 100 万次转移的数据集。

In Figure 1, we graph the performance of the agents as they train with each batch, as well as their value estimates. Straight lines represent the average return of episodes contained in the batch. Additionally, we graph the learning performance of the behavioral agent for the relevant tasks.

在图 1 中，我们绘制各批次训练过程中智能体的表现及其价值估计。直线表示批次中所含回合的平均回报。此外还绘制相关任务上行为智能体的学习曲线。

Our experiments demonstrate several surprising facts about off-policy deep reinforcement learning agents. In each task, the off-policy agent performances significantly worse than the behavioral agent. Even in the concurrent experiment, where both agents are trained with the same dataset, there is a large gap in performance in every single trial. This result suggests that differences in the state distribution under the initial policies is enough for extrapolation error to drastically offset the performance of the off-policy agent. Additionally, the corresponding value estimate exhibits divergent behavior, while the value estimate of the behavioral agent is highly stable. In the final buffer experiment, the off-policy agent is provided with a large and diverse dataset, with the aim of providing sufficient coverage of the initial policy. Even in this instance, the value estimate is highly unstable, and the performance suffers. In the imitation setting, the agent is provided with expert data. However, the agent quickly learns to take non-expert actions, under the guise of optimistic extrapolation. As a result, the value estimates rapidly diverge and the agent fails to learn.

实验揭示了关于离策略深度强化学习智能体的若干令人惊讶的事实。在每项任务上，离策略智能体都显著差于行为智能体。即便在并发实验中两者用同一数据集训练，每次试验仍有很大表现差距。这表明初始策略下状态分布的差异就足以让外推误差大幅抵消离策略智能体的表现。同时，对应的价值估计呈现发散行为，而行为智能体的价值估计高度稳定。在最终缓冲实验中，离策略智能体得到大而多样的数据集，本意是充分覆盖初始策略；即便如此，价值估计仍高度不稳，表现受损。在模仿设定中，智能体得到专家数据，却很快在乐观外推的假象下学会采取非专家动作，价值估计迅速发散，学习失败。

Although extrapolation error is not necessarily positively biased, when combined with maximization in reinforcement learning algorithms, extrapolation error provides a source of noise that can induce a persistent overestimation bias (Thrun & Schwartz 1993; Van Hasselt et al. 2016; Fujimoto et al. 2018). In an on-policy setting, extrapolation error may be a source of beneficial exploration through an implicit “optimism in the face of uncertainty” strategy (Lai & Robbins 1985; Jaksch et al. 2010). In this case, if the value function overestimates an unknown state-action pair, the policy will collect data in the region of uncertainty, and the value estimate will be corrected. However, when learning off-policy, or in a batch setting, extrapolation error will never be corrected due to the inability to collect new data.

尽管外推误差未必带正偏，但与强化学习算法中的最大化结合时，它会成为噪声源，诱发持续的高估偏差（Thrun & Schwartz 1993；Van Hasselt et al. 2016；Fujimoto et al. 2018）。在同策略设定下，外推误差可能通过隐式「面对不确定性时乐观」策略成为有益探索（Lai & Robbins 1985；Jaksch et al. 2010）：若价值函数高估未知状态–动作对，策略会在不确定区域采集数据，价值估计得到校正。但在离策略或批次设定下，因无法采集新数据，外推误差永远得不到校正。

These experiments show extrapolation error can be highly detrimental to learning off-policy in a batch reinforcement learning setting. While the continuous state space and multi-dimensional action space in MuJoCo environments are contributing factors to extrapolation error, the scale of these tasks is small compared to real world settings. As a result, even with a sufficient amount of data collection, extrapolation error may still occur due to the concern of catastrophic forgetting (McCloskey & Cohen 1989; Goodfellow et al. 2013). Consequently, off-policy reinforcement learning algorithms used in the real-world will require practical guarantees without exhaustive amounts of data.

这些实验表明，外推误差对批次强化学习设定下的离策略学习极为有害。MuJoCo 环境的连续状态空间与多维动作空间虽是外推误差的促成因素，但这些任务的规模相对真实世界仍偏小。因此，即便采集了足够数据，仍可能因灾难性遗忘（McCloskey & Cohen 1989；Goodfellow et al. 2013）出现外推误差。结果是，用于真实世界的离策略强化学习算法需要在不依赖穷尽数据量的前提下给出实用保证。

4 Batch-Constrained Reinforcement Learning

4 批次约束强化学习

Current off-policy deep reinforcement learning algorithms fail to address extrapolation error by selecting actions with respect to a learned value estimate, without consideration of the accuracy of the estimate. As a result, certain out-of-distribution actions can be erroneously extrapolated to higher values. However, the value of an off-policy agent can be accurately evaluated in regions where data is available. We propose a conceptually simple idea: to avoid extrapolation error a policy should induce a similar state-action visitation to the batch. We denote policies which satisfy this notion as batch-constrained. To optimize off-policy learning for a given batch, batch-constrained policies are trained to select actions with respect to three objectives:

当前离策略深度强化学习算法按学到的价值估计选动作，不考虑估计是否准确，因而未能解决外推误差。结果是某些分布外动作会被错误外推到更高价值。但在有数据的区域，离策略智能体的价值可以被准确评估。我们提出一个概念上简单的想法：为避免外推误差，策略应诱导与批次相似的状态–动作访问。满足这一概念的策略称为批次约束策略。为对给定批次优化离策略学习，批次约束策略按三个目标选动作：

(1) Minimize the distance of selected actions to the data in the batch.

(1) 最小化所选动作与批次数据之间的距离。

(2) Lead to states where familiar data can be observed.

(2) 导向可观察到熟悉数据的状态。

(3) Maximize the value function.

(3) 最大化价值函数。

We note the importance of objective (1) above the others, as the value function and estimates of future states may be arbitrarily poor without access to the corresponding transitions. That is, we cannot correctly estimate (2) and (3) unless (1) is sufficiently satisfied. As a result, we propose optimizing the value function, along with some measure of future certainty, with a constraint limiting the distance of selected actions to the batch. This is achieved in our deep reinforcement learning algorithm through a state-conditioned generative model, to produce likely actions under the batch. This generative model is combined with a network which aims to optimally perturb the generated actions in a small range, along with a Q-network, used to select the highest valued action. Finally, we train a pair of Q-networks, and take the minimum of their estimates during the value update. This update penalizes states which are unfamiliar, and pushes the policy to select actions which lead to certain data.

我们强调目标 (1) 相对其他目标的重要性：若无法访问相应转移，价值函数与未来状态估计可能任意差。也就是说，除非 (1) 被充分满足，否则无法正确估计 (2) 与 (3)。因此，我们建议在限制所选动作到批次距离的约束下，优化价值函数以及某种未来确定性度量。深度强化学习算法通过状态条件生成模型实现这一点，以产生批次下似然高的动作。该生成模型与一个旨在小范围内最优扰动生成动作的网络、以及用于选出最高价值动作的 Q 网络相结合。最后，我们训练一对 Q 网络，并在价值更新时取它们估计的最小值。该更新惩罚不熟悉的状态，并推动策略选择导向确定数据的动作。

We begin by analyzing the theoretical properties of batch-constrained policies in a finite MDP setting, where we are able to quantify extrapolation error precisely. We then introduce our deep reinforcement learning algorithm in detail, Batch-Constrained deep Q-learning (BCQ) by drawing inspiration from the tabular analogue.

我们先在有限 MDP 设定下分析批次约束策略的理论性质，以便精确量化外推误差。随后详细介绍受表格情形启发的深度强化学习算法——批次约束深度 Q 学习（BCQ）。

4.1 Addressing Extrapolation Error in Finite MDPs

4.1 在有限 MDP 中应对外推误差

In the finite MDP setting, extrapolation error can be described by the bias from the mismatch between the transitions contained in the buffer and the true MDP. We find that by inducing a data distribution that is contained entirely within the batch, batch-constrained policies can eliminate extrapolation error entirely for deterministic MDPs. In addition, we show that the batch-constrained variant of Q-learning converges to the optimal policy under the same conditions as the standard form of Q-learning. Moreover, we prove that for a deterministic MDP, batch-constrained Q-learning is guaranteed to match, or outperform, the behavioral policy when starting from any state contained in the batch. All of the proofs for this section can be found in the Supplementary Material.

在有限 MDP 设定下，外推误差可由缓冲中所含转移与真实 MDP 之间的偏差来描述。我们发现：通过诱导完全包含在批次内的数据分布，批次约束策略可对确定性 MDP 完全消除外推误差。此外，我们证明批次约束版 Q-learning 在与标准 Q-learning 相同的条件下收敛到最优策略。更进一步，我们证明：对确定性 MDP，从批次中任一状态出发，批次约束 Q-learning 保证匹配或超过行为策略。本节全部证明见补充材料。

A value estimate $Q$ can be learned using an experience replay buffer $\mathcal{B}$. This involves sampling transition tuples $(s,a,r,s^{\prime})$ with uniform probability, and applying the temporal difference update (Sutton 1988; Watkins 1989):

可用经验回放缓冲 $\mathcal{B}$ 学习价值估计 $Q$。做法是以均匀概率采样转移元组 $(s,a,r,s^{\prime})$，并应用时序差分更新（Sutton 1988；Watkins 1989）：

$$Q(s,a)\leftarrow(1-\alpha)Q(s,a)+\alpha\left(r+\gamma Q(s^{\prime},\pi(s^{\prime}))\right). \tag{6}$$

If $\pi(s^{\prime})=\argmax_{a^{\prime}}Q(s^{\prime},a^{\prime})$, this is known as Q-learning. Assuming a non-zero probability of sampling any possible transition tuple from the buffer and infinite updates, Q-learning converges to the optimal value function.

若 $\pi(s^{\prime})=\argmax_{a^{\prime}}Q(s^{\prime},a^{\prime})$，即 Q-learning。假设能以非零概率从缓冲采样任意可能转移元组且更新无穷多次，则 Q-learning 收敛到最优价值函数。

We begin by showing that the value function $Q$ learned with the batch $\mathcal{B}$ corresponds to the value function for an alternate MDP $M_{\mathcal{B}}$. From the true MDP $M$ and initial values $Q(s,a)$, we define the new MDP $M_{\mathcal{B}}$ with the same action and state space as $M$, along with an additional terminal state $s_{\text{init}}$. $M_{\mathcal{B}}$ has transition probabilities $p_{\mathcal{B}}(s^{\prime}|s,a)=\frac{N(s,a,s^{\prime})}{\sum_{\tilde{s}}N(s,a,\tilde{s})}$, where $N(s,a,s^{\prime})$ is the number of times the tuple $(s,a,s^{\prime})$ is observed in $\mathcal{B}$. If $\sum_{\tilde{s}}N(s,a,\tilde{s})=0$, then $p_{\mathcal{B}}(s_{\text{init}}|s,a)=1$, where $r(s,a,s_{\text{init}})$ is set to the initialized value of $Q(s,a)$.

我们首先证明：用批次 $\mathcal{B}$ 学到的价值函数 $Q$ 对应于另一 MDP $M_{\mathcal{B}}$ 的价值函数。从真实 MDP $M$ 与初始值 $Q(s,a)$ 出发，定义新 MDP $M_{\mathcal{B}}$：动作与状态空间与 $M$ 相同，并额外增加终止状态 $s_{\text{init}}$。$M_{\mathcal{B}}$ 的转移概率为 $p_{\mathcal{B}}(s^{\prime}|s,a)=\frac{N(s,a,s^{\prime})}{\sum_{\tilde{s}}N(s,a,\tilde{s})}$，其中 $N(s,a,s^{\prime})$ 是元组 $(s,a,s^{\prime})$ 在 $\mathcal{B}$ 中出现的次数。若 $\sum_{\tilde{s}}N(s,a,\tilde{s})=0$，则 $p_{\mathcal{B}}(s_{\text{init}}|s,a)=1$，并把 $r(s,a,s_{\text{init}})$ 设为 $Q(s,a)$ 的初始化值。

Theorem 1. Performing Q-learning by sampling from a batch $\mathcal{B}$ converges to the optimal value function under the MDP $M_{\mathcal{B}}$.

定理 1。 通过从批次 $\mathcal{B}$ 采样做 Q-learning，收敛到 MDP $M_{\mathcal{B}}$ 下的最优价值函数。

We define $\epsilon_{\text{MDP}}$ as the tabular extrapolation error, which accounts for the discrepancy between the value function $Q^{\pi}_{\mathcal{B}}$ computed with the batch $\mathcal{B}$ and the value function $Q^{\pi}$ computed with the true MDP $M$:

我们将 $\epsilon_{\text{MDP}}$ 定义为表格外推误差，用以刻画用批次 $\mathcal{B}$ 计算的价值函数 $Q^{\pi}_{\mathcal{B}}$ 与用真实 MDP $M$ 计算的价值函数 $Q^{\pi}$ 之间的差异：

$$\epsilon_{\text{MDP}}(s,a)=Q^{\pi}(s,a)-Q^{\pi}_{\mathcal{B}}(s,a). \tag{7}$$

For any policy $\pi$, the exact form of $\epsilon_{\text{MDP}}(s,a)$ can be computed through a Bellman-like equation:

对任意策略 $\pi$，$\epsilon_{\text{MDP}}(s,a)$ 的精确形式可通过类 Bellman 方程计算：

$$\begin{split}\epsilon_{\text{MDP}}(s,a)~&=\sum_{s^{\prime}}\left(p_{M}(s^{\prime}|s,a)-p_{\mathcal{B}}(s^{\prime}|s,a)\right)\
&\left(r(s,a,s^{\prime})+\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})Q^{\pi}{\mathcal{B}}(s^{\prime},a^{\prime})\right)\
&+p{M}(s^{\prime}|s,a)\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})\epsilon_{\text{MDP}}(s^{\prime},a^{\prime}).\end{split} \tag{8}$$

This means extrapolation error is a function of divergence in the transition distributions, weighted by value, along with the error at succeeding states. If the policy is chosen carefully, the error between value functions can be minimized by visiting regions where the transition distributions are similar. For simplicity, we denote

这意味着外推误差是转移分布差异（按价值加权）以及后续状态误差的函数。若谨慎选择策略，可通过访问转移分布相近的区域来最小化价值函数之间的误差。为简便，记

$$\epsilon^{\pi}_{\text{MDP}}=\sum_{s}\mu_{\pi}(s)\sum_{a}\pi(a|s)|\epsilon_{\text{MDP}}(s,a)|. \tag{9}$$

To evaluate a policy $\pi$ exactly at relevant state-action pairs, only $\epsilon^{\pi}_{\text{MDP}}=0$ is required. We can then determine the condition required to evaluate the exact expected return of a policy without extrapolation error.

要在相关状态–动作对上精确评估策略 $\pi$，只需 $\epsilon^{\pi}_{\text{MDP}}=0$。由此可确定在无外推误差下精确评估策略期望回报所需的条件。

Lemma 1. For all reward functions, $\epsilon^{\pi}_{\text{MDP}}=0$ if and only if $p_{\mathcal{B}}(s^{\prime}|s,a)=p_{M}(s^{\prime}|s,a)$ for all $s^{\prime}\in\mathcal{S}$ and $(s,a)$ such that $\mu_{\pi}(s)>0$ and $\pi(a|s)>0$.

引理 1。 对所有奖励函数，$\epsilon^{\pi}_{\text{MDP}}=0$ 当且仅当对所有满足 $\mu_{\pi}(s)>0$ 且 $\pi(a|s)>0$ 的 $(s,a)$ 以及所有 $s^{\prime}\in\mathcal{S}$，有 $p_{\mathcal{B}}(s^{\prime}|s,a)=p_{M}(s^{\prime}|s,a)$。

Lemma 1 states that if $M_{\mathcal{B}}$ and $M$ exhibit the same transition probabilities in regions of relevance, the policy can be accurately evaluated. For a stochastic MDP this may require an infinite number of samples to converge to the true distribution, however, for a deterministic MDP this requires only a single transition. This means a policy which only traverses transitions contained in the batch, can be evaluated without error. More formally, we denote a policy $\pi\in\Pi_{\mathcal{B}}$ as batch-constrained if for all $(s,a)$ where $\mu_{\pi}(s)>0$ and $\pi(a|s)>0$ then $(s,a)\in\mathcal{B}$. Additionally, we define a batch $\mathcal{B}$ as coherent if for all $(s,a,s^{\prime})\in\mathcal{B}$ then $s^{\prime}\in\mathcal{B}$ unless $s^{\prime}$ is a terminal state. This condition is trivially satisfied if the data is collected in trajectories, or if all possible states are contained in the batch. With a coherent batch, we can guarantee the existence of a batch-constrained policy.

引理 1 表明：若 $M_{\mathcal{B}}$ 与 $M$ 在相关区域展现相同转移概率，则策略可被准确评估。对随机 MDP，这可能需要无穷多样本才能收敛到真实分布；但对确定性 MDP，只需单次转移。这意味着只遍历批次中所含转移的策略可无误差评估。更形式地，若对所有满足 $\mu_{\pi}(s)>0$ 且 $\pi(a|s)>0$ 的 $(s,a)$ 都有 $(s,a)\in\mathcal{B}$，则称策略 $\pi\in\Pi_{\mathcal{B}}$ 为批次约束。此外，若对所有 $(s,a,s^{\prime})\in\mathcal{B}$，除非 $s^{\prime}$ 为终止状态否则 $s^{\prime}\in\mathcal{B}$，则称批次 $\mathcal{B}$ 为连贯。若数据按轨迹采集，或批次包含全部可能状态，该条件平凡满足。有了连贯批次，即可保证批次约束策略存在。

Theorem 2. For a deterministic MDP and all reward functions, $\epsilon^{\pi}_{\text{MDP}}=0$ if and only if the policy $\pi$ is batch-constrained. Furthermore, if $\mathcal{B}$ is coherent, then such a policy must exist if the start state $s_{0}\in\mathcal{B}$.

定理 2。 对确定性 MDP 与所有奖励函数，$\epsilon^{\pi}_{\text{MDP}}=0$ 当且仅当策略 $\pi$ 是批次约束的。此外，若 $\mathcal{B}$ 连贯，且起始状态 $s_{0}\in\mathcal{B}$，则这样的策略必然存在。

Batch-constrained policies can be used in conjunction with Q-learning to form batch-constrained Q-learning (BCQL), which follows the standard tabular Q-learning update while constraining the possible actions with respect to the batch:

批次约束策略可与 Q-learning 结合形成批次约束 Q-learning（BCQL）：遵循标准表格 Q-learning 更新，同时按批次约束可能动作：

$$Q(s,a)\leftarrow(1-\alpha)Q(s,a)+\alpha(r+\gamma\max_{a^{\prime}\text{s.t.}(s^{\prime},a^{\prime})\in\mathcal{B}}Q(s^{\prime},a^{\prime})). \tag{10}$$

BCQL converges under the same conditions as the standard form of Q-learning, noting the batch-constraint is nonrestrictive given infinite state-action visitation.

BCQL 在与标准 Q-learning 相同的条件下收敛；注意在无穷状态–动作访问下，批次约束并无限制性。

Theorem 3. Given the Robbins-Monro stochastic convergence conditions on the learning rate $\alpha$, and standard sampling requirements from the environment, BCQL converges to the optimal value function $Q^{}$.*

定理 3。 给定学习率 $\alpha$ 的 Robbins-Monro 随机收敛条件，以及环境的标准采样要求，BCQL 收敛到最优价值函数 $Q^{}$。*

The more interesting property of BCQL is that for a deterministic MDP and any coherent batch $\mathcal{B}$, BCQL converges to the optimal batch-constrained policy $\pi^{*}\in\Pi_{\mathcal{B}}$ such that $Q^{\pi^{*}}(s,a)\geq Q^{\pi}(s,a)$ for all $\pi\in\Pi_{\mathcal{B}}$ and $(s,a)\in\mathcal{B}$.

BCQL 更有趣的性质是：对确定性 MDP 与任意连贯批次 $\mathcal{B}$，BCQL 收敛到最优批次约束策略 $\pi^{*}\in\Pi_{\mathcal{B}}$，使得对所有 $\pi\in\Pi_{\mathcal{B}}$ 与 $(s,a)\in\mathcal{B}$，有 $Q^{\pi^{*}}(s,a)\geq Q^{\pi}(s,a)$。

Theorem 4. Given a deterministic MDP and coherent batch $\mathcal{B}$, along with the Robbins-Monro stochastic convergence conditions on the learning rate $\alpha$ and standard sampling requirements on the batch $\mathcal{B}$, BCQL converges to $Q^{\pi}_{\mathcal{B}}(s,a)$ where $\pi^{}(s)=\argmax_{a\text{ s.t.}(s,a)\in\mathcal{B}}Q^{\pi}_{\mathcal{B}}(s,a)$ is the optimal batch-constrained policy.*

定理 4。 给定确定性 MDP 与连贯批次 $\mathcal{B}$，以及学习率 $\alpha$ 的 Robbins-Monro 随机收敛条件与对批次 $\mathcal{B}$ 的标准采样要求，BCQL 收敛到 $Q^{\pi}_{\mathcal{B}}(s,a)$，其中 $\pi^{}(s)=\argmax_{a\text{ s.t.}(s,a)\in\mathcal{B}}Q^{\pi}_{\mathcal{B}}(s,a)$ 是最优批次约束策略。*

This means that BCQL is guaranteed to outperform any behavioral policy when starting from any state contained in the batch, effectively outperforming imitation learning. Unlike standard Q-learning, there is no condition on state-action visitation, other than coherency in the batch.

这意味着从批次中任一状态出发，BCQL 保证超过任意行为策略，从而有效优于模仿学习。与标准 Q-learning 不同，除批次连贯性外，对状态–动作访问没有其他条件。

4.2 Batch-Constrained Deep Reinforcement Learning

4.2 批次约束深度强化学习

We introduce our approach to off-policy batch reinforcement learning, Batch-Constrained deep Q-learning (BCQ). BCQ approaches the notion of batch-constrained through a generative model. For a given state, BCQ generates plausible candidate actions with high similarity to the batch, and then selects the highest valued action through a learned Q-network. Furthermore, we bias this value estimate to penalize rare, or unseen, states through a modification to Clipped Double Q-learning (Fujimoto et al. 2018). As a result, BCQ learns a policy with a similar state-action visitation to the data in the batch, as inspired by the theoretical benefits of its tabular counterpart.

我们提出离策略批次强化学习方法——批次约束深度 Q 学习（BCQ）。BCQ 通过生成模型实现批次约束概念：对给定状态，生成与批次高度相似的候选动作，再经学到的 Q 网络选出最高价值动作。此外，我们修改 Clipped Double Q-learning（Fujimoto et al. 2018），使价值估计偏向惩罚罕见或未见状态。结果是，BCQ 学到与批次数据具有相似状态–动作访问的策略，呼应其表格对应物的理论好处。

To maintain the notion of batch-constraint, we define a similarity metric by making the assumption that for a given state $s$, the similarity between $(s,a)$ and the state-action pairs in the batch $\mathcal{B}$ can be modelled using a learned state-conditioned marginal likelihood $P_{\mathcal{B}}^{G}(a|s)$. In this case, it follows that the policy maximizing $P_{\mathcal{B}}^{G}(a|s)$ would minimize the error induced by extrapolation from distant, or unseen, state-action pairs, by only selecting the most likely actions in the batch with respect to a given state. Given the difficulty of estimating $P_{\mathcal{B}}^{G}(a|s)$ in high-dimensional continuous spaces, we instead train a parametric generative model of the batch $G_{\omega}(s)$, which we can sample actions from, as a reasonable approximation to $\argmax_{a}P_{\mathcal{B}}^{G}(a|s)$.

为维持批次约束概念，我们定义相似度量，并假设：对给定状态 $s$，$(s,a)$ 与批次 $\mathcal{B}$ 中状态–动作对的相似性可用学到的状态条件边缘似然 $P_{\mathcal{B}}^{G}(a|s)$ 建模。此时，最大化 $P_{\mathcal{B}}^{G}(a|s)$ 的策略会通过只选批次中相对给定状态最可能的动作，最小化由遥远或未见状态–动作对外推引入的误差。鉴于在高维连续空间估计 $P_{\mathcal{B}}^{G}(a|s)$ 很难，我们改为训练批次的参数生成模型 $G_{\omega}(s)$，从中采样动作，作为对 $\argmax_{a}P_{\mathcal{B}}^{G}(a|s)$ 的合理近似。

For our generative model we use a conditional variational auto-encoder (VAE) (Kingma & Welling 2013; Sohn et al. 2015), which models the distribution by transforming an underlying latent space[^1]. The generative model $G_{\omega}$, alongside the value function $Q_{\theta}$, can be used as a policy by sampling $n$ actions from $G_{\omega}$ and selecting the highest valued action according to the value estimate $Q_{\theta}$. To increase the diversity of seen actions, we introduce a perturbation model $\xi_{\phi}(s,a,\Phi)$, which outputs an adjustment to an action $a$ in the range $[-\Phi,\Phi]$. This enables access to actions in a constrained region, without having to sample from the generative model a prohibitive number of times. This results in the policy $\pi$:

生成模型我们使用条件变分自编码器（VAE）（Kingma & Welling 2013；Sohn et al. 2015），通过变换潜在空间来建模分布[^1]。生成模型 $G_{\omega}$ 与价值函数 $Q_{\theta}$ 可组成策略：从 $G_{\omega}$ 采样 $n$ 个动作，再按价值估计 $Q_{\theta}$ 选最高价值者。为增加所见动作的多样性，我们引入扰动模型 $\xi_{\phi}(s,a,\Phi)$，在 $[-\Phi,\Phi]$ 范围内对动作 $a$ 输出调整。这样可在受约束区域内访问动作，而不必从生成模型采样过多次数。所得策略 $\pi$ 为：

$$\begin{split}\pi(s)=&\argmax_{a_{i}+\xi_{\phi}(s,a_{i},\Phi)}Q_{\theta}(s,a_{i}+\xi_{\phi}(s,a_{i},\Phi)),\
&{a_{i}\sim G_{\omega}(s)}_{i=1}^{n}.\end{split} \tag{11}$$

The choice of $n$ and $\Phi$ creates a trade-off between an imitation learning and reinforcement learning algorithm. If $\Phi=0$, and the number of sampled actions $n=1$, then the policy resembles behavioral cloning and as $\Phi\rightarrow a_{\text{max}}-a_{\text{min}}$ and $n\rightarrow\infty$, then the algorithm approaches Q-learning, as the policy begins to greedily maximize the value function over the entire action space.

$n$ 与 $\Phi$ 的选择在模仿学习与强化学习算法之间形成权衡。若 $\Phi=0$ 且采样动作数 $n=1$，策略类似于行为克隆；当 $\Phi\rightarrow a_{\text{max}}-a_{\text{min}}$ 且 $n\rightarrow\infty$ 时，算法逼近 Q-learning，因为策略开始在整个动作空间贪婪最大化价值函数。

Input: Batch $\mathcal{B}$, horizon $T$, target network update rate $\tau$, mini-batch size $N$, max perturbation $\Phi$, number of sampled actions $n$, minimum weighting $\lambda$.
Initialize Q-networks $Q_{\theta_{1}},Q_{\theta_{2}}$, perturbation network $\xi_{\phi}$, and VAE $G_{\omega}=\{E_{\omega_{1}},D_{\omega_{2}}\}$, with random parameters $\theta_{1}$, $\theta_{2}$, $\phi$, $\omega$, and target networks $Q_{\theta^{\prime}_{1}},Q_{\theta^{\prime}_{2}}$, $\xi_{\phi^{\prime}}$ with $\theta^{\prime}_{1}\leftarrow\theta_{1},\theta^{\prime}_{2}\leftarrow\theta_{2}$, $\phi^{\prime}\leftarrow\phi$.
for $t=1$ to $T$ do
Sample mini-batch of $N$ transitions $(s,a,r,s^{\prime})$ from $\mathcal{B}$
$\mu,\sigma=E_{\omega_{1}}(s,a),\quad\tilde{a}=D_{\omega_{2}}(s,z),\quad z\sim\mathcal{N}(\mu,\sigma)$
$\omega\leftarrow\argmin_{\omega}\sum(a-\tilde{a})^{2}+D_{\text{KL}}(\mathcal{N}(\mu,\sigma)||\mathcal{N}(0,1))$
Sample $n$ actions: $\{a_{i}\sim G_{\omega}(s^{\prime})\}_{i=1}^{n}$
Perturb each action: $\{a_{i}=a_{i}+\xi_{\phi}(s^{\prime},a_{i},\Phi)\}_{i=1}^{n}$
Set value target $y$ (Eqn. 13)
$\theta\leftarrow\argmin_{\theta}\sum(y-Q_{\theta}(s,a))^{2}$
$\phi\leftarrow\argmax_{\phi}\sum Q_{\theta_{1}}(s,a+\xi_{\phi}(s,a,\Phi)),a\sim G_{\omega}(s)$
Update target networks: $\theta^{\prime}_{i}\leftarrow\tau\theta+(1-\tau)\theta^{\prime}_{i}$
$\phi^{\prime}\leftarrow\tau\phi+(1-\tau)\phi^{\prime}$
end for

Algorithm 1 BCQ

输入： 批次 $\mathcal{B}$，时域 $T$，目标网络更新率 $\tau$，小批次大小 $N$，最大扰动 $\Phi$，采样动作数 $n$，最小权重 $\lambda$。
初始化 Q 网络 $Q_{\theta_{1}},Q_{\theta_{2}}$、扰动网络 $\xi_{\phi}$ 与 VAE $G_{\omega}=\{E_{\omega_{1}},D_{\omega_{2}}\}$，参数 $\theta_{1}$、$\theta_{2}$、$\phi$、$\omega$ 随机，目标网络 $Q_{\theta^{\prime}_{1}},Q_{\theta^{\prime}_{2}}$、$\xi_{\phi^{\prime}}$ 满足 $\theta^{\prime}_{1}\leftarrow\theta_{1},\theta^{\prime}_{2}\leftarrow\theta_{2}$，$\phi^{\prime}\leftarrow\phi$。
for $t=1$ to $T$ do
从 $\mathcal{B}$ 采样 $N$ 条转移 $(s,a,r,s^{\prime})$
$\mu,\sigma=E_{\omega_{1}}(s,a),\quad\tilde{a}=D_{\omega_{2}}(s,z),\quad z\sim\mathcal{N}(\mu,\sigma)$
$\omega\leftarrow\argmin_{\omega}\sum(a-\tilde{a})^{2}+D_{\text{KL}}(\mathcal{N}(\mu,\sigma)||\mathcal{N}(0,1))$
采样 $n$ 个动作：$\{a_{i}\sim G_{\omega}(s^{\prime})\}_{i=1}^{n}$
扰动每个动作：$\{a_{i}=a_{i}+\xi_{\phi}(s^{\prime},a_{i},\Phi)\}_{i=1}^{n}$
设价值目标 $y$（公式 13）
$\theta\leftarrow\argmin_{\theta}\sum(y-Q_{\theta}(s,a))^{2}$
$\phi\leftarrow\argmax_{\phi}\sum Q_{\theta_{1}}(s,a+\xi_{\phi}(s,a,\Phi)),a\sim G_{\omega}(s)$
更新目标网络：$\theta^{\prime}_{i}\leftarrow\tau\theta+(1-\tau)\theta^{\prime}_{i}$
$\phi^{\prime}\leftarrow\tau\phi+(1-\tau)\phi^{\prime}$
end for

算法 1 BCQ

The perturbation model $\xi_{\phi}$ can be trained to maximize $Q_{\theta}(s,a)$ through the deterministic policy gradient algorithm (Silver et al. 2014) by sampling $a\sim G_{\omega}(s)$:

扰动模型 $\xi_{\phi}$ 可通过确定性策略梯度算法（Silver et al. 2014），在采样 $a\sim G_{\omega}(s)$ 时训练以最大化 $Q_{\theta}(s,a)$：

$$\phi\leftarrow\argmax_{\phi}\sum_{(s,a)\in\mathcal{B}}Q_{\theta}(s,a+\xi_{\phi}(s,a,\Phi)). \tag{12}$$

To penalize uncertainty over future states, we modify Clipped Double Q-learning (Fujimoto et al. 2018), which estimates the value by taking the minimum between two Q-networks $\{Q_{\theta_{1}},Q_{\theta_{2}}\}$. Although originally used as a countermeasure to overestimation bias (Thrun & Schwartz 1993; Van Hasselt 2010), the minimum operator also penalizes high variance estimates in regions of uncertainty, and pushes the policy to favor actions which lead to states contained in the batch. In particular, we take a convex combination of the two values, with a higher weight on the minimum, to form a learning target which is used by both Q-networks:

为惩罚对未来状态的不确定性，我们修改 Clipped Double Q-learning（Fujimoto et al. 2018）：取两个 Q 网络 $\{Q_{\theta_{1}},Q_{\theta_{2}}\}$ 的最小值来估计价值。该最小值算子最初用于对抗高估偏差（Thrun & Schwartz 1993；Van Hasselt 2010），同时也惩罚不确定区域的高方差估计，并推动策略偏好导向批次中状态的动作。具体地，我们对两个价值做凸组合，并对最小值赋更高权重，形成两个 Q 网络共用的学习目标：

$$r+\gamma\max_{a_{i}}\left[\lambda\min_{j=1,2}Q_{\theta^{\prime}_{j}}(s^{\prime},a_{i})+(1-\lambda)\max_{j=1,2}Q_{\theta^{\prime}_{j}}(s^{\prime},a_{i})\right] \tag{13}$$

where $a_{i}$ corresponds to the perturbed actions, sampled from the generative model. If we set $\lambda=1$, this update corresponds to Clipped Double Q-learning. We use this weighted minimum as the constrained updates produces less overestimation bias than a purely greedy policy update, and enables control over how heavily uncertainty at future time steps is penalized through the choice of $\lambda$.

其中 $a_{i}$ 对应从生成模型采样并经扰动的动作。若取 $\lambda=1$，该更新即 Clipped Double Q-learning。我们使用这种加权最小值，是因为受约束更新比纯贪婪策略更新产生的高估偏差更小，并可通过 $\lambda$ 控制对未来时间步不确定性的惩罚强度。

This forms Batch-Constrained deep Q-learning (BCQ), which maintains four parametrized networks: a generative model $G_{\omega}(s)$, a perturbation model $\xi_{\phi}(s,a)$, and two Q-networks $Q_{\theta_{1}}(s,a),Q_{\theta_{2}}(s,a)$. We summarize BCQ in Algorithm 1. In the following section, we demonstrate BCQ results in stable value learning and a strong performance in the batch setting. Furthermore, we find that only a single choice of hyper-parameters is necessary for a wide range of tasks and environments.

由此构成批次约束深度 Q 学习（BCQ），维护四个参数化网络：生成模型 $G_{\omega}(s)$、扰动模型 $\xi_{\phi}(s,a)$，以及两个 Q 网络 $Q_{\theta_{1}}(s,a),Q_{\theta_{2}}(s,a)$。算法 1 总结 BCQ。下一节表明 BCQ 带来稳定的价值学习与批次设定下的强表现。此外，我们发现对广泛任务与环境只需一组超参选择。

5 Experiments

5 实验

(a) Final buffer performance

To evaluate the effectiveness of Batch-Constrained deep Q-learning (BCQ) in a high-dimensional setting, we focus on MuJoCo environments in OpenAI gym (Todorov et al. 2012; Brockman et al. 2016). For reproducibility, we make no modifications to the original environments or reward functions. We compare our method with DDPG (Lillicrap et al. 2015), DQN (Mnih et al. 2015) using an independently discretized action space, a feed-forward behavioral cloning method (BC), and a variant with a VAE (VAE-BC), using $G_{\omega}(s)$ from BCQ. Exact implementation and experimental details are provided in the Supplementary Material.

为在高维设定下评估批次约束深度 Q 学习（BCQ）的有效性，我们聚焦 OpenAI gym 中的 MuJoCo 环境（Todorov et al. 2012；Brockman et al. 2016）。为可复现，不对原始环境或奖励函数做任何修改。对比方法包括 DDPG（Lillicrap et al. 2015）、在独立离散化动作空间上的 DQN（Mnih et al. 2015）、前馈行为克隆方法（BC），以及使用 BCQ 中 $G_{\omega}(s)$ 的 VAE 变体（VAE-BC）。精确实现与实验细节见补充材料。

(a) Final Buffer

We evaluate each method following the three experiments defined in Section 3.1. In final buffer the off-policy agents learn from the final replay buffer gathered by training a DDPG agent over a million time steps. In concurrent the off-policy agents learn concurrently, with the same replay buffer, as the behavioral DDPG policy, and in imitation, the agents learn from a dataset collected by an expert policy. Additionally, to study the robustness of BCQ to noisy and multi-modal data, we include an imperfect demonstrations task, in which the agents are trained with a batch of 100k transitions collected by an expert policy, with two sources of noise. The behavioral policy selects actions randomly with probability $0.3$ and with high exploratory noise $\mathcal{N}(0,0.3)$ added to the remaining actions. The experimental results for these tasks are reported in Figure 2. Furthermore, the estimated values of BCQ, DDPG and DQN, and the true value of BCQ are displayed in Figure 3.

我们按第 3.1 节定义的三项实验评估各方法。在 final buffer 中，离策略智能体从训练 DDPG 一百万步后得到的最终回放缓冲学习。在 concurrent 中，离策略智能体与行为 DDPG 策略共享同一回放缓冲并发学习。在 imitation 中，智能体从专家策略采集的数据集学习。此外，为研究 BCQ 对噪声与多模态数据的稳健性，我们加入 imperfect demonstrations 任务：智能体用专家策略采集的 10 万次转移训练，并含两类噪声——行为策略以概率 $0.3$ 随机选动作，其余动作加高探索噪声 $\mathcal{N}(0,0.3)$。这些任务的实验结果见图 2。此外，BCQ、DDPG 与 DQN 的估计价值以及 BCQ 的真实价值见图 3。

Our approach, BCQ, is the only algorithm which succeeds at all tasks, matching or outperforming the behavioral policy in each instance, and outperforming all other agents, besides in the imitation learning task where behavioral cloning unsurprisingly performs the best. These results demonstrate that our algorithm can be used as a single approach for both imitation learning and off-policy reinforcement learning, with a single set of fixed hyper-parameters. Furthermore, unlike the deep reinforcement learning algorithms, DDPG and DQN, BCQ exhibits a highly stable value function in the presence of off-policy samples, suggesting extrapolation error has been successfully mitigated through the batch-constraint. In the imperfect demonstrations task, we find that both deep reinforcement learning and imitation learning algorithms perform poorly. BCQ, however, is able to strongly outperform the noisy demonstrator, disentangling poor and expert actions. Furthermore, compared to current deep reinforcement learning algorithms, which can require millions of time steps (Duan et al. 2016; Henderson et al. 2017), BCQ attains a high performance in remarkably few iterations. This suggests our approach effectively leverages expert transitions, even in the presence of noise.

我们的方法 BCQ 是唯一在全部任务上成功的算法：每次都匹配或超过行为策略，并超过所有其他智能体——除模仿学习任务上行为克隆理所当然最好之外。这些结果表明，我们的算法可用固定的一组超参，同时作为模仿学习与离策略强化学习的统一方法。此外，与深度强化学习算法 DDPG、DQN 不同，BCQ 在离策略样本下展现高度稳定的价值函数，说明外推误差已通过批次约束得到缓解。在不完美示范任务中，深度强化学习与模仿学习算法都表现不佳；BCQ 却能大幅超过嘈杂示范者，拆开差动作与专家动作。再者，相对常需数百万步的当前深度强化学习算法（Duan et al. 2016；Henderson et al. 2017），BCQ 在极少迭代内就达到高表现，说明即便存在噪声，我们的方法也能有效利用专家转移。

6 Related Work

6 相关工作

Batch Reinforcement Learning. While batch reinforcement learning algorithms have been shown to be convergent with non-parametric function approximators such as averagers (Gordon 1995) and kernel methods (Ormoneit & Sen 2002), they make no guarantees on the quality of the policy without infinite data. Other batch algorithms, such as fitted Q-iteration, have used other function approximators, including decision trees (Ernst et al. 2005) and neural networks (Riedmiller 2005), but come without convergence guarantees. Unlike many previous approaches to off-policy policy evaluation (Peshkin & Shelton 2002; Thomas et al. 2015; Liu et al. 2018), our work focuses on constraining the policy to a subset of policies which can be adequately evaluated, rather than the process of evaluation itself. Additionally, off-policy algorithms which rely on importance sampling (Precup et al. 2001; Jiang & Li 2016; Munos et al. 2016) may not be applicable in a batch setting, requiring access to the action probabilities under the behavioral policy, and scale poorly to multi-dimensional action spaces. Reinforcement learning with a replay buffer (Lin 1992) can be considered a form of batch reinforcement learning, and is a standard tool for off-policy deep reinforcement learning algorithms (Mnih et al. 2015). It has been observed that a large replay buffer can be detrimental to performance (de Bruin et al. 2015; Zhang & Sutton 2017) and the diversity of states in the buffer is an important factor for performance (de Bruin et al. 2016). Isele & Cosgun 2018 observed the performance of an agent was strongest when the distribution of data in the replay buffer matched the test distribution. These results defend the notion that extrapolation error is an important factor in the performance off-policy reinforcement learning.

批次强化学习。 批次强化学习算法已证明可与平均器（Gordon 1995）、核方法（Ormoneit & Sen 2002）等非参数函数逼近器收敛，但在无无穷数据时不对策略质量做保证。其他批次算法如 fitted Q-iteration 使用决策树（Ernst et al. 2005）与神经网络（Riedmiller 2005）等逼近器，但无收敛保证。与许多先前离策略策略评估工作（Peshkin & Shelton 2002；Thomas et al. 2015；Liu et al. 2018）不同，我们的工作聚焦于把策略约束到可被充分评估的策略子集，而非评估过程本身。此外，依赖重要性采样的离策略算法（Precup et al. 2001；Jiang & Li 2016；Munos et al. 2016）在批次设定下可能不适用：需要行为策略下的动作概率，且难以扩展到多维动作空间。带回放缓冲的强化学习（Lin 1992）可视为一种批次强化学习，也是离策略深度强化学习算法的标准工具（Mnih et al. 2015）。已有观察表明过大回放缓冲可能损害表现（de Bruin et al. 2015；Zhang & Sutton 2017），且缓冲中状态多样性是重要因素（de Bruin et al. 2016）。Isele & Cosgun 2018 观察到：当回放缓冲中数据分布匹配测试分布时，智能体表现最强。这些结果支持外推误差是离策略强化学习表现重要因素的观点。

Imitation Learning. Imitation learning and its variants are well studied problems (Schaal 1999; Argall et al. 2009; Hussein et al. 2017). Imitation has been combined with reinforcement learning, via learning from demonstrations methods (Kim et al. 2013; Piot et al. 2014; Chemali & Lazaric 2015), with deep reinforcement learning extensions (Hester et al. 2017; Večerík et al. 2017), and modified policy gradient approaches (Ho et al. 2016; Sun et al. 2017; Cheng et al. 2018; Sun et al. 2018). While effective, these interactive methods are inadequate for batch reinforcement learning as they require either an explicit distinction between expert and non-expert data, further on-policy data collection or access to an oracle. Research in imitation, and inverse reinforcement learning, with robustness to noise is an emerging area (Evans 2016; Nair et al. 2018), but relies on some form of expert data. Gao et al. 2018 introduced an imitation learning algorithm which learned from imperfect demonstrations, by favoring seen actions, but is limited to discrete actions. Our work also connects to residual policy learning (Johannink et al. 2018; Silver et al. 2018), where the initial policy is the generative model, rather than an expert or feedback controller.

模仿学习。 模仿学习及其变体是充分研究的问题（Schaal 1999；Argall et al. 2009；Hussein et al. 2017）。模仿已与强化学习结合：通过从示范学习的方法（Kim et al. 2013；Piot et al. 2014；Chemali & Lazaric 2015）、深度强化学习扩展（Hester et al. 2017；Večerík et al. 2017），以及修改的策略梯度方法（Ho et al. 2016；Sun et al. 2017；Cheng et al. 2018；Sun et al. 2018）。这些交互式方法虽有效，但对批次强化学习不够：要么需要明确区分专家与非专家数据，要么需要继续同策略采集，要么需要访问 oracle。对噪声稳健的模仿与逆强化学习是新兴方向（Evans 2016；Nair et al. 2018），但仍依赖某种形式的专家数据。Gao et al. 2018 提出通过偏好已见动作从不完美示范学习的模仿算法，但限于离散动作。我们的工作也与残差策略学习（Johannink et al. 2018；Silver et al. 2018）相关，其中初始策略是生成模型，而非专家或反馈控制器。

Uncertainty in Reinforcement Learning. Uncertainty estimates in deep reinforcement learning have generally been used to encourage exploration (Dearden et al. 1998; Strehl & Littman 2008; O’Donoghue et al. 2018; Azizzadenesheli et al. 2018). Other methods have examined approximating the Bayesian posterior of the value function (Osband et al. 2016; Osband et al. 2018; Touati et al. 2018), again using the variance to encourage exploration to unseen regions of the state space. In model-based reinforcement learning, uncertainty has been used for exploration, but also for the opposite effect–to push the policy towards regions of certainty in the model. This is used to combat the well-known problems with compounding model errors, and is present in policy search methods (Deisenroth & Rasmussen 2011; Gal et al. 2016; Higuera et al. 2018; Xu et al. 2018), or combined with trajectory optimization (Chua et al. 2018) or value-based methods (Buckman et al. 2018). Our work connects to policy methods with conservative updates (Kakade & Langford 2002), such as trust region (Schulman et al. 2015; Achiam et al. 2017; Pham et al. 2018) and information-theoretic methods (Peters & Mülling 2010; Van Hoof et al. 2017), which aim to keep the updated policy similar to the previous policy. These methods avoid explicit uncertainty estimates, and rather force policy updates into a constrained range before collecting new data, limiting errors introduced by large changes in the policy. Similarly, our approach can be thought of as an off-policy variant, where the policy aims to be kept close, in output space, to any combination of the previous policies which performed data collection.

强化学习中的不确定性。 深度强化学习中的不确定性估计通常用于鼓励探索（Dearden et al. 1998；Strehl & Littman 2008；O'Donoghue et al. 2018；Azizzadenesheli et al. 2018）。其他方法近似价值函数的贝叶斯后验（Osband et al. 2016；Osband et al. 2018；Touati et al. 2018），同样用方差鼓励探索未见状态区域。在基于模型的强化学习中，不确定性既用于探索，也用于相反效果——把策略推向模型中确定的区域。这用于对抗复合模型误差的已知问题，出现在策略搜索方法（Deisenroth & Rasmussen 2011；Gal et al. 2016；Higuera et al. 2018；Xu et al. 2018）、与轨迹优化结合（Chua et al. 2018）或与基于价值的方法结合（Buckman et al. 2018）中。我们的工作与带保守更新的策略方法（Kakade & Langford 2002）相关，如信赖域（Schulman et al. 2015；Achiam et al. 2017；Pham et al. 2018）与信息论方法（Peters & Mülling 2010；Van Hoof et al. 2017），目标是让更新后的策略与先前策略保持相似。这些方法避免显式不确定性估计，而是在采集新数据前把策略更新限制在约束范围内，以限制策略大幅变化引入的误差。类似地，我们的方法可视为一种离策略变体：策略旨在在输出空间上贴近执行过数据采集的先前策略的任意组合。

7 Conclusion

7 结论

In this work, we demonstrate a critical problem in off-policy reinforcement learning with finite data, where the value target introduces error by including an estimate of unseen state-action pairs. This phenomenon, which we denote extrapolation**error, has important implications for off-policy and batch reinforcement learning, as it is generally implausible to have complete state-action coverage in any practical setting. We present batch-constrained reinforcement learning–acting close to on-policy with respect to the available data, as an answer to extrapolation error. When extended to a deep reinforcement learning setting, our algorithm, Batch-Constrained deep Q-learning (BCQ), is the first continuous control algorithm capable of learning from arbitrary batch data, without exploration. Due to the importance of batch reinforcement learning for practical applications, we believe BCQ will be a strong foothold for future algorithms to build on, while furthering our understanding of the systematic risks in Q-learning (Thrun & Schwartz 1993; Lu et al. 2018).

本文揭示有限数据下离策略强化学习的一个关键问题：价值目标因包含未见状态–动作对的估计而引入误差。我们称该现象为外推误差；它对离策略与批次强化学习有重要含义，因为在任何实际设定中通常都不太可能有完整的状态–动作覆盖。我们提出批次约束强化学习——相对可用数据表现得接近同策略——作为应对外推误差的答案。扩展到深度强化学习设定后，我们的算法批次约束深度 Q 学习（BCQ）是首个能从任意批次数据学习、无需探索的连续控制算法。鉴于批次强化学习对实际应用的重要性，我们相信 BCQ 将成为未来算法的坚实立足点，并加深对 Q-learning 系统性风险的理解（Thrun & Schwartz 1993；Lu et al. 2018）。

References

参考文献

Achiam et al. (2017) Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International Conference on Machine Learning , pp. 22–31, 2017.

Argall et al. (2009) Argall, B. D., Chernova, S., Veloso, M., and Browning, B. A survey of robot learning from demonstration. Robotics and Autonomous Systems , 57(5):469–483, 2009.

Azizzadenesheli et al. (2018) Azizzadenesheli, K., Brunskill, E., and Anandkumar, A. Efficient exploration through bayesian deep q-networks. arXiv preprint arXiv:1802.04412 , 2018.

Bertsekas & Tsitsiklis (1996) Bertsekas, D. P. and Tsitsiklis, J. N. Neuro-Dynamic Programming . Athena scientific Belmont, MA, 1996.

Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016.

Buckman et al. (2018) Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. Sample-efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems , pp. 8234–8244, 2018.

Chemali & Lazaric (2015) Chemali, J. and Lazaric, A. Direct policy iteration with demonstrations. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence , 2015.

Cheng et al. (2018) Cheng, C.-A., Yan, X., Wagener, N., and Boots, B. Fast policy learning through imitation and reinforcement. arXiv preprint arXiv:1805.10413 , 2018.

Chua et al. (2018) Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31 , pp. 4759–4770, 2018.

Dayan & Watkins (1992) Dayan, P. and Watkins, C. J. C. H. Q-learning. Machine learning , 8(3):279–292, 1992.

de Bruin et al. (2015) de Bruin, T., Kober, J., Tuyls, K., and Babuška, R. The importance of experience replay database composition in deep reinforcement learning. In Deep Reinforcement Learning Workshop, NIPS , 2015.

de Bruin et al. (2016) de Bruin, T., Kober, J., Tuyls, K., and Babuška, R. Improved deep reinforcement learning for robotics through distribution-based experience retention. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 3947–3952. IEEE, 2016.

Dearden et al. (1998) Dearden, R., Friedman, N., and Russell, S. Bayesian q-learning. In AAAI/IAAI , pp. 761–768, 1998.

Deisenroth & Rasmussen (2011) Deisenroth, M. and Rasmussen, C. E. Pilco: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning , pp. 465–472, 2011.

Duan et al. (2016) Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning , pp. 1329–1338, 2016.

Ernst et al. (2005) Ernst, D., Geurts, P., and Wehenkel, L. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research , 6(Apr):503–556, 2005.

Evans (2016) Evans, O. Learning the preferences of ignorant, inconsistent agents. In AAAI , pp. 323–329, 2016.

Fujimoto et al. (2018) Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning , volume 80, pp. 1587–1596. PMLR, 2018.

Gal et al. (2016) Gal, Y., McAllister, R., and Rasmussen, C. E. Improving pilco with bayesian neural network dynamics models. In Data-Efficient Machine Learning workshop, International Conference on Machine Learning , 2016.

Gao et al. (2018) Gao, Y., Lin, J., Yu, F., Levine, S., and Darrell, T. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.05313 , 2018.

Goodfellow et al. (2013) Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 , 2013.

Gordon (1995) Gordon, G. J. Stable function approximation in dynamic programming. In Machine Learning Proceedings 1995 , pp. 261–268. Elsevier, 1995.

Henderson et al. (2017) Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep Reinforcement Learning that Matters. arXiv preprint arXiv:1709.06560 , 2017.

Hester et al. (2017) Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Dulac-Arnold, G., et al. Deep q-learning from demonstrations. arXiv preprint arXiv:1704.03732 , 2017.

Higuera et al. (2018) Higuera, J. C. G., Meger, D., and Dudek, G. Synthesizing neural network controllers with probabilistic model based reinforcement learning. arXiv preprint arXiv:1803.02291 , 2018.

Ho et al. (2016) Ho, J., Gupta, J., and Ermon, S. Model-free imitation learning with policy optimization. In International Conference on Machine Learning , pp. 2760–2769, 2016.

Hussein et al. (2017) Hussein, A., Gaber, M. M., Elyan, E., and Jayne, C. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR) , 50(2):21, 2017.

Isele & Cosgun (2018) Isele, D. and Cosgun, A. Selective experience replay for lifelong learning. arXiv preprint arXiv:1802.10269 , 2018.

Jaksch et al. (2010) Jaksch, T., Ortner, R., and Auer, P. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research , 11(Apr):1563–1600, 2010.

Jiang & Li (2016) Jiang, N. and Li, L. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning , pp. 652–661, 2016.

Johannink et al. (2018) Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J. A., Solowjow, E., and Levine, S. Residual reinforcement learning for robot control. arXiv preprint arXiv:1812.03201 , 2018.

Kakade & Langford (2002) Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning , volume 2, pp. 267–274, 2002.

Kim et al. (2013) Kim, B., Farahmand, A.-m., Pineau, J., and Precup, D. Learning from limited demonstrations. In Advances in Neural Information Processing Systems , pp. 2859–2867, 2013.

Kingma & Ba (2014) Kingma, D. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Kingma & Welling (2013) Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Konda & Tsitsiklis (2003) Konda, V. R. and Tsitsiklis, J. N. On actor-critic algorithms. SIAM journal on Control and Optimization , 42(4):1143–1166, 2003.

Lai & Robbins (1985) Lai, T. L. and Robbins, H. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics , 6(1):4–22, 1985.

Lange et al. (2012) Lange, S., Gabel, T., and Riedmiller, M. Batch reinforcement learning. In Reinforcement learning , pp. 45–73. Springer, 2012.

Lillicrap et al. (2015) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Lin (1992) Lin, L.-J. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning , 8(3-4):293–321, 1992.

Liu et al. (2018) Liu, Y., Gottesman, O., Raghu, A., Komorowski, M., Faisal, A. A., Doshi-Velez, F., and Brunskill, E. Representation balancing mdps for off-policy policy evaluation. In Advances in Neural Information Processing Systems , pp. 2644–2653, 2018.

Lu et al. (2018) Lu, T., Schuurmans, D., and Boutilier, C. Non-delusional q-learning and value-iteration. In Advances in Neural Information Processing Systems , pp. 9971–9981, 2018.

McCloskey & Cohen (1989) McCloskey, M. and Cohen, N. J. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation , volume 24, pp. 109–165. Elsevier, 1989.

Melo (2001) Melo, F. S. Convergence of q-learning: A simple proof. Institute Of Systems and Robotics, Tech. Rep , pp. 1–4, 2001.

Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529–533, 2015.

Munos et al. (2016) Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems , pp. 1054–1062, 2016.

Nair et al. (2018) Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pp. 6292–6299. IEEE, 2018.

O’Donoghue et al. (2018) O’Donoghue, B., Osband, I., Munos, R., and Mnih, V. The uncertainty Bellman equation and exploration. In International Conference on Machine Learning , volume 80, pp. 3839–3848. PMLR, 2018.

Ormoneit & Sen (2002) Ormoneit, D. and Sen, Ś. Kernel-based reinforcement learning. Machine learning , 49(2-3):161–178, 2002.

Osband et al. (2016) Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems , pp. 4026–4034, 2016.

Osband et al. (2018) Osband, I., Aslanides, J., and Cassirer, A. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems 31 , pp. 8626–8638, 2018.

Peshkin & Shelton (2002) Peshkin, L. and Shelton, C. R. Learning from scarce experience. In International Conference on Machine Learning , pp. 498–505, 2002.

Peters & Mülling (2010) Peters, J. and Mülling, K. Relative entropy policy search. In AAAI , pp. 1607–1612, 2010.

Pham et al. (2018) Pham, T.-H., De Magistris, G., Agravante, D. J., Chaudhury, S., Munawar, A., and Tachibana, R. Constrained exploration and recovery from experience shaping. arXiv preprint arXiv:1809.08925 , 2018.

Piot et al. (2014) Piot, B., Geist, M., and Pietquin, O. Boosted bellman residual minimization handling expert demonstrations. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pp. 549–564. Springer, 2014.

Precup et al. (2001) Precup, D., Sutton, R. S., and Dasgupta, S. Off-policy temporal-difference learning with function approximation. In International Conference on Machine Learning , pp. 417–424, 2001.

Rezende et al. (2014) Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082 , 2014.

Riedmiller (2005) Riedmiller, M. Neural fitted q iteration–first experiences with a data efficient neural reinforcement learning method. In European Conference on Machine Learning , pp. 317–328. Springer, 2005.

Schaal (1999) Schaal, S. Is imitation learning the route to humanoid robots? Trends in Cognitive Sciences , 3(6):233–242, 1999.

Schulman et al. (2015) Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning , pp. 1889–1897, 2015.

Silver et al. (2014) Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International Conference on Machine Learning , pp. 387–395, 2014.

Silver et al. (2018) Silver, T., Allen, K., Tenenbaum, J., and Kaelbling, L. Residual policy learning. arXiv preprint arXiv:1812.06298 , 2018.

Singh et al. (2000) Singh, S., Jaakkola, T., Littman, M. L., and Szepesvári, C. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine learning , 38(3):287–308, 2000.

Sohn et al. (2015) Sohn, K., Lee, H., and Yan, X. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems , pp. 3483–3491, 2015.

Strehl & Littman (2008) Strehl, A. L. and Littman, M. L. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences , 74(8):1309–1331, 2008.

Sun et al. (2017) Sun, W., Venkatraman, A., Gordon, G. J., Boots, B., and Bagnell, J. A. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International Conference on Machine Learning , pp. 3309–3318, 2017.

Sun et al. (2018) Sun, W., Bagnell, J. A., and Boots, B. Truncated horizon policy search: Combining reinforcement learning & imitation learning. arXiv preprint arXiv:1805.11240 , 2018.

Sutton (1988) Sutton, R. S. Learning to predict by the methods of temporal differences. Machine learning , 3(1):9–44, 1988.

Sutton & Barto (1998) Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998.

Thomas et al. (2015) Thomas, P., Theocharous, G., and Ghavamzadeh, M. High confidence policy improvement. In International Conference on Machine Learning , pp. 2380–2388, 2015.

Thrun & Schwartz (1993) Thrun, S. and Schwartz, A. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum , 1993.

Todorov et al. (2012) Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 5026–5033. IEEE, 2012.

Touati et al. (2018) Touati, A., Satija, H., Romoff, J., Pineau, J., and Vincent, P. Randomized value functions via multiplicative normalizing flows. arXiv preprint arXiv:1806.02315 , 2018.

Van Hasselt (2010) Van Hasselt, H. Double q-learning. In Advances in Neural Information Processing Systems , pp. 2613–2621, 2010.

Van Hasselt et al. (2016) Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In AAAI , pp. 2094–2100, 2016.

Van Hoof et al. (2017) Van Hoof, H., Neumann, G., and Peters, J. Non-parametric policy search with limited information loss. The Journal of Machine Learning Research , 18(1):2472–2517, 2017.

Večerík et al. (2017) Večerík, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Rothörl, T., Lampe, T., and Riedmiller, M. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 , 2017.

Watkins (1989) Watkins, C. J. C. H. Learning from delayed rewards . PhD thesis, King’s College, Cambridge, 1989.

Xu et al. (2018) Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T. Algorithmic framework for model-based reinforcement learning with theoretical guarantees. arXiv preprint arXiv:1807.03858 , 2018.

Zhang & Sutton (2017) Zhang, S. and Sutton, R. S. A deeper look at experience replay. arXiv preprint arXiv:1712.01275 , 2017.

Appendix A Missing Proofs

附录 A 缺失证明

A.1 Proofs and Details from Section 4.1

A.1 第 4.1 节的证明与细节

Definition 1. We define a coherent batch $\mathcal{B}$ as a batch such that if $(s,a,s^{\prime})\in\mathcal{B}$ then $s^{\prime}\in\mathcal{B}$ unless $s^{\prime}$ is a terminal state.

定义 1。 若 $(s,a,s^{\prime})\in\mathcal{B}$ 则除非 $s^{\prime}$ 为终止状态否则 $s^{\prime}\in\mathcal{B}$，称这样的批次 $\mathcal{B}$ 为连贯批次。

Definition 2. We define $\epsilon_{\text{MDP}}(s,a)=Q^{\pi}(s,a)-Q^{\pi}_{\mathcal{B}}(s,a)$ as the error between the true value of a policy $\pi$ in the MDP $M$ and the value of $\pi$ when learned with a batch $\mathcal{B}$.

定义 2。 定义 $\epsilon_{\text{MDP}}(s,a)=Q^{\pi}(s,a)-Q^{\pi}_{\mathcal{B}}(s,a)$ 为真实 MDP $M$ 中策略 $\pi$ 的价值与用批次 $\mathcal{B}$ 学到的 $\pi$ 价值之间的误差。

Definition 3. For simplicity in notation, we denote

定义 3。 为记号简便，记

$$\epsilon^{\pi}_{\text{MDP}}=\sum_{s}\mu_{\pi}(s)\sum_{a}\pi(a|s)|\epsilon_{\text{MDP}}(s,a)|. \tag{14}$$

To evaluate a policy $\pi$ exactly at relevant state-action pairs, only $\epsilon^{\pi}_{\text{MDP}}=0$ is required.

要在相关状态–动作对上精确评估策略 $\pi$，只需 $\epsilon^{\pi}_{\text{MDP}}=0$。

Definition 4. We define the optimal batch-constrained policy $\pi^{*}\in\Pi_{\mathcal{B}}$ such that $Q^{\pi^{*}}(s,a)\geq Q^{\pi}(s,a)$ for all $\pi\in\Pi_{\mathcal{B}}$ and $(s,a)\in\mathcal{B}$.

定义 4。 定义最优批次约束策略 $\pi^{*}\in\Pi_{\mathcal{B}}$，使得对所有 $\pi\in\Pi_{\mathcal{B}}$ 与 $(s,a)\in\mathcal{B}$，有 $Q^{\pi^{*}}(s,a)\geq Q^{\pi}(s,a)$。

Algorithm 1. Batch-Constrained Q-learning (BCQL) maintains a tabular value function $Q(s,a)$ for each possible state-action pair $(s,a)$. A transition tuple $(s,a,r,s^{\prime})$ is sampled from the batch $\mathcal{B}$ with uniform probability and the following update rule is applied, with learning rate $\alpha$:

算法 1。 批次约束 Q-learning（BCQL）为每个可能状态–动作对 $(s,a)$ 维护表格价值函数 $Q(s,a)$。以均匀概率从批次 $\mathcal{B}$ 采样转移元组 $(s,a,r,s^{\prime})$，并以学习率 $\alpha$ 应用如下更新规则：

$$Q(s,a)\leftarrow(1-\alpha)Q(s,a)+\alpha(r+\gamma\max_{a^{\prime}\text{s.t.}(s^{\prime},a^{\prime})\in\mathcal{B}}Q(s^{\prime},a^{\prime})). \tag{15}$$

Theorem 1. Performing Q-learning by sampling from a batch $\mathcal{B}$ converges to the optimal value function under the MDP $M_{\mathcal{B}}$.

定理 1。 通过从批次 $\mathcal{B}$ 采样做 Q-learning，收敛到 MDP $M_{\mathcal{B}}$ 下的最优价值函数。

Proof. Again, the MDP $M_{\mathcal{B}}$ is defined by the same action and state space as $M$, with an additional terminal state $s_{\text{init}}$. $M_{\mathcal{B}}$ has transition probabilities $p_{\mathcal{B}}(s^{\prime}|s,a)=\frac{N(s,a,s^{\prime})}{\sum_{\tilde{s}}N(s,a,\tilde{s})}$, where $N(s,a,s^{\prime})$ is the number of times the tuple $(s,a,s^{\prime})$ is observed in $\mathcal{B}$. If $\sum_{\tilde{s}}N(s,a,\tilde{s})=0$, then $p_{\mathcal{B}}(s_{\text{init}}|s,a)=1$, where $r(s_{\text{init}},s,a)$ is to the initialized value of $Q(s,a)$.

证明。 同样，$M_{\mathcal{B}}$ 由与 $M$ 相同的动作与状态空间定义，并额外增加终止状态 $s_{\text{init}}$。$M_{\mathcal{B}}$ 的转移概率为 $p_{\mathcal{B}}(s^{\prime}|s,a)=\frac{N(s,a,s^{\prime})}{\sum_{\tilde{s}}N(s,a,\tilde{s})}$，其中 $N(s,a,s^{\prime})$ 是元组 $(s,a,s^{\prime})$ 在 $\mathcal{B}$ 中出现次数。若 $\sum_{\tilde{s}}N(s,a,\tilde{s})=0$，则 $p_{\mathcal{B}}(s_{\text{init}}|s,a)=1$，并把 $r(s_{\text{init}},s,a)$ 设为 $Q(s,a)$ 的初始化值。

For any given MDP Q-learning converges to the optimal value function given infinite state-action visitation and some standard assumptions (see Section A.2). Now note that sampling under a batch $\mathcal{B}$ with uniform probability satisfies the infinite state-action visitation assumptions of the MDP $M_{\mathcal{B}}$, where given $(s,a)$, the probability of sampling $(s,a,s^{\prime})$ corresponds to $p(s^{\prime}|s,a)=\frac{N(s,a,s^{\prime})}{\sum_{\tilde{s}}N(s,a,\tilde{s})}$ in the limit. We remark that for $(s,a)\notin\mathcal{B}$, $Q(s,a)$ will never be updated, and will return the initialized value, which corresponds to the terminal transition $s_{\text{init}}$. It follows that sampling from $\mathcal{B}$ is equivalent to sampling from the MDP $M_{\mathcal{B}}$, and Q-learning converges to the optimal value function under $M_{\mathcal{B}}$.

对任意给定 MDP，在无穷状态–动作访问与若干标准假设下，Q-learning 收敛到最优价值函数（见第 A.2 节）。注意：以均匀概率从批次 $\mathcal{B}$ 采样满足 MDP $M_{\mathcal{B}}$ 的无穷状态–动作访问假设；给定 $(s,a)$，采样 $(s,a,s^{\prime})$ 的概率在极限下对应 $p(s^{\prime}|s,a)=\frac{N(s,a,s^{\prime})}{\sum_{\tilde{s}}N(s,a,\tilde{s})}$。我们指出：对 $(s,a)\notin\mathcal{B}$，$Q(s,a)$ 永不更新，返回初始化值，对应终止转移 $s_{\text{init}}$。因此从 $\mathcal{B}$ 采样等价于从 MDP $M_{\mathcal{B}}$ 采样，Q-learning 收敛到 $M_{\mathcal{B}}$ 下的最优价值函数。

Remark 1. For any policy $\pi$ and state-action pair $(s,a)$, the error term $\epsilon_{\text{MDP}}(s,a)$ satisfies the following Bellman-like equation:

注记 1。 对任意策略 $\pi$ 与状态–动作对 $(s,a)$，误差项 $\epsilon_{\text{MDP}}(s,a)$ 满足如下类 Bellman 方程：

$$\begin{split}\epsilon_{\text{MDP}}(s,a)=~&\sum_{s^{\prime}}\left(p_{M}(s^{\prime}|s,a)-p_{\mathcal{B}}(s^{\prime}|s,a)\right)\left(r(s,a,s^{\prime})+\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})\left(Q^{\pi}{\mathcal{B}}(s^{\prime},a^{\prime})\right)\right)\
&+p{M}(s^{\prime}|s,a)\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})\epsilon_{\text{MDP}}(s^{\prime},a^{\prime}).\end{split} \tag{16}$$

Proof. Proof follows by expanding each $Q$, rearranging terms and then simplifying the expression.

证明。 证明通过对每个 $Q$ 展开、重排项并化简表达式得到。

$$\begin{split}\epsilon_{\text{MDP}}(s,a)=&~Q^{\pi}(s,a)-Q^{\pi}{\mathcal{B}}(s,a)\
=&\sum{s^{\prime}}p_{M}(s^{\prime}|s,a)\left(r(s,a,s^{\prime})+\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})Q^{\pi}(s^{\prime},a^{\prime})\right)-Q^{\pi}{\mathcal{B}}(s,a)\
=&\sum{s^{\prime}}p_{M}(s^{\prime}|s,a)\left(r(s,a,s^{\prime})+\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})Q^{\pi}(s^{\prime},a^{\prime})\right)\
&-\left(\sum_{s^{\prime}}p_{\mathcal{B}}(s^{\prime}|s,a)\left(r(s,a,s^{\prime})+\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})Q^{\pi}{\mathcal{B}}(s^{\prime},a^{\prime})\right)\right)\
=&\sum{s^{\prime}}\left(p_{M}(s^{\prime}|s,a)-p_{\mathcal{B}}(s^{\prime}|s,a)\right)r(s,a,s^{\prime})+p_{M}(s^{\prime}|s,a)\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})\left(Q^{\pi}{\mathcal{B}}(s^{\prime},a^{\prime})+\epsilon{\text{MDP}}(s^{\prime},a^{\prime})\right)\
&-p_{\mathcal{B}}(s^{\prime}|s,a)\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})Q^{\pi}{\mathcal{B}}(s^{\prime},a^{\prime})\
=&\sum{s^{\prime}}\left(p_{M}(s^{\prime}|s,a)-p_{\mathcal{B}}(s^{\prime}|s,a)\right)r(s,a,s^{\prime})+p_{M}(s^{\prime}|s,a)\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})\left(Q^{\pi}{\mathcal{B}}(s^{\prime},a^{\prime})+\epsilon{\text{MDP}}(s^{\prime},a^{\prime})\right)\
&+p_{M}(s^{\prime}|s,a)\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})\left(\epsilon_{\text{MDP}}(s^{\prime},a^{\prime})-\epsilon_{\text{MDP}}(s^{\prime},a^{\prime})\right)-p_{\mathcal{B}}(s^{\prime}|s,a)\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})Q^{\pi}{\mathcal{B}}(s^{\prime},a^{\prime})\
=&\sum{s^{\prime}}\left(p_{M}(s^{\prime}|s,a)-p_{\mathcal{B}}(s^{\prime}|s,a)\right)\left(r(s,a,s^{\prime})+\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})Q^{\pi}{\mathcal{B}}(s^{\prime},a^{\prime})\right)\
&+p{M}(s^{\prime}|s,a)\gamma\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})\epsilon_{\text{MDP}}(s^{\prime},a^{\prime})\end{split} \tag{17}$$

Lemma 1. For all reward functions, $\epsilon^{\pi}_{\text{MDP}}=0$ if and only if $p_{\mathcal{B}}(s^{\prime}|s,a)=p_{M}(s^{\prime}|s,a)$ for all $s^{\prime}\in\mathcal{S}$ and $(s,a)$ such that $\mu_{\pi}(s)>0$ and $\pi(a|s)>0$.

引理 1。 对所有奖励函数，$\epsilon^{\pi}_{\text{MDP}}=0$ 当且仅当对所有满足 $\mu_{\pi}(s)>0$ 且 $\pi(a|s)>0$ 的 $(s,a)$ 以及所有 $s^{\prime}\in\mathcal{S}$，有 $p_{\mathcal{B}}(s^{\prime}|s,a)=p_{M}(s^{\prime}|s,a)$。