Mastering Atari with Discrete World Models

用离散世界模型精通 Atari

Danijar Hafner † † thanks: Correspondence to: Danijar Hafner mail@danijar.com. Affiliation: Google Research    Timothy Lillicrap Affiliation: DeepMind    Mohammad Norouzi Affiliation: Google Research    Jimmy Ba Affiliation: University of Toronto

Abstract

摘要

Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of a powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitutes the first agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. With the same computational budget and wall-clock time, Dreamer V2 reaches 200M frames and surpasses the final performance of the top single-GPU agents IQN and Rainbow. DreamerV2 is also applicable to tasks with continuous actions, where it learns an accurate world model of a complex humanoid robot and solves stand-up and walking from only pixel inputs.

智能体需要从过往经验中泛化，才能在复杂环境中达成目标。世界模型有助于这种泛化，并允许从想象出的结果中学习行为，从而提高样本效率。尽管从图像输入学习世界模型在部分任务上已变得可行，但把 Atari 游戏建模得足够准、从而导出成功行为，多年来仍是开放挑战。我们提出 DreamerV2：一种强化学习智能体，它完全在强大世界模型的紧凑潜空间预测上学习行为。该世界模型使用离散表示，并与策略分开训练。DreamerV2 是首个在 55 个 Atari 任务基准上，通过在单独训练的世界模型内部学习行为而达到人类水平表现的智能体。在相同计算预算与墙钟时间下，DreamerV2 达到 2 亿帧，并超过顶级单 GPU 智能体 IQN 与 Rainbow 的最终表现。DreamerV2 也适用于连续动作任务：它能学到复杂人形机器人的准确世界模型，并仅从像素输入解决起立与行走。

1 Introduction

1 引言

Figure 1: Gamer normalized median score on the Atari benchmark of 55 games with sticky actions at 200M steps. DreamerV2 is the first agent that learns purely within a world model to achieve human-level Atari performance, demonstrating the high accuracy of its learned world model. DreamerV2 further outperforms the top single-GPU agents Rainbow and IQN, whose scores are provided by Dopamine (Castro et al., 2018). According to its authors, SimPLe (Kaiser et al., 2019) was only evaluated on an easier subset of 36 games and trained for fewer steps and additional training does not further increase its performance.

图 1：在含粘性动作、2 亿步设定下，55 款 Atari 游戏的玩家归一化中位分数。DreamerV2 是首个纯在世界模型内学习并达到人类水平 Atari 表现的智能体，说明其学到的世界模型精度很高。DreamerV2 还超过了顶级单 GPU 智能体 Rainbow 与 IQN（分数由 Dopamine 提供，Castro et al., 2018）。按其作者说明，SimPLe（Kaiser et al., 2019）仅在更简单的 36 款子集上评估、训练步数更少，且继续训练不会再提升表现。

To successfully operate in unknown environments, reinforcement learning agents need to learn about their environments over time. World models are an explicit way to represent an agent’s knowledge about its environment. Compared to model-free reinforcement learning that learns through trial and error, world models facilitate generalization and can predict the outcomes of potential actions to enable planning (Sutton, 1991). Capturing general aspects of the environment, world models have been shown to be effective for transfer to novel tasks (Byravan et al., 2019), directed exploration (Sekar et al., 2020), and generalization from offline datasets (Yu et al., 2020). When the inputs are high-dimensional images, latent dynamics models predict ahead in an abstract latent space (Watter et al., 2015; Ha and Schmidhuber, 2018; Hafner et al., 2018; Zhang et al., 2019). Predicting compact representations instead of images has been hypothesized to reduce accumulating errors and their small memory footprint enables thousands of parallel predictions on a single GPU (Hafner et al., 2018; Hafner et al., 2019). Leveraging this approach, the recent Dreamer agent (Hafner et al., 2019) has solved a wide range of continuous control tasks from image inputs.

要在未知环境中成功运作，强化学习智能体需要随时间学习环境。世界模型是显式表示智能体环境知识的一种方式。相较通过试错学习的无模型强化学习，世界模型便于泛化，并能预测潜在动作的结果以支持规划（Sutton, 1991）。由于捕捉环境的一般方面，世界模型已被证明对迁移到新任务（Byravan et al., 2019）、有向探索（Sekar et al., 2020）以及从离线数据集泛化（Yu et al., 2020）有效。当输入是高维图像时，潜动态模型在抽象潜空间中向前预测（Watter et al., 2015；Ha and Schmidhuber, 2018；Hafner et al., 2018；Zhang et al., 2019）。预测紧凑表示而非图像，被认为可减少误差累积；其较小内存占用也使单块 GPU 上可并行数千条预测（Hafner et al., 2018；Hafner et al., 2019）。借助这一思路，近期的 Dreamer 智能体（Hafner et al., 2019）已从图像输入解决了大量连续控制任务。

Despite their intriguing properties, world models have so far not been accurate enough to compete with the state-of-the-art model-free algorithms on the most competitive benchmarks. The well-established Atari benchmark (Bellemare et al., 2013) historically required model-free algorithms to achieve human-level performance, such as DQN (Mnih et al., 2015), A3C (Mnih et al., 2016), or Rainbow (Hessel et al., 2018). Several attempts at learning accurate world models of Atari games have been made, without achieving competitive performance (Oh et al., 2015; Chiappa et al., 2017; Kaiser et al., 2019). On the other hand, the recently proposed MuZero agent (Schrittwieser et al., 2019) shows that planning can achieve impressive performance on board games and deterministic Atari games given extensive engineering effort and a vast computational budget. However, its implementation is not available to the public and it would require over 2 months of computation to train even one agent on a GPU, rendering it impractical for most research groups.

尽管性质诱人，世界模型迄今仍不够准确，难以在最具竞争力的基准上与最先进的无模型算法抗衡。成熟的 Atari 基准（Bellemare et al., 2013）历史上需要无模型算法才能达到人类水平，例如 DQN（Mnih et al., 2015）、A3C（Mnih et al., 2016）或 Rainbow（Hessel et al., 2018）。已有多次尝试学习准确的 Atari 世界模型，但未达到有竞争力的表现（Oh et al., 2015；Chiappa et al., 2017；Kaiser et al., 2019）。另一方面，近期提出的 MuZero（Schrittwieser et al., 2019）表明，在大量工程与巨大算力下，规划可在棋类与确定性 Atari 上取得令人印象深刻的表现。但其实现未公开，即便在 GPU 上训练一个智能体也需逾两个月，对多数研究组不现实。

In this paper, we introduce DreamerV2, the first reinforcement learning agent that achieves human-level performance on the Atari benchmark by learning behaviors purely within a separately trained world model, as shown in Figure 1. Learning successful behaviors purely within the world model demonstrates that the world model learns to accurately represent the environment. To achieve this, we apply small modifications to the Dreamer agent (Hafner et al., 2019), such as using discrete latents and balancing terms within the KL loss. Using a single GPU and a single environment instance, DreamerV2 outperforms top single-GPU Atari agents Rainbow (Hessel et al., 2018) and IQN (Dabney et al., 2018), which rest upon years of model-free reinforcement learning research (Van Hasselt et al., 2015; Schaul et al., 2015; Wang et al., 2016; Bellemare et al., 2017; Fortunato et al., 2017). Moreover, aspects of these algorithms are complementary to our world model and could be integrated into the Dreamer framework in the future. To rigorously compare the algorithms, we report scores normalized by both a human gamer (Mnih et al., 2015) and the human world record (Toromanoff et al., 2019) and make a suggestion for reporting scores going forward.

本文提出 DreamerV2：首个通过完全在单独训练的世界模型内学习行为、从而在 Atari 基准上达到人类水平的强化学习智能体，见图 1。行为完全在世界模型内学成，说明世界模型学会了准确表示环境。为此，我们对 Dreamer（Hafner et al., 2019）做了若干小改动，例如使用离散潜变量，并在 KL 损失中平衡各项。使用单块 GPU 与单个环境实例，DreamerV2 超过顶级单 GPU Atari 智能体 Rainbow（Hessel et al., 2018）与 IQN（Dabney et al., 2018）；这些方法建立在多年无模型强化学习研究之上（Van Hasselt et al., 2015；Schaul et al., 2015；Wang et al., 2016；Bellemare et al., 2017；Fortunato et al., 2017）。此外，这些算法的某些要素与我们的世界模型互补，未来可并入 Dreamer 框架。为严格比较，我们同时报告按人类玩家（Mnih et al., 2015）与人类世界纪录（Toromanoff et al., 2019）归一化的分数，并对今后如何汇报分数给出建议。

2 DreamerV2

We present DreamerV2, an evolution of the Dreamer agent (Hafner et al., 2019). We refer to the original Dreamer agent as DreamerV1 throughout this paper. This section describes the complete DreamerV2 algorithm, consisting of the three typical components of a model-based agent (Sutton, 1991). We learn the world model from a dataset of past experience, learn an actor and critic from imagined sequences of compact model states, and execute the actor in the environment to grow the experience dataset. In Appendix C, we include a list of changes that we applied to DreamerV1 and which of them we found to increase empirical performance.

我们提出 DreamerV2，作为 Dreamer 智能体（Hafner et al., 2019）的演进。本文将原 Dreamer 称作 DreamerV1。本节描述完整的 DreamerV2 算法，包含基于模型智能体的三个典型组件（Sutton, 1991）。我们从过往经验数据集学习世界模型，从紧凑模型状态的想象序列学习演员与评论家，并在环境中执行演员以扩充经验数据集。附录 C 列出我们对 DreamerV1 所做改动，以及其中哪些在经验上提升了表现。

2.1 World Model Learning

2.1 世界模型学习

World models summarize an agent’s experience into a predictive model that can be used in place of the environment to learn behaviors. When inputs are high-dimensional images, it is beneficial to learn compact state representations of the inputs to predict ahead in this learned latent space (Watter et al., 2015; Karl et al., 2016; Ha and Schmidhuber, 2018). These models are called latent dynamics models. Predicting ahead in latent space not only facilitates long-term predictions, it also allows to efficiently predict thousands of compact state sequences in parallel in a single batch, without having to generate images. DreamerV2 builds upon the world model that was introduced by PlaNet (Hafner et al., 2018) and used in DreamerV1, by replacing its Gaussian latents with categorical variables.

世界模型把智能体的经验总结为可替代环境、用于学习行为的预测模型。当输入是高维图像时，学习输入的紧凑状态表示并在该潜空间中向前预测是有益的（Watter et al., 2015；Karl et al., 2016；Ha and Schmidhuber, 2018）。这类模型称为潜动态模型。在潜空间中向前预测不仅利于长期预测，还能在单个批次中高效并行预测数千条紧凑状态序列，而无需生成图像。DreamerV2 建立在 PlaNet（Hafner et al., 2018）提出、并用于 DreamerV1 的世界模型之上，将其高斯潜变量替换为类别变量。

Figure 2: World Model Learning. The training sequence of images $x_{t}$ is encoded using the CNN. The RSSM uses a sequence of deterministic recurrent states $h_{t}$. At each step, it computes a posterior stochastic state $z_{t}$ that incorporates information about the current image $x_{t}$, as well as a prior stochastic state $\hat{z}_{t}$ that tries to predict the posterior without access to the current image. Unlike in PlaNet and DreamerV1, the stochastic state of DreamerV2 is a vector of multiple categorical variables. The learned prior is used for imagination, as shown in Figure 3. The KL loss both trains the prior and regularizes how much information the posterior incorporates from the image. The regularization increases robustness to novel inputs. It also encourages reusing existing information from past steps to predict rewards and reconstruct images, thus learning long-term dependencies.

图 2：世界模型学习。训练图像序列 $x_{t}$ 经 CNN 编码。RSSM 使用一串确定性循环状态 $h_{t}$。每一步计算纳入当前图像 $x_{t}$ 信息的后验随机状态 $z_{t}$，以及试图在看不到当前图像时预测后验的先验随机状态 $\hat{z}_{t}$。与 PlaNet 与 DreamerV1 不同，DreamerV2 的随机状态是多个类别变量组成的向量。学到的先验用于想象，见图 3。KL 损失既训练先验，也正则化后验从图像中纳入多少信息。该正则化增强了对新颖输入的鲁棒性，并鼓励复用过去步骤中的已有信息来预测奖励与重建图像，从而学习长期依赖。

Experience dataset

The world model is trained from the agent’s growing dataset of past experience that contains sequences of images $x_{1:T}$, actions $a_{1:T}$, rewards $r_{1:T}$, and discount factors $\gamma_{1:T}$. The discount factors equal a fixed hyper parameter $\gamma=0.999$ for time steps within an episode and are set to zero for terminal time steps. For training, we use batches of $B=50$ sequences of fixed length $L=50$ that are sampled randomly within the stored episodes. To observe enough episode ends during training, we sample the start index of each training sequence uniformly within the episode and then clip it to not exceed the episode length minus the training sequence length.

经验数据集

世界模型从智能体不断增长的过往经验数据集训练，其中包含图像序列 $x_{1:T}$、动作 $a_{1:T}$、奖励 $r_{1:T}$ 与折扣因子 $\gamma_{1:T}$。回合内时间步的折扣因子等于固定超参 $\gamma=0.999$，终止时间步设为零。训练时使用 $B=50$ 条、固定长度 $L=50$ 的序列批次，在已存储回合内随机采样。为在训练中观察到足够多的回合结束，我们将每条训练序列的起始下标在回合内均匀采样，再裁剪使其不超过回合长度减去训练序列长度。

Model components

The world model consists of an image encoder, a Recurrent State-Space Model (Hafner et al., 2018, RSSM;) to learn the dynamics, and predictors for the image, reward, and discount factor. The world model is summarized in Figure 2. The RSSM uses a sequence of deterministic recurrent states $h_{t}$, from which it computes two distributions over stochastic states at each step. The posterior state $z_{t}$ incorporates information about the current image $x_{t}$, while the prior state $\hat{z}_{t}$ aims to predict the posterior without access to the current image. The concatenation of deterministic and stochastic states forms the compact model state. From the posterior model state, we reconstruct the current image $x_{t}$ and predict the reward $r_{t}$ and discount factor $\gamma_{t}$. The model components are:

模型组件

世界模型由图像编码器、用于学习动态的循环状态空间模型（Hafner et al., 2018，RSSM）以及图像、奖励与折扣因子的预测器组成，概括见图 2。RSSM 使用一串确定性循环状态 $h_{t}$，并在每一步计算两组随机状态分布。后验状态 $z_{t}$ 纳入当前图像 $x_{t}$ 的信息，先验状态 $\hat{z}_{t}$ 则试图在看不到当前图像时预测后验。确定性与随机状态的拼接构成紧凑模型状态。从后验模型状态重建当前图像 $x_{t}$，并预测奖励 $r_{t}$ 与折扣因子 $\gamma_{t}$。各组件为：

$\displaystyle\begin{aligned} \begin{aligned} \raisebox{8.39578pt}{\hbox to0.0pt{\hss\vbox to0.0pt{\hbox{\text{RSSM}\hskip 4.30554pt\begin{cases}\hphantom{A}\
\hphantom{A}\
\hphantom{A}\end{cases}\hskip-10.33327pt$}\vss}}}&\text{Recurrent model:}\hskip 35.00005pt&&h_{t}&\ =&\ f_{\phi}(h_{t-1},z_{t-1},a_{t-1})\
&\text{Representation model:}\hskip 35.00005pt&&z_{t}&\ \sim&\ q_{\phi}(z_{t};|;h_{t},x_{t})\
&\text{Transition predictor:}\hskip 35.00005pt&&\hat{z}{t}&\ \sim&\ p{\phi}(\hat{z}{t};|;h{t})\
&\text{Image predictor:}\hskip 35.00005pt&&\hat{x}{t}&\ \sim&\ p{\phi}(\hat{x}{t};|;h{t},z_{t})\
&\text{Reward predictor:}\hskip 35.00005pt&&\hat{r}{t}&\ \sim&\ p{\phi}(\hat{r}{t};|;h{t},z_{t})\
&\text{Discount predictor:}\hskip 35.00005pt&&\hat{\gamma}{t}&\ \sim&\ p{\phi}(\hat{\gamma}{t};|;h{t},z_{t}).\end{aligned}\end{aligned} \tag{1}$$

All components are implemented as neural networks and $\phi$ describes their combined parameter vector. The transition predictor guesses the next model state only from the current model state and the action but without using the next image, so that we can later learn behaviors by predicting sequences of model states without having to observe or generate images. The discount predictor lets us estimate the probability of an episode ending when learning behaviors from model predictions.

所有组件均实现为神经网络，$\phi$ 表示其合并参数向量。转移预测器仅根据当前模型状态与动作猜测下一模型状态，而不使用下一张图像，从而我们之后可在无需观察或生成图像的情况下，通过预测模型状态序列来学习行为。折扣预测器使我们在从模型预测学习行为时，能够估计回合结束的概率。

3 sample = one_hot(draw(logits)****) # sample has no gradient probs = softmax(logits) # want gradient of this sample = sample + probs - stop_grad(probs) # has gradient of probs
3 sample = one_hot(draw(logits)****) # sample has no gradient
probs = softmax(logits) # want gradient of this
sample = sample + probs - stop_grad(probs) # has gradient of probs

3 sample = one_hot(draw(logits)****) # sample has no gradient
probs = softmax(logits) # want gradient of this
sample = sample + probs - stop_grad(probs) # has gradient of probs

Algorithm 1 Straight-Through Gradients with Automatic Differentiation

3 sample = one_hot(draw(logits)****) # sample 无梯度 probs = softmax(logits) # 需要该量的梯度 sample = sample + probs - stop_grad(probs) # 带有 probs 的梯度
3 sample = one_hot(draw(logits)****) # sample 无梯度
probs = softmax(logits) # 需要该量的梯度
sample = sample + probs - stop_grad(probs) # 带有 probs 的梯度

3 sample = one_hot(draw(logits)****) # sample 无梯度
probs = softmax(logits) # 需要该量的梯度
sample = sample + probs - stop_grad(probs) # 带有 probs 的梯度

算法 1 配合自动微分的直通梯度

Neural networks

The representation model is implemented as a Convolutional Neural Network (LeCun et al., 1989, CNN;) followed by a Multi-Layer Perceptron (MLP) that receives the image embedding and the deterministic recurrent state. The RSSM uses a Gated Recurrent Unit (Cho et al., 2014, GRU;) to compute the deterministic recurrent states. The model state is the concatenation of deterministic GRU state and a sample of the stochastic state. The image predictor is a transposed CNN and the transition, reward, and discount predictors are MLPs. We down-scale the $84\times 84$ grayscale images to $64\times 64$ pixels so that we can apply the convolutional architecture of DreamerV1. We use the ELU activation function for all components of the model (Clevert et al., 2015). The world model uses a total of 20M trainable parameters.

神经网络

表示模型实现为卷积神经网络（LeCun et al., 1989，CNN），后接接收图像嵌入与确定性循环状态的多层感知机（MLP）。RSSM 使用门控循环单元（Cho et al., 2014，GRU）计算确定性循环状态。模型状态是确定性 GRU 状态与随机状态样本的拼接。图像预测器是转置 CNN，转移、奖励与折扣预测器是 MLP。我们将 $84\times 84$ 灰度图像下采样到 $64\times 64$，以便沿用 DreamerV1 的卷积结构。模型各组件均使用 ELU 激活（Clevert et al., 2015）。世界模型共有约 2000 万可训练参数。

3 kl_loss = alpha ***** compute_kl(stop_grad(approx_posterior), prior**)** + (1 - alpha) ***** compute_kl(approx_posterior, stop_grad(prior)****)
3 kl_loss = alpha ***** compute_kl(stop_grad(approx_posterior), prior**)**
+ (1 - alpha) ***** compute_kl(approx_posterior, stop_grad(prior)****)

3 kl_loss = alpha ***** compute_kl(stop_grad(approx_posterior), prior**)**
+ (1 - alpha) ***** compute_kl(approx_posterior, stop_grad(prior)****)

Algorithm 2 KL Balancing with Automatic Differentiation

3 kl_loss = alpha * compute_kl(stop_grad(approx_posterior), prior**)** + (1 - alpha) * compute_kl(approx_posterior, stop_grad(prior)****)
3 kl_loss = alpha * compute_kl(stop_grad(approx_posterior), prior**)**
+ (1 - alpha) * compute_kl(approx_posterior, stop_grad(prior)****)

3 kl_loss = alpha * compute_kl(stop_grad(approx_posterior), prior**)**
+ (1 - alpha) * compute_kl(approx_posterior, stop_grad(prior)****)

算法 2 配合自动微分的 KL 平衡

Distributions

The image predictor outputs the mean of a diagonal Gaussian likelihood with unit variance, the reward predictor outputs a univariate Gaussian with unit variance, and the discount predictor outputs a Bernoulli likelihood. In prior work, the latent variable in the model state was a diagonal Gaussian that used reparameterization gradients during backpropagation (Kingma and Welling, 2013; Rezende et al., 2014). In DreamerV2, we instead use a vector of several categorical variables and optimize them using straight-through gradients (Bengio et al., 2013), which are easy to implement using automatic differentiation as shown in Algorithm 1. We discuss possible benefits of categorical over Gaussian latents in the experiments section.

分布

图像预测器输出单位方差对角高斯似然的均值，奖励预测器输出单位方差一元高斯，折扣预测器输出伯努利似然。先前工作中，模型状态里的潜变量是对角高斯，并在反向传播中使用重参数化梯度（Kingma and Welling, 2013；Rezende et al., 2014）。在 DreamerV2 中，我们改为使用若干类别变量组成的向量，并用直通梯度优化（Bengio et al., 2013）；如算法 1 所示，这在自动微分下很容易实现。类别潜变量相对高斯潜变量的可能好处，我们在实验部分讨论。

Loss function

All components of the world model are optimized jointly. The distributions produced by the image predictor, reward predictor, discount predictor, and transition predictor are trained to maximize the log-likelihood of their corresponding targets. The representation model is trained to produce model states that facilitates these prediction tasks, through the expectation below. Moreover, it is regularized to produce model states with high entropy, such that the model becomes robust to many different model states during training. The loss function for learning the world model is:

损失函数

世界模型的所有组件联合优化。图像、奖励、折扣与转移预测器产生的分布被训练以最大化对应目标的对数似然。表示模型被训练为产生便于这些预测任务的模型状态，见下方期望。此外，它被正则化为产生高熵的模型状态，使模型在训练中对许多不同模型状态更鲁棒。学习世界模型的损失函数为：

$$\displaystyle\begin{aligned} \mathcal{L}(\phi)\doteq\operatorname{E}{q{\phi}(z_{1:T};|;a_{1:T},x_{1:T})}\Big[\textstyle\sum_{t=1}^{T}&\hskip 1.19995pt\underbracket{-\ln p_{\phi}(x_{t};|;h_{t},z_{t})\hskip 0.0pt}{\text{image log loss}}\hskip 1.19995pt\underbracket{-\ln p{\phi}(r_{t};|;h_{t},z_{t})\hskip 0.0pt}{\text{reward log loss}}\hskip 1.19995pt\underbracket{-\ln p{\phi}(\gamma_{t};|;h_{t},z_{t})\hskip 0.0pt}{\text{discount log loss}}\
&\hskip 1.19995pt\underbracket{+\beta\operatorname{KL}!\big[q{\phi}(z_{t};|;h_{t},x_{t});\big|;p_{\phi}(z_{t};|;h_{t})\big]\hskip 0.0pt}_{\text{KL loss}}\Big].\end{aligned} \tag{2}$$

We jointly minimize the loss function with respect to the vector $\phi$ that contains all parameters of the world model using the Adam optimizer (Kingma and Ba, 2014). We scale the KL loss by $\beta=0.1$ for Atari and by $\beta=1.0$ for continuous control (Higgins et al., 2016).

我们用 Adam 优化器（Kingma and Ba, 2014）对包含世界模型全部参数的向量 $\phi$ 联合最小化该损失。Atari 上 KL 损失缩放为 $\beta=0.1$，连续控制上为 $\beta=1.0$（Higgins et al., 2016）。

KL balancing

The world model loss function in Equation 2 is the ELBO or variational free energy of a hidden Markov model that is conditioned on the action sequence. The world model can thus be interpreted as a sequential VAE, where the representation model is the approximate posterior and the transition predictor is the temporal prior. In the ELBO objective, the KL loss serves two purposes: it trains the prior toward the representations, and it regularizes the representations toward the prior. However, learning the transition function is difficult and we want to avoid regularizing the representations toward a poorly trained prior. To solve this problem, we minimize the KL loss faster with respect to the prior than the representations by using different learning rates, $\alpha=0.8$ for the prior and $1-\alpha$ for the approximate posterior. We implement this technique as shown in Algorithm 2 and refer to it as KL balancing. KL balancing encourages learning an accurate prior over increasing posterior entropy, so that the prior better approximates the aggregate posterior. KL balancing is different from and orthogonal to beta-VAEs (Higgins et al., 2016).

KL 平衡

方程 2 中的世界模型损失是以动作序列为条件的隐马尔可夫模型的 ELBO 或变分自由能。因而世界模型可解释为序列 VAE：表示模型是近似后验，转移预测器是时间先验。在 ELBO 目标中，KL 损失有两个作用：把先验拉向表示，并把表示正则化向先验。然而学习转移函数很难，我们希望避免把表示正则化到训练不佳的先验上。为此，我们对先验使用比表示更大的学习率来最小化 KL：先验用 $\alpha=0.8$，近似后验用 $1-\alpha$。实现如算法 2，我们称之为 KL 平衡。KL 平衡鼓励在提高后验熵之前先学到准确先验，使先验更好逼近聚合后验。KL 平衡与 beta-VAE（Higgins et al., 2016）不同且正交。

2.2 Behavior Learning

2.2 行为学习

DreamerV2 learns long-horizon behaviors purely within its world model using an actor and a critic. The actor chooses actions for predicting imagined sequences of compact model states. The critic accumulates the future predicted rewards to take into account rewards beyond the planning horizon. Both the actor and critic operate on top of the learned model states and thus benefit from the representations learned by the world model. The world model is fixed during behavior learning, so the actor and value gradients do not affect its representations. Not predicting images during behavior learning lets us efficiently simulate 2500 latent trajectories in parallel on a single GPU.

DreamerV2 使用演员与评论家，完全在其世界模型内学习长视野行为。演员选择动作用于预测紧凑模型状态的想象序列。评论家累积未来预测奖励，以计入规划视野之外的奖励。演员与评论家都在学到的模型状态上运作，因而受益于世界模型学到的表示。行为学习期间世界模型固定，因此演员与价值梯度不会影响其表示。行为学习时不预测图像，使我们能在单块 GPU 上高效并行模拟 2500 条潜轨迹。

Imagination MDP

To learn behaviors within the latent space of the world model, we define the imagination MPD as follows. The distribution of initial states $\hat{z}_{0}$ in the imagination MDP is the distribution of compact model states encountered during world model training. From there, the transition predictor $p_{\phi}(\hat{z}_{t}\;|\;\hat{z}_{t-1},\hat{a}_{t-1})$ outputs sequences $\hat{z}_{1:H}$ of compact model states up to the imagination horizon $H=15$. The mean of the reward predictor $p_{\phi}(\hat{r}_{t}\;|\;\hat{z}_{t})$ is used as reward sequence $\hat{r}_{1:H}$. The discount predictor $p_{\phi}(\hat{\gamma}_{t}\;|\;\hat{z}_{t})$ outputs the discount sequence $\hat{\gamma}_{1:H}$ that is used to down-weight rewards. Moreover, we weigh the loss terms of the actor and critic by the cumulative predicted discount factors to softly account for the possibility of episode ends.

想象 MDP

为在世界模型潜空间中学习行为，我们如下定义想象 MDP。想象 MDP 中初始状态 $\hat{z}_{0}$ 的分布，是世界模型训练中遇到的紧凑模型状态分布。由此，转移预测器 $p_{\phi}(\hat{z}_{t}\;|\;\hat{z}_{t-1},\hat{a}_{t-1})$ 输出直至想象视野 $H=15$ 的紧凑模型状态序列 $\hat{z}_{1:H}$。奖励预测器 $p_{\phi}(\hat{r}_{t}\;|\;\hat{z}_{t})$ 的均值用作奖励序列 $\hat{r}_{1:H}$。折扣预测器 $p_{\phi}(\hat{\gamma}_{t}\;|\;\hat{z}_{t})$ 输出折扣序列 $\hat{\gamma}_{1:H}$，用于下调奖励权重。此外，我们用累积预测折扣因子对演员与评论家的损失项加权，以软性计入回合结束的可能性。

Figure 3: Actor Critic Learning. The world model learned in Figure 2 is used for learning a policy from trajectories imagined in the compact latent space. The trajectories start from posterior states computed during model training and predict forward by sampling actions from the actor network. The critic network predicts the expected sum of future rewards for each state. The critic uses temporal difference learning on the imagined rewards. The actor is trained to maximize the critic prediction, via reinforce gradients, straight-through gradients of the world model, or a combination of them.

图 3：演员–评论家学习。图 2 学到的世界模型用于从紧凑潜空间中想象的轨迹学习策略。轨迹从模型训练期间计算的后验状态出发，通过从演员网络采样动作向前预测。评论家网络预测每个状态的期望未来奖励和。评论家在想象奖励上做时序差分学习。演员被训练以最大化评论家预测，可通过 reinforce 梯度、世界模型的直通梯度，或其组合。

Model components

To learn long-horizon behaviors in the imagination MDP, we leverage a stochastic actor that chooses actions and a deterministic critic. The actor and critic are trained cooperatively, where the actor aims to output actions that lead to states that maximize the critic output, while the critic aims to accurately estimate the sum of future rewards achieved by the actor from each imagined state. The actor and critic use the parameter vectors $\psi$ and $\xi$, respectively:

模型组件

为在想象 MDP 中学习长视野行为，我们使用选择动作的随机演员与确定性评论家。二者协作训练：演员旨在输出导向最大化评论家输出之状态的动作，评论家旨在准确估计演员从每个想象状态获得的未来奖励和。演员与评论家分别使用参数向量 $\psi$ 与 $\xi$：

$$\displaystyle\begin{aligned} \begin{aligned} &\text{Actor:}\hskip 35.00005pt&&\hat{a}{t}\sim p{\psi}(\hat{a}{t};|;\hat{z}{t})\
&\text{Critic:}\hskip 35.00005pt&&v_{\xi}(\hat{z}{t})\approx\operatorname{E}{p_{\phi},p_{\psi}}\Big[\textstyle\sum_{\tau\geq t}\hat{\gamma}^{\tau-t}\hat{r}_{\tau}\Big].\
\end{aligned}\end{aligned} \tag{3}$$

In contrast to the actual environment, the latent state sequence is Markovian, so that there is no need for the actor and critic to condition on more than the current model state. The actor and critic are both MLPs with ELU activations (Clevert et al., 2015) and use 1M trainable parameters each. The actor outputs a categorical distribution over actions and the critic has a deterministic output. The two components are trained from the same imagined trajectories but optimize separate loss functions.

与真实环境不同，潜状态序列是马尔可夫的，因此演员与评论家无需以多于当前模型状态为条件。二者均为带 ELU 激活的 MLP（Clevert et al., 2015），各约 100 万可训练参数。演员输出动作上的类别分布，评论家为确定性输出。两组件从同一批想象轨迹训练，但优化各自的损失函数。

Critic loss function

The critic aims to predict the discounted sum of future rewards that the actor achieves in a given model state, known as the state value. For this, we leverage temporal-difference learning, where the critic is trained toward a value target that is constructed from intermediate rewards and critic outputs for later states. A common choice is the 1-step target that sums the current reward and the critic output for the following state. However, the imagination MDP lets us generate on-policy trajectories of multiple steps, suggesting the use of n-step targets that incorporate reward information into the critic more quickly. We follow DreamerV1 in using the more general $\lambda$-target (Sutton and Barto, 2018; Schulman et al., 2015) that is defined recursively as follows:

评论家损失函数

评论家旨在预测演员在给定模型状态下获得的折扣未来奖励和，即状态价值。为此我们利用时序差分学习：评论家朝由中间奖励与更晚状态的评论家输出构造的价值目标训练。常见选择是 1 步目标：当前奖励加下一状态的评论家输出。但想象 MDP 允许我们生成多步同策略轨迹，因而宜使用能更快把奖励信息纳入评论家的 n 步目标。我们沿用 DreamerV1，使用更一般的 $\lambda$-目标（Sutton and Barto, 2018；Schulman et al., 2015），递归定义为：

$$\displaystyle\begin{aligned} V^{\lambda}{t}\doteq\hat{r}{t}+\hat{\gamma}{t}\begin{cases}(1-\lambda)v{\xi}(\hat{z}{t+1})+\lambda V^{\lambda}{t+1}&\text{if}\quad t<H,\
v_{\xi}(\hat{z}_{H})&\text{if}\quad t=H.\
\end{cases}\end{aligned} \tag{4}$$

Intuitively, the $\lambda$-target is a weighted average of n-step returns for different horizons, where longer horizons are weighted exponentially less. We set $\lambda=0.95$ in practice, to focus more on long horizon targets than on short horizon targets. Given a trajectory of model states, rewards, and discount factors, we train the critic to regress the $\lambda$-return using a squared loss:

直观上，$\lambda$-目标是不同视野 n 步回报的加权平均，较长视野的权重指数衰减。实践中设 $\lambda=0.95$，以更侧重长视野目标。给定模型状态、奖励与折扣因子轨迹，我们用平方损失训练评论家回归 $\lambda$-回报：

$$\displaystyle\begin{aligned} \mathcal{L}(\xi)\doteq\operatorname{E}_{p_{\phi},p_{\psi}}\Big[\textstyle\sum_{t=1}^{H-1}\frac{1}{2}\big(v_{\xi}(\hat{z}_{t})-\operatorname{sg}(V^{\lambda}_{t})\big)^{2}\Big].\end{aligned} \tag{5}$$

We optimize the critic loss with respect to the critic parameters $\xi$ using the Adam optimizer. There is no loss term for the last time step because the target equals the critic at that step. We stop the gradients around the targets, denoted by the $\operatorname{sg}(\cdot)$ function, as typical in the literature. We stabilize value learning using a target network (Mnih et al., 2015), namely, we compute the targets using a copy of the critic that is updated every $100$ gradient steps.

我们用 Adam 对评论家参数 $\xi$ 优化该损失。最后一步没有损失项，因为目标等于该步的评论家。我们按文献惯例对目标停止梯度，记为 $\operatorname{sg}(\cdot)$。用目标网络稳定价值学习（Mnih et al., 2015）：用每 100 次梯度步更新一次的评论家副本计算目标。

Actor loss function

The actor aims to output actions that maximize the prediction of long-term future rewards made by the critic. To incorporate intermediate rewards more directly, we train the actor to maximize the same $\lambda$-return that was computed for training the critic. There are different gradient estimators for maximizing the targets with respect to the actor parameters. DreamerV2 combines unbiased but high-variance Reinforce gradients with biased but low-variance straight-through gradients. Moreover, we regularize the entropy of the actor to encourage exploration where feasible while allowing the actor to choose precise actions when necessary.

演员损失函数

演员旨在输出最大化评论家所预测长期未来奖励的动作。为更直接纳入中间奖励，我们训练演员最大化为训练评论家而计算的同一 $\lambda$-回报。相对演员参数最大化目标有不同梯度估计器。DreamerV2 将无偏但高方差的 Reinforce 梯度与有偏但低方差的直通梯度结合。此外，我们对演员熵正则化，以在可行时鼓励探索，同时允许演员在必要时选择精确动作。

Learning by Reinforce (Williams, 1992) maximizes the actor’s probability of its own sampled actions weighted by the values of those actions. The variance of this estimator can be reduced by subtracting the state value as baseline, which does not depend on the current action. Intuitively, subtracting the baseline centers the weights and leads to faster learning. The benefit of Reinforce is that it produced unbiased gradients and the downside is that it can have high variance, even with baseline.

按 Reinforce（Williams, 1992）学习时，以动作价值为权重最大化演员对其自身采样动作的概率。减去不依赖当前动作的状态价值作为基线，可降低该估计器的方差。直观上，减去基线使权重居中并加快学习。Reinforce 的好处是产生无偏梯度，代价是即使有基线也可能方差较大。

DreamerV1 relied entirely on reparameterization gradients (Kingma and Welling, 2013; Rezende et al., 2014) to train the actor directly by backpropagating value gradients through the sequence of sampled model states and actions. DreamerV2 uses both discrete latents and discrete actions. To backpropagate through the sampled actions and state sequences, we leverage straight-through gradients (Bengio et al., 2013). This results in a biased gradient estimate with low variance. The combined actor loss function is:

DreamerV1 完全依赖重参数化梯度（Kingma and Welling, 2013；Rezende et al., 2014），通过沿采样的模型状态与动作序列反向传播价值梯度来直接训练演员。DreamerV2 同时使用离散潜变量与离散动作。为穿过采样动作与状态序列反向传播，我们利用直通梯度（Bengio et al., 2013）。这得到低方差但有偏的梯度估计。合并后的演员损失为：

$\displaystyle\begin{aligned} \begin{aligned} \mathcal{L}(\psi)\doteq\operatorname{E}_{p_{\phi},p_{\psi}}\Big[\textstyle\sum_{t=1}^{H-1}\big(\hskip 1.19995pt\underbracket{-\rho\ln p_{\psi}(\hat{a}_{t}\;|\;\hat{z}_{t})\operatorname{sg}(V^{\lambda}_{t}-v_{\xi}({\hat{z}_{t}}))\hskip 0.0pt}_{\text{reinforce}}\hskip 1.19995pt\underbracket{-(1-\rho)V^{\lambda}_{t}\hskip 0.0pt}_{\text{\begin{subarray}{c}\text{dynamics}\
\text{backprop}\end{subarray}\vskip-6.02777pt}}\hskip 1.19995pt\underbracket{-\eta\operatorname{H}[a_{t}|\hat{z}_{t}]\hskip 0.0pt}_{\text{entropy regularizer}}\big)\Big].\end{aligned}\end{aligned} \tag{6}$

We optimize the actor loss with respect to the actor parameters $\psi$ using the Adam optimizer. We consider both Reinforce gradients and straight-through gradients, which backpropagate directly through the learned dynamics. Intuitively, the low-variance but biased dynamics backpropagation could learn faster initially and the unbiased but high-variance could to converge to a better solution. For Atari, we find Reinforce gradients to work substantially better and use $\rho=1$ and $\eta=10^{-3}$. For continuous control, we find dynamics backpropagation to work substantially better and use $\rho=0$ and $\eta=10^{-4}$. Annealing these hyper parameters can improve performance slightly but to avoid the added complexity we report the scores without annealing.

我们用 Adam 对演员参数 $\psi$ 优化该损失。我们同时考虑 Reinforce 梯度与穿过所学动态直接反向传播的直通梯度。直观上，低方差但有偏的动态反向传播可能初期学得更快，而无偏但高方差的方法可能收敛到更好解。对 Atari，我们发现 Reinforce 明显更好，取 $\rho=1$、$\eta=10^{-3}$。对连续控制，动态反向传播明显更好，取 $\rho=0$、$\eta=10^{-4}$。退火这些超参可略微提升，但为避免额外复杂性，我们报告不退火的分数。

3 Experiments

3 实验

We evaluate DreamerV2 on the well-established Atari benchmark with sticky actions, comparing to four strong model-free algorithms. DreamerV2 outperforms the four model-free algorithms in all scenarios. For an extensive comparison, we report four scores according to four aggregation protocols and give a recommendation for meaningfully aggregating scores across games going forward. We also ablate the importance of discrete representations in the world model. Our implementation of DreamerV2 reaches 200M environment steps in under 10 days, while using only a single NVIDIA V100 GPU and a single environment instance. During the 200M environment steps, DreamerV2 learns its policy from 468B compact states imagined under the model, which is 10,000$\times$ more than the 50M inputs received from the real environment after action repeat. Refer to the project website for videos, the source code, and training curves in JSON format.[^1]

我们在含粘性动作的成熟 Atari 基准上评估 DreamerV2，并与四种强无模型算法比较。DreamerV2 在所有场景下均超过这四种算法。为充分比较，我们按四种聚合协议报告四个分数，并就今后如何有意义地跨游戏聚合分数给出建议。我们也消融了世界模型中离散表示的重要性。我们的 DreamerV2 实现在不到 10 天内达到 2 亿环境步，且仅使用单块 NVIDIA V100 GPU 与单个环境实例。在这 2 亿环境步期间，DreamerV2 从模型下想象的 4680 亿紧凑状态学习策略，是动作重复后真实环境所给 5000 万输入的 10,000 倍。视频、源码与 JSON 格式训练曲线见项目网站。[^1]

Experimental setup

We select the 55 games that prior works in the literature from different research labs tend to agree on (Mnih et al., 2016; Brockman et al., 2016; Hessel et al., 2018; Castro et al., 2018; Badia et al., 2020) and recommend this set of games for evaluation going forward. We follow the evaluation protocol of Machado et al. (2018) with 200M environment steps, action repeat of 4, a time limit of 108,000 steps per episode that correspond to 30 minutes of game play, no access to life information, full action space, and sticky actions. Because the world model integrates information over time, DreamerV2 does not use frame stacking. The experiments use a single-task setup where a separate agent is trained for each game. Moreover, each agent uses only a single environment instance. We compare the algorithms based on both human gamer and human world record normalization (Toromanoff et al., 2019).

实验设置

我们选取不同实验室文献中倾向于一致的 55 款游戏（Mnih et al., 2016；Brockman et al., 2016；Hessel et al., 2018；Castro et al., 2018；Badia et al., 2020），并建议今后用该集合评估。我们遵循 Machado et al.（2018）的评估协议：2 亿环境步、动作重复 4、每回合时限 108,000 步（对应 30 分钟游戏）、不使用生命信息、完整动作空间、粘性动作。由于世界模型随时间整合信息，DreamerV2 不使用帧堆叠。实验为单任务设置：每款游戏单独训练一个智能体，且每个智能体只用单个环境实例。比较时同时采用人类玩家与人类世界纪录归一化（Toromanoff et al., 2019）。

Model-free baselines

We compare the learning curves and final scores of DreamerV2 to four model-free algorithms, IQN (Dabney et al., 2018), Rainbow (Hessel et al., 2018), C51 (Bellemare et al., 2017), and DQN (Mnih et al., 2015). We use the scores of these agents provided by the Dopamine framework (Castro et al., 2018) that use sticky actions. These may differ from the reported results in the papers that introduce these algorithms in the deterministic Atari setup. The training time of Rainbow was reported at 10 days on a single GPU and using one environment instance.

无模型基线

我们将 DreamerV2 的学习曲线与最终分数与四种无模型算法比较：IQN（Dabney et al., 2018）、Rainbow（Hessel et al., 2018）、C51（Bellemare et al., 2017）与 DQN（Mnih et al., 2015）。使用 Dopamine 框架（Castro et al., 2018）提供的、采用粘性动作的分数；它们可能与这些算法在确定性 Atari 设定下论文中报告的结果不同。Rainbow 的训练时间据报为单 GPU、单环境实例下约 10 天。

3.1 Atari Performance

3.1 Atari 表现

Figure 4: Atari performance over 200M steps. See Table 1 for numeric scores. The standards in the literature to aggregate over tasks are shown in the left two plots. These normalize scores by a professional gamer and compute the median or mean over tasks (Mnih et al., 2015; Mnih et al., 2016). In Section 3, we point out limitations of this methodology. As a robust measure of performance, we recommend the metric in the right-most plot. We normalize scores by the human world record (Toromanoff et al., 2019) and then clip them, such that exceeding the record does not further increase the score, before averaging over tasks.

图 4：2 亿步上的 Atari 表现。数值分数见表 1。文献中跨任务聚合的惯用做法见左两图：按职业玩家归一化分数后取任务中位数或均值（Mnih et al., 2015；Mnih et al., 2016）。第 3 节指出该方法的局限。作为稳健的表现度量，我们推荐最右图的指标：先按人类世界纪录归一化（Toromanoff et al., 2019），再裁剪使超过纪录不再加分，然后对任务取平均。

Agent	Gamer Median	Gamer Mean	Record Mean	Clipped Record Mean
DreamerV2	2.15	11.33	0.44	0.28
DreamerV2 (schedules)	2.64	10.45	0.43	0.28
IQN	1.29	08.85	0.21	0.21
Rainbow	1.47	09.12	0.17	0.17
C51	1.09	07.70	0.15	0.15
DQN	0.65	02.84	0.12	0.12

Table 1: Atari performance at 200M steps. The scores of the 55 games are aggregated using the four different protocols described in Section 3. To overcome limitations of the previous metrics, we recommend the task mean of clipped record normalized scores as a robust measure of algorithm performance, shown in the right-most column. DreamerV2 outperforms previous single-GPU agents across all metrics. The baseline scores are taken from Dopamine Baselines (Castro et al., 2018).

智能体	玩家中位数	玩家均值	纪录均值	裁剪纪录均值
DreamerV2	2.15	11.33	0.44	0.28
DreamerV2（调度）	2.64	10.45	0.43	0.28
IQN	1.29	08.85	0.21	0.21
Rainbow	1.47	09.12	0.17	0.17
C51	1.09	07.70	0.15	0.15
DQN	0.65	02.84	0.12	0.12

表 1：2 亿步时的 Atari 表现。55 款游戏的分数按第 3 节所述四种协议聚合。为克服先前指标的局限，我们推荐裁剪纪录归一化分数的任务均值作为稳健度量，见最右列。DreamerV2 在所有指标上超过先前单 GPU 智能体。基线分数取自 Dopamine Baselines（Castro et al., 2018）。

The performance curves of DreamerV2 and four standard model-free algorithms are visualized in Figure 4. The final scores at 200M environment steps are shown in Table 1 and the scores on individual games are included in Table K.1. There are different approaches for aggregating the scores across the 55 games and we show that this choice can have a substantial impact on the relative performance between algorithms. To extensively compare DreamerV2 to the model-free algorithms, we consider the following four aggregation approaches:

DreamerV2 与四种标准无模型算法的表现曲线见图 4。2 亿环境步时的最终分数见表 1，各游戏分数见附录表 K.1。跨 55 款游戏聚合分数有不同做法，我们表明这一选择可显著影响算法间的相对表现。为全面比较 DreamerV2 与无模型算法，我们考虑以下四种聚合方式：

• • Gamer Median  Atari scores are commonly normalized based on a random policy and a professional gamer, averaged over seeds, and the median over tasks is reported (Mnih et al., 2015; Mnih et al., 2016). However, if almost half of the scores would be zero, the median would not be affected. Thus, we argue that median scores are not reflective of the robustness of an algorithm and results in wasted computational resources for games that will not affect the score.

• • 玩家中位数 Atari 分数常按随机策略与职业玩家归一化，对种子取平均，再报告任务中位数（Mnih et al., 2015；Mnih et al., 2016）。然而若近一半分数为零，中位数仍不受影响。因此我们认为中位数不能反映算法稳健性，并导致不会影响分数的游戏浪费算力。

• • Gamer Mean  Compared to the task median, the task mean considers all tasks. However, the gamer performed poorly on a small number of games, such as Crazy Climber, James Bond, and Video Pinball. This makes it easy for algorithms to achieve a high normalized score on these few games, which then dominate the task mean so it is not informative of overall performance.

• • 玩家均值 相较任务中位数，任务均值考虑所有任务。但玩家在少数游戏（如 Crazy Climber、James Bond、Video Pinball）上表现很差，算法容易在这些游戏上拿到很高归一化分数，进而主导任务均值，使其不能反映整体表现。

• • Record Mean  Instead of normalizing based on the professional gamer, Toromanoff et al. (2019) suggest to normalize based on the registered human world record of each game. This partially addresses the outlier problem but the mean is still dominated by games where the algorithms easily achieve superhuman performance.

• • 纪录均值 Toromanoff et al.（2019）建议不按职业玩家、而按各游戏登记的人类世界纪录归一化。这部分缓解了离群问题，但均值仍被算法轻易达到超人类表现的游戏主导。

• • Clipped Record Mean  To overcome these limitations, we recommend normalizing by the human world record and then clipping the scores to not exceed a value of 1, so that performance above the record does not further increase the score. The result is a robust measure of algorithm performance on the Atari suite that considers performance across all games.

• • 裁剪纪录均值 为克服上述局限，我们建议按人类世界纪录归一化后，将分数裁剪不超过 1，使超过纪录的表现不再加分。结果是考虑全部游戏、稳健衡量 Atari 套件上算法表现的指标。

From Figure 4 and Table 1, we see that the different aggregation approaches let us examine agent performance from different angles. Interestingly, Rainbow clearly outperforms IQN in the first aggregation method but IQN clearly outperforms Rainbow in the remaining setups. DreamerV2 outperforms the model-free agents in all four metrics, with the largest margin in record normalized mean performance. Despite this, we recommend clipped record normalized mean as the most meaningful aggregation method, as it considers all tasks to a similar degree without being dominated by a small number of outlier scores. In Table 1, we also include DreamerV2 with schedules that anneal the actor entropy loss scale and actor gradient mixing over the course of training, which further increases the gamer median score of DreamerV2.

从图 4 与表 1 可见，不同聚合方式让我们从不同角度审视智能体表现。有趣的是，Rainbow 在第一种聚合上明显超过 IQN，但在其余设定上 IQN 明显超过 Rainbow。DreamerV2 在全部四个指标上超过无模型智能体，在纪录归一化均值上差距最大。尽管如此，我们仍推荐裁剪纪录归一化均值作为最有意义的聚合方法，因为它以相近程度考虑所有任务，而不被少数离群分数主导。表 1 也包含在训练过程中退火演员熵损失尺度与演员梯度混合的 DreamerV2，这进一步提升了其玩家中位分数。

Individual games

The scores on individual Atari games at 200M environment steps are included in Table K.1, alongside the model-free algorithms and the baselines of random play, human gamer, and human world record. We filled in reasonable values for the 2 out of 55 games that have no registered world record. Figure E.1 compares the score differences between DreamerV2 and each model-free algorithm for the individual games. DreamerV2 achieves comparable or higher performance on most games except for Video Pinball. We hypothesize that the reconstruction loss of the world model does not encourage learning a meaningful latent representation because the most important object in the game, the ball, occupies only a single pixel. One the other hand, DreamerV2 achieves the strongest improvements over the model-free agents on the games James Bond, Up N Down, and Assault.

各游戏

2 亿环境步时各 Atari 游戏的分数见附录表 K.1，并与无模型算法以及随机游玩、人类玩家、人类世界纪录基线并列。对 55 款中没有登记世界纪录的 2 款，我们填入合理数值。图 E.1 比较 DreamerV2 与各无模型算法在各游戏上的分数差。DreamerV2 在多数游戏上达到相当或更高表现，Video Pinball 除外。我们猜测世界模型的重建损失未鼓励学到有意义的潜表示，因为该游戏最重要的物体——球——只占一个像素。另一方面，DreamerV2 相对无模型智能体提升最大的游戏是 James Bond、Up N Down 与 Assault。

Figure 5: Clipped record normalized scores of various ablations of the DreamerV2 agent. This experiment uses a slightly earlier version of DreamerV2. The score curves for individual tasks are shown in Figure H.1. The ablations highlight the benefit of using categorical over Gaussian latent variables and of using KL balancing. Moreover, they show that the world model relies on image gradients for learning its representations. Stopping reward gradients even improves performance on some tasks, suggesting that representations that are not specifically trained to predict previously experienced rewards may generalize better to new situations.

图 5：DreamerV2 智能体多种消融的裁剪纪录归一化分数。本实验使用稍早版本的 DreamerV2。各任务分数曲线见图 H.1。消融凸显了类别潜变量相对高斯潜变量以及 KL 平衡的好处。此外表明世界模型依赖图像梯度学习表示。停止奖励梯度甚至在部分任务上提升表现，说明并非专门为预测已见奖励而训练的表示，可能对新情境泛化更好。

3.2 Ablation Study

3.2 消融研究

Agent	Gamer Median	Gamer Mean	Record Mean	Clipped Record Mean
DreamerV2	1.64	11.33	0.36	0.25
No Layer Norm	1.66	05.95	0.38	0.25
No Reward Gradients	1.68	06.18	0.37	0.24
No Discrete Latents	1.08	03.71	0.24	0.19
No KL Balancing	0.84	03.49	0.19	0.16
No Policy Reinforce	0.69	02.74	0.16	0.15
No Image Gradients	0.04	00.31	0.01	0.01

Table 2: Ablations to DreamerV2 measured by their Atari performance at 200M frames, sorted by the last column. The this experiment uses a slightly earlier version of DreamerV2 compared to Table 1. Each ablation only removes one part of the DreamerV2 agent. Discrete latent variables and KL balancing substantially contribute to the success of DreamerV2. Moreover, the world model relies on image gradients to learn general representations that lead to successful behaviors, even if the representations are not specifically learned for predicting past rewards.

智能体	玩家中位数	玩家均值	纪录均值	裁剪纪录均值
DreamerV2	1.64	11.33	0.36	0.25
无层归一化	1.66	05.95	0.38	0.25
无奖励梯度	1.68	06.18	0.37	0.24
无离散潜变量	1.08	03.71	0.24	0.19
无 KL 平衡	0.84	03.49	0.19	0.16
无策略 Reinforce	0.69	02.74	0.16	0.15
无图像梯度	0.04	00.31	0.01	0.01

表 2：以 2 亿帧 Atari 表现衡量的 DreamerV2 消融，按最后一列排序。本实验使用相较表 1 稍早的 DreamerV2 版本。每次消融只去掉 DreamerV2 的一部分。离散潜变量与 KL 平衡对成功贡献很大。此外，世界模型依赖图像梯度学习通向成功行为的一般表示，即使这些表示并非专门为预测过往奖励而学。

To understand which ingredients of DreamerV2 are responsible for its success, we conduct an extensive ablation study. We compare equipping the world model with categorical latents, as in DreamerV2, to Gaussian latents, as in DreamerV1. Moreover, we study the importance of KL balancing. Finally, we investigate the importance of gradients from image reconstruction and reward prediction for learning the model representations, by stopping one of the two gradient signals before entering the model states. The results of the ablation study are summarized in Figure 5 and Table 2. Refer to the appendix for the score curves of the individual tasks.

为理解 DreamerV2 成功的关键要素，我们做了广泛消融。比较世界模型配备 DreamerV2 式类别潜变量与 DreamerV1 式高斯潜变量；研究 KL 平衡的重要性；并通过在进入模型状态前停止二者之一的梯度信号，考察图像重建与奖励预测梯度对学习模型表示的重要性。消融结果概括见图 5 与表 2。各任务分数曲线见附录。

Categorical latents

Categorical latent variables outperform than Gaussian latent variables on 42 tasks, achieve lower performance on 8 tasks, and are tied on 5 tasks. We define a tie as being within $5\%$ of another. While we do not know the reason why the categorical variables are beneficial, we state several hypotheses that can be investigated in future work:

类别潜变量

类别潜变量在 42 个任务上优于高斯潜变量，在 8 个任务上更差，5 个任务持平。我们将差距在 $5\%$ 以内视为持平。虽不确定类别变量有益的原因，我们提出若干可供未来工作检验的假说：

• • A categorical prior can perfectly fit the aggregate posterior, because a mixture of categoricals is again a categorical. In contrast, a Gaussian prior cannot match a mixture of Gaussian posteriors, which could make it difficult to predict multi-modal changes between one image and the next.

• • 类别先验可完美拟合聚合后验，因为类别分布的混合仍是类别分布。相比之下，高斯先验无法匹配高斯后验的混合，可能难以预测相邻图像之间的多峰变化。

• • The level of sparsity enforced by a vector of categorical latent variables could be beneficial for generalization. Flattening the sample from the 32 categorical with 32 classes each results in a sparse binary vector of length 1024 with 32 active bits.

• • 由类别潜变量向量施加的稀疏程度可能有利于泛化。将 32 个各含 32 类的类别样本展平，得到长度 1024、含 32 个激活比特的稀疏二值向量。

• • Despite common intuition, categorical variables may be easier to optimize than Gaussian variables, possibly because the straight-through gradient estimator ignores a term that would otherwise scale the gradient. This could reduce exploding and vanishing gradients.

• • 尽管常见直觉相反，类别变量可能比高斯变量更易优化，或许因为直通梯度估计器忽略了本会缩放梯度的一项，从而减轻梯度爆炸与消失。

• • Categorical variables could be a better inductive bias than unimodal continuous latent variables for modeling the non-smooth aspects of Atari games, such as when entering a new room, or when collected items or defeated enemies disappear from the image.

• • 相对单峰连续潜变量，类别变量可能是对 Atari 非平滑方面更好的归纳偏置，例如进入新房间，或收集物与击败的敌人从画面消失时。

KL balancing

KL balancing outperforms the standard KL regularizer on 44 tasks, achieves lower performance on 6 tasks, and is tied on 5 tasks. Learning accurate prior dynamics of the world model is critical because it is used for imagining latent state trajectories using policy optimization. By scaling up the prior cross entropy relative to the posterior entropy, the world model is encouraged to minimize the KL by improving its prior dynamics toward the more informed posteriors, as opposed to reducing the KL by increasing the posterior entropy. KL balancing may also be beneficial for probabilistic models with learned priors beyond world models.

KL 平衡

KL 平衡在 44 个任务上优于标准 KL 正则，6 个任务更差，5 个持平。学到准确的世界模型先验动态至关重要，因为要用它通过策略优化想象潜状态轨迹。相对后验熵放大先验交叉熵，会鼓励世界模型通过把先验动态改进到更知情的后验来减小 KL，而不是靠抬高后验熵减小 KL。KL 平衡也可能有益于世界模型之外、带可学习先验的概率模型。

Model gradients

Stopping the image gradients increases performance on 3 tasks, decreases performance on 51 tasks, and is tied on 1 task. The world model of DreamerV2 thus heavily relies on the learning signal provided by the high-dimensional images. Stopping the reward gradients increases performance on 15 tasks, decreases performance on 22 tasks, and is tied on 18 tasks. Figure H.1 further shows that the difference in scores is small. In contrast to MuZero, DreamerV2 thus learns general representations of the environment state from image information alone. Stopping reward gradients improved performance on a number of tasks, suggesting that the representations that are not specific to previously experienced rewards may generalize better to unseen situations.

模型梯度

停止图像梯度在 3 个任务上提升、51 个任务下降、1 个持平。因此 DreamerV2 的世界模型严重依赖高维图像提供的学习信号。停止奖励梯度在 15 个任务上提升、22 个下降、18 个持平。图 H.1 进一步显示分数差距较小。与 MuZero 不同，DreamerV2 因而仅从图像信息学习环境状态的一般表示。停止奖励梯度在若干任务上提升表现，说明不特化于已见奖励的表示可能对未见情境泛化更好。

Policy gradients

Using only Reinforce gradients to optimize the policy increases performance on 18 tasks, decreases performance on 24 tasks, and is tied on 13 tasks. This shows that DreamerV2 relies mostly on Reinforce gradients to learn the policy. However, mixing Reinforce and straight-through gradients yields a substantial improvement on James Bond and Seaquest, leading to a higher gamer normalized task mean score. Using only straight-through gradients to optimize the policy increases performance on 5 tasks, decreases performance on 44 tasks, and is tied on 6 tasks. We conjecture that straight-through gradients alone are not well suited for policy optimization because of their bias.

策略梯度

仅用 Reinforce 梯度优化策略在 18 个任务上提升、24 个下降、13 个持平，说明 DreamerV2 主要依赖 Reinforce 学习策略。但混合 Reinforce 与直通梯度在 James Bond 与 Seaquest 上带来显著改进，从而提高玩家归一化任务均值。仅用直通梯度在 5 个任务上提升、44 个下降、6 个持平。我们猜测仅靠直通梯度不适合策略优化，因其有偏。

4 Related Work

4 相关工作

Model-free Atari

The majority of agents applied to the Atari benchmark have been trained using model-free algorithms. DQN (Mnih et al., 2015) showed that deep neural network policies can be trained using Q-learning by incorporating experience replay and target networks. Several works have extended DQN to incorporate bias correction as in DDQN (Van Hasselt et al., 2015), prioritized experience replay (Schaul et al., 2015), architectural improvements (Wang et al., 2016), and distributional value learning (Bellemare et al., 2017; Dabney et al., 2017; Dabney et al., 2018). Besides value learning, agents based on policy gradients have targeted the Atari benchmark, such as ACER (Schulman et al., 2017a), PPO (Schulman et al., 2017a), ACKTR (Wu et al., 2017), and Reactor (Gruslys et al., 2017). Another line of work has focused on improving performance by distributing data collection, often while increasing the budget of environment steps beyond 200M (Mnih et al., 2016; Schulman et al., 2017b; Horgan et al., 2018; Kapturowski et al., 2018; Badia et al., 2020).

无模型 Atari

应用于 Atari 基准的多数智能体用无模型算法训练。DQN（Mnih et al., 2015）表明通过经验回放与目标网络，可用 Q 学习训练深度神经网络策略。若干工作扩展 DQN，纳入如 DDQN 的偏差校正（Van Hasselt et al., 2015）、优先经验回放（Schaul et al., 2015）、架构改进（Wang et al., 2016）以及分布价值学习（Bellemare et al., 2017；Dabney et al., 2017；Dabney et al., 2018）。除价值学习外，基于策略梯度的智能体也面向 Atari，如 ACER（Schulman et al., 2017a）、PPO（Schulman et al., 2017a）、ACKTR（Wu et al., 2017）与 Reactor（Gruslys et al., 2017）。另一路线通过分布式数据收集提升表现，常把环境步预算提高到超过 2 亿（Mnih et al., 2016；Schulman et al., 2017b；Horgan et al., 2018；Kapturowski et al., 2018；Badia et al., 2020）。

World models

Several model-based agents focus on proprioceptive inputs (Watter et al., 2015; Gal et al., 2016; Higuera et al., 2018; Henaff et al., 2018; Chua et al., 2018; Wang et al., 2019; Wang and Ba, 2019), model images without using them for planning (Oh et al., 2015; Krishnan et al., 2015; Karl et al., 2016; Chiappa et al., 2017; Babaeizadeh et al., 2017; Gemici et al., 2017; Denton and Fergus, 2018; Buesing et al., 2018; Doerr et al., 2018; Gregor and Besse, 2018), or combine the benefits of model-based and model-free approaches (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 2018; Lee et al., 2019). Risi and Stanley (2019) optimize discrete latents using evolutionary search. Parmas et al. (2019) combine reinforce and reparameterization gradients. Most world model agents with image inputs have thus far been limited to relatively simple control tasks (Watter et al., 2015; Ebert et al., 2017; Ha and Schmidhuber, 2018; Hafner et al., 2018; Zhang et al., 2019; Hafner et al., 2019). We explain the two model-based approaches that were applied to Atari in detail below.

世界模型

若干基于模型的智能体关注本体感受输入（Watter et al., 2015；Gal et al., 2016；Higuera et al., 2018；Henaff et al., 2018；Chua et al., 2018；Wang et al., 2019；Wang and Ba, 2019），建模图像但不用于规划（Oh et al., 2015；Krishnan et al., 2015；Karl et al., 2016；Chiappa et al., 2017；Babaeizadeh et al., 2017；Gemici et al., 2017；Denton and Fergus, 2018；Buesing et al., 2018；Doerr et al., 2018；Gregor and Besse, 2018），或结合基于模型与无模型方法的优点（Kalweit and Boedecker, 2017；Nagabandi et al., 2017；Weber et al., 2017；Kurutach et al., 2018；Buckman et al., 2018；Ha and Schmidhuber, 2018；Wayne et al., 2018；Igl et al., 2018；Srinivas et al., 2018；Lee et al., 2019）。Risi and Stanley（2019）用进化搜索优化离散潜变量。Parmas et al.（2019）结合 reinforce 与重参数化梯度。带图像输入的多数世界模型智能体迄今限于相对简单的控制任务（Watter et al., 2015；Ebert et al., 2017；Ha and Schmidhuber, 2018；Hafner et al., 2018；Zhang et al., 2019；Hafner et al., 2019）。下面详细说明两种应用于 Atari 的基于模型方法。

Algorithm	Reward Modeling	Image Modeling	Latent Transitions	Single GPU	Trainable Parameters	Atari Frames	Accelerator Days
DreamerV2	✓	✓	✓	✓	22M	200M	10
SimPLe	✓	✓	✗	✓	74M	4M	40
MuZero	✓	✗	✓	✗	40M	20B	80
MuZero Reanalyze	✓	✗	✓	✗	40M	200M	80

Table 3: Conceptual comparison of recent RL algorithms that leverage planning with a learned model. DreamerV2 and SimPLe learn complete models of the environment by leveraging the learning signal provided by the image inputs, while MuZero learns its model through value gradients that are specific to an individual task. The Monte-Carlo tree search used by MuZero is effective but adds complexity and is challenging to parallelize. This component is orthogonal to the world model proposed here.

算法	奖励建模	图像建模	潜转移	单 GPU	可训练参数	Atari 帧数	加速器天数
DreamerV2	✓	✓	✓	✓	22M	200M	10
SimPLe	✓	✓	✗	✓	74M	4M	40
MuZero	✓	✗	✓	✗	40M	20B	80
MuZero Reanalyze	✓	✗	✓	✗	40M	200M	80

表 3：利用所学模型做规划的近期 RL 算法概念比较。DreamerV2 与 SimPLe 借助图像输入的学习信号学习完整环境模型，而 MuZero 通过针对单个任务的价值梯度学习其模型。MuZero 使用的蒙特卡洛树搜索有效，但增加复杂性且难以并行。该组件与本文提出的世界模型正交。

SimPLe

The SimPLe agent (Kaiser et al., 2019) learns a video prediction model in pixel-space and uses its predictions to train a PPO agent (Schulman et al., 2017a), as shown in Table 3. The model directly predicts each frame from the previous four frames and receives an additional discrete latent variable as input. The authors evaluate SimPLe on a subset of Atari games for 400k and 2M environment steps, after which they report diminishing returns. Some recent model-free methods have followed the comparison at 400k steps (Srinivas et al., 2020; Kostrikov et al., 2020). However, the highest performance achieved in this data-efficient regime is a gamer normalized median score of 0.28 (Kostrikov et al., 2020) that is far from human-level performance. Instead, we focus on the well-established and competitive evaluation after 200M frames, where many successful model-free algorithms are available for comparison.

SimPLe

SimPLe 智能体（Kaiser et al., 2019）在像素空间学习视频预测模型，并用其预测训练 PPO 智能体（Schulman et al., 2017a），见表 3。模型直接由前四帧预测每一帧，并额外接收离散潜变量作为输入。作者在 Atari 子集上以 40 万与 200 万环境步评估 SimPLe，并报告此后收益递减。一些近期无模型方法沿用 40 万步比较（Srinivas et al., 2020；Kostrikov et al., 2020）。但该数据高效体制下最高表现是玩家归一化中位 0.28（Kostrikov et al., 2020），远未达到人类水平。我们转而关注成熟且竞争激烈的 2 亿帧评估，此处有许多成功的无模型算法可供比较。

MuZero

The MuZero agent (Schrittwieser et al., 2019) learns a sequence model of rewards and values (Oh et al., 2017) to solve reinforcement learning tasks via Monte-Carlo Tree Search (Coulom, 2006; Silver et al., 2017, MCTS;). The sequence model is trained purely by predicting task-specific information and does not incorporate explicit representation learning using the images, as shown in Table 3. MuZero shows that with significant engineering effort and a vast computational budget, planning can achieve impressive performance on several board games and deterministic Atari games. However, MuZero is not publicly available, and it would require over 2 months to train an Atari agent on one GPU. By comparison, DreamerV2 is a simple algorithm that achieves human-level performance on Atari on a single GPU in 10 days, making it reproducible for many researchers. Moreover, the advanced planning components of MuZero are complementary and could be applied to the accurate world models learned by DreamerV2. DreamerV2 leverages the additional learning signal provided by the input images, analogous to recent successes by semi-supervised image classification (Chen et al., 2020; He et al., 2020; Grill et al., 2020).

MuZero

MuZero 智能体（Schrittwieser et al., 2019）学习奖励与价值的序列模型（Oh et al., 2017），通过蒙特卡洛树搜索（Coulom, 2006；Silver et al., 2017，MCTS）解决强化学习任务。该序列模型纯靠预测任务特定信息训练，不借助图像做显式表示学习，见表 3。MuZero 表明，在大量工程与巨大算力下，规划可在多种棋类与确定性 Atari 上取得出色表现。但 MuZero 未公开，在单 GPU 上训练一个 Atari 智能体需逾两个月。相较之下，DreamerV2 是简单算法，在单 GPU 上约 10 天即可在 Atari 达到人类水平，便于许多研究者复现。此外，MuZero 的高级规划组件与 DreamerV2 学到的准确世界模型互补。DreamerV2 利用输入图像提供的额外学习信号，类似半监督图像分类的近期成功（Chen et al., 2020；He et al., 2020；Grill et al., 2020）。

5 Discussion

5 讨论

We present DreamerV2, a model-based agent that achieves human-level performance on the Atari 200M benchmark by learning behaviors purely from the latent-space predictions of a separately trained world model. Using a single GPU and a single environment instance, DreamerV2 outperforms top model-free single-GPU agents Rainbow and IQN using the same computational budget and training time. To develop DreamerV2, we apply several small modifications to the Dreamer agent (Hafner et al., 2019). We confirm experimentally that learning a categorical latent space and using KL balancing improves the performance of the agent. Moreover, we find the DreamerV2 relies on image information for learning generally useful representations — its performance is not impacted by whether the representations are especially learned for predicting rewards.

我们提出 DreamerV2：一种基于模型的智能体，通过完全从单独训练的世界模型的潜空间预测学习行为，在 Atari 2 亿帧基准上达到人类水平。使用单 GPU 与单个环境实例，DreamerV2 在相同计算预算与训练时间下超过顶级无模型单 GPU 智能体 Rainbow 与 IQN。为发展 DreamerV2，我们对 Dreamer（Hafner et al., 2019）做了若干小改动。实验证实学习类别潜空间与使用 KL 平衡提升了智能体表现。此外发现 DreamerV2 依赖图像信息学习一般有用的表示——表现不受表示是否专门为预测奖励而学的影响。

DreamerV2 serves as proof of concept, showing that model-based RL can outperform top model-free algorithms on the most competitive RL benchmarks, despite the years of research and engineering effort that modern model-free agents rest upon. Beyond achieving strong performance on individual tasks, world models open avenues for efficient transfer and multi-task learning, sample-efficient learning on physical robots, and global exploration based on uncertainty estimates.

DreamerV2 作为概念验证表明：尽管现代无模型智能体建立在多年研究与工程之上，基于模型的 RL 仍可在最具竞争力的 RL 基准上超过顶级无模型算法。除在单个任务上取得强表现外，世界模型还为高效迁移与多任务学习、物理机器人上的样本高效学习，以及基于不确定性估计的全局探索开辟道路。

Acknowledgements

We thank our anonymous reviewers for their feedback and Nick Rhinehart for an insightful discussion about the potential benefits of categorical latent variables.

致谢

感谢匿名审稿人的反馈，以及 Nick Rhinehart 关于类别潜变量潜在好处的深入讨论。

References

参考文献

Babaeizadeh et al. (2017) M Babaeizadeh, C Finn, D Erhan, RH Campbell, S Levine. Stochastic Variational Video Prediction. ArXiv Preprint ArXiv:1710.11252 , 2017.

Badia et al. (2020) AP Badia, B Piot, S Kapturowski, P Sprechmann, A Vitvitskyi, D Guo, C Blundell. Agent57: Outperforming the Atari Human Benchmark. ArXiv Preprint ArXiv:2003.13350 , 2020.

Bellemare et al. (2013) MG Bellemare, Y Naddaf, J Veness, M Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research , 47, 2013.

Bellemare et al. (2017) MG Bellemare, W Dabney, R Munos. A Distributional Perspective on Reinforcement Learning. ArXiv Preprint ArXiv:1707.06887 , 2017.

Bengio et al. (2013) Y Bengio, N Léonard, A Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. ArXiv Preprint ArXiv:1308.3432 , 2013.

Brockman et al. (2016) G Brockman, V Cheung, L Pettersson, J Schneider, J Schulman, J Tang, W Zaremba. Openai Gym, 2016.

Buckman et al. (2018) J Buckman, D Hafner, G Tucker, E Brevdo, H Lee. Sample-Efficient Reinforcement Learning With Stochastic Ensemble Value Expansion. Advances in Neural Information Processing Systems , 2018.

Buesing et al. (2018) L Buesing, T Weber, S Racaniere, S Eslami, D Rezende, DP Reichert, F Viola, F Besse, K Gregor, D Hassabis, et al. Learning and Querying Fast Generative Models for Reinforcement Learning. ArXiv Preprint ArXiv:1802.03006 , 2018.

Byravan et al. (2019) A Byravan, JT Springenberg, A Abdolmaleki, R Hafner, M Neunert, T Lampe, N Siegel, N Heess, M Riedmiller. Imagined Value Gradients: Model-Based Policy Optimization With Transferable Latent Dynamics Models. ArXiv Preprint ArXiv:1910.04142 , 2019.

Castro et al. (2018) PS Castro, S Moitra, C Gelada, S Kumar, MG Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. ArXiv Preprint ArXiv:1812.06110 , 2018.

Chen et al. (2020) T Chen, S Kornblith, M Norouzi, G Hinton. A Simple Framework for Contrastive Learning of Visual Representations. ArXiv Preprint ArXiv:2002.05709 , 2020.

Chiappa et al. (2017) S Chiappa, S Racaniere, D Wierstra, S Mohamed. Recurrent Environment Simulators. ArXiv Preprint ArXiv:1704.02254 , 2017.

Cho et al. (2014) K Cho, B Van Merriënboer, C Gulcehre, D Bahdanau, F Bougares, H Schwenk, Y Bengio. Learning Phrase Representations Using Rnn Encoder-Decoder for Statistical Machine Translation. ArXiv Preprint ArXiv:1406.1078 , 2014.

Chua et al. (2018) K Chua, R Calandra, R McAllister, S Levine. Deep Reinforcement Learning in a Handful of Trials Using Probabilistic Dynamics Models. Advances in Neural Information Processing Systems , 2018.

Clevert et al. (2015) DA Clevert, T Unterthiner, S Hochreiter. Fast and Accurate Deep Network Learning by Exponential Linear Units (Elus). ArXiv Preprint ArXiv:1511.07289 , 2015.

Coulom (2006) R Coulom. Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. International Conference on Computers and Games . Springer, 2006.

Dabney et al. (2017) W Dabney, M Rowland, MG Bellemare, R Munos. Distributional Reinforcement Learning With Quantile Regression. ArXiv Preprint ArXiv:1710.10044 , 2017.

Dabney et al. (2018) W Dabney, G Ostrovski, D Silver, R Munos. Implicit Quantile Networks for Distributional Reinforcement Learning. ArXiv Preprint ArXiv:1806.06923 , 2018.

Denton and Fergus (2018) E Denton R Fergus. Stochastic Video Generation With a Learned Prior. ArXiv Preprint ArXiv:1802.07687 , 2018.

Doerr et al. (2018) A Doerr, C Daniel, M Schiegg, D Nguyen-Tuong, S Schaal, M Toussaint, S Trimpe. Probabilistic Recurrent State-Space Models. ArXiv Preprint ArXiv:1801.10395 , 2018.

Ebert et al. (2017) F Ebert, C Finn, AX Lee, S Levine. Self-Supervised Visual Planning With Temporal Skip Connections. ArXiv Preprint ArXiv:1710.05268 , 2017.

Fortunato et al. (2017) M Fortunato, MG Azar, B Piot, J Menick, I Osband, A Graves, V Mnih, R Munos, D Hassabis, O Pietquin, et al. Noisy Networks for Exploration. ArXiv Preprint ArXiv:1706.10295 , 2017.

Gal et al. (2016) Y Gal, R McAllister, CE Rasmussen. Improving Pilco With Bayesian Neural Network Dynamics Models. Data-Efficient Machine Learning Workshop, ICML , 2016.

Gemici et al. (2017) M Gemici, CC Hung, A Santoro, G Wayne, S Mohamed, DJ Rezende, D Amos, T Lillicrap. Generative Temporal Models With Memory. ArXiv Preprint ArXiv:1702.04649 , 2017.

Gregor and Besse (2018) K Gregor F Besse. Temporal Difference Variational Auto-Encoder. ArXiv Preprint ArXiv:1806.03107 , 2018.

Grill et al. (2020) JB Grill, F Strub, F Altché, C Tallec, PH Richemond, E Buchatskaya, C Doersch, BA Pires, ZD Guo, MG Azar, et al. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. ArXiv Preprint ArXiv:2006.07733 , 2020.

Gruslys et al. (2017) A Gruslys, W Dabney, MG Azar, B Piot, M Bellemare, R Munos. The Reactor: A Fast and Sample-Efficient Actor-Critic Agent for Reinforcement Learning. ArXiv Preprint ArXiv:1704.04651 , 2017.

Ha and Schmidhuber (2018) D Ha J Schmidhuber. World Models. ArXiv Preprint ArXiv:1803.10122 , 2018.

Hafner et al. (2018) D Hafner, T Lillicrap, I Fischer, R Villegas, D Ha, H Lee, J Davidson. Learning Latent Dynamics for Planning From Pixels. ArXiv Preprint ArXiv:1811.04551 , 2018.

Hafner et al. (2019) D Hafner, T Lillicrap, J Ba, M Norouzi. Dream to Control: Learning Behaviors by Latent Imagination. ArXiv Preprint ArXiv:1912.01603 , 2019.

He et al. (2020) K He, H Fan, Y Wu, S Xie, R Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020.

Henaff et al. (2018) M Henaff, WF Whitney, Y LeCun. Model-Based Planning With Discrete and Continuous Actions. ArXiv Preprint ArXiv:1705.07177 , 2018.

Hessel et al. (2018) M Hessel, J Modayil, H Van Hasselt, T Schaul, G Ostrovski, W Dabney, D Horgan, B Piot, M Azar, D Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning. Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Higgins et al. (2016) I Higgins, L Matthey, A Pal, C Burgess, X Glorot, M Botvinick, S Mohamed, A Lerchner. Beta-Vae: Learning Basic Visual Concepts With a Constrained Variational Framework. International Conference on Learning Representations , 2016.

Higuera et al. (2018) JCG Higuera, D Meger, G Dudek. Synthesizing Neural Network Controllers With Probabilistic Model Based Reinforcement Learning. ArXiv Preprint ArXiv:1803.02291 , 2018.

Horgan et al. (2018) D Horgan, J Quan, D Budden, G Barth-Maron, M Hessel, H Van Hasselt, D Silver. Distributed Prioritized Experience Replay. ArXiv Preprint ArXiv:1803.00933 , 2018.

Igl et al. (2018) M Igl, L Zintgraf, TA Le, F Wood, S Whiteson. Deep Variational Reinforcement Learning for Pomdps. ArXiv Preprint ArXiv:1806.02426 , 2018.

Kaiser et al. (2019) L Kaiser, M Babaeizadeh, P Milos, B Osinski, RH Campbell, K Czechowski, D Erhan, C Finn, P Kozakowski, S Levine, et al. Model-Based Reinforcement Learning for Atari. ArXiv Preprint ArXiv:1903.00374 , 2019.

Kalweit and Boedecker (2017) G Kalweit J Boedecker. Uncertainty-Driven Imagination for Continuous Deep Reinforcement Learning. Conference on Robot Learning , 2017.

Kapturowski et al. (2018) S Kapturowski, G Ostrovski, J Quan, R Munos, W Dabney. Recurrent Experience Replay in Distributed Reinforcement Learning. International Conference on Learning Representations , 2018.

Karl et al. (2016) M Karl, M Soelch, J Bayer, P van der Smagt. Deep Variational Bayes Filters: Unsupervised Learning of State Space Models From Raw Data. ArXiv Preprint ArXiv:1605.06432 , 2016.

Kingma and Ba (2014) DP Kingma J Ba. Adam: A Method for Stochastic Optimization. ArXiv Preprint ArXiv:1412.6980 , 2014.

Kingma and Welling (2013) DP Kingma M Welling. Auto-Encoding Variational Bayes. ArXiv Preprint ArXiv:1312.6114 , 2013.

Kostrikov et al. (2020) I Kostrikov, D Yarats, R Fergus. Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning From Pixels. ArXiv Preprint ArXiv:2004.13649 , 2020.

Krishnan et al. (2015) RG Krishnan, U Shalit, D Sontag. Deep Kalman Filters. ArXiv Preprint ArXiv:1511.05121 , 2015.

Kurutach et al. (2018) T Kurutach, I Clavera, Y Duan, A Tamar, P Abbeel. Model-Ensemble Trust-Region Policy Optimization. ArXiv Preprint ArXiv:1802.10592 , 2018.

LeCun et al. (1989) Y LeCun, B Boser, JS Denker, D Henderson, RE Howard, W Hubbard, LD Jackel. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation , 1(4), 1989.

Lee et al. (2019) AX Lee, A Nagabandi, P Abbeel, S Levine. Stochastic Latent Actor-Critic: Deep Reinforcement Learning With a Latent Variable Model. ArXiv Preprint ArXiv:1907.00953 , 2019.

Machado et al. (2018) MC Machado, MG Bellemare, E Talvitie, J Veness, M Hausknecht, M Bowling. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. Journal of Artificial Intelligence Research , 61, 2018.

Mnih et al. (2015) V Mnih, K Kavukcuoglu, D Silver, AA Rusu, J Veness, MG Bellemare, A Graves, M Riedmiller, AK Fidjeland, G Ostrovski, et al. Human-Level Control Through Deep Reinforcement Learning. Nature , 518(7540), 2015.

Mnih et al. (2016) V Mnih, AP Badia, M Mirza, A Graves, T Lillicrap, T Harley, D Silver, K Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. International Conference on Machine Learning , 2016.

Nagabandi et al. (2017) A Nagabandi, G Kahn, RS Fearing, S Levine. Neural Network Dynamics for Model-Based Deep Reinforcement Learning With Model-Free Fine-Tuning. ArXiv Preprint ArXiv:1708.02596 , 2017.

Oh et al. (2015) J Oh, X Guo, H Lee, RL Lewis, S Singh. Action-Conditional Video Prediction Using Deep Networks in Atari Games. Advances in Neural Information Processing Systems , 2015.

Oh et al. (2017) J Oh, S Singh, H Lee. Value Prediction Network. Advances in Neural Information Processing Systems , 2017.

Parmas et al. (2019) P Parmas, CE Rasmussen, J Peters, K Doya. Pipps: Flexible Model-Based Policy Search Robust to the Curse of Chaos. ArXiv Preprint ArXiv:1902.01240 , 2019.

Pathak et al. (2017) D Pathak, P Agrawal, AA Efros, T Darrell. Curiosity-Driven Exploration by Self-Supervised Prediction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , 2017.

Rezende et al. (2014) DJ Rezende, S Mohamed, D Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. ArXiv Preprint ArXiv:1401.4082 , 2014.

Risi and Stanley (2019) S Risi KO Stanley. Deep Neuroevolution of Recurrent and Discrete World Models. Proceedings of the Genetic and Evolutionary Computation Conference , 2019.

Schaul et al. (2015) T Schaul, J Quan, I Antonoglou, D Silver. Prioritized Experience Replay. ArXiv Preprint ArXiv:1511.05952 , 2015.

Schrittwieser et al. (2019) J Schrittwieser, I Antonoglou, T Hubert, K Simonyan, L Sifre, S Schmitt, A Guez, E Lockhart, D Hassabis, T Graepel, et al. Mastering Atari, Go, Chess and Shogi by Planning With a Learned Model. ArXiv Preprint ArXiv:1911.08265 , 2019.

Schulman et al. (2015) J Schulman, P Moritz, S Levine, M Jordan, P Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ArXiv Preprint ArXiv:1506.02438 , 2015.

Schulman et al. (2017a) J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov. Proximal Policy Optimization Algorithms. ArXiv Preprint ArXiv:1707.06347 , 2017a.

Schulman et al. (2017b) J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov. Proximal Policy Optimization Algorithms. ArXiv Preprint ArXiv:1707.06347 , 2017b.

Sekar et al. (2020) R Sekar, O Rybkin, K Daniilidis, P Abbeel, D Hafner, D Pathak. Planning to Explore via Self-Supervised World Models. ArXiv Preprint ArXiv:2005.05960 , 2020.

Silver et al. (2017) D Silver, J Schrittwieser, K Simonyan, I Antonoglou, A Huang, A Guez, T Hubert, L Baker, M Lai, A Bolton, et al. Mastering the Game of Go Without Human Knowledge. Nature , 550(7676), 2017.

Srinivas et al. (2018) A Srinivas, A Jabri, P Abbeel, S Levine, C Finn. Universal Planning Networks. ArXiv Preprint ArXiv:1804.00645 , 2018.

Srinivas et al. (2020) A Srinivas, M Laskin, P Abbeel. Curl: Contrastive Unsupervised Representations for Reinforcement Learning. ArXiv Preprint ArXiv:2004.04136 , 2020.

Sutton (1991) RS Sutton. Dyna, an Integrated Architecture for Learning, Planning, and Reacting. ACM SIGART Bulletin , 2(4), 1991.

Sutton and Barto (2018) RS Sutton AG Barto. Reinforcement Learning: An Introduction . MIT press, 2018.

Taiga et al. (2019) AA Taiga, W Fedus, MC Machado, A Courville, MG Bellemare. On Bonus Based Exploration Methods in the Arcade Learning Environment. International Conference on Learning Representations , 2019.

Toromanoff et al. (2019) M Toromanoff, E Wirbel, F Moutarde. Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the Playing Field. ArXiv Preprint ArXiv:1908.04683 , 2019.

Van Hasselt et al. (2015) H Van Hasselt, A Guez, D Silver. Deep Reinforcement Learning With Double Q-Learning. ArXiv Preprint ArXiv:1509.06461 , 2015.

Wang and Ba (2019) T Wang J Ba. Exploring Model-Based Planning With Policy Networks. ArXiv Preprint ArXiv:1906.08649 , 2019.

Wang et al. (2019) T Wang, X Bao, I Clavera, J Hoang, Y Wen, E Langlois, S Zhang, G Zhang, P Abbeel, J Ba. Benchmarking Model-Based Reinforcement Learning. CoRR , abs/1907.02057, 2019.

Wang et al. (2016) Z Wang, T Schaul, M Hessel, H Hasselt, M Lanctot, N Freitas. Dueling Network Architectures for Deep Reinforcement Learning. International Conference on Machine Learning , 2016.

Watter et al. (2015) M Watter, J Springenberg, J Boedecker, M Riedmiller. Embed to Control: A Locally Linear Latent Dynamics Model for Control From Raw Images. Advances in Neural Information Processing Systems , 2015.

Wayne et al. (2018) G Wayne, CC Hung, D Amos, M Mirza, A Ahuja, A Grabska-Barwinska, J Rae, P Mirowski, JZ Leibo, A Santoro, et al. Unsupervised Predictive Memory in a Goal-Directed Agent. ArXiv Preprint ArXiv:1803.10760 , 2018.

Weber et al. (2017) T Weber, S Racanière, DP Reichert, L Buesing, A Guez, DJ Rezende, AP Badia, O Vinyals, N Heess, Y Li, et al. Imagination-Augmented Agents for Deep Reinforcement Learning. ArXiv Preprint ArXiv:1707.06203 , 2017.

Williams (1992) RJ Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning , 8(3-4), 1992.

Wu et al. (2017) Y Wu, E Mansimov, RB Grosse, S Liao, J Ba. Scalable Trust-Region Method for Deep Reinforcement Learning Using Kronecker-Factored Approximation. Advances in Neural Information Processing Systems , 2017.

Yu et al. (2020) T Yu, G Thomas, L Yu, S Ermon, J Zou, S Levine, C Finn, T Ma. Mopo: Model-Based Offline Policy Optimization. ArXiv Preprint ArXiv:2005.13239 , 2020.

Zhang et al. (2019) M Zhang, S Vikram, L Smith, P Abbeel, M Johnson, S Levine. Solar: Deep Structured Representations for Model-Based Reinforcement Learning. International Conference on Machine Learning , 2019.

Appendix A Humanoid from Pixels

附录 A 像素输入下的人形机器人

Figure A.1: Behavior learned by DreamerV2 on the Humanoid Walk task from pixel inputs only. The task is provided by the DeepMind Control Suite and uses a continuous action space with 21 dimensions. The frames show the agent inputs.

图 A.1：DreamerV2 仅从像素输入在 Humanoid Walk 任务上学到的行为。任务来自 DeepMind Control Suite，动作为 21 维连续空间。帧为智能体输入。

Figure A.2: Performance on the humanoid walking task from only pixel inputs.

图 A.2：仅从像素输入在人形行走任务上的表现。

While the main experiments of this paper focus on the Atari benchmark with discrete actions, DreamerV2 is also applicable to control tasks with continuous actions. For this, we the actor outputs a truncated normal distribution instead of a categorical distribution. To demonstrate the abilities of DreamerV2 for continuous control, we choose the challenging humanoid environment with only image inputs, shown in Figure A.1. We find that for continuous control tasks, dynamics backpropagation substantially outperforms reinforce gradients and thus set $\rho=0$. We also set $\eta=10^{-5}$ and $\beta=2$ to further accelerate learning. We find that DreamerV2 reliably solves both the stand-up motion required at the beginning of the episode and the subsequent walking. The score is shown in Figure A.2. To the best of our knowledge, this constitutes the first published result of solving the humanoid environment from only pixel inputs.

尽管本文主实验聚焦离散动作的 Atari 基准，DreamerV2 也适用于连续动作控制任务。为此，演员输出截断正态分布而非类别分布。为展示连续控制能力，我们选择仅图像输入的高难度人形环境，见图 A.1。我们发现对连续控制，动态反向传播明显优于 reinforce，因而设 $\rho=0$。同时设 $\eta=10^{-5}$、$\beta=2$ 以进一步加速学习。DreamerV2 可靠解决回合开始所需的起立动作以及随后的行走。分数见图 A.2。据我们所知，这是首个公开发表的、仅从像素输入解决人形环境的结果。

Appendix B Montezuma’s Revenge

附录 B Montezuma’s Revenge

Figure B.1: Behavior learned by DreamerV2 on the Atari game Montezuma’s Revenge, that poses a hard exploration challenge. Without any explicit exploration mechanism, DreamerV2 reaches about the same performance as the exploration method ICM.

图 B.1：DreamerV2 在具有高难度探索挑战的 Atari 游戏 Montezuma’s Revenge 上学到的行为。在没有任何显式探索机制的情况下，DreamerV2 达到与探索方法 ICM 大致相当的表现。

Figure B.2: Performance on the Atari game Montezuma’s Revenge.

图 B.2：在 Atari 游戏 Montezuma’s Revenge 上的表现。

While our main experiments use the same hyper parameters across all tasks, we find that DreamerV2 achieves higher performance on Montezuma’s Revenge by using a lower discount factor of $\gamma=0.99$, possibly to stabilize value learning under sparse rewards. Figure B.2 shows the resulting performance, with all other hyper parameters left at their defaults. DreamerV2 outperforms existing model-free approaches on the hard-exploration game Montezuma’s Revenge and matches the performance of the explicit exploration algorithm ICM (Pathak et al., 2017) that was applied on top of Rainbow by Taiga et al. (2019). This suggests that the world model may help with solving sparse reward tasks, for example due to improved generalization, efficient policy optimization in the compact latent space enabling more actor critic updates, or because the reward predictor generalizes and thus smooths out the sparse rewards.

尽管主实验在所有任务上使用相同超参，我们发现 DreamerV2 在 Montezuma’s Revenge 上通过使用更低的折扣因子 $\gamma=0.99$ 可获得更高表现，可能有助于在稀疏奖励下稳定价值学习。图 B.2 给出结果，其余超参保持默认。DreamerV2 在该高难度探索游戏上超过已有无模型方法，并与 Taiga et al.（2019）叠在 Rainbow 上的显式探索算法 ICM（Pathak et al., 2017）表现相当。这表明世界模型可能有助于稀疏奖励任务，例如由于更好的泛化、在紧凑潜空间中高效的策略优化带来更多演员–评论家更新，或因为奖励预测器泛化从而平滑稀疏奖励。

Appendix C Summary of Modifications

附录 C 改动摘要

To develop DreamerV2, we used the Dreamer agent (Hafner et al., 2019) as a starting point. This subsection describes the changes that we applied to the agent to achieve high performance on the Atari benchmark, as well as the changes that were tried but not found to increase performance and thus were not not included in DreamerV2.

为发展 DreamerV2，我们以 Dreamer（Hafner et al., 2019）为起点。本小节描述为在 Atari 基准上取得高表现而对智能体所做的改动，以及尝试过但未发现显著提升、因而未纳入 DreamerV2 的改动。

Summary of changes that were tried and were found to help:

尝试过并发现有帮助的改动摘要：

• • Categorical latents  Using categorical latent states using straight-through gradients in the world model instead of Gaussian latents with reparameterized gradients.

• • 类别潜变量 在世界模型中使用带直通梯度的类别潜状态，替代带重参数化梯度的高斯潜变量。

• • KL balancing  Separately scaling the prior cross entropy and the posterior entropy in the KL loss to encourage learning an accurate temporal prior, instead of using free nats.

• • KL 平衡 在 KL 损失中分别缩放先验交叉熵与后验熵，以鼓励学习准确的时间先验，而非使用 free nats。

• • Reinforce only  Reinforce gradients worked substantially better for Atari than dynamics backpropagation. For continuous control, dynamics backpropagation worked substantially better.

• • 仅 Reinforce 对 Atari，Reinforce 梯度明显优于动态反向传播。对连续控制，动态反向传播明显更好。

• • Model size  Increasing the number of units or feature maps per layer of all model components, resulting in a change from 13M parameters to 22M parameters.

• • 模型规模 增加所有模型组件每层的单元数或特征图数，参数量从约 1300 万增至约 2200 万。

• • Policy entropy  Regularizing the policy entropy for exploration both in imagination and during data collection, instead of using external action noise during data collection.

• • 策略熵 在想象与数据采集中对策略熵正则化以鼓励探索，而非在数据采集时使用外部动作噪声。

Summary of changes that were tried but were found to not help substantially:

尝试过但未发现显著帮助的改动摘要：

• • Binary latents  Using a larger number of binary latents for the world model instead of categorical latents, which could have encouraged a more disentangled representation, was worse.

• • 二值潜变量 用更多二值潜变量替代类别潜变量（本可鼓励更解耦的表示），结果更差。

• • Long-term entropy  Including the policy entropy into temporal-difference loss of the value function, so that the actor seeks out states with high action entropy beyond the planning horizon.

• • 长期熵 把策略熵纳入价值函数的时序差分损失，使演员在规划视野之外也寻找高动作熵状态。

• • Mixed actor gradients  Combining Reinforce and dynamics backpropagation gradients for learning the actor instead of Reinforce provided marginal or no benefits.

• • 混合演员梯度 结合 Reinforce 与动态反向传播训练演员，相对纯 Reinforce 收益边际或没有。

• • Scheduling  Scheduling the learning rates, KL scale, actor entropy loss scale, and actor gradient mixing (from 0.1 to 0) provided marginal or no benefits.

• • 调度 调度学习率、KL 尺度、演员熵损失尺度以及演员梯度混合（从 0.1 到 0），收益边际或没有。

• • Layer norm  Using layer normalization in the GRU that is used as part of the RSSM latent transition model, instead of no normalization, provided no or marginal benefits.

• • 层归一化 在作为 RSSM 潜转移模型一部分的 GRU 中使用层归一化，相对无归一化收益没有或边际。

Due to the large computational requirements, a comprehensive ablation study on this list of all changes is unfortunately infeasible for us. This would require 55 tasks times 5 seeds for 10 days per change to run, resulting in over 60,000 GPU hours per change. However, we include ablations for the most important design choices in the main text of the paper.

由于算力需求很大，我们对这份全部改动清单做全面消融不可行：每项改动需 55 任务×5 种子×约 10 天，超过每项 60,000 GPU 小时。但正文已对最重要的设计选择给出消融。

Appendix D Hyper Parameters

附录 D 超参数

Name	Symbol	Value
World Model
Dataset size (FIFO)	—	$2\cdot 10^{6}$
Batch size	$B$	50
Sequence length	$L$	50
Discrete latent dimensions	—	32
Discrete latent classes	—	32
RSSM number of units	—	600
KL loss scale	$\beta$	0.1
KL balancing	$\alpha$	0.8
World model learning rate	—	$2\cdot 10^{-4}$
Reward transformation	—	$\operatorname{tanh}$
Behavior
Imagination horizon	$H$	15
Discount	$\gamma$	0.995
$\lambda$-target parameter	$\lambda$	0.95
Actor gradient mixing	$\rho$	$1$
Actor entropy loss scale	$\eta$	$1\cdot 10^{-3}$
Actor learning rate	—	$4\cdot 10^{-5}$
Critic learning rate	—	$1\cdot 10^{-4}$
Slow critic update interval	—	$100$
Common
Policy steps per gradient step	—	4
MPL number of layers	—	4
MPL number of units	—	400
Gradient clipping	—	100
Adam epsilon	$\epsilon$	$10^{-5}$
Weight decay (decoupled)	—	$10^{-6}$

Table D.1: Atari hyper parameters of DreamerV2. When tuning the agent for a new task, we recommend searching over the KL loss scale $\beta\in\{0.1,0.3,1,3\}$, actor entropy loss scale $\eta\in\{3\cdot 10^{-5},10^{-4},3\cdot 10^{-4},10^{-3}\}$, and the discount factor $\gamma\in\{0.99,0.999\}$. The training frequency update should be increased when aiming for higher data-efficiency.

名称	符号	取值
世界模型
数据集大小（FIFO）	—	$2\cdot 10^{6}$
批大小	$B$	50
序列长度	$L$	50
离散潜维度	—	32
离散潜类别数	—	32
RSSM 单元数	—	600
KL 损失尺度	$\beta$	0.1
KL 平衡	$\alpha$	0.8
世界模型学习率	—	$2\cdot 10^{-4}$
奖励变换	—	$\operatorname{tanh}$
行为
想象视野	$H$	15
折扣	$\gamma$	0.995
$\lambda$-目标参数	$\lambda$	0.95
演员梯度混合	$\rho$	$1$
演员熵损失尺度	$\eta$	$1\cdot 10^{-3}$
演员学习率	—	$4\cdot 10^{-5}$
评论家学习率	—	$1\cdot 10^{-4}$
慢评论家更新间隔	—	$100$
共用
每梯度步策略步数	—	4
MLP 层数	—	4
MLP 单元数	—	400
梯度裁剪	—	100
Adam epsilon	$\epsilon$	$10^{-5}$
权重衰减（解耦）	—	$10^{-6}$

表 D.1：DreamerV2 的 Atari 超参数。为新任务调参时，建议搜索 KL 损失尺度 $\beta\in\{0.1,0.3,1,3\}$、演员熵损失尺度 $\eta\in\{3\cdot 10^{-5},10^{-4},3\cdot 10^{-4},10^{-3}\}$ 以及折扣因子 $\gamma\in\{0.99,0.999\}$。若追求更高数据效率，应提高训练更新频率。

Appendix E Agent Comparison

附录 E 智能体比较

DreamerV2 vs IQN

DreamerV2 vs Rainbow

DreamerV2 vs C51

DreamerV2 vs DQN

Figure E.1: Atari agent comparison. The bars show the difference in gamer normalized scores at 200M steps. DreamerV2 outperforms the four model-free algorithms IQN, Rainbow, C51, and DQN while learning behaviors purely by planning within a separately learned world model. DreamerV2 achieves higher or similar performance on all tasks besides Video Pinball, where we hypothesize that the reconstruction loss does not focus on the ball that makes up only one pixel on the screen.

DreamerV2 vs IQN

DreamerV2 vs Rainbow

DreamerV2 vs C51

DreamerV2 vs DQN

图 E.1：Atari 智能体比较。柱状图显示 2 亿步时玩家归一化分数之差。DreamerV2 在完全通过单独学习的世界模型内规划来学习行为的同时，超过 IQN、Rainbow、C51 与 DQN 四种无模型算法。除 Video Pinball 外，DreamerV2 在所有任务上达到更高或相近表现；我们猜测重建损失未聚焦于屏幕上仅占一个像素的球。

Appendix F Model-Free Comparison

附录 F 与无模型方法比较

Figure F.1: Comparison of DreamerV2 to the top model-free RL methods IQN and Rainbow. The curves show mean and standard deviation over 5 seeds. IQN and Rainbow additionally average each point over 10 evaluation episodes, explaining the smoother curves. DreamerV2 outperforms IQN and Rainbow in all four aggregated scores. While IQN and Rainbow tend to succeed on the same tasks, DreamerV2 shows a different performance profile.

图 F.1：DreamerV2 与顶级无模型 RL 方法 IQN、Rainbow 的比较。曲线为 5 个种子的均值与标准差。IQN 与 Rainbow 每个点还对 10 次评估回合取平均，因而曲线更平滑。DreamerV2 在全部四种聚合分数上超过 IQN 与 Rainbow。IQN 与 Rainbow 往往在相同任务上成功，而 DreamerV2 呈现不同的表现轮廓。

Appendix G Latents and KL Balancing Ablations

附录 G 潜变量与 KL 平衡消融

Figure G.1: Comparison of DreamerV2, Gaussian instead of categorical latent variables, and no KL balancing. The ablation experiments use a slightly earlier version of the agent. The curves show mean and standard deviation across two seeds. Categorical latent variables and KL balancing both substantially improve performance across many of the tasks. The importance of the two techniques is reflected in all four aggregated scores.

图 G.1：比较 DreamerV2、用高斯替代类别潜变量，以及无 KL 平衡。消融实验使用稍早版本的智能体。曲线为两个种子的均值与标准差。类别潜变量与 KL 平衡均在许多任务上显著提升表现。这两种技术的重要性反映在全部四种聚合分数上。

Appendix H Representation Learning Ablations

附录 H 表示学习消融

Figure H.1: Comparison of leveraging image prediction, reward prediction, or both for learning the model representations. While image gradients are crucial, reward gradients are not necessary for our world model to succeed and their gradients can be stopped. Representations learned purely from images are not biased toward previously encountered rewards and outperform reward-specific representations on a number of tasks, suggesting that they may generalize better to unseen situations.

图 H.1：比较利用图像预测、奖励预测或二者学习模型表示。图像梯度至关重要，而奖励梯度对世界模型成功并非必要，可以停止。纯从图像学习的表示不偏向先前遇到的奖励，并在若干任务上超过奖励特化表示，说明它们可能对未见情境泛化更好。

Appendix I Policy Learning Ablations

附录 I 策略学习消融

Figure I.1: Comparison of leveraging Reinforce gradients, straight-through gradients, or both for training the actor. While Reinforce gradients are crucial, straight-through gradients are not important for most of the tasks. Nonetheless, combining both gradients yields substantial improvements on a small number of games, most notably on Seaquest. We conjecture that straight-through gradients have low variance and thus help the agent start learning, whereas Reinforce gradients are unbiased and help converging to a better solution.

图 I.1：比较利用 Reinforce 梯度、直通梯度或二者训练演员。Reinforce 梯度至关重要，直通梯度对多数任务不重要。尽管如此，组合两者在少数游戏上带来显著改进，尤其是 Seaquest。我们猜测直通梯度方差低因而有助起步，而 Reinforce 无偏有助收敛到更好解。

Appendix J Additional Ablations

附录 J 额外消融

Figure J.1: Comparison of DreamerV2 to a version without layer norm in the GRU and to training from experience collected over time by a uniform random policy. We find that the benefit of layer norm depends on the task at hand, increasing and decreasing performance on a roughly equal number of tasks. The comparison to random data collection highlights which of the tasks require non-trivial exploration, which can help guide future work on directed exploration using world models.

图 J.1：比较 DreamerV2、GRU 中无层归一化的版本，以及从均匀随机策略随时间采集的经验上训练。我们发现层归一化的益处依赖具体任务，提升与下降的任务数大致相当。与随机数据采集的比较凸显哪些任务需要非平凡探索，可指导今后利用世界模型做有向探索的工作。

Appendix K Atari Task Scores

附录 K Atari 任务分数

	Baselines	Algorithms
Task	Random	Gamer	Record	Rainbow	IQN	DreamerV2
Alien	229	7128	251916	3457	4961	3967
Amidar	6	1720	104159	2529	2393	2577
Assault	222	742	8647	3229	4885	23625
Asterix	210	8503	1000000	18367	10374	72311
Asteroids	719	47389	10506650	1484	1585	41526
Atlantis	12850	29028	10604840	802548	890214	978778
Bank Heist	14	753	82058	1075	1052	1126
Battle Zone	2360	37188	801000	40061	40953	40325
Beam Rider	364	16926	999999	6290	7130	18646
Berzerk	124	2630	1057940	833	648	810
Bowling	23	161	300	43	39	49
Boxing	0	12	100	99	98	92
Breakout	2	30	864	120	79	312
Centipede	2091	12017	1301709	6510	3728	11883
Chopper Command	811	7388	999999	12338	9282	2861
Crazy Climber	10780	35829	219900	145389	132738	161839
Demon Attack	152	1971	1556345	17071	15350	82263
Double Dunk	-19	-16	22	22	21	17
Enduro	0	860	9500	2200	2203	1656
Fishing Derby	-92	-39	71	42	45	65
Freeway	0	30	38	34	34	33
Frostbite	65	4335	454830	8208	7812	11384
Gopher	258	2412	355040	10641	12108	92282
Gravitar	173	3351	162850	1272	1347	3789
Hero	1027	30826	1000000	46675	36058	21868
Ice Hockey	-11	1	36	0	-5	26
James Bond	7	303	45550	1097	3166	40445
Kangaroo	52	3035	1424600	12748	12602	14064
Krull	1598	2666	104100	4066	8844	50061
Kung Fu Master	258	22736	1000000	26475	31653	62741
Montezuma Revenge	0	4753	1219200	500	500	81
Ms Pacman	307	6952	290090	3861	5218	5652
Name This Game	2292	8049	25220	9026	6639	14649
Phoenix	761	7243	4014440	8545	5102	49375
Pitfall	-229	6464	114000	-20	-13	0
Pong	-21	15	21	20	20	20
Private Eye	25	69571	101800	21334	4181	2198
Qbert	164	13455	2400000	17383	16730	94688
Riverraid	1338	17118	1000000	20756	15183	16351
Road Runner	12	7845	2038100	54662	58966	203576
Robotank	2	12	76	66	66	78
Seaquest	68	42055	999999	9903	17039	7480
Skiing	-17098	-4337	-3272	-28708	-11162	-9299
Solaris	1236	12327	111420	1583	1684	922
Space Invaders	148	1669	621535	4131	4530	2474
Star Gunner	664	10250	77400	57909	80003	7800
Tennis	-24	-8	21	0	23	14
Time Pilot	3568	5229	65300	12051	11666	37945
Tutankham	11	168	5384	239	251	264
Up N Down	533	11693	82840	34888	59944	653662
Venture	0	1188	38900	1529	1313	2
Video Pinball	16257	17668	89218328	466895	415833	41860
Wizard Of Wor	564	4756	395300	7879	5671	12851
Yars Revenge	3093	54577	15000105	45542	84144	156748
Zaxxon	32	9173	83700	14603	11023	50699

Table K.1: Atari individual scores. We select the 55 games that are common among most papers in the literature. We compare the algorithms DreamerV2, IQN, and Rainbow to the baselines of random actions, DeepMind’s human gamer, and the human world record. Algorithm scores are highlighted in bold when they fall within 5% of the best algorithm. Note that these scores are already averaged across seeds, whereas any aggregated scores must be computed before averaging across seeds.

	基线	算法
任务	随机	玩家	纪录	Rainbow	IQN	DreamerV2
Alien	229	7128	251916	3457	4961	3967
Amidar	6	1720	104159	2529	2393	2577
Assault	222	742	8647	3229	4885	23625
Asterix	210	8503	1000000	18367	10374	72311
Asteroids	719	47389	10506650	1484	1585	41526
Atlantis	12850	29028	10604840	802548	890214	978778
Bank Heist	14	753	82058	1075	1052	1126
Battle Zone	2360	37188	801000	40061	40953	40325
Beam Rider	364	16926	999999	6290	7130	18646
Berzerk	124	2630	1057940	833	648	810
Bowling	23	161	300	43	39	49
Boxing	0	12	100	99	98	92
Breakout	2	30	864	120	79	312
Centipede	2091	12017	1301709	6510	3728	11883
Chopper Command	811	7388	999999	12338	9282	2861
Crazy Climber	10780	35829	219900	145389	132738	161839
Demon Attack	152	1971	1556345	17071	15350	82263
Double Dunk	-19	-16	22	22	21	17
Enduro	0	860	9500	2200	2203	1656
Fishing Derby	-92	-39	71	42	45	65
Freeway	0	30	38	34	34	33
Frostbite	65	4335	454830	8208	7812	11384
Gopher	258	2412	355040	10641	12108	92282
Gravitar	173	3351	162850	1272	1347	3789
Hero	1027	30826	1000000	46675	36058	21868
Ice Hockey	-11	1	36	0	-5	26
James Bond	7	303	45550	1097	3166	40445
Kangaroo	52	3035	1424600	12748	12602	14064
Krull	1598	2666	104100	4066	8844	50061
Kung Fu Master	258	22736	1000000	26475	31653	62741
Montezuma Revenge	0	4753	1219200	500	500	81
Ms Pacman	307	6952	290090	3861	5218	5652
Name This Game	2292	8049	25220	9026	6639	14649
Phoenix	761	7243	4014440	8545	5102	49375
Pitfall	-229	6464	114000	-20	-13	0
Pong	-21	15	21	20	20	20
Private Eye	25	69571	101800	21334	4181	2198
Qbert	164	13455	2400000	17383	16730	94688
Riverraid	1338	17118	1000000	20756	15183	16351
Road Runner	12	7845	2038100	54662	58966	203576
Robotank	2	12	76	66	66	78
Seaquest	68	42055	999999	9903	17039	7480
Skiing	-17098	-4337	-3272	-28708	-11162	-9299
Solaris	1236	12327	111420	1583	1684	922
Space Invaders	148	1669	621535	4131	4530	2474
Star Gunner	664	10250	77400	57909	80003	7800
Tennis	-24	-8	21	0	23	14
Time Pilot	3568	5229	65300	12051	11666	37945
Tutankham	11	168	5384	239	251	264
Up N Down	533	11693	82840	34888	59944	653662
Venture	0	1188	38900	1529	1313	2
Video Pinball	16257	17668	89218328	466895	415833	41860
Wizard Of Wor	564	4756	395300	7879	5671	12851
Yars Revenge	3093	54577	15000105	45542	84144	156748
Zaxxon	32	9173	83700	14603	11023	50699