Dream to Control: Learning Behaviors by Latent Imagination

Dream to Control：通过潜空间想象学习行为

Danijar Hafner † † thanks: Correspondence to: Danijar Hafner mail@danijar.com. Affiliation: University of Toronto Affiliation: Google Brain    Timothy Lillicrap Affiliation: DeepMind    Jimmy Ba Affiliation: University of Toronto    Mohammad Norouzi Affiliation: Google Brain

Abstract

摘要

Learned world models summarize an agent’s experience to facilitate learning complex behaviors. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks from images purely by latent imagination. We efficiently learn behaviors by propagating analytic gradients of learned state values back through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance.

学到的世界模型能概括智能体的经验，便于学习复杂行为。虽已能借助深度学习从高维感官输入学习世界模型，但从中导出行为仍有多种路径。我们提出 Dreamer：一种强化学习智能体，纯靠潜空间想象从图像解决长时程任务。我们在学到的世界模型的紧凑状态空间中想象轨迹，并把学到的状态价值的解析梯度沿这些轨迹反向传播，从而高效学习行为。在 20 个有挑战性的视觉控制任务上，Dreamer 在数据效率、计算时间与最终性能上均超过现有方法。

1 Introduction

1 引言

Figure 1: Dreamer learns a world model from past experience and efficiently learns farsighted behaviors in its latent space by backpropagating value estimates back through imagined trajectories.

图 1：Dreamer 从过往经验学习世界模型，并在其潜空间中通过把价值估计沿想象轨迹反向传播，高效学习有远见的行为。

Intelligent agents can achieve goals in complex environments even though they never encounter the exact same situation twice. This ability requires building representations of the world from past experience that enable generalization to novel situations. World models offer an explicit way to represent an agent’s knowledge about the world in a parametric model that can make predictions about the future.

智能智能体即便从未两次遇到完全相同的情境，也能在复杂环境中达成目标。这要求从过往经验建立世界的表示，以便泛化到新情境。世界模型提供了一种显式方式：用参数模型表示智能体对世界的知识，并对未来做预测。

When the sensory inputs are high-dimensional images, latent dynamics models can abstract observations to predict forward in compact state spaces (Watter et al., 2015, Oh et al., 2017, Gregor et al., 2019). Compared to predictions in image space, latent states have a small memory footprint that enables imagining thousands of trajectories in parallel. Learning effective latent dynamics models is becoming feasible through advances in deep learning and latent variable models (Krishnan et al., 2015, Karl et al., 2016, Doerr et al., 2018, Buesing et al., 2018).

当感官输入是高维图像时，潜动力学模型可以把观测抽象到紧凑状态空间中向前预测（Watter et al., 2015；Oh et al., 2017；Gregor et al., 2019）。相对在图像空间预测，潜状态内存占用小，可并行想象数千条轨迹。借助深度学习与潜变量模型的进展，学习有效的潜动力学正变得可行（Krishnan et al., 2015；Karl et al., 2016；Doerr et al., 2018；Buesing et al., 2018）。

Behaviors can be derived from dynamics models in many ways. Often, imagined rewards are maximized with a parametric policy (Sutton, 1991, Ha and Schmidhuber, 2018, Zhang et al., 2019) or by online planning (Chua et al., 2018, Hafner et al., 2018). However, considering only rewards within a fixed imagination horizon results in shortsighted behaviors (Wang et al., 2019). Moreover, prior work commonly resorts to derivative-free optimization for robustness to model errors (Ebert et al., 2017, Chua et al., 2018, Parmas et al., 2019), rather than leveraging analytic gradients offered by neural network dynamics (Henaff et al., 2019, Srinivas et al., 2018).

从动力学模型导出行为的方式很多。常见做法是用参数策略最大化想象奖励（Sutton, 1991；Ha and Schmidhuber, 2018；Zhang et al., 2019），或做在线规划（Chua et al., 2018；Hafner et al., 2018）。但只考虑固定想象时域内的奖励会导致短视行为（Wang et al., 2019）。此外，先前工作常为鲁棒模型误差而采用无导数优化（Ebert et al., 2017；Chua et al., 2018；Parmas et al., 2019），而非利用神经网络动力学提供的解析梯度（Henaff et al., 2019；Srinivas et al., 2018）。

We present Dreamer, an agent that learns long-horizon behaviors from images purely by latent imagination. A novel actor critic algorithm accounts for rewards beyond the imagination horizon while making efficient use of the neural network dynamics. For this, we predict state values and actions in the learned latent space as summarized in Figure 1. The values optimize Bellman consistency for imagined rewards and the policy maximizes the values by propagating their analytic gradients back through the dynamics.

我们提出 Dreamer：一种纯靠潜空间想象、从图像学习长时程行为的智能体。一种新的 actor-critic 算法在高效利用神经网络动力学的同时，计入想象时域之外的奖励。为此，我们在学到的潜空间中预测状态价值与动作，如图 1。价值对想象奖励优化 Bellman 一致性；策略通过把价值的解析梯度沿动力学反向传播来最大化这些价值。

In comparison to actor critic algorithms that learn online or by experience replay (Lillicrap et al., 2015, Mnih et al., 2016, Schulman et al., 2017, Haarnoja et al., 2018, Lee et al., 2019), world models can interpolate past experience and offer analytic gradients of multi-step returns for efficient policy optimization.

相对在线或经验回放学习的 actor-critic 算法（Lillicrap et al., 2015；Mnih et al., 2016；Schulman et al., 2017；Haarnoja et al., 2018；Lee et al., 2019），世界模型可以插值过往经验，并提供多步回报的解析梯度，便于高效优化策略。

The key contributions of this paper are summarized as follows:

本文主要贡献概括如下：

• • Learning long-horizon behaviors by latent imagination  Model-based agents can be shortsighted if they use a finite imagination horizon. We approach this limitation by predicting both actions and state values. Training purely by imagination in a latent space lets us efficiently learn the policy by propagating analytic value gradients back through the latent dynamics.
• • Empirical performance for visual control  We pair Dreamer with existing representation learning methods and evaluate it on the DeepMind Control Suite with image inputs, illustrated in Figure 2. Using the same hyper parameters for all tasks, Dreamer exceeds previous model-based and model-free agents in terms of data-efficiency, computation time, and final performance.

• • 通过潜空间想象学习长时程行为　基于模型的智能体若只用有限想象时域，可能短视。我们通过同时预测动作与状态价值来应对：纯在潜空间中靠想象训练，得以把解析价值梯度沿潜动力学反向传播，高效学习策略。
• • 视觉控制上的实证表现　我们把 Dreamer 与现有表示学习方法配对，在带图像输入的 DeepMind Control Suite 上评估，见图 2。全部任务共用同一套超参，Dreamer 在数据效率、计算时间与最终性能上超过先前基于模型与无模型智能体。

2 Control with World Models

2 用世界模型做控制

(a) Cup

Reinforcement learning

强化学习

We formulate visual control as a partially observable Markov decision process (POMDP) with discrete time step $t\in[1;T]$, continuous vector-valued actions $a_{t}\sim p(a_{t}\mido_{\leq t},a_{<t})$ generated by the agent, and high-dimensional observations and scalar rewards $o_{t},r_{t}\sim p(o_{t},r_{t}\mido_{<t},a_{<t})$ generated by the unknown environment. The goal is to develop an agent that maximizes the expected sum of rewards $\smash{\mathbb{\operatorname{E}}_{p}\big(\sum_{t=1}^{T}r_{t}\big)}$. Figure 2 shows a selection of our tasks.

我们把视觉控制表述为部分可观测马尔可夫决策过程（POMDP）：离散时间步 $t\in[1;T]$，智能体生成连续向量动作 $a_{t}\sim p(a_{t}\mido_{\leq t},a_{<t})$，未知环境生成高维观测与标量奖励 $o_{t},r_{t}\sim p(o_{t},r_{t}\mido_{<t},a_{<t})$。目标是设计最大化期望累计奖励 $\smash{\mathbb{\operatorname{E}}_{p}\big(\sum_{t=1}^{T}r_{t}\big)}$ 的智能体。图 2 展示部分任务。

Agent components

智能体组件

The classical components of agents that learn in imagination are dynamics learning, behavior learning, and environment interaction (Sutton, 1991). In the case of Dreamer, the behavior is learned by predicting hypothetical trajectories in the compact latent space of the world model. As outlined in Figure 3 and detailed in Algorithm 1, Dreamer performs the following operations throughout the agent’s life time, either interleaved or in parallel:

在想象中学习的智能体，经典组件是动力学学习、行为学习与环境交互（Sutton, 1991）。对 Dreamer，行为通过在世界模型的紧凑潜空间中预测假设轨迹来学习。如图 3 与算法 1，Dreamer 在生命周期中交错或并行执行下列操作：

• • Learning the latent dynamics model from the dataset of past experience to predict future rewards from actions and past observations. Any learning objective for the world model can be incorporated with Dreamer. We review existing methods for learning latent dynamics in Section 4.
• • Learning action and value models from predicted latent trajectories, as described in Section 3. The value model optimizes Bellman consistency for imagined rewards and the action model is updated by propagating gradients of value estimates back through the neural network dynamics.
• • Executing the learned action model in the world to collect new experience for growing the dataset.

• • 从过往经验数据集学习潜动力学模型，以便由动作与过往观测预测未来奖励。任意世界模型学习目标都可接入 Dreamer。第 4 节回顾现有潜动力学学习方法。
• • 如第 3 节所述，从预测的潜轨迹学习动作与价值模型。价值模型对想象奖励优化 Bellman 一致性；动作模型通过把价值估计的梯度沿神经网络动力学反向传播来更新。
• • 在真实世界中执行学到的动作模型，收集新经验以扩充数据集。

Latent dynamics

潜动力学

Dreamer uses a latent dynamics model that consists of three components. The representation model encodes observations and actions to create continuous vector-valued model states $s_{t}$ with Markovian transitions (Watter et al., 2015, Zhang et al., 2019, Hafner et al., 2018). The transition model predicts future model states without seeing the corresponding observations that will later cause them. The reward model predicts the rewards given the model states,

Dreamer 的潜动力学模型含三块。表示模型编码观测与动作，生成具有马尔可夫转移的连续向量模型状态 $s_{t}$（Watter et al., 2015；Zhang et al., 2019；Hafner et al., 2018）。转移模型在不看见将导致这些状态的对应观测时预测未来模型状态。奖励模型在给定模型状态时预测奖励，

$$\displaystyle\begin{aligned} \makebox[120.00018pt][l]{Representation model:}&&&p(s_{t}\mids_{t-1},a_{t-1},o_{t})\
\makebox[120.00018pt][l]{Transition model:}&&&q(s_{t}\mids_{t-1},a_{t-1})\
\makebox[120.00018pt][l]{Reward model:}&&&q(r_{t}\mids_{t}).\end{aligned} \tag{1}$$

We use $p$ for distributions that generate samples in the real environment and $q$ for their approximations that enable latent imagination. Specifically, the transition model lets us predict ahead in the compact latent space without having to observe or imagine the corresponding images. This results in a low memory footprint and fast predictions of thousands of imagined trajectories in parallel.

我们用 $p$ 表示在真实环境中产生样本的分布，用 $q$ 表示使其能做潜空间想象的近似。具体地，转移模型让我们在紧凑潜空间中向前预测，而不必观测或想象对应图像。由此内存占用低，可并行快速预测数千条想象轨迹。

The model mimics a non-linear Kalman filter (Kalman, 1960), latent state space model, or HMM with real-valued states. However, it is conditioned on actions and predicts rewards, allowing the agent to imagine the outcomes of potential action sequences without executing them in the environment.

该模型类似非线性卡尔曼滤波（Kalman, 1960）、潜状态空间模型，或带实值状态的 HMM。但它以动作为条件并预测奖励，使智能体无需在环境中执行就能想象潜在动作序列的后果。

3 Learning Behaviors by Latent Imagination

3 通过潜空间想象学习行为

(a) Learn dynamics from experience

(a) 从经验学习动力学

Initialize dataset $\mathcal{D}$ with $S$ random seed episodes. ;
Initialize neural network parameters $\theta,\phi,\psi$ randomly. ;
while not converged do
for update step $c=1..C$ do
// Dynamics learning
Draw $B$ data sequences $\{(a_{t},o_{t},r_{t})\}_{t=k}^{k+L}\sim\mathcal{D}$. ;
Compute model states $s_{t}\sim p_{\theta}(s_{t}\mids_{t-1},a_{t-1},o_{t})$. ;
Update $\theta$ using representation learning. ;
// Behavior learning
Imagine trajectories $\{(s_{\tau},a_{\tau})\}_{\tau=t}^{t+H}$ from each $s_{t}$. ;
Predict rewards $\mathbb{\operatorname{E}}\big(q_{\theta}(r_{\tau}\mids_{\tau})\big)$ and values $v_{\psi}(s_{\tau}).$ ;
Compute value estimates $\mathrm{V}_{\mathrm{\lambda}}(s_{\tau})$ via Equation 6. ;
Update $\phi\leftarrow\phi+\alpha\nabla_{\phi}\sum_{\tau=t}^{t+H}\mathrm{V}_{\mathrm{\lambda}}(s_{\tau})$. ;
Update \makebox[5.95834pt][l]{\psi}\leftarrow\makebox[5.95834pt][l]{\psi}-\alpha\nabla_{\psi}\sum_{\tau=t}^{t+H}\frac{1}{2}\big\|v_{\psi}(s_{\tau})\!\shortminus\!\mathrm{V}_{\mathrm{\lambda}}(s_{\tau})\big\|^{2}.
// Environment interaction
$o_{1}\leftarrow\texttt{env.reset()}$ ;
for time step $t=1..T$ do
Compute $s_{t}\sim p_{\theta}(s_{t}\mids_{t-1},a_{t-1},o_{t})$ from history. ;
Compute $a_{t}\sim q_{\phi}(a_{t}\mids_{t})$ with the action model. ;
Add exploration noise to action. ;
$r_{t},o_{t+1}\leftarrow\texttt{env.step(}a_{t}\texttt{)}.$ ;
Add experience to dataset $\mathcal{D}\leftarrow\mathcal{D}\cup\{(o_{t},a_{t},r_{t})_{t=1}^{T}\}.$ ;
Model components Representation $p_{\theta}(s_{t}\mids_{t\text{-}1},a_{t\text{-}1},o_{t})$ Transition $q_{\theta}(s_{t}\mids_{t\text{-}1},a_{t\text{-}1})$ Reward $q_{\theta}(r_{t}\mids_{t})$ Action $q_{\phi}(a_{t}\mids_{t})$ Value $v_{\psi}(s_{t})$ Hyper parameters Seed episodes $S$ Collect interval $C$ Batch size $B$ Sequence length $L$ Imagination horizon $H$ Learning rate $\alpha$

Algorithm 1 Dreamer

用 $S$ 条随机种子回合初始化数据集 $\mathcal{D}$。
随机初始化神经网络参数 $\theta,\phi,\psi$。
while 未收敛 do
for 更新步 $c=1..C$ do
// 动力学学习
从 $\mathcal{D}$ 抽取 $B$ 条数据序列 $\{(a_{t},o_{t},r_{t})\}_{t=k}^{k+L}$。
计算模型状态 $s_{t}\sim p_{\theta}(s_{t}\mids_{t-1},a_{t-1},o_{t})$。
用表示学习更新 $\theta$。
// 行为学习
从每个 $s_{t}$ 想象轨迹 $\{(s_{\tau},a_{\tau})\}_{\tau=t}^{t+H}$。
预测奖励 $\mathbb{\operatorname{E}}\big(q_{\theta}(r_{\tau}\mids_{\tau})\big)$ 与价值 $v_{\psi}(s_{\tau})$。
按式 6 计算价值估计 $\mathrm{V}_{\mathrm{\lambda}}(s_{\tau})$。
更新 $\phi\leftarrow\phi+\alpha\nabla_{\phi}\sum_{\tau=t}^{t+H}\mathrm{V}_{\mathrm{\lambda}}(s_{\tau})$。
更新 $\psi\leftarrow\psi-\alpha\nabla_{\psi}\sum_{\tau=t}^{t+H}\frac{1}{2}\big\|v_{\psi}(s_{\tau})\!\shortminus\!\mathrm{V}_{\mathrm{\lambda}}(s_{\tau})\big\|^{2}$。
// 环境交互
$o_{1}\leftarrow\texttt{env.reset()}$
for 时间步 $t=1..T$ do
由历史计算 $s_{t}\sim p_{\theta}(s_{t}\mids_{t-1},a_{t-1},o_{t})$。
用动作模型计算 $a_{t}\sim q_{\phi}(a_{t}\mids_{t})$。
给动作加探索噪声。
$r_{t},o_{t+1}\leftarrow\texttt{env.step(}a_{t}\texttt{)}$。
把经验加入数据集 $\mathcal{D}\leftarrow\mathcal{D}\cup\{(o_{t},a_{t},r_{t})_{t=1}^{T}\}$。
模型组件 表示 $p_{\theta}(s_{t}\mids_{t\text{-}1},a_{t\text{-}1},o_{t})$ 转移 $q_{\theta}(s_{t}\mids_{t\text{-}1},a_{t\text{-}1})$ 奖励 $q_{\theta}(r_{t}\mids_{t})$ 动作 $q_{\phi}(a_{t}\mids_{t})$ 价值 $v_{\psi}(s_{t})$ 超参 种子回合 $S$ 收集间隔 $C$ 批大小 $B$ 序列长度 $L$ 想象时域 $H$ 学习率 $\alpha$

算法 1 Dreamer

Figure 4: Imagination horizons. We compare the final performance of Dreamer, learning an action model without value prediction, and online planning using PlaNet. Learning a state value model to estimate rewards beyond the imagination horizon makes Dreamer more robust to the horizon length. The agents use pixel reconstruction for representation learning and an action repeat of $R=2$.

图 4：想象时域。比较 Dreamer、不做价值预测而只学动作模型、以及用 PlaNet 做在线规划的最终性能。学习状态价值模型以估计想象时域之外的奖励，使 Dreamer 对时域长度更稳健。智能体用像素重建做表示学习，动作重复 $R=2$。

Dreamer learns long-horizon behaviors in the compact latent space of a learned world model by efficiently leveraging the neural network latent dynamics. For this, we propagate stochastic gradients of multi-step returns through neural network predictions of actions, states, rewards, and values using reparameterization. This section describes the main contribution of our paper.

Dreamer 在学到的世界模型的紧凑潜空间中学习长时程行为，并高效利用神经网络潜动力学。为此，我们用重参数化，把多步回报的随机梯度沿动作、状态、奖励与价值的神经网络预测反向传播。本节描述本文主要贡献。

Imagination environment

想象环境

The latent dynamics define a Markov decision process (Sutton, 1991, MDP;) that is fully observed because the compact model states $s_{t}$ are Markovian. We denote imagined quantities with $\tau$ as the time index. Imagined trajectories start at the true model states $s_{t}$ of observation sequences drawn from the agent’s past experience. They follow predictions of the transition model $s_{\tau}\sim q(s_{\tau}\mids_{\tau-1},a_{\tau-1})$, reward model $r_{\tau}\sim q(r_{\tau}\mids_{\tau})$, and a policy $a_{\tau}\sim q(a_{\tau}\mids_{\tau})$. The objective is to maximize expected imagined rewards $\smash{\mathbb{\operatorname{E}}_{q}\big(\sum_{\tau=t}^{\infty}\gamma^{\tau-t}r_{\tau}\big)}$ with respect to the policy.

潜动力学定义一个马尔可夫决策过程（Sutton, 1991, MDP）：因紧凑模型状态 $s_{t}$ 具有马尔可夫性，该过程完全可观测。想象量的时间下标记为 $\tau$。想象轨迹从智能体过往经验中抽取的观测序列所对应的真实模型状态 $s_{t}$ 出发，遵循转移模型 $s_{\tau}\sim q(s_{\tau}\mids_{\tau-1},a_{\tau-1})$、奖励模型 $r_{\tau}\sim q(r_{\tau}\mids_{\tau})$ 与策略 $a_{\tau}\sim q(a_{\tau}\mids_{\tau})$ 的预测。目标是相对策略最大化期望想象奖励 $\smash{\mathbb{\operatorname{E}}_{q}\big(\sum_{\tau=t}^{\infty}\gamma^{\tau-t}r_{\tau}\big)}$。

Action and value models

动作与价值模型

Consider imagined trajectories with a finite horizon $H$. Dreamer uses an actor critic approach to learn behaviors that consider rewards beyond the horizon. We learn an action model and a value model in the latent space of the world model for this. The action model implements the policy and aims to predict actions that solve the imagination environment. The value model estimates the expected imagined rewards that the action model achieves from each state $s_{\tau}$,

考虑有限时域 $H$ 的想象轨迹。Dreamer 用 actor-critic 方法学习能计入时域之外奖励的行为。为此我们在世界模型的潜空间中学习动作模型与价值模型。动作模型实现策略，目标是预测能解决想象环境的动作；价值模型估计动作模型从各状态 $s_{\tau}$ 起获得的期望想象奖励，

$$\displaystyle\makebox[120.00018pt][l]{Action model:} \\ \displaystyle a_{\tau}\sim\ q_{\phi}(a_{\tau}\mids_{\tau}) \\ \displaystyle\makebox[120.00018pt][l]{Value model:} \\ \displaystyle v_{\psi}(s_{\tau})\approx\mathbb{\operatorname{E}}_{q(\cdot|s_{\tau})}\big(\textstyle\sum_{\tau=t}^{t+H}\gamma^{\tau-t}r_{\tau}\big).$$

The action and value models are trained cooperatively as typical in policy iteration: the action model aims to maximize an estimate of the value, while the value model aims to match an estimate of the value that changes as the action model changes.

动作与价值模型按策略迭代的典型方式协同训练：动作模型力图最大化价值估计，价值模型则力图匹配随动作模型变化而变化的价值估计。

We use dense neural networks for the action and value models with parameters $\phi$ and $\psi$, respectively. The action model outputs a tanh-transformed Gaussian (Haarnoja et al., 2018) with sufficient statistics predicted by the neural network. This allows for reparameterized sampling (Kingma and Welling, 2013, Rezende et al., 2014) that views sampled actions as deterministically dependent on the neural network output, allowing us to backpropagate analytic gradients through the sampling operation,

动作与价值模型分别用参数为 $\phi$、$\psi$ 的稠密神经网络。动作模型输出经 tanh 变换的高斯（Haarnoja et al., 2018），充分统计量由网络预测。这允许重参数化采样（Kingma and Welling, 2013；Rezende et al., 2014）：把采样动作视为确定性依赖于网络输出，从而可把解析梯度穿过采样操作反向传播，

$$a_{\tau}=\operatorname{tanh}\!\big(\mu_{\phi}(s_{\tau})+\sigma_{\phi}(s_{\tau})\,\epsilon\big),\quad\epsilon\sim\operatorname{Normal}(0,\mathop{}\!\mathbb{I}). \tag{3}$$

Value estimation

价值估计

To learn the action and value models, we need to estimate the state values of imagined trajectories $\{s_{\tau},a_{\tau},r_{\tau}\}_{\tau=t}^{t+H}$. These trajectories branch off of the model states $s_{t}$ of sequence batches drawn from the agent’s dataset of experience and predict forward for the imagination horizon $H$ using actions sampled from the action model. State values can be estimated in multiple ways that trade off bias and variance (Sutton and Barto, 2018),

为学习动作与价值模型，需要估计想象轨迹 $\{s_{\tau},a_{\tau},r_{\tau}\}_{\tau=t}^{t+H}$ 的状态价值。这些轨迹从智能体经验数据集中抽取的序列批次的模型状态 $s_{t}$ 分支，用动作模型采样的动作向前预测想象时域 $H$。状态价值可用多种方式估计，在偏差与方差之间权衡（Sutton and Barto, 2018），

$$\displaystyle\mathrm{V}_{\mathrm{R}}(s_{\tau}) \\ \displaystyle\doteq\mathbb{\operatorname{E}}_{q_{\theta},q_{\phi}}\bigg(\sum_{n=\tau}^{t+H}r_{n}\bigg), \\ \displaystyle\mathrm{V}_{\mathrm{N}}^{k}(s_{\tau}) \\ \displaystyle\doteq\mathbb{\operatorname{E}}_{q_{\theta},q_{\phi}}\bigg(\sum_{n=\tau}^{h-1}\gamma^{n-\tau}r_{n}+\gamma^{h-\tau}v_{\psi}(s_{h})\bigg)\quad\text{with}\quad h=\min(\tau+k,t+H), \\ \displaystyle\mathrm{V}_{\mathrm{\lambda}}(s_{\tau}) \\ \displaystyle\doteq(1-\lambda)\sum_{n=1}^{H-1}\lambda^{n-1}\mathrm{V}_{\mathrm{N}}^{n}(s_{\tau})+\lambda^{H-1}\mathrm{V}_{\mathrm{N}}^{H}(s_{\tau}), \tag{4}$$

where the expectations are estimated under the imagined trajectories. $\mathrm{V}_{\mathrm{R}}$ simply sums the rewards from $\tau$ until the horizon and ignores rewards beyond it. This allows learning the action model without a value model, an ablation we compare to in our experiments. $\mathrm{V}_{\mathrm{N}}^{k}$ estimates rewards beyond $k$ steps with the learned value model. Dreamer uses $\mathrm{V}_{\mathrm{\lambda}}$, an exponentially-weighted average of the estimates for different $k$ to balance bias and variance. Figure 4 shows that learning a value model in imagination enables Dreamer to solve long-horizon tasks while being robust to the imagination horizon. The experimental details and results on all tasks are described in Section 6.

其中期望在想象轨迹下估计。$\mathrm{V}_{\mathrm{R}}$ 只从 $\tau$ 累加奖励到时域为止，忽略之后的奖励；这允许不建价值模型就学动作模型，实验中我们做了这一消融。$\mathrm{V}_{\mathrm{N}}^{k}$ 用学到的价值模型估计 $k$ 步之后的奖励。Dreamer 使用 $\mathrm{V}_{\mathrm{\lambda}}$——对不同 $k$ 的估计做指数加权平均以平衡偏差与方差。图 4 显示：在想象中学习价值模型，使 Dreamer 能解决长时程任务，并对想象时域稳健。实验细节与全部任务结果见第 6 节。

Learning objective

学习目标

To update the action and value models, we first compute the value estimates $\mathrm{V}_{\mathrm{\lambda}}(s_{\tau})$ for all states $s_{\tau}$ along the imagined trajectories. The objective for the action model $q_{\phi}(a_{\tau}\mids_{\tau})$ is to predict actions that result in state trajectories with high value estimates. The objective for the value model $v_{\psi}(s_{\tau})$, in turn, is to regress the value estimates,

更新动作与价值模型时，先对想象轨迹上所有状态 $s_{\tau}$ 计算价值估计 $\mathrm{V}_{\mathrm{\lambda}}(s_{\tau})$。动作模型 $q_{\phi}(a_{\tau}\mids_{\tau})$ 的目标是预测能带来高价值估计状态轨迹的动作；价值模型 $v_{\psi}(s_{\tau})$ 的目标则是回归这些价值估计，

$$\max_{\phi}\mathbb{\operatorname{E}}{q{\theta},q_{\phi}}\bigg(\sum_{\tau=t}^{t+H}\mathrm{V}{\mathrm{\lambda}}(s{\tau})\bigg),\
\tag{7}$$

$$\min_{\psi}\mathbb{\operatorname{E}}_{q_{\theta},q_{\phi}}\bigg(\sum_{\tau=t}^{t+H}\frac{1}{2}\Big\|v_{\psi}(s_{\tau})-\mathrm{V}_{\mathrm{\lambda}}(s_{\tau}))\Big\|^{2}\bigg). \tag{8}$$

The value model is updated to regress the targets, around which we stop the gradient as typical (Sutton and Barto, 2018). The action model uses analytic gradients through the learned dynamics to maximize the value estimates. To understand this, we note that the value estimates depend on the reward and value predictions, which depend on the imagined states, which in turn depend on the imagined actions. Since all steps are implemented as neural networks, we analytically compute $\smash{\nabla_{\phi}\mathbb{\operatorname{E}}_{q_{\theta},q_{\phi}}\big(\sum_{\tau=t}^{t+H}\mathrm{V}_{\mathrm{\lambda}}(s_{\tau})\big)}$ by stochastic backpropagation (Kingma and Welling, 2013, Rezende et al., 2014). We use reparameterization for continuous actions and latent states and straight-through gradients (Bengio et al., 2013) for discrete actions. The world model is fixed while learning behaviors.

价值模型更新为回归目标，目标周围按惯例停止梯度（Sutton and Barto, 2018）。动作模型利用穿过学到动力学的解析梯度来最大化价值估计。理解这一点需注意：价值估计依赖奖励与价值预测，后者依赖想象状态，而想象状态又依赖想象动作。所有步骤均为神经网络实现，我们通过随机反向传播（Kingma and Welling, 2013；Rezende et al., 2014）解析计算 $\smash{\nabla_{\phi}\mathbb{\operatorname{E}}_{q_{\theta},q_{\phi}}\big(\sum_{\tau=t}^{t+H}\mathrm{V}_{\mathrm{\lambda}}(s_{\tau})\big)}$。连续动作与潜状态用重参数化，离散动作用直通梯度（Bengio et al., 2013）。学习行为时固定世界模型。

In tasks with early termination, the world model also predicts the discount factor from each latent state to weigh the time steps in Equations 7 and 8 by the cumulative product of the predicted discount factors, so terms are weighted down based on how likely the imagined trajectory would have ended.

在有提前终止的任务中，世界模型还从每个潜状态预测折扣因子，用以按预测折扣因子的累积乘积对式 7、8 中各时间步加权，从而按想象轨迹可能已结束的程度下调权重。

Comparison to actor critic methods

与 actor-critic 方法的比较

Agents using Reinforce gradients (Williams, 1992), such as A3C and PPO (Mnih et al., 2016, Schulman et al., 2017), employ value baselines to reduce gradient variance, while Dreamer backpropagates through the value model. This is similar to deterministic or reparameterized actor critics (Silver et al., 2014), such as DDPG and SAC (Lillicrap et al., 2015, Haarnoja et al., 2018). However, these do not leverage gradients through transitions and only maximize immediate Q-values. MVE and STEVE (Feinberg et al., 2018, Buckman et al., 2018) extend them to multi-step Q-learning with learned dynamics to provide more accurate Q-value targets. We predict state values, which is sufficient for policy optimization since we backpropagate through the dynamics. Refer to Section 5 for a more detailed comparison to related work.

使用 Reinforce 梯度（Williams, 1992）的智能体，如 A3C 与 PPO（Mnih et al., 2016；Schulman et al., 2017），用价值基线降低梯度方差，而 Dreamer 穿过价值模型反向传播。这类似于确定性或重参数化 actor-critic（Silver et al., 2014），如 DDPG 与 SAC（Lillicrap et al., 2015；Haarnoja et al., 2018）。不过后者不利用穿过转移的梯度，只最大化即时 Q 值。MVE 与 STEVE（Feinberg et al., 2018；Buckman et al., 2018）把它们扩展为带学到动力学的多步 Q 学习，以提供更准的 Q 目标。我们预测状态价值——因穿过动力学反向传播，这对策略优化已足够。更细的相关工作比较见第 5 节。

4 Learning Latent Dynamics

4 学习潜动力学

Figure 5: Reconstructions of long-term predictions. We apply the representation model to the first $5$ images of two hold-out trajectories and predict forward for $45$ steps using the latent dynamics, given only the actions. The recurrent state space model (Hafner et al., 2018, RSSM;) performs accurate long-term predictions, enabling Dreamer to learn successful behaviors in a compact latent space.

图 5：长期预测的重建。对两条留出轨迹的前 5 帧应用表示模型，再仅给定动作用潜动力学向前预测 45 步。循环状态空间模型（Hafner et al., 2018, RSSM）能做准确的长期预测，使 Dreamer 能在紧凑潜空间中学到成功行为。

Figure 6: Performance comparison to existing methods. Dreamer inherits the data-efficiency of PlaNet while exceeding the asymptotic performance of the best model-free agents. After $5\times 10^{6}$ environment steps, Dreamer reaches an average performance of $823$ across tasks, compared to PlaNet at $332$ and the top model-free D4PG agent at $786$ after $10^{8}$ steps. Results are averages over 5 seeds.

图 6：与现有方法的性能比较。Dreamer 继承 PlaNet 的数据效率，同时超过最佳无模型智能体的渐近性能。在 $5\times 10^{6}$ 环境步后，Dreamer 跨任务平均性能达 $823$，相比之下 PlaNet 为 $332$，顶尖无模型 D4PG 在 $10^{8}$ 步后为 $786$。结果为 5 个种子的平均。

Figure 7: Dreamer succeeds at visual control tasks that require long-horizon credit assignment, such as the acrobot and hopper tasks. Optimizing only imagined rewards within the horizon via an action model or by online planning yields shortsighted behaviors that only succeed in reactive tasks, such as in the walker domain. The performance on all 20 tasks is summarized in Figure 6 and training curves are shown in Appendix D. See Tassa et al. (2018) for performance curves of D4PG and A3C.

图 7：Dreamer 在需要长时程信用分配的视觉控制任务上成功，如 acrobot 与 hopper。仅在时域内优化想象奖励（用动作模型或在线规划）会产生短视行为，只在反应式任务（如 walker 域）上成功。全部 20 个任务的性能汇总见图 6，训练曲线见附录 D。D4PG 与 A3C 的性能曲线见 Tassa et al. (2018)。

Learning behaviors in imagination requires a world model that generalizes well. We focus on latent dynamics models that predict forward in a compact latent space, facilitating long-term predictions and allowing the agent to imagine thousands of trajectories in parallel. Several objectives for learning representations for control have been proposed (Watter et al., 2015, Jaderberg et al., 2016, Oord et al., 2018, Eslami et al., 2018). We review three approaches for learning representations to use with Dreamer: reward prediction, image reconstruction, and contrastive estimation.

在想象中学习行为，要求世界模型泛化良好。我们聚焦在紧凑潜空间中向前预测的潜动力学模型，便于长期预测，并允许智能体并行想象数千条轨迹。已有多种为控制学习表示的目标（Watter et al., 2015；Jaderberg et al., 2016；Oord et al., 2018；Eslami et al., 2018）。我们回顾三种可与 Dreamer 搭配的表示学习途径：奖励预测、图像重建与对比估计。

Reward prediction

奖励预测

Latent imagination requires a representation model $p(s_{t}\mids_{t-1},a_{t-1},o_{t})$, transition model $q(s_{t}\mids_{t-1},a_{t-1},)$, and reward model $q(r_{t}\mids_{t})$, as described in Section 2. In principle, this could be achieved by simply learning to predict future rewards given actions and past observations (Oh et al., 2017, Gelada et al., 2019, Schrittwieser et al., 2019). With a large and diverse dataset, such representations should be sufficient for solving a control task. However, with a finite dataset and especially when rewards are sparse, learning about observations that correlate with rewards is likely to improve the world model (Jaderberg et al., 2016, Gregor et al., 2019).

潜空间想象需要表示模型 $p(s_{t}\mids_{t-1},a_{t-1},o_{t})$、转移模型 $q(s_{t}\mids_{t-1},a_{t-1})$ 与奖励模型 $q(r_{t}\mids_{t})$，见第 2 节。原则上，只需学会在给定动作与过往观测时预测未来奖励即可（Oh et al., 2017；Gelada et al., 2019；Schrittwieser et al., 2019）。数据量大且多样时，这类表示应足以解决控制任务。但在有限数据下，尤其奖励稀疏时，学习与奖励相关的观测很可能改善世界模型（Jaderberg et al., 2016；Gregor et al., 2019）。

Reconstruction

重建

We first describe the world model used by PlaNet (Hafner et al., 2018) that learns latent dynamics by reconstructing images as shown in Figure 3(a). The world model consists of the following components, where the observation model is only used to provide a learning signal,

我们先描述 PlaNet（Hafner et al., 2018）所用、通过重建图像学习潜动力学的世界模型，见图 3(a)。世界模型含下列组件，其中观测模型仅用于提供学习信号，

$$\displaystyle\begin{aligned} \makebox[120.00018pt][l]{Representation model:}&&&p_{\theta}(s_{t}\mids_{t-1},a_{t-1},o_{t})\
\makebox[120.00018pt][l]{Observation model:}&&&q_{\theta}(o_{t}\mids_{t})\
\makebox[120.00018pt][l]{Reward model:}&&&q_{\theta}(r_{t}\mids_{t})\
\makebox[120.00018pt][l]{Transition model:}&&&q_{\theta}(s_{t}\mids_{t-1},a_{t-1}).\end{aligned} \tag{9}$$

The components are optimized jointly to increase the variational lower bound (Jordan et al., 1999, ELBO;) or more generally the variational information bottleneck (Tishby et al., 2000, Alemi et al., 2016, VIB;). As derived in Appendix B, the bound includes reconstruction terms for observations and rewards and a KL regularizer. The expectation is taken under the dataset and representation model,

各组件联合优化以增大变分下界（Jordan et al., 1999, ELBO），或更一般的变分信息瓶颈（Tishby et al., 2000；Alemi et al., 2016, VIB）。如附录 B 所导，该界含观测与奖励的重建项以及 KL 正则。期望在数据集与表示模型下取，

$$\displaystyle\mathcal{J}_{\mathrm{REC}}\doteq\mathbb{\operatorname{E}}_{p}\bigg(\sum_{t}\Big(\mathcal{J}_{\mathrm{O}}^{t}+\mathcal{J}_{\mathrm{R}}^{t}+\mathcal{J}_{\mathrm{D}}^{t}\Big)\bigg)+\text{const}\qquad\mathcal{J}_{\mathrm{O}}^{t}\doteq\ \ln q(o_{t}\mids_{t}) \\ \displaystyle\mathcal{J}_{\mathrm{R}}^{t}\doteq\ln q(r_{t}\mids_{t})\qquad\mathcal{J}_{\mathrm{D}}^{t}\doteq\ -\beta\operatorname{KL}\big(p(s_{t}\mids_{t-1},a_{t-1},o_{t})\;\big\|\;q(s_{t}\mids_{t-1},a_{t-1})\big).$$

We implement the transition model as a recurrent state space model (Hafner et al., 2018, RSSM;), the representation model by combining the RSSM with a convolutional neural network (LeCun et al., 1989, CNN;) applied to the image observation, the observation model as a transposed CNN, and the reward model as a dense network. The combined parameter vector $\theta$ is updated by stochastic backpropagation (Kingma and Welling, 2013, Rezende et al., 2014). Figure 5 shows video predictions of this model. We refer to Appendix A and Hafner et al. (2018) model details.

转移模型实现为循环状态空间模型（Hafner et al., 2018, RSSM）；表示模型把 RSSM 与作用于图像观测的卷积神经网络（LeCun et al., 1989, CNN）结合；观测模型为转置 CNN；奖励模型为稠密网络。组合参数向量 $\theta$ 用随机反向传播更新（Kingma and Welling, 2013；Rezende et al., 2014）。图 5 给出该模型的视频预测。模型细节见附录 A 与 Hafner et al. (2018)。

Contrastive estimation

对比估计

Predicting pixels can require high model capacity. We can also encourage mutual information between model states and observations by instead predicting the states from the images (Guo et al., 2018). This replaces the observation model with a state model,

预测像素可能需要很高模型容量。我们也可改为从图像预测状态，以鼓励模型状态与观测之间的互信息（Guo et al., 2018）。这用状态模型替换观测模型，

$$\displaystyle\begin{aligned} \makebox[120.00018pt][l]{State model:}&&&q_{\theta}(s_{t}\mido_{t}).\end{aligned} \tag{11}$$

While the reconstruction objective used the fact that the observation marginal is a constant, we now face the state marginal. As shown in Appendix B, this can be estimated via noise contrastive estimation (Gutmann and Hyvärinen, 2010, Oord et al., 2018, NCE;) by averaging the state model over observations $o^{\prime}$ of the current sequence batch. Intuitively, $q(s_{t}\mido_{t})$ makes the state predictable from the current image while $\ln\sum_{o^{\prime}}q(s_{t}\mido^{\prime})$ keeps it diverse to prevent collapse,

重建目标利用了观测边缘是常数这一事实，现在则面对状态边缘。如附录 B 所示，可通过噪声对比估计（Gutmann and Hyvärinen, 2010；Oord et al., 2018, NCE）对当前序列批次中的观测 $o^{\prime}$ 平均状态模型来估计。直观上，$q(s_{t}\mido_{t})$ 使状态可由当前图像预测，而 $\ln\sum_{o^{\prime}}q(s_{t}\mido^{\prime})$ 保持多样性以防坍塌，

$$\displaystyle\begin{aligned} \mathcal{J}_{\mathrm{NCE}}\doteq\ \mathbb{\operatorname{E}}\bigg(\sum_{t}\Big(\mathcal{J}_{\mathrm{S}}^{t}+\mathcal{J}_{\mathrm{R}}^{t}+\mathcal{J}_{\mathrm{D}}^{t}\Big)\bigg)\quad\mathcal{J}_{\mathrm{S}}^{t}\doteq\ &\ln q(s_{t}\mido_{t})-\ln\bigg(\sum_{o^{\prime}}q(s_{t}\mido^{\prime})\bigg).\end{aligned} \tag{12}$$

We implement the state model as a CNN and again optimize the bound with respect to the combined parameter vector $\theta$ using stochastic backpropagation. While avoiding pixel prediction, the amount of information this bound can extract efficiently is limited (McAllester and Statos, 2018). We empirically compare reward, reconstruction, and contrastive objectives in our experiments in Figure 8.

状态模型实现为 CNN，同样用随机反向传播相对组合参数 $\theta$ 优化该界。虽避免像素预测，该界能高效提取的信息量有限（McAllester and Statos, 2018）。我们在图 8 的实验中实证比较奖励、重建与对比目标。

5 Related Work

5 相关工作

Prior works learn latent dynamics for visual control by derivative-free policy learning or online planning, augment model-free agents with multi-step predictions, or use analytic gradients of Q-values or multi-step rewards, often for low-dimensional tasks. In comparison, Dreamer uses analytic gradients to efficiently learn long-horizon behaviors for visual control purely by latent imagination.

先前工作为视觉控制学习潜动力学时，或用无导数策略学习与在线规划，或用多步预测增强无模型智能体，或利用 Q 值或多步奖励的解析梯度（常针对低维任务）。相较之下，Dreamer 用解析梯度，纯靠潜空间想象高效学习视觉控制的长时程行为。

Control with latent dynamics

用潜动力学做控制

E2C (Watter et al., 2015) and RCE (Banijamali et al., 2017) embed images to predict forward in a compact space to solve simple tasks. World Models (Ha and Schmidhuber, 2018) learn latent dynamics in a two-stage process to evolve linear controllers in imagination. PlaNet (Hafner et al., 2018) learns them jointly and solves visual locomotion tasks by latent online planning. SOLAR (Zhang et al., 2019) solves robotic tasks via guided policy search in latent space. I2A (Weber et al., 2017) hands imagined trajectories to a model-free policy, while Lee et al. (2019) and Gregor et al. (2019) learn belief representations to accelerate model-free agents.

E2C（Watter et al., 2015）与 RCE（Banijamali et al., 2017）把图像嵌入紧凑空间向前预测以解决简单任务。World Models（Ha and Schmidhuber, 2018）分两阶段学习潜动力学，在想象中演化线性控制器。PlaNet（Hafner et al., 2018）联合学习它们，并用潜空间在线规划解决视觉运动任务。SOLAR（Zhang et al., 2019）经潜空间引导策略搜索解决机器人任务。I2A（Weber et al., 2017）把想象轨迹交给无模型策略；Lee et al. (2019) 与 Gregor et al. (2019) 学习信念表示以加速无模型智能体。

Imagined multi-step returns

想象的多步回报

VPN (Oh et al., 2017), MVE (Feinberg et al., 2018), and STEVE (Buckman et al., 2018) learn dynamics for multi-step Q-learning from a replay buffer. AlphaGo (Silver et al., 2017) combines predictions of actions and state values with planning, assuming access to the true dynamics. Also assuming access to the dynamics, POLO (Lowrey et al., 2018) plans to explore by learning a value ensemble. MuZero (Schrittwieser et al., 2019) learns task-specific reward and value models to solve challenging tasks but requires large amounts of experience. PETS (Chua et al., 2018), VisualMPC (Ebert et al., 2017), and PlaNet (Hafner et al., 2018) plan online using derivative-free optimization. POPLIN (Wang and Ba, 2019) improves over online planning by self-imitation. Piergiovanni et al. (2018) learn robot policies by imagination with a latent dynamics model. Planning with neural network gradients was shown on small problems (Schmidhuber, 1990, Henaff et al., 2018) but has been challenging to scale (Parmas et al., 2019).

VPN（Oh et al., 2017）、MVE（Feinberg et al., 2018）与 STEVE（Buckman et al., 2018）从回放缓冲学习动力学以做多步 Q 学习。AlphaGo（Silver et al., 2017）把动作与状态价值预测与规划结合，并假定可访问真实动力学。同样假定可访问动力学时，POLO（Lowrey et al., 2018）通过学习价值集成规划探索。MuZero（Schrittwieser et al., 2019）学习任务专用的奖励与价值模型以解决挑战性任务，但需要大量经验。PETS（Chua et al., 2018）、VisualMPC（Ebert et al., 2017）与 PlaNet（Hafner et al., 2018）用无导数优化做在线规划。POPLIN（Wang and Ba, 2019）经自模仿改进在线规划。Piergiovanni et al. (2018) 用潜动力学模型通过想象学习机器人策略。用神经网络梯度做规划在小问题上已有展示（Schmidhuber, 1990；Henaff et al., 2018），但扩展到大规模仍有挑战（Parmas et al., 2019）。

Analytic value gradients

解析价值梯度

DPG (Silver et al., 2014), DDPG (Lillicrap et al., 2015), and SAC (Haarnoja et al., 2018) leverage gradients of learned immediate action values to learn a policy by experience replay. SVG (Heess et al., 2015) reduces the variance of model-free on-policy algorithms by analytic value gradients of one-step model predictions. Concurrent work by Byravan et al. (2019) uses latent imagination with deterministic models for navigation and manipulation tasks. ME-TRPO (Kurutach et al., 2018) accelerates an otherwise model-free agent via gradients of predicted rewards for proprioceptive inputs. DistGBP (Henaff et al., 2017, Henaff et al., 2019) uses model gradients for online planning in simple tasks.

DPG（Silver et al., 2014）、DDPG（Lillicrap et al., 2015）与 SAC（Haarnoja et al., 2018）利用学到的即时动作价值的梯度，经经验回放学习策略。SVG（Heess et al., 2015）用一步模型预测的解析价值梯度降低无模型同策略算法的方差。Byravan et al. (2019) 的同期工作用确定性模型的潜空间想象做导航与操作。ME-TRPO（Kurutach et al., 2018）经预测奖励的梯度加速否则为无模型的智能体，面向本体感觉输入。DistGBP（Henaff et al., 2017；Henaff et al., 2019）在简单任务上用模型梯度做在线规划。

6 Experiments

6 实验

Figure 8: Comparison of representation learning objectives to be used with Dreamer. Pixel reconstruction performs best for the majority of tasks. The contrastive objective solves about half of the tasks, while predicting rewards alone was not sufficient in our experiments. The results suggest that future developments in learning representations are likely to translate into improved task performance for Dreamer. The performance curves for all tasks are included in Appendix E.

图 8：与 Dreamer 搭配的表示学习目标比较。像素重建在多数任务上表现最好。对比目标约解决一半任务；仅预测奖励在我们实验中不足。结果表明：表示学习的未来进展很可能转化为 Dreamer 更高的任务性能。全部任务的性能曲线见附录 E。

We experimentally evaluate Dreamer on a variety of control tasks. We designed the experiments to compare Dreamer to current best methods in the literature, and to evaluate its ability to solve tasks with long horizons, continuous actions, discrete actions, and early termination. We further compare the orthogonal choice of learning objective for the world model. The source code for all our experiments and videos of Dreamer are available at https://danijar.com/dreamer.

我们在多种控制任务上实验评估 Dreamer。实验旨在与文献中当前最佳方法比较，并评估其解决长时程、连续动作、离散动作与提前终止任务的能力。我们还比较世界模型学习目标这一正交选择。全部实验源码与 Dreamer 视频见 https://danijar.com/dreamer。

Control tasks

控制任务

We evaluate Dreamer on 20 visual control tasks of the DeepMind Control Suite Tassa et al. (2018), illustrated in Figure 2. These tasks pose a variety of challenges, including sparse rewards, contact dynamics, and 3D scenes. We selected the tasks on which Tassa et al. (2018) report non-zero performance from image inputs. Agent observations are images of shape $64\times 64\times 3$, actions range from $1$ to $12$ dimensions, rewards range from $0$ to $1$, episodes last for $1000$ steps and have randomized initial states. We use a fixed action repeat of $R=2$ across tasks. We further evaluate the applicability of Dreamer to discrete actions and early termination on a subset of Atari games (Bellemare et al., 2013) and DeepMind Lab levels (Beattie et al., 2016) as detailed in Appendix C.

我们在 DeepMind Control Suite（Tassa et al., 2018）的 20 个视觉控制任务上评估 Dreamer，见图 2。这些任务涵盖稀疏奖励、接触动力学与三维场景等挑战。我们选取 Tassa et al. (2018) 报告从图像输入可获非零性能的任务。智能体观测为 $64\times 64\times 3$ 图像，动作维度 1 到 12，奖励 0 到 1，回合长 1000 步且初始状态随机。跨任务固定动作重复 $R=2$。我们还在部分 Atari 游戏（Bellemare et al., 2013）与 DeepMind Lab 关卡（Beattie et al., 2016）上评估 Dreamer 对离散动作与提前终止的适用性，细节见附录 C。

Implementation

实现

Our implementation uses TensorFlow Probability (Dillon et al., 2017). We use a single Nvidia V100 GPU and 10 CPU cores for each training run. The training time for our Dreamer implementation is about $3$ hours per $10^{6}$ environment steps on the control suite, compared to $11$ hours for online planning using PlaNet, and the $24$ hours used by D4PG to reach similar performance. We use the same hyper parameters across all continuous tasks, and similarly across all discrete tasks, detailed in Appendix A. The world models are learned via reconstruction unless specified.

实现使用 TensorFlow Probability（Dillon et al., 2017）。每次训练跑用单块 Nvidia V100 GPU 与 10 个 CPU 核。在 control suite 上，我们的 Dreamer 实现约每 $10^{6}$ 环境步训练 3 小时，相比之下 PlaNet 在线规划约 11 小时，D4PG 达到相近性能约需 24 小时。全部连续任务共用同一套超参，离散任务同理，细节见附录 A。除非另说明，世界模型经重建学习。

Baseline methods

基线方法

The highest reported performance on the continuous tasks is achieved by D4PG (Barth-Maron et al., 2018), an improved variant of DDPG (Lillicrap et al., 2015) that uses distributed collection, distributional Q-learning, multi-step returns, and prioritized replay. We include the scores for D4PG with pixel inputs and A3C (Mnih et al., 2016) with state inputs from Tassa et al. (2018). PlaNet (Hafner et al., 2018) learns the same world model as Dreamer and selects actions via online planning without an action model and drastically improves over D4PG and A3C in data efficiency. We re-run PlaNet with $R=2$ for a unified experimental setup. For Atari, we show the final performance of SimPLe (Kaiser et al., 2019), DQN (Mnih et al., 2015) and Rainbow (Hessel et al., 2018) reported by Castro et al. (2018), and for DeepMind Lab that of IMPALA (Espeholt et al., 2018) as a guideline.

连续任务上报告最高性能的是 D4PG（Barth-Maron et al., 2018）——DDPG（Lillicrap et al., 2015）的改进变体，含分布式采集、分布 Q 学习、多步回报与优先回放。我们纳入 Tassa et al. (2018) 中带像素输入的 D4PG 与带状态输入的 A3C（Mnih et al., 2016）分数。PlaNet（Hafner et al., 2018）学习与 Dreamer 相同的世界模型，经在线规划选动作而无动作模型，在数据效率上大幅超过 D4PG 与 A3C。我们以 $R=2$ 重跑 PlaNet 以统一实验设置。对 Atari，展示 Castro et al. (2018) 报告的 SimPLe（Kaiser et al., 2019）、DQN（Mnih et al., 2015）与 Rainbow（Hessel et al., 2018）最终性能；对 DeepMind Lab，以 IMPALA（Espeholt et al., 2018）作参考。

Performance

性能

To evaluate the performance of Dreamer, we compare it to state-of-the-art reinforcement learning agents. The results are summarized in Figure 6. With an average score of $823$ across tasks after $5\times 10^{6}$ environment steps, Dreamer exceeds the performance of the strong model-free D4PG agent that achieves an average of $786$ within $10^{8}$ environment steps. At the same time, Dreamer inherits the data-efficiency of PlaNet, confirming that the learned world model can help to generalize from small amounts of experience. The empirical success of Dreamer shows that learning behaviors by latent imagination with world models can outperform top methods based on experience replay.

为评估 Dreamer 性能，我们与当时最优强化学习智能体比较。结果汇总于图 6。在 $5\times 10^{6}$ 环境步后跨任务平均分 $823$，Dreamer 超过强无模型 D4PG——后者在 $10^{8}$ 环境步内平均 $786$。同时 Dreamer 继承 PlaNet 的数据效率，确认学到的世界模型有助于从少量经验泛化。Dreamer 的实证成功表明：用世界模型经潜空间想象学习行为，可超过基于经验回放的顶尖方法。

Long horizons

长时程

To investigate its ability to learn long-horizon behaviors, we compare Dreamer to alternatives for deriving behaviors from the world model at various horizon lengths. For this, we learn an action model to maximize imagined rewards without a value model and compare to online planning using PlaNet. Figure 4 shows the final performance for different imagination horizons, confirming that the value model makes Dreamer more robust to the horizon and performs well even for short horizons. Performance curves for all 19 tasks with horizon of 20 are shown in Appendix D, where Dreamer outperforms the alternatives on 16 of 20 tasks, with 4 ties.

为考察其学习长时程行为的能力，我们在不同时域长度下，比较从世界模型导出行为的替代方案。为此，我们学习不带价值模型、仅最大化想象奖励的动作模型，并与 PlaNet 在线规划比较。图 4 给出不同想象时域下的最终性能，确认价值模型使 Dreamer 对时域更稳健，即便短时域也表现良好。时域为 20 时全部 19 个任务的性能曲线见附录 D：Dreamer 在 20 个任务中的 16 个上超过替代方案，另有 4 个持平。

Representation learning

表示学习

Dreamer can be used with any differentiable dynamics model that predicts future rewards given actions and past observations. Since the representation learning objective is orthogonal to our algorithm, we compare three natural choices described in Section 4: pixel reconstruction, contrastive estimation, and pure reward prediction. Figure 8 shows clear differences in task performance for different representation learning approaches, with pixel reconstruction outperforming contrastive estimation on most tasks. This suggests that future improvements in representation learning are likely to translate to higher task performance with Dreamer. Reward prediction alone was not sufficient in our experiments. Further ablations are included in the appendix of the paper.

Dreamer 可与任意能在给定动作与过往观测时预测未来奖励的可微动力学模型搭配。表示学习目标与我们的算法正交，我们比较第 4 节所述三种自然选择：像素重建、对比估计与纯奖励预测。图 8 显示不同表示学习途径在任务性能上差异明显，像素重建在多数任务上超过对比估计。这表明表示学习的未来改进很可能转化为 Dreamer 更高的任务性能。仅奖励预测在我们实验中不足。更多消融见论文附录。

7 Conclusion

7 结论

We present Dreamer, an agent that learns long-horizon behaviors purely by latent imagination. For this, we propose an actor critic method that optimizes a parametric policy by propagating analytic gradients of multi-step values back through learned latent dynamics. Dreamer outperforms previous methods in data-efficiency, computation time, and final performance on a variety of challenging continuous control tasks with image inputs. We further show that Dreamer is applicable to tasks with discrete actions and early episode termination. Future research on representation learning can likely scale latent imagination to environments of higher visual complexity.

我们提出 Dreamer：一种纯靠潜空间想象学习长时程行为的智能体。为此，我们提出一种 actor-critic 方法：把多步价值的解析梯度沿学到的潜动力学反向传播，以优化参数策略。在多种有挑战性的带图像输入的连续控制任务上，Dreamer 在数据效率、计算时间与最终性能上超过先前方法。我们还表明 Dreamer 适用于离散动作与回合提前终止的任务。表示学习的未来研究很可能把潜空间想象扩展到视觉复杂度更高的环境。

Acknowledgements

致谢

We thank Simon Kornblith, Benjamin Eysenbach, Ian Fischer, Amy Zhang, Geoffrey Hinton, Shane Gu, Adam Kosiorek, Brandon Amos, Jacob Buckman, Calvin Luo, and Rishabh Agarwal, and our anonymous reviewers for feedback and discussions. We thank Yuval Tassa for adding the quadruped environment to the control suite.

感谢 Simon Kornblith、Benjamin Eysenbach、Ian Fischer、Amy Zhang、Geoffrey Hinton、Shane Gu、Adam Kosiorek、Brandon Amos、Jacob Buckman、Calvin Luo、Rishabh Agarwal 以及匿名审稿人的反馈与讨论。感谢 Yuval Tassa 把 quadruped 环境加入 control suite。

References

参考文献

Alemi et al. (2016) A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 , 2016.

Banijamali et al. (2017) E. Banijamali, R. Shu, M. Ghavamzadeh, H. Bui, and A. Ghodsi. Robust locally-linear controllable embedding. arXiv preprint arXiv:1710.05373 , 2017.

Barth-Maron et al. (2018) G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, A. Muldal, N. Heess, and T. Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617 , 2018.

Beattie et al. (2016) C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801 , 2016.

Bellemare et al. (2013) M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253–279, 2013.

Bengio et al. (2013) Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.

Buckman et al. (2018) J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems , pages 8224–8234, 2018.

Buesing et al. (2018) L. Buesing, T. Weber, S. Racaniere, S. Eslami, D. Rezende, D. P. Reichert, F. Viola, F. Besse, K. Gregor, D. Hassabis, et al. Learning and querying fast generative models for reinforcement learning. arXiv preprint arXiv:1802.03006 , 2018.

Byravan et al. (2019) A. Byravan, J. T. Springenberg, A. Abdolmaleki, R. Hafner, M. Neunert, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller. Imagined value gradients: Model-based policy optimization with transferable latent dynamics models. arXiv preprint arXiv:1910.04142 , 2019.

Castro et al. (2018) P. S. Castro, S. Moitra, C. Gelada, S. Kumar, and M. G. Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110 , 2018.

Chua et al. (2018) K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems , pages 4754–4765, 2018.

Clevert et al. (2015) D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 , 2015.

Dillon et al. (2017) J. V. Dillon, I. Langmore, D. Tran, E. Brevdo, S. Vasudevan, D. Moore, B. Patton, A. Alemi, M. Hoffman, and R. A. Saurous. Tensorflow distributions. arXiv preprint arXiv:1711.10604 , 2017.

Doerr et al. (2018) A. Doerr, C. Daniel, M. Schiegg, D. Nguyen-Tuong, S. Schaal, M. Toussaint, and S. Trimpe. Probabilistic recurrent state-space models. arXiv preprint arXiv:1801.10395 , 2018.

Ebert et al. (2017) F. Ebert, C. Finn, A. X. Lee, and S. Levine. Self-supervised visual planning with temporal skip connections. arXiv preprint arXiv:1710.05268 , 2017.

Eslami et al. (2018) S. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, I. Danihelka, K. Gregor, et al. Neural scene representation and rendering. Science , 360(6394):1204–1210, 2018.

Espeholt et al. (2018) L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561 , 2018.

Feinberg et al. (2018) V. Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101 , 2018.

Gelada et al. (2019) C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare. Deepmdp: Learning continuous latent space models for representation learning. arXiv preprint arXiv:1906.02736 , 2019.

Gregor et al. (2019) K. Gregor, D. J. Rezende, F. Besse, Y. Wu, H. Merzic, and A. v. d. Oord. Shaping belief states with generative environment models for rl. arXiv preprint arXiv:1906.09237 , 2019.

Guo et al. (2018) Z. D. Guo, M. G. Azar, B. Piot, B. A. Pires, T. Pohlen, and R. Munos. Neural predictive belief representations. arXiv preprint arXiv:1811.06407 , 2018.

Gutmann and Hyvärinen (2010) M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 297–304, 2010.

Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber. World models. arXiv preprint arXiv:1803.10122 , 2018.

Haarnoja et al. (2018) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 , 2018.

Hafner et al. (2018) D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551 , 2018.

Heess et al. (2015) N. Heess, G. Wayne, D. Silver, T. Lillicrap, T. Erez, and Y. Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems , pages 2944–2952, 2015.

Henaff et al. (2017) M. Henaff, W. F. Whitney, and Y. LeCun. Model-based planning in discrete action spaces. CoRR , abs/1705.07177, 2017.

Henaff et al. (2018) M. Henaff, W. F. Whitney, and Y. LeCun. Model-based planning with discrete and continuous actions. arXiv preprint arXiv:1705.07177 , 2018.

Henaff et al. (2019) M. Henaff, A. Canziani, and Y. LeCun. Model-predictive policy learning with uncertainty regularization for driving in dense traffic. arXiv preprint arXiv:1901.02705 , 2019.

Hessel et al. (2018) M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Jaderberg et al. (2016) M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 , 2016.

Jordan et al. (1999) M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine learning , 37(2):183–233, 1999.

Kaiser et al. (2019) L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374 , 2019.

Kalman (1960) R. E. Kalman. A new approach to linear filtering and prediction problems. Journal of basic Engineering , 82(1):35–45, 1960.

Karl et al. (2016) M. Karl, M. Soelch, J. Bayer, and P. van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint arXiv:1605.06432 , 2016.

Kingma and Ba (2014) D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Kingma and Welling (2013) D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Krishnan et al. (2015) R. G. Krishnan, U. Shalit, and D. Sontag. Deep kalman filters. arXiv preprint arXiv:1511.05121 , 2015.

Kurutach et al. (2018) T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592 , 2018.

LeCun et al. (1989) Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4):541–551, 1989.

Lee et al. (2019) A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv:1907.00953 , 2019.

Lillicrap et al. (2015) T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Lowrey et al. (2018) K. Lowrey, A. Rajeswaran, S. Kakade, E. Todorov, and I. Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint arXiv:1811.01848 , 2018.

Machado et al. (2018) M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research , 61:523–562, 2018.

McAllester and Statos (2018) D. McAllester and K. Statos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv:1811.04251 , 2018.

Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529, 2015.

Mnih et al. (2016) V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning , pages 1928–1937, 2016.

Oh et al. (2017) J. Oh, S. Singh, and H. Lee. Value prediction network. In Advances in Neural Information Processing Systems , pages 6118–6128, 2017.

Oord et al. (2018) A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Parmas et al. (2019) P. Parmas, C. E. Rasmussen, J. Peters, and K. Doya. Pipps: Flexible model-based policy search robust to the curse of chaos. arXiv preprint arXiv:1902.01240 , 2019.

Piergiovanni et al. (2018) A. Piergiovanni, A. Wu, and M. S. Ryoo. Learning real-world robot policies by dreaming. arXiv preprint arXiv:1805.07813 , 2018.

Poole et al. (2019) B. Poole, S. Ozair, A. v. d. Oord, A. A. Alemi, and G. Tucker. On variational bounds of mutual information. arXiv preprint arXiv:1905.06922 , 2019.

Rezende et al. (2014) D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082 , 2014.

Schmidhuber (1990) J. Schmidhuber. Making the world differentiable: On using self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments. 1990.

Schrittwieser et al. (2019) J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265 , 2019.

Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Silver et al. (2014) D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning , 2014.

Silver et al. (2017) D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature , 550(7676):354, 2017.

Srinivas et al. (2018) A. Srinivas, A. Jabri, P. Abbeel, S. Levine, and C. Finn. Universal planning networks. arXiv preprint arXiv:1804.00645 , 2018.

Sutton (1991) R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin , 2(4):160–163, 1991.

Sutton and Barto (2018) R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction . MIT press, 2018.

Tassa et al. (2018) Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Tishby et al. (2000) N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. arXiv preprint physics/0004057 , 2000.

Wang and Ba (2019) T. Wang and J. Ba. Exploring model-based planning with policy networks. arXiv preprint arXiv:1906.08649 , 2019.

Wang et al. (2019) T. Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois, S. Zhang, G. Zhang, P. Abbeel, and J. Ba. Benchmarking model-based reinforcement learning. CoRR , abs/1907.02057, 2019.

Watter et al. (2015) M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems , pages 2746–2754, 2015.

Weber et al. (2017) T. Weber, S. Racanière, D. P. Reichert, L. Buesing, A. Guez, D. J. Rezende, A. P. Badia, O. Vinyals, N. Heess, Y. Li, et al. Imagination-augmented agents for deep reinforcement learning. arXiv preprint arXiv:1707.06203 , 2017.

Williams (1992) R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3-4):229–256, 1992.

Zhang et al. (2019) M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning , 2019.

Appendix A Hyper Parameters

附录 A 超参数

Model components

模型组件

We use the convolutional encoder and decoder networks from Ha and Schmidhuber (2018), the RSSM of Hafner et al. (2018), and implement all other functions as three dense layers of size $300$ with ELU activations (Clevert et al., 2015). Distributions in latent space are 30-dimensional diagonal Gaussians. The action model outputs a tanh mean scaled by a factor of 5 and a softplus standard deviation for the Normal distribution that is then transformed using tanh (Haarnoja et al., 2018). The scaling factor allows the agent to saturate the action distribution.

我们使用 Ha and Schmidhuber (2018) 的卷积编码器与解码器网络、Hafner et al. (2018) 的 RSSM，其余函数均实现为三层大小 300、ELU 激活（Clevert et al., 2015）的稠密层。潜空间分布为 30 维对角高斯。动作模型输出经因子 5 缩放的 tanh 均值，以及对随后经 tanh 变换的 Normal 分布的 softplus 标准差（Haarnoja et al., 2018）。缩放因子使智能体能饱和动作分布。

Learning updates

学习更新

We draw batches of 50 sequences of length 50 to train the world model, value model, and action model models using Adam (Kingma and Ba, 2014) with learning rates $6\times 10^{-4}$, $8\times 10^{-5}$, $8\times 10^{-5}$, respectively and scale down gradient norms that exceed $100$. We do not scale the KL regularizers ($\beta=1$) but clip them below $3$ free nats as in PlaNet. The imagination horizon is $H=15$ and the same trajectories are used to update both action and value models. We compute the $\mathrm{V}_{\lambda}$ targets with $\gamma=0.99$ and $\lambda=0.95$. We did not find latent overshooting for learning the model, an entropy bonus for the action model, or target networks for the value model necessary.

我们抽取 50 条长度为 50 的序列批次，用 Adam（Kingma and Ba, 2014）分别以学习率 $6\times 10^{-4}$、$8\times 10^{-5}$、$8\times 10^{-5}$ 训练世界模型、价值模型与动作模型，梯度范数超过 100 时缩小。不缩放 KL 正则（$\beta=1$），但按 PlaNet 做法将其下截断在 3 free nats。想象时域 $H=15$，同一批轨迹同时更新动作与价值模型。用 $\gamma=0.99$、$\lambda=0.95$ 计算 $\mathrm{V}_{\lambda}$ 目标。我们未发现学习模型时的潜 overshooting、动作模型熵奖励或价值模型目标网络是必要的。

Environment interaction

环境交互

The dataset is initialized with $S=5$ episodes collected using random actions. We iterate between $100$ training steps and collecting $1$ episode by executing the predicted mode action with $\operatorname{Normal}(0,0.3)$ exploration noise. Instead of manually selecting the action repeat for each environment as in Hafner et al. (2018) and Lee et al. (2019), we fix it to 2 for all environments. See Figure 12 for an assessment of the robustness to different action repeat values.

数据集用随机动作收集的 $S=5$ 条回合初始化。我们在 100 次训练步与收集 1 条回合之间迭代：执行预测的众数动作，并加 $\operatorname{Normal}(0,0.3)$ 探索噪声。不像 Hafner et al. (2018) 与 Lee et al. (2019) 那样为每个环境手动选动作重复，我们对所有环境固定为 2。对不同动作重复取值的稳健性评估见图 12。

Discrete control

离散控制

For experiments on Atari games and DeepMind Lab levels, the action model predicts the logits of a categorical distribution. We use straight-through gradients for the sampling step during latent imagination. The action noise is epsilon greedy where $\epsilon$ is linearly scheduled from $0.4\rightarrow 0.1$ over the first $200,000$ gradient steps. To account for the higher complexity of these tasks, we use an imagination horizon of $H=10$, scale the KL regularizers by $\beta=0.1$, and bound rewards using tanh. We predict the discount factor from the latent state with a binary classifier that is trained towards the soft labels of $0$ and $\gamma$.

在 Atari 游戏与 DeepMind Lab 关卡实验中，动作模型预测类别分布的 logits。潜空间想象中的采样步使用直通梯度。动作噪声为 $\epsilon$-greedy，$\epsilon$ 在前 200,000 次梯度步从 $0.4\rightarrow 0.1$ 线性调度。为应对这些任务更高的复杂度，取想象时域 $H=10$，KL 正则乘 $\beta=0.1$，并用 tanh 界住奖励。我们用二分类器从潜状态预测折扣因子，训练目标为软标签 0 与 $\gamma$。

Appendix B Derivations

附录 B 推导

We define the information bottleneck objective (Tishby et al., 2000) for latent dynamics models,

我们对潜动力学模型定义信息瓶颈目标（Tishby et al., 2000），

$$\max\mathbb{\operatorname{I}}(s_{1:T};(o_{1:T},r_{1:T})\mid a_{1:T})-\beta\,\mathbb{\operatorname{I}}(s_{1:T},i_{1:T}\mid a_{1:T}), \tag{13}$$

where $\beta$ is scalar and $i_{t}$ are dataset indices that determine the observations $p(o_{t}\midi_{t})\doteq\delta(o_{t}-\bar{o}_{t})$ as in Alemi et al. (2016).

其中 $\beta$ 为标量，$i_{t}$ 为决定观测的数据集索引 $p(o_{t}\midi_{t})\doteq\delta(o_{t}-\bar{o}_{t})$，如 Alemi et al. (2016)。

Maximizing the objective leads to model states that can predict the sequence of observations and rewards while limiting the amount of information extracted at each time step. This encourages the model to reconstruct each image by relying on information extracted at preceeding time steps to the extent possible, and only accessing additional information from the current image when necessary. As a result, the information regularizer encourages the model to learn long-term dependencies.

最大化该目标使模型状态既能预测观测与奖励序列，又限制每步提取的信息量。这鼓励模型在重建每张图像时尽可能依赖先前时间步提取的信息，仅在必要时从当前图像获取额外信息。因此，信息正则鼓励模型学习长期依赖。

For the generative objective, we lower bound the first term using the non-negativity of the KL divergence and drop the marginal data probability as it does not depend on the representation model,

对生成目标，我们用 KL 散度非负性对第一项下界，并因边缘数据概率不依赖表示模型而将其丢掉，

$$\displaystyle\mathbb{\operatorname{I}}(s_{1:T};(o_{1:T},r_{1:T})\mid a_{1:T}) \\ \displaystyle= \\ \displaystyle\mathbb{\operatorname{E}}_{p(o_{1:T},r_{1:T},s_{1:T},a_{1:T})}\Big(\sum_{t}\ln p(o_{1:T},r_{1:T}\mids_{1:T},a_{1:T})-\underbracket{\ln p(o_{1:T},r_{1:T}\mida_{1:T})}_{\text{const}}\Big) \\ \displaystyle\stackrel{{\scriptstyle+}}{{=}} \\ \displaystyle\mathbb{\operatorname{E}}\Big(\sum_{t}\ln p(o_{1:T},r_{1:T}\mids_{1:T},a_{1:T})\Big) \\ \displaystyle\geq \\ \displaystyle\mathbb{\operatorname{E}}\Big(\sum_{t}\ln p(o_{1:T},r_{1:T}\mids_{1:T},a_{1:T})\Big)-\operatorname{KL}\Big(p(o_{1:T},r_{1:T}\mids_{1:T},a_{1:T})\;\Big\|\;\prod_{t}q(o_{t}\mids_{t})q(r_{t}\mids_{t})\Big) \\ \displaystyle= \\ \displaystyle\mathbb{\operatorname{E}}\Big(\sum_{t}\ln q(o_{t}\mids_{t})+\ln q(r_{t}\mids_{t})\Big).$$

For the contrastive objective, we subtract the constant marginal probability of the data under the variational encoder, apply Bayes rule, and use the InfoNCE mini-batch bound (Poole et al., 2019),

对对比目标，我们减去变分编码器下数据的常数边缘概率，应用贝叶斯法则，并用 InfoNCE 小批量界（Poole et al., 2019），

$$\displaystyle\mathbb{\operatorname{E}}\big(\ln q(o_{t}\mids_{t})+\ln q(r_{t}\mids_{t})\big) \\ \displaystyle\stackrel{{\scriptstyle+}}{{=}} \\ \displaystyle\mathbb{\operatorname{E}}\big(\ln q(o_{t}\mids_{t})-\ln q(o_{t})+\ln q(r_{t}\mids_{t})\big) \\ \displaystyle= \\ \displaystyle\mathbb{\operatorname{E}}\big(\ln q(s_{t}\mido_{t})-\ln q(s_{t})+\ln q(r_{t}\mids_{t})\big) \\ \displaystyle\geq \\ \displaystyle\mathbb{\operatorname{E}}\bigg(\ln q(s_{t}\mido_{t})-\ln\sum_{o^{\prime}}q(s_{t}\mido^{\prime})+\ln q(r_{t}\mids_{t})\bigg).$$

For the second term, we use the non-negativity of the KL divergence to obtain an upper bound,

对第二项，我们用 KL 散度非负性得到上界，

$$\displaystyle\mathbb{\operatorname{I}}(s_{1:T};i_{1:T}\mid a_{1:T}) \\ \displaystyle= \\ \displaystyle\mathbb{\operatorname{E}}_{p(o_{1:T},r_{1:T},s_{1:T},a_{1:T},i_{1:T})}\Big(\sum_{t}\ln p(s_{t}\mids_{t-1},a_{t-1},i_{t})-\ln p(s_{t}\mids_{t-1},a_{t-1})\Big) \\ \displaystyle= \\ \displaystyle\mathbb{\operatorname{E}}\Big(\sum_{t}\ln p(s_{t}\mids_{t-1},a_{t-1},o_{t})-\ln p(s_{t}\mids_{t-1},a_{t-1})\Big) \\ \displaystyle\leq \\ \displaystyle\mathbb{\operatorname{E}}\Big(\sum_{t}\ln p(s_{t}\mids_{t-1},a_{t-1},o_{t})-\ln q(s_{t}\mids_{t-1},a_{t-1})\Big) \\ \displaystyle= \\ \displaystyle\mathbb{\operatorname{E}}\Big(\sum_{t}\operatorname{KL}\big(p(s_{t}\mids_{t-1},a_{t-1},o_{t})\;\big\|\;q(s_{t}\mids_{t-1},a_{t-1})\big)\Big).$$

This lower bounds the objective.

这给出目标的下界。

Appendix C Discrete Control

附录 C 离散控制

We evaluate Dreamer on a subset of tasks with discrete actions from the Atari suite (Bellemare et al., 2013) and DeepMind Lab (Beattie et al., 2016). While agents that purely learn through world models are not yet competitive in these domains (Kaiser et al., 2019), the tasks offer a diverse test bed with visual complexity, sparse rewards, and early termination. Agents observe $64\times 64\times 3$ images and select one of between $3$ and $18$ actions. For Atari, we follow the evaluation protocol of Machado et al. (2018) with sticky actions. Refer to Figure 9 for these experiments.

我们在 Atari suite（Bellemare et al., 2013）与 DeepMind Lab（Beattie et al., 2016）的部分离散动作任务上评估 Dreamer。虽纯靠世界模型学习的智能体在这些领域尚未具竞争力（Kaiser et al., 2019），这些任务仍提供含视觉复杂度、稀疏奖励与提前终止的多样试验台。智能体观测 $64\times 64\times 3$ 图像，在 3 到 18 个动作中选一。对 Atari，我们遵循 Machado et al. (2018) 的 sticky actions 评估协议。实验见图 9。

Figure 9: Performance of Dreamer in environments with discrete actions and early termination. Dreamer learns successful behaviors on this subset of Atari games and the object collection level of DMLab. We highlight representation learning for these environments as a direction of future work that could enable competitive performance across all Atari games and DMLab levels using Dreamer.

图 9：Dreamer 在离散动作与提前终止环境中的性能。Dreamer 在该 Atari 子集与 DMLab 的物体收集关卡上学会成功行为。我们把这些环境上的表示学习标为未来工作方向，或可使 Dreamer 在全部 Atari 游戏与 DMLab 关卡上具竞争力。

Appendix D Behavior Learning

附录 D 行为学习

Figure 10: Comparison of action selection schemes on the continuous control tasks of the DeepMind Control Suite from pixel inputs. The lines show mean scores over environment steps and the shaded areas show the standard deviation across 5 seeds. We compare Dreamer that learns both actions and values in imagination, to only learning actions in imagination, and Planet that selects actions by online planning instead of learning a policy. The baselines include the top model-free algorithm D4PG, the well-known A3C agent, and the hybrid SLAC agent.

图 10：在 DeepMind Control Suite 像素输入连续控制任务上比较动作选择方案。曲线为相对环境步的平均分数，阴影为 5 个种子的标准差。我们比较在想象中同时学习动作与价值的 Dreamer、仅在想象中学习动作，以及用在线规划而非学策略选动作的 PlaNet。基线包括顶尖无模型算法 D4PG、知名的 A3C，以及混合的 SLAC。

Appendix E Representation Learning

附录 E 表示学习

Figure 11: Comparison of representation learning methods for Dreamer. The lines show mean scores and the shaded areas show the standard deviation across 5 seeds. We compare generating both images and rewards, generating rewards and using a contrastive loss to learn about the images, and only predicting rewards. Image reconstruction provides the best learning signal across most of the tasks, followed by the contrastive objective. Learning purely from rewards was not sufficient in our experiments and might require larger amounts of experience.

图 11：Dreamer 的表示学习方法比较。曲线为平均分数，阴影为 5 个种子的标准差。我们比较同时生成图像与奖励、生成奖励并用对比损失学习图像，以及仅预测奖励。图像重建在多数任务上提供最佳学习信号，其次是对比目标。纯从奖励学习在我们实验中不足，或需更多经验。

Appendix F Action Repeat

附录 F 动作重复

Figure 12: Robustness of Dreamer to different control frequencies. Reinforcement learning methods can be sensitive to this hyper parameter, which could be amplified when learning dynamics models at the control frequency of the environment. For this experiment, we train Dreamer with different amounts of action repeat. The areas show one standard deviation across 2 seeds. We used a previous hyper parameter setting for this experiment. We find that a value of $R=2$ works best across tasks.

图 12：Dreamer 对不同控制频率的稳健性。强化学习方法可能对该超参敏感；在环境控制频率上学习动力学模型时，敏感性可能被放大。本实验用不同动作重复量训练 Dreamer。区域显示 2 个种子的一倍标准差。本实验使用先前的超参设置。我们发现跨任务以 $R=2$ 最佳。

Appendix G Continuous Control Scores

附录 G 连续控制分数