Curiosity-driven Exploration by Self-supervised Prediction

用自监督预测做好奇心驱动的探索

Deepak Pathak Affiliation:  University of California, Berkeley Correspondence to: pathak@berkeley.edu    Pulkit Agrawal Affiliation:  University of California, Berkeley    Alexei A. Efros Affiliation:  University of California, Berkeley    Trevor Darrell Affiliation:  University of California, Berkeley

Abstract

摘要

In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent’s ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, critically, ignores the aspects of the environment that cannot affect the agent. The proposed approach is evaluated in two environments: VizDoom and Super Mario Bros. Three broad settings are investigated: 1) sparse extrinsic reward, where curiosity allows for far fewer interactions with the environment to reach the goal; 2) exploration with no extrinsic reward, where curiosity pushes the agent to explore more efficiently; and 3) generalization to unseen scenarios (e.g. new levels of the same game) where the knowledge gained from earlier experience helps the agent explore new places much faster than starting from scratch.

在许多现实场景里，对智能体而言的外在奖励极为稀疏，甚至完全没有。这时，好奇心可以充当内在奖励信号，让智能体去探索环境，并学会日后可能用得上的技能。我们把好奇心表述为：智能体在由自监督逆动力学模型学到的视觉特征空间里，预测自身动作后果时的误差。这一表述可以扩展到图像这类高维连续状态空间，避开直接预测像素的困难，并且——这一点很关键——会忽略那些无法影响智能体的环境因素。我们在两个环境中评估该方法：VizDoom 和 Super Mario Bros。我们考察了三类设定：1）外在奖励稀疏时，好奇心能显著减少到达目标所需的环境交互次数；2）完全没有外在奖励的探索中，好奇心会推动智能体更有效地探索；3）泛化到未见过的场景（例如同一款游戏的新关卡）时，先前经验帮助智能体比从零开始更快地探索新地方。

1 Introduction

1 引言

Keywords:

Reinforcement learning algorithms aim at learning policies for achieving target tasks by maximizing rewards provided by the environment. In some scenarios, these rewards are supplied to the agent continuously, e.g. the running score in an Atari game (Mnih et al., 2015), or the distance between a robot arm and an object in a reaching task (Lillicrap et al., 2016). However, in many real-world scenarios, rewards extrinsic to the agent are extremely sparse or missing altogether, and it is not possible to construct a shaped reward function. This is a problem as the agent receives reinforcement for updating its policy only if it succeeds in reaching a pre-specified goal state. Hoping to stumble into a goal state by chance (i.e. random exploration) is likely to be futile for all but the simplest of environments.

关键词：

强化学习算法的目标，是通过最大化环境给出的奖励，学到完成目标任务的策略。有些场景里，这些奖励会持续提供给智能体，例如 Atari 游戏中不断变化的分数（Mnih et al., 2015），或是机械臂在伸手任务中与物体的距离（Lillicrap et al., 2016）。但在许多现实场景中，对智能体而言的外在奖励极为稀疏，甚至完全缺失，而且无法构造出塑形后的奖励函数。问题在于：智能体只有成功到达预先指定的目标状态时，才会得到用于更新策略的强化信号。指望靠碰巧撞进目标状态（也就是随机探索）就能成功，除了最简单的环境之外，多半都会落空。

(a) learn to explore in Level-1

（a）在 Level-1 中学习探索

As human agents, we are accustomed to operating with rewards that are so sparse that we only experience them once or twice in a lifetime, if at all. To a three-year-old enjoying a sunny Sunday afternoon on a playground, most trappings of modern life – college, good job, a house, a family – are so far into the future, they provide no useful reinforcement signal. Yet, the three-year-old has no trouble entertaining herself in that playground using what psychologists call intrinsic motivation (Ryan, 2000) or curiosity (Silvia, 2012). Motivation/curiosity have been used to explain the need to explore the environment and discover novel states. The French word flâneur perfectly captures the notion of a curiosity-driven observer, the “deliberately aimless pedestrian, unencumbered by any obligation or sense of urgency” (Cornelia Otis Skinner). More generally, curiosity is a way of learning new skills which might come handy for pursuing rewards in the future.

作为人类，我们早已习惯在极为稀疏的奖励下行动——有些奖励一辈子也只会遇到一两次，甚至从未遇到。对一个周日下午在游乐场玩耍的三岁孩子来说，现代生活里的许多东西——上大学、找到好工作、房子、家庭——都远在未来，提供不了有用的强化信号。可这个三岁孩子依然能在游乐场里自己玩得很开心，靠的是心理学家所说的内在动机（Ryan, 2000）或好奇心（Silvia, 2012）。动机/好奇心常被用来解释人们为何要探索环境、发现新的状态。法语词 flâneur 恰好抓住了这种由好奇心驱动的观察者形象：那个“有意漫无目的的行人，不被任何义务或紧迫感所束缚”（Cornelia Otis Skinner）。更一般地说，好奇心是一种学习新技能的方式，而这些技能日后追求奖励时可能会派上用场。

Similarly, in reinforcement learning, intrinsic motivation/rewards become critical whenever extrinsic rewards are sparse. Most formulations of intrinsic reward can be grouped into two broad classes: 1) encourage the agent to explore “novel” states (Bellemare et al., 2016; Poupart et al., 2006; Lopes et al., 2012) or, 2) encourage the agent to perform actions that reduce the error/uncertainty in the agent’s ability to predict the consequence of its own actions (i.e. its knowledge about the environment) (Schmidhuber, 1991; Schmidhuber, 2010; Singh et al., 2005; Mohamed & Rezende, 2015; Stadie et al., 2015; Houthooft et al., 2016).

同样，在强化学习里，只要外在奖励稀疏，内在动机/奖励就变得关键。现有的内在奖励表述大多可以归为两大类：1）鼓励智能体去探索“新奇”的状态（Bellemare et al., 2016; Poupart et al., 2006; Lopes et al., 2012）；或者 2）鼓励智能体采取能降低其预测自身动作后果时的误差/不确定性的动作（也就是降低它对环境的认识缺口）（Schmidhuber, 1991; Schmidhuber, 2010; Singh et al., 2005; Mohamed & Rezende, 2015; Stadie et al., 2015; Houthooft et al., 2016）。

Measuring “novelty” requires a statistical model of the distribution of the environmental states, whereas measuring prediction error/uncertainty requires building a model of environmental dynamics that predicts the next state ($s_{t+1}$) given the current state ($s_{t}$) and the action ($a_{t}$) executed at time $t$. Both these models are hard to build in high-dimensional continuous state spaces such as images. An additional challenge lies in dealing with the stochasticity of the agent-environment system, both due to the noise in the agent’s actuation, which causes its end-effectors to move in a stochastic manner, and, more fundamentally, due to the inherent stochasticity in the environment. To give the example from (Schmidhuber, 2010), if the agent receiving images as state inputs is observing a television screen displaying white noise, every state will be novel and it would be impossible to predict the value of any pixel in the future. Other examples of such stochasticity include appearance changes due to shadows from other moving entities, presence of distractor objects, or other agents in the environment whose motion is not only hard to predict but is also irrelevant to the agent’s goals. Somewhat different, but related, is the challenge of generalization across physically (and perhaps also visually) distinct but functionally similar parts of an environment, which is crucial for large-scale problems. One proposed solution to all these problems is to only reward the agent when it encounters states that are hard to predict but are “learnable” (Schmidhuber, 1991). However, estimating learnability is a non-trivial problem (Lopes et al., 2012).

度量“新奇性”需要环境状态分布的统计模型；度量预测误差/不确定性，则需要建立一个环境动力学模型，在给定当前状态（$s_{t}$）和时刻 $t$ 执行的动作（$a_{t}$）时预测下一状态（$s_{t+1}$）。这两种模型在图像这类高维连续状态空间里都很难建立。另一个挑战是应对智能体—环境系统的随机性：既来自智能体执行机构的噪声（末端执行器会以随机方式运动），更根本地，也来自环境固有的随机性。借用（Schmidhuber, 2010）的例子：如果以图像为状态输入的智能体正在观察一块播放白噪声的电视屏幕，那么每个状态都会是新奇的，也无法预测未来任何一个像素的取值。这类随机性的其他例子还包括：其他运动实体投下的阴影造成的外观变化、干扰物体的存在，或环境中其他智能体的运动——这些运动不仅难以预测，而且与智能体自身的目标无关。与此相关、但略有不同的，是如何在环境中那些物理上（也许视觉上也）不同、功能上却相似的部分之间做泛化，这对大规模问题至关重要。针对上述问题，一种已有提议是：只在智能体遇到难以预测、但又“可学习”的状态时才给予奖励（Schmidhuber, 1991）。不过，估计可学习性本身并不是一个容易的问题（Lopes et al., 2012）。

This work belongs to the broad category of methods that generate an intrinsic reward signal based on how hard it is for the agent to predict the consequences of its own actions, i.e. predict the next state given the current state and the executed action. However, we manage to escape most pitfalls of previous prediction approaches with the following key insight: we only predict those changes in the environment that could possibly be due to the actions of our agent or affect the agent, and ignore the rest. That is, instead of making predictions in the raw sensory space (e.g. pixels), we transform the sensory input into a feature space where only the information relevant to the action performed by the agent is represented. We learn this feature space using self-supervision – training a neural network on a proxy inverse dynamics task of predicting the agent’s action given its current and next states. Since the neural network is only required to predict the action, it has no incentive to represent within its feature embedding space the factors of variation in the environment that do not affect the agent itself. We then use this feature space to train a forward dynamics model that predicts the feature representation of the next state, given the feature representation of the current state and the action. We provide the prediction error of the forward dynamics model to the agent as an intrinsic reward to encourage its curiosity.

本工作属于这样一类方法：根据智能体预测自身动作后果有多难（也就是在给定当前状态和已执行动作时预测下一状态有多难），来生成内在奖励信号。不过，我们靠下面这条关键认识，避开了以往预测方法的大多数陷阱：我们只预测那些可能由智能体自身动作引起、或会影响到智能体的环境变化，其余一概忽略。也就是说，我们不在原始感官空间（例如像素）里做预测，而是把感官输入变换到一个特征空间，其中只保留与智能体所执行动作相关的信息。我们用自监督来学习这个特征空间——让神经网络去做一项代理性的逆动力学任务：根据当前状态和下一状态，预测智能体采取了什么动作。由于网络只需要预测动作，它没有动机在特征嵌入空间里表示那些并不影响智能体自身的环境变化因素。随后，我们在这个特征空间上训练一个前向动力学模型：根据当前状态的特征表示和动作，预测下一状态的特征表示。我们把前向动力学模型的预测误差作为内在奖励提供给智能体，以激发它的好奇心。

The role of curiosity has been widely studied in the context of solving tasks with sparse rewards. In our opinion, curiosity has two other fundamental uses. Curiosity helps an agent explore its environment in the quest for new knowledge (a desirable characteristic of exploratory behavior is that it should improve as the agent gains more knowledge). Further, curiosity is a mechanism for an agent to learn skills that might be helpful in future scenarios. In this paper, we evaluate the effectiveness of our curiosity formulation in all three of these roles.

好奇心在解决稀疏奖励任务中的作用已经得到广泛研究。在我们看来，好奇心还有另外两项根本用途。好奇心帮助智能体在寻求新知识的过程中探索环境（好的探索行为应当随着智能体获得更多知识而改进）。此外，好奇心也是智能体学习那些在未来场景中可能有用的技能的一种机制。本文会在这三种角色上分别评估我们这种好奇心表述的效果。

We first compare the performance of an A3C agent (Mnih et al., 2016) with and without the curiosity signal on 3-D navigation tasks with sparse extrinsic reward in the VizDoom environment. We show that a curiosity-driven intrinsic reward is crucial in accomplishing these tasks (see Section 4.1). Next, we show that even in the absence of any extrinsic rewards, a curious agent learns good exploration policies. For instance, an agent trained only with curiosity as its reward is able to cross a significant portion of Level-1 in Super Mario Bros. Similarly in VizDoom, the agent learns to walk intelligently along the corridors instead of bumping into walls or getting stuck in corners (see Section 4.2). A question that naturally follows is whether the learned exploratory behavior is specific to the physical space that the agent trained itself on, or if it enables the agent to perform better in unseen scenarios too? We show that the exploration policy learned in the first level of Mario helps the agent explore subsequent levels faster (shown in Figure 1), while the intelligent walking behavior learned by the curious VizDoom agent transfers to a completely new map with new textures (see Section  4.3). These results suggest that the proposed method enables an agent to learn generalizable skills even in the absence of an explicit goal.

我们首先在 VizDoom 环境中、外在奖励稀疏的三维导航任务上，比较带与不带好奇心信号的 A3C 智能体（Mnih et al., 2016）。结果表明，由好奇心驱动的内在奖励对这些任务至关重要（见第 4.1 节）。接着我们说明，即便完全没有外在奖励，好奇的智能体也能学到良好的探索策略。例如，只用好奇心作为奖励训练的智能体，能够通过 Super Mario Bros 第一关的相当一部分。类似地，在 VizDoom 中，智能体会学着沿着走廊智能地行走，而不是撞墙或卡在角落里（见第 4.2 节）。接下来一个自然的问题是：学到的探索行为是只针对它训练时所在的物理空间，还是也能让智能体在未见过的场景中表现更好？我们表明，在 Mario 第一关学到的探索策略，能帮助智能体更快地探索后续关卡（如图 1 所示）；而好奇的 VizDoom 智能体学到的智能行走行为，也能迁移到一张纹理全新的地图上（见第 4.3 节）。这些结果说明，即便没有显式目标，所提方法也能让智能体学到可泛化的技能。

Figure 2: The agent in state $s_{t}$ interacts with the environment by executing an action $a_{t}$ sampled from its current policy $\pi$ and ends up in the state $s_{t+1}$. The policy $\pi$ is trained to optimize the sum of the extrinsic reward ($r_{t}^{e}$) provided by the environment E and the curiosity based intrinsic reward signal ($r_{t}^{i}$) generated by our proposed Intrinsic Curiosity Module (ICM). ICM encodes the states $s_{t}$, $s_{t+1}$ into the features $\phi(s_{t}),\phi(s_{t+1})$ that are trained to predict $a_{t}$ (i.e. inverse dynamics model). The forward model takes as inputs $\phi(s_{t})$ and $a_{t}$ and predicts the feature representation $\hat{\phi}(s_{t+1})$ of $s_{t+1}$. The prediction error in the feature space is used as the curiosity based intrinsic reward signal. As there is no incentive for $\phi(s_{t})$ to encode any environmental features that can not influence or are not influenced by the agent’s actions, the learned exploration strategy of our agent is robust to uncontrollable aspects of the environment.

图 2：处于状态 $s_{t}$ 的智能体，从当前策略 $\pi$ 中采样动作 $a_{t}$ 与环境交互，并到达状态 $s_{t+1}$。策略 $\pi$ 的训练目标是最大化环境 $E$ 提供的外在奖励（$r_{t}^{e}$）与我们提出的内在好奇心模块（ICM）生成的、基于好奇心的内在奖励信号（$r_{t}^{i}$）之和。ICM 把状态 $s_{t}$、$s_{t+1}$ 编码成特征 $\phi(s_{t}),\phi(s_{t+1})$，这些特征被训练用来预测 $a_{t}$（即逆动力学模型）。前向模型以 $\phi(s_{t})$ 和 $a_{t}$ 为输入，预测 $s_{t+1}$ 的特征表示 $\hat{\phi}(s_{t+1})$。特征空间中的预测误差被用作基于好奇心的内在奖励信号。由于 $\phi(s_{t})$ 没有动机去编码任何不能影响、也不受智能体动作影响的环境特征，我们的智能体所学到的探索策略，对无法控制的环境因素是稳健的。

2 Curiosity-Driven Exploration

2 好奇心驱动的探索

Our agent is composed of two subsystems: a reward generator that outputs a curiosity-driven intrinsic reward signal and a policy that outputs a sequence of actions to maximize that reward signal. In addition to intrinsic rewards, the agent optionally may also receive some extrinsic reward from the environment. Let the intrinsic curiosity reward generated by the agent at time $t$ be $r_{t}^{i}$ and the extrinsic reward be $r_{t}^{e}$. The policy sub-system is trained to maximize the sum of these two rewards $r_{t}=r_{t}^{i}+r_{t}^{e}$, with $r_{t}^{e}$ mostly (if not always) zero.

我们的智能体由两个子系统组成：一个奖励生成器，输出由好奇心驱动的内在奖励信号；一个策略，输出一连串动作来最大化该奖励信号。除了内在奖励，智能体还可以选择性地从环境接收一些外在奖励。记智能体在时刻 $t$ 生成的内在好奇奖励为 $r_{t}^{i}$，外在奖励为 $r_{t}^{e}$。策略子系统被训练来最大化这两种奖励之和 $r_{t}=r_{t}^{i}+r_{t}^{e}$，其中 $r_{t}^{e}$ 多数时候（如果不是总是）为零。

We represent the policy $\pi(s_{t};\theta_{P})$ by a deep neural network with parameters $\theta_{P}$. Given the agent in state $s_{t}$, it executes the action $a_{t}\sim\pi(s_{t};\theta_{P})$ sampled from the policy. $\theta_{P}$ is optimized to maximize the expected sum of rewards,

我们用参数为 $\theta_{P}$ 的深度神经网络来表示策略 $\pi(s_{t};\theta_{P})$。智能体处于状态 $s_{t}$ 时，执行从策略中采样的动作 $a_{t}\sim\pi(s_{t};\theta_{P})$。$\theta_{P}$ 被优化以最大化期望累积奖励，

$$\max_{\theta_{P}}\mathbb{E}_{\pi(s_{t};\theta_{P})}[\Sigma_{t}r_{t}] \tag{1}$$

Unless specified otherwise, we use the notation $\pi(s)$ to denote the parameterized policy $\pi(s;\theta_{P})$. Our curiosity reward model can potentially be used with a range of policy learning methods; in the experiments discussed here, we use the asynchronous advantage actor critic policy gradient (A3C) (Mnih et al., 2016) for policy learning. Our main contribution is in designing an intrinsic reward signal based on prediction error of the agent’s knowledge about its environment that scales to high-dimensional continuous state spaces like images, bypasses the hard problem of predicting pixels and is unaffected by the unpredictable aspects of the environment that do not affect the agent.

除非另有说明，我们用记号 $\pi(s)$ 表示参数化策略 $\pi(s;\theta_{P})$。我们的好奇奖励模型原则上可以配合多种策略学习方法使用；在本文讨论的实验中，我们用异步优势演员—评论家策略梯度（A3C）（Mnih et al., 2016）来学习策略。我们的主要贡献在于设计一种基于智能体对环境认识之预测误差的内在奖励信号：它能扩展到图像这类高维连续状态空间，避开预测像素这一难题，并且不受那些不影响智能体、又难以预测的环境因素干扰。

2.1 Prediction error as curiosity reward

2.1 把预测误差当作好奇奖励

Making predictions in the raw sensory space (e.g. when $s_{t}$ corresponds to images) is undesirable not only because it is hard to predict pixels directly, but also because it is unclear if predicting pixels is even the right objective to optimize. To see why, consider using prediction error in the pixel space as the curiosity reward. Imagine a scenario where the agent is observing the movement of tree leaves in a breeze. Since it is inherently hard to model breeze, it is even harder to predict the pixel location of each leaf. This implies that the pixel prediction error will remain high and the agent will always remain curious about the leaves. But the motion of the leaves is inconsequential to the agent and therefore its continued curiosity about them is undesirable. The underlying problem is that the agent is unaware that some parts of the state space simply cannot be modeled and thus the agent can fall into an artificial curiosity trap and stall its exploration. Novelty-seeking exploration schemes that record the counts of visited states in a tabular form (or their extensions to continuous state spaces) also suffer from this issue. Measuring learning progress instead of prediction error has been proposed in the past as one solution (Schmidhuber, 1991). Unfortunately, there are currently no known computationally feasible mechanisms for measuring learning progress.

在原始感官空间里做预测（例如当 $s_{t}$ 对应图像时）并不可取：不仅因为直接预测像素很难，也因为预测像素到底是不是该优化的目标，本身并不清楚。为理解这一点，设想把像素空间中的预测误差当作好奇奖励。想象智能体正在观察微风中树叶的晃动。由于微风本身就很难建模，要预测每一片叶子的像素位置就更难。这意味着像素预测误差会一直偏高，智能体也就会对树叶永远保持好奇。但树叶的运动对智能体并无影响，因此它持续对此保持好奇是不应当的。深层问题在于：智能体并不知道状态空间的某些部分根本无法被建模，于是可能掉进人为的好奇心陷阱，探索停滞不前。那些以表格形式记录已访问状态计数的求新探索方案（以及它们向连续状态空间的扩展）也会受这个问题困扰。过去有人提出，用学习进展而不是预测误差来度量（Schmidhuber, 1991）。可惜，目前还没有已知的、在计算上可行的机制来度量学习进展。

If not the raw observation space, then what is the right feature space for making predictions so that the prediction error provides a good measure of curiosity? To answer this question, let us divide all sources that can modify the agent’s observations into three cases: (1) things that can be controlled by the agent; (2) things that the agent cannot control but that can affect the agent (e.g. a vehicle driven by another agent), and (3) things out of the agent’s control and not affecting the agent (e.g. moving leaves). A good feature space for curiosity should model (1) and (2) and be unaffected by (3). This latter is because, if there is a source of variation that is inconsequential for the agent, then the agent has no incentive to know about it.

如果不是原始观测空间，那么什么样的特征空间才适合用来做预测，从而使预测误差成为衡量好奇心的良好指标？为回答这个问题，不妨把所有可能改变智能体观测的来源分成三类：(1) 智能体能够控制的事物；(2) 智能体无法控制、但会影响到智能体的事物（例如另一智能体驾驶的车辆）；以及 (3) 既不受智能体控制、也不影响智能体的事物（例如晃动的树叶）。好的好奇特征空间应当建模 (1) 和 (2)，并不受 (3) 影响。后者是因为：如果某种变化来源对智能体无关紧要，智能体就没有动机去了解它。

2.2 Self-supervised prediction for exploration

2.2 用自监督预测来探索

Instead of hand-designing a feature representation for every environment, our aim is to come up with a general mechanism for learning feature representations such that the prediction error in the learned feature space provides a good intrinsic reward signal. We propose that such a feature space can be learned by training a deep neural network with two sub-modules: the first sub-module encodes the raw state $(s_{t})$ into a feature vector $\phi(s_{t})$ and the second sub-module takes as inputs the feature encoding $\phi(s_{t}),\phi(s_{t+1})$ of two consequent states and predicts the action $(a_{t})$ taken by the agent to move from state $s_{t}$ to $s_{t+1}$. Training this neural network amounts to learning function $g$ defined as:

我们不想为每个环境手工设计特征表示，而是希望找到一种通用的特征学习机制，使得在所学特征空间中的预测误差能提供良好的内在奖励信号。我们提出，这样的特征空间可以通过训练一个包含两个子模块的深度神经网络来得到：第一个子模块把原始状态 $(s_{t})$ 编码成特征向量 $\phi(s_{t})$；第二个子模块以两个相继状态的特征编码 $\phi(s_{t}),\phi(s_{t+1})$ 为输入，预测智能体从状态 $s_{t}$ 走到 $s_{t+1}$ 时所采取的动作 $(a_{t})$。训练这个神经网络，相当于学习如下定义的函数 $g$：

$$\hat{a}_{t}=g\Big(s_{t},s_{t+1};\theta_{I}\Big) \tag{2}$$

where, $\hat{a}_{t}$ is the predicted estimate of the action $a_{t}$ and the the neural network parameters $\theta_{I}$ are trained to optimize,

其中，$\hat{a}_{t}$ 是对动作 $a_{t}$ 的预测估计，神经网络参数 $\theta_{I}$ 被训练以优化

$$\min_{\theta_{I}}L_{I}(\hat{a}_{t},a_{t}) \tag{3}$$

where, $L_{I}$ is the loss function that measures the discrepancy between the predicted and actual actions. In case $a_{t}$ is discrete, the output of $g$ is a soft-max distribution across all possible actions and minimizing $L_{I}$ amounts to maximum likelihood estimation of $\theta_{I}$ under a multinomial distribution. The learned function $g$ is also known as the inverse dynamics model and the tuple $(s_{t},a_{t},s_{t+1})$ required to learn $g$ is obtained while the agent interacts with the environment using its current policy $\pi(s)$.

其中，$L_{I}$ 是衡量预测动作与实际动作差异的损失函数。当 $a_{t}$ 为离散时，$g$ 的输出是所有可能动作上的 soft-max 分布，最小化 $L_{I}$ 相当于在多项分布下对 $\theta_{I}$ 做极大似然估计。学到的函数 $g$ 也称作逆动力学模型；学习 $g$ 所需的三元组 $(s_{t},a_{t},s_{t+1})$，是智能体用当前策略 $\pi(s)$ 与环境交互时得到的。

In addition to inverse dynamics model, we train another neural network that takes as inputs $a_{t}$ and $\phi(s_{t})$ and predicts the feature encoding of the state at time step $t+1$,

除了逆动力学模型，我们还训练另一个神经网络：它以 $a_{t}$ 和 $\phi(s_{t})$ 为输入，预测时刻 $t+1$ 状态的特征编码，

$$\hat{\phi}(s_{t+1})=f\Big(\phi(s_{t}),a_{t};\theta_{F}\Big) \tag{4}$$

where $\hat{\phi}(s_{t+1})$ is the predicted estimate of $\phi(s_{t+1})$ and the neural network parameters $\theta_{F}$ are optimized by minimizing the loss function $L_{F}$:

其中 $\hat{\phi}(s_{t+1})$ 是对 $\phi(s_{t+1})$ 的预测估计，神经网络参数 $\theta_{F}$ 通过最小化损失函数 $L_{F}$ 来优化：

$$L_{F}\Big(\phi(s_{t}),\hat{\phi}(s_{t+1})\Big)=\frac{1}{2}\|\hat{\phi}(s_{t+1})-\phi(s_{t+1})\|_{2}^{2} \tag{5}$$

The learned function $f$ is also known as the forward dynamics model. The intrinsic reward signal $r_{t}^{i}$ is computed as,

学到的函数 $f$ 也称作前向动力学模型。内在奖励信号 $r_{t}^{i}$ 计算为：

$$r_{t}^{i}=\frac{\eta}{2}\|\hat{\phi}(s_{t+1})-\phi(s_{t+1})\|_{2}^{2} \tag{6}$$

where $\eta>0$ is a scaling factor. In order to generate the curiosity based intrinsic reward signal, we jointly optimize the forward and inverse dynamics loss described in equations 3 and 5 respectively. The inverse model learns a feature space that encodes information relevant for predicting the agent’s actions only and the forward model makes predictions in this feature space. We refer to this proposed curiosity formulation as Intrinsic Curiosity Module (ICM). As there is no incentive for this feature space to encode any environmental features that are not influenced by the agent’s actions, our agent will receive no rewards for reaching environmental states that are inherently unpredictable and its exploration strategy will be robust to the presence of distractor objects, changes in illumination, or other nuisance sources of variation in the environment. See Figure 2 for illustration of the formulation.

其中 $\eta>0$ 是一个缩放因子。为了生成基于好奇心的内在奖励信号，我们联合优化方程 3 和方程 5 中描述的前向与逆动力学损失。逆模型学到一个只编码预测智能体动作所需信息的特征空间，前向模型则在这个特征空间里做预测。我们把这种好奇心表述称为内在好奇心模块（Intrinsic Curiosity Module，ICM）。由于这个特征空间没有动机去编码那些不受智能体动作影响的环境特征，我们的智能体到达本质上不可预测的环境状态时不会获得奖励，其探索策略因此对干扰物体、光照变化或其他恼人的变化来源保持稳健。图 2 展示了这一表述。

The use of inverse models has been investigated to learn features for recognition tasks (Jayaraman & Grauman, 2015; Agrawal et al., 2015). Agrawal et al. (2016) constructed a joint inverse-forward model to learn feature representation for the task of pushing objects. However, they only used the forward model as a regularizer for training the inverse model features, while we make use of the error in the forward model predictions as the curiosity reward for training our agent’s policy.

用逆模型来学习识别任务的特征，此前已有研究（Jayaraman & Grauman, 2015; Agrawal et al., 2015）。Agrawal 等人（2016）构建了联合的逆—前向模型，为推物体这一任务学习特征表示。不过，他们只把前向模型当作训练逆模型特征的正则项；而我们则把前向模型预测误差用作训练智能体策略的好奇奖励。

The overall optimization problem that is solved for learning the agent is a composition of equations 1, 3 and 5 and can be written as,

为学习智能体而求解的总体优化问题，是方程 1、3 和 5 的组合，可以写成：

$$\min_{\theta_{P},\theta_{I},\theta_{F}}\Bigg[-\lambda\mathbb{E}_{\pi(s_{t};\theta_{P})}[\Sigma_{t}r_{t}]+(1-\beta)L_{I}+\beta L_{F}\Bigg] \tag{7}$$

where $0\leq\beta\leq 1$ is a scalar that weighs the inverse model loss against the forward model loss and $\lambda>0$ is a scalar that weighs the importance of the policy gradient loss against the importance of learning the intrinsic reward signal.

其中 $0\leq\beta\leq 1$ 是权衡逆模型损失与前向模型损失的标量，$\lambda>0$ 则权衡策略梯度损失与学习内在奖励信号的相对重要性。

3 Experimental Setup

3 实验设置

To evaluate our curiosity module on its ability to improve exploration and provide generalization to novel scenarios, we will use two simulated environments. This section describes the details of the environments and the experimental setup.

为评估我们的好奇心模块能否改进探索、并泛化到新场景，我们将使用两个仿真环境。本节说明环境细节与实验设置。

Environments

The first environment we evaluate on is the VizDoom (Kempka et al., 2016) game. We consider the Doom 3-D navigation task where the action space of the agent consists of four discrete actions – move forward, move left, move right and no-action. Our testing setup in all the experiments is the ‘DoomMyWayHome-v0’ environment which is available as part of OpenAI Gym (Brockman et al., 2016). Episodes are terminated either when the agent finds the vest or if the agent exceeds a maximum of 2100 time steps. The map consists of $9$ rooms connected by corridors and the agent is tasked to reach some fixed goal location from its spawning location. The agent is only provided a sparse terminal reward of +1 if it finds the vest and zero otherwise. For generalization experiments, we pre-train on a different map with different random textures from (Dosovitskiy & Koltun, 2016) and each episode lasts for 2100 time steps. Sample frames from VizDoom are shown in Figure 3(a), and maps are explained in Figure 4. It takes approximately 350 steps for an optimal policy to reach the vest location from the farthest room in this map (sparse reward).

环境

我们评估的第一个环境是 VizDoom 游戏（Kempka et al., 2016）。我们考虑 Doom 的三维导航任务，智能体的动作空间由四个离散动作组成：前进、左移、右移和不行动。所有实验中的测试设置都是 ‘DoomMyWayHome-v0’ 环境，它是 OpenAI Gym 的一部分（Brockman et al., 2016）。回合在智能体找到背心、或超过最多 2100 个时间步时终止。地图由 $9$ 个房间经走廊相连，智能体的任务是从出生点到达某个固定的目标位置。智能体只在找到背心时获得 +1 的稀疏终止奖励，否则为零。在泛化实验中，我们在一张来自（Dosovitskiy & Koltun, 2016）、带有不同随机纹理的另一张地图上做预训练，每回合持续 2100 个时间步。VizDoom 的示例帧见图 3(a)，地图说明见图 4。在这张地图上，最优策略从最远的房间走到背心位置大约需要 350 步（稀疏奖励）。

Our second environment is the classic Nintendo game Super Mario Bros (Paquette, 2016). We consider four levels of the game: pre-training on the first level and showing generalization on the subsequent levels. In this setup, we reparametrize the action space of the agent into 14 unique actions following (Paquette, 2016). This game is played using a joystick allowing for multiple simultaneous button presses, where the duration of the press affects what action is being taken. This property makes the game particularly hard, e.g. to make a long jump over tall pipes or wide gaps, the agent needs to predict the same action up to 12 times in a row, introducing long-range dependencies. All our experiments on Mario are trained using curiosity signal only, without any reward from the game.

第二个环境是经典任天堂游戏 Super Mario Bros（Paquette, 2016）。我们考虑游戏的四个关卡：在第一关上预训练，并在后续关卡上展示泛化。在这一设置中，我们按照（Paquette, 2016）把智能体的动作空间重新参数化为 14 个独特动作。这款游戏用手柄游玩，允许多个按键同时按下，而且按下的持续时间会影响实际执行的动作。这一特性使游戏特别困难：例如，要越过高水管或宽缺口做一次远跳，智能体需要连续预测同一动作多达 12 次，从而引入了长程依赖。我们在 Mario 上的全部实验都只使用好奇心信号训练，不使用游戏提供的任何奖励。

(a) Input snapshot in VizDoom

（a）VizDoom 中的输入截图

Training details

All agents in this work are trained using visual inputs that are pre-processed in manner similar to (Mnih et al., 2016). The input RGB images are converted into gray-scale and re-sized to $42\times 42$. In order to model temporal dependencies, the state representation ($s_{t}$) of the environment is constructed by concatenating the current frame with the three previous frames. Closely following (Mnih et al., 2015; Mnih et al., 2016), we use action repeat of four during training time in VizDoom and action repeat of six in Mario. However, we sample the policy without any action repeat during inference. Following the asynchronous training protocol in A3C, all the agents were trained asynchronously with twenty workers using stochastic gradient descent. We used ADAM optimizer with its parameters not shared across the workers.

训练细节

本文中的所有智能体都使用与（Mnih et al., 2016）类似的方式预处理的视觉输入进行训练。输入的 RGB 图像被转换成灰度，并缩放为 $42\times 42$。为了建模时间依赖，环境的状态表示（$s_{t}$）由当前帧与之前三帧拼接而成。我们紧密遵循（Mnih et al., 2015; Mnih et al., 2016）：训练时在 VizDoom 中使用四步动作重复，在 Mario 中使用六步动作重复。不过，推理时我们从策略中采样，不再做动作重复。按照 A3C 的异步训练协议，所有智能体都用二十个 worker、以随机梯度下降异步训练。我们使用 ADAM 优化器，其参数不在 worker 之间共享。

(a) Train Map Scenario

（a）训练地图场景

A3C architecture

The input state $s_{t}$ is passed through a sequence of four convolution layers with 32 filters each, kernel size of 3x3, stride of 2 and padding of 1. An exponential linear unit (ELU; (Clevert et al., 2015)) is used after each convolution layer. The output of the last convolution layer is fed into a LSTM with 256 units. Two seperate fully connected layers are used to predict the value function and the action from the LSTM feature representation.

A3C 架构

输入状态 $s_{t}$ 依次通过四个卷积层，每层 32 个滤波器，卷积核大小为 3x3，步幅为 2，填充为 1。每个卷积层之后使用指数线性单元（ELU；（Clevert et al., 2015））。最后一个卷积层的输出送入一个有 256 个单元的 LSTM。再用两个分开的全连接层，分别从 LSTM 特征表示预测价值函数和动作。

Intrinsic Curiosity Module (ICM) architecture

The intrinsic curiosity module consists of the forward and the inverse model. The inverse model first maps the input state $(s_{t})$ into a feature vector $\phi(s_{t})$ using a series of four convolution layers, each with 32 filters, kernel size 3x3, stride of 2 and padding of 1. ELU non-linearity is used after each convolution layer. The dimensionality of $\phi(s_{t})$ (i.e. the output of the fourth convolution layer) is 288. For the inverse model, $\phi(s_{t})$ and $\phi(s_{t+1})$ are concatenated into a single feature vector and passed as inputs into a fully connected layer of 256 units followed by an output fully connected layer with 4 units to predict one of the four possible actions. The forward model is constructed by concatenating $\phi(s_{t})$ with $a_{t}$ and passing it into a sequence of two fully connected layers with 256 and 288 units respectively. The value of $\beta$ is $0.2$, and $\lambda$ is 0.1. The Equation (7) is minimized with learning rate of $1e-3$.

内在好奇心模块（ICM）架构

内在好奇心模块由前向模型和逆模型组成。逆模型首先用一系列四个卷积层把输入状态 $(s_{t})$ 映射成特征向量 $\phi(s_{t})$，每层 32 个滤波器，卷积核大小 3x3，步幅为 2，填充为 1。每个卷积层之后使用 ELU 非线性。$\phi(s_{t})$ 的维度（也就是第四个卷积层的输出）为 288。对逆模型而言，$\phi(s_{t})$ 和 $\phi(s_{t+1})$ 被拼接成一个特征向量，送入一个 256 单元的全连接层，再接一个有 4 个单元的输出全连接层，用以预测四个可能动作中的一个。前向模型则把 $\phi(s_{t})$ 与 $a_{t}$ 拼接，送入两个依次相连、分别有 256 和 288 个单元的全连接层。$\beta$ 的取值为 $0.2$，$\lambda$ 为 0.1。方程 (7) 以 $1e-3$ 的学习率最小化。

(a) “dense reward” setting

（a）“稠密奖励”设定

Baseline Methods

‘ICM + A3C’ denotes our full algorithm which combines intrinsic curiosity model with A3C. Across different experiments, we compare our approach with three baselines. First is the vanilla ‘A3C’ algorithm with $\epsilon$-greedy exploration. Second is ‘ICM-pixels + A3C’, which is a variant of our ICM without the inverse model, and has curiosity reward dependent only on the forward model loss in predicting next observation in pixel space. To design this, we remove the inverse model layers and append deconvolution layers to the forward model. ICM-pixels is close to ICM in architecture but incapable of learning embedding that is invariant to the uncontrollable part of environment. Note that ICM-pixels is representative of previous methods which compute information gain by directly using the observation space (Stadie et al., 2015; Schmidhuber, 2010). We show that directly using observation space for computing curiosity is significantly worse than learning an embedding as in ICM. Finally, we include comparison with state-of-the-art exploration methods based on variational information maximization (VIME) (Houthooft et al., 2016) which is trained with TRPO.

基线方法

‘ICM + A3C’ 表示我们把内在好奇心模型与 A3C 结合的完整算法。在不同实验中，我们与三条基线进行比较。第一条是带 $\epsilon$-greedy 探索的原版 ‘A3C’ 算法。第二条是 ‘ICM-pixels + A3C’，它是去掉逆模型的 ICM 变体，好奇奖励只依赖于在像素空间中预测下一观测的前向模型损失。为设计这一变体，我们去掉逆模型各层，并在前向模型后接上反卷积层。ICM-pixels 在架构上接近 ICM，但无法学到对不可控环境部分不变的嵌入。需要指出，ICM-pixels 代表了以往那些直接在观测空间上计算信息增益的方法（Stadie et al., 2015; Schmidhuber, 2010）。我们会说明，直接用观测空间来计算好奇心，显著差于像 ICM 那样学习一个嵌入。最后，我们还与基于变分信息最大化（VIME）（Houthooft et al., 2016）、并用 TRPO 训练的先进探索方法做了比较。

4 Experiments

4 实验

We qualitatively and quantitatively evaluate the performance of the learned policy with and without the proposed intrinsic curiosity signal in two environments, VizDoom and Super Mario Bros. Three broad settings are evaluated: a) sparse extrinsic reward on reaching a goal (Section 4.1); b) exploration with no extrinsic reward (Section 4.2); and c) generalization to novel scenarios (Section 4.3). In VizDoom generalization is evaluated on a novel map with novel textures, while in Mario it is evaluated on subsequent game levels.

我们在 VizDoom 和 Super Mario Bros 两个环境中，定性和定量地评估带与不带所提内在好奇信号的所学策略表现。评估覆盖三类设定：a）到达目标时获得稀疏外在奖励（第 4.1 节）；b）没有任何外在奖励的探索（第 4.2 节）；以及 c）泛化到新场景（第 4.3 节）。在 VizDoom 中，泛化在一张带有新纹理的新地图上评估；在 Mario 中，则在后续游戏关卡上评估。

4.1 Sparse Extrinsic Reward Setting

4.1 稀疏外在奖励设定

We perform extrinsic reward experiments on VizDoom using ‘DoomMyWayHome-v0’ setup described in Section 3. The extrinsic reward is sparse and only provided when the agent finds the goal (a vest) located at a fixed location in the map. We systematically varied the difficulty of this goal-directed exploration task by varying the distance between the initial spawning location of the agent and the location of the goal. A larger distance means that the chances of reaching the goal location by random exploration is lower and consequently the reward is said to be sparser.

我们在 VizDoom 上、使用第 3 节描述的 ‘DoomMyWayHome-v0’ 设置进行外在奖励实验。外在奖励是稀疏的，只有当智能体找到位于地图固定位置的目标（一件背心）时才会给出。我们通过改变智能体初始出生位置与目标位置之间的距离，系统地调整这一目标导向探索任务的难度。距离越大，靠随机探索到达目标位置的机会就越低，奖励也就被说成更稀疏。

Varying the degree of reward sparsity:

We consider three setups with “dense”, “sparse” and “very-sparse” rewards (see Figure 4(b)). In these settings, the reward is always terminal and the episode terminates upon reaching goal or after a maximum of 2100 steps. In the “dense” reward case, the agent is randomly spawned in any of the 17 possible spawning locations uniformly distributed across the map. This is not a hard exploration task because sometimes the agent is randomly initialized close to the goal and therefore by random $\epsilon$-greedy exploration it can reach the goal with reasonably high probability. In the “sparse” and “very sparse” reward cases, the agent is always spawned in Room-13 and Room-17 respectively which are $~270$ and $~350$ steps away from the goal under an optimal policy. A long sequence of directed actions is required to reach the goals from these rooms, making these settings hard goal directed exploration problems.

改变奖励稀疏程度：

我们考虑“稠密”“稀疏”和“非常稀疏”三种设置（见图 4(b)）。在这些设置中，奖励始终是终止性的，回合在到达目标或最多 2100 步后结束。在“稠密”奖励情形下，智能体在地图上均匀分布的 17 个可能出生点中被随机放置。这并不是一项很难的探索任务，因为有时智能体会被随机初始化到目标附近，因此靠基线的随机 $\epsilon$-greedy 探索，它有相当高的概率到达目标。在“稀疏”和“非常稀疏”奖励情形下，智能体分别总是出生在 Room-13 和 Room-17，在最优策略下距目标分别约 $~270$ 步和 $~350$ 步。要从这些房间到达目标，需要一长串有方向的动作，因此这些设置是困难的目标导向探索问题。

Figure 6: Evaluating the robustness of ICM to the presence of uncontrollable distractors in the environment. We created such a distractor by replacing 40% of the visual observation of the agent by white noise (see Figure 3(b)). The results show that while ICM succeeds most of the times, the pixel prediction model struggles.

图 6：评估 ICM 在环境中存在不可控干扰因素时的稳健性。我们把智能体视觉观测的 40% 替换成白噪声，来制造这样一种干扰（见图 3(b)）。结果表明，ICM 在多数时候都能成功，而像素预测模型则十分吃力。

Figure 7: Each column in the figure shows the visitation pattern of an agent exploring the environment. The red arrow shows the initial location and orientation of the agent at the start of the episode. Each room that the agent visits during its exploration of maximum 2100 steps has been colored. The first three columns (with maps colored in yellow) show the exploration strategy of an agent trained with curiosity driven internal reward signal only. The last two columns show the rooms visited by an agent conducting random exploration. The results clearly show that the curious agent trained with intrinsic rewards explores a significantly larger number of rooms as compared to a randomly exploring agent.

图 7：图中每一列展示一个智能体探索环境时的访问模式。红色箭头标出回合开始时智能体的初始位置和朝向。智能体在最多 2100 步的探索中访问过的每个房间都被着色。前三列（地图以黄色着色）展示只用好奇心驱动的内部奖励信号训练的智能体的探索策略。后两列展示进行随机探索的智能体所访问的房间。结果清楚地表明，用内在奖励训练的好奇智能体，探索到的房间数显著多于随机探索的智能体。

Results shown in Figure 5 indicate that while the performance of the baseline A3C degrades with sparser rewards, curious A3C agents are superior in all cases. In the “dense” reward case, curious agents learn much faster indicating more efficient exploration of the environment as compared to $\epsilon$-greedy exploration of the baseline agent. One possible explanation of the inferior performance of ICM-pixels in comparison to ICM is that in every episode the agent is spawned in one out of seventeen rooms with different textures. It is hard to learn a pixel-prediction model as the number of textures increases.

图 5 的结果表明：基线 A3C 的表现会随着奖励变稀疏而下降，而带好奇心的 A3C 智能体在所有情形下都更优。在“稠密”奖励情形中，好奇智能体学得更快，说明与基线智能体的 $\epsilon$-greedy 探索相比，它对环境的探索更有效。ICM-pixels 相对 ICM 表现较差，一种可能的解释是：每一回合智能体都会在十七个房间之一中出生，而这些房间有不同的纹理。随着纹理种类增加，学习像素预测模型会变得更难。

In the “sparse” reward case, as expected, the baseline A3C agent fails to solve the task, while the curious A3C agent is able to learn the task quickly. Note that ICM-pixels and ICM have similar convergence because, with a fixed spawning location of the agent, the ICM-pixels encounters the same textures at the starting of each episode which makes learning the pixel-prediction model easier as compared to the “dense” reward case. Finally, in the “very sparse” reward case, both the A3C agent and ICM-pixels never succeed, while the ICM agent achieves a perfect score in $66\%$ of the random runs. This indicates that ICM is better suited than ICM-pixels and vanilla A3C for hard goal directed exploration tasks.

在“稀疏”奖励情形中，正如所料，基线 A3C 智能体无法完成任务，而带好奇心的 A3C 智能体能够很快学会。需要指出，ICM-pixels 与 ICM 的收敛相近，因为智能体的出生位置固定后，ICM-pixels 在每回合开始时都会遇到相同的纹理，这使学习像素预测模型比在“稠密”奖励情形下更容易。最后，在“非常稀疏”的奖励情形中，A3C 智能体和 ICM-pixels 从未成功，而 ICM 智能体在 $66\%$ 的随机运行中取得了满分。这说明，对于困难的目标导向探索任务，ICM 比 ICM-pixels 和原版 A3C 更合适。

Robustness to uncontrollable dynamics

For testing the robustness of the proposed ICM formulation to changes in the environment that do not affect the agent, we augmented the agent’s observation with a fixed region of white noise which made up $40\%$ of the image (see Figure 3(b)). In VizDoom 3-D navigation, ideally the agent should be unaffected by this noise as the noise does not affect the agent in anyway and is merely a nuisance. Figure 6 compares the performance of ICM against some baseline methods on the “sparse” reward setup described above. While, the proposed ICM agent achieves a perfect score, ICM-pixels suffers significantly despite having succeeded at the “sparse reward” task when the inputs were not augmented with any noise (see Figure 5(b)). This indicates that in contrast to ICM-pixels, ICM is insensitive to nuisance changes in the environment.

对不可控动态的稳健性

为检验所提 ICM 表述对那些不影响智能体的环境变化是否稳健，我们在智能体的观测上叠加了一块固定的白噪声区域，约占图像的 $40\%$（见图 3(b)）。在 VizDoom 三维导航中，理想情况下智能体不应当受这一噪声影响，因为它无论如何都不会影响到智能体，只是一种干扰。图 6 在上文描述的“稀疏”奖励设置上，比较了 ICM 与若干基线方法的表现。所提 ICM 智能体取得了满分，而 ICM-pixels 尽管在输入未叠加任何噪声时已经成功完成了“稀疏奖励”任务（见图 5(b)），此时却明显受损。这说明，与 ICM-pixels 不同，ICM 对环境中恼人的变化并不敏感。

Comparison to TRPO-VIME

We now compare our curious agent against variational information maximization agent trained with TRPO (Houthooft et al., 2016) for the VizDoom “sparse” reward setup described above. TRPO is in general more sample efficient than A3C but takes a lot more wall-clock time. We do not show these results in plots because TRPO and A3C have different setups. The hyper-parameters and accuracy for the TRPO and VIME results follow from the concurrent work (Fu et al., 2017). Despite the sample efficiency of TRPO, we see that our ICM agents work significantly better than TRPO and TRPO-VIME, both in terms of convergence rate and accuracy. Results are shown in the Table below:

与 TRPO-VIME 的比较

我们接下来把好奇智能体，与用 TRPO 训练的变分信息最大化智能体（Houthooft et al., 2016）在上文描述的 VizDoom“稀疏”奖励设置上进行比较。TRPO 通常比 A3C 更样本高效，但需要多得多的墙上时钟时间。由于 TRPO 和 A3C 的设置不同，我们没有把这些结果画在图里。TRPO 和 VIME 结果的超参数与精度沿用同期工作（Fu et al., 2017）。尽管 TRPO 更样本高效，我们看到 ICM 智能体在收敛速度和精度上都显著优于 TRPO 和 TRPO-VIME。结果见下表：

As a sanity check, we replaced the curiosity network with random noise sources and used them as the curiosity reward. We performed systematic sweep across different distribution parameters in the “sparse” reward case: uniform, Gaussian and Laplacian. We found that none of these agents were able to reach the goal showing that our curiosity module does not learn degenerate solutions.

作为健全性检查，我们把好奇心网络替换成随机噪声源，并把它们当作好奇奖励。我们在“稀疏”奖励情形下，对不同分布参数做了系统扫描：均匀分布、高斯分布和拉普拉斯分布。我们发现，这些智能体没有一个能够到达目标，说明我们的好奇心模块并没有学到退化解。

4.2 No Reward Setting

4.2 无奖励设定

A good exploration policy is one which allows the agent to visit as many states as possible even without any goals. In the case of 3-D navigation, we expect a good exploration policy to cover as much of the map as possible; in the case of playing a game, we expect it to visit as many game states as possible. In order to test if our agent can learn a good exploration policy, we trained it on VizDoom and Mario without any rewards from the environment. We then evaluated what portion of the map was explore (for VizDoom), and how much progress it made (for Mario) in this setting. To our surprise, we have found that in both cases, the no-reward agent was able to perform quote well (see video at http://pathak22.github.io/noreward_rl/).

好的探索策略应当让智能体在没有任何目标时也能访问尽可能多的状态。对三维导航而言，我们期望好的探索策略尽可能覆盖地图；对玩游戏而言，我们期望它访问尽可能多的游戏状态。为了检验我们的智能体能否学到好的探索策略，我们在 VizDoom 和 Mario 上、不使用环境提供的任何奖励来训练它。然后我们评估在这一设置下地图被探索了多大一部分（对 VizDoom），以及它取得了多大进展（对 Mario）。出乎我们意料的是，在这两种情形下，无奖励智能体都表现得相当好（视频见 http://pathak22.github.io/noreward_rl/）。

VizDoom: Coverage during Exploration. An agent trained with no extrinsic rewards was able to learn to navigate corridors, walk between rooms and explore many rooms in the 3-D Doom environment. On many occasions the agent traversed the entire map and reached rooms that were farthest away from the room it was initialized in. Given that the episode terminates in 2100 steps and farthest rooms are over 250 steps away (for an optimally-moving agent), this result is quite remarkable, demonstrating that it is possible to learn useful skills without the requirement of any external supervision of rewards. Example explorations are shown in Figure 7. The first 3 maps show our agent explore a much larger state space without any extrinsic signal, compared to a random exploration agent (last two maps), which often has hard time getting around local minima of state spaces, e.g. getting stuck against a wall and not able to move (see video).

VizDoom：探索过程中的覆盖。 一个不使用任何外在奖励训练的智能体，能够学会在三维 Doom 环境中沿走廊导航、在房间之间行走，并探索许多房间。很多时候，智能体走遍了整张地图，到达了距离其初始化房间最远的那些房间。考虑到回合在 2100 步时终止，而最远的房间（对最优移动的智能体而言）也在 250 步以外，这一结果相当突出：它表明，不需要任何外部的奖励监督，也可以学到有用的技能。探索示例如图 7 所示。前三张地图显示，我们的智能体在没有任何外在信号时探索了大得多的状态空间；相比之下，随机探索智能体（后两张地图）常常难以摆脱状态空间中的局部极小，例如卡在墙边无法移动（见视频）。

Mario: Learning to play with no rewards. We train our agent in the Super Mario World using only curiosity based signal. Without any extrinsic reward from environment, our Mario agent can learn to cross over $30\%$ of Level-1. The agent received no reward for killing or dodging enemies or avoiding fatal events, yet it automatically discovered these behaviors (see video). One possible reason is because getting killed by the enemy will result in only seeing a small part of the game space, making its curiosity saturate. In order to remain curious, it is in the agent’s interest to learn how to kill and dodge enemies so that it can reach new parts of the game space. This suggests that curiosity provides indirect supervision for learning interesting behaviors.

Mario：在没有奖励的情况下学习游玩。 我们在 Super Mario World 中只用基于好奇心的信号训练智能体。在没有任何来自环境的外在奖励时，我们的 Mario 智能体能学会通过 Level-1 的 $30\%$ 以上。智能体并没有因为击杀或躲避敌人、或避免致命事件而获得奖励，却自动发现了这些行为（见视频）。一种可能的原因是：被敌人杀死后，它只能看到游戏空间的一小部分，好奇心很快就会饱和。为了保持好奇，学会击杀和躲避敌人以便到达游戏空间的新区域，符合智能体的利益。这说明，好奇心为学习有趣的行为提供了间接监督。

To the best of our knowledge, this is the first demonstration where the agent learns to navigate in a 3D environment and discovers how to play a game by making use of relatively complex visual imagery directly from pixels, without any extrinsic rewards. There are several prior works that use reinforcement learning to navigate in 3D environments from pixel inputs or playing ATARI games such as (Mnih et al., 2015; Mnih et al., 2016; Mirowski et al., 2017), but they rely on intermediate external rewards provided by the environment.

据我们所知，这是首次有演示表明：智能体可以直接从像素出发、利用相对复杂的视觉图像，在三维环境中学会导航，并发现如何玩游戏，而且不使用任何外在奖励。此前已有若干工作用强化学习从像素输入在三维环境中导航或玩 ATARI 游戏，例如（Mnih et al., 2015; Mnih et al., 2016; Mirowski et al., 2017），但它们依赖于环境提供的中间外部奖励。

4.3 Generalization to Novel Scenarios

4.3 泛化到新场景

In the previous section we showed that our agent learns to explore large parts of the space where its curiosity-driven exploration policy was trained. However, it remains unclear whether the agent has done this by learning “generalized skills” for efficiently exploring its environment, or if it simply memorized the training set. In other words we would like to know, when exploring a space, how much of the learned behavior is specific to that particular space and how much is general enough to be useful in novel scenarios? To investigate this question, we train a no reward exploratory behavior in one scenario (e.g. Level-1 of Mario) and then evaluate the resulting exploration policy in three different ways: a) apply the learned policy “as is” to a new scenario; b) adapt the policy by fine-tuning with curiosity reward only; c) adapt the policy to maximize some extrinsic reward. Happily, in all three cases, we observe some promising generalization results:

上一节我们说明，智能体能够在训练其好奇心驱动探索策略的空间中，探索很大一部分区域。不过，目前仍不清楚：智能体是靠学会了用于高效探索环境的“可泛化技能”做到这一点的，还是只是记住了训练集。换句话说，我们想知道：在探索一个空间时，所学行为有多少是针对那个特定空间的，又有多少足够通用，从而能在新场景中有用？为研究这个问题，我们在一种场景中（例如 Mario 的 Level-1）训练无奖励的探索行为，然后用三种不同方式评估得到的探索策略：a）把学到的策略“原样”应用到新场景；b）只用好奇奖励做微调来适应策略；c）调整策略以最大化某些外在奖励。令人高兴的是，在这三种情形下，我们都观察到了一些有希望的泛化结果：

Level Ids	Level-1	Level-2	Level-3
Accuracy	Scratch	Run as is	Fine-tuned	Scratch	Scratch	Run as is	Fine-tuned	Scratch	Scratch
Iterations	1.5M	0	1.5M	1.5M	3.5M	0	1.5M	1.5M	5.0M
Mean $\pm$ stderr	711 $\pm$ 59.3	31.9 $\pm$ 4.2	466 $\pm$ 37.9	399.7 $\pm$ 22.5	455.5 $\pm$ 33.4	319.3 $\pm$ 9.7	97.5 $\pm$ 17.4	11.8 $\pm$ 3.3	42.2 $\pm$ 6.4
$\%$ distance $>$ 200	50.0 $\pm$ 0.0	0	64.2 $\pm$ 5.6	88.2 $\pm$ 3.3	69.6 $\pm$ 5.7	50.0 $\pm$ 0.0	1.5 $\pm$ 1.4	0	0
$\%$ distance $>$ 400	35.0 $\pm$ 4.1	0	63.6 $\pm$ 6.6	33.2 $\pm$ 7.1	51.9 $\pm$ 5.7	8.4 $\pm$ 2.8	0	0	0
$\%$ distance $>$ 600	35.8 $\pm$ 4.5	0	42.6 $\pm$ 6.1	14.9 $\pm$ 4.4	28.1 $\pm$ 5.4	0	0	0	0

Table 1: Quantitative evaluation of the agent trained to play Super Mario Bros. using only curiosity signal without any rewards from the game. Our agent was trained with no rewards in Level-1. We then evaluate the agent’s policy both when it is run “as is”, and further fine-tuned on subsequent levels. The results are compared to settings when Mario agent is train from scratch in Level-2,3 using only curiosity without any extrinsic rewards. Evaluation metric is based on the distance covered by the Mario agent.

关卡编号	Level-1	Level-2	Level-3
精度	Scratch	Run as is	Fine-tuned	Scratch	Scratch	Run as is	Fine-tuned	Scratch	Scratch
迭代次数	1.5M	0	1.5M	1.5M	3.5M	0	1.5M	1.5M	5.0M
均值 $\pm$ stderr	711 $\pm$ 59.3	31.9 $\pm$ 4.2	466 $\pm$ 37.9	399.7 $\pm$ 22.5	455.5 $\pm$ 33.4	319.3 $\pm$ 9.7	97.5 $\pm$ 17.4	11.8 $\pm$ 3.3	42.2 $\pm$ 6.4
$\%$ 距离 $> 200$	50.0 $\pm$ 0.0	0	64.2 $\pm$ 5.6	88.2 $\pm$ 3.3	69.6 $\pm$ 5.7	50.0 $\pm$ 0.0	1.5 $\pm$ 1.4	0	0
$\%$ 距离 $> 400$	35.0 $\pm$ 4.1	0	63.6 $\pm$ 6.6	33.2 $\pm$ 7.1	51.9 $\pm$ 5.7	8.4 $\pm$ 2.8	0	0	0
$\%$ 距离 $> 600$	35.8 $\pm$ 4.5	0	42.6 $\pm$ 6.1	14.9 $\pm$ 4.4	28.1 $\pm$ 5.4	0	0	0	0

表 1：对只用好奇心信号、不使用游戏中任何奖励来训练、玩 Super Mario Bros 的智能体所做的定量评估。我们的智能体在 Level-1 上不使用奖励进行训练。随后，我们分别评估把该策略“原样”运行、以及在后续关卡上进一步微调时的表现。结果与 Mario 智能体在 Level-2、3 上从零开始、只用好奇心、不使用任何外在奖励训练的设置进行比较。评估指标基于 Mario 智能体所走过的距离。

Evaluate “as is”:

We evaluate the policy trained by maximizing curiosity on Level-1 of Mario on subsequent levels without adapting the learned policy in any way. We measure the distance covered by the agent as a result of executing this policy on Levels 1, 2, and 3, as shown in Table 1. We note that the policy performs surprisingly well on Level 3, suggesting good generalization, despite the fact that Level-3 has different structures and enemies compared to Level-1. However, note that the running “as is” on Level-2 does not do well. At first, this seems to contradict the generalization results on Level-3. However, note that Level-3 has similar global visual appearance (day world with sunlight) to Level-1, whereas Level-2 is significantly different (night world). If this is indeed the issue, then it should be possible to quickly adapt the exploration policy to Level-2 with a little bit of “fine-tuning”. We will explore this below.

“原样”评估：

我们把在 Mario 的 Level-1 上通过最大化好奇心训练得到的策略，不经任何适应，直接在后续关卡上评估。我们测量执行该策略时智能体在 Level-1、2 和 3 上覆盖的距离，如表 1 所示。我们注意到，该策略在 Level-3 上的表现出奇地好，说明泛化良好，尽管 Level-3 与 Level-1 相比有不同的结构和敌人。不过需要指出，“原样”跑在 Level-2 上表现并不好。这起初似乎与 Level-3 上的泛化结果相矛盾。但请注意，Level-3 的整体视觉外观（有阳光的白昼世界）与 Level-1 相似，而 Level-2 则显著不同（夜晚世界）。如果这确实是症结所在，那么只需稍作“微调”，探索策略就应当能很快适应 Level-2。我们将在下面探讨这一点。

Fine-tuning with curiosity only:

From Table 1 we see that when the agent pre-trained (using only curiosity as reward) on Level-1 is fine-tuned (using only curiosity as reward) on Level-2 it quickly overcomes the mismatch in global visual appearance and achieves a higher score than training from scratch with the same number of iterations. Interestingly, training “from scratch” on Level-2 is worse than the fine-tuned policy, even when training for more iterations than pre-training + fine-tuning combined. One possible reason is that Level-2 is more difficult than Level-1, so learning the basic skills such as moving, jumping, and killing enemies from scratch is much more dangerous than in the relative “safety” of Level-1. This result, therefore might suggest that first pre-training on an earlier level and then fine-tuning on a later one produces a form of curriculum which aids learning and generalization. In other words, the agent is able to use the knowledge it acquired by playing Level-1 to better explore the subsequent levels. Of course, the game designers do this on purpose to allow the human players to gradually learn to play the game.

只用好奇心微调：

从表 1 可以看到，当智能体先在 Level-1 上预训练（只用好奇心作为奖励），再在 Level-2 上微调（同样只用好奇心作为奖励）时，它能很快克服整体视觉外观的不匹配，并在相同迭代次数下取得高于从零训练的分数。有意思的是，即便从零训练的迭代次数超过“预训练 + 微调”之和，Level-2 上“从零开始”的训练仍差于微调后的策略。一种可能的原因是 Level-2 比 Level-1 更难，因此从零学习移动、跳跃和击杀敌人等基本技能，要比在相对“安全”的 Level-1 中危险得多。这一结果因此可能说明：先在较早关卡上预训练、再在较晚关卡上微调，会产生一种有助于学习和泛化的课程。换句话说，智能体能够利用通过玩 Level-1 获得的知识，更好地探索后续关卡。当然，游戏设计者正是有意这样做，好让人类玩家逐步学会玩游戏。

However, interestingly, fine-tuning the exploration policy pre-trained on Level-1 to Level-3 deteriorates the performance, compared to running “as is”. This is because Level-3 is very hard for the agent to cross beyond a certain point – the agent hits a curiosity blockade and is unable to make any progress. As the agent has already learned about parts of the environment before the hard point, it receives almost no curiosity reward and as a result it attempts to update its policy with almost zero intrinsic rewards and the policy slowly degenerates. This behavior is vaguely analogous to boredom, where if the agent is unable to make progress it gets bored and stops exploring.

不过有意思的是，把在 Level-1 上预训练的探索策略微调到 Level-3，反而比“原样”运行表现更差。这是因为 Level-3 对智能体来说，超过某一点之后就非常难以跨越——智能体撞上了一道好奇心封锁，无法再取得任何进展。由于智能体已经对那个困难点之前的环境部分有所了解，它几乎收不到好奇奖励，于是试图用几乎为零的内在奖励来更新策略，策略便慢慢退化。这种行为隐约类似于厌倦：如果智能体无法取得进展，它就会感到无聊并停止探索。

Figure 8: Performance of ICM + A3C agents on the test set of VizDoom in the “very sparse” reward case. Fine-tuned models learn the exploration policy without any external rewards on the training maps and are then fine-tuned on the test map. The scratch models are directly trained on the test map. The fine-tuned ICM + A3C significantly outperforms ICM + A3C indicating that our curiosity formulation is able to learn generalizable exploration policies. The pixel prediction based ICM agent completely fail. Note that textures are also different in train and test.

图 8：ICM + A3C 智能体在 VizDoom 测试集、“非常稀疏”奖励情形下的表现。微调模型先在训练地图上不使用任何外部奖励学习探索策略，再在测试地图上微调。从零训练的模型则直接在测试地图上训练。微调后的 ICM + A3C 显著优于从零训练的 ICM + A3C，说明我们的好奇心表述能够学到可泛化的探索策略。基于像素预测的 ICM 智能体则完全失败。需要指出，训练与测试中的纹理也不同。

Fine-tuning with extrinsic rewards:

If it is the case that the agent has actually learned useful exploratory behavior, then it should be able to learn quicker than starting from scratch even when external rewards are provided by environment. We perform this evaluation on VizDoom where we pre-train the agent using curiosity reward on a map showed in Figure 4(a). We then test on the “very sparse” reward setting of ‘DoomMyWayHome-v0’ environment which uses a different map with novel textures (see Figure 4(b)) as described earlier in Section 4.1.

用外在奖励微调：

如果智能体确实已经学到了有用的探索行为，那么即便环境提供外部奖励，它也应当比从零开始学得更快。我们在 VizDoom 上做这一评估：先在图 4(a) 所示的地图上用好奇奖励预训练智能体，然后在 ‘DoomMyWayHome-v0’ 环境的“非常稀疏”奖励设置上测试——该设置使用一张带有全新纹理的不同地图（见图 4(b)），细节见前文第 4.1 节。

Results in Figure 8 show that the ICM agent pre-trained only with curiosity and then fine-tuned with external reward learns faster and achieves higher reward than an ICM agent trained from scratch to jointly maximize curiosity and the external rewards. This result confirms that the learned exploratory behavior is also useful when the agent is required to achieve goals specified by the environment. It is also worth noting that ICM-pixels does not generalize to this test environment. This indicates that the proposed mechanism of measuring curiosity is significantly better for learning skills that generalize as compared to measuring curiosity in the raw sensory space.

图 8 的结果表明：先只用好奇心预训练、再用外部奖励微调的 ICM 智能体，学得更快，取得的奖励也高于从零开始、联合最大化好奇心与外部奖励训练的 ICM 智能体。这一结果证实，学到的探索行为在智能体需要完成由环境指定的目标时同样有用。同样值得指出的是，ICM-pixels 无法泛化到这一测试环境。这说明，与在原始感官空间中度量好奇心相比，所提的好奇心度量机制更有利于学习能够泛化的技能。

5 Related Work

5 相关工作

Curiosity-driven exploration is a well studied topic in the reinforcement learning literature and a good summary can be found in (Oudeyer et al., 2007; Oudeyer & Kaplan, 2009). Schmidhuber  (1991; 2010) and Sun et al. (2011) use surprise and compression progress as intrinsic rewards. Classic work of Kearns et al. (1999) and Brafman et al. (2002) propose exploration algorithms polynomial in the number of state space parameters. Others have used empowerment, which is the information gain based on entropy of actions, as intrinsic rewards (Klyubin et al., 2005; Mohamed & Rezende, 2015). Stadie et al. (2015) use prediction error in the feature space of an auto-encoder as a measure of interesting states to explore. State visitation counts have also been investigated for exploration (Oh et al., 2015; Bellemare et al., 2016; Tang et al., 2016). Osband et al. (2016) train multiple value functions and makes use of bootstrapping and Thompson sampling for exploration. Many approaches measure information gain for exploration (Still & Precup, 2012; Little & Sommer, 2014; Storck et al., 1995). Houthooft et al. (2016) use an exploration strategy that maximizes information gain about the agent’s belief of the environment’s dynamics. Our approach of jointly training forward and inverse models for learning a feature space has similarities to (Jordan & Rumelhart, 1992; Wolpert et al., 1995; Agrawal et al., 2016), but these works use the learned models of dynamics for planning a sequence of actions instead of exploration. The idea of using a proxy task to learn a semantic feature embedding has been used in a number of works on self-supervised learning in computer vision (Goroshin et al., 2015; Agrawal et al., 2015; Doersch et al., 2015; Jayaraman & Grauman, 2015; Wang & Gupta, 2015; Pathak et al., 2016).

好奇心驱动的探索是强化学习文献中一个研究充分的主题，较好的综述可见（Oudeyer et al., 2007; Oudeyer & Kaplan, 2009）。Schmidhuber（1991; 2010）以及 Sun 等人（2011）把惊讶和压缩进展用作内在奖励。Kearns 等人（1999）和 Brafman 等人（2002）的经典工作提出了在状态空间参数数量上多项式级的探索算法。另一些人使用赋权（empowerment）——基于动作熵的信息增益——作为内在奖励（Klyubin et al., 2005; Mohamed & Rezende, 2015）。Stadie 等人（2015）把自编码器特征空间中的预测误差，用作值得探索的有趣状态的度量。状态访问计数也被用于探索（Oh et al., 2015; Bellemare et al., 2016; Tang et al., 2016）。Osband 等人（2016）训练多个价值函数，并利用自助法和 Thompson 采样来探索。许多方法用信息增益来做探索（Still & Precup, 2012; Little & Sommer, 2014; Storck et al., 1995）。Houthooft 等人（2016）使用一种最大化关于智能体对环境动力学之信念的信息增益的探索策略。我们联合训练前向与逆模型来学习特征空间，这一做法与（Jordan & Rumelhart, 1992; Wolpert et al., 1995; Agrawal et al., 2016）有相似之处，但这些工作把所学的动力学模型用于规划一连串动作，而不是用于探索。用代理任务来学习语义特征嵌入，已出现在计算机视觉中许多自监督学习工作里（Goroshin et al., 2015; Agrawal et al., 2015; Doersch et al., 2015; Jayaraman & Grauman, 2015; Wang & Gupta, 2015; Pathak et al., 2016）。

Concurrent work: A number of interesting related papers have appeared on Arxiv while the present work was in submission. Sukhbaatar et al. (2017) generates supervision for pre-training via asymmetric self-play between two agents to improve data efficiency during fine-tuning. Several methods propose improving data efficiency of RL algorithms using self-supervised prediction based auxiliary tasks (Jaderberg et al., 2017; Shelhamer et al., 2017). Fu et al. (2017) learn discriminative models, and Gregor et al. (2017) use empowerment based measure to tackle exploration in sparse reward setups.

同期工作： 本文投稿期间，Arxiv 上出现了若干有意思的相关论文。Sukhbaatar 等人（2017）通过两个智能体之间不对称的自我对弈来生成预训练监督，以提高微调时的数据效率。若干方法提出用基于自监督预测的辅助任务来提高强化学习算法的数据效率（Jaderberg et al., 2017; Shelhamer et al., 2017）。Fu 等人（2017）学习判别模型，Gregor 等人（2017）则使用基于赋权的度量，来应对稀疏奖励设置中的探索。

6 Discussion

6 讨论

In this work we propose a mechanism for generating curiosity-driven intrinsic reward signal that scales to high dimensional visual inputs, bypasses the difficult problem of predicting pixels and ensures that the exploration strategy of the agent is unaffected by nuisance factors in the environment. We demonstrate that our agent significantly outperforms the baseline A3C with no curiosity, a recently proposed VIME (Houthooft et al., 2016) formulation for exploration, and a baseline pixel-predicting formulation.

在本工作中，我们提出一种生成好奇心驱动的内在奖励信号的机制：它能扩展到高维视觉输入，避开预测像素这一难题，并确保智能体的探索策略不受环境中恼人因素的影响。我们表明，我们的智能体显著优于不带好奇心的基线 A3C、最近提出的用于探索的 VIME 表述（Houthooft et al., 2016），以及一条预测像素的基线表述。

In VizDoom our agent learns the exploration behavior of moving along corridors and across rooms without any rewards from the environment. In Mario our agent crosses more than 30% of Level-1 without any rewards from the game. One reason why our agent is unable to go beyond this limit is the presence of a pit at 38% of the game that requires a very specific sequence of 15-20 key presses in order to jump across it. If the agent is unable to execute this sequence, it falls in the pit and dies, receiving no further rewards from the environment. Therefore it receives no gradient information indicating that there is a world beyond the pit that could potentially be explored. This issue is somewhat orthogonal to developing models of curiosity, but presents a challenging problem for policy learning.

在 VizDoom 中，我们的智能体在不使用环境任何奖励的情况下，学会了沿着走廊、穿过房间移动的探索行为。在 Mario 中，我们的智能体在不使用游戏任何奖励的情况下，通过了 Level-1 的 30% 以上。它之所以难以越过这一限度，一个原因是游戏 38% 处有一个坑，需要一串非常特定的 15–20 次按键才能跳过去。如果智能体无法执行这一序列，它就会掉进坑里死去，从而不再从环境得到任何奖励。因此它收不到任何梯度信息，告诉它坑的那边还有一个世界可以探索。这个问题与发展好奇心模型多少有些正交，但对策略学习提出了一个困难的挑战。

It is common practice to evaluate reinforcement learning approaches in the same environment that was used for training. However, we feel that it is also important to evaluate on a separate “testing set” as well. This allows us to gauge how much of what has been learned is specific to the training environment (i.e. memorized), and how much might constitute “generalizable skills” that could be applied to new settings. In this paper, we evaluate generalization in two ways: 1) by applying the learned policy to a new scenario “as is” (no further learning), and 2) by fine-tuning the learned policy on a new scenario (we borrow the pre-training/fine-tuning nomenclature from the deep feature learning literature). We believe that evaluating generalization is a valuable tool and will allow the community to better understand the performance of various reinforcement learning algorithms. To further aid in this effort, we will make the code for our algorithm, as well as testing and environment setups freely available online.

评估强化学习方法时，常见做法是在用于训练的同一环境中评估。不过我们觉得，在单独的“测试集”上评估同样重要。这让我们能够衡量：所学内容有多少是针对训练环境的（也就是被记住的），又有多少可能构成可以应用到新设置中的“可泛化技能”。本文用两种方式评估泛化：1）把学到的策略“原样”应用到新场景（不再继续学习）；2）在新场景上微调学到的策略（预训练/微调这一术语借自深度特征学习文献）。我们相信，评估泛化是一种有价值的工具，将帮助社区更好地理解各种强化学习算法的表现。为进一步推动这一努力，我们将把算法代码以及测试与环境设置免费公开到网上。

An interesting direction of future research is to use the learned exploration behavior/skill as a motor primitive/low-level policy in a more complex, hierarchical system. For example, our VizDoom agent learns to walk along corridors instead of bumping into walls. This could be a useful primitive for a navigation system.

一个有意思的未来研究方向，是把学到的探索行为/技能，作为更复杂的层次系统中的运动原语/低层策略。例如，我们的 VizDoom 智能体学会了沿着走廊行走而不是撞墙。这对导航系统来说可能是一种有用的原语。

While the rich and diverse real world provides ample opportunities for interaction, reward signals are sparse. Our approach excels in this setting and converts unexpected interactions that affect the agent into intrinsic rewards. However our approach does not directly extend to the scenarios where “opportunities for interactions” are also rare. In theory, one could save such events in a replay memory and use them to guide exploration. However, we leave this extension for future work.

丰富而多样的现实世界提供了充足的交互机会，但奖励信号却很稀疏。我们的方法在这一设置中表现出色，并把那些影响到智能体的意外交互转化成内在奖励。不过，当“交互机会”本身也很少时，我们的方法并不能直接扩展。理论上，可以把这类事件存进回放记忆，并用它们来指导探索。不过，我们把这一扩展留给未来工作。

Acknowledgements:

We would like to thank Sergey Levine, Evan Shelhamer, Saurabh Gupta, Phillip Isola and other members of the BAIR lab for fruitful discussions and comments. We thank Jacob Huh for help with Figure 2 and Alexey Dosovitskiy for VizDoom maps. This work was supported in part by NSF IIS-1212798, IIS-1427425, IIS-1536003, IIS-1633310, ONR MURI N00014-14-1-0671, Berkeley DeepDrive, equipment grant from Nvidia, and the Valrhona Reinforcement Learning Fellowship.

致谢：

我们感谢 Sergey Levine、Evan Shelhamer、Saurabh Gupta、Phillip Isola 以及 BAIR 实验室的其他成员所给予的富有成效的讨论和意见。我们感谢 Jacob Huh 对图 2 的帮助，以及 Alexey Dosovitskiy 提供的 VizDoom 地图。本工作部分得到 NSF IIS-1212798、IIS-1427425、IIS-1536003、IIS-1633310、ONR MURI N00014-14-1-0671、Berkeley DeepDrive、Nvidia 设备赠款以及 Valrhona Reinforcement Learning Fellowship 的支持。

References

参考文献

Agrawal et al. (2015) Agrawal, Pulkit, Carreira, Joao, and Malik, Jitendra. Learning to see by moving. In ICCV , 2015.

Agrawal et al. (2016) Agrawal, Pulkit, Nair, Ashvin, Abbeel, Pieter, Malik, Jitendra, and Levine, Sergey. Learning to poke by poking: Experiential learning of intuitive physics. NIPS , 2016.

Bellemare et al. (2016) Bellemare, Marc, Srinivasan, Sriram, Ostrovski, Georg, Schaul, Tom, Saxton, David, and Munos, Remi. Unifying count-based exploration and intrinsic motivation. In NIPS , 2016.

Brafman & Tennenholtz (2002) Brafman, Ronen I and Tennenholtz, Moshe. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. JMLR , 2002.

Brockman et al. (2016) Brockman, Greg, Cheung, Vicki, Pettersson, Ludwig, Schneider, Jonas, Schulman, John, Tang, Jie, and Zaremba, Wojciech. Openai gym. arXiv:1606.01540 , 2016.

Clevert et al. (2015) Clevert, Djork-Arné, Unterthiner, Thomas, and Hochreiter, Sepp. Fast and accurate deep network learning by exponential linear units (elus). arXiv:1511.07289 , 2015.

Doersch et al. (2015) Doersch, Carl, Gupta, Abhinav, and Efros, Alexei A. Unsupervised visual representation learning by context prediction. In ICCV , 2015.

Dosovitskiy & Koltun (2016) Dosovitskiy, Alexey and Koltun, Vladlen. Learning to act by predicting the future. ICLR , 2016.

Fu et al. (2017) Fu, Justin, Co-Reyes, John D, and Levine, Sergey. Ex2: Exploration with exemplar models for deep reinforcement learning. arXiv:1703.01260 , 2017.

Goroshin et al. (2015) Goroshin, Ross, Bruna, Joan, Tompson, Jonathan, Eigen, David, and LeCun, Yann. Unsupervised feature learning from temporal data. arXiv:1504.02518 , 2015.

Gregor et al. (2017) Gregor, Karol, Rezende, Danilo Jimenez, and Wierstra, Daan. Variational intrinsic control. ICLR Workshop , 2017.

Houthooft et al. (2016) Houthooft, Rein, Chen, Xi, Duan, Yan, Schulman, John, De Turck, Filip, and Abbeel, Pieter. Vime: Variational information maximizing exploration. In NIPS , 2016.

Jaderberg et al. (2017) Jaderberg, Max, Mnih, Volodymyr, Czarnecki, Wojciech Marian, Schaul, Tom, Leibo, Joel Z, Silver, David, and Kavukcuoglu, Koray. Reinforcement learning with unsupervised auxiliary tasks. ICLR , 2017.

Jayaraman & Grauman (2015) Jayaraman, Dinesh and Grauman, Kristen. Learning image representations tied to ego-motion. In ICCV , 2015.

Jordan & Rumelhart (1992) Jordan, Michael I and Rumelhart, David E. Forward models: Supervised learning with a distal teacher. Cognitive science , 1992.

Kearns & Koller (1999) Kearns, Michael and Koller, Daphne. Efficient reinforcement learning in factored mdps. In IJCAI , 1999.

Kempka et al. (2016) Kempka, Michał, Wydmuch, Marek, Runc, Grzegorz, Toczek, Jakub, and Jaśkowski, Wojciech. Vizdoom: A doom-based ai research platform for visual reinforcement learning. arXiv:1605.02097 , 2016.

Klyubin et al. (2005) Klyubin, Alexander S, Polani, Daniel, and Nehaniv, Chrystopher L. Empowerment: A universal agent-centric measure of control. In Evolutionary Computation , 2005.

Lillicrap et al. (2016) Lillicrap, Timothy P, Hunt, Jonathan J, Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. ICLR , 2016.

Little & Sommer (2014) Little, Daniel Y and Sommer, Friedrich T. Learning and exploration in action-perception loops. Closing the Loop Around Neural Systems , 2014.

Lopes et al. (2012) Lopes, Manuel, Lang, Tobias, Toussaint, Marc, and Oudeyer, Pierre-Yves. Exploration in model-based reinforcement learning by empirically estimating learning progress. In NIPS , 2012.

Mirowski et al. (2017) Mirowski, Piotr, Pascanu, Razvan, Viola, Fabio, Soyer, Hubert, Ballard, Andy, Banino, Andrea, Denil, Misha, Goroshin, Ross, Sifre, Laurent, Kavukcuoglu, Koray, et al. Learning to navigate in complex environments. ICLR , 2017.

Mnih et al. (2015) Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, et al. Human-level control through deep reinforcement learning. Nature , 2015.

Mnih et al. (2016) Mnih, Volodymyr, Badia, Adria Puigdomenech, Mirza, Mehdi, Graves, Alex, Lillicrap, Timothy P, Harley, Tim, Silver, David, and Kavukcuoglu, Koray. Asynchronous methods for deep reinforcement learning. In ICML , 2016.

Mohamed & Rezende (2015) Mohamed, Shakir and Rezende, Danilo Jimenez. Variational information maximisation for intrinsically motivated reinforcement learning. In NIPS , 2015.

Oh et al. (2015) Oh, Junhyuk, Guo, Xiaoxiao, Lee, Honglak, Lewis, Richard L, and Singh, Satinder. Action-conditional video prediction using deep networks in atari games. In NIPS , 2015.

Osband et al. (2016) Osband, Ian, Blundell, Charles, Pritzel, Alexander, and Van Roy, Benjamin. Deep exploration via bootstrapped dqn. In NIPS , 2016.

Oudeyer & Kaplan (2009) Oudeyer, Pierre-Yves and Kaplan, Frederic. What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics , 2009.

Oudeyer et al. (2007) Oudeyer, Pierre-Yves, Kaplan, Frdric, and Hafner, Verena V. Intrinsic motivation systems for autonomous mental development. Evolutionary Computation , 2007.

Paquette (2016) Paquette, Philip. Super mario bros. in openai gym. github:ppaquette/gym-super-mario , 2016.

Pathak et al. (2016) Pathak, Deepak, Krahenbuhl, Philipp, Donahue, Jeff, Darrell, Trevor, and Efros, Alexei A. Context encoders: Feature learning by inpainting. In CVPR , 2016.

Poupart et al. (2006) Poupart, Pascal, Vlassis, Nikos, Hoey, Jesse, and Regan, Kevin. An analytic solution to discrete bayesian reinforcement learning. In ICML , 2006.

Ryan (2000) Ryan, Richard; Deci, Edward L. Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology , 2000.

Schmidhuber (1991) Schmidhuber, Jurgen. A possibility for implementing curiosity and boredom in model-building neural controllers. In From animals to animats: Proceedings of the first international conference on simulation of adaptive behavior , 1991.

Schmidhuber (2010) Schmidhuber, Jürgen. Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE Transactions on Autonomous Mental Development , 2010.

Shelhamer et al. (2017) Shelhamer, Evan, Mahmoudieh, Parsa, Argus, Max, and Darrell, Trevor. Loss is its own reward: Self-supervision for reinforcement learning. arXiv:1612.07307 , 2017.

Silvia (2012) Silvia, Paul J. Curiosity and motivation. In The Oxford Handbook of Human Motivation , 2012.

Singh et al. (2005) Singh, Satinder P, Barto, Andrew G, and Chentanez, Nuttapong. Intrinsically motivated reinforcement learning. In NIPS , 2005.

Stadie et al. (2015) Stadie, Bradly C, Levine, Sergey, and Abbeel, Pieter. Incentivizing exploration in reinforcement learning with deep predictive models. NIPS Workshop , 2015.

Still & Precup (2012) Still, Susanne and Precup, Doina. An information-theoretic approach to curiosity-driven reinforcement learning. Theory in Biosciences , 2012.

Storck et al. (1995) Storck, Jan, Hochreiter, Sepp, and Schmidhuber, Jürgen. Reinforcement driven information acquisition in non-deterministic environments. In ICANN , 1995.

Sukhbaatar et al. (2017) Sukhbaatar, Sainbayar, Kostrikov, Ilya, Szlam, Arthur, and Fergus, Rob. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv:1703.05407 , 2017.

Sun et al. (2011) Sun, Yi, Gomez, Faustino, and Schmidhuber, Jürgen. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In AGI , 2011.

Tang et al. (2016) Tang, Haoran, Houthooft, Rein, Foote, Davis, Stooke, Adam, Chen, Xi, Duan, Yan, Schulman, John, De Turck, Filip, and Abbeel, Pieter. # exploration: A study of count-based exploration for deep reinforcement learning. arXiv:1611.04717 , 2016.

Wang & Gupta (2015) Wang, Xiaolong and Gupta, Abhinav. Unsupervised learning of visual representations using videos. In ICCV , 2015.

Wolpert et al. (1995) Wolpert, Daniel M, Ghahramani, Zoubin, and Jordan, Michael I. An internal model for sensorimotor integration. Science-AAAS-Weekly Paper Edition , 1995.