First return, then explore
First return, then explore
1Uber AI Labs, San Francisco, CA, USA
2OpenAI, San Francisco, CA, USA
∗These authors contributed equally to this work
Correspondence should be addressed to Adrien Ecoffet (email: adrienecoffet@gmail.com), Joost Huizinga (email: joost.hui@gmail.com), and Jeff Clune (email: jclune@gmail.com).
Please cite as: Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K.O. and Clune, J. First return, then explore. Nature 590, 580–586 (2021). https://doi.org/10.1038/s41586-020-03157-9
The promise of reinforcement learning is to solve complex sequential decision problems autonomously by specifying a high-level reward function only. However, reinforcement learning algorithms struggle when, as is often the case, simple and intuitive rewards provide sparse1 and deceptive2 feedback. Avoiding these pitfalls requires thoroughly exploring the environment, but creating algorithms that can do so remains one of the central challenges of the field. We hypothesise that the main impediment to effective exploration originates from algorithms forgetting how to reach previously visited states (“detachment”) and from failing to first return to a state before exploring from it (“derailment”). We introduce Go-Explore, a family of algorithms that addresses these two challenges directly through the simple principles of explicitly remembering promising states and first returning to such states before intentionally exploring. Go-Explore solves all heretofore unsolved Atari games and surpasses the state of the art on all hard-exploration games1, with orders of magnitude improvements on the grand challenges Montezuma’s Revenge and Pitfall. We also demonstrate the practical potential of Go-Explore on a sparse-reward pick-and-place robotics task. Additionally, we show that adding a goal-conditioned policy can further improve Go-Explore’s exploration efficiency and enable it to handle stochasticity throughout training. The substantial performance gains from Go-Explore suggest that the simple principles of remembering states, returning to them, and exploring from them are a powerful and general approach to exploration, an insight that may prove critical to the creation of truly intelligent learning agents.
强化学习的承诺是:只需给出一个高层奖励函数,就能自主解决复杂的序列决策问题。可是,当简单直观的奖励像常见情形那样只提供稀疏1而又带欺骗性2的反馈时,强化学习算法就会陷入困境。要避开这些陷阱,必须把环境充分探索一遍,但怎样做出能做到这一点的算法,至今仍是这个领域的核心难题之一。我们提出一个假设:有效探索之所以受阻,主要是因为算法会忘掉如何回到曾经访问过的状态(“脱离”),以及没能先回到某个状态再从那里探索(“脱轨”)。我们引入 Go-Explore 这一族算法,正是用两条简单原则直接应对这两个困难:明确记住有希望的状态,并在有意探索之前先回到这些状态。Go-Explore 解决了此前所有尚未解决的 Atari 游戏,并在所有难探索游戏1上超过当时最好成绩,在蒙特祖玛的复仇和 Pitfall 这两项公认难题上更有数量级的提升。我们还在一个稀疏奖励的取放机器人任务上展示了它的实用潜力。此外,我们表明,加入目标条件策略还能进一步提高 Go-Explore 的探索效率,并使其在整个训练过程中都能应对随机性。Go-Explore 带来的大幅性能提升说明,记住状态、回到这些状态、再从那里探索,是一套有力且通用的探索原则;这一认识,对真正智能的学习智能体的诞生可能至关重要。
Recent years have yielded impressive achievements in Reinforcement Learning (RL), including world-champion level performance in Go3, Starcraft II4, and Dota II5, as well as autonomous learning of robotic skills such as running, jumping, and grasping6,7. Many of these successes were enabled by carefully-designed, highly-informative reward functions. However, for many practical problems, defining a good reward function is non-trivial; to guide a robot to a refrigerator, one might provide a reward only when the refrigerator is reached, but doing so makes the reward sparse if many actions are required to reach the refrigerator. Unfortunately, a denser reward (e.g. the Euclidean distance to the refrigerator), can be deceptive; naively following the reward function may lead the robot into a dead end and can also produce unintended (and potentially unsafe) behaviour (e.g. the robot not detouring around obstacles like pets)8–10.
近年来,强化学习(RL)取得了令人瞩目的成绩,包括在围棋3、星际争霸 II4 和 Dota II5 中达到世界冠军水平,以及让机器人自主学会跑、跳和抓取等技能6,7。这些成功,很多都依赖于精心设计、信息量很大的奖励函数。可是对许多实际问题来说,定义一个好的奖励函数并不容易:若要引导机器人走到冰箱前,一种做法是只在到达冰箱时给奖励,可如果需要很多步才能走到,奖励就会非常稀疏。不幸的是,更密的奖励(例如到冰箱的欧氏距离)又可能带有欺骗性;若只是跟着奖励走,机器人可能闯进死胡同,还可能做出并非本意、甚至不安全的行为(例如不绕开宠物这类障碍)8–10。
These challenges motivate designing RL algorithms that better handle sparsity and deception. A key observation is that sufficient exploration of the state space enables discovering sparse rewards and avoiding deceptive local optima11,12. We argue that two major issues have hindered the ability of previous algorithms to explore. The first is detachment, wherein the algorithm prematurely stops returning to certain areas of the state space despite having evidence that those areas are promising (SI “Detachment”). Detachment is especially likely when (as is common) there are multiple areas to explore because the algorithm may partially explore one area, switch to a second area, and forget how to visit the first area. The second is derailment, wherein the exploratory mechanisms of the algorithm prevent it from returning to previously visited states, preventing exploration directly and/or forcing practitioners to make exploratory mechanisms so minimal that effective exploration does not occur (SI “Derailment”). For example, if a long string of correct actions is required to reach a particular area, a high probability of exploratory actions prevents the area from being reached while a low probability of exploratory actions results in little exploration in general. We present Go-Explore, a family of algorithms designed to explicitly avoid detachment and derailment, and demonstrate that it thoroughly explores environments. Go-Explore surpasses human performance on (solves) all previously unsolved Atari games*, which has been posited as a major milestone in previous work13,15,16. It also surpasses the state of the art on all hard-exploration Atari games (i.e. where obtaining rewards requires long sequences of correct actions, meaning randomly sampling actions rarely produces rewards and thus more intelligent exploration is needed).
这些困难促使人们去设计更能处理稀疏性和欺骗性的强化学习算法。一个关键观察是:只要把状态空间探索得足够充分,就更容易发现稀疏奖励,也更容易避开带欺骗性的局部最优11,12。我们认为,以往算法之所以探不深,主要卡在两件事上。第一件是脱离:尽管已有证据表明某些区域有希望,算法却过早不再回到这些区域(SI “Detachment”)。当(如常见情形)同时有多片区域需要探索时,脱离尤其容易发生——算法可能先在一片区域探了一部分,再转到第二片,随后就忘了怎么回到第一片。第二件是脱轨:算法的探索机制反而妨碍它回到曾经到过的状态,既直接阻碍探索,又迫使使用者把探索机制调得极弱,结果有效探索几乎不会发生(SI “Derailment”)。例如,若到达某一区域需要一长串正确动作,探索动作的概率太高就到不了那里,概率太低又几乎无法探索。我们提出 Go-Explore 这一族算法,就是要明确避免脱离和脱轨,并表明它能够把环境探索得很充分。Go-Explore 在所有此前未解决的 Atari 游戏上都超过了人类水平(即“解决”了这些游戏)*,而这曾被先前工作视为一项重要里程碑13,15,16。它也在所有难探索 Atari 游戏上超过当时最好成绩(所谓难探索,是指要获得奖励需要一长串正确动作,随机采样几乎得不到奖励,因而需要更有针对性的探索)。
*Concurrent work13 similarly reached this milestone (SI “Comparing Go-Explore and Agent57”), but under easier, mostly deterministic conditions that do not meet community-defined standards14 for evaluation on Atari. Our descriptions of prior results include only evaluations meeting these standards, unless explicitly mentioned (Methods “State of the art on Atari”).
*同时期的工作13同样达到了这一里程碑(SI “Comparing Go-Explore and Agent57”),但其评估条件更容易,且多为确定性环境,并不符合社区为 Atari 评估制定的标准14。除非另行说明,我们对先前结果的描述只包括符合这些标准的评估(Methods “State of the art on Atari”)。
Additionally, we demonstrate that it can solve a practical simulated robotics problem with an extremely sparse reward. Finally, we show that its performance can be greatly increased by incorporating minimal domain knowledge and examine how harnessing learned skills during exploration can improve exploration efficiency, highlighting the versatility of the Go-Explore family.
此外,我们表明它还能解决一个奖励极其稀疏的实用仿真机器人问题。最后,我们展示:只需加入很少的领域知识,性能就能大幅提高;并考察在探索过程中调用已学会的技能,如何提升探索效率,以此说明 Go-Explore 这一族方法的灵活性。
The Go-Explore family of algorithms
Go-Explore 算法家族
To avoid detachment, Go-Explore builds an archive of the different states it has visited in the environment, thus ensuring that states cannot be forgotten. Starting from an archive containing only the initial state, it builds this archive iteratively: first, it probabilistically selects a state to return to from the archive (Fig. 1a), returns to that state (the “go” step; Fig. 1b), then explores from that state (the “explore” step; Fig. 1c) and updates the archive with all novel states encountered (Fig. 1e). The overall process is reminiscent of classical planning algorithms (e.g. the archive can be considered a frontier, the “explore” step represents expanding a node, etc.), the potential of which have been relatively unappreciated within deep RL. However, for problems focused on by the RL community (like hard-exploration Atari games), which are high-dimensional with sparse rewards and/or stochasticity, no known planning method works14,17. Among other reasons (SI “Go-Explore, Planning, and Model-based RL”), such state spaces are too large to search exhaustively (requiring hard-to-invent heuristics to prune search) and stochastic transitions make it impossible to know whether a node has been fully expanded. Go-Explore can be seen as porting the principles of planning algorithms to these challenging problems.
为避免脱离,Go-Explore 会为它在环境中访问过的不同状态建立一份档案,从而确保这些状态不会被忘掉。档案起初只包含初始状态,随后迭代扩充:先按概率从档案中选出一个要回去的状态(图 1a),回到该状态(“go”一步;图 1b),再从该状态出发探索(“explore”一步;图 1c),并把途中遇到的所有新状态写入档案(图 1e)。整个过程让人想起经典规划算法(例如,档案可以看作前沿,“explore”一步相当于扩展一个节点,等等),而这些方法的潜力在深度强化学习里相对没有被充分重视。不过,对强化学习社区所关注的那类问题(如难探索的 Atari 游戏)——状态空间维度高,奖励稀疏,并且可能带有随机性——目前还没有已知的规划方法能够奏效14,17。原因不止一端(SI “Go-Explore, Planning, and Model-based RL”):这样的状态空间大到无法穷尽搜索(需要很难发明的启发式来剪枝),而随机转移又让人无法判断一个节点是否已经完全展开。Go-Explore 可以被看作把规划算法的原则移植到这些困难问题上。
Previous RL algorithms do not separate returning from exploring, and instead mix in exploration throughout an episode, usually by adding random actions a fraction of the time15,18 or by sampling from a stochastic policy (a function that decides which action to take in each state, often a neural network)19,20. By first returning before exploring, Go-Explore avoids derailment by minimising exploration when returning (thus minimising failure to return) after which it can purely focus on exploration.
以往的强化学习算法并不把“回去”和“探索”分开,而是在整段回合里混入探索,通常是按一定比例加入随机动作15,18,或从随机策略(一个决定在每个状态下采取哪个动作的函数,常常是神经网络)中采样19,20。Go-Explore 先回去再探索,在回去时尽量减少探索(从而尽量避免回不去),然后再专注于探索,以此避免脱轨。
Because non-trivial environments have too many states to store explicitly, Go-Explore groups similar states into cells, and states are only considered novel if they are in a cell that does not yet exist in the archive (Fig. 1d). The archive stores one state per cell, and to maximise performance, if a state maps to an already known cell, but is associated with a better trajectory (higher performing or shorter; Methods), that state and its associated trajectory will replace the state and trajectory currently associated with that cell. Go-Explore selects states to return to (Fig. 1a) proportionally to weights it assigns to their associated cells in the archive (Methods).
由于非平凡环境里的状态太多,无法一一显式存储,Go-Explore 把相似状态归入格子(cell),只有当状态落在档案中尚不存在的格子里时,才被视为新状态(图 1d)。档案里每个格子只存一个状态;为了尽量提高表现,如果某个状态映射到已有格子,但对应的轨迹更好(分数更高或更短;Methods),就会用该状态及其轨迹替换这个格子当前保存的状态和轨迹。Go-Explore 按它给档案中各格子所赋的权重,成比例地选择要回去的状态(图 1a)(Methods)。
Figure 1: Overview of Go-Explore. (a) Probabilistically select a state from the archive, preferring states associated with promising cells. (b) Return to the selected state, such as by restoring simulator state or by running a goal-conditioned policy. (c) Explore from that state by taking random actions or sampling from a trained policy. (d) Map every state encountered during returning and exploring to a low-dimensional cell representation. (e) Add states that map to new cells to the archive and update other archive entries.
图 1:Go-Explore 概览。(a) 按概率从档案中选出一个状态,优先选择与有希望格子相关联的状态。(b) 回到所选状态,例如通过恢复模拟器状态,或运行目标条件策略。(c) 从该状态出发探索,采取随机动作,或从已训练的策略中采样。(d) 把回去和探索过程中遇到的每个状态,映射到一个低维的格子表示。(e) 把映射到新格子的状态加入档案,并更新档案中的其他条目。
While returning to a previously found state can be done with a trained policy (demonstrated in Sec. “Policy-based Go-Explore”), Go-Explore provides a unique opportunity to leverage the availability and widespread use of simulators in RL tasks7,21–23. Simulators are restorable environments because previous states can be saved and instantly returned to, thus completely negating derailment.
回到先前找到的状态,可以用训练好的策略来完成(见 “Policy-based Go-Explore” 一节),但 Go-Explore 也提供了一个独特机会,去利用强化学习任务中模拟器既常用又容易获得这一条件7,21–23。模拟器是可恢复环境:先前的状态可以保存下来,并立刻回到那里,从而彻底消除脱轨。
When exploiting this property of restorable environments, Go-Explore thoroughly explores the environment during its exploration phase by continually restoring (and subsequently taking exploratory actions from) one of the states in its archive (Fig. 1). It eventually returns the highest-scoring trajectory (sequence of actions) it found. Such trajectories are not robust to stochasticity or unexpected outcomes (e.g. a robot may slip and miss a crucial turn, invalidating the entire trajectory). To resolve this issue, Go-Explore trains a robust policy by Learning from Demonstrations (LfD)24, where the exploration phase trajectories replace the usual human expert demonstrations (similar to Guo et al. (2014)25), in a variant of the environment featuring sufficient stochasticity to ensure robustness. The exploration-phase trajectories will be informative in the stochastic environment as long as following a close approximation to the example trajectory still leads to a high cumulative reward (SI “Go-Explore and Stochasticity”). Because it produces robust policies from open-loop (i.e. predetermined) trajectories, we call this LfD process the robustification phase (Fig. 1 “Robustification Phase”).
在利用可恢复环境的这一性质时,Go-Explore 在探索阶段会不断恢复档案中的某个状态(随后从该状态采取探索动作),从而把环境探索得很充分(图 1)。它最终返回所找到的得分最高的轨迹(动作序列)。这样的轨迹对随机性或意外结果并不稳健(例如机器人可能打滑、错过一个关键转弯,整条轨迹就失效了)。为解决这个问题,Go-Explore 用演示学习(Learning from Demonstrations, LfD)24 训练一个稳健策略,用探索阶段的轨迹代替通常由人类专家提供的演示(类似于 Guo 等人(2014)25),并在带有足够随机性、从而能保证稳健性的环境变体中进行训练。只要沿着与示例轨迹相近的路径走,仍然能获得高累计奖励,这些探索阶段轨迹在随机环境中就仍有信息量(SI “Go-Explore and Stochasticity”)。由于它能从开环(即预先确定的)轨迹中得到稳健策略,我们把这一 LfD 过程称为稳健化阶段(图 1 “Robustification Phase”)。
Learning Atari with state restoration
在可恢复状态的 Atari 中学习
The Atari benchmark suite26, a prominent benchmark for RL algorithms15,27,28, is an appropriate test-bed for Go-Explore because it contains a diverse set of games with varying levels of reward sparsity and deceptiveness. The following experiment highlights the benefit of a “go” step that restores the simulator state directly. In this experiment, the “explore” step happens through random actions, meaning the exploration phase operates entirely without a trained policy, which assumes that random actions have a sufficiently high probability of discovering new cells; more complex problems may require policy-based exploration (explored below). The state-to-cell mapping for Go-Explore’s archive consists of downscaling the current game frame from the original 210 × 160 colour frame to a much smaller grayscale image, which (in contrast to most RL pre-processing that reduces dimensionality to save compute while minimising conflation15) aggregates similar-looking frames into the same cell (Fig. 1d). This mapping does not require game-specific knowledge and proves to be efficient across the entire Atari benchmark, though more complex environments may require more sophisticated (e.g. learned) representations. Good state-to-cell-mapping parameters result in a representation that strikes a balance between two extremes: lack of aggregation (e.g. one cell for every frame, which is computationally inefficient) and excessive aggregation (e.g. assigning all frames to a single cell, which prevents exploration). Because appropriate downscaling parameters (width, height, and number of possible grayscale values) vary across Atari games (SI “Exploration phase without dynamic representations”) as well as when exploration progresses within a given game, these parameters are optimised dynamically at regular intervals (Methods “Downscaling on Atari”). The hyperparameters of this optimisation procedure are robust and generalise (SI “Ablations”).
Atari 基准套件26是强化学习算法的重要评测平台15,27,28,也适合用来检验 Go-Explore,因为它包含一组风格多样、奖励稀疏程度和欺骗性各不相同的游戏。下面的实验突出了“go”一步直接恢复模拟器状态所带来的好处。在该实验中,“explore”一步通过随机动作完成,也就是说探索阶段完全不依赖训练好的策略;这假定随机动作发现新格子的概率足够高。更复杂的问题可能需要基于策略的探索(下文会讨论)。Go-Explore 档案所用的状态到格子映射,是把当前游戏画面从原来 210 × 160 的彩色帧缩小成一张小得多的灰度图;与大多数为节省计算、同时尽量减少不同画面被混为一谈而做的强化学习预处理15不同,这种映射会把看上去相似的帧归入同一个格子(图 1d)。该映射不需要针对某一款游戏的专门知识,并且在整个 Atari 基准上都相当有效,不过更复杂的环境可能需要更精巧(例如学出来的)表示。好的状态到格子映射参数,要在两个极端之间取得平衡:一端是几乎不聚合(例如一帧一个格子,计算上效率很低),另一端是过度聚合(例如把所有帧都分到同一个格子,探索就无法进行)。合适的缩小参数(宽度、高度和可能的灰度级数)不仅在不同 Atari 游戏之间不同(SI “Exploration phase without dynamic representations”),在同一款游戏的探索进程中也会变化,因此这些参数会按固定间隔动态优化(Methods “Downscaling on Atari”)。该优化过程的超参数相当稳健,也具有泛化性(SI “Ablations”)。
Here, the robustification phase consists of a modified version of the “backward algorithm”29, currently the highest performing LfD algorithm on Montezuma’s Revenge. Due to the large computational expense of the robustification process, this work focuses on the set of eleven games that have been considered hard-exploration challenges by the community1 or for which the state-of-the-art performance was still below human performance (Methods “State of the art on Atari”). To ensure the trained policy becomes robust to environmental perturbations, during robustification stochasticity is added to these environments following current community standards14. The demonstrations provided by the exploration phase provide enough information about available rewards to allow Go-Explore to eschew standard reward clipping, which overemphasises small rewards30, in favour of automatically scaling rewards to an appropriate range (Methods).
这里的稳健化阶段,采用的是经过修改的“反向算法”29,它目前是蒙特祖玛的复仇上表现最好的演示学习算法。由于稳健化计算开销很大,本工作把重点放在社区视为难探索挑战的那 11 款游戏1,以及当时最好成绩仍低于人类水平的游戏上(Methods “State of the art on Atari”)。为了让训练出的策略对环境扰动稳健,稳健化过程会按当前社区标准14 向这些环境加入随机性。探索阶段提供的演示,已经包含足够的奖励信息,使 Go-Explore 可以不再使用通常的奖励裁剪——那种做法会过分强调小奖励30——改为把奖励自动缩放到合适的范围(Methods)。
At test time, the mean performance of Go-Explore is both superhuman and surpasses the state of the art in all eleven games (except in Freeway where both Go-Explore and the state of the art reach the maximum score; Fig. 2b). These games include the grand challenges of Montezuma’s Revenge, where Go-Explore quadruples the state-of-the-art score, and Pitfall, where Go-Explore surpasses average human performance while previous algorithms were unable to score any points. The number of frames processed in these experiments is 30 billion (Extended Data Fig. 2 and 4), similar to that of recent distributed RL algorithms13,27,31. While older algorithms often processed fewer frames, many of them show signs of convergence (meaning no further progress is expected), and it is often unclear whether these algorithms would be able to process billions of frames in a reasonable amount of time.
在测试时,Go-Explore 的平均表现既超过人类,也在全部 11 款游戏上超过当时最好成绩(Freeway 除外,因为 Go-Explore 和当时最好成绩都达到了满分;图 2b)。这些游戏包括两项公认难题:在蒙特祖玛的复仇上,Go-Explore 把当时最好成绩提高了约三倍(即达到原来的四倍);在 Pitfall 上,Go-Explore 超过了人类平均水平,而此前的算法一分都拿不到。这些实验处理的帧数为 300 亿(Extended Data Fig. 2 和 4),与近期的分布式强化学习算法相当13,27,31。较老的算法处理的帧数往往更少,其中不少已出现收敛迹象(意味着很难再有进一步提升),而且常常不清楚它们能否在合理时间内处理数十亿帧。
The ability of the exploration phase to find high-performing trajectories is not limited to hard-exploration problems; it finds trajectories with superhuman scores for all of the 55 Atari games provided by OpenAI gym32, a feat that has not been performed before (save concurrent work13). In 85.5% of these games the trajectories reach scores higher than those achieved by state-of-the-art RL algorithms (Fig. 3). Go-Explore’s performance also exceeds that of planning algorithms (which similarly restore simulator states) that were evaluated on Atari14,17.
探索阶段找出高分轨迹的能力,并不限于难探索问题;它在 OpenAI gym 提供的全部 55 款 Atari 游戏上都找到了超过人类水平的轨迹32,这是此前从未做到的(同时期工作13除外)。在其中 85.5% 的游戏里,这些轨迹的得分高于当时最好的强化学习算法(图 3)。Go-Explore 的表现也超过了那些同样会恢复模拟器状态、并在 Atari 上做过评估的规划算法14,17。
Figure 2: Performance of robustified Go-Explore on Atari games. (a) Go-Explore produces substantial improvements over previous methods on Montezuma’s Revenge, a grand challenge which has been the primary focus of hard-exploration research for many years. Different methods use different amounts of computation. Go-Explore processed a similar number of frames (30B) as other distributed RL algorithms like Ape-X (22B) and NGU (35B). (b) It exceeds the average human score in each of the 11 hard-exploration and unsolved games in the Atari suite, and matches or beats (often by a factor of 2 or more) the state of the art in each of these games. Bold indicates the best scores with stochastic evaluation. Score differences between the exploration and robustification phases are discussed in SI “Robustification scores analysis”. A video of high performing runs can be found at https://youtu.be/e_aqRq59-Ns.
图 2:稳健化后的 Go-Explore 在 Atari 游戏上的表现。(a) Go-Explore 在蒙特祖玛的复仇上相对此前方法有大幅提升。这款游戏多年以来一直是难探索研究的主要焦点。不同方法所用的计算量并不相同。Go-Explore 处理的帧数(300 亿)与 Ape-X(220 亿)和 NGU(350 亿)等其他分布式强化学习算法相近。(b) 它在 Atari 套件中全部 11 款难探索和尚未解决的游戏上都超过人类平均分,并在每一款上达到或超过(常常是 2 倍或更多)当时最好成绩。加粗表示带随机性评估下的最好分数。探索阶段与稳健化阶段的分数差异,见 SI “Robustification scores analysis”。高水平回合的视频见 https://youtu.be/e_aqRq59-Ns。
Figure 3: Human-normalised performance of the exploration phase and state-of-the-art algorithms on all Atari games. The exploration phase of Go-Explore exceeds human performance in every game, often by orders of magnitude, and outperforms the prior state of the art in most games (details in Extended Data Fig. 2a and Extended Data Table 3).
图 3:探索阶段以及当时最好算法在全部 Atari 游戏上相对人类水平的归一化表现。Go-Explore 的探索阶段在每一款游戏上都超过人类水平,常常是几个数量级,并且在大多数游戏上超过此前的最好成绩(细节见 Extended Data Fig. 2a 和 Extended Data Table 3)。
In practical applications, it is often possible to define helpful features based on domain knowledge. Go-Explore can harness such easy-to-provide domain knowledge to substantially boost performance by constructing a cell representation (for the archive, not policy inputs) that only contains features relevant for exploration. The domain-knowledge features are the discretised position of the agent and relevant items held (Methods). With this domain knowledge cell representation, Go-Explore produces robustified policies that achieve a mean score of over 1.7 million on Montezuma’s Revenge, surpassing the state of the art by a factor of 150 as well as the human world record of 1.2 million33 (Fig. 2a). On Pitfall, the addition of domain knowledge produces robustified policies with a mean score of 102,571, close to the maximum possible of 112,000 and far above the state of the art of 0. The exploration phase explores both games extensively (Extended Data Fig. 3b), discovering effectively every unique location in each game (SI “Exploration in Atari”). Previous work suggests that intrinsic motivation algorithms benefit far less from domain knowledge; a count-based exploration algorithm with the same domain knowledge representation scores 12,240 on Montezuma’s Revenge34.
在实际应用中,往往可以根据领域知识定义出有用的特征。Go-Explore 能够利用这类不难提供的领域知识,显著提高表现:做法是构造一种只包含与探索相关特征的格子表示(用于档案,而不是作为策略输入)。这里的领域知识特征是智能体的离散化位置,以及持有的相关物品(Methods)。采用这种带领域知识的格子表示后,Go-Explore 得到的稳健化策略在蒙特祖玛的复仇上平均超过 170 万分,约为当时最好成绩的 150 倍,也超过 120 万的人类世界纪录33(图 2a)。在 Pitfall 上,加入领域知识后,稳健化策略平均分为 102,571,接近满分 112,000,并远高于当时最好成绩 0。探索阶段把这两款游戏都探得很开(Extended Data Fig. 3b),几乎发现了每款游戏里每一个独特位置(SI “Exploration in Atari”)。已有工作表明,内在动机算法从领域知识中获益要少得多:用同样的领域知识表示,一种基于计数的探索算法在蒙特祖玛的复仇上只有 12,240 分34。
A hard-exploration robotics environment
一个难探索的机器人环境
While robotics is a promising application for RL and it is often easy to define the high-level goal of a robotics task (e.g. to put a cup in a cupboard), it is much more difficult to define a sufficiently dense reward function10 (e.g. reward all of the low-level motor commands to move toward the cup, grasp it, etc.). Go-Explore enables forgoing such a dense reward function in favour of a sparse reward function that only considers the high-level task. Additionally, robot policies are usually trained in simulation before being transferred to the real world7,21–23, making robotics a natural domain to demonstrate the usefulness of harnessing the ability to restore simulator state.
机器人是强化学习一个很有前景的应用,而且任务的高层目标通常不难定义(例如把杯子放进橱柜),但要定义足够密的奖励函数却困难得多10(例如,对靠近杯子、抓住杯子等所有底层运动指令都给奖励)。有了 Go-Explore,就可以放弃这种稠密奖励,改用只考虑高层任务的稀疏奖励。此外,机器人策略通常先在仿真里训练,再迁移到真实世界7,21–23,因此机器人领域很适合用来展示:利用模拟器可恢复状态这一能力,究竟有什么用处。
The following experiment, featuring a realistic simulation of a real-world robot35, demonstrates that Go-Explore can solve a practical hard-exploration task where a robot arm must pick up an object and put it inside of one of four shelves, two of which are behind latched doors (Fig. 4a). A reward is given only when the object is put into a specified target shelf. A state-of-the-art RL algorithm for continuous control (PPO20) does not encounter a single reward after training in this environment for a billion frames, showcasing the hard-exploration nature of this problem. Go-Explore’s “explore” step takes random actions and states are assigned to cells with an easy-to-provide domain-knowledge-based mapping (Methods).
下面的实验采用一台真实机器人的仿真模型35,表明 Go-Explore 能够解决一项实用的难探索任务:机械臂需要抓起一个物体,并把它放进四个架子中的一个,其中两个架子还挡在带闩的门后面(图 4a)。只有当物体被放进指定的目标架子时,才给出奖励。一种用于连续控制的当时最好强化学习算法(PPO20)在这个环境中训练了 10 亿帧,却一次奖励都没有遇到,足见这个问题的难探索性质。Go-Explore 的“explore”一步采取随机动作,状态则用一种不难提供的、基于领域知识的映射分到各个格子中(Methods)。
The exploration phase quickly and reliably discovers trajectories for putting the object in each of the four shelves (Fig. 4b and Extended Data Fig. 5a). Go-Explore succeeds because it thoroughly explores its environment without suffering from detachment (e.g. once each cupboard is opened, Go-Explore never forgets about those states) or derailment (Go-Explore can directly restore to difficult-to-reach states like grasping). In contrast, a count-based intrinsic motivation algorithm with the same representation as the exploration phase is incapable of discovering any reward (Fig. 4c), and discovers only a fraction of the cells discovered by the exploration phase after 2 billion frames of training, 100 times more than the exploration phase (Fig. 4b). Despite receiving intrinsic rewards for touching the object, this control was incapable of learning to reliably grasp objects. Evidence suggests that this failure to grasp is due to the problem of derailment (SI “Derailment in robotics”), which Go-Explore is specifically designed to solve. Robustifying the trajectories found by Go-Explore produces robust policies in 99% of cases (Fig. 4c).
探索阶段能够迅速而可靠地找到把物体放进四个架子中每一个的轨迹(图 4b 和 Extended Data Fig. 5a)。Go-Explore 之所以成功,是因为它把环境探索得很充分,又不会遇到脱离(例如,每个橱柜一旦打开,Go-Explore 就不会忘掉那些状态)或脱轨(Go-Explore 可以直接恢复到像抓住物体这样难以到达的状态)。相比之下,一种采用与探索阶段相同表示的基于计数的内在动机算法,完全发现不了任何奖励(图 4c);即便训练了 20 亿帧——相当于探索阶段的 100 倍——它所发现的格子也只占探索阶段的一小部分(图 4b)。尽管接触物体就能获得内在奖励,这一对照仍无法学会可靠地抓住物体。证据表明,抓取失败正是由于脱轨问题(SI “Derailment in robotics”),而 Go-Explore 正是为解决这个问题而设计的。对 Go-Explore 找到的轨迹做稳健化后,在 99% 的情形中都能得到稳健策略(图 4c)。
Figure 4: Go-Explore can Solve a Challenging, Sparse-Reward, Simulated Robotics Task. (a) A simulated Fetch robot needs to grasp an object and put it in one of four shelves. (b) The exploration phase significantly outperforms an intrinsic motivation control using the same cell representation. (c) For each of four different target locations, including the two with a door, the robot is able to learn to pick the object up and place it on the shelf in 99% of trials. Shaded areas show 95% bootstrap CIs of the mean with 1,000 samples.
图 4:Go-Explore 能够解决一项具有挑战性、奖励稀疏的仿真机器人任务。(a) 仿真中的 Fetch 机器人需要抓住一个物体,并把它放进四个架子中的一个。(b) 探索阶段明显优于采用相同格子表示的内在动机对照。(c) 对四个不同的目标位置,包括两个带门的位置,机器人都能在 99% 的试验中学会捡起物体并放到架子上。阴影区域表示对均值做 1,000 次抽样得到的 95% 自助法置信区间。
Policy-based Go-Explore
基于策略的 Go-Explore
Leveraging the ability of simulators to restore states increases Go-Explore’s efficiency, but is not a requirement. When returning, instead of restoring simulator state, it is possible to execute a policy conditioned on (i.e. told to go to) the cell to return to, which we call policy-based Go-Explore. There are advantages to doing so. First, it enables sampling from the policy during the “explore” step, which can substantially increase exploration efficiency vs. taking random actions because the policy can generalise to new situations (e.g. it need only learn to overcome a type of obstacle once instead of solving that problem again each time via random actions). To test this hypothesis, our implementation commits with equal probability to either taking random actions or sampling from the policy for the duration of the “explore” step, making it possible to compare random and policy-based exploration (Methods). Second, training a policy in the exploration phase obviates the need for robustification and thus removes its associated additional complexity, hyperparameters, and overhead. Finally, policy-based Go-Explore can explore directly in a stochastic environment (which we do in our experiments) and can potentially handle forms of stochasticity not explored in our experiments (e.g. stochastic rewards; SI “Policy-based Go-Explore and Stochasticity”).
利用模拟器恢复状态的能力可以提高 Go-Explore 的效率,但这并不是必要条件。在回去时,可以不恢复模拟器状态,而是执行一个以要回去的格子为条件(即被要求走到该格子)的策略,我们称之为基于策略的 Go-Explore。这样做有若干好处。第一,它可以在“explore”一步中从策略里采样,与采取随机动作相比,探索效率可以明显提高,因为策略能够推广到新情境(例如,只需学会克服某一类障碍一次,而不必每次都靠随机动作重新解决)。为了检验这一假设,我们的实现在整个“explore”一步期间,会以相等概率固定采用随机动作或从策略中采样,从而可以比较随机探索与基于策略的探索(Methods)。第二,在探索阶段就训练策略,便不必再做稳健化,也就去掉了稳健化带来的额外复杂性、超参数和开销。最后,基于策略的 Go-Explore 可以直接在随机环境中探索(我们的实验便是如此),并且有可能处理我们实验中未曾涉及的其他随机性(例如随机奖励;SI “Policy-based Go-Explore and Stochasticity”)。
The goal-conditioned policy is trained during the exploration phase with a common RL algorithm (PPO20). Because goal-conditioned policies often struggle to reach distant states36 (SI “Ablations”), the policy is guided towards the selected state by being presented with intermediate goals along the best trajectory that previously led to the selected state (Methods). Policy-based Go-Explore includes additional innovations to promote exploration and stabilise learning, the most important of which are Self-Imitation Learning37 (SI “Ablations”), dynamic entropy increase, soft-trajectories, and dynamic episode limits, all discussed in detail in Methods.
目标条件策略在探索阶段用一种常见的强化学习算法(PPO20)训练。由于目标条件策略往往难以到达远处的状态36(SI “Ablations”),策略会沿着先前通往所选状态的最佳轨迹,被依次给出中间目标,从而被引导到所选状态(Methods)。基于策略的 Go-Explore 还包含若干促进探索、稳定学习的额外设计,其中最重要的是自我模仿学习(Self-Imitation Learning)37(SI “Ablations”)、动态增大熵、软轨迹,以及动态回合长度限制,这些都在 Methods 中详细讨论。
Policy-based Go-Explore was tested on Montezuma’s Revenge and Pitfall with the domain-knowledge cell representation for the archive (and to represent the goal to the policy; the game state is input as pixels). It beats the state of the art and human performance with a mean reward of 97,728 points on Montezuma’s Revenge and 20,093 points on Pitfall (Fig. 5), demonstrating that Go-Explore’s performance is not merely a result of its ability to leverage simulator restorability, but is a function of its overall design. Policy-based Go-Explore also outperforms a concurrently developed, similar algorithm34 in terms of performance and sample efficiency (SI “Comparing Policy-based Go-Explore and DTSIL”). Furthermore, confirming our hypothesis, sampling from the policy is more effective at discovering new cells than taking random actions, and becomes increasingly effective across training because the policy gains new, generally useful skills, ultimately resulting in the discovery of over four times more cells than random actions on both Montezuma’s Revenge and Pitfall (Extended Data Fig. 7), highlighting the potential of goal-conditioned, policy-based exploration over the usual random actions used in RL.
基于策略的 Go-Explore 在蒙特祖玛的复仇和 Pitfall 上做了测试,档案采用带领域知识的格子表示(同时也用它向策略表示目标;游戏状态则以像素输入)。它以蒙特祖玛的复仇上平均 97,728 分、Pitfall 上平均 20,093 分的成绩,超过了当时最好成绩和人类水平(图 5),说明 Go-Explore 的表现并不仅仅来自利用模拟器可恢复性,而是来自它的整体设计。在表现和样本效率上,基于策略的 Go-Explore 也优于一种同时期开发的相似算法34(SI “Comparing Policy-based Go-Explore and DTSIL”)。此外,正如我们的假设,从策略中采样比采取随机动作更善于发现新格子,并且随着训练进行会越来越有效,因为策略逐步获得了新的、普遍有用的技能,最终在蒙特祖玛的复仇和 Pitfall 上发现的格子都超过随机动作四倍以上(Extended Data Fig. 7),这凸显了目标条件、基于策略的探索,相对于强化学习中常用随机动作所具有的潜力。
Figure 5: Policy-based Go-Explore with domain knowledge outperforms state-of-the-art and human performance in Montezuma’s Revenge and Pitfall. On both Montezuma’s Revenge (a) and Pitfall (b), performance increases throughout the run, suggesting even higher performance is possible with additional training time. Shaded areas show 95% bootstrap CIs of the mean with 1,000 samples.
图 5:带领域知识的基于策略的 Go-Explore,在蒙特祖玛的复仇和 Pitfall 上超过当时最好成绩和人类水平。在蒙特祖玛的复仇 (a) 和 Pitfall (b) 上,表现在整个运行过程中持续上升,说明若再增加训练时间,还有可能达到更高分数。阴影区域表示对均值做 1,000 次抽样得到的 95% 自助法置信区间。
Conclusion
结论
The effectiveness of the Go-Explore family of algorithms presented in this work suggests that it will enable progress in many domains that can be framed as sequential decision making problems, including robotics7,21–23, language understanding38, and drug design39. However, these instantiations represent only a fraction of the possible ways in which the Go-Explore paradigm can be implemented, opening up many exciting possibilities for future research. A key direction for future work is to learn cell representations, such as through compression-based methods40,41, contrastive-predictive encodings42 or auxiliary tasks43, which would allow Go-Explore to generalise to even more complex domains. Other future extensions could learn to choose which cells to return to, learn which cells to try to reach during the exploration step, learn a specialised policy for exploration in the “explore” step, learn to explore safely in the real world by mining diverse catastrophes in simulation, maintain a continuous density-based archive rather than a discrete cell-based one, improve sample efficiency by leveraging multiple trajectories (or even all transitions) from a single exploration-phase run or improve the robustification phase to work from a single demonstration, and so on. Furthermore, the planning-like nature of the Go-Explore exploration phase highlights the potential of porting other powerful planning algorithms like MCTS44, RRT45, A*46, or conformant planning47 to high-dimensional state-spaces. These new directions offer rich possibilities to improve the generality, performance, robustness, and efficiency of algorithms inspired by Go-Explore. Finally, the insights presented in this work extend broadly; the simple decomposition of remembering previously found states, returning to them, and then exploring from them appears to be especially powerful, suggesting it may be a fundamental feature of learning in general. Harnessing these insights, either within or outside of the context of Go-Explore, may be essential to improve our ability to create generally intelligent agents.
本文提出的 Go-Explore 算法家族所展现出的效果表明,它将推动许多可以表述为序列决策问题的领域取得进展,包括机器人7,21–23、语言理解38 和药物设计39。不过,这些实例只是 Go-Explore 范式可能实现方式中的一小部分,从而为后续研究打开了许多令人兴奋的方向。未来工作的一个关键方向是学习格子表示,例如通过基于压缩的方法40,41、对比预测编码42 或辅助任务43,这将使 Go-Explore 能够推广到更复杂的领域。其他可能的扩展包括:学习选择回到哪些格子,学习在探索一步中尝试到达哪些格子,为“explore”一步学习专门的探索策略,通过在仿真中挖掘多样的灾难事件来学习如何在真实世界中安全探索,维护一份基于连续密度的档案而不是离散的格子档案,利用同一次探索阶段运行中的多条轨迹(甚至全部转移)来提高样本效率,或者让稳健化阶段只需一条演示就能工作,等等。此外,Go-Explore 探索阶段近似规划的性质,也提示人们可以把 MCTS44、RRT45、A*46 或 conformant planning47 等其他有力的规划算法移植到高维状态空间中。这些新方向为提高受 Go-Explore 启发的算法的通用性、表现、稳健性和效率提供了丰富可能。最后,本文提出的认识其实适用范围更广:先记住已经找到的状态,再回到这些状态,然后从那里探索,这一简单分解显得格外有力,暗示它可能是一般学习的一项基本特征。无论是在 Go-Explore 之内还是之外,把握这些认识,都可能对提高我们创造通用智能智能体的能力至关重要。
References
参考文献
- Bellemare, M. et al. Unifying count-based exploration and intrinsic motivation in NIPS (2016), 1471–1479.
- Lehman, J. & Stanley, K. O. Novelty Search and the Problem with Objectives in Genetic Programming Theory and Practice IX (GPTP 2011) (2011).
- Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).
- Vinyals, O. et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019).
- OpenAI et al. Dota 2 with Large Scale Deep Reinforcement Learning. arXiv preprint arXiv:1912.06680 (2019).
- Merel, J. et al. Hierarchical visuomotor control of humanoids. arXiv preprint arXiv:1811.09656 (2018).
- OpenAI et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research 39, 3–20 (2020).
- Lehman, J. et al. The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities. CoRR abs/1803.03453 (2018).
- Amodei, D. et al. Concrete Problems in AI Safety. CoRR abs/1606.06565 (2016).
- Smart, W. D. & Kaelbling, L. P. Effective reinforcement learning for mobile robots in Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292) 4 (2002), 3404–3410.
- Lehman, J. & Stanley, K. O. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation 19, 189–223 (2011).
- Conti, E. et al. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents in Advances in neural information processing systems (2018), 5027–5038.
- Puigdomènech Badia, A. et al. Agent57: Outperforming the Atari Human Benchmark. arXiv preprint arXiv:2003.13350 (2020).
- Machado, M. C. et al. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. J. Artif. Intell. Res. 61, 523–562 (2018).
- Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).
- Aytar, Y. et al. Playing hard exploration games by watching YouTube. arXiv preprint arXiv:1805.11592 (2018).
- Lipovetzky, N., Ramı́rez, M. & Geffner, H. Classical Planning with Simulators: Results on the Atari Video Games in IJCAI (2015).
- Sutton, R. S. & Barto, A. G. Reinforcement learning: An introduction (Bradford, 1998).
- Mnih, V. et al. Asynchronous methods for deep reinforcement learning in International conference on machine learning (2016), 1928–1937.
- Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. Proximal Policy Optimization Algorithms. CoRR abs/1707.06347 (2017).
- Cully, A., Clune, J., Tarapore, D. & Mouret, J.-B. Robots that can adapt like animals. Nature 521, 503–507 (2015).
- Peng, X. B., Andrychowicz, M., Zaremba, W. & Abbeel, P. Sim-to-real transfer of robotic control with dynamics randomization in 2018 IEEE international conference on robotics and automation (ICRA) (2018), 1–8.
- Tan, J. et al. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804.10332 (2018).
- Hester, T. et al. Deep Q-learning From Demonstrations in AAAI (2018).
- Guo, X., Singh, S. P., Lee, H., Lewis, R. L. & Wang, X. Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning in NIPS (2014).
- Bellemare, M. G., Naddaf, Y., Veness, J. & Bowling, M. The Arcade Learning Environment: An evaluation platform for general agents. J. Artif. Intell. Res.(JAIR) 47, 253–279 (2013).
- Horgan, D. et al. Distributed Prioritized Experience Replay. CoRR abs/1803.00933 (2018).
- Espeholt, L. et al. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures in ICML (2018).
- Salimans, T. & Chen, R. Learning Montezuma’s Revenge from a Single Demonstration. arXiv preprint arXiv:1812.03381 (2018).
- Van Hasselt, H. P., Guez, A., Hessel, M., Mnih, V. & Silver, D. Learning values across many orders of magnitude in Advances in Neural Information Processing Systems (2016), 4287–4295.
- Puigdomènech Badia, A. et al. Never Give Up: Learning Directed Exploration Strategies. arXiv (2020).
- Brockman, G. et al. OpenAI gym. arXiv preprint arXiv:1606.01540 (2016).
- ATARI VCS/2600 Scoreboard 2018. http://www.ataricompendium.com/game_library/high_scores/high_scores.html.
- Guo, Y. et al. Efficient Exploration with Self-Imitation Learning via Trajectory-Conditioned Policy. arXiv preprint arXiv:1907.10247 (2019).
- Wise, M., Ferguson, M., King, D., Diehr, E. & Dymesich, D. Fetch and freight: Standard platforms for service robot applications in Workshop on autonomous mobile service robots (2016).
- Eysenbach, B., Salakhutdinov, R. R. & Levine, S. Search on the replay buffer: Bridging planning and reinforcement learning in Advances in Neural Information Processing Systems (2019), 15220–15231.
- Oh, J., Guo, Y., Singh, S. & Lee, H. Self-Imitation Learning in ICML (2018).
- Madotto, A. et al. Exploration Based Language Learning for Text-Based Games. arXiv preprint arXiv:2001.08868 (2020).
- Popova, M., Isayev, O. & Tropsha, A. Deep reinforcement learning for de novo drug design. Science advances 4, eaap7885 (2018).
- Alvernaz, S. & Togelius, J. Autoencoder-augmented neuroevolution for visual doom playing in 2017 IEEE Conference on Computational Intelligence and Games (CIG) (2017), 1–8.
- Cuccu, G., Togelius, J. & Cudré-Mauroux, P. Playing atari with six neurons. arXiv preprint arXiv:1806.01363 (2018).
- Oord, A. v. d., Li, Y. & Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018).
- Jaderberg, M. et al. Reinforcement Learning with Unsupervised Auxiliary Tasks. CoRR abs/1611.05397 (2016).
- Chaslot, G., Bakkes, S., Szita, I. & Spronck, P. Monte-Carlo Tree Search: A New Framework for Game AI in AIIDE (2008).
- Lavalle, S. M. Rapidly-Exploring Random Trees: A New Tool for Path Planning tech. rep. (Iowa State University, 1998).
- Hart, P. E., Nilsson, N. J. & Raphael, B. A Formal Basis for the Heuristic Determination of Minimum Cost Paths. IEEE Transactions on Systems Science and Cybernetics 4, 100–107. ISSN : 0536-1567 (July 1968).
- Smith, D. E. & Weld, D. S. Conformant graphplan in AAAI/IAAI (1998), 889–896.
Acknowledgements We thank Ashley Edwards, Sanyam Kapoor, Felipe Petroski Such and Jiale Zhi for their ideas, feedback, technical support, and work on aspects of Go-Explore not presented in this work. We are grateful to the Colorado Data Center and OpusStack Teams at Uber for providing our computing platform. We thank Vikash Kumar for creating the MuJoCo files that served as the basis for our robotics environment (https://github.com/vikashplus/fetch).
致谢 我们感谢 Ashley Edwards、Sanyam Kapoor、Felipe Petroski Such 和 Jiale Zhi 提供的想法、反馈和技术支持,以及他们在未写入本文的 Go-Explore 相关工作上的贡献。我们感谢 Uber 的 Colorado Data Center 和 OpusStack 团队提供计算平台。我们感谢 Vikash Kumar 创建了作为本机器人环境基础的 MuJoCo 文件(https://github.com/vikashplus/fetch)。
Author contributions A.E. and J.H. contributed equally and are responsible for the technical work (with J.H. focusing primarily on policy-based Go-Explore and A.E. on most other technical contributions) as well as the initial draft of the paper. J.C. and K.O.S. led the team. All authors (A.E., J.H., J.L., K.O.S. and J.C.) significantly contributed to ideation, experimental design, analysing data, strategic decisions, developing the philosophical motivation for the algorithm, and editing the paper.
作者贡献 A.E. 与 J.H. 贡献相同,共同负责技术工作(J.H. 主要负责基于策略的 Go-Explore,A.E. 负责大部分其他技术内容)以及论文初稿。J.C. 与 K.O.S. 领导团队。所有作者(A.E.、J.H.、J.L.、K.O.S. 和 J.C.)都在提出想法、实验设计、数据分析、战略决策、形成算法的思想动机以及修改论文方面作出了重要贡献。
Competing Interests Uber Technologies, Inc. has filed a publicly available provisional patent application 16/696,893 about some Go-Explore variants featuring a deep reinforcement learning model, with all authors (A.E., J.H., J.L, K.O.S. and J.C.) listed as inventors.
竞争性利益 Uber Technologies, Inc. 已就部分带有深度强化学习模型的 Go-Explore 变体,提交了可公开查阅的临时专利申请 16/696,893,全部作者(A.E.、J.H.、J.L、K.O.S. 和 J.C.)均列为发明人。
Supplementary information is available for this paper.
本文提供补充信息(Supplementary information)。
Correspondence should be addressed to Adrien Ecoffet (email: adrienecoffet@gmail.com), Joost Huizinga (email: joost.hui@gmail.com), and Jeff Clune (email: jclune@gmail.com).
Methods
方法
State of the art on Atari. With new work on RL for Atari being published on a regular basis, and with reporting methods often varying significantly, it can be difficult to establish the state-of-the-art score for each Atari game. At the same time, it is important to compare new algorithms with previous ones to evaluate progress.
Atari 上的当时最好成绩。 关于 Atari 强化学习的新工作在不断发表,报告方式又常常差别很大,因而要确定每一款 Atari 游戏的当时最好分数并不容易。与此同时,把新算法与先前算法加以比较,对评估进展来说又很重要。
For determining the state-of-the-art score for each game, we considered a set of notable, recently published papers that cover at least the particular subset of games this paper focuses on, namely hard-exploration games. Community guidelines advocate sticky actions, which approximate the minor lack of precision in control that a human might have (e.g. continuing to tilt the joystick for a fraction of a second longer than intended), as a way to evaluate agents on Atari14. There is substantial evidence to show that sticky actions can decrease performance substantially compared to the now deprecated no-ops evaluation strategy14,48,49. As a result, we exclude work which was only evaluated with no-ops from our definition of state of the art. Fig. 2a includes works tested only with no-ops as they help bring context to the amount of effort expended by the community on solving Montezuma’s Revenge. We did not include work that does not provide individualised scores for each game. To avoid cherry-picking lucky rollouts that can bias scores upward substantially, we also exclude work that only provided the maximum score achieved in an entire run as opposed to the average score achieved by a particular instance of the agent.
为确定每款游戏的当时最好分数,我们考察了一组近期发表的重要论文,它们至少覆盖了本文所关注的那一类游戏,也就是难探索游戏。社区指南主张用黏滞动作来评估 Atari 上的智能体14,这种设置近似于人类控制中那种轻微的不精确(例如摇杆多倾斜了零点几秒)。已有充分证据表明,与现已不再推荐的 no-ops 评估策略相比,黏滞动作会明显降低成绩14,48,49。因此,我们把只在 no-ops 下评估过的工作排除在“当时最好成绩”的定义之外。图 2a 仍收入了只在 no-ops 下测试的工作,因为它们有助于说明社区为解决蒙特祖玛的复仇投入了多少努力。没有为每款游戏提供单独分数的工作,我们没有纳入。为避免挑出运气好的回合从而把分数明显抬高,我们也排除了只报告整次运行中最高分、而不给出某一智能体实例平均分的工作。
In total, state of the art results were extracted from the following papers: Burda et al. (2018)50, Castro et al. (2018)48, Choi et al. (2018)51, Fedus et al. (2019)52, Taiga et al. (2020)53, Tang et al. (2020)54, and Toromanoff et al. (2019)49. Because these works themselves report scores for several algorithms and variants, including reproductions of previous algorithms, a total of 23 algorithms and variants were included in the state of the art assessment. For each game, the state-of-the-art score was the highest score achieved across all algorithms.
当时最好成绩总共摘自以下论文:Burda 等人(2018)50、Castro 等人(2018)48、Choi 等人(2018)51、Fedus 等人(2019)52、Taiga 等人(2020)53、Tang 等人(2020)54,以及 Toromanoff 等人(2019)49。这些工作本身就报告了多种算法和变体的分数,包括对先前算法的复现,因此当时最好成绩的评估一共纳入了 23 种算法和变体。对每一款游戏,当时最好分数取所有这些算法中的最高分。
Downscaling on Atari. In the first variant of Go-Explore presented in this work (Sec. “Learning Atari with state restoration”), the cell representation is a downscaled version of the original game frame, which can be applied in any domain where the state is a visual observation (SI “Generality of downscaling”).
Atari 上的画面缩小。 在本文介绍的第一种 Go-Explore 变体中(“Learning Atari with state restoration”一节),格子表示是把原始游戏画面缩小后的结果,这一做法可用于任何状态为视觉观测的领域(SI “Generality of downscaling”)。
To obtain the downscaled representation, (1) the original frame is converted to grayscale, (2) its resolution is reduced with pixel area relation interpolation to a width and a height , and (3) the pixel depth is reduced to using the formula , where is the value of the pixel after step (2). A fixed set of values for the parameters , and would not generalise across games because visuals (e.g. the amount of detail shown on screen and how much it varies across frames) vary substantially between games (SI “Ablations”). Therefore these parameters are updated dynamically by proposing different values for each, calculating how a sample of recent frames would be grouped into cells under these proposed parameters, and then selecting the values that result in the best cell distribution (as determined by the objective function defined below).
要得到缩小后的表示,(1) 先把原始帧转为灰度,(2) 用像素面积关系插值把分辨率降到宽度 、高度 ,(3) 再用公式 把像素深度降到 ,其中 是第 (2) 步之后的像素值。若对参数 、 和 使用固定取值,将难以在不同游戏间通用,因为画面风格(例如屏幕上细节的多少,以及它随帧变化的程度)在不同游戏之间差别很大(SI “Ablations”)。因此这些参数会动态更新:对每个参数提出不同取值,计算在这些候选参数下近期帧样本会被如何分入格子,然后选出能给出最佳格子分布的取值(由下文定义的目标函数判定)。
The objective function for candidate downscaling parameters is calculated based on a target number of cells (where is a fixed fraction of the number of cells in the sample, 12.5% in our experiment, though the algorithm is relatively robust to different values, see SI “Downscaling target proportion”), the actual number of cells produced by the parameters currently considered , and the distribution of sample frames over cells . Its general form is
候选缩小参数的目标函数,依据目标格子数 ( 是样本中格子数的一个固定比例,本实验中为 12.5%,不过算法对这一取值相对稳健,见 SI “Downscaling target proportion”)、当前所考虑参数实际产生的格子数 ,以及样本帧在各格子上的分布 来计算。其一般形式为
measures the discrepancy between the number of cells under the current parameters and the target number of cells . It prevents the representation that is discovered from aggregating too many frames together, which would result in low exploration, or from aggregating too few frames together, which would result in an intractable time and memory complexity, and is defined as
衡量当前参数下的格子数 与目标格子数 之间的差距。它防止所得到的表示把过多帧聚在一起(那样探索会不足),或把过少帧聚在一起(那样时间和内存复杂度会难以承受),其定义为
is the ratio of the entropy of how frames were distributed across cells to the entropy of the discrete uniform distribution of size , i.e. the normalised entropy. In this way, the loss encourages frames to be distributed as uniformly as possible across cells, which is important because highly non-uniform distributions may suffer from the same lack of exploration that excessive aggregation can produce or the same intractability that lack of aggregation can produce. Unlike unnormalized entropy, normalised entropy is comparable across different numbers of cells, allowing the number of cells to be controlled solely by . Its form is
是帧在各格子上分布的熵,与大小为 的离散均匀分布的熵之比,也就是归一化熵。这样,损失会鼓励帧在各格子上尽可能均匀分布;这一点很重要,因为高度不均匀的分布,可能出现与过度聚合同样的探索不足,或与聚合不足同样的计算上难以承受的问题。与未归一化的熵不同,归一化熵在不同格子数之间可以比较,从而可以把格子数单独交给 来控制。其形式为
At each step of the randomised search, new values of each parameter , and are proposed by sampling from a geometric distribution whose mean is the current best known value of the given parameter. If the current best known value is lower than a minimum mean (set to approximately 1/20th of the maximum value of each parameter: 8 for , 10.5 for and 12 for ), the minimum mean is used as the mean of the geometric distribution (SI “Downscaling distribution minimum means” shows that the algorithm is not overly sensitive to the particular setting of the minimum means). New parameter values are re-sampled if they fall outside of the valid range for that parameter. In our implementation, the randomised search runs for 3,000 iterations.
在随机搜索的每一步,会对参数 、 和 各自提出新的取值,方法是从几何分布中采样,该分布的均值取为该参数当前已知的最好值。若当前最好值低于一个最小均值(设为各参数最大值的约 1/20:对 为 8,对 为 10.5,对 为 12),就用这个最小均值作为几何分布的均值(SI “Downscaling distribution minimum means” 表明,算法对最小均值的具体设定并不过分敏感)。若新参数值超出该参数的有效范围,则会重新采样。在我们的实现中,随机搜索运行 3,000 次迭代。
The recent frames that constitute the sample over which parameter search is done are obtained by maintaining a set of recently seen sample frames as Go-Explore runs: each time a frame not already in the set is seen during the explore step, it is added to the running set with a probability of 1%, ensuring that the set contains a diverse set of frames rather than just the most recent frames (SI “Downscaling sampling rate” shows that the algorithm is not overly sensitive to the value of this parameter). If the resulting set contains more than 10,000 frames, the oldest frame it contains is removed. This set is reminiscent of a FIFO replay buffer, except that, because it is not used for training the network, it only stores individual frames, rather than complete state transitions.
构成参数搜索所用样本的近期帧,是通过在 Go-Explore 运行过程中维护一组最近见过的样本帧得到的:每当探索一步中看到一个尚不在该集合中的帧,就以 1% 的概率把它加入这个滚动集合,从而保证集合里是多样的帧,而不仅仅是最新的那些帧(SI “Downscaling sampling rate” 表明算法对该参数的取值并不过分敏感)。若集合超过 10,000 帧,就去掉其中最旧的一帧。这个集合让人想起 FIFO 回放缓冲区,只不过因为它并不用于训练网络,所以只存储单独的帧,而不是完整的状态转移。
The first downscaling parameters are computed after running with a single-cell representation for 40,000 frames. To handle changes in frame distribution as exploration progresses and to avoid being stuck with a bad representation, the search for a new representation is performed every 40 million frames. To avoid excessive memory usage, the representation is also recomputed if the number of cells in the archive exceeds 50,000. When switching to a new representation, a new archive is created and initialised by converting all previous archives to the new representation using the frames corresponding to each state in the previous archives (the number of cells in the archive on Atari over time can be seen in Extended Data Fig. 3a).
第一组缩小参数,是在用单一格子表示运行 40,000 帧之后计算的。为了应对探索推进时帧分布的变化,并避免卡在不好的表示上,每隔 4,000 万帧会重新搜索一次新的表示。为避免占用过多内存,若档案中的格子数超过 50,000,也会重新计算表示。切换到新表示时,会新建一份档案,并用先前各份档案中每个状态所对应的帧,把这些旧档案转换到新表示上来初始化(Atari 上档案格子数随时间的变化见 Extended Data Fig. 3a)。
Hyperparameter values were found by an initial randomised sweep on Montezuma’s Revenge, with the 10 best combinations then tested on Gravitar to ensure their generalisability (it is the norm in hard-exploration work to include Montezuma’s Revenge as part of the tuning set1,13,31,34,50,51,53,55–57 , though which other games are included, if any, varies). Aside from the hyperparameters examined in SI “Ablations” (the target proportion, minimum means and buffer sampling rate), the hyperparameters control the tradeoff between computational and memory efficiency and the quality of downscaling parameters obtained (e.g. increasing the number of search iterations is likely to produce better parameters at the cost of more time spent searching for parameters), and should thus be set according to the computational constraints of the user. In our experiments, approximately 25% of the computation spent on the exploration phase was spent searching for new downscaling parameters.
超参数取值先在蒙特祖玛的复仇上做了一轮随机扫描,再把其中最好的 10 组组合拿到 Gravitar 上测试,以确认其泛化性(在难探索研究中,把蒙特祖玛的复仇纳入调参集合是惯例1,13,31,34,50,51,53,55–57,至于是否以及纳入哪些其他游戏,则各不相同)。除 SI “Ablations” 中考察的那些超参数(目标比例、最小均值和缓冲区采样率)以外,其余超参数主要权衡计算与内存效率,以及所得到缩小参数的质量(例如,增加搜索迭代次数,往往能得到更好的参数,但搜索参数本身也更费时间),因而应根据使用者的计算条件来设置。在我们的实验中,探索阶段大约 25% 的计算量花在搜索新的缩小参数上。
Domain knowledge representations. The domain knowledge representation for Pitfall consists of the current room (out of 255) the agent is currently located in, as well as the discretised x, y position of the agent. In Go-Explore without a return policy, the x, y position is discretised in 8 by 16 pixel cells. Policy-based Go-Explore uses the coarser-grained 18 by 18 pixel cell representation from Guo et al. (2019)34 , which reduces training time (there are fewer cells the policy needs to learn how to reach) without hindering exploration. In Montezuma’s Revenge, the representation also includes the keys currently held by the agent (including which room they were found in) as well as the current level. Most of these features (level, room and x, y position) serve to specify the location of the agent, capturing the intuition that exploration requires discovering the different available locations within a space, while the keys held by the agent are important affordances that allow the agent to reach new locations. Although this information can in principle be extracted from RAM, in this work it was extracted from pixels through small hand-written classifiers, showing that domain knowledge representations need not require access to the inner state of a simulator. For practical applications, the features that help with exploration are often easier to identify and obtain than the features that are necessary for a policy to successfully execute a task. For example, in a task where a robot has to pick up an object, it is clear that the robot should explore different positions for its end effector in order to find a good grip on the object and the end effector position is generally easy to obtain58,59 , but a policy executing such a task will also need to recognise the object itself under a wide range of circumstances, which may require advanced image processing that benefits from being learned60 .
领域知识表示。 Pitfall 的领域知识表示包括智能体当前所在的房间(共 255 个),以及智能体离散化后的 、 位置。在不使用返回策略的 Go-Explore 中,、 位置按 8×16 像素的格子离散化。基于策略的 Go-Explore 采用 Guo 等人(2019)34 中更粗的 18×18 像素格子表示,这样可以缩短训练时间(策略需要学会到达的格子更少),同时又不会妨碍探索。在蒙特祖玛的复仇中,表示还包括智能体当前持有的钥匙(以及它们是在哪个房间找到的)和当前关卡。这些特征中的大多数(关卡、房间和 、 位置)用来标明智能体的位置,体现的直觉是:探索需要发现环境中各个可到达的位置。钥匙则捕捉智能体当前拥有哪些能力(例如能否打开某扇门)。这类位置和物品类特征在许多领域都容易提供:在机器人中,末端执行器的位置通常不难获得58,59,但要执行这类任务的策略,还需要在多种多样的情形下认出物体本身,这可能需要受益于学习的高级图像处理60。
In robotics, the domain knowledge representation is extracted from the internal state of the MuJoCo61 simulator. However, similar information has been extracted from raw camera footage for real robots by previous work7 . It consists of the current 3D position of the robot’s gripper, discretised in voxels with sides of length 0.5 meters, whether the robot is currently touching (with a single grip) or grasping (touching with both grips) the object, and whether the object is currently in the target shelf. In the case of the two target shelves with doors, the positions of the door and its latch are also included. The discretization for latches and doors follows the following formula, given that is the distance of the latch/door from its starting position in meters: .
在机器人任务中,领域知识表示取自 MuJoCo61 模拟器的内部状态。不过,先前工作已经表明,类似信息也可以从真实机器人的原始摄像画面中提取7。它包括机器人夹爪当前的三维位置(按边长 0.5 米的体素离散化)、机器人当前是单侧接触还是双侧夹住物体,以及物体是否已在目标架子中。对于两个带门的目标架子,门及其闩的位置也被纳入。闩和门的离散化采用如下公式,其中 是闩或门相对起始位置的距离(以米计):。
Exploration phase. During the exploration phase (SI Algorithm 1), the selection probability of a cell at each step is proportional to its selection weight, which unless otherwise specified is calculated as:
探索阶段。 在探索阶段(SI Algorithm 1),每一步选择某个格子的概率与它的选择权重成正比;除非另行说明,该权重的计算方式为:
where is the number of exploration steps in which that cell is visited (i.e. the count of a cell is increased by one when it is visited in the exploration step, even if the cell was visited multiple times in that step). This reciprocal square root weight is similar to the exploration bonus used in algorithms such as UCT62 and count-based intrinsic motivation algorithms1,63 .
其中 是该格子在探索步中被访问的次数(也就是说,只要它在探索一步中被访问到,即使该步内被访问了多次, 计数也只加一)。这种倒数平方根权重,类似于 UCT62 以及基于计数的内在动机算法1,63 中使用的探索奖励。
One advantage of introducing domain knowledge into cell representations is that we can leverage our semantic understanding of domain features to improve cell selection. We demonstrate this advantage on Montezuma’s Revenge with domain knowledge but without a return policy, where we define the cell selection weight based on: (1) The number of horizontal neighbours to the cell present in the archive (h); (2) A key bonus: for each location (defined by level, room, and x,y position), the cell with the largest number of keys at that location gets a bonus of k = 1 (k = 0 for other cells); (3) the current level. The first two values contribute to the location weight
把领域知识引入格子表示的一个好处是,我们可以利用对这些领域特征的语义理解,来改进格子选择。我们在带领域知识、但不使用返回策略的蒙特祖玛的复仇上展示了这一好处:格子选择权重由以下三项决定:(1) 档案中与该格子相邻的水平邻居数 ();(2) 钥匙奖励:对每个位置(由关卡、房间和 位置定义),该位置上钥匙数最多的格子获得 的奖励(其他格子 );(3) 当前关卡。前两项构成位置权重
This value captures the intuition that a cell that lacks neighbours in the archive is likely to be at the current frontier of search (vertical neighbours do not have the same effect as it is often more difficult to move from one vertical level to another, requiring e.g. a ladder to be present), and that an agent has more exploration capacity (i.e. affordances) if it is holding more keys. is then combined with W above as well as the level of the given cell l and the maximum level in the archive L to obtain the final weight for Montezuma’s Revenge with domain knowledge:
这一取值体现的直觉是:若某个格子在档案中缺少邻居,它很可能处于当前搜索的前沿(垂直方向的邻居没有同样的效果,因为在垂直层面之间移动通常更困难,例如需要有梯子),以及智能体持有的钥匙越多,探索能力(即可供使用的行动条件)就越强。随后将 与前面的 ,以及该格子的关卡 和档案中的最高关卡 结合起来,得到带领域知识时蒙特祖玛的复仇的最终权重:
This level-weighing puts a much stronger weight on cells in the highest level reached so far, thus focusing exploration on the frontier of search. These domain knowledge features substantially improve sample complexity in Montezuma’s Revenge relative to the default selection weight W defined above, but Go-Explore with the default selection weight is still able to get to the end of level 3, and thus still finds trajectories that traverse Montezuma’s Revenge in its entirety (SI “Ablations”). While it is possible to produce an analogous domain knowledge cell selection weight for Pitfall with domain knowledge, no such weight produced any substantial improvement over W alone.
这种按关卡加权会给目前到达的最高关卡中的格子以强得多的权重,从而把探索集中在搜索前沿。相对上面的默认选择权重 ,这些领域知识特征在蒙特祖玛的复仇上显著改善了样本复杂度;不过,即使用默认选择权重,Go-Explore 仍然能够到达第 3 关的尽头,从而仍然能找到贯穿整款蒙特祖玛的复仇的轨迹(SI “Ablations”)。虽然也可以为带领域知识的 Pitfall 构造类似的格子选择权重,但没有任何一种这样的权重,能比单独使用 带来实质性的改进。
Unless otherwise specified, once a cell is returned to, exploration proceeds with random actions for a number of steps (100 in Atari, 30 in robotics), or until the end of episode signal is received from the environment. In Atari, where the action set is discrete, actions are chosen uniformly at random. In robotics, each of the 9 continuous-valued components of the action is sampled independently and uniformly from the interval from -1 to 1. To help explore in a consistent direction, the probability of repeating the previous action is 95% for Atari and 90% for robotics. The effect of action repetition is investigated in SI “Ablations”.
除非另行说明,一旦回到某个格子,探索就会以随机动作进行若干步(Atari 中为 100 步,机器人中为 30 步),或直到环境发出回合结束信号。在动作集合为离散的 Atari 中,动作按均匀随机选择。在机器人中,动作的 9 个连续值分量各自独立地在 -1 到 1 的区间上均匀采样。为了帮助沿较为一致的方向探索,重复上一个动作的概率在 Atari 中为 95%,在机器人中为 90%。动作重复的影响见 SI “Ablations”。
For increased efficiency, the exploration phase is processed in parallel by selecting a batch of return cells and exploring from each one of them across multiple processes. In all runs without a return policy, the batch size is 100.
为提高效率,探索阶段采用并行处理:一次选出一批要回去的格子,并在多个进程中分别从这些格子出发探索。在所有不使用返回策略的运行中,批量大小为 100。
All reported experiments, except those involving policy-based Go-Explore, return by directly restoring simulator state. This method of returning is available whenever a simulator is available, which is the case for most RL experiments; due to the large number of training trials current RL algorithms require, as well as the safety concerns that arise when running RL directly in the real world, simulators have played a key role in training the most compelling applications of RL, and will likely continue to be harnessed for the foreseeable future.
除基于策略的 Go-Explore 相关实验外,所有报告的实验都通过直接恢复模拟器状态来完成回去。只要有模拟器,就可以用这种方式回去,而这对大多数强化学习实验都成立;由于当前强化学习算法需要大量训练试验,以及直接在真实世界中运行强化学习所带来的安全顾虑,模拟器在训练最引人注目的强化学习应用中一直起着关键作用,并且在可预见的未来很可能仍会被继续使用。
The backward algorithm. The “backward algorithm”29 places the agent close to the end of the trajectory and runs PPO (SI “PPO and SIL”) until the performance of the agent matches that of the demonstration. Once that is achieved, the agent’s starting point is moved closer to the trajectory’s beginning and the process is repeated.
反向算法。 “反向算法”29 把智能体放在接近轨迹末端的位置,并运行 PPO(SI “PPO and SIL”),直到智能体的表现与演示相当。一旦达到这一点,就把智能体的起点沿轨迹向起始端移近一些,再重复这一过程。
The algorithm was modified to support multiple (10, in our experiments) demonstrations by selecting a demonstration uniformly at random at the start of each episode, which stabilises learning. The demonstrations can be obtained cheaply by running the exploration phase multiple times. In Atari, the agent may be able to find rewards from the starting position before it has worked backwards all the way to the start in a way that matches the demonstration performance. To track such partial progress, a virtual “demonstration” corresponding to starting the agent at the true starting point was added (SI “Multiple demonstrations”). This process was not performed in the robotics environment as there is only one point to score, making partial success impossible. Self-Imitation Learning (SIL)37 was performed on the demonstrations provided to the backward algorithm (SI “PPO and SIL”). In Atari, we normalise the rewards based on the mean absolute returns found in the demonstrations to allow a single set of hyperparameters to be used across all games, including those with widely varying reward magnitudes (SI “Reward scaling”). Pseudo-code, which includes the modifications above, is shown in SI Algorithm 2, and the neural-network architectures that were trained are shown in Extended Data Fig. 1.
该算法经过修改以支持多条演示(本实验中为 10 条):在每个回合开始时均匀随机选择一条演示,从而稳定学习。这些演示可以通过多次运行探索阶段便宜地获得。在 Atari 中,智能体有可能在还没有沿着轨迹一路匹配到演示表现、退回到真正起点之前,就已经能从起始位置找到奖励。为了跟踪这种部分进展,我们加入了一条虚拟“演示”,对应于把智能体放在真正的起点(SI “Multiple demonstrations”)。机器人环境中没有做这一步,因为那里只有一个得分点,不存在部分成功。对提供给反向算法的演示,还进行了自我模仿学习(SIL)37(SI “PPO and SIL”)。在 Atari 中,我们根据演示中平均绝对回报来归一化奖励,从而可以在奖励量级差异很大的全部游戏上使用同一组超参数(SI “Reward scaling”)。包含上述修改的伪代码见 SI Algorithm 2,所训练的神经网络结构见 Extended Data Fig. 1。
Evaluation. In Atari, the score of an exploration phase run is measured as the highest score ever achieved at episode end (SI “Score tracking in the exploration phase”). For the 11 focus games, exploration phase scores are averaged across 50 exploration phase runs. For the other games, they are averaged across 5 runs. For domain knowledge, they are averaged across 100 runs.
评估。 在 Atari 中,一次探索阶段运行的分数,定义为回合结束时曾经达到的最高分(SI “Score tracking in the exploration phase”)。对 11 款重点游戏,探索阶段分数在 50 次探索阶段运行上取平均;对其余游戏,在 5 次运行上取平均。对带领域知识的情形,在 100 次运行上取平均。
On Atari, only the 11 focus games are robustified and evaluated in a stochastic setting. Modern RL algorithms are already able to adequately solve the games not included in the 11 focus games in this work, as demonstrated by previous work (Extended Data Table 3). Thus, because robustifying these already solved games would have been prohibitively expensive, we did not perform robustification experiments for these 44 games.
在 Atari 上,只有这 11 款重点游戏在随机设置中做了稳健化并加以评估。现代强化学习算法已经能够充分解决未被纳入这 11 款重点游戏的那些游戏,先前工作也表明了这一点(Extended Data Table 3)。因此,再对这些已经解决的游戏做稳健化代价过高,我们没有对这 44 款游戏进行稳健化实验。
During robustification, a checkpoint is produced every 100 training iterations (13,926,400 frames). A subset of checkpoints corresponding to points during which the rolling average of scores seen during training was at its highest are tested by averaging their scores across 100 test episodes. Then the highest scoring checkpoint found is retested with 1,000 new test episodes to eliminate selection bias. For the downscaled representation, robustification scores are averaged over 5 runs. For domain knowledge, they are averaged across 10 runs. All testing is performed with sticky actions (see “State of the art on Atari”). To accurately compare against the human world record of 1.2 million33 , we patched an ALE bug that prevents the score from exceeding 1 million (SI “ALE issues”).
稳健化过程中,每 100 次训练迭代(13,926,400 帧)保存一个检查点。从训练过程中滚动平均分数最高的那些时段里选出一部分检查点,各用 100 个测试回合的平均分来评估。然后再用 1,000 个新的测试回合,对其中得分最高的检查点重新测试,以消除选择偏差。对缩小表示,稳健化分数在 5 次运行上取平均;对领域知识,在 10 次运行上取平均。所有测试都使用黏滞动作(见 “State of the art on Atari”)。为了与 120 万的人类世界纪录33做准确比较,我们修补了一个会阻止分数超过 100 万的 ALE 缺陷(SI “ALE issues”)。
The exploration phase for robotics was evaluated across 50 runs per target shelf, for a total of 200 runs. The reported metric is the proportion of runs that discovered a successful trajectory. Because the outcome of a robotics episode is binary (success or failure), there is no reason to continue robustification once the agent is reliably successful (unlike with Atari where it is usually possible to further improve the score). Thus, robustification runs for robotics are terminated once they keep a success rate greater than 98.5% for over 150 training iterations (19,660,800 frames), and the runs are then considered successful. To ensure that the agent learns to keep the object inside of the shelf, a penalty of -1 is given for taking the object outside of the shelf, and during robustification the agent is given up to 54 additional steps after successfully putting the object in the shelf (see “Extra frame coef” in Extended Data Table 1a), forcing it to ensure the object doesn’t leave the shelf. Out of 200 runs (50 per target shelf), 2 runs did not succeed after running for over 3 billion frames (whereas all other runs succeeded in fewer than 2 billion) and were thus considered unsuccessful (one for the bottom left shelf and the other for the bottom right shelf), resulting in a 99% overall success rate.
机器人任务的探索阶段按每个目标架子 50 次运行来评估,共 200 次。报告的指标是发现成功轨迹的运行比例。由于机器人回合的结果是二元的(成功或失败),一旦智能体已经能够稳定成功,就没有理由再继续稳健化(这与 Atari 不同,那里通常还能继续提高分数)。因此,机器人的稳健化运行在成功率连续超过 150 次训练迭代(19,660,800 帧)保持在 98.5% 以上时就会终止,并被视为成功。为了让智能体学会把物体留在架子里,若把物体拿出架子会给予 -1 的惩罚;稳健化时,智能体在成功把物体放进架子后最多再获得 54 步(见 Extended Data Table 1a 中的 “Extra frame coef”),从而迫使它确保物体不会离开架子。在 200 次运行(每个目标架子 50 次)中,有 2 次在运行超过 30 亿帧后仍未成功(而其余运行都在 20 亿帧以内成功),因而被视为失败(一次对应左下架子,一次对应右下架子),总体成功率为 99%。
The robotics results are compared to two controls. First, to confirm the hard-exploration nature of the environment, 5 runs per target shelf of ordinary PPO20 with no exploration mechanism were run for 1 billion frames. At no point during these runs were any rewards found, confirming that the robotics problem in this paper constitutes a hard-exploration challenge. Secondly, we ran 10 runs per target shelf for 2 billion frames of ordinary PPO augmented with count-based intrinsic rewards, one of the best modern versions of intrinsic motivation1,53,63,64 designed to deal with hard-exploration challenges. The representation for this control is identical to the one used in the exploration phase, so as to provide a fair comparison. Similar to the exploration phase, the counts for each cell are incremented each time the agent enters a cell for the first time in an episode, and the intrinsic reward is given by , similar to . Because it is possible (though rare) for the agent to place the object out of reach, a per-episode time limit is necessary to ensure that not too many training frames are wasted on such unrecoverable states. In robustification, the time limit is implicitly given by the length of the demonstration combined with the additional time described above and in Extended Data Table 1a. For the controls, a limit of 300 time steps was given as it provides ample time to solve the environment (Extended Data Fig. 5b), while ensuring that the object is almost always in range of the robot arm throughout training. As shown in Fig. 4b, this control was unable to find anywhere near the number of cells found by the exploration phase, despite of running for significantly longer, and as shown in Fig. 4c, it also was unable to find any rewards in spite of running for longer than any successful Go-Explore run (counting both the exploration phase and robustification phase combined).
机器人结果与两种对照进行了比较。第一,为确认该环境的难探索性质,对每个目标架子各做了 5 次普通 PPO20、且不带任何探索机制的运行,每次 10 亿帧。这些运行过程中始终没有获得任何奖励,从而确认本文中的机器人问题构成一项难探索挑战。第二,我们对每个目标架子各做了 10 次、每次 20 亿帧的运行,在普通 PPO 上加入基于计数的内在奖励;这是为应对难探索挑战而设计的、当代最好的内在动机方法之一1,53,63,64。该对照所用的表示与探索阶段相同,以便公平比较。与探索阶段类似,每当智能体在某一回合中首次进入某个格子时,该格子的计数加一,内在奖励为 ,这与 类似。由于智能体有可能(尽管很少见)把物体放到够不着的地方,需要设置每回合时间上限,以免在这类无法恢复的状态上浪费过多训练帧。在稳健化中,时间上限由演示长度再加上文及 Extended Data Table 1a 中所述的额外时间隐含给出。对对照而言,上限设为 300 个时间步,因为这已足够完成该环境(Extended Data Fig. 5b),同时又保证训练过程中物体几乎总是处在机械臂可及范围内。如图 4b 所示,尽管运行时间明显更长,该对照发现的格子数量仍远不及探索阶段;如图 4c 所示,即便运行时间超过任何一次成功的 Go-Explore 运行(把探索阶段和稳健化阶段合计在内),它也未能获得任何奖励。
Hyperparameters. Hyperparameters are reported in Extended Data Table 1. Extended Data Table 1b reports the hyperparameters specific to the Atari environment. Of note are the use of sticky actions as recommended by Machado et al. (2018)14, and the fact that the agent acts every 4 frames, as is typical in RL for Atari15. In this work, sample complexity is always reported in terms of raw Atari frames, so that the number of actions can be obtained by dividing by 4. In robotics, the agent acts 12.5 times per second. Each action is simulated with a timestep granularity of 0.001 seconds, corresponding to 80 simulator steps for every action taken.
超参数。 超参数见 Extended Data Table 1。Extended Data Table 1b 报告了 Atari 环境特有的超参数。值得注意的是,按 Machado 等人(2018)14 的建议使用了黏滞动作,以及智能体每隔 4 帧才行动一次,这在面向 Atari 的强化学习中很常见15。在本工作中,样本复杂度一律按原始 Atari 帧数报告,因此动作次数可以用该帧数除以 4 得到。在机器人任务中,智能体每秒行动 12.5 次。每个动作以 0.001 秒的时间步粒度进行仿真,相当于每采取一个动作对应 80 个模拟器步。
While the robustification algorithm originates from Salimans & Chen (2018)29, it was modified in various ways (Methods “The backward algorithm”). Extended Data Table 1a shows the hyperparameters for this algorithm used in this work, to the extent that they are different from those in the original paper, or were added due to the modifications in this work. Extended Data Tables 2a and 2b show the state representation for robotics robustification.
稳健化算法虽然源自 Salimans 与 Chen(2018)29,但在多个方面做了修改(Methods “The backward algorithm”)。Extended Data Table 1a 给出了本工作中该算法所用的超参数,范围限于与原论文不同、或因本文修改而新增的那些。Extended Data Tables 2a 和 2b 给出了机器人稳健化所用的状态表示。
With the downscaled representation on Atari, the exploration phase was run for 2 billion frames prior to extracting demonstrations for robustification. Because exploration phase performance was slightly below human on Pitfall, Skiing and Private Eye, the exploration phase was allowed to run longer on these three games (5 billion for Pitfall and Skiing, 15 billion for Private Eye) to demonstrate that it can exceed human performance on all Atari games. The demonstrations used to robustify these three games were still extracted after 2 billion frames, and the robustified policies still exceeded average human performance thanks to the ability of robustification to improve upon demonstration performance. With the domain knowledge representation on Atari, the exploration phase ran for 1 billion frames. Robustification ran for 10 billion frames on all Atari games except Solaris (20 billion) and Pitfall when using domain knowledge demonstrations (15 billion). On robotics, the exploration phase ran for 20 million frames and details for the robustification phase are given in the Evaluation section.
在 Atari 的缩小表示下,探索阶段在提取用于稳健化的演示之前先运行 20 亿帧。由于探索阶段在 Pitfall、Skiing 和 Private Eye 上的表现略低于人类水平,这三款游戏被允许运行更长时间(Pitfall 和 Skiing 为 50 亿帧,Private Eye 为 150 亿帧),以表明探索阶段能够在全部 Atari 游戏上超过人类水平。用于稳健化这三款游戏的演示仍在 20 亿帧时提取,而稳健化后的策略仍然超过了平均人类水平,这得益于稳健化能够在演示表现之上再作改进。在 Atari 的领域知识表示下,探索阶段运行了 10 亿帧。稳健化在全部 Atari 游戏上运行 100 亿帧,但 Solaris 为 200 亿帧,以及使用领域知识演示时的 Pitfall 为 150 亿帧。在机器人任务中,探索阶段运行 2,000 万帧,稳健化阶段的细节见评估一节。
Policy-based Go-Explore. The idea in policy-based Go-Explore is to learn how to return (rather than to restore archived simulator states to return). The algorithm builds off the popular PPO algorithm20 (SI “PPO and SIL”) and pseudo-code for the algorithm is shown in SI Algorithm 3. At the heart of policy-based Go-Explore lies a goal-conditioned policy (Extended Data Fig. 1c), parameterised by , that takes a state and a goal and defines a probability distribution over actions . Policy-based Go-Explore includes all PPO loss functions described in SI “PPO and SIL”, except that instances of the state are replaced with the state-goal tuple . The total reward is the sum of the trajectory reward (defined below) and the environment reward , where is clipped to the range. Because most rewards in Atari have an absolute value greater than 2, this clip range effectively sets the magnitude of in-game rewards to 2. Given that trajectory rewards are 1 (see below), this clipping implements the intuition that in-game rewards should be more important than following the trajectory. We implement this intuition in the form of clipping so as to not increase the importance of the smallest Atari rewards. Policy-based Go-Explore also includes self-imitation learning (SIL)37 (SI “PPO and SIL”), where SIL actors follow the same procedure as regular actors, except that they replay the trajectory associated with the cell they select from the archive. Hyperparameters are listed in Extended Data Table 1a.
基于策略的 Go-Explore。 基于策略的 Go-Explore 的想法是学会如何回去(而不是通过恢复档案中的模拟器状态来回去)。该算法建立在广泛使用的 PPO 算法20之上(SI “PPO and SIL”),伪代码见 SI Algorithm 3。基于策略的 Go-Explore 的核心是一个由参数 刻画的目标条件策略 (Extended Data Fig. 1c),它接收状态 和目标 ,并给出动作 上的概率分布。基于策略的 Go-Explore 包含 SI “PPO and SIL” 中描述的全部 PPO 损失函数,只不过其中出现的状态 被替换成状态-目标元组 。总奖励 是轨迹奖励 (定义见下文)与环境奖励 之和,其中 被裁剪到 范围。由于 Atari 中大多数奖励的绝对值都大于 2,这一裁剪范围实际上是把游戏内奖励的量级设为 2。鉴于轨迹奖励为 1(见下文),这种裁剪体现的直觉是:游戏内奖励应当比跟随轨迹更重要。我们用裁剪而不是放大的方式来实现这一直觉,是为了不提高那些最小的 Atari 奖励的重要性。基于策略的 Go-Explore 也包括自我模仿学习(SIL)37(SI “PPO and SIL”):SIL 行动器遵循与普通行动器相同的流程,只不过它们回放自己从档案中选出的那个格子所对应的轨迹。超参数见 Extended Data Table 1a。
To fit the batch-oriented paradigm, policy-based Go-Explore updates its archive after every mini-batch (Extended Data Fig. 6). In addition, the “go” step now involves executing actions in the environment (as explained below), and each actor independently tracks whether it is in the “go” step or the “explore” step of the algorithm. For the purpose of updating the archive, no distinction is made between data gathered during the “go” step and data gathered during the “explore” step, meaning policy-based Go-Explore can discover new cells or update existing cells while returning.
为了适配面向批量的范式,基于策略的 Go-Explore 在每个小批量之后更新档案(Extended Data Fig. 6)。此外,“回去”步骤现在需要在环境中执行动作(说明见下),并且每个行动器独立跟踪自己当前处于算法的“回去”步骤还是“探索”步骤。就更新档案而言,在“回去”步骤收集的数据和在“探索”步骤收集的数据不加区分,这意味着基于策略的 Go-Explore 可以在回去的过程中发现新格子,或更新已有格子。
For the experiments presented in this paper, data is gathered in episodes. Whenever an actor starts a new episode, it selects a state from the archive with a cell selection weight of:
就本文报告的实验而言,数据按回合收集。每当一个行动器开始新的回合,它按如下格子选择权重从档案中选出一个状态:
where is the total number of steps the agent spend in the cell. This equation is different from the one in the exploration-phase without a policy ( grows much faster than ) because policy-based Go-Explore benefits from focusing more strongly on the most recently discovered cells for two reasons: (1) after a new cell is discovered in policy-based Go-Explore the policy may first need to learn how to return there reliably; focusing on new cells helps the agent collect the necessary experience to do so, and (2) policy-based Go-Explore will visit many cells along the way to a target cell, allowing it to explore from those intermediate cells without selecting them explicitly (Extended Data Fig. 7). After a cell is selected, policy-based Go-Explore runs its goal-conditioned policy to reach the selected state, which enables it to be applied without assuming access to a deterministic or restorable environment during the exploration phase. It is exceedingly difficult and practically unnecessary to reach a particular state exactly, so instead, the policy is conditioned to reach the cell associated with this state, referred to as the goal cell, provided to the policy in the form of a concatenated one-hot encoding for every attribute characterizing the cell. Directly providing the goal cell to the goal-conditioned policy did not perform well (SI “Ablations”), presumably because goal-conditioned policies tend to falter when goals become distant36. Instead, the actor is iteratively conditioned on the successive cells traversed by the archived trajectory that leads to the goal cell.
其中 是智能体在该格子中度过的总步数。这一公式与不带策略的探索阶段所用公式不同( 比 增长快得多),因为基于策略的 Go-Explore 从更集中地关注最近发现的格子中获益,原因有二:(1)在基于策略的 Go-Explore 中发现新格子之后,策略可能首先需要学会如何可靠地回到那里;关注新格子有助于智能体收集为此所需的经验;(2)基于策略的 Go-Explore 在通往目标格子的路上会经过许多格子,从而可以在这些中间格子上探索,而不必显式选中它们(Extended Data Fig. 7)。选出格子之后,基于策略的 Go-Explore 运行其目标条件策略去到达所选状态,因此在探索阶段不必假定环境是确定性的、或可以恢复。要精确到达某一个特定状态极为困难,实际上也没有必要,因此策略被条件化为到达与该状态对应的格子,称为目标格子,并以对该格子每一个属性做拼接独热编码的形式提供给策略。把目标格子直接提供给目标条件策略效果不好(SI “Ablations”),大概是因为目标一旦变远,目标条件策略往往就会失效36。取而代之的是,行动器被迭代地条件化到通往目标格子的那条已归档轨迹所经过的一连串格子上。
Here, we allow the agent to follow the archived trajectory in a soft-order, a method similar to the one described in Guo et al. (2019)34. To prevent the soft-trajectory from being affected by the time an agent spends in a cell, the algorithm first constructs a trajectory of non-repeated cells, collapsing any consecutive sequence of identical cells into a single cell. Then, given a window size , if the agent is supposed to reach a specific goal cell in this trajectory and it reaches that or any of the subsequent 9 cells in this trajectory, the goal is considered met. When a goal is met, the agent receives a trajectory reward of 1 and the subsequent goal in the non-repeated trajectory (i.e. the goal that comes after the cell that was actually reached) is set as the next goal. When the cell that was reached occurs multiple times in the window (indicating cycles) the next goal is the one that follows the last occurrence of this repeated goal cell.
在这里,我们允许智能体以软顺序跟随已归档的轨迹,这种方法与 Guo 等人(2019)34 所描述的方法类似。为了避免软轨迹受到智能体在某个格子中停留时间的影响,算法先构造一条不含重复的格子轨迹,把任何连续相同的格子序列折叠成单个格子。随后,给定窗口大小 ,如果智能体本应到达这条轨迹中的某个特定目标格子,而它到达了该格子或其后 9 个格子中的任何一个,则视为目标已达成。目标达成时,智能体获得数值为 1 的轨迹奖励 ,并且把无重复轨迹中的下一个目标(也就是实际到达的那个格子之后的目标)设为下一目标。若实际到达的格子在窗口中出现多次(表明存在循环),则下一目标是该重复目标格子最后一次出现之后的那个目标。
When an agent reaches the last cell in the trajectory, it receives a trajectory reward of 3, which is higher than the intermediate trajectory reward of 1 to implement the general practice of having a higher reward for reaching a desired final state than for completing any intermediate objectives65,66: this practice improved performance (SI “Ablations”). Then the agent executes the “explore” step, either through policy exploration or random exploration. With policy exploration, the agent will select a goal for the policy according to one of three rules: (1) with 10% probability, randomly select an adjacent cell (see “Exploration phase”) not in the archive, (2) with 22.5% probability, select any adjacent cell, whether already in the archive or not, and (3), in the remaining 67.5% of cases, select a cell from the archive according to the standard cell-selection weights. If the first rule does not apply because all adjacent cells are already in the archive, rules 2 and 3 are selected with proportionally scaled probabilities. Note that, in the exploration step, the agent is presented directly with the goal, rather than with a trajectory. Whenever the current exploration goal is reached, or if the goal is not reached for some number of steps (here 100), a new exploration goal is chosen. With random exploration, the agent takes random actions according to the random exploration procedure described in Methods “Exploration phase”. All gathered data is ignored with respect to calculating the loss of the policy.
当智能体到达轨迹中的最后一个格子时,它获得数值为 3 的轨迹奖励 ,高于中间轨迹奖励 1,这是为了落实一种通行做法:到达期望的最终状态,应比完成任何中间目标获得更高的奖励65,66;这一做法改进了表现(SI “Ablations”)。随后智能体执行“探索”步骤,方式可以是策略探索或随机探索。在策略探索中,智能体按以下三条规则之一为策略选择目标:(1)以 10% 的概率,随机选择一个不在档案中的相邻格子(见 “Exploration phase”);(2)以 22.5% 的概率,选择任意相邻格子,不论它是否已在档案中;(3)在其余 67.5% 的情形中,按标准的格子选择权重从档案中选一个格子。若由于所有相邻格子都已在档案中而使第一条规则不适用,则第 2、第 3 条规则按比例放大后的概率被选中。需要注意的是,在探索步骤中,直接把目标呈现给智能体,而不是呈现一条轨迹。一旦当前探索目标被到达,或在若干步内(这里是 100 步)仍未到达该目标,就另选一个新的探索目标。在随机探索中,智能体按照 Methods “Exploration phase” 中描述的随机探索程序采取随机动作。所收集的全部数据在计算策略损失时都被忽略。
While following a trajectory or during exploration, it is possible for the agent to fail to make progress towards the current goal cell because the policy has converged towards putting all its probability mass on a small set of actions, meaning the policy performs insufficient exploration to discover the goal and observe its reward. To alleviate this issue, in addition to having the entropy bonus , the policy is extended with an entropy term that divides the logits of the policy right before the softmax activation function is applied. If the agent fails to reach the current goal for some number of steps (defined below), this entropy term is increased following:
在跟随轨迹或进行探索时,智能体有可能无法朝当前目标格子取得进展,原因是策略已经收敛到把全部概率质量放在一小撮动作上,从而探索不足,发现不了目标、也观察不到其奖励。为缓解这一问题,除了熵奖励 之外,还为策略增加一个熵项 ,在应用 softmax 激活函数之前,用它去除以策略的对数几率。如果智能体在若干步 (定义见下)内仍未到达当前目标,该熵项按如下方式增大:
where is the number of steps the agent has taken since it last reached a goal (for returning) or discovered a new cell (for exploring), is the entropy increase factor and is the entropy increase power. While executing the “explore” step, the threshold has a fixed value of 50. While returning, the threshold equals the number of actions that the followed trajectory required to move from the previously reached goal cell to the current goal cell. Here, the previously reached goal cell refers to the first cell in the soft-trajectory window that matched the cell occupied by the agent at the time the previous goal was considered met.
其中 是自智能体上次到达一个目标(回去时)或发现一个新格子(探索时)以来已走的步数, 是熵增大因子, 是熵增大幂次。在执行“探索”步骤时,阈值 取固定值 50。在回去时,阈值 等于所跟随的轨迹从先前到达的目标格子走到当前目标格子所需的动作数。这里,先前到达的目标格子是指:在上一目标被视为达成时,软轨迹窗口中与智能体当时所在格子相匹配的第一个格子。
Lastly, to prevent actors from spending many time steps without making any progress (possibly because the agent reached a state from which further progress is impossible), we terminate the episode early if the current goal is not reached within 1,000 steps after we have started to increase entropy (while returning), or if no new cells are discovered for 1,000 steps (while exploring). For Montezuma’s Revenge with policy-based Go-Explore only, we also terminate the episode on death to deal with an ALE bug (details in Supplemental Information).
最后,为了防止行动器在许多时间步里毫无进展(有可能是因为智能体到达了一个无法再前进的状态),若在开始增大熵之后 1,000 步内仍未到达当前目标(回去时),或连续 1,000 步都没有发现新格子(探索时),我们就提前结束该回合。仅对使用基于策略的 Go-Explore 的蒙特祖玛的复仇,我们还会在智能体死亡时结束回合,以应对一个 ALE 缺陷(细节见 Supplemental Information)。
Robotics environment. The environment, from https://github.com/vikashplus/fetch, features a realistic model67 of the Fetch Mobile Manipulator35 and was minimally modified to implement a sparse-reward pick-and-place task. The modified environment is included with the Go-Explore code.
机器人环境。 该环境来自 https://github.com/vikashplus/fetch,包含 Fetch 移动操作器35 的一个逼真模型67,并做了尽可能少的修改,以实现一项稀疏奖励的抓取放置任务。修改后的环境包含在 Go-Explore 的代码中。
References
参考文献
- Castro, P. S., Moitra, S., Gelada, C., Kumar, S. & Bellemare, M. G. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110 (2018).
- Toromanoff, M., Wirbel, E. & Moutarde, F. Is Deep Reinforcement Learning Really Superhuman on Atari? arXiv preprint arXiv:1908.04683 (2019).
- Burda, Y., Edwards, H., Storkey, A. & Klimov, O. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 (2018).
- Choi, J. et al. Contingency-Aware Exploration in Reinforcement Learning. CoRR abs/1811.01483 (2018).
- Fedus, W., Gelada, C., Bengio, Y., Bellemare, M. G. & Larochelle, H. Hyperbolic discounting and learning over multiple horizons. arXiv preprint arXiv:1902.06865 (2019).
- Taiga, A. A., Fedus, W., Machado, M. C., Courville, A. & Bellemare, M. G. On Bonus Based Exploration Methods In The Arcade Learning Environment in International Conference on Learning Representations (2020).
- Tang, Y., Valko, M. & Munos, R. Taylor expansion policy optimization. arXiv preprint arXiv:2003.06259 (2020).
- Ostrovski, G., Bellemare, M. G., van den Oord, A. & Munos, R. Count-Based Exploration with Neural Density Models in ICML (2017).
- Martin, J., Sasikumar, S. N., Everitt, T. & Hutter, M. Count-Based Exploration in Feature Space for Reinforcement Learning in IJCAI (2017).
- O’Donoghue, B., Osband, I., Munos, R. & Mnih, V. The Uncertainty Bellman Equation and Exploration in ICML (2018).
- Goldenberg, A., Benhabib, B. & Fenton, R. A complete generalized solution to the inverse kinematics of robots. IEEE Journal on Robotics and Automation 1, 14–20 (1985).
- Spong, M. W., Hutchinson, S., Vidyasagar, M., et al. Robot modeling and control (2006).
- Zhao, Z.-Q., Zheng, P., Xu, S.-t. & Wu, X. Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems 30, 3212–3232 (2019).
- Todorov, E., Erez, T. & Tassa, Y. MuJoCo: A physics engine for model-based control in IROS (2012), 5026–5033.
- Kocsis, L. & Szepesvári, C. Bandit Based Monte-Carlo Planning in ECML (2006).
- Strehl, A. L. & Littman, M. L. An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences 74, 1309–1331 (2008).
- Tang, H. et al. # Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning in NIPS (2017), 2750–2759.
- Ng, A. Y., Harada, D. & Russell, S. Policy invariance under reward transformations: Theory and application to reward shaping in ICML 99 (1999), 278–287.
- Hussein, A., Gaber, M. M., Elyan, E. & Jayne, C. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR) 50, 1–35 (2017).
- Plappert, M. et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464 (2018).
Data availability. The data that support the findings of this study (including the raw data for all figures and tables in the manuscript, extended data and SI, as well as the demonstration trajectories used in robustification) are available from the corresponding authors upon reasonable request.
数据可用性。 支持本研究结论的数据(包括稿件、扩展数据与 SI 中全部图和表的原始数据,以及稳健化所用的演示轨迹)可应合理请求向通讯作者索取。
Code availability. The Go-Explore code is available at: https://github.com/uber-research/go-explore.
代码可用性。 Go-Explore 的代码发布于:https://github.com/uber-research/go-explore。
Extended Data Figure 1: Neural network architectures. (a) The Atari architecture is based on the architecture provided with the backward algorithm implementation. The input consists of the RGB channels of the last four frames (re-scaled to 80 by 105 pixels) concatenated, resulting in 12 input channels. The network consists of 3 convolutional layers, 2 fully connected layers, and a layer of Gated Recurrent Units (GRUs)68. The network has a policy head and a value head . (b) For the robotics problem, the architecture consists of two separate networks, each with 2 fully connected layers and a GRU layer. One network specifies the policy by returning a mean and variance for the actuator torques of the arm and the desired position of each of the two fingers of the gripper (gripper fingers are implemented as Mujoco position actuators61 with and a control range of ). The other network implements the value function . (c) The architecture for policy-based Go-Explore is identical to the Atari architecture, except that the goal representation is concatenated with the input of the first fully connected layer.
扩展数据图 1:神经网络结构。(a) Atari 结构基于反向算法实现所提供的结构。输入由最近四帧的 RGB 通道(重新缩放到 80×105 像素)拼接而成,共 12 个输入通道。网络包含 3 个卷积层、2 个全连接层,以及一层门控循环单元(GRU)68。网络有一个策略头 和一个价值头 。(b) 对机器人问题,结构由两个独立网络组成,每个网络有 2 个全连接层和一层 GRU。一个网络给出策略 ,为机械臂的执行器力矩以及夹爪两根手指各自的目标位置返回均值 和方差 (夹爪手指实现为 Mujoco 位置执行器61,,控制范围为 )。另一个网络实现价值函数 。(c) 基于策略的 Go-Explore 的结构与 Atari 结构相同,只是把目标表示 拼接到第一个全连接层的输入上。
Extended Data Figure 2: Maximum end-of-episode score found by the exploration phase on Atari. Because only scores achieved at the episode end are reported, the plots for some games (e.g. Solaris) begin after the start of the run, when the episode end is first reached. In (a), averaging is over 50 runs for the 11 focus games and 5 runs for other games. In (b), averaging is over 100 runs. Shaded areas show 95% bootstrap CIs of the mean with 1,000 samples.
扩展数据图 2:探索阶段在 Atari 上发现的回合结束最高分。由于只报告回合结束时取得的分数,一些游戏(例如 Solaris)的曲线会在运行开始之后、首次到达回合结束时才出现。在 (a) 中,11 款重点游戏对 50 次运行取平均,其余游戏对 5 次运行取平均。在 (b) 中,对 100 次运行取平均。阴影区域表示对均值做 1,000 次抽样得到的 95% 自助法置信区间。
Extended Data Figure 3: Number of cells in archive during the exploration phase on Atari. In (a), archive size can decrease when the representation is recomputed. Previous archives are converted to the new format when the representation is recomputed, possibly leading to an archive larger than 50K. In this case, one iteration of the exploration phase runs and the representation is recomputed again. Shaded areas show 95% bootstrap CIs of the mean with 1,000 samples.
扩展数据图 3:Atari 探索阶段中档案里的格子数量。在 (a) 中,当表示被重新计算时,档案规模可能会下降。表示重新计算时,先前的档案会被转换成新的格式,有可能使档案大于 50K。此时会再运行一轮探索阶段,并再次重新计算表示。阴影区域表示对均值做 1,000 次抽样得到的 95% 自助法置信区间。
Extended Data Figure 4: Progress of robustification phase on Atari. Shown are the scores achieved by robustifying agents across training time for the exploration phase (a) with representations informed by domain knowledge, and (b) representations without domain knowledge. In particular, the rolling mean is shown of performance across the past 100 episodes when starting from the virtual demonstration (which corresponds to the domain’s traditional starting state). Note that in (a) averaging is over 5 independent runs, while in (b) averaging is over 10 runs. Because the final performance is obtained by testing the highest-performing network checkpoint for each run over 1,000 additional episodes, rather than directly extracted from the curves above, the performance reported in Fig. 2b does not necessarily match any particular point along these curves (Methods). Shaded areas show 95% bootstrap CIs of the mean with 1,000 samples.
扩展数据图 4:Atari 上稳健化阶段的进展。图中给出了稳健化智能体在训练过程中取得的分数,分别对应探索阶段采用 (a) 带领域知识的表示,以及 (b) 不带领域知识的表示。具体画出的是,从虚拟演示出发(对应于该领域传统的起始状态)时,过去 100 个回合表现的滚动平均。需要注意,(a) 是对 5 次独立运行取平均,而 (b) 是对 10 次运行取平均。由于最终表现是通过对每次运行中表现最好的网络检查点再测 1,000 个额外回合得到的,而不是直接从上述曲线中读取,图 2b 中报告的表现并不必然与这些曲线上的某一点吻合(Methods)。阴影区域表示对均值做 1,000 次抽样得到的 95% 自助法置信区间。
Extended Data Figure 5: Progress of the exploration phase in the robotics environment. In (a), the exploration phase quickly achieves 100% success rate for all shelves in the robotics environment. However, (b) shows that while success is achieved quickly, it is useful to keep the exploration phase running longer to reduce the length of the successful trajectories, thus making robustification easier. Shaded areas show 95% bootstrap CIs of the mean with 1,000 samples.
扩展数据图 5:机器人环境中探索阶段的进展。在 (a) 中,探索阶段很快就在该机器人环境的全部架子上达到 100% 成功率。不过 (b) 表明,尽管成功来得很快,让探索阶段继续运行更长时间仍然有用,因为这样可以缩短成功轨迹的长度,从而使稳健化更容易。阴影区域表示对均值做 1,000 次抽样得到的 95% 自助法置信区间。
Extended Data Figure 6: Policy-based Go-Explore overview. With respect to their practical implementation, the main difference between policy-based Go-Explore and Go-Explore when restoring simulator state is that in policy-based Go-Explore there exist separate actors that each have an internal loop switching between the “select”, “go”, and “explore” steps, rather than one outer loop in which the “select”, “go”, and “explore” steps are executed in synchronised batches. This structure allows policy-based Go-Explore to be easily combined with popular RL algorithms like A3C19, PPO20 or DQN15, which already divide data gathering over many actors.
扩展数据图 6:基于策略的 Go-Explore 概览。就实际实现而言,基于策略的 Go-Explore 与通过恢复模拟器状态来回去的 Go-Explore 的主要差别在于:前者有多个独立的行动器,每个行动器内部都有一个在“选择”、“回去”和“探索”步骤之间切换的循环,而不是在一个外层循环里按同步批量执行“选择”、“回去”和“探索”。这种结构使基于策略的 Go-Explore 易于与 A3C19、PPO20 或 DQN15 等已经把数据收集分配到许多行动器上的流行强化学习算法相结合。
Extended Data Figure 7: Method by which cells are found. In both (a) Montezuma’s Revenge and (b) Pitfall, sampling from the goal-conditioned policy results in the discovery of roughly four times more cells than when taking random actions. At the start of training there is effectively no difference between random actions and sampling from the policy, supporting the intuition that sampling from the policy only becomes more efficient than random actions after the policy has acquired the basic skills for moving towards the indicated goal. Lastly, the number of cells that are discovered while returning is about twice that of the cells discovered when taking random actions after returning, indicating that the frames spent while returning to a previously visited cell are not just overhead required for moving towards the frontier of yet undiscovered states and training the policy network, but actually provide a substantial contribution towards exploration as well. Shaded areas show 95% bootstrap CIs of the mean with 1,000 samples.
扩展数据图 7:格子被发现的方式。在 (a) 蒙特祖玛的复仇和 (b) Pitfall 中,从目标条件策略采样所发现的格子,大约是采取随机动作时的四倍。训练开始时,随机动作与从策略采样实际上没有差别,这支持如下直觉:只有在策略已经掌握朝所指目标移动的基本技能之后,从策略采样才会比随机动作更有效。最后,在回去过程中发现的格子数量,大约是回去之后再采取随机动作所发现格子的两倍,说明花在回到先前访问过的格子上的那些帧,并不仅仅是走向尚未发现状态的前沿、以及训练策略网络所需的开销,它们对探索本身也有可观贡献。阴影区域表示对均值做 1,000 次抽样得到的 95% 自助法置信区间。
Extended Data Table 1: Hyperparameters.
(a) Parameters for PPO, SIL, and the backward algorithm.
| Parameter | Robustification Atari | Robustification Robotics | Policy-Based Atari |
|---|---|---|---|
| Discount factor () | 0.999 | 0.99 | 0.99 |
| N-step return factor () | 0.95 | 0.95 | 0.95 |
| Num. workers | 8 | 8 | 16 |
| Num. actors per worker | 32 + 2 SIL | 120 + 8 SIL | 15 + 1 SIL |
| Num. actors () | 256 + 16 SIL | 960 + 64 SIL | 240 + 16 SIL |
| Steps per batch () | 128 | 128 | 128 |
| PPO Clip () | 0.1 | 0.1 | 0.1 |
| PPO Epochs | 4 | 4 | 4 |
| Value coef. () | 0.5 | 0.5 | 0.5 |
| Ent. coef. () | |||
| L2 coef. () | |||
| SIL coef. () | 0.1 | 0.1 | 0.1 |
| SIL ent. coef. () | 0 | ||
| SIL value coef. () | 0.01 | 0.1 | 0.01 |
| Allowed lag | 50 | 10 | — |
| Extra frame coef | 7 | 4 | — |
| Move threshold | 0.1 | 0.1 | — |
| Num. demonstrations | 10 + 1 virtual | 10 | — |
| SIL from start prob. | 0.3 | 0 | — |
| Window size (frames) | 160 | 40 | — |
(b) Atari environment parameters
| Parameter | Value |
|---|---|
| Sticky actions | True |
| Length limit | 400K frames* |
| End of episode | All lives lost† |
| Action repeat (i.e. frame skip) | 4 |
| Frame max pool | 2 or 4‡ |
(a) Parameters above the dividing line are applicable to PPO with SIL, while parameters below the line are specific to the backward algorithm. “Allowed lag” is the number of frames the agent may lag the demonstration before being considered unsuccessful. When the agent matches the demonstration, it runs for additional frames, controlled by “Extra frame coef” : . Window size is the number of starting points below the maximum starting point of the demonstration that the algorithm may start from. (b) For the exploration phase when restoring simulator state, only “Max episode length”, “End of episode”, and “Action repeat” apply. (*) OpenAI Gym default. (†) Except for Montezuma’s Revenge with a return policy (Methods “Policy-based Go-Explore”). (‡) 4 for Gravitar and Venture.
扩展数据表 1:超参数。
(a) PPO、SIL 与反向算法的参数。
| 参数 | 稳健化 Atari | 稳健化 机器人 | 基于策略 Atari |
|---|---|---|---|
| 折扣因子 () | 0.999 | 0.99 | 0.99 |
| N 步回报因子 () | 0.95 | 0.95 | 0.95 |
| 工作进程数 | 8 | 8 | 16 |
| 每个工作进程的行动器数 | 32 + 2 SIL | 120 + 8 SIL | 15 + 1 SIL |
| 行动器数 () | 256 + 16 SIL | 960 + 64 SIL | 240 + 16 SIL |
| 每批步数 () | 128 | 128 | 128 |
| PPO 裁剪 () | 0.1 | 0.1 | 0.1 |
| PPO 轮数 | 4 | 4 | 4 |
| 价值系数 () | 0.5 | 0.5 | 0.5 |
| 熵系数 () | |||
| L2 系数 () | |||
| SIL 系数 () | 0.1 | 0.1 | 0.1 |
| SIL 熵系数 () | 0 | ||
| SIL 价值系数 () | 0.01 | 0.1 | 0.01 |
| 允许滞后 | 50 | 10 | — |
| 额外帧系数 | 7 | 4 | — |
| 移动阈值 | 0.1 | 0.1 | — |
| 演示条数 | 10 + 1 条虚拟 | 10 | — |
| 从起点做 SIL 的概率 | 0.3 | 0 | — |
| 窗口大小(帧) | 160 | 40 | — |
(b) Atari 环境参数
| 参数 | 取值 |
|---|---|
| 黏滞动作 | True |
| 长度上限 | 400K 帧* |
| 回合结束 | 失去全部生命† |
| 动作重复(即帧跳过) | 4 |
| 帧最大池化 | 2 或 4‡ |
(a) 分界线以上的参数适用于带 SIL 的 PPO,分界线以下则是反向算法所特有的。“允许滞后”是在被视为不成功之前,智能体可以落后演示的帧数。当智能体与演示匹配后,它还会再运行由“额外帧系数” 控制的额外帧:()。窗口大小是算法可以从演示最大起点之下开始的起点数目。(b) 对通过恢复模拟器状态来回去的探索阶段,只适用“最大回合长度”、“回合结束”和“动作重复”。(*) OpenAI Gym 默认值。(†) 使用返回策略的蒙特祖玛的复仇除外(Methods “Policy-based Go-Explore”)。(‡) 对 Gravitar 和 Venture 为 4。
Extended Data Table 2: Robotics state representation.
(a) Position and velocity objects. (b) Collision and bounding box objects.
| (a) Object | (b) Object |
|---|---|
| door1 | DoorLR |
| door | DoorUR |
| elbow_flex_link | Shelf |
| forearm_roll_link | Table |
| gripper_link | door1 |
| head_camera_link | door |
| head_pan_link | frameL1 |
| head_tilt_link | frameL |
| l_gripper_finger_link | frameR1 |
| latch1 | frameR |
| latch | gripper_link |
| obj0 | l_gripper_finger_link |
| r_gripper_finger_link | latch1 |
| shoulder_lift_link | latch |
| shoulder_pan_link | obj0 |
| upperarm_roll_link | r_gripper_finger_link |
| wrist_flex_link | world |
| wrist_roll_link |
Position and velocities of the objects in (a) are included in the state representation for robotics. Collisions between any two objects in (b) as well as whether each object is currently inside the bounding boxes for the table and shelves are also included in the state representation. Objects are given by their MuJoCo61 entity names in the source code for the environment. Door-related objects ending with a 1 correspond to the lower door while door-related objects not ending with anything correspond to the upper door. The frame objects are the unmovable wooden blocks situated on either side of the movable part of the door. “L” and “R” correspond to “left” and “right” whereas “L” and “U” correspond to “lower” and “upper”. The difference between “door” and “DoorUR” as well as “door1” and “DoorLR” is that in each case the latter object corresponds to the entire door structure, including the frames, while the former corresponds only to the movable part of the door. A link to the original source code for the MuJoCo description files defining these entities is given in “Acknowledgements”, and a link to the Go-Explore codebase containing our modified version is provided in Methods “Code availability”.
扩展数据表 2:机器人状态表示。
(a) 位置与速度对象。(b) 碰撞与包围盒对象。
| (a) 对象 | (b) 对象 |
|---|---|
| door1 | DoorLR |
| door | DoorUR |
| elbow_flex_link | Shelf |
| forearm_roll_link | Table |
| gripper_link | door1 |
| head_camera_link | door |
| head_pan_link | frameL1 |
| head_tilt_link | frameL |
| l_gripper_finger_link | frameR1 |
| latch1 | frameR |
| latch | gripper_link |
| obj0 | l_gripper_finger_link |
| r_gripper_finger_link | latch1 |
| shoulder_lift_link | latch |
| shoulder_pan_link | obj0 |
| upperarm_roll_link | r_gripper_finger_link |
| wrist_flex_link | world |
| wrist_roll_link |
(a) 中各对象的位置和速度被纳入机器人的状态表示。(b) 中任意两个对象之间的碰撞,以及每个对象当前是否位于桌子和架子的包围盒内,也被纳入状态表示。对象名称采用环境源代码中的 MuJoCo61 实体名。以 1 结尾的与门相关的对象对应于下门,不以任何数字结尾的与门相关的对象对应于上门。frame 对象是位于门的可活动部分两侧、不可移动的木块。“L”和“R”分别对应“左”和“右”,而“L”和“U”分别对应“下”和“上”。“door”与“DoorUR”、以及“door1”与“DoorLR”的差别在于:在每一对中,后者对应包括门框在内的整个门结构,前者只对应门的可活动部分。定义这些实体的 MuJoCo 描述文件原始源代码链接见 “Acknowledgements”,包含我们修改版本的 Go-Explore 代码库链接见 Methods “Code availability”。
Extended Data Table 3: Full scores on Atari.
| Game | Expl. Phase | Robust. Phase | SOTA | Avg. Human | Agent57 |
|---|---|---|---|---|---|
| Alien | 959,312 | 11,358 | 7,128 | 297,638 | |
| Amidar | 19,083 | 3,092 | 1,720 | 29,660 | |
| Assault | 30,773 | 13,759 | 742 | 67,213 | |
| Asterix | 999,500 | 274,491 | 8,503 | 991,384 | |
| Asteroids | 112,952 | 159,426 | 47,389 | 150,855 | |
| Atlantis | 286,460 | 937,558 | 29,028 | 1,528,842 | |
| BankHeist | 3,668 | 1,563 | 753 | 23,072 | |
| BattleZone | 998,800 | 45,610 | 37,188 | 934,135 | |
| BeamRider | 371,723 | 24,031 | 16,927 | 300,510 | |
| Berzerk | 131,417 | 197,376 | 1,383 | 2,630 | 61,508 |
| Bowling | 247 | 260 | 69 | 161 | 251 |
| Boxing | 91 | 99 | 12 | 100 | |
| Breakout | 774 | 637 | 31 | 790 | |
| Centipede | 613,815 | 1,422,628 | 10,166 | 12,017 | 412,848 |
| ChopperCommand | 996,220 | 19,256 | 7,388 | 999,900 | |
| CrazyClimber | 235,600 | 160,161 | 35,829 | 565,910 | |
| DemonAttack | 239,895 | 133,030 | 1,971 | 143,161 | |
| DoubleDunk | 24 | 23 | -16 | 24 | |
| Enduro | 1,031 | 2,338 | 861 | 2,368 | |
| FishingDerby | 67 | 49 | -39 | 87 | |
| Freeway | 34 | 34 | 34 | 30 | 33 |
| Frostbite | 999,990 | 10,003 | 4,335 | 541,281 | |
| Gopher | 134,244 | 26,123 | 2,413 | 117,777 | |
| Gravitar | 13,385 | 7,588 | 3,906 | 3,351 | 19,214 |
| Hero | 37,783 | 50,142 | 30,826 | 114,736 | |
| IceHockey | 33 | 14 | 1 | 64 | |
| Jamesbond | 200,810 | 4,303 | 303 | 135,785 | |
| Kangaroo | 24,300 | 13,982 | 3,035 | 24,034 | |
| Krull | 63,149 | 9,971 | 2,666 | 251,997 | |
| KungFuMaster | 24,320 | 44,920 | 22,736 | 206,846 | |
| MontezumaRevenge | 24,758 | 43,791 | 11,618 | 4,753 | 9,352 |
| MsPacman | 456,123 | 9,901 | 6,952 | 63,994 | |
| NameThisGame | 212,824 | 18,084 | 8,049 | 54,387 | |
| Phoenix | 19,200 | 148,840 | 7,243 | 908,264 | |
| Pitfall | 7,875 | 6,954 | 0 | 6,464 | 18,756 |
| Pong | 21 | 21 | 15 | 21 | |
| PrivateEye | 69,976 | 95,756 | 26,364 | 69,571 | 79,716 |
| Qbert | 999,975 | 26,172 | 13,455 | 580,328 | |
| Riverraid | 35,588 | 24,116 | 17,118 | 63,319 | |
| RoadRunner | 999,900 | 67,962 | 7,845 | 243,026 | |
| Robotank | 143 | 70 | 12 | 127 | |
| Seaquest | 539,456 | 64,985 | 42,055 | 999,998 | |
| Skiing | -4,185 | -3,660 | -10,386 | -4,337 | -4,203 |
| Solaris | 20,306 | 19,671 | 3,282 | 12,327 | 44,200 |
| SpaceInvaders | 93,147 | 24,183 | 1,669 | 48,681 | |
| StarGunner | 609,580 | 265,480 | 10,250 | 839,574 | |
| Tennis | 24 | 23 | -8 | 24 | |
| TimePilot | 183,620 | 32,813 | 5,229 | 405,425 | |
| Tutankham | 528 | 288 | 168 | 2,355 | |
| UpNDown | 553,718 | 193,520 | 11,693 | 623,806 | |
| Venture | 3,074 | 2,281 | 1,916 | 1,188 | 2,624 |
| VideoPinball | 999,999 | 656,572 | 17,668 | 992,341 | |
| WizardOfWor | 199,900 | 10,980 | 4,757 | 157,306 | |
| YarsRevenge | 999,998 | 93,680 | 54,577 | 998,532 | |
| Zaxxon | 18,340 | 25,603 | 9,173 | 249,809 |
Go-Explore outperforms SOTA on all focus games (Freeway’s score is at its maximum). The exploration phase similarly finds trajectories that frequently exceed SOTA scores. Finally, Go-Explore outperforms Agent57 on 7 of the 11 focus games, despite Go-Explore being evaluated in a harder environment. Agent57 was included because it is the only other algorithm that has achieved superhuman scores on all unsolved and hard-exploration games, but it is listed separately because it was evaluated under easier, mostly deterministic conditions (Methods “State of the art on Atari”).
扩展数据表 3:Atari 上的全部分数。
| Game | Expl. Phase | Robust. Phase | SOTA | Avg. Human | Agent57 |
|---|---|---|---|---|---|
| Alien | 959,312 | 11,358 | 7,128 | 297,638 | |
| Amidar | 19,083 | 3,092 | 1,720 | 29,660 | |
| Assault | 30,773 | 13,759 | 742 | 67,213 | |
| Asterix | 999,500 | 274,491 | 8,503 | 991,384 | |
| Asteroids | 112,952 | 159,426 | 47,389 | 150,855 | |
| Atlantis | 286,460 | 937,558 | 29,028 | 1,528,842 | |
| BankHeist | 3,668 | 1,563 | 753 | 23,072 | |
| BattleZone | 998,800 | 45,610 | 37,188 | 934,135 | |
| BeamRider | 371,723 | 24,031 | 16,927 | 300,510 | |
| Berzerk | 131,417 | 197,376 | 1,383 | 2,630 | 61,508 |
| Bowling | 247 | 260 | 69 | 161 | 251 |
| Boxing | 91 | 99 | 12 | 100 | |
| Breakout | 774 | 637 | 31 | 790 | |
| Centipede | 613,815 | 1,422,628 | 10,166 | 12,017 | 412,848 |
| ChopperCommand | 996,220 | 19,256 | 7,388 | 999,900 | |
| CrazyClimber | 235,600 | 160,161 | 35,829 | 565,910 | |
| DemonAttack | 239,895 | 133,030 | 1,971 | 143,161 | |
| DoubleDunk | 24 | 23 | -16 | 24 | |
| Enduro | 1,031 | 2,338 | 861 | 2,368 | |
| FishingDerby | 67 | 49 | -39 | 87 | |
| Freeway | 34 | 34 | 34 | 30 | 33 |
| Frostbite | 999,990 | 10,003 | 4,335 | 541,281 | |
| Gopher | 134,244 | 26,123 | 2,413 | 117,777 | |
| Gravitar | 13,385 | 7,588 | 3,906 | 3,351 | 19,214 |
| Hero | 37,783 | 50,142 | 30,826 | 114,736 | |
| IceHockey | 33 | 14 | 1 | 64 | |
| Jamesbond | 200,810 | 4,303 | 303 | 135,785 | |
| Kangaroo | 24,300 | 13,982 | 3,035 | 24,034 | |
| Krull | 63,149 | 9,971 | 2,666 | 251,997 | |
| KungFuMaster | 24,320 | 44,920 | 22,736 | 206,846 | |
| MontezumaRevenge | 24,758 | 43,791 | 11,618 | 4,753 | 9,352 |
| MsPacman | 456,123 | 9,901 | 6,952 | 63,994 | |
| NameThisGame | 212,824 | 18,084 | 8,049 | 54,387 | |
| Phoenix | 19,200 | 148,840 | 7,243 | 908,264 | |
| Pitfall | 7,875 | 6,954 | 0 | 6,464 | 18,756 |
| Pong | 21 | 21 | 15 | 21 | |
| PrivateEye | 69,976 | 95,756 | 26,364 | 69,571 | 79,716 |
| Qbert | 999,975 | 26,172 | 13,455 | 580,328 | |
| Riverraid | 35,588 | 24,116 | 17,118 | 63,319 | |
| RoadRunner | 999,900 | 67,962 | 7,845 | 243,026 | |
| Robotank | 143 | 70 | 12 | 127 | |
| Seaquest | 539,456 | 64,985 | 42,055 | 999,998 | |
| Skiing | -4,185 | -3,660 | -10,386 | -4,337 | -4,203 |
| Solaris | 20,306 | 19,671 | 3,282 | 12,327 | 44,200 |
| SpaceInvaders | 93,147 | 24,183 | 1,669 | 48,681 | |
| StarGunner | 609,580 | 265,480 | 10,250 | 839,574 | |
| Tennis | 24 | 23 | -8 | 24 | |
| TimePilot | 183,620 | 32,813 | 5,229 | 405,425 | |
| Tutankham | 528 | 288 | 168 | 2,355 | |
| UpNDown | 553,718 | 193,520 | 11,693 | 623,806 | |
| Venture | 3,074 | 2,281 | 1,916 | 1,188 | 2,624 |
| VideoPinball | 999,999 | 656,572 | 17,668 | 992,341 | |
| WizardOfWor | 199,900 | 10,980 | 4,757 | 157,306 | |
| YarsRevenge | 999,998 | 93,680 | 54,577 | 998,532 | |
| Zaxxon | 18,340 | 25,603 | 9,173 | 249,809 |
Go-Explore 在全部重点游戏上都超过当时最好成绩(Freeway 的分数已达到其上限)。探索阶段同样经常找到超过当时最好分数的轨迹。最后,尽管 Go-Explore 是在更难的环境中评估的,它仍在 11 款重点游戏中的 7 款上超过 Agent57。列入 Agent57 是因为它是唯一另一个在全部未解决和难探索游戏上都达到超人类分数的算法,但将它单独列出,是因为它是在更容易、大多为确定性的条件下评估的(Methods “State of the art on Atari”)。











