CURL: Contrastive Unsupervised Representations for Reinforcement Learning

CURL：面向强化学习的对比无监督表征

Aravind Srinivas ∗ Affiliation:  University of California, Berkeley, BAIR    Michael Laskin ∗ Affiliation:  University of California, Berkeley, BAIR    Pieter Abbeel Affiliation:  University of California, Berkeley, BAIR

Abstract

摘要

We present CURL: Contrastive Unsupervised Representations for Reinforcement Learning. CURL extracts high-level features from raw pixels using contrastive learning and performs off-policy control on top of the extracted features. CURL outperforms prior pixel-based methods, both model-based and model-free, on complex tasks in the DeepMind Control Suite and Atari Games showing 1.9x and 1.2x performance gains at the 100K environment and interaction steps benchmarks respectively. On the DeepMind Control Suite, CURL is the first image-based algorithm to nearly match the sample-efficiency of methods that use state-based features. Our code is open-sourced and available at https://www.github.com/MishaLaskin/curl.

本文提出 CURL：Contrastive Unsupervised Representations for Reinforcement Learning。CURL 用对比学习从原始像素抽取高层特征，并在其上做离策略控制。在 DeepMind Control Suite 与 Atari 的复杂任务上，它优于先前基于像素的方法（含基于模型与无模型）：在 100K 环境步与交互步基准上分别取得约 1.9× 与 1.2× 的性能提升。在 DMControl 上，CURL 是首个几乎追上基于状态特征方法样本效率的图像算法。代码开源：https://www.github.com/MishaLaskin/curl。

1 Introduction

1 引言

Keywords:

Developing agents that can perform complex control tasks from high dimensional observations such as pixels has been possible by combining the expressive power of deep neural networks with the long-term credit assignment power of reinforcement learning algorithms. Notable successes include learning to play a diverse set of video games from raw pixels (Mnih et al. 2015), continuous control tasks such as controlling a simulated car from a dashboard camera (Lillicrap et al. 2015) and subsequent algorithmic developments and applications to agents that successfully navigate mazes and solve complex tasks from first-person camera observations (Jaderberg et al. 2016; Espeholt et al. 2018; Jaderberg et al. 2019); and robots that successfully grasp objects in the real world (Kalashnikov et al. 2018).

关键词：

把深度神经网络的表达力与强化学习的长期信用分配结合起来，已能从像素等高维观测完成复杂控制。代表性成功包括：从原始像素玩多样电子游戏（Mnih et al. 2015）、从仪表盘相机做仿真车连续控制（Lillicrap et al. 2015），以及后续在第一人称相机观测下导航迷宫、求解复杂任务的算法与应用（Jaderberg et al. 2016；Espeholt et al. 2018；Jaderberg et al. 2019）；还有在真实世界成功抓取物体的机器人（Kalashnikov et al. 2018）。

Figure 1: Contrastive Unsupervised Representations for Reinforcement Learning (CURL) combines instance contrastive learning and reinforcement learning. CURL trains a visual representation encoder by ensuring that the embeddings of data-augmented versions $o_{q}$ and $o_{k}$ of observation $o$ match using a contrastive loss. The query observations $o_{q}$ are treated as the anchor while the key observations $o_{k}$ contain the positive and negatives, all constructed from the minibatch sampled for the RL update. The keys are encoded with a momentum averaged version of the query encoder. The RL policy and (or) value function are built on top of the query encoder which is jointly trained with the contrastive and reinforcement learning objectives. CURL is a generic framework that can be plugged into any RL algorithm that relies on learning representations from high dimensional images.

图 1：Contrastive Unsupervised Representations for Reinforcement Learning（CURL）把实例对比学习与强化学习结合。CURL 用对比损失迫使观测 $o$ 的数据增强版本 $o_{q}$ 与 $o_{k}$ 的嵌入一致，从而训练视觉表征编码器。查询观测 $o_{q}$ 作锚点，键观测 $o_{k}$ 提供正样本与负样本，均来自为 RL 更新采样的小批量。键由查询编码器的动量平均版本编码。RL 策略与（或）价值函数建在查询编码器之上，并与对比、强化学习目标联合训练。CURL 是通用框架，可插入任何依赖从高维图像学习表征的 RL 算法。

However, it has been empirically observed that reinforcement learning from high dimensional observations such as raw pixels is sample-inefficient (Lake et al. 2017; Kaiser et al. 2019). Moreover, it is widely accepted that learning policies from physical state based features is significantly more sample-efficient than learning from pixels (Tassa et al. 2018). In principle, if the state information is present in the pixel data, then we should be able to learn representations that extract the relevant state information. For this reason, it may be possible to learn from pixels as fast as from state given the right representation.

但经验表明：从原始像素等高维观测做强化学习样本效率很低（Lake et al. 2017；Kaiser et al. 2019）。普遍认为，从物理状态特征学策略远比从像素学更省样本（Tassa et al. 2018）。原则上，若状态信息包含在像素中，就应能学到抽取相关状态的表征；因此在表征得当的前提下，从像素学习有望与从状态一样快。

From a practical standpoint, although high rendering speeds in simulated environments enable RL agents to solve complex tasks within reasonable wall clock time, learning in the real world means that agents are bound to work within the limitations of physics. Kalashnikov et al. 2018 needed a farm of robotic arms that collected large scale robot interaction data over several months to develop their robot grasp value functions and policies. The data-efficiency of the whole pipeline thus has significant room for improvement. Similarly, in simulated worlds which are limited by rendering speeds in the absence of GPU accelerators, data efficiency is extremely crucial to have a fast experimental turnover and iteration. Therefore, improving the sample efficiency of reinforcement learning (RL) methods that operate from high dimensional observations is of paramount importance to RL research both in simulation and the real world and allows for faster progress towards the broader goal of developing intelligent autonomous agents.

实践上，仿真环境渲染很快，智能体可在合理墙钟时间内解复杂任务；真实世界则受物理约束。Kalashnikov et al. 2018 需要一排机械臂数月采集大规模交互数据，才训出抓取值函数与策略——整条管线的数据效率仍有很大提升空间。在缺乏 GPU 加速、渲染受限的仿真世界中，数据效率对快速实验迭代同样关键。因此，提高从高维观测出发的强化学习样本效率，对仿真与真实世界 RL 研究都至关重要，也有助于更快走向智能自主智能体。

A number of approaches have been proposed in the literature to address the sample inefficiency of deep RL algorithms. Broadly, they can be classified into two streams of research, though not mutually exclusive: (i) Auxiliary tasks on the agent’s sensory observations; (ii) World models that predict the future. While the former class of methods use auxiliary self-supervision tasks to accelerate the learning progress of model-free RL methods (Jaderberg et al. 2016; Mirowski et al. 2016), the latter class of methods build explicit predictive models of the world and use those models to plan through or collect fictitious rollouts for model-free methods to learn from (Sutton 1990; Ha & Schmidhuber 2018; Kaiser et al. 2019; Schrittwieser et al. 2019).

文献中已有多种提升深度 RL 样本效率的思路，大致可分两类（并不互斥）：（i）智能体感官观测上的辅助任务；（ii）预测未来的世界模型。前者用辅助自监督任务加速无模型 RL（Jaderberg et al. 2016；Mirowski et al. 2016）；后者显式建世界预测模型，用于规划或生成虚构轨迹供无模型方法学习（Sutton 1990；Ha & Schmidhuber 2018；Kaiser et al. 2019；Schrittwieser et al. 2019）。

Our work falls into the first class of models, which use auxiliary tasks to improve sample efficiency. Our hypothesis is simple: If an agent learns a useful semantic representation from high dimensional observations, control algorithms built on top of those representations should be significantly more data-efficient. Self-supervised representation learning has seen dramatic progress in the last couple of years with huge advances in masked language modeling (Devlin et al. 2018) and contrastive learning (Hénaff et al. 2019; He et al. 2019a; Chen et al. 2020) for language and vision respectively. The representations uncovered by these objectives improve the performance of any supervised learning system especially in scenarios where the amount of labeled data available for the downstream task is really low.

本文属于第一类：用辅助任务提升样本效率。假设很简单：若智能体能从高维观测学到有用语义表征，建在其上的控制算法应显著更省数据。 自监督表征学习近年进展迅猛，语言侧有掩码语言建模（Devlin et al. 2018），视觉侧有对比学习（Hénaff et al. 2019；He et al. 2019a；Chen et al. 2020）。这些目标得到的表征能提升各类监督学习，尤其下游标注很少时。

We take inspiration from the contrastive pre-training successes in computer vision. However, there are a couple of key differences: (i) There is no giant unlabeled dataset of millions of images available beforehand - the dataset is collected online from the agent’s interactions and changes dynamically with the agent’s experience; (ii) The agent has to perform unsupervised and reinforcement learning simultaneously as opposed to fine-tuning a pre-trained network for a specific downstream task. These two differences introduce a different challenge: How can we use contrastive learning for improving agents that can learn to control effectively and efficiently from online interactions?

我们受计算机视觉对比预训练成功启发，但有两点关键差异：（i）事先没有百万级未标注图像库——数据来自智能体在线交互，并随经验动态变化；（ii）智能体必须同时做无监督与强化学习，而非为某一下游任务微调预训练网络。于是问题变成：如何用对比学习提升能从在线交互中有效、高效学会控制的智能体？

To address this challenge, we propose CURL - Contrastive Uunsupervised Representations for Reinforcement Learning. CURL uses a form of contrastive learning that maximizes agreement between augmented versions of the same observation, where each observation is a stack of temporally sequential frames. We show that CURL significantly improves sample-efficiency over prior pixel-based methods by performing contrastive learning simultaneously with an off-policy RL algorithm. CURL coupled with the Soft-Actor-Critic (SAC) (Haarnoja et al. 2018) results in 1.9x median higher performance over Dreamer, a prior state-of-the-art algorithm on DMControl environments, benchmarked at 100k environment steps and matches the performance of state-based SAC on the majority of 16 environments tested, a first for pixel-based methods. In the Atari setting benchmarked at 100k interaction steps, we show that CURL coupled with a data-efficient version of Rainbow DQN (van Hasselt et al. 2019) results in 1.2x median higher performance over prior methods such as SimPLe (Kaiser et al. 2019), improving upon Efficient Rainbow (van Hasselt et al. 2019) on 19 out of 26 Atari games, surpassing human efficiency on two games.

为此我们提出 CURL——Contrastive Unsupervised Representations for Reinforcement Learning。CURL 采用一种对比学习：最大化同一观测的增强版本之间的一致性，每个观测是时间上连续的帧堆叠。我们表明：与离策略 RL 同时做对比学习，可相对先前像素方法显著提升样本效率。CURL 与 Soft Actor-Critic（SAC）（Haarnoja et al. 2018）结合后，在 DMControl 环境、以 100k 环境步 为基准时，相对先前前沿 Dreamer 的中位性能高 1.9×，并在所测 16 个环境的多数上对齐基于状态的 SAC——对像素方法而言是首次。在 Atari、以 100k 交互步 为基准时，CURL 与数据高效版 Rainbow DQN（van Hasselt et al. 2019）结合，相对 SimPLe（Kaiser et al. 2019）等先前方法中位性能高 1.2×，在 26 款游戏中的 19 款上超过 Efficient Rainbow（van Hasselt et al. 2019），并在两款游戏上超过人类效率。

While contrastive learning in aid of model-free RL has been studied in the past by van den Oord et al. 2018 using Contrastive Predictive Coding (CPC), the results were mixed with marginal gains in a few DMLab (Espeholt et al. 2018) environments. CURL is the first model to show substantial data-efficiency gains from using a contrastive self-supervised learning objective for model-free RL agents across a multitude of pixel based continuous and discrete control tasks in DMControl and Atari.

用对比学习辅助无模型 RL 既往已有 van den Oord et al. 2018 的 Contrastive Predictive Coding（CPC），但结果好坏参半，仅在少数 DMLab（Espeholt et al. 2018）环境上有边际收益。CURL 是首个在 DMControl 与 Atari 大量像素连续/离散控制任务上，用对比自监督目标给无模型 RL 带来显著数据效率收益的模型。

We prioritize designing a simple and easily reproducible pipeline. While the promise of auxiliary tasks and learning world models for RL agents has been demonstrated in prior work, there’s an added layer of complexity when introducing components like modeling the future in a latent space (van den Oord et al. 2018; Ha & Schmidhuber 2018). CURL is designed to add minimal overhead in terms of architecture and model learning. The contrastive learning objective in CURL operates with the same latent space and architecture typically used for model-free RL and seamlessly integrates with the training pipeline without the need to introduce multiple additional hyperparameters.

我们优先设计简单、易复现的管线。辅助任务与世界模型的价值已有展示，但在隐空间建模未来（van den Oord et al. 2018；Ha & Schmidhuber 2018）会再加一层复杂度。CURL 在结构与模型学习上的额外开销尽量小：对比目标与无模型 RL 常用的同一隐空间与结构一起工作，无缝接入训练管线，无需引入大量额外超参。

Our paper makes the following key contributions: We present CURL, a simple framework that integrates contrastive learning with model-free RL with minimal changes to the architecture and training pipeline. Using 16 complex control tasks from the DeepMind control (DMControl) suite and 26 Atari games, we empirically show that contrastive learning combined with model-free RL outperforms the prior state-of-the-art by 1.9x on DMControl and 1.2x on Atari compared across leading prior pixel-based methods. CURL is also the first algorithm across both model-based and model-free methods that operates purely from pixels, and nearly matches the performance and sample-efficiency of a SAC algorithm trained from the state based features on the DMControl suite. Finally, our design is simple and does not require any custom architectural choices or hyperparameters which is crucial for reproducible end-to-end training. Through these strong empirical results, we demonstrate that a contrastive objective is the preferred self-supervised auxiliary task for achieving sample-efficiency compared to reconstruction based methods, and enables model-free methods to outperform state-of-the-art model-based methods in terms of data-efficiency.

本文主要贡献如下：提出 CURL——以最少结构与训练改动，把对比学习接入无模型 RL 的简单框架。在 DeepMind Control（DMControl）套件 16 项复杂控制任务与 26 款 Atari 游戏上，经验表明对比学习与无模型 RL 结合后，相对先前前沿像素方法在 DMControl 上提升 1.9×、在 Atari 上提升 1.2×。CURL 也是在基于模型与无模型方法中首个纯像素运作、并几乎对齐 DMControl 上基于状态特征的 SAC 性能与样本效率的算法。设计简单，无需定制结构或超参，对可复现的端到端训练至关重要。这些强经验结果表明：相对基于重建的方法，对比目标是达成样本效率更受欢迎的自监督辅助任务，并能使无模型方法在数据效率上超过当时前沿基于模型的方法。

2 Related Work

2 相关工作

Self-Supervised Learning: Self-Supervised Learning is aimed at learning rich representations of high dimensional unlabeled data to be useful for a wide variety of tasks. The fields of natural language processing and computer vision have seen dramatic advances in self-supervised methods such as BERT (Devlin et al. 2018), CPC, MoCo, SimCLR (Hénaff et al. 2019; He et al. 2019a; Chen et al. 2020).

自监督学习： 自监督学习旨在从未标注高维数据学到丰富表征，以服务多样任务。自然语言处理与计算机视觉已出现 BERT（Devlin et al. 2018）、CPC、MoCo、SimCLR（Hénaff et al. 2019；He et al. 2019a；Chen et al. 2020）等显著进展。

Contrastive Learning: Contrastive Learning is a framework to learn representations that obey similarity constraints in a dataset typically organized by similar and dissimilar pairs. This is often best understood as performing a dictionary lookup task wherein the positive and negatives represent a set of keys with respect to a query (or an anchor). A simple instantiation of contrastive learning is Instance Discrimination (Wu et al. 2018) wherein a query and key are positive pairs if they are data-augmentations of the same instance (example, image) and negative otherwise. A key challenge in contrastive learning is the choice of negatives which can decide the quality of the underlying representations learned. The loss functions used to contrast could be among several choices such as InfoNCE (van den Oord et al. 2018), Triplet (Wang & Gupta 2015), Siamese (Chopra et al. 2005) and so forth.

对比学习： 对比学习在按相似/不相似对组织的数据上学习服从相似性约束的表征，常被理解为相对查询（或锚点）的字典查找：正负样本构成一组键。简单实例是 Instance Discrimination（Wu et al. 2018）：若查询与键是同一实例（如图像）的数据增强则视为正对，否则为负。负样本选择会决定表征质量。对比损失可选 InfoNCE（van den Oord et al. 2018）、Triplet（Wang & Gupta 2015）、Siamese（Chopra et al. 2005）等。

Self-Supervised Learning for RL: Auxiliary tasks such as predicting the future conditioned on the past observation(s) and action(s) (Jaderberg et al. 2016; Shelhamer et al. 2016; van den Oord et al. 2018; Schmidhuber 1990) are a few representative examples of using auxiliary tasks to improve the sample-efficiency of model-free RL algorithms. The future prediction is either done in a pixel space (Jaderberg et al. 2016) or latent space (van den Oord et al. 2018). The sample-efficiency gains from reconstruction-based auxiliary losses have been benchmarked in Jaderberg et al. 2016; Higgins et al. 2017; Yarats et al. 2019. Contrastive learning has been used to extract reward signals in the latent space (Sermanet et al. 2018; Dwibedi et al. 2018; Warde-Farley et al. 2018); and study representation learning on Atari games by Anand et al. 2019.

面向 RL 的自监督学习： 以过去观测（与动作）为条件预测未来（Jaderberg et al. 2016；Shelhamer et al. 2016；van den Oord et al. 2018；Schmidhuber 1990）等辅助任务，是提升无模型 RL 样本效率的代表性做法。未来预测可在像素空间（Jaderberg et al. 2016）或隐空间（van den Oord et al. 2018）进行。基于重建的辅助损失带来的样本效率收益见 Jaderberg et al. 2016；Higgins et al. 2017；Yarats et al. 2019。对比学习也曾用于在隐空间抽取奖励信号（Sermanet et al. 2018；Dwibedi et al. 2018；Warde-Farley et al. 2018）；Anand et al. 2019 则在 Atari 上研究表征学习。

World Models for sample-efficiency: While joint learning of an auxiliary unsupervised task with model-free RL is one way to improve the sample-efficiency of agents, there has also been another line of research that has tried to learn world models of the environment and use them to sample rollouts and plan. An early instantiation of the generic principle was put forth by Sutton 1990 in Dyna where fictitious samples rolled out from a learned world model are used in addition to the agent’s experience for sample-efficient learning. Planning through a learned world model (Srinivas et al. 2018) is another way to improve sample-efficiency. While Jaderberg et al. 2016; van den Oord et al. 2018; Lee et al. 2019 also learn pixel and latent space forward models, the models are learned to shape the latent representations, and there is no explicit Dyna or planning. Planning through learned world models has been successfully demonstrated in Ha & Schmidhuber 2018; Hafner et al. 2018; Hafner et al. 2019. Kaiser et al. 2019 introduce SimPLe which implements Dyna with expressive deep neural networks for the world model for sample-efficiency on Atari games.

面向样本效率的世界模型： 除与无模型 RL 联合学辅助无监督任务外，另一路线是学习环境世界模型并用其采样轨迹与规划。Sutton 1990 的 Dyna 给出一般原则：用学到的世界模型滚动出的虚构样本，补充智能体经验以提升样本效率。经学到的世界模型做规划（Srinivas et al. 2018）是另一条路径。Jaderberg et al. 2016；van den Oord et al. 2018；Lee et al. 2019 也学习像素或隐空间前向模型，但主要用于塑造隐表征，并无显式 Dyna 或规划。Ha & Schmidhuber 2018；Hafner et al. 2018；Hafner et al. 2019 成功展示了经学到世界模型的规划。Kaiser et al. 2019 提出 SimPLe，用表达力强的深度网络实现 Dyna，以提升 Atari 样本效率。

Sample-efficient RL for image-based control: CURL encompasses the areas of self-supervision, contrastive learning and using auxiliary tasks for sample-efficient RL. We benchmark for sample-efficiency on the DMControl suite (Tassa et al. 2018) and Atari Games benchmarks (Bellemare et al. 2013). The DMControl suite has been used widely by Yarats et al. 2019, Hafner et al. 2018, Hafner et al. 2019 and Lee et al. 2019 for benchmarking sample-efficiency for image based continuous control methods. As for Atari, Kaiser et al. 2019 propose to use the 100k interaction steps benchmark for sample-efficiency which has been adopted in Kielak 2020; van Hasselt et al. 2019. The Rainbow DQN (Hessel et al. 2017) was originally proposed for maximum sample-efficiency on the Atari benchmark and in recent times has been adapted to a version known as Data-Efficient Rainbow (van Hasselt et al. 2019) with competitive performance to SimPLe without learning world models. We benchmark extensively against both model-based and model-free algorithms in our experiments. For the DMControl experiments, we compare our method to Dreamer, PlaNet, SLAC, SAC+AE whereas for Atari experiments we compare to SimPLe, Rainbow, and OverTrained Rainbow (OTRainbow) and Efficient Rainbow (Eff. Rainbow).

面向图像控制的样本高效 RL： CURL 涵盖自监督、对比学习与用辅助任务做样本高效 RL。我们在 DMControl 套件（Tassa et al. 2018）与 Atari Games（Bellemare et al. 2013）上评测样本效率。DMControl 被 Yarats et al. 2019、Hafner et al. 2018、Hafner et al. 2019 与 Lee et al. 2019 广泛用于基准图像连续控制的样本效率。Atari 方面，Kaiser et al. 2019 提出用 100k 交互步作样本效率基准，并为 Kielak 2020；van Hasselt et al. 2019 所采用。Rainbow DQN（Hessel et al. 2017）本为 Atari 最大样本效率提出，近期又有 Data-Efficient Rainbow（van Hasselt et al. 2019），在不学世界模型时仍可与 SimPLe 竞争。我们在实验中广泛对比基于模型与无模型算法：DMControl 上对比 Dreamer、PlaNet、SLAC、SAC+AE；Atari 上对比 SimPLe、Rainbow、OverTrained Rainbow（OTRainbow）与 Efficient Rainbow（Eff. Rainbow）。

3 Background

3 背景

Figure 2: CURL Architecture: A batch of transitions is sampled from the replay buffer. Observations are then data-augmented twice to form query and key observations, which are then encoded with the query encoder and key encoders, respectively. The queries are passed to the RL algorithm while query-key pairs are passed to the contrastive learning objective. During the gradient update step, only the query encoder is updated. The key encoder weights are the moving average (EMA) of the query weights similar to MoCo (He et al. 2019a).

图 2：CURL 结构。从回放缓冲采样一批转移；对观测做两次数据增强得到查询与键，分别用查询编码器与键编码器编码。查询送入 RL 算法，查询–键对送入对比学习目标。梯度更新时只更新查询编码器；键编码器权重为查询权重的滑动平均（EMA），类似 MoCo（He et al. 2019a）。

CURL is a general framework for combining contrastive learning with RL. In principle, one could use any RL algorithm in the CURL pipeline, be it on-policy or off-policy. We use the widely adopted Soft Actor Critic (SAC) (Haarnoja et al. 2018) for continuous control benchmarks (DM Control) and Rainbow DQN (Hessel et al. 2017; van Hasselt et al. 2019) for discrete control benchmarks (Atari). Below, we review SAC, Rainbow DQN and Contrastive Learning.

CURL 是把对比学习与 RL 结合的通用框架。原则上可用任意 RL 算法，不论同策略或离策略。连续控制基准（DM Control）我们用广泛采用的 Soft Actor Critic（SAC）（Haarnoja et al. 2018）；离散控制基准（Atari）用 Rainbow DQN（Hessel et al. 2017；van Hasselt et al. 2019）。下文回顾 SAC、Rainbow DQN 与对比学习。

3.1 Soft Actor Critic

SAC is an off-policy RL algorithm that optimizes a stochastic policy for maximizing the expected trajectory returns. Like other state-of-the-art end-to-end RL algorithms, SAC is effective when solving tasks from state observations but fails to learn efficient policies from pixels. SAC is an actor-critic method that learns a policy $\pi_{\psi}$ and critics $Q_{\phi_{1}}$ and $Q_{\phi_{2}}$. The parameters $\phi_{i}$ are learned by minimizing the Bellman error:

SAC 是离策略 RL 算法，优化随机策略以最大化期望轨迹回报。与其他端到端前沿 RL 一样，SAC 在状态观测上有效，但从像素学高效策略则失败。SAC 是演员–评论家方法，学习策略 $\pi_{\psi}$ 与评论家 $Q_{\phi_{1}}$、$Q_{\phi_{2}}$。参数 $\phi_{i}$ 通过最小化贝尔曼误差学习：

$$\mathcal{L}(\phi_{i},\mathcal{B})=\mathbb{E}_{t\sim\mathcal{B}}\left[\left(Q_{\phi_{i}}(o,a)-\left(r+\gamma(1-d)\mathcal{T}\right)\right)^{2}\right] \tag{1}$$

where $t=(o,a,o^{\prime},r,d)$ is a tuple with observation $o$, action $a$, reward $r$ and done signal $d$, $\mathcal{B}$ is the replay buffer, and $\mathcal{T}$ is the target, defined as:

其中 $t=(o,a,o^{\prime},r,d)$ 为含观测 $o$、动作 $a$、奖励 $r$ 与终止信号 $d$ 的元组，$\mathcal{B}$ 为回放缓冲，$\mathcal{T}$ 为目标：

$$\mathcal{T}=\left(\min_{i=1,2}Q^{*}_{\phi_{i}}(o^{\prime},a^{\prime})-\alpha\log\pi_{\psi}(a^{\prime}|o^{\prime})\right) \tag{2}$$

In the target equation (2), $Q^{*}_{\phi_{i}}$ denotes the exponential moving average (EMA) of the parameters of $Q_{\phi_{i}}$. Using the EMA has empirically shown to improve training stability in off-policy RL algorithms. The parameter $\alpha$ is a positive entropy coefficient that determines the priority of the entropy maximization over value function optimization.

目标式 (2) 中，$Q^{*}_{\phi_{i}}$ 表示 $Q_{\phi_{i}}$ 参数的指数滑动平均（EMA）。经验上，EMA 能提升离策略 RL 训练稳定性。正熵系数 $\alpha$ 决定相对价值函数优化，熵最大化的优先级。

While the critic is given by $Q_{\phi_{i}}$, the actor samples actions from policy $\pi_{\psi}$ and is trained by maximizing the expected return of its actions as in:

评论家由 $Q_{\phi_{i}}$ 给出；演员从策略 $\pi_{\psi}$ 采样动作，并最大化其动作的期望回报：

$$\mathcal{L}(\psi)=\mathbb{E}_{a\sim\pi}\left[Q^{\pi}(o,a)-\alpha\log\pi_{\psi}(a|o)\right] \tag{3}$$

where actions are sampled stochastically from the policy $a_{\psi}(o,\xi)\sim\tanh\left(\mu_{\psi}(o)+\sigma_{\psi}(o)\odot\xi\right)$ and $\xi\sim\mathcal{N}(0,I)$ is a standard normalized noise vector.

其中动作按 $a_{\psi}(o,\xi)\sim\tanh\left(\mu_{\psi}(o)+\sigma_{\psi}(o)\odot\xi\right)$ 随机采样，$\xi\sim\mathcal{N}(0,I)$ 为标准正态噪声。

3.2 Rainbow

Rainbow DQN (Hessel et al. 2017) is best summarized as multiple improvements on top of the original Nature DQN (Mnih et al. 2015) applied together. Specifically, Deep Q Network (DQN) (Mnih et al. 2015) combines the off-policy algorithm Q-Learning with a convolutional neural network as the function approximator to map raw pixels to action value functions. Since then, multiple improvements have been proposed such as Double Q Learning (Van Hasselt et al. 2016), Dueling Network Architectures (Wang et al. 2015), Prioritized Experience Replay (Schaul et al. 2015), and Noisy Networks (Fortunato et al. 2017). Additionally, distributional reinforcement learning (Bellemare et al. 2017) proposed the technique of predicting a distribution over possible value function bins through the C51 Algorithm. Rainbow DQN combines all of the above techniques into a single off-policy algorithm for state-of-the-art sample efficiency on Atari benchmarks. Additionally, Rainbow also makes use of multi-step returns (Sutton et al. 1998). van Hasselt et al. 2019 propose a data-efficient version of the Rainbow which can be summarized as an improved configuration of hyperparameters that is optimized for performance benchmarked at 100K interaction steps.

Rainbow DQN（Hessel et al. 2017）可概括为在原始 Nature DQN（Mnih et al. 2015）上叠加多项改进。Deep Q Network（DQN）（Mnih et al. 2015）把离策略 Q-Learning 与卷积网络函数逼近结合，将原始像素映到动作价值。此后出现 Double Q Learning（Van Hasselt et al. 2016）、Dueling 网络（Wang et al. 2015）、优先经验回放（Schaul et al. 2015）、Noisy Networks（Fortunato et al. 2017）等。分布强化学习（Bellemare et al. 2017）提出用 C51 预测价值函数分箱上的分布。Rainbow 把上述技术并入单一离策略算法，以求 Atari 上当时最佳样本效率；并使用多步回报（Sutton et al. 1998）。van Hasselt et al. 2019 提出面向数据效率的 Rainbow 配置，针对 100K 交互步基准优化超参。

3.3 Contrastive Learning

3.3 对比学习

A key component of CURL is the ability to learn rich representations of high dimensional data using contrastive unsupervised learning. Contrastive learning (Hadsell et al. 2006; LeCun et al. 2006; van den Oord et al. 2018; Wu et al. 2018; He et al. 2019a) can be understood as learning a differentiable dictionary look-up task. Given a query $q$ and keys $\mathbb{K}=\{k_{0},k_{1},\dots\}$ and an explicitly known partition of $\mathbb{K}$ (with respect to $q$) $P(\mathbb{K})=(\{k_{+}\},\mathbb{K}\setminus\{k_{+}\})$, the goal of contrastive learning is to ensure that $q$ matches with $k_{+}$ relatively more than any of the keys in $\mathbb{K}\setminus\{k_{+}\}$. $q,\mathbb{K},k_{+},$ and $\mathbb{K}\setminus\{k_{+}\}$ are also referred to as anchor, targets, positive, negatives respectively in the parlance of contrastive learning (van den Oord et al. 2018; He et al. 2019a). Similarities between the anchor and targets are best modeled with dot products ($q^{T}k)$ (Wu et al. 2018; He et al. 2019a) or bilinear products ($q^{T}Wk$) (van den Oord et al. 2018; Hénaff et al. 2019) though other forms like euclidean distances are also common (Schroff et al. 2015; Wang & Gupta 2015). To learn embeddings that respect these similarity relations, van den Oord et al. 2018 propose the InfoNCE loss:

CURL 的关键是用对比无监督学习高维数据的丰富表征。对比学习（Hadsell et al. 2006；LeCun et al. 2006；van den Oord et al. 2018；Wu et al. 2018；He et al. 2019a）可理解为学习可微的字典查找。给定查询 $q$、键集 $\mathbb{K}=\{k_{0},k_{1},\dots\}$ 以及相对 $q$ 已知的划分 $P(\mathbb{K})=(\{k_{+}\},\mathbb{K}\setminus\{k_{+}\})$，目标是使 $q$ 与 $k_{+}$ 的匹配强于与 $\mathbb{K}\setminus\{k_{+}\}$ 中任一键。$q,\mathbb{K},k_{+},\mathbb{K}\setminus\{k_{+\}}$ 在对比学习术语中亦称锚点、目标、正样本、负样本（van den Oord et al. 2018；He et al. 2019a）。锚点与目标的相似性常用点积 ($q^{T}k$)（Wu et al. 2018；He et al. 2019a）或双线性积 ($q^{T}Wk$)（van den Oord et al. 2018；Hénaff et al. 2019）建模，也可用欧氏距离等形式（Schroff et al. 2015；Wang & Gupta 2015）。为学习尊重这些相似性的嵌入，van den Oord et al. 2018 提出 InfoNCE 损失：

$$\mathcal{L}_{q}=\log\frac{\exp(q^{T}Wk_{+})}{\exp(q^{T}Wk_{+})+\sum_{i=0}^{K-1}\exp(q^{T}Wk_{i})} \tag{4}$$

The loss 4 can be interpreted as the log-loss of a $K$-way softmax classifier whose label is $k_{+}$.

损失 (4) 可理解为标签为 $k_{+}$ 的 $K$ 类 softmax 分类器的对数损失。

4 CURL Implementation

4 CURL 实现

CURL minimally modifies a base RL algorithm by training the contrastive objective as an auxiliary loss during the batch update. In our experiments, we train CURL alongside two model-free RL algorithms — SAC for DMControl experiments and Rainbow DQN (data-efficient version) for Atari experiments. To specify a contrastive learning objective, we need to define (i) the discrimination objective (ii) the transformation for generating query-key observations (iii) the embedding procedure for transforming observations into queries and keys and (iv) the inner product used as a similarity measure between the query-key pairs in the contrastive loss. The exact specification these aspects largely determine the quality of the learned representations.

CURL 在批次更新时把对比目标作为辅助损失训练，从而对基线 RL 算法做最小改动。实验中，DMControl 用 SAC，Atari 用 Rainbow DQN（数据高效版）与 CURL 一起训。要指定对比学习目标，需定义：（i）判别目标；（ii）生成查询–键观测的变换；（iii）把观测映成查询与键的嵌入过程；（iv）对比损失中查询–键对的内积相似度。这些选择很大程度上决定表征质量。

We first summarize the CURL architecture, and then cover each architectural choice in detail.

先概述 CURL 结构，再逐项说明设计选择。

4.1 Architectural Overview

4.1 结构概览

CURL uses instance discrimination with similarities to SimCLR (Chen et al. 2020), MoCo (He et al. 2019a) and CPC (Hénaff et al. 2019). Most Deep RL architectures operate with a stack of temporally consecutive frames as input (Hessel et al. 2017). Therefore, instance discrimination is performed across the frame stacks as opposed to single image instances. We use a momentum encoding procedure for targets similar to MoCo (He et al. 2019b) which we found to be better performing for RL. Finally, for the InfoNCE score function, we use a bi-linear inner product similar to CPC (van den Oord et al. 2018) which we found to work better than unit norm vector products used in MoCo and SimCLR. Ablations for both the encoder and the similarity measure choices are shown in Figure 5. The contrastive representation is trained jointly with the RL algorithm, and the latent code receives gradients from both the contrastive objective and the Q-function. An overview of the architecture is shown in in Figure 2.

CURL 做实例判别，思路接近 SimCLR（Chen et al. 2020）、MoCo（He et al. 2019a）与 CPC（Hénaff et al. 2019）。多数深度 RL 结构以时间连续帧堆叠为输入（Hessel et al. 2017），因此实例判别在帧堆叠而非单图上进行。目标编码采用类似 MoCo（He et al. 2019b）的动量编码，我们发现这对 RL 更有效。InfoNCE 打分使用类似 CPC（van den Oord et al. 2018）的双线性内积，效果优于 MoCo 与 SimCLR 的单位范数向量积。编码器与相似度选择的消融见图 5。对比表征与 RL 联合训练，隐码同时接收对比目标与 Q 函数梯度。结构概览见图 2。

4.2 Discrimination Objective

4.2 判别目标

A key component of contrastive representation learning is the choice of positives and negative samples relative to an anchor (Bachman et al. 2019; Tian et al. 2019; Hénaff et al. 2019; He et al. 2019a; Chen et al. 2020). Contrastive Predictive Coding (CPC) based pipelines (Hénaff et al. 2019; van den Oord et al. 2018) use groups of image patches separated by a carefully chosen spatial offset for anchors and positives while the negatives come from other patches within the image and from other images.

对比表征学习的关键是相对锚点选择正负样本（Bachman et al. 2019；Tian et al. 2019；Hénaff et al. 2019；He et al. 2019a；Chen et al. 2020）。基于 Contrastive Predictive Coding（CPC）的管线（Hénaff et al. 2019；van den Oord et al. 2018）用经仔细选取空间偏移的图像块组作锚点与正样本，负样本来自同图其他块及其他图像。

While patches are a powerful way to incorporate spatial and instance discrimination together, they introduce extra hyperparameters and architectural design choices which may be hard to adapt for a new problem. SimCLR (Chen et al. 2020) and MoCo (He et al. 2019a) opt for a simpler design where there is no patch extraction.

图像块能同时纳入空间与实例判别，但会引入额外超参与结构设计，换任务时往往难适配。SimCLR（Chen et al. 2020）与 MoCo（He et al. 2019a）选择更简单设计：不做块抽取。

Discriminating transformed image instances as opposed to image-patches within the same image optimizes a simpler instance discrimination objective (Wu et al. 2018) with the InfoNCE loss and requires minimal architectural adjustments (He et al. 2019b; Chen et al. 2020). It is preferable to pick a simpler discrimination objective in the RL setting for two reasons. First, considering the brittleness of reinforcement learning algorithms (Henderson et al. 2018), complex discrimination may destabilize the RL objective. Second, since RL algorithms are trained on dynamically generated datasets, a complex discrimination objective may significantly increase the wall-clock training time. CURL therefore uses instance discrimination rather than patch discrimination. One could view contrastive instance discrimination setups like SimCLR and MoCo as maximizing mutual information between an image and its augmented version. The reader is encouraged to refer to van den Oord et al. 2018; Hjelm et al. 2018; Tschannen et al. 2019 for connections between contrastive learning and mutual information.

对变换后的图像实例而非同图内图像块做判别，是用 InfoNCE 优化更简单的实例判别目标（Wu et al. 2018），结构改动最少（He et al. 2019b；Chen et al. 2020）。在 RL 设定下更偏好简单判别，理由有二。第一，强化学习算法脆弱（Henderson et al. 2018），复杂判别可能破坏 RL 目标。第二，RL 数据动态生成，复杂判别会显著增加墙钟训练时间。故 CURL 用实例判别而非块判别。也可把 SimCLR、MoCo 这类对比实例判别视为最大化图像与其增强版本之间的互信息。对比学习与互信息的联系见 van den Oord et al. 2018；Hjelm et al. 2018；Tschannen et al. 2019。

4.3 Query-Key Pair Generation

4.3 查询–键对生成

Similar to instance discrimination in the image setting (He et al. 2019b; Chen et al. 2020), the anchor and positive observations are two different augmentations of the same image while negatives come from other images. CURL primarily relies on the random crop data augmentation, where a random square patch is cropped from the original rendering.

与图像设定下的实例判别类似（He et al. 2019b；Chen et al. 2020），锚点与正样本是同一图像的两种不同增强，负样本来自其他图像。CURL 主要依赖随机裁剪数据增强：从原始渲染中裁出随机方形块。

A significant difference between RL and computer vision settings is that an instance ingested by a model-free RL algorithm that operates from pixels is not just a single image but a stack of frames (Mnih et al. 2015). For example, one typically feeds in a stack of 4 frames in Atari experiments and a stack of 3 frames in DMControl. This way, performing instance discrimination on frame stacks allows CURL to learn both spatial and temporal discriminative features. For details regarding the extent to which CURL captures temporal features, see Appendix E.

RL 与计算机视觉的显著差别是：从像素运作的无模型 RL 摄入的实例往往不是单张图，而是帧堆叠（Mnih et al. 2015）。例如 Atari 实验通常输入 4 帧堆叠，DMControl 为 3 帧。因此在帧堆叠上做实例判别，使 CURL 能学到空间与时间判别特征。时间特征捕获程度见附录 E。

We apply the random augmentations across the batch but consistently across each stack of frames to retain information about the temporal structure of the observation. The augmentation procedure is shown in Figure 3. For more details, refer to Appendix A.

随机增强跨批次施加，但在每个帧堆叠内部保持一致，以保留观测的时间结构。增强过程见图 3，细节见附录 A。

Figure 3: Visually illustrating the process of generating an anchor and its positive using stochastic random crops. Our aspect ratio for cropping is 0.84, i.e, we crop a $84\times 84$ image from a $100\times 100$ simulation-rendered image. Applying the same random crop coordinates across all frames in the stack ensures time-consistent spatial jittering.

图 3：用随机裁剪生成锚点及其正样本的示意。裁剪宽高比为 0.84，即从 $100\times 100$ 仿真渲染图裁出 $84\times 84$。对堆叠中所有帧使用相同随机裁剪坐标，以保证时间一致的空间抖动。

4.4 Similarity Measure

4.4 相似度度量

Another determining factor in the discrimination objective is the inner product used to measure agreement between query-key pairs. CURL employs the bi-linear inner-product $\text{sim}(q,k)=q^{T}Wk$, where $W$ is a learned parameter matrix. We found this similarity measure to outperform the normalized dot-product (see Figure 5 in Appendix A) used in recent state-of-the-art contrastive learning methods in computer vision like MoCo and SimCLR.

判别目标的另一决定因素是衡量查询–键一致性的内积。CURL 使用双线性内积 $\text{sim}(q,k)=q^{T}Wk$，其中 $W$ 为可学习参数矩阵。我们发现该度量优于近期 MoCo、SimCLR 等视觉对比方法所用的归一化点积（见附录 A 图 5）。

4.5 Target Encoding with Momentum

4.5 带动量的目标编码

The motivation for using contrastive learning in CURL is to train encoders that map from high dimensional pixels to more semantic latents. InfoNCE is an unsupervised loss that learns encoders $f_{q}$ and $f_{k}$ mapping the raw anchors (query) $x_{q}$ and targets (keys) $x_{k}$ into latents $q=f_{q}(x_{q})$ and $k=f_{k}(x_{k})$, on which we apply the similarity dot products. It is common to share the same encoder between the anchor and target mappings, that is, to have $f_{q}=f_{k}$ (van den Oord et al. 2018; Hénaff et al. 2019).

CURL 使用对比学习，是为了训练把高维像素映到更语义隐变量的编码器。InfoNCE 是无监督损失，学习编码器 $f_{q}$ 与 $f_{k}$，把原始锚点（查询）$x_{q}$ 与目标（键）$x_{k}$ 映到隐变量 $q=f_{q}(x_{q})$、$k=f_{k}(x_{k})$，再施加相似点积。锚点与目标映射常共享编码器，即 $f_{q}=f_{k}$（van den Oord et al. 2018；Hénaff et al. 2019）。

From the perspective of viewing contrastive learning as building differentiable dictionary lookups over high dimensional entities, increasing the size of the dictionary and enriching the set of negatives is helpful in learning rich representations. He et al. 2019a propose momentum contrast (MoCo), which uses the exponentially moving average (momentum averaged) version of the query encoder $f_{q}$ for encoding the keys in $\mathbb{K}$. Given $f_{q}$ parametrized by $\theta_{q}$ and $f_{k}$ parametrized by $\theta_{k}$, MoCo performs the update $\theta_{k}=m\theta_{k}+(1-m)\theta_{q}$ and encodes any target $x_{k}$ using $\textrm{SG}(f_{k}(x_{k}))$ [SG : Stop Gradient].

若把对比学习视为在高维实体上建可微字典查找，增大字典、丰富负样本有助于学到丰富表征。He et al. 2019a 提出动量对比（MoCo）：用查询编码器 $f_{q}$ 的指数滑动平均（动量平均）版本编码 $\mathbb{K}$ 中的键。设 $f_{q}$ 参数为 $\theta_{q}$、$f_{k}$ 参数为 $\theta_{k}$，MoCo 更新 $\theta_{k}=m\theta_{k}+(1-m)\theta_{q}$，并用 $\textrm{SG}(f_{k}(x_{k}))$ 编码任意目标 $x_{k}$（SG：停止梯度）。

CURL couples frame-stack instance discrimination with momentum encoding for the targets during contrastive learning, and RL is performed on top of the encoder features.

CURL 在对比学习中把帧堆叠实例判别与目标的动量编码相结合，并在编码器特征之上做 RL。

4.6 Differences Between CURL and Prior Contrastive Methods in RL

4.6 CURL 与先前 RL 对比方法的差异

van den Oord et al. 2018 use Contastive Predictive Coding (CPC) as an auxiliary task wherein an LSTM operates on a latent space of a convolutional encoder; and both the CPC and A2C (Mnih et al. 2015) objectives are jointly optimized. CURL avoids using pipelines that predict the future in a latent space such as van den Oord et al. 2018; Hafner et al. 2019. In CURL, we opt for a simple instance discrimination style contrastive auxiliary task.

van den Oord et al. 2018 用 Contrastive Predictive Coding（CPC）作辅助任务：LSTM 在卷积编码器隐空间上运作，并联合优化 CPC 与 A2C（Mnih et al. 2015）目标。CURL 避免 van den Oord et al. 2018；Hafner et al. 2019 这类在隐空间预测未来的管线，而采用简单的实例判别式对比辅助任务。

4.7 CURL Contrastive Learning Pseudocode (PyTorch-like)

4.7 CURL 对比学习伪代码（类 PyTorch）

f_q, f_k: encoder networks for anchor

(query) and target (keys) respectively.

loader: minibatch sampler from ReplayBuffer

B-batch_size, C-channels, H,W-spatial_dims

x : shape : [B, C, H, W]

C = c * num_frames; c=3 (R/G/B) or 1 (gray)

m: momentum, e.g. 0.95

z_dim: latent dimension

f_k.params = f_q.params
W = rand(z_dim, z_dim) # bilinear product.
for x in loader: # load minibatch from buffer
x_q = aug(x) # random augmentation
x_k = aug(x) # different random augmentation
z_q = f_q.forward(x_q)
z_k = f_k.forward(x_k)
z_k = z_k.detach() # stop gradient
proj_k = matmul(W, z_k.T) # bilinear product
logits = matmul(z_q, proj_k) # B x B

subtract max from logits for stability

logits = logits - max(logits, axis=1)
labels = arange(logits.shape[0])
loss = CrossEntropyLoss(logits, labels)
loss.backward()
update(f_q.params) # Adam
update(W) # Adam
f_k.params = m*f_k.params+(1-m)*f_q.params

5 Experiments

5 实验

5.1 Evaluation

5.1 评测

We measure the data-efficiency and performance of our method and baselines at 100k and 500k environment steps on DMControl and 100k interaction steps (400k environment steps with action repeat of 4) on Atari, which we will henceforth refer to as DMControl100k, DMControl500k and Atari100k for clarity. While Atari100k benchmark has been common practice when investigating data-efficiency on Atari (Kaiser et al. 2019; van Hasselt et al. 2019; Kielak 2020), the DMControl benchmark was set at 500k environment steps because state-based RL approaches asymptotic performance on many environments at this point, and 100k steps to measure the speed of initial learning. A broader motivation is that while RL algorithms can achieve super-human performance on Atari games, they are still far less efficient than a human learner. Training for 100-500k environment steps corresponds to a few hours of human time.

我们在 DMControl 的 100k 与 500k 环境步、以及 Atari 的 100k 交互步（动作重复 4 时对应 400k 环境步）上衡量方法与基线的数据效率与性能，后文分别记为 DMControl100k、DMControl500k 与 Atari100k。Atari100k 已是考察 Atari 数据效率的常见做法（Kaiser et al. 2019；van Hasselt et al. 2019；Kielak 2020）；DMControl 取 500k 环境步，是因为基于状态的 RL 此时常接近渐近性能，另取 100k 衡量初期学习速度。更广动机是：虽 RL 可在 Atari 上超人类，效率仍远低于人类学习者；训 100–500k 环境步约对应人类数小时。

We evaluate (i) sample-efficiency by measuring how many steps it takes the best performing baselines to match CURL performance at a fixed $T$ (100k or 500k) steps and (ii) performance by measuring the ratio of the episode returns achieved by CURL versus the best performing baseline at $T$ steps. To be explicit, when we say data or sample-efficiency we’re referring to (i) and when we say performance we’re referring to (ii).

我们评测（i）样本效率：最佳基线达到 CURL 在固定 $T$（100k 或 500k）步性能所需步数；（ii）性能：CURL 与最佳基线在 $T$ 步时的回合回报比。明确说，谈数据或样本效率指（i），谈性能指（ii）。

500K step scores	CURL	PlaNet	Dreamer	SAC+AE	SLACv1	Pixel SAC	State SAC
Finger, spin	926 $\pm$ 45	561 $\pm$ 284	796 $\pm$ 183	884 $\pm$ 128	673 $\pm$ 92	179 $\pm$ 166	923 $\pm$ 21
Cartpole, swingup	841 $\pm$ 45	475 $\pm$ 71	762 $\pm$ 27	735 $\pm$ 63	-	419 $\pm$ 40	848 $\pm$ 15
Reacher, easy	929 $\pm$ 44	210 $\pm$ 390	793 $\pm$ 164	627 $\pm$ 58	-	145 $\pm$ 30	923 $\pm$ 24
Cheetah, run	518 $\pm$ 28	305 $\pm$ 131	570 $\pm$ 253	550 $\pm$ 34	640 $\pm$ 19	197 $\pm$ 15	795 $\pm$ 30
Walker, walk	902 $\pm$ 43	351 $\pm$ 58	897 $\pm$ 49	847 $\pm$ 48	842 $\pm$ 51	42 $\pm$ 12	948 $\pm$ 54
Ball in cup, catch	959 $\pm$ 27	460 $\pm$ 380	879 $\pm$ 87	794$\pm$ 58	852 $\pm$ 71	312$\pm$ 63	974 $\pm$ 33
100K step scores
Finger, spin	767 $\pm$ 56	136 $\pm$ 216	341 $\pm$ 70	740 $\pm$ 64	693 $\pm$ 141	179 $\pm$ 66	811$\pm$46
Cartpole, swingup	582$\pm$146	297$\pm$39	326$\pm$27	311$\pm$11	-	419$\pm$40	835$\pm$22
Reacher, easy	538$\pm$233	20$\pm$50	314$\pm$155	274$\pm$14	-	145$\pm$30	746$\pm$25
Cheetah, run	299 $\pm$48	138$\pm$88	235$\pm$ 137	267$\pm$24	319$\pm$56	197$\pm$15	616$\pm$18
Walker, walk	403$\pm$24	224$\pm$48	277$\pm$12	394$\pm$22	361$\pm$73	42$\pm$12	891$\pm$82
Ball in cup, catch	769 $\pm$ 43	0 $\pm$ 0	246 $\pm$ 174	391$\pm$ 82	512 $\pm$ 110	312$\pm$ 63	746$\pm$91

Table 1: Scores achieved by CURL (mean & standard deviation for 10 seeds) and baselines on DMControl500k and 1DMControl100k. CURL achieves state-of-the-art performance on the majority (5 out of 6) environments benchmarked on DMControl500k. These environments were selected based on availability of data from baseline methods (we run CURL experiments on 16 environments in total and show results in Figure 7). The baselines are PlaNet (Hafner et al. 2018), Dreamer (Hafner et al. 2019), SAC+AE (Yarats et al. 2019), SLAC (Lee et al. 2019), pixel-based SAC and state-based SAC (Haarnoja et al. 2018). SLAC results were reported with one and three gradient updates per agent step, which we refer to as SLACv1 and SLACv2 respectively. We compare to SLACv1 since all other baselines and CURL only make one gradient update per agent step. We also ran CURL with three gradient updates per step and compare results to SLACv2 in Table 5.

500K 步得分	CURL	PlaNet	Dreamer	SAC+AE	SLACv1	Pixel SAC	State SAC
Finger, spin	926 $\pm$ 45	561 $\pm$ 284	796 $\pm$ 183	884 $\pm$ 128	673 $\pm$ 92	179 $\pm$ 166	923 $\pm$ 21
Cartpole, swingup	841 $\pm$ 45	475 $\pm$ 71	762 $\pm$ 27	735 $\pm$ 63	-	419 $\pm$ 40	848 $\pm$ 15
Reacher, easy	929 $\pm$ 44	210 $\pm$ 390	793 $\pm$ 164	627 $\pm$ 58	-	145 $\pm$ 30	923 $\pm$ 24
Cheetah, run	518 $\pm$ 28	305 $\pm$ 131	570 $\pm$ 253	550 $\pm$ 34	640 $\pm$ 19	197 $\pm$ 15	795 $\pm$ 30
Walker, walk	902 $\pm$ 43	351 $\pm$ 58	897 $\pm$ 49	847 $\pm$ 48	842 $\pm$ 51	42 $\pm$ 12	948 $\pm$ 54
Ball in cup, catch	959 $\pm$ 27	460 $\pm$ 380	879 $\pm$ 87	794$\pm$ 58	852 $\pm$ 71	312$\pm$ 63	974 $\pm$ 33
100K 步得分
Finger, spin	767 $\pm$ 56	136 $\pm$ 216	341 $\pm$ 70	740 $\pm$ 64	693 $\pm$ 141	179 $\pm$ 66	811$\pm$46
Cartpole, swingup	582$\pm$146	297$\pm$39	326$\pm$27	311$\pm$11	-	419$\pm$40	835$\pm$22
Reacher, easy	538$\pm$233	20$\pm$50	314$\pm$155	274$\pm$14	-	145$\pm$30	746$\pm$25
Cheetah, run	299 $\pm$48	138$\pm$88	235$\pm$ 137	267$\pm$24	319$\pm$56	197$\pm$15	616$\pm$18
Walker, walk	403$\pm$24	224$\pm$48	277$\pm$12	394$\pm$22	361$\pm$73	42$\pm$12	891$\pm$82
Ball in cup, catch	769 $\pm$ 43	0 $\pm$ 0	246 $\pm$ 174	391$\pm$ 82	512 $\pm$ 110	312$\pm$ 63	746$\pm$91

表 1：CURL（10 个种子的均值与标准差）与基线在 DMControl500k 与 DMControl100k 上的得分。CURL 在 DMControl500k 所评多数（5 出 6）环境上达到当时最佳。环境选取取决于基线是否有数据（我们共在 16 个环境上跑 CURL，结果见图 7）。基线包括 PlaNet（Hafner et al. 2018）、Dreamer（Hafner et al. 2019）、SAC+AE（Yarats et al. 2019）、SLAC（Lee et al. 2019）、像素 SAC 与状态 SAC（Haarnoja et al. 2018）。SLAC 报告了每智能体步 1 次与 3 次梯度更新，分别记为 SLACv1 与 SLACv2；我们与 SLACv1 对比，因其他基线与 CURL 每步仅 1 次更新。我们也跑了每步 3 次更新的 CURL，并与表 5 中 SLACv2 对比。

Game	Human	Random	Rainbow	SimPLe	OTRainbow	Eff. Rainbow	CURL
Alien	7127.7	227.8	318.7	616.9	824.7	739.9	558.2
Amidar	1719.5	5.8	32.5	88.0	82.8	188.6	142.1
Assault	742.0	222.4	231	527.2	351.9	431.2	600.6
Asterix	8503.3	210.0	243.6	1128.3	628.5	470.8	734.5
Bank Heist	753.1	14.2	15.55	34.2	182.1	51.0	131.6
Battle Zone	37187.5	2360.0	2360.0	5184.4	4060.6	10124.6	14870.0
Boxing	12.1	0.1	-24.8	9.1	2.5	0.2	1.2
Breakout	30.5	1.7	1.2	16.4	9.84	1.9	4.9
Chopper Command	7387.8	811.0	120.0	1246.9	1033.33	861.8	1058.5
crazy_climber	35829.4	10780.5	2254.5	62583.6	21327.8	16185.3	12146.5
demon_attack	1971.0	152.1	163.6	208.1	711.8	508.0	817.6
freeway	29.6	0.0	0.0	20.3	25.0	27.9	26.7
frostbite	4334.7	65.2	60.2	254.7	231.6	866.8	1181.3
gopher	2412.5	257.6	431.2	771.0	778.0	349.5	669.3
hero	30826.4	1027.0	487	2656.6	6458.8	6857.0	6279.3
jamesbond	302.8	29.0	47.4	125.3	112.3	301.6	471.0
kangaroo	3035.0	52.0	0.0	323.1	605.4	779.3	872.5
krull	2665.5	1598.0	1468	4539.9	3277.9	2851.5	4229.6
kung_fu_master	22736.3	258.5	0.	17257.2	5722.2	14346.1	14307.8
ms_pacman	6951.6	307.3	67	1480.0	941.9	1204.1	1465.5
Pong	14.6	-20.7	-20.6	12.8	1.3	-19.3	-16.5
Private eye	69571.3	24.9	0	58.3	100.0	97.8	218.4
Qbert	13455.0	163.9	123.46	1288.8	509.3	1152.9	1042.4
Road_Runner	7845.0	11.5	1588.46	5640.6	2696.7	9600.0	5661.0
seaquest	42054.7	68.4	131.69	683.3	286.92	354.1	384.5
Up_n_Down	11693.2	533.4	504.6	3350.3	2847.6	2877.4	2955.2

Table 2: Scores achieved by CURL (coupled with Eff. Rainbow) and baselines on Atari benchmarked at 100k time-steps (Atari100k). CURL achieves state-of-the-art performance on 7 out of 26 environments. Our baselines are SimPLe (Kaiser et al. 2019), OverTrained Rainbow (OTRainbow) (Kielak 2020), Data-Efficient Rainbow (Eff. Rainbow) (van Hasselt et al. 2019), Rainbow (Hessel et al. 2017), Random Agent and Human Performance (Human). We see that CURL implemented on top of Eff. Rainbow improves over Eff. Rainbow on 19 out of 26 games. We also run CURL with 20 random seeds given that this benchmark is susceptible to high variance across multiple runs. We also see that CURL achieves superhuman performance on JamesBond and Krull.

游戏	Human	Random	Rainbow	SimPLe	OTRainbow	Eff. Rainbow	CURL
Alien	7127.7	227.8	318.7	616.9	824.7	739.9	558.2
Amidar	1719.5	5.8	32.5	88.0	82.8	188.6	142.1
Assault	742.0	222.4	231	527.2	351.9	431.2	600.6
Asterix	8503.3	210.0	243.6	1128.3	628.5	470.8	734.5
Bank Heist	753.1	14.2	15.55	34.2	182.1	51.0	131.6
Battle Zone	37187.5	2360.0	2360.0	5184.4	4060.6	10124.6	14870.0
Boxing	12.1	0.1	-24.8	9.1	2.5	0.2	1.2
Breakout	30.5	1.7	1.2	16.4	9.84	1.9	4.9
Chopper Command	7387.8	811.0	120.0	1246.9	1033.33	861.8	1058.5
crazy_climber	35829.4	10780.5	2254.5	62583.6	21327.8	16185.3	12146.5
demon_attack	1971.0	152.1	163.6	208.1	711.8	508.0	817.6
freeway	29.6	0.0	0.0	20.3	25.0	27.9	26.7
frostbite	4334.7	65.2	60.2	254.7	231.6	866.8	1181.3
gopher	2412.5	257.6	431.2	771.0	778.0	349.5	669.3
hero	30826.4	1027.0	487	2656.6	6458.8	6857.0	6279.3
jamesbond	302.8	29.0	47.4	125.3	112.3	301.6	471.0
kangaroo	3035.0	52.0	0.0	323.1	605.4	779.3	872.5
krull	2665.5	1598.0	1468	4539.9	3277.9	2851.5	4229.6
kung_fu_master	22736.3	258.5	0.	17257.2	5722.2	14346.1	14307.8
ms_pacman	6951.6	307.3	67	1480.0	941.9	1204.1	1465.5
Pong	14.6	-20.7	-20.6	12.8	1.3	-19.3	-16.5
Private eye	69571.3	24.9	0	58.3	100.0	97.8	218.4
Qbert	13455.0	163.9	123.46	1288.8	509.3	1152.9	1042.4
Road_Runner	7845.0	11.5	1588.46	5640.6	2696.7	9600.0	5661.0
seaquest	42054.7	68.4	131.69	683.3	286.92	354.1	384.5
Up_n_Down	11693.2	533.4	504.6	3350.3	2847.6	2877.4	2955.2

表 2：CURL（与 Eff. Rainbow 结合）与基线在 Atari100k（100k 时间步）上的得分。CURL 在 26 个环境中的 7 个达到当时最佳。基线为 SimPLe（Kaiser et al. 2019）、OverTrained Rainbow（OTRainbow）（Kielak 2020）、Data-Efficient Rainbow（Eff. Rainbow）（van Hasselt et al. 2019）、Rainbow（Hessel et al. 2017）、随机智能体与人类表现（Human）。可见建在 Eff. Rainbow 上的 CURL 在 26 款游戏中的 19 款超过 Eff. Rainbow。因该基准跨多次运行方差很大，我们用 20 个随机种子跑 CURL。CURL 还在 JamesBond 与 Krull 上达到超人类表现。

5.2 Environments

5.2 环境

Our primary goal for CURL is sample-efficient control from pixels that is broadly applicable across a range of environments. We benchmark the performance of CURL for both discrete and continuous control environments. Specifically, we focus on DMControl suite for continuous control tasks and the Atari Games benchmark for discrete control tasks with inputs being raw pixels rendered by the environments.

CURL 的主要目标是从像素做样本高效控制，并能广泛适用于多种环境。我们在离散与连续控制环境上评测：连续控制用 DMControl 套件，离散控制用 Atari Games 基准，输入均为环境渲染的原始像素。

DeepMind Control: Recently, there have been a number of papers that have benchmarked for sample efficiency on challenging visual continuous control tasks belonging to the DMControl suite (Tassa et al. 2018) where the agent operates purely from pixels. The reason for operating in these environments is multi fold: (i) they present a reasonably challenging and diverse set of tasks; (ii) sample-efficiency of pure model-free RL algorithms operating from pixels on these benchmarks is poor; (iii) multiple recent efforts to improve the sample efficiency of both model-free and model-based methods on these benchmarks thereby giving us sufficient baselines to compare against; (iv) performance on the DM control suite is relevant to robot learning in real world benchmarks.

DeepMind Control： 近期多篇工作在 DMControl 套件（Tassa et al. 2018）中具有挑战的视觉连续控制任务上评测样本效率，智能体纯从像素运作。选这些环境的理由包括：（i）任务集合有一定难度与多样性；（ii）纯无模型像素 RL 样本效率差；（iii）近期有大量提升基于模型/无模型样本效率的工作，便于对比；（iv）DMControl 表现与真实世界机器人学习相关。

We run experiments on sixteen environments from DMControl to examine the performance of CURL on pixels relative to SAC with access to the ground truth state, shown in Figure 7. For more extensive benchmarking, we compare CURL to five leading pixel-based methods across the the six environments presented in Yarats et al. 2019: ball-in-cup, finger-spin, reacher-easy, cheetah-run, walker-walk, cartpole-swingup for benchmarking.

我们在 16 个 DMControl 环境上实验，考察相对可访问真值状态的 SAC，CURL 在像素上的表现，见图 7。更细对比上，按 Yarats et al. 2019 给出的六个环境（ball-in-cup、finger-spin、reacher-easy、cheetah-run、walker-walk、cartpole-swingup）与五种领先像素方法比较。

Atari: Similar to DMControl sample-efficiency benchmarks, there have been a number of recent papers that have benchmarked for sample-efficiency on the Atari 2600 Games. Kaiser et al. 2019 proposed comparing various algorithms in terms of performance achieved within $100$K timesteps ($400$K frames, frame skip of $4$) of interaction with the environments (games). The method proposed by Kaiser et al. 2019 called SimPLe is a model-based RL algorithm. SimPLe is compared to a random agent, model-free Rainbow DQN (Hessel et al. 2017) and human performance for the same amount of interaction time. Recently, van Hasselt et al. 2019 and Kielak 2020 proposed data-efficient versions of Rainbow DQN which are competitive with SimPLe on the same benchmark. Given that the same benchmark has been established in multiple recent papers and that there is a human baseline to compare to, we benchmark CURL on all the 26 Atari Games (Table 2).

Atari： 类似地，近期多篇工作在 Atari 2600 上评测样本效率。Kaiser et al. 2019 建议比较各算法在与环境交互 $100$K 时间步（$400$K 帧、帧跳跃 $4$）内的表现；其提出的 SimPLe 是基于模型的 RL。SimPLe 与随机智能体、无模型 Rainbow DQN（Hessel et al. 2017）及同等交互时间下的人类表现对比。van Hasselt et al. 2019 与 Kielak 2020 又提出与 SimPLe 竞争的数据高效 Rainbow。鉴于该基准已被多篇采用且有人类基线，我们在全部 26 款 Atari 游戏上评测 CURL（表 2）。

5.3 Baselines for benchmarking sample efficiency

5.3 样本效率基准的基线

DMControl baselines: We present a number of baselines for continuous control within the DMControl suite: (i) SAC-AE (Yarats et al. 2019) where the authors attempt to use a $\beta$-VAE (Higgins et al. 2017), VAE (Kingma & Welling 2013) and a regualrized autoencoder Vincent et al. 2008; Ghosh et al. 2019 jointly with SAC; (ii) SLAC (Lee et al. 2019) which learns a latent space world model on top of VAE features Ha & Schmidhuber 2018 and builds value functions on top; (iii) PlaNet and (iv) Dreamer (Hafner et al. 2018; Hafner et al. 2019) both of which learn a latent space world model and explicitly plan through it; (v) Pixel SAC: Vanilla SAC operating purely from pixels (Haarnoja et al. 2018). These baselines are competitive methods for benchmarking control from pixels. In addition to these, we also present the baseline State-SAC where the assumption is that the agent has access to low level state based features and does not operate from pixels. This baseline acts as an oracle in that it approximates the upper bound of how sample-efficient a pixel-based agent can get in these environments.

DMControl 基线： 我们给出套件内连续控制的若干基线：（i）SAC-AE（Yarats et al. 2019），作者尝试把 $\beta$-VAE（Higgins et al. 2017）、VAE（Kingma & Welling 2013）与正则自编码器（Vincent et al. 2008；Ghosh et al. 2019）与 SAC 联合；（ii）SLAC（Lee et al. 2019）在 VAE 特征上建隐空间世界模型并在其上建价值函数；（iii）PlaNet 与（iv）Dreamer（Hafner et al. 2018；Hafner et al. 2019）均学隐空间世界模型并显式规划；（v）Pixel SAC：纯像素上的原版 SAC（Haarnoja et al. 2018）。这些都是像素控制的有竞争力方法。另外给出 State-SAC：假设智能体可访问低层状态特征、不从像素运作，作为预言机上界，近似像素智能体在这些环境上能达到的样本效率上限。

Atari baselines: For benchmarking performance on Atari, we compare CURL to (i) SimPLe (Kaiser et al. 2019), the top performing model-based method in terms of data-efficiency on Atari and (ii) Rainbow DQN (Hessel et al. 2017), a top-performing model-free baseline for Atari, (iii) OTRainbow (Kielak 2020) which is an OverTrained version of Rainbow for data-efficiency, (iv) Efficient Rainbow (van Hasselt et al. 2019) which is a modification of Rainbow hyperparameters for data-efficiency, (v) Random Agent (Kaiser et al. 2019), (vi) Human Performance (Kaiser et al. 2019; van Hasselt et al. 2019). All the baselines and our method are evaluated for performance after 100K interaction steps (400K frames with a frame skip of 4) which corresponds to roughly two hours of gameplay. These benchmarks help us understand how the state-of-the-art pixel based RL algorithms compare in terms of sample efficiency and also to human efficiency. Note: Scores for SimPLe and Human baselines have been reported differently in prior work (Kielak 2020; van Hasselt et al. 2019). To be rigorous, we take the best reported score for each individual game reported in prior work.

Atari 基线： 对比（i）SimPLe（Kaiser et al. 2019）——Atari 数据效率上表现最好的基于模型方法；（ii）Rainbow DQN（Hessel et al. 2017）——Atari 上表现突出的无模型基线；（iii）面向数据效率的 OverTrained Rainbow（OTRainbow）（Kielak 2020）；（iv）修改超参的 Efficient Rainbow（van Hasselt et al. 2019）；（v）随机智能体（Kaiser et al. 2019）；（vi）人类表现（Kaiser et al. 2019；van Hasselt et al. 2019）。所有方法在 100K 交互步（帧跳跃 4 时 400K 帧，约两小时游戏）后评测，以比较前沿像素 RL 相对人类的样本效率。注意： 先前工作对 SimPLe 与人类基线的报告不完全一致（Kielak 2020；van Hasselt et al. 2019）；为严谨起见，我们对每款游戏取先前工作中报告的最佳分数。

6 Results

6 结果

6.1 DMControl

Sample-efficiency results for DMControl experiments are shown in Table 1 and in Figures 4, 6, and 7. Below are the key findings:

DMControl 实验的样本效率结果见表 1 与图 4、6、7。主要发现如下：

(i) CURL is the state-of-the-art image-based RL algorithm on the majority (5 out of 6) DMControl environments that we benchmark on for sample-efficiency against existing pixel-based baselines. On DMControl100k, CURL achieves 1.9x higher median performance than Dreamer (Hafner et al. 2019), a leading model-based method, and is 4.5x more data-efficient shown in Figure 6.

（i）在所评样本效率的多数（5 出 6）DMControl 环境上，CURL 是相对既有像素基线的当时最佳图像 RL 算法。在 DMControl100k 上，CURL 相对领先基于模型方法 Dreamer（Hafner et al. 2019）取得 1.9× 更高中位性能，数据效率约 4.5×（见图 6）。

(ii) CURL operating purely from pixels nearly matches (and sometimes surpasses) the sample efficiency of SAC operating from state on the majority of 16 DMControl environments tested shown in Figure 7 and matches the median state-based score on DMControl500k shown in Figure 4. This is a first for any image-based RL algorithm, be it model-based, model-free, with or without auxiliary tasks.

（ii）纯像素运作的 CURL 在所测 16 个 DMControl 环境的多数上几乎对齐（有时甚至超过）基于状态的 SAC 的样本效率（图 7），并在 DMControl500k 上对齐状态方法的中位分数（图 4）。这对任何图像 RL 算法（不论基于模型/无模型、有无辅助任务）都是首次。

(iii) CURL solves (converges close to optimal score of 1000) on the majority of 16 DMControl experiments within 500k steps. It also matches the state-based median score across the 6 extensively benchmarked environments in this regime.

（iii）CURL 在 500k 步内于多数 16 个 DMControl 实验上求解（收敛到接近最优分 1000），并在该区间对齐六个重点评测环境上的状态中位分数。

6.2 Atari

Results for Atari100k are shown in Table 2. Below are the key findings:

Atari100k 结果见表 2。主要发现如下：

(i) CURL achieves a median human-normalized score (HNS) of 17.5% while SimPLe and Efficient Rainbow DQN achieve 14.4% and 16.1% respectively. The mean HNS is 38.1%, 44.3%, and 28.5% for CURL, SimPLe, and Efficient Rainbow DQN respectively.

（i）CURL 的中位人类归一化分数（HNS）为 17.5%，SimPLe 与 Efficient Rainbow DQN 分别为 14.4% 与 16.1%。三者平均 HNS 分别为 38.1%、44.3% 与 28.5%。

(ii) CURL improves on top of Efficient Rainbow on 19 out of 26 Atari games. Averaged across 26 games, CURL improves on top of Efficient Rainbow by 1.3x, while the median performance improvement over SimPLE and Efficient Rainbow are 1.2x and 1.1x respectively.

（ii）CURL 在 26 款 Atari 游戏中的 19 款上超过 Efficient Rainbow。对 26 款游戏平均，CURL 相对 Efficient Rainbow 提升 1.3×；相对 SimPLe 与 Efficient Rainbow 的中位提升分别为 1.2× 与 1.1×。

(iii) CURL surpasses human performance on two games JamesBond (1.6 HNS), Krull (2.5 HNS).

（iii）CURL 在 JamesBond（1.6 HNS）与 Krull（2.5 HNS）两款游戏上超过人类表现。

Figure 4: Performance of CURL coupled to SAC averaged across 10 seeds relative to SLACv1, PlaNet, Pixel SAC and State SAC baselines. At the 500k benchmark CURL matches the median score of state-based SAC. At 100k environment steps CURL achieves a 1.9x higher median score than Dreamer. For a direct comparison, we only compute the median across the 6 environments in 1 (4 for SLAC) and show learning curves for CURL across 16 DMControl experiments in 7.

图 4：CURL 与 SAC 结合、在 10 个种子上平均后，相对 SLACv1、PlaNet、Pixel SAC 与 State SAC 的表现。500k 基准上 CURL 对齐基于状态 SAC 的中位分数；100k 环境步上 CURL 相对 Dreamer 取得 1.9× 更高中位分数。为直接对比，中位仅对表 1 中的 6 个环境计算（SLAC 为 4 个）；CURL 在 16 个 DMControl 实验上的学习曲线见图 7。

7 Ablation Studies

7 消融研究

In Appendix E, we present the results of ablation studies carried out to answer the following questions: (i) Does CURL learn only visual features or does it also capture temporal dynamics of the environment? (ii) How well does the RL policy perform if CURL representations are learned solely with the contrastive objective and no signal from RL? (iii) Why does CURL match state-based RL performance on some DMControl environments but not on others?

附录 E 给出消融，回答：（i）CURL 是否只学视觉特征，还是也捕获环境时间动态？（ii）若表征仅用对比目标、无 RL 信号，RL 策略表现如何？（iii）为何 CURL 在部分 DMControl 环境对齐状态 RL，在另一些上却不能？

8 Conclusion

8 结论

In this work, we proposed CURL, a contrastive unsupervised representation learning method for RL, that achieves state-of-the-art data-efficiency on pixel-based RL tasks across a diverse set of benchmark environments. CURL is the first model-free RL pipeline accelerated by contrastive learning with minimal architectural changes to demonstrate state-of-the-art performance on complex tasks so far dominated by approaches that have relied on learning world models and (or) decoder-based objectives. We hope that progress like CURL enables avenues for real-world deployment of RL in areas like robotics where data-efficiency is paramount.

本文提出面向 RL 的对比无监督表征学习方法 CURL，在多样基准环境的像素 RL 任务上达到当时最佳数据效率。CURL 是首个以最少结构改动、用对比学习加速的无模型 RL 管线，在此前由世界模型与（或）基于解码器目标主导的复杂任务上展示当时最佳表现。我们希望 CURL 一类进展能为机器人等数据效率至关重要的真实世界 RL 部署打开空间。

9 Acknowledgements

9 致谢

This research is supported in part by DARPA through the Learning with Less Labels (LwLL) Program and by ONR through PECASE N000141612723. We also thank Wendy Shang for her help with Section E.4; Zak Stone and Google TFRC for cloud credits; Danijar Hafner, Alex Lee, and Denis Yarats for sharing data for baselines; and Lerrel Pinto, Adam Stooke, Will Whitney, and Ankesh Anand for insightful discussions.

本研究部分得到 DARPA Learning with Less Labels（LwLL）计划以及 ONR PECASE N000141612723 支持。感谢 Wendy Shang 对第 E.4 节的帮助；感谢 Zak Stone 与 Google TFRC 提供云积分；感谢 Danijar Hafner、Alex Lee 与 Denis Yarats 分享基线数据；感谢 Lerrel Pinto、Adam Stooke、Will Whitney 与 Ankesh Anand 的深入讨论。

References

参考文献

Anand et al. (2019) Anand, A., Racah, E., Ozair, S., Bengio, Y., Côté, M.-A., and Hjelm, R. D. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems , pp. 8766–8779, 2019.

Bachman et al. (2019) Bachman, P., Hjelm, R. D., and Buchwalter, W. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems , pp. 15509–15519, 2019.

Bellemare et al. (2013) Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253–279, 2013.

Bellemare et al. (2017) Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pp. 449–458. JMLR. org, 2017.

Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations, 2020.

Chopra et al. (2005) Chopra, S., Hadsell, R., and LeCun, Y. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) , volume 1, pp. 539–546. IEEE, 2005.

Cubuk et al. (2019) Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmentation with a reduced search space, 2019.

Devlin et al. (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Dwibedi et al. (2018) Dwibedi, D., Tompson, J., Lynch, C., and Sermanet, P. Learning actionable representations from visual observations. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 1577–1584. IEEE, 2018.

Espeholt et al. (2018) Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561 , 2018.

Fortunato et al. (2017) Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295 , 2017.

Ghosh et al. (2019) Ghosh, P., Sajjadi, M. S. M., Vergari, A., Black, M., and Schölkopf, B. From variational to deterministic autoencoders, 2019.

Ha & Schmidhuber (2018) Ha, D. and Schmidhuber, J. World models. arXiv preprint arXiv:1803.10122 , 2018.

Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Hadsell et al. (2006) Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) , volume 2, pp. 1735–1742. IEEE, 2006.

Hafner et al. (2018) Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551 , 2018.

Hafner et al. (2019) Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 , 2019.

He et al. (2019a) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 , 2019a.

He et al. (2019b) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. 2019b.

Hénaff et al. (2019) Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S., and Oord, A. v. d. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272 , 2019.

Henderson et al. (2018) Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Hessel et al. (2017) Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning, 2017.

Higgins et al. (2017) Higgins, I., Pal, A., Rusu, A., Matthey, L., Burgess, C., Pritzel, A., Botvinick, M., Blundell, C., and Lerchner, A. Darla: Improving zero-shot transfer in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pp. 1480–1490. JMLR. org, 2017.

Hjelm et al. (2018) Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2018.

Jaderberg et al. (2016) Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 , 2016.

Jaderberg et al. (2019) Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castaneda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science , 364(6443):859–865, 2019.

Kaiser et al. (2019) Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374 , 2019.

Kalashnikov et al. (2018) Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293 , 2018.

Kielak (2020) Kielak, K. Do recent advancements in model-based deep reinforcement learning really improve data efficiency?, 2020.

Kingma & Welling (2013) Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Kostrikov et al. (2020) Kostrikov, I., Yarats, D., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 , 2020.

Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 25 , pp. 1097–1105. Curran Associates, Inc., 2012.

Lake et al. (2017) Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J. Building machines that learn and think like people. Behavioral and brain sciences , 40, 2017.

Laskin et al. (2020) Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data. arXiv preprint arXiv:2004.14990 , 2020.

LeCun et al. (2006) LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F. A tutorial on energy-based learning. 2006.

Lee et al. (2019) Lee, A. X., Nagabandi, A., Abbeel, P., and Levine, S. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv:1907.00953 , 2019.

Lillicrap et al. (2015) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Mirowski et al. (2016) Mirowski, P., Pascanu, R., Viola, F., Soyer, H., Ballard, A. J., Banino, A., Denil, M., Goroshin, R., Sifre, L., Kavukcuoglu, K., et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.03673 , 2016.

Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529–533, 2015.

Schaul et al. (2015) Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. arXiv preprint arXiv:1511.05952 , 2015.

Schmidhuber (1990) Schmidhuber, J. Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments. Technical Report FKI-126-90, TUM , 1990.

Schrittwieser et al. (2019) Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265 , 2019.

Schroff et al. (2015) Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 815–823, 2015.

Sermanet et al. (2018) Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., and Brain, G. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pp. 1134–1141. IEEE, 2018.

Shelhamer et al. (2016) Shelhamer, E., Mahmoudieh, P., Argus, M., and Darrell, T. Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint arXiv:1612.07307 , 2016.

Srinivas et al. (2018) Srinivas, A., Jabri, A., Abbeel, P., Levine, S., and Finn, C. Universal planning networks. arXiv preprint arXiv:1804.00645 , 2018.

Sutton (1990) Sutton, R. S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990 , pp. 216–224. Elsevier, 1990.

Sutton et al. (1998) Sutton, R. S. et al. Introduction to reinforcement learning , volume 135. 1998.

Szegedy et al. (2015) Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Computer Vision and Pattern Recognition (CVPR) , 2015. URL http://arxiv.org/abs/1409.4842 .

Tassa et al. (2018) Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Tian et al. (2019) Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. arXiv preprint arXiv:1906.05849 , 2019.

Tschannen et al. (2019) Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., and Lucic, M. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625 , 2019.

van den Oord et al. (2018) van den Oord, A., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Van Hasselt et al. (2016) Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In Thirtieth AAAI conference on artificial intelligence , 2016.

van Hasselt et al. (2019) van Hasselt, H. P., Hessel, M., and Aslanides, J. When to use parametric models in reinforcement learning? In Advances in Neural Information Processing Systems , pp. 14322–14333, 2019.

Vincent et al. (2008) Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning , pp. 1096–1103, 2008.

Wang & Gupta (2015) Wang, X. and Gupta, A. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision , pp. 2794–2802, 2015.

Wang et al. (2015) Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., and De Freitas, N. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581 , 2015.

Warde-Farley et al. (2018) Warde-Farley, D., Van de Wiele, T., Kulkarni, T., Ionescu, C., Hansen, S., and Mnih, V. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv:1811.11359 , 2018.

Wu et al. (2018) Wu, Z., Xiong, Y., Yu, S., and Lin, D. Unsupervised feature learning via non-parametric instance-level discrimination. arXiv preprint arXiv:1805.01978 , 2018.

Yarats et al. (2019) Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., and Fergus, R. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741 , 2019.

Appendix A Implementation Details

附录 A 实现细节

Below, we explain the implementation details for CURL in the DMControl setting. Specifically, we use the SAC algorithm as the RL objective coupled with CURL and build on top of the publicly released implementation from Yarats et al. 2019. We present in detail the hyperparameters for the architecture and optimization. We do not use any extra hyperparameter for balancing the contrastive loss and the reinforcement learning losses. Both the objectives are weighed equally in the gradient updates.

下面说明 DMControl 设定下 CURL 的实现细节。具体以 SAC 为 RL 目标并与 CURL 结合，基于 Yarats et al. 2019 公开实现。我们详列结构与优化超参。对比损失与强化学习损失之间不设额外平衡超参，梯度更新中两者等权。

Hyperparameter	Value
Random crop	True
Observation rendering	$(100,100)$
Observation downsampling	$(84,84)$
Replay buffer size	$100000$
Initial steps	$1000$
Stacked frames	$3$
Action repeat	$2$ finger, spin; walker, walk
	$8$ cartpole, swingup
	$4$ otherwise
Hidden units (MLP)	$1024$
Evaluation episodes	$10$
Optimizer	Adam
$(\beta_{1},\beta_{2})\rightarrow(f_{\theta},\pi_{\psi},Q_{\phi})$	$(.9,.999)$
$(\beta_{1},\beta_{2})\rightarrow(\alpha)$	$(.5,.999)$
Learning rate $(f_{\theta},\pi_{\psi},Q_{\phi})$	$2e-4$ cheetah, run
	$1e-3$ otherwise
Learning rate ($\alpha$)	$1e-4$
Batch Size	$512$
$Q$ function EMA $\tau$	$0.01$
Critic target update freq	$2$
Convolutional layers	$4$
Number of filters	$32$
Non-linearity	ReLU
Encoder EMA $\tau$	$0.05$
Latent dimension	$50$
Discount $\gamma$	$.99$
Initial temperature	$0.1$

Table 3: Hyperparameters used for DMControl CURL experiments. Most hyperparameters values are unchanged across environments with the exception for action repeat, learning rate, and batch size.

超参数	取值
随机裁剪	True
观测渲染	$(100,100)$
观测下采样	$(84,84)$
回放缓冲大小	$100000$
初始步数	$1000$
堆叠帧数	$3$
动作重复	$2$ finger, spin; walker, walk
	$8$ cartpole, swingup
	$4$ otherwise
隐藏单元（MLP）	$1024$
评测回合	$10$
优化器	Adam
$(\beta_{1},\beta_{2})\rightarrow(f_{\theta},\pi_{\psi},Q_{\phi})$	$(.9,.999)$
$(\beta_{1},\beta_{2})\rightarrow(\alpha)$	$(.5,.999)$
学习率 $(f_{\theta},\pi_{\psi},Q_{\phi})$	$2e-4$ cheetah, run
	$1e-3$ otherwise
学习率 ($\alpha$)	$1e-4$
批大小	$512$
$Q$ function EMA $\tau$	$0.01$
评论家目标更新频率	$2$
卷积层数	$4$
滤波器数	$32$
非线性	ReLU
编码器 EMA $\tau$	$0.05$
隐变量维数	$50$
折扣 $\gamma$	$.99$
初始温度	$0.1$

表 3：DMControl CURL 实验超参。多数取值跨环境不变，例外是动作重复、学习率与批大小。

Architecture: We use an encoder architecture that is similar to (Yarats et al. 2019), which we sketch in PyTorch-like pseuodocode below. The actor and critic both use the same encoder to embed image observations. A full list of hyperparameters is displayed in Table 3.

结构： 编码器结构类似 Yarats et al. 2019，下面用类 PyTorch 伪代码简述。演员与评论家共用同一编码器嵌入图像观测。完整超参见表 3。

For contrastive learning, CURL utilizes momentum for the key encoder (He et al. 2019b) and a bi-linear inner product as the similarity measure (van den Oord et al. 2018). Performance curves ablating these two architectural choices are shown in Figure 5.

对比学习上，CURL 对键编码器使用动量（He et al. 2019b），相似度用双线性内积（van den Oord et al. 2018）。这两项结构选择的性能曲线消融见图 5。

Figure 5: Performance on cheetah-run environment ablated two-ways: (left) using the query encoder or exponentially moving average of the query encoder for encoding keys (right) using the bi-linear inner product as in (van den Oord et al. 2018) or the cosine inner product as in He et al. 2019b; Chen et al. 2020

图 5：在 cheetah-run 环境上的双向消融：（左）用查询编码器或其指数滑动平均编码键；（右）用 van den Oord et al. 2018 的双线性内积，或 He et al. 2019b；Chen et al. 2020 的余弦内积。

Pseudo-code for the architecture is provided below:

结构伪代码如下：

def encode(x,z_dim):
"""
␣␣␣␣ConvNet␣encoder
␣␣␣␣args:
␣␣␣␣␣␣␣␣B-batch_size,␣C-channels
␣␣␣␣␣␣␣␣H,W-spatial_dims
␣␣␣␣␣␣␣␣x␣:␣shape␣:␣[B,␣C,␣H,␣W]
␣␣␣␣␣␣␣␣C␣=␣3␣*␣num_frames;␣3␣-␣R/G/B
␣␣␣␣␣␣␣␣z_dim:␣latent␣dimension
␣␣␣␣"""

x = x / 255.

c: channels, f: filters

k: kernel, s: stride

z = Conv2d(c=x.shape[1], f=32, k=3, s=2)])(x)
z = ReLU(z)

for _ in range(num_layers - 1):
z = Conv2d((c=32, f=32, k=3, s=1))(z)
z = ReLU(z)

z = flatten(z)

in: input dim, out: output_dim, h: hiddens

z = mlp(in=z.size(),out=z_dim,h=1024)
z = LayerNorm(z)
z = tanh(z)

Terminology: A common point of confusion is the meaning “training steps.” We use the term environment steps to denote the amount of times the simulator environment is stepped through and interaction steps to denote the number of times the agent steps through its policy. The terms action repeat or frame skip refer to the number of times an action is repeated when it’s drawn from the agent’s policy. For example, if action repeat is set to 4, then 100k interaction steps is equivalent to 400k environment steps.

术语： 「训练步」易混淆。我们用 环境步 表示仿真器被步进的次数，用 交互步 表示智能体按其策略步进的次数。动作重复 或 帧跳跃 指从策略抽出动作后重复执行的次数。例如动作重复为 4 时，100k 交互步等价于 400k 环境步。

Batch Updates: After initializing the replay buffer with observations extracted by a random agent, we sample a batch of observations, compute the CURL objectives, and step through the optimizer. Note that since queries and keys are generated by data-augmenting an observation, we can generate arbitrarily many keys to increase the contrastive batch size without sampling any additional observations.

批次更新： 用随机智能体采集的观测初始化回放缓冲后，采样一批观测，计算 CURL 目标并步进优化器。注意查询与键由对观测做数据增强得到，因而可不额外采样观测、任意增大对比批大小以增加键的数量。

Shared Representations: The objective of performing contrastive learning together with RL is to ensure that the shared encoder learns rich features that facilitate sample efficient control. There is a subtle coincidental connection between MoCo and off-policy RL. Both the frameworks adopt the usage of a momentum averaged (EMA) version of the underlying model. In MoCo, the EMA encoder is used for encoding the keys (targets) while in off-policy RL, the EMA version of the Q-networks are used as targets in the Bellman error (Mnih et al. 2015; Haarnoja et al. 2018). Thanks to this connection, CURL shares the convolutional encoder, momentum coefficient and EMA update between contrastive and reinforcement learning updates for the shared parameters. The MLP part of the critic that operates on top of these convolutional features has a separate momentum coefficient and update decoupled from the image encoder parameters.

共享表征： 与 RL 一起做对比学习，是为了让共享编码器学到利于样本高效控制的丰富特征。MoCo 与离策略 RL 有一处巧合联系：二者都对底层模型使用动量平均（EMA）版本。MoCo 用 EMA 编码器编码键（目标）；离策略 RL 用 Q 网络的 EMA 版本作贝尔曼误差目标（Mnih et al. 2015；Haarnoja et al. 2018）。借助这一联系，CURL 在对比与强化学习更新之间共享卷积编码器、动量系数与对共享参数的 EMA 更新。评论家中作用于这些卷积特征之上的 MLP 部分，另有独立于图像编码器参数的动量系数与更新。

Balancing Contrastive and RL Updates: While past work has learned hyperparameters to balance the auxiliary loss coefficient or learning rate relative to the RL objective (Jaderberg et al. 2016; Yarats et al. 2019), CURL does not need any such adjustments. We use both the contrastive and RL objectives together with equal weight and learning rate. This simplifies the training process compared to other methods, such as training a VAE jointly (Hafner et al. 2018; Hafner et al. 2019; Lee et al. 2019), that require careful tuning of coefficients for representation learning.

平衡对比与 RL 更新： 既往工作常学习相对 RL 目标的辅助损失系数或学习率（Jaderberg et al. 2016；Yarats et al. 2019），CURL 无需此类调整。对比与 RL 目标等权、同学习率一起优化，相对联合训 VAE 等方法（Hafner et al. 2018；Hafner et al. 2019；Lee et al. 2019）简化了训练，后者常需仔细调表征学习系数。

Differences in Data Collection between Computer Vision and RL Settings: There are two key differences between contrastive learning in the computer vision and RL settings because of their different goals. Unsupervised feature learning methods built for downstream vision tasks like image classification assume a setting where there is a large static dataset of unlabeled images. On the other hand, in RL, the dataset changes over time to account for the agent’s new experiences. Secondly, the size of the memory bank of labeled images and dataset of unlabeled ones in vision-based settings are 65K and 1M (or 1B) respectively. The goal in vision-based methods is to learn from millions of unlabeled images. On the other hand, the goal in CURL is to develop sample-efficient RL algorithms. For example, to be able to solve a task within 100K timesteps (approximately 2 hours in real-time), an agent can only ingest 100K image frames.

计算机视觉与 RL 设定下数据采集的差异： 目标不同导致对比学习在两类设定下有两点关键差别。服务图像分类等下游视觉任务的无监督特征学习，假定存在大型静态未标注图像集；RL 中数据集随时间变化以纳入智能体新经验。其次，视觉设定中有标签记忆库与未标注集规模常为 65K 与 1M（或 1B），目标是从百万级未标注图像学习；CURL 的目标则是样本高效 RL——例如要在 100K 时间步（约实时 2 小时）内解任务，智能体最多摄入 100K 图像帧。

Therefore, unlike MoCo, CURL does not use a memory bank for contrastive learning. Instead, the negatives are constructed on the fly for every minibatch sampled from the agent’s replay buffer for an RL update similar to SimCLR. The exact implementation is provided as a PyTorch-like code snippet in 4.7.

因此与 MoCo 不同，CURL 对比学习不使用记忆库；负样本在每次为 RL 更新从智能体回放缓冲采样的小批量上即时构造，类似 SimCLR。具体实现见第 4.7 节类 PyTorch 代码片段。

Data Augmentation:

数据增强：

Random crop data augmentation has been crucial for the performance of deep learning based computer vision systems in object recognition, detection and segmentation (Krizhevsky et al. 2012; Szegedy et al. 2015; Cubuk et al. 2019; Chen et al. 2020). However, similar augmentation methods have not seen much adoption in the field of RL even though several benchmarks use raw pixels as inputs to the model.

随机裁剪数据增强对基于深度学习的目标识别、检测与分割至关重要（Krizhevsky et al. 2012；Szegedy et al. 2015；Cubuk et al. 2019；Chen et al. 2020）。但在 RL 领域，即便多个基准以原始像素为模型输入，类似增强仍少见采用。

CURL adopts the random crop data augmentation as the stochastic data augmentation applied to a frame stack. To make it easier for the model to correlate spatio-temporal patterns in the input, we apply the same random crop (in terms of box coordinates) across all four frames in the stack as opposed to extracting different random crop positions from each frame in the stack. Further, unlike in computer vision systems where the aspect ratio for random crop is allowed to be as low as 0.08, we preserve much of the spatial information as possible and use a constant aspect ratio of 0.84 between the original and cropped. In our experiments, data augmented samples for CURL are formed by cropping $84\times 84$ frames from an input frame of $100\times 100$.

CURL 把随机裁剪作为施加于帧堆叠的随机数据增强。为便于模型关联输入中的时空模式，我们对堆叠中全部四帧施加相同随机裁剪（框坐标一致），而非从每帧抽取不同位置。此外，视觉系统允许随机裁剪宽高比低至 0.08，我们尽量保留空间信息，固定原图与裁剪图宽高比为 0.84。实验中，CURL 的增强样本由从 $100\times 100$ 输入帧裁出 $84\times 84$ 得到。

DMControl: We render observations at $100\times 100$ and randomly crop $84\times 84$ frames. For evaluation, we render observations at $100\times 100$ and center crop to $84\times 84$ pixels. We found that implementing random crop efficiently was extremely important to the success of the algorithm. We provide pseudocode below:

DMControl： 以 $100\times 100$ 渲染观测，随机裁出 $84\times 84$；评测时 $100\times 100$ 渲染后中心裁到 $84\times 84$。我们发现高效实现随机裁剪对算法成功极为重要。伪代码如下：

from skimage import view_as_windows
import numpy as np

def random_crop(imgs, out):
"""
␣␣␣␣Vectorized␣random␣crop
␣␣␣␣args:
␣␣␣␣␣␣␣␣imgs:␣shape␣(B,C,H,W)
␣␣␣␣␣␣␣␣out:␣output␣size␣(e.g.␣84)
␣␣␣␣"""

n: batch size.

n = imgs.shape[0]
img_size = imgs.shape[-1] # e.g. 100
crop_max = img_size - out

imgs = np.transpose(imgs, (0, 2, 3, 1))

w1 = np.random.randint(0, crop_max, n)
h1 = np.random.randint(0, crop_max, n)

creates all sliding window

combinations of size (out)

windows = view_as_windows(
imgs, (1, out, out, 1))[..., 0,:,:, 0]

selects a random window

for each batch element

cropped = windows[np.arange(n), w1, h1]
return cropped

Appendix B Atari100k Implementation Details

附录 B Atari100k 实现细节

The flexibility of CURL allows us to apply it to discrete control setting with minimal modifications. Similar to our rationale for picking SAC as the baseline RL algorithm to couple CURL with (for continuous control), we pick the data-efficient version of Rainbow DQN (Efficient Rainbow) (van Hasselt et al. 2019) for Atari100K which performs competitively with an older version of SimPLe (most recent version has improved numbers). In order to understand specifically what the gains from CURL are without any other changes, we adopt the exact same hyperparameters specified in the paper (van Hasselt et al. 2019) (including a modified convolutional encoder that uses larger kernel size and stride of 5). We present the details in Table 4. Similar to DMControl, the contrastive objective and the RL objective are weighted equally for learning (except for Pong, Freeway, Boxing and PrivateEye for which we used a coefficient of $0.05$ for the momentum contastive loss. On a large majority (22 out of 26) of the games, we do not use this adjustment. While it is standard practice to use the same hyperparameters for all games in Atari, papers proposing auxiliary losses have adopted a different practice of using game specific coefficients (Jaderberg et al. 2016).). We use the Efficient Rainbow codebase from https://github.com/Kaixhin/Rainbow which has a reproduced version of van Hasselt et al. 2019. We evaluate with $20$ random seeds and report the mean score for each game given the high variance nature of the Atari100k steps benchmark. We restrict ourselves to using grayscale renderings of image observations and use random crop of frame stack as data augmentation.

CURL 的灵活性使其只需最少改动即可用于离散控制。与连续控制选 SAC 作基线类似，Atari100K 我们选与较早版 SimPLe（新版分数更高）竞争的数据高效 Rainbow DQN（Efficient Rainbow）（van Hasselt et al. 2019）。为专门看清 CURL 的增益、不做其他改动，我们采用该文（van Hasselt et al. 2019）给出的完全相同超参（含使用更大核与步长 5 的改装卷积编码器）。细节见表 4。与 DMControl 相同，对比目标与 RL 目标等权学习（Pong、Freeway、Boxing 与 PrivateEye 除外：这四款游戏动量对比损失系数为 $0.05$。26 款中的绝大多数（22 款）不做该调整。Atari 通常对所有游戏用同一超参，但提出辅助损失的论文常采用按游戏设系数的做法（Jaderberg et al. 2016）。）。我们使用 https://github.com/Kaixhin/Rainbow 上的 Efficient Rainbow 代码库，其中复现了 van Hasselt et al. 2019。鉴于 Atari100k 步基准方差很高，用 $20$ 个随机种子评测并报告每款游戏平均分。输入限制为灰度渲染，数据增强为对帧堆叠做随机裁剪。

Hyperparameter	Value
Random crop	True
Image size	$(84,84)$
Data Augmentation	Random Crop (Train)
Replay buffer size	$100000$
Training frames	$400000$
Training steps	$100000$
Frame skip	$4$
Stacked frames	$4$
Action repeat	$4$
Replay period every	$1$
Q network: channels	$32$, $64$
Q network: filter size	$5\times 5,5\times 5$
Q network: stride	$5$, $5$
Q network: hidden units	$256$
Momentum (EMA for CURL) $\tau$	$0.001$
Non-linearity	ReLU
Reward Clipping	$[-1,1]$
Multi step return	$20$
Minimum replay size for sampling	$1600$
Max frames per episode	$108$K
Update	Distributional Double Q
Target Network Update Period	every $2000$ updates
Support-of-Q-distribution	$51$ bins
Discount $\gamma$	$0.99$
Batch Size	$32$
Optimizer	Adam
Optimizer: learning rate	$0.0001$
Optimizer: $\beta 1$	$0.9$
Optimizer: $\beta 2$	$0.999$
Optimizer $\epsilon$	$0.000015$
Max gradient norm	$10$
Exploration	Noisy Nets
Noisy nets parameter	$0.1$
Priority exponent	$0.5$
Priority correction	$0.4\rightarrow 1$
Hardware	CPU

Table 4: Hyperparameters used for Atari100K CURL experiments. Hyperparameters are unchanged across games.

超参数	取值
随机裁剪	True
图像大小	$(84,84)$
数据增强	Random Crop (Train)
回放缓冲大小	$100000$
训练帧数	$400000$
训练步数	$100000$
帧跳跃	$4$
堆叠帧数	$4$
动作重复	$4$
回放周期	$1$
Q network: channels	$32$, $64$
Q network: filter size	$5\times 5,5\times 5$
Q network: stride	$5$, $5$
Q network: hidden units	$256$
动量（CURL 的 EMA） $\tau$	$0.001$
非线性	ReLU
奖励裁剪	$[-1,1]$
多步回报	$20$
开始采样的最小回放大小	$1600$
每回合最大帧数	$108$K
Update	Distributional Double Q
目标网络更新周期	every $2000$ updates
Q 分布支撑	$51$ bins
折扣 $\gamma$	$0.99$
批大小	$32$
优化器	Adam
优化器: learning rate	$0.0001$
优化器: $\beta 1$	$0.9$
优化器: $\beta 2$	$0.999$
优化器 $\epsilon$	$0.000015$
最大梯度范数	$10$
探索	Noisy Nets
Noisy nets 参数	$0.1$
优先级指数	$0.5$
优先级校正	$0.4\rightarrow 1$
硬件	CPU

表 4：Atari100K CURL 实验超参。超参跨游戏不变。

Appendix C Benchmarking Data Efficiency

附录 C 数据效率基准

Tables 1 and 2 show the episode returns of DMControl100k, DMControl500k, and Atari100k across CURL and a number of pixel-based baselines. CURL outperforms all baseline pixel-based methods across experiments on both DMControl100k and DMControl500k. On Atari100k experiments, CURL coupled with Eff Rainbow outperforms the baseline on the majority of games tested (19 out of 26 games).

表 1 与表 2 给出 DMControl100k、DMControl500k 与 Atari100k 上 CURL 与多种像素基线的回合回报。CURL 在 DMControl100k 与 DMControl500k 全部实验上优于所有像素基线；Atari100k 上，CURL 与 Eff Rainbow 结合在多数游戏（26 中的 19）上超过基线。

Figure 6: The number of steps it takes a prior leading pixel-based method, Dreamer, to achieve the same score that CURL achieves at 100k training steps (clipped at 1M steps). On average, CURL is 4.5x more data-efficient. We chose Dreamer because the authors (Hafner et al. 2019) report performance for all of the above environments while other baselines like SLAC and SAC+AE only benchmark on 4 and 6 environments, respectively. For further comparison of CURL with these methods, the reader is referred to Table 1 and Figure 4.

图 6：先前领先像素方法 Dreamer 达到 CURL 在 100k 训练步所得分数所需的步数（截断于 1M 步）。平均而言 CURL 数据效率高 4.5×。选择 Dreamer 是因为作者（Hafner et al. 2019）报告了上述全部环境的表现，而 SLAC、SAC+AE 等基线分别只评 4 与 6 个环境。与这些方法的进一步对比见表 1 与图 4。

Appendix D Further Investigation of Data-Efficiency in Contrastive RL

附录 D 对比 RL 数据效率的进一步考察

To further benchmark CURL’s sample-efficiency, we compare it to state-based SAC on a total of 16 DMControl environments. Shown in Figure 7, CURL matches state-based data-efficiency on most of the environments, but lags behind state-based SAC on more challenging environments.

为进一步基准 CURL 的样本效率，我们在共 16 个 DMControl 环境上与基于状态的 SAC 对比。如图 7 所示，CURL 在多数环境上对齐状态方法的数据效率，但在更具挑战的环境上落后于状态 SAC。

Figure 7: CURL compared to state-based SAC run for 3 seeds on each of 16 selected DMControl environments. For the 6 environments in 4, CURL performance is averaged over 10 seeds.

图 7：CURL 与基于状态的 SAC 在 16 个选定 DMControl 环境上各跑 3 个种子的对比。对图 4 中的 6 个环境，CURL 性能按 10 个种子平均。

Appendix E Ablations

附录 E 消融

E.1 Learning Temporal Dynamics

E.1 学习时间动态

Figure 8: CURL with temporal and visual discrimination (red) compared to CURL with only visual discrimination (green). In most settings, the variant with temporal variant outperforms the purely visual variant of CURL. The two exceptions are reacher and ball in cup environments, suggesting that learning dynamics is not necessary for those two environments. Note that the walker environment was run with action repeat of 4, whereas walker walk in the main results Table 1 and Figure 7 was run with action repeat of 2.

图 8：同时做时间与视觉判别的 CURL（红）对比仅做视觉判别的 CURL（绿）。多数设定下带时间变体优于纯视觉变体；例外是 reacher 与 ball in cup，暗示这两项不必学动态。注意 walker 环境此处动作重复为 4，而主结果表 1 与图 7 中的 walker walk 动作重复为 2。

To gain insight as to whether CURL learns temporal dynamics across the stacked frames, we also train a variant of CURL where the discriminants are individual frames as opposed to stacked ones. This can be done by sampling stacked frames from the replay buffer but only using the first frame to update the contrastive loss:

为考察 CURL 是否学到堆叠帧上的时间动态，我们另训一变体：判别对象是单帧而非堆叠。做法是从回放缓冲采样帧堆叠，但更新对比损失时只用第一帧：

During the actor-critic update, frames in the batch are encoded individually into latent codes, which are then concatenated before being passed to a dense network.

演员–评论家更新时，批次中各帧单独编码成隐码，再拼接后送入稠密网络。

Encoding each frame indiviudally ensures that the contrastive objective only has access to visual discriminants. Comparing the visual and spatiotemporal variants of CURL in Figure 8 shows that the variant trained on stacked frames outperforms the visual-only version in most environments. The only exceptions are reacher and ball-in-cup environments. Indeed, in those environments the visual signal is strong enough to solve the task optimally, whereas in other environments, such as walker and cheetah, where balance or coordination is required, visual information alone is insufficient.

逐帧编码保证对比目标只能利用视觉判别。图 8 比较视觉与时空变体：多数环境上在堆叠帧上训练的变体更优；例外是 reacher 与 ball-in-cup——这两项视觉信号已足以最优求解，而 walker、cheetah 等需要平衡或协调的环境，仅靠视觉不够。

E.2 Increasing Gradient Updates per Agent Step

E.2 提高每智能体步的梯度更新次数

Although most baselines we benchmark against use one gradient update per agent step, it was recently empirically shown that increasing the ratio of gradients per step improves data-efficiency in RL (Kielak 2020). This finding is also supported by SLAC (Lee et al. 2019), where results are shown with a ratio of 1:1 (SLACv1) and 3:1 (SLACv2). We

多数基线每智能体步做一次梯度更新，但近期经验表明提高每步梯度比可改善 RL 数据效率（Kielak 2020）。SLAC（Lee et al. 2019）也支持这一点：分别报告了 1:1（SLACv1）与 3:1（SLACv2）的结果。我们

DMControl500k	CURL	SLACv2
Finger, spin	923 $\pm$ 50	884 $\pm$ 98
Walker, walk	911 $\pm$ 35	891 $\pm$ 60
Cheetah, run	545 $\pm$ 39	791 $\pm$ 37
Ball in cup, catch	948 $\pm$ 21	885 $\pm$ 154
DMControl100k	CURL	SLACv2
Finger, spin	741$\pm$ 118	728 $\pm$212
Walker, walk	428 $\pm$ 59	513 $\pm$ 41
Cheetah, run	314 $\pm$ 46	438 $\pm$ 76
Ball in cup, catch	899 $\pm$ 47	837 $\pm$ 147

Table 5: Scores achieved by CURL and SLAC when run with a 3:1 ratio of gradient updates per agent step on DMControl500k and DMControl100k. CURL achieves state-of-the-art performance on the majority (3 out of 4) environments on DMControl500k. Performance of both algorithms is improved relative to the 1:1 ratio reported for all baselines in Table 1 but at the cost of significant compute and wall-clock time overhead.

DMControl500k	CURL	SLACv2
Finger, spin	923 $\pm$ 50	884 $\pm$ 98
Walker, walk	911 $\pm$ 35	891 $\pm$ 60
Cheetah, run	545 $\pm$ 39	791 $\pm$ 37
Ball in cup, catch	948 $\pm$ 21	885 $\pm$ 154
DMControl100k	CURL	SLACv2
Finger, spin	741$\pm$ 118	728 $\pm$212
Walker, walk	428 $\pm$ 59	513 $\pm$ 41
Cheetah, run	314 $\pm$ 46	438 $\pm$ 76
Ball in cup, catch	899 $\pm$ 47	837 $\pm$ 147

表 5：每智能体步梯度更新比为 3:1 时，CURL 与 SLAC 在 DMControl500k 与 DMControl100k 上的得分。CURL 在 DMControl500k 多数（3 出 4）环境上达到当时最佳。相对表 1 全部基线报告的 1:1 比例，两种算法性能都有提升，但计算与墙钟开销显著增加。

E.3 Decoupling Representation Learning from Reinforcement Learning

E.3 把表征学习与强化学习解耦

Typically, Deep RL representations depend almost entirely on the reward function specific to a task. However, hand-crafted representations such as the proprioceptive state are independent of the reward function. It is much more desirable to learn reward-agnostic representations, so that the same representation can be re-used across different RL tasks. We test whether CURL can learn such representations by comparing CURL to a variant where the critic gradients are backpropagated through the critic and contrastive dense feedforward networks but stopped before reaching the convolutional neural network (CNN) part of the encoder.

通常深度 RL 表征几乎完全依赖任务特定奖励；而本体感觉状态等手工特征与奖励无关。更理想的是学到与奖励无关的表征，以便跨 RL 任务复用。我们比较 CURL 与一变体：评论家梯度经评论家与对比稠密前馈网络回传，但在到达编码器卷积（CNN）部分前停止。

Scores displayed in Figure 9 show that for many environments, the detached CNN representations are sufficient to learn an optimal policy. The major exception is the cheetah environment, where the detached representation significantly under-performs. Though promising, we leave further exploration of task-agnostic representations for future work.

图 9 显示：许多环境上，分离的 CNN 表征已足以学到近优策略；主要例外是 cheetah，分离表征明显落后。虽有希望，任务无关表征的进一步探索留待未来工作。

Figure 9: CURL where the CNN part of the encoder receives gradients from both the contrastive loss and critic (red) compared to CURL with the convolutional part of the encoder trained only with the contrastive objective (green). The detached encoder variant is able to learn representations that enable near-optimal learning on most environments, except for cheetah. As in Figure 8, the walker environment was run with action repeat of 4.

图 9：编码器 CNN 部分同时接收对比损失与评论家梯度的 CURL（红），对比卷积部分仅用对比目标训练的 CURL（绿）。分离编码器变体在多数环境上能学到支持近优学习的表征，cheetah 除外。与图 8 一样，walker 环境动作重复为 4。

E.4 Removing Data Augmentation for the Actor Critic

E.4 去掉演员–评论家的数据增强

Our main results involve the use of data augmentations to regularize both the contrastive and SAC objectives. Here, we investigate whether the contrastive representations alone are sufficient for learning effective policies. In these experiments, we only augment the data for the contrastive objective but not for the SAC agent. As a result, data augmentation is used only to learn features but does not influence the control policy. The pseudocode is shown below:

主结果对对比与 SAC 目标都用数据增强做正则。这里考察仅靠对比表征是否足以学到有效策略：只对对比目标增强数据，不对 SAC 智能体增强，因而增强只用于学特征、不影响控制策略。伪代码如下：

o = original unaugmented observation

aug = augmentation

contrastive = InfoNCE loss

o_anchor, o_target = aug(o), aug(o)
curl_loss = contrastive(o_anchor, o_target)
sac_loss = critic_loss(o) + actor_loss(o)
loss = curl_loss + sac_loss
params = update(params, grad(loss, params))

Figure 10: CURL with no data augmentations passed to the SAC agent improves the performance of the baseline pixel SAC by a mean of 2.0x / median of 1.7x on DMControl500k. For these runs we use a smaller batch size of 128 than the 512 batch size used for results in Table 4. While the constastive loss alone improves over the pixel SAC baseline, most environments benefit from data augmentation also being passed to the SAC agent.

图 10：不对 SAC 智能体传入数据增强的 CURL，在 DMControl500k 上相对基线像素 SAC 平均提升 2.0× / 中位 1.7×。这些运行批大小为 128，小于表 4 结果所用的 512。仅对比损失已能提升像素 SAC，但多数环境在把数据增强也传给 SAC 智能体时受益更大。

DMControl500k results plotted in Figure 10 show that, on average, features learned through the contrastive loss alone improve the pixel SAC baseline by 2x. Augmenting the input passed to the SAC algorithm further improves performance.

图 10 中的 DMControl500k 结果表明：平均而言，仅通过对比损失学到的特征相对像素 SAC 基线提升约 2×；再把增强输入传给 SAC 还可进一步提升。

E.5 Predicting State from Pixels

E.5 从像素预测状态

Despite improved sample-efficiency on most DMControl tasks, there is still a visible gap between the performance of SAC on state and SAC with CURL in some environments. Since CURL learns representations by performing instance discrimination across stacks of three frames, it’s possible that the reason for degraded sample-efficiency on more challenging tasks is due to partial-observability of the ground truth state.

尽管多数 DMControl 任务样本效率提升，部分环境上状态 SAC 与带 CURL 的 SAC 仍有可见差距。由于 CURL 通过对三帧堆叠做实例判别学习表征，更具挑战任务上样本效率下降，可能源于真值状态的部分可观测性。

To test this hypothesis, we perform supervised regression $(X,Y)$ from pixels $X$ to the proprioceptive state $Y$, where each data point $x\in X$ is a stack of three consecutive frames and $y\in Y$ is the corresponding state extracted from the simulator. We find that the error in predicting the state from pixels correlates with the policy performance of pixel-based methods. Test-time error rates displayed in Figure 11 show that environments that CURL solves as efficiently as state-based SAC have low error-rates in predicting the state from stacks of pixels. The prediction error increases for more challenging environments, such as cheetah-run and walker-walk. Finally, the error is highest for environments where current pixel-based methods, CURL included, make no progress at all (Tassa et al. 2018), such as humanoid and swimmer.

为检验该假设，我们做从像素 $X$ 到本体感觉状态 $Y$ 的有监督回归 $(X,Y)$：每个 $x\in X$ 为连续三帧堆叠，$y\in Y$ 为仿真器给出的对应状态。发现从像素预测状态的误差与像素方法的策略表现相关。图 11 的测试误差显示：CURL 能与状态 SAC 同样高效求解的环境，从像素堆叠预测状态的误差较低；cheetah-run、walker-walk 等更具挑战环境误差升高；而当前像素方法（含 CURL）完全无进展的环境（Tassa et al. 2018）如 humanoid 与 swimmer，误差最高。

This investigation suggests that degraded policy performance on challenging tasks may result from the lack of requisite information about the underlying state in the pixel data used for learning representations. We leave further investigation for future work.

该考察表明：挑战任务上策略变差，可能因为用于学表征的像素数据缺少关于底层状态的必要信息。进一步调查留待未来工作。

Figure 11: Test-time mean squared error for predicting the proprioceptive state from pixels on a number of DMControl environments. In DMControl, environments fall into two groups - where the state corresponds to either (a) positions and velocities of the robot joints or (b) the joint angles and angular velocities.

图 11：若干 DMControl 环境上从像素预测本体感觉状态的测试均方误差。DMControl 中环境分两类——状态对应（a）机器人关节的位置与速度，或（b）关节角与角速度。

E.6 CURL + Efficient Rainbow Atari runs

E.6 CURL + Efficient Rainbow 的 Atari 运行

We report the scores (Tables 6 and 7) for 20 seeds across the 26 Atari games in the Atari100k benchmark for CURL coupled with Efficient Rainbow. The variance across multiple seeds is considerably high in this benchmark. Therefore, we report the scores for each of the seeds along with the mean and standard deviation for each game.

我们报告 Atari100k 基准上、CURL 与 Efficient Rainbow 结合、26 款游戏各 20 个种子的分数（表 6 与表 7）。该基准跨种子方差相当高，故报告每个种子分数以及每款游戏的均值与标准差。

Pacman	Frostbite	Asterix	KungFuMaster	Kangaroo	Gopher	RoadRunner	JamesBond	BattleZone	Seaquest	Assault	Krull	Qbert
1287	2292	850	8470	600	1036	2820	305	18100	322	634.2	3404.3	1020
1608	1046	525	10870	2280	574	3190	265	18200	236	696.8	2443.5	650
1466	1209	655	10920	1940	540	7840	335	26800	352	655.2	6791.4	830
1430	255	565	7730	1140	618	12060	145	21300	386	443	3022.5	902.5
1114	426	715	17525	520	534	8340	565	7900	458	546	3892.2	3957.5
1083	2280	715	3560	600	596	6920	565	8100	224	564.9	3505.5	772.5
2301	259	770	10940	600	502	2230	350	12000	282	514.4	2564.1	782.5
1128	335	980	23420	900	998	4250	365	16500	339	516.6	4079.7	727.5
1184	1409	665	15160	600	950	1570	140	23900	526	661.5	2376.4	705
1510	258	610	15370	730	544	6300	425	19900	436	664.5	4161.8	757.5
2343	335	905	22260	600	796	3100	315	10000	272	529	3311.1	647.5
1063	1062	800	17320	880	522	1060	335	11200	428	445.2	2517.3	562.5
2040	1542	675	31820	220	392	6050	735	9700	358	573.3	3764.7	2425
1195	1102	795	23360	920	780	11810	950	23500	533	531.3	10150.2	1112.5
1343	2461	585	27460	600	792	4630	520	10500	968	663.6	2883.6	527.5
1354	257	865	7770	2300	454	2530	755	18100	314	795.3	5123.7	472.5
1925	513	730	8820	320	564	6840	750	9000	378	633	3652.5	610
1228	1826	680	2980	600	522	6580	795	8900	168	674.1	2376.4	697.5
1099	1889	965	10100	600	496	10720	450	10700	242	604.8	11745	1847.5
1608	2869	640	10300	500	1176	4380	355	13100	467	665.7	2826	840
1465.5	1181.3	734.5	14307.8	872.5	669.3	5661	471	14870	384.5	600.6	4229.6	1042.4
397.5	856.2	129.8	7919.3	600.1	220.6	3289.3	226.2	5964.3	170.2	89.5	2540.6	828.4

Table 6: CURL implemented on top of Efficient Rainbow - Scores reported for $20$ random seeds for each of the above games, with the last two rows being the mean and standard deviation across the runs.

Pacman	Frostbite	Asterix	KungFuMaster	Kangaroo	Gopher	RoadRunner	JamesBond	BattleZone	Seaquest	Assault	Krull	Qbert
1287	2292	850	8470	600	1036	2820	305	18100	322	634.2	3404.3	1020
1608	1046	525	10870	2280	574	3190	265	18200	236	696.8	2443.5	650
1466	1209	655	10920	1940	540	7840	335	26800	352	655.2	6791.4	830
1430	255	565	7730	1140	618	12060	145	21300	386	443	3022.5	902.5
1114	426	715	17525	520	534	8340	565	7900	458	546	3892.2	3957.5
1083	2280	715	3560	600	596	6920	565	8100	224	564.9	3505.5	772.5
2301	259	770	10940	600	502	2230	350	12000	282	514.4	2564.1	782.5
1128	335	980	23420	900	998	4250	365	16500	339	516.6	4079.7	727.5
1184	1409	665	15160	600	950	1570	140	23900	526	661.5	2376.4	705
1510	258	610	15370	730	544	6300	425	19900	436	664.5	4161.8	757.5
2343	335	905	22260	600	796	3100	315	10000	272	529	3311.1	647.5
1063	1062	800	17320	880	522	1060	335	11200	428	445.2	2517.3	562.5
2040	1542	675	31820	220	392	6050	735	9700	358	573.3	3764.7	2425
1195	1102	795	23360	920	780	11810	950	23500	533	531.3	10150.2	1112.5
1343	2461	585	27460	600	792	4630	520	10500	968	663.6	2883.6	527.5
1354	257	865	7770	2300	454	2530	755	18100	314	795.3	5123.7	472.5
1925	513	730	8820	320	564	6840	750	9000	378	633	3652.5	610
1228	1826	680	2980	600	522	6580	795	8900	168	674.1	2376.4	697.5
1099	1889	965	10100	600	496	10720	450	10700	242	604.8	11745	1847.5
1608	2869	640	10300	500	1176	4380	355	13100	467	665.7	2826	840
1465.5	1181.3	734.5	14307.8	872.5	669.3	5661	471	14870	384.5	600.6	4229.6	1042.4
397.5	856.2	129.8	7919.3	600.1	220.6	3289.3	226.2	5964.3	170.2	89.5	2540.6	828.4

表 6：建在 Efficient Rainbow 上的 CURL——上列各游戏 20 个随机种子的分数，最后两行为各次运行的均值与标准差。

UpNDown	Hero	CrazyClimber	ChopperComm.	DemonAttack	Amidar	Alien	BankHeist	Breakout	Freeway	Pong	PrivateEye	Boxing
3529	8747.5	19090	560	611.5	150.9	616	95	3.6	29.2	-19.3	100	-0.5
772	3026	8290	1530	707.5	131.2	923	184	5	25.4	-16.9	100	-11.4
5972	7146	12160	1390	843.5	141.5	467	75	3.2	27.6	-12	100	4
2793	7686	8920	1100	330.5	133.7	441	232	5.1	28.6	-19.6	100	3.6
3546	7335	11360	500	759	157.1	716	187	2.9	22.8	-17.8	1357.4	6.2
4552	7325	4110	990	940	125.4	453	367	6.3	29.6	-18.9	100	5
2972	7275.5	9460	780	1136	183.2	273	186	5.9	23.3	-15.9	0	-1.7
2865	3115	20630	1180	758	153.6	540	68	2.6	27.6	-15.2	100	0.1
3098	7424	6780	1380	772.5	127.8	499	60	5.9	26.1	-18.7	100	3.5
1953	7475	13570	970	820	149.4	475	123	4.3	28.3	-13.3	100	-0.5
1467	3135	11890	1200	784	125.7	553	72	3.2	21.8	-17.2	1510	-22.1
2912	5060.5	9160	1130	1080	130.4	446	53	4.8	21.8	-20.1	100	-1.8
4123	4409	10960	1380	847	133	533	68	6.3	28.9	-16.5	100	1.6
2334	6979	17360	1230	771.5	140.5	968	36	7.3	28.2	-14.9	100	3.6
2605	4159	8930	1350	907.5	133.8	499	53	4.8	28.3	-19.3	100	-17.6
2432	7560	11510	1080	1095.5	191.8	523	105	3.7	26.8	-15.6	0	21.7
3826	8587	22690	1210	700	115.5	616	276	6.6	27.5	-21	100	2
3052	4683.5	8120	840	803.5	164	475	69	5.5	26.5	-10.5	0	5.9
3131	7317	13500	730	818	131.7	525	50	4.3	26.8	-13.3	100	18.7
1169	7141	14440	640	866	122.4	622	273	6.2	28.6	-13.1	100	3.7
2955.2	6279.3	12146.5	1058.5	817.6	142.1	558.2	131.6	4.9	26.7	-16.5	218.4	1.2
1181.1	1871.5	4765.6	299.1	176.6	20.0	160.3	94.4	1.4	2.4	2.9	417.9	10.0

Table 7: CURL implemented on top of Efficient Rainbow - Scores reported for $20$ random seeds for each of the above games, with the last two rows being the mean and standard deviation across the runs.