Distributed Prioritized Experience Replay

分布式优先经验回放

Dan Horgan Affiliation:  DeepMind Email:  horgan@google.com    John Quan Affiliation:  DeepMind Email:  johnquan@google.com    David Budden Affiliation:  DeepMind Email:  budden@google.com    Gabriel Barth-Maron Affiliation:  DeepMind Email:  gabrielbm@google.com    Matteo Hessel Affiliation:  DeepMind Email:  mtthss@google.com    Hado van Hasselt Affiliation:  DeepMind Email:  hado@google.com    David Silver Affiliation:  DeepMind Email:  davidsilver@google.com

Abstract

摘要

We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

我们提出一套可规模化的深度强化学习分布式架构，使智能体能够有效学习比以往高出若干数量级的数据。算法把执行与学习解耦：各执行器在各自的环境实例中按共享神经网络选动作，并把所得经验写入共享经验回放；学习者回放经验样本并更新网络。架构依赖优先经验回放，只聚焦执行器产生的最有价值数据。该架构在 Arcade Learning Environment 上大幅刷新当时表现，以更短墙钟训练时间取得更好最终性能。

1 Introduction

1 引言

A broad trend in deep learning is that combining more computation (Dean et al., 2012) with more powerful models (Kaiser et al., 2017) and larger datasets (Deng et al., 2009) yields more impressive results. It is reasonable to hope that a similar principle holds for deep reinforcement learning. There are a growing number of examples to justify this optimism: effective use of greater computational resources has been a critical factor in the success of such algorithms as Gorila (Nair et al., 2015), A3C (Mnih et al., 2016), GPU Advantage Actor Critic (Babaeizadeh et al., 2017), Distributed PPO (Heess et al., 2017) and AlphaGo (Silver et al., 2016).

深度学习的一条宽趋势是：更多算力（Dean et al., 2012）、更强模型（Kaiser et al., 2017）与更大数据（Deng et al., 2009）叠加，往往带来更醒目的结果。有理由期望深度强化学习也遵循类似原则。越来越多例子支撑这一乐观：Gorila（Nair et al., 2015）、A3C（Mnih et al., 2016）、GPU Advantage Actor Critic（Babaeizadeh et al., 2017）、Distributed PPO（Heess et al., 2017）与 AlphaGo（Silver et al., 2016）等算法的成功，都离不开更有效地动用算力。

Deep learning frameworks such as TensorFlow (Abadi et al., 2016) support distributed training, making large scale machine learning systems easier to implement and deploy. Despite this, much current research in deep reinforcement learning concerns itself with improving performance within the computational budget of a single machine, and the question of how to best harness more resources is comparatively underexplored.

TensorFlow（Abadi et al., 2016）等深度学习框架已支持分布式训练，使大规模机器学习系统更易实现与部署。尽管如此，深度强化学习的许多研究仍盯着单机算力预算内的提升，如何更好利用更多资源仍相对少被探索。

In this paper we describe an approach to scaling up deep reinforcement learning by generating more data and selecting from it in a prioritized fashion (Schaul et al., 2016). Standard approaches to distributed training of neural networks focus on parallelizing the computation of gradients, to more rapidly optimize the parameters (Dean et al., 2012). In contrast, we distribute the generation and selection of experience data, and find that this alone suffices to improve results. This is complementary to distributing gradient computation, and the two approaches can be combined, but in this work we focus purely on data-generation.

本文描述一种扩展深度强化学习的路径：生成更多数据，再按优先级从中挑选（Schaul et al., 2016）。神经网络分布式训练的标准做法侧重并行算梯度以更快优化参数（Dean et al., 2012）。我们则分布式地生成并筛选经验数据，并发现仅此一项就足以提升结果。它与分布式梯度计算互补，二者可以结合；本文则专注于数据生成。

We use this distributed architecture to scale up variants of Deep Q-Networks (DQN) and Deep Deterministic Policy Gradient (DDPG), and we evaluate these on the Arcade Learning Environment benchmark (Bellemare et al., 2013), and on a range of continuous control tasks. Our architecture achieves a new state of the art performance on Atari games, using a fraction of the wall-clock time compared to the previous state of the art, and without per-game hyperparameter tuning.

我们用该分布式架构扩展 Deep Q-Networks（DQN）与 Deep Deterministic Policy Gradient（DDPG）的变体，并在 Arcade Learning Environment 基准（Bellemare et al., 2013）以及一系列连续控制任务上评测。架构在 Atari 游戏上达到新的前沿表现，墙钟时间仅为此前前沿的一小部分，且无需按游戏调超参。

We empirically investigate the scalability of our framework, analysing how prioritization affects performance as we increase the number of data-generating workers. Our experiments include an analysis of factors such as the replay capacity, the recency of the experience, and the use of different data-generating policies for different workers. Finally, we discuss implications for deep reinforcement learning agents that may apply beyond our distributed framework.

我们经验地考察框架的可扩展性，分析在增加数据生成工作进程时优先级如何影响性能。实验涵盖回放容量、经验新旧程度，以及不同工作进程采用不同数据生成策略等因素。最后讨论若干可能超出本分布式框架的深度强化学习启示。

2 Background

2 背景

Distributed Stochastic Gradient Descent

Distributed stochastic gradient descent is widely used in supervised learning to speed up training of deep neural networks, by parallelizing the computation of the gradients used to update their parameters. The resulting parameter updates may be applied synchronously (Krizhevsky, 2014) or asynchronously (Dean et al., 2012). Both approaches have proven effective and are an increasingly standard part of the deep learning toolbox. Inspired by this, Nair et al. (2015) applied distributed asynchronous parameter updates and distributed data generation to deep reinforcement learning. Asynchronous parameter updates and parallel data generation have also been successfully used within a single-machine, in a multi-threaded rather than a distributed context (Mnih et al., 2016). GPU Asynchronous Actor-Critic (Babaeizadeh et al., 2017, GA3C;) and Parallel Advantage Actor-Critic (Clemente et al., 2017, PAAC;) adapt this approach to make efficient use of GPUs.

分布式随机梯度下降

分布式随机梯度下降在监督学习中广泛用于加速深度网络训练：并行计算用于更新参数的梯度。参数更新可以同步（Krizhevsky, 2014）或异步（Dean et al., 2012）。两种做法都已证明有效，并日益成为深度学习标配。受此启发，Nair et al.（2015）把分布式异步参数更新与分布式数据生成用于深度强化学习。异步参数更新与并行数据生成也曾在单机多线程而非分布式情境下成功使用（Mnih et al., 2016）。GPU Asynchronous Actor-Critic（Babaeizadeh et al., 2017, GA3C）与 Parallel Advantage Actor-Critic（Clemente et al., 2017, PAAC）把该思路改造成更高效利用 GPU。

Distributed Importance Sampling

A complementary family of techniques for speeding up training is based on variance reduction by means of importance sampling (Hastings, 1970, cf.). This has been shown to be useful in the context of neural networks (Hinton, 2007). Sampling non-uniformly from a dataset and weighting updates according to the sampling probability in order to counteract the bias thereby introduced can increase the speed of convergence by reducing the variance of the gradients. One way of doing this is to select samples with probability proportional to the $L_{2}$ norm of the corresponding gradients. In supervised learning, this approach has been successfully extended to the distributed setting (Alain et al., 2015). An alternative is to rank samples according to their latest known loss value and make the sampling probability a function of the rank rather than of the loss itself (Loshchilov & Hutter, 2015).

分布式重要性采样

另一类加速训练的技术族，借助重要性采样降低方差（Hastings, 1970, cf.），在神经网络情境下也已显示有用（Hinton, 2007）。对数据集非均匀采样，并按采样概率加权更新以抵消由此引入的偏差，可通过降低梯度方差加快收敛。做法之一是按对应梯度的 $L_{2}$ 范数成比例选样本。监督学习中，该思路已成功扩展到分布式设定（Alain et al., 2015）。另一种做法是按最近已知损失值给样本排序，使采样概率成为秩的函数而非损失本身（Loshchilov & Hutter, 2015）。

Prioritized Experience Replay

Experience replay (Lin, 1992) has long been used in reinforcement learning to improve data efficiency. It is particularly useful when training neural network function approximators with stochastic gradient descent algorithms, as in Neural Fitted Q-Iteration (Riedmiller, 2005) and Deep Q-Learning (Mnih et al., 2015). Experience replay may also help to prevent overfitting by allowing the agent to learn from data generated by previous versions of the policy. Prioritized experience replay (Schaul et al., 2016) extends classic prioritized sweeping ideas (Moore & Atkeson, 1993) to work with deep neural network function approximators. The approach is strongly related to the importance sampling techniques discussed in the previous section, but using a more general class of biased sampling procedures that focus learning on the most ‘surprising’ experiences. Biased sampling can be particularly helpful in reinforcement learning, since the reward signal may be sparse and the data distribution depends on the agent’s policy. As a result, prioritized experience replay is used in many agents, such as Prioritized Dueling DQN (Wang et al., 2016), UNREAL (Jaderberg et al., 2017), DQfD (Hester et al., 2017), and Rainbow (Hessel et al., 2017). In an ablation study conducted to investigate the relative importance of several algorithmic ingredients (Hessel et al., 2017), prioritization was found to be the most important ingredient contributing to the agent’s performance.

优先经验回放

经验回放（Lin, 1992）在强化学习中长期用于提高数据效率，在用随机梯度下降训练神经网络函数逼近器时尤其有用，如 Neural Fitted Q-Iteration（Riedmiller, 2005）与 Deep Q-Learning（Mnih et al., 2015）。经验回放也有助于防止过拟合：智能体可从先前版本策略产生的数据中学习。优先经验回放（Schaul et al., 2016）把经典优先扫描思想（Moore & Atkeson, 1993）扩展到深度网络函数逼近器，与上一节的重要性采样密切相关，但采用更一般的有偏采样，把学习聚焦到最「出人意料」的经验。强化学习中奖励可能稀疏、数据分布又依赖策略，有偏采样因而特别有帮助。结果是许多智能体都采用优先经验回放，如 Prioritized Dueling DQN（Wang et al., 2016）、UNREAL（Jaderberg et al., 2017）、DQfD（Hester et al., 2017）与 Rainbow（Hessel et al., 2017）。在考察多种算法成分相对重要性的消融中（Hessel et al., 2017），优先级被认定为对性能贡献最大的成分。

3 Our Contribution: Distributed Prioritized Experience Replay

3 我们的贡献：分布式优先经验回放

In this paper we extend prioritized experience replay to the distributed setting and show that this is a highly scalable approach to deep reinforcement learning. We introduce a few key modifications that enable this scalability, and we refer to our approach as Ape-X.

本文将优先经验回放扩展到分布式设定，并表明这是深度强化学习高度可扩展的路径。我们引入若干实现可扩展性的关键改动，并将方法称为 Ape-X。

As in Gorila (Nair et al., 2015), we decompose the standard deep reinforcement learning algorithm into two parts, which run concurrently with no high-level synchronization. The first part consists of stepping through an environment, evaluating a policy implemented as a deep neural network, and storing the observed data in a replay memory. We refer to this as acting. The second part consists of sampling batches of data from the memory to update the policy parameters. We term this learning.

与 Gorila（Nair et al., 2015）类似，我们把标准深度强化学习算法拆成两部分并发运行、不做高层同步。第一部分在环境中步进，评估实现为深度网络的策略，并把观测数据存入回放记忆，称为 执行（acting）。第二部分从记忆中采样批次以更新策略参数，称为 学习（learning）。

Figure 1: The Ape-X architecture in a nutshell: multiple actors, each with its own instance of the environment, generate experience, add it to a shared experience replay memory, and compute initial priorities for the data. The (single) learner samples from this memory and updates the network and the priorities of the experience in the memory. The actors’ networks are periodically updated with the latest network parameters from the learner.

图 1：Ape-X 架构一览：多个执行器各有环境实例，生成经验、写入共享经验回放，并计算数据的初始优先级；单一学习者从该记忆采样，更新网络以及记忆中经验的优先级；执行器网络定期用学习者的最新参数更新。

1: procedure Actor($B$, $T$)$\triangleright$ Run agent in environment instance, storing experiences.
2:   $\theta_{0}\leftarrow$ learner.Parameters( )$\triangleright$ Remote call to obtain latest network parameters.
3:   $s_{0}\leftarrow$ environment.Initialize( ) $\triangleright$ Get initial state from environment.
4:   for $t=1$ to $T$ do
5:    $a_{t-1}\leftarrow\pi_{\theta_{t-1}}(s_{t-1})$$\triangleright$ Select an action using the current policy.
6:    $(r_{t},\gamma_{t},s_{t})\leftarrow\textsc{environment.Step}(a_{t-1})$$\triangleright$ Apply the action in the environment.
7:    localBuffer.Add($(s_{t-1},a_{t-1},r_{t},\gamma_{t})$) $\triangleright$ Add data to local buffer.
8:    if $\textsc{localBuffer.Size}(\ )\geq B$ then $\triangleright$ In a background thread, periodically send data to replay.
9:      $\tau\leftarrow\textsc{localBuffer.Get}(B)$ $\triangleright$ Get buffered data (e.g. batch of multi-step transitions).
10:      $p\leftarrow\textsc{ComputePriorities}(\tau)$ $\triangleright$ Calculate priorities for experience (e.g. absolute TD error).
11:      $\textsc{replay.Add}(\tau,p)$ $\triangleright$ Remote call to add experience to replay memory.
12:    end if
13:    $\textsc{Periodically}(\theta_{t}\leftarrow\textsc{learner.Parameters}())$ $\triangleright$ Obtain latest network parameters.
14:   end for
15: end procedure

Algorithm 1 Actor

1: procedure Actor($B$, $T$)$\triangleright$ 在环境实例中运行智能体并存储经验。
2:   $\theta_{0}\leftarrow$ learner.Parameters( )$\triangleright$ 远程调用以获取最新网络参数。
3:   $s_{0}\leftarrow$ environment.Initialize( ) $\triangleright$ 从环境获取初始状态。
4:   for $t=1$ to $T$ do
5:    $a_{t-1}\leftarrow\pi_{\theta_{t-1}}(s_{t-1})$$\triangleright$ 用当前策略选择动作。
6:    $(r_{t},\gamma_{t},s_{t})\leftarrow\textsc{environment.Step}(a_{t-1})$$\triangleright$ 在环境中执行该动作。
7:    localBuffer.Add($(s_{t-1},a_{t-1},r_{t},\gamma_{t})$) $\triangleright$ 将数据加入本地缓冲。
8:    if $\textsc{localBuffer.Size}(\ )\geq B$ then $\triangleright$ 在后台线程中定期把数据发到回放。
9:      $\tau\leftarrow\textsc{localBuffer.Get}(B)$ $\triangleright$ 取出缓冲数据（例如多步转移批次）。
10:      $p\leftarrow\textsc{ComputePriorities}(\tau)$ $\triangleright$ 计算经验优先级（例如绝对 TD 误差）。
11:      $\textsc{replay.Add}(\tau,p)$ $\triangleright$ 远程调用，把经验加入回放记忆。
12:    end if
13:    $\textsc{Periodically}(\theta_{t}\leftarrow\textsc{learner.Parameters}())$ $\triangleright$ 获取最新网络参数。
14:   end for
15: end procedure

算法 1 执行器（Actor）

1: procedure Learner($T$)$\triangleright$ Update network using batches sampled from memory.
2:   $\theta_{0}\leftarrow$ InitializeNetwork( )
3:   for $t=1$ to $T$ do$\triangleright$ Update the parameters $T$ times.
4:    $id,\tau\leftarrow$ replay.Sample( )$\triangleright$ Sample a prioritized batch of transitions (in a background thread).
5:    $l_{t}\leftarrow\textsc{ComputeLoss}(\tau;\theta_{t})$ $\triangleright$ Apply learning rule; e.g. double Q-learning or DDPG
6:    $\theta_{t+1}\leftarrow\textsc{UpdateParameters}(l_{t};\theta_{t})$
7:    $p\leftarrow$ ComputePriorities( )$\triangleright$ Calculate priorities for experience, (e.g. absolute TD error).
8:    $\textsc{replay.SetPriority}(id,p)$ $\triangleright$ Remote call to update priorities.
9:    $\textsc{Periodically}(\textsc{replay.RemoveToFit}())$ $\triangleright$ Remove old experience from replay memory.
10:   end for
11: end procedure

Algorithm 2 Learner

1: procedure Learner($T$)$\triangleright$ 用从记忆采样的批次更新网络。
2:   $\theta_{0}\leftarrow$ InitializeNetwork( )
3:   for $t=1$ to $T$ do$\triangleright$ 将参数更新 $T$ 次。
4:    $id,\tau\leftarrow$ replay.Sample( )$\triangleright$ 采样一批优先转移（后台线程）。
5:    $l_{t}\leftarrow\textsc{ComputeLoss}(\tau;\theta_{t})$ $\triangleright$ 应用学习规则；例如 double Q-learning 或 DDPG
6:    $\theta_{t+1}\leftarrow\textsc{UpdateParameters}(l_{t};\theta_{t})$
7:    $p\leftarrow$ ComputePriorities( )$\triangleright$ 计算经验优先级（例如绝对 TD 误差）。
8:    $\textsc{replay.SetPriority}(id,p)$ $\triangleright$ 远程调用以更新优先级。
9:    $\textsc{Periodically}(\textsc{replay.RemoveToFit}())$ $\triangleright$ 从回放记忆中移除旧经验。
10:   end for
11: end procedure

算法 2 学习者（Learner）

In principle, both acting and learning may be distributed across multiple workers. In our experiments, hundreds of actors run on CPUs to generate data, and a single learner running on a GPU samples the most useful experiences (Figure 1). Pseudocode for the actors and learners is shown in Algorithms 1 and 2. Updated network parameters are periodically communicated to the actors from the learner.

原则上，执行与学习都可分布到多个工作进程。实验中，数百个执行器在 CPU 上生成数据，单一学习者在 GPU 上采样最有用的经验（图 1）。执行器与学习者的伪代码见算法 1 与 2。更新后的网络参数由学习者定期传给执行器。

In contrast to Nair et al. (2015), we use a shared, centralized replay memory, and instead of sampling uniformly, we prioritize, to sample the most useful data more often. Since priorities are shared, high priority data discovered by any actor can benefit the whole system. Priorities can be defined in various ways, depending on the learning algorithm; two instances are described in the next sections.

与 Nair et al.（2015）不同，我们使用共享的集中式回放记忆，并且不采用均匀采样，而是按优先级更频繁地采最有用数据。优先级共享后，任一执行器发现的高优先级数据都能惠及整个系统。优先级可按学习算法以多种方式定义；下一节给出两个实例。

In Prioritized DQN (Schaul et al., 2016) priorities for new transitions were initialized to the maximum priority seen so far, and only updated once they were sampled. This does not scale well: due to the large number of actors in our architecture, waiting for the learner to update priorities would result in a myopic focus on the most recent data, which has maximum priority by construction. Instead, we take advantage of the computation the actors in Ape-X are already doing to evaluate their local copies of the policy, by making them also compute suitable priorities for new transitions online. This ensures that data entering the replay has more accurate priorities, at no extra cost.

在 Prioritized DQN（Schaul et al., 2016）中，新转移的优先级初始化为迄今最大优先级，只有被采样后才更新。这在大规模下并不好用：执行器很多时，若等学习者更新优先级，系统会短视地盯住「按构造就有最大优先级」的最新数据。Ape-X 改为利用执行器本已在做的本地策略评估，在线计算新转移的合适优先级，使进入回放的数据一开始就有更准的优先级，且几乎不增加额外开销。

Sharing experiences has certain advantages compared to sharing gradients. Low latency communication is not as important as in distributed SGD, because experience data becomes outdated less rapidly than gradients, provided the learning algorithm is robust to off-policy data. Across the system, we take advantage of this by batching all communications with the centralized replay, increasing the efficiency and throughput at the cost of some latency. With this approach it is even possible for actors and learners to run in different data-centers without limiting performance.

共享经验相对共享梯度有若干好处。低延迟通信不如分布式 SGD 那么关键，因为经验数据过时速度慢于梯度——只要学习算法对离策略数据鲁棒。系统里因此可对与集中式回放的所有通信做批处理，以略增延迟换更高效率与吞吐。按此做法，执行器与学习者甚至可位于不同数据中心而不明显限制性能。

Finally, by learning off-policy (Sutton & Barto, 1998; Sutton & Barto, 2017, cf.), we can further take advantage of Ape-X’s ability to combine data from many distributed actors, by giving the different actors different exploration policies, broadening the diversity of the experience they jointly encounter. As we will see in the results, this can be sufficient to make progress on difficult exploration problems.

最后，通过离策略学习（Sutton & Barto, 1998；Sutton & Barto, 2017, cf.），我们还能进一步利用 Ape-X 汇聚众多分布式执行器数据的能力：让不同执行器采用不同探索策略，拓宽它们共同遭遇的经验多样性。结果将显示，这足以在困难探索问题上取得进展。

3.1 Ape-X DQN

The general framework we have described may be combined with different learning algorithms. First, we combined it with a variant of DQN (Mnih et al., 2015) with some of the components of Rainbow (Hessel et al., 2017). More specifically, we used double Q-learning (van Hasselt, 2010; van Hasselt et al., 2016) with multi-step bootstrap targets (Sutton, 1988; Sutton & Barto, 1998; Sutton & Barto, 2017; Mnih et al., 2016, cf.) as the learning algorithm, and a dueling network architecture (Wang et al., 2016) as the function approximator $q(\cdot,\cdot,{\bm{\theta}})$.

上述一般框架可与不同学习算法结合。我们首先把它与带部分 Rainbow 成分（Hessel et al., 2017）的 DQN 变体（Mnih et al., 2015）结合：学习算法采用 double Q-learning（van Hasselt, 2010；van Hasselt et al., 2016）与多步自举目标（Sutton, 1988；Sutton & Barto, 1998；Sutton & Barto, 2017；Mnih et al., 2016, cf.），函数逼近器 $q(\cdot,\cdot,{\bm{\theta}})$ 采用决斗网络结构（Wang et al., 2016）。

This results in computing for all elements in the batch the loss $l_{t}({\bm{\theta}})=\frac{1}{2}(G_{t}-q(S_{t},A_{t},{\bm{\theta}}))^{2}$ with

由此对批次中每个元素计算损失 $l_{t}({\bm{\theta}})=\frac{1}{2}(G_{t}-q(S_{t},A_{t},{\bm{\theta}}))^{2}$，其中

$$G_{t}=\underbrace{R_{t+1}+\gamma R_{t+2}+\ldots+\gamma^{n-1}R_{t+n}+\gamma^{n}\overbrace{q(S_{t+n},\argmax_{a}q(S_{t+n},a,{\bm{\theta}}),{\bm{\theta}}^{-})}^{\mbox{double-Q bootstrap value}}}_{\mbox{multi-step return}}\,,$$

where $t$ is a time index for an experience sampled from the replay starting with state $S_{t}$ and action $A_{t}$, and ${\bm{\theta}}^{-}$ denotes parameters of the target network (Mnih et al., 2015), a slow moving copy of the online parameters. Multi-step returns are truncated if the episode ends in fewer than $n$ steps.

其中 $t$ 是从回放采到的、以状态 $S_{t}$ 与动作 $A_{t}$ 起始的经验时间下标，${\bm{\theta}}^{-}$ 表示 目标网络（Mnih et al., 2015）参数——在线参数的慢速副本。若回合在不足 $n$ 步内结束，多步回报截断。

In principle, Q-learning variants are off-policy methods, so we are free to choose the policies we use to generate data. However, in practice, the choice of behaviour policy does affect both exploration and the quality of function approximation. Furthermore, we are using a multi-step return with no off-policy correction, which in theory could adversely affect the value estimation. Nonetheless, in Ape-X DQN, each actor executes a different policy, and this allows experience to be generated from a variety of strategies, relying on the prioritization mechanism to pick out the most effective experiences. In our experiments, the actors use $\epsilon$-greedy policies with different values of $\epsilon$. Low $\epsilon$ policies allow exploring deeper in the environment, while high $\epsilon$ policies prevent over-specialization.

原则上 Q-learning 变体是离策略方法，数据生成策略可自由选择；实践中行为策略仍影响探索与函数逼近质量。此外我们使用未做离策略校正的多步回报，理论上可能损害价值估计。尽管如此，Ape-X DQN 中每个执行器执行不同策略，使经验来自多种策略，并依靠优先级机制挑出最有效经验。实验中执行器使用不同 $\epsilon$ 的 $\epsilon$-贪婪策略：低 $\epsilon$ 利于深入探索环境，高 $\epsilon$ 防止过度特化。

3.2 Ape-X DPG

To test the generality of the framework we also combined it with a continuous-action policy gradient system based on DDPG (Lillicrap et al., 2016), an implementation of deterministic policy gradients Silver et al. (2014) also similar to older methods (Werbos, 1990; Prokhorov & Wunsch, 1997), and tested it on continuous control tasks from the DeepMind Control Suite (Tassa et al., 2018).

为检验框架一般性，我们还把它与基于 DDPG（Lillicrap et al., 2016）的连续动作策略梯度系统结合——DDPG 是确定性策略梯度（Silver et al., 2014）的一种实现，也类似更早方法（Werbos, 1990；Prokhorov & Wunsch, 1997）——并在 DeepMind Control Suite（Tassa et al., 2018）的连续控制任务上测试。

The Ape-X DPG setup is similar to Ape-X DQN, but the actor’s policy is now represented explicitly by a separate policy network, in addition to the Q-network. The two networks are optimized separately, by minimizing different losses on the sampled experience. We denote the policy and Q-network parameters by $\phi$ and $\psi$ respectively, and adopt the same convention as above to denote target networks. The Q-network outputs an action-value estimate $q(s,a,\psi)$ for a given state $s$, and multi-dimensional action $a\in\mathbb{R}^{m}$. It is updated using temporal-difference learning with a multi-step bootstrap target. The Q-network loss can be written as $l_{t}(\psi)=\frac{1}{2}(G_{t}-q(S_{t},A_{t},\psi))^{2}$, where

Ape-X DPG 设定类似 Ape-X DQN，但执行器策略改由独立的策略网络显式表示，另有 Q 网络。两网络分别优化，在采样经验上最小化不同损失。记策略与 Q 网络参数为 $\phi$ 与 $\psi$，目标网络记号同上。Q 网络对给定状态 $s$ 与多维动作 $a\in\mathbb{R}^{m}$ 输出动作价值估计 $q(s,a,\psi)$，用带多步自举目标的时序差分学习更新。Q 网络损失可写为 $l_{t}(\psi)=\frac{1}{2}(G_{t}-q(S_{t},A_{t},\psi))^{2}$，其中

$$G_{t}=\underbrace{R_{t+1}+\gamma R_{t+2}+\ldots+\gamma^{n-1}R_{t+n}+\gamma^{n}q(S_{t+n},\pi(S_{t+n},\phi^{-}),\psi^{-})}_{\mbox{multi-step return}}\,.$$

The policy network outputs an action $A_{t}=\pi(S_{t},\phi)\in\mathbb{R}^{m}$. The policy parameters are updated using policy gradient ascent on the estimated Q-value, using gradient $\nabla_{\phi}q(S_{t},\pi(S_{t},\phi),\psi)$ — note that this depends on the policy parameters $\phi$ only through the action $A_{t}=\pi(S_{t},\phi)$ that is input to the critic network. Further details of the Ape-X DPG algorithm are available in the appendix.

策略网络输出动作 $A_{t}=\pi(S_{t},\phi)\in\mathbb{R}^{m}$。策略参数通过对估计 Q 值做策略梯度 上升 更新，梯度为 $\nabla_{\phi}q(S_{t},\pi(S_{t},\phi),\psi)$——注意它仅通过输入到评论家网络的动作 $A_{t}=\pi(S_{t},\phi)$ 依赖策略参数 $\phi$。Ape-X DPG 的更多细节见附录。

4 Experiments

4 实验

4.1 Atari

Figure 2: Left: Atari results aggregated across 57 games, evaluated from random no-op starts. Right: Atari training curves for selected games, against baselines. Blue: Ape-X DQN with 360 actors; Orange: A3C; Purple: Rainbow; Green: DQN. See appendix for longer runs over all games.

图 2：左：57 款游戏聚合的 Atari 结果，从随机 no-op 开局评估。右：若干游戏相对基线的训练曲线。蓝：360 执行器的 Ape-X DQN；橙：A3C；紫：Rainbow；绿：DQN。全部游戏的更长运行见附录。

In our first set of experiments we evaluate Ape-X DQN on Atari, and show state of the art results on this standard reinforcement learning benchmark. We use 360 actor machines (each using one CPU core) to feed data into the replay memory as fast as they can generate it; approximately 139 frames per second (FPS) each, for a total of $\sim$50K FPS, which corresponds to $\sim$12.5K transitions (because of a fixed action repeat of 4). The actors batch experience data locally before sending it to the replay: up to 100 transitions may be buffered at a time, which are then sent asynchronously in batches of $B=50$. The learner asynchronously prefetches up to 16 batches of 512 transitions, and computes updates for 19 such batches each second, meaning that gradients are computed for $\sim$9.7K transitions per second on average. To reduce memory and bandwidth requirements, observation data is compressed using a PNG codec when sent and when stored in the replay. The learner decompresses data as it prefetches it, in parallel with computing and applying gradients. The learner also asynchronously handles any requests for parameters from actors.

第一组实验在 Atari 上评测 Ape-X DQN，并在该标准强化学习基准上展示前沿结果。我们用 360 台执行器机器（各一颗 CPU 核）尽快向回放灌数据：每台约 139 帧/秒，合计约 50K FPS，对应约 12.5K 转移/秒（固定动作重复 4）。执行器先在本地批处理经验再发给回放：最多缓冲 100 条转移，再异步以 $B=50$ 的批次发送。学习者异步预取最多 16 个、每批 512 条转移的批次，每秒为约 19 个这样的批次算更新，即平均每秒对约 9.7K 条转移算梯度。为降低内存与带宽，观测在发送与存入回放时用 PNG 编解码压缩；学习者在预取时并行解压，同时算并应用梯度，并异步处理执行器的参数请求。

Actors copy the network parameters from the learner every 400 frames ($\sim$2.8 seconds). Each actor $i\in\{0,...,N-1\}$ executes an $\epsilon_{i}$-greedy policy where $\epsilon_{i}=\epsilon^{1+\frac{i}{N-1}\alpha}$ with $\epsilon=0.4$, $\alpha=7$. Each $\epsilon_{i}$ is held constant throughout training. The episode length is limited to 50000 frames during training.

执行器每 400 帧（约 2.8 秒）从学习者拷贝网络参数。每个执行器 $i\in\{0,...,N-1\}$ 执行 $\epsilon_{i}$-贪婪策略，其中 $\epsilon_{i}=\epsilon^{1+\frac{i}{N-1}\alpha}$，$\epsilon=0.4$，$\alpha=7$。各 $\epsilon_{i}$ 在训练全程保持不变。训练时回合长度上限为 50000 帧。

The capacity of the shared experience replay memory is soft-limited to 2 million transitions: adding new data is always permitted, to not slow down the actors, but every 100 learning steps any excess data above this capacity threshold is removed en masse, in FIFO order. The median actual size of the memory is 2035050. Data is sampled according to proportional prioritization, with a priority exponent of 0.6 and an importance sampling exponent set to 0.4.

共享经验回放容量软限制为 200 万条转移：始终允许写入以免拖慢执行器，但每 100 次学习步会按 FIFO 一次性清掉超出阈值的多余数据。记忆实际大小中位数为 2035050。按比例优先级采样，优先级指数 0.6，重要性采样指数 0.4。

In Figure 2, on the left, we compare the median human normalized score across all 57 games to several baselines: DQN, Prioritized DQN, Distributional DQN (Bellemare et al., 2017), Rainbow, and Gorila. In all cases the performance is measured at the end of training under the no-op starts testing regime (Mnih et al., 2015). On the right, we show initial learning curves (taken from the greediest actor) for a selection of 6 games (full learning curves for all games are in the appendix). Given that Ape-X can harness substantially more computation than most baselines, one might expect it to train faster. Figure 2 shows that this was indeed the case. Perhaps more surprisingly, our agent achieved a substantially higher final performance.

图 2 左侧把 57 款游戏的人类归一化分数中位数与若干基线比较：DQN、Prioritized DQN、Distributional DQN（Bellemare et al., 2017）、Rainbow 与 Gorila。表现均在训练结束、no-op starts 测试体制下度量（Mnih et al., 2015）。右侧给出 6 款游戏的初期学习曲线（取自最贪婪执行器；全部游戏完整曲线见附录）。Ape-X 可用算力远超多数基线，预期训得更快——图 2 确如此；或许更意外的是，最终表现也显著更高。

In Table 1 we compare the median human-normalized performance of Ape-X DQN on the Atari benchmark to corresponding metrics as reported for other baseline agents in their respective publications. Whenever available we report results both for no-op starts and for human starts. The human-starts regime (Nair et al., 2015) corresponds to a more challenging generalization test, as the agent is initialized from random starts drawn from games played by human experts. Ape-X’s performance is higher than the performance of any of the baselines according to both metrics.

表 1 把 Ape-X DQN 在 Atari 基准上的人类归一化中位表现，与各基线论文报告的对应指标比较。凡可用处，我们同时报告 no-op starts 与 human starts。human-starts 体制（Nair et al., 2015）是更难的泛化测试：智能体从人类专家对局中抽取的随机开局起步。按两项指标，Ape-X 的表现都高于任一基线。

Algorithm	Training	Environment	Resources	Median	Median
	Time	Frames	(per game)	(no-op starts)	(human starts)
Ape-X DQN	5 days	22800M	376 cores, 1 GPU a	434%	358%
Rainbow	10 days	200M	1 GPU	223%	153%
Distributional (C51)	10 days	200M	1 GPU	178%	125%
A3C	4 days	—	16 cores	—	117%
Prioritized Dueling	9.5 days	200M	1 GPU	172%	115%
DQN	9.5 days	200M	1 GPU	79%	68%
Gorila DQN c	$\sim$4 days	—	unknown b	96%	78%
UNREAL d	—	250M	16 cores	331% d	250% d

Table 1: Median normalized scores across 57 Atari games. a Tesla P100. b $>$100 CPUs, with a mixed number of cores per CPU machine. c Only evaluated on 49 games. d Hyper-parameters were tuned per game.

算法	训练	环境	资源	中位数	中位数
	时间	帧数	（每游戏）	（no-op starts）	（human starts）
Ape-X DQN	5 天	22800M	376 核，1 GPU a	434%	358%
Rainbow	10 天	200M	1 GPU	223%	153%
Distributional (C51)	10 天	200M	1 GPU	178%	125%
A3C	4 天	—	16 核	—	117%
Prioritized Dueling	9.5 天	200M	1 GPU	172%	115%
DQN	9.5 天	200M	1 GPU	79%	68%
Gorila DQN c	$\sim$4 天	—	未知 b	96%	78%
UNREAL d	—	250M	16 核	331% d	250% d

表 1：57 款 Atari 游戏的中位归一化分数。a Tesla P100。b 超过 100 颗 CPU，每台机器核数不一。c 仅在 49 款游戏上评测。d 超参按游戏调优。

4.2 Continuous Control

4.2 连续控制

In a second set of experiments we evaluated Ape-X DPG on four continuous control tasks. In the manipulator domain the agent must learn to bring a ball to a specified location. In the humanoid domain the agent must learn to control a humanoid body to solve three distinct tasks of increasing complexity: Standing, Walking and Running. Since here we learn from features, rather than from pixels, the observation space is much smaller than it is in the Atari domain. We therefore use small, fully-connected networks (details in the appendix). With 64 actors on this domain, we obtain $\sim$14K total FPS (the same number of transitions per second; here we do not use action repeats). We process 86 batches of 256 transitions per second, or $\sim$22K transitions processed per second.

第二组实验在四项连续控制任务上评测 Ape-X DPG。manipulator 域要求把球带到指定位置；humanoid 域要控制人形身体完成站立、行走、奔跑三项难度递增的任务。此处从特征而非像素学习，观测空间远小于 Atari，因而使用小型全连接网络（细节见附录）。该域用 64 个执行器时总 FPS 约 14K（转移/秒相同；此处不用动作重复）。每秒处理 86 个、每批 256 条转移的批次，即约 22K 转移/秒。

Figure 3 shows that Ape-X DPG achieved very good performance on all four tasks. The figure shows the performance of Ape-X DPG for different numbers of actors: as the number of actors increases our agent becomes increasingly effective at solving these problems rapidly and reliably, outperforming a standard DDPG baseline trained for over 10 times longer. A parallel paper (Barth-Maron et al., 2018) builds on this work by combining Ape-X DPG with distributional value functions, and the resulting algorithm is successfully applied to further continuous control tasks.

图 3 显示 Ape-X DPG 在四项任务上都取得很好表现。图中给出不同执行器数量下的性能：执行器增多时，智能体更快、更可靠地解决这些问题，并超过训练逾十倍时长的标准 DDPG 基线。并行论文（Barth-Maron et al., 2018）在此基础上把 Ape-X DPG 与分布价值函数结合，并成功用于更多连续控制任务。

Figure 3: Performance of Ape-X DPG on four continuous control tasks, as a function of wall clock time. Performance improves as we increase the numbers of actors. The black dashed line indicates the maximum performance reached by a standard DDPG baseline over 5 days of training.

图 3：Ape-X DPG 在四项连续控制任务上相对墙钟时间的表现。执行器增多时性能提升。黑色虚线为标准 DDPG 基线在 5 天训练内达到的最高表现。

5 Analysis

5 分析

Figure 4: Scaling the number of actors. Performance consistently improves as we scale the number of actors from 8 to 256, note that the number of learning updates performed does not depend on the number of actors.

图 4：扩展执行器数量。执行器从 8 增到 256 时性能持续提升；注意学习更新次数并不随执行器数变化。

In this section we describe additional Ape-X DQN experiments on Atari that helped improve our understanding of the framework, and we investigate the contribution of different components.

本节描述在 Atari 上的额外 Ape-X DQN 实验，以加深对框架的理解，并考察不同成分的贡献。

First, we investigated how the performance scales with the number of actors. We trained our agent with different numbers of actors (8, 16, 32, 64, 128 and 256) for 35 hours on a subset of 6 Atari games. In all experiments we kept the size of the shared experience replay memory fixed at 1 million transitions. Figure 4 shows that the performance consistently improved as the number of actors increased. The appendix contains learning curves for additional games, and a comparison of the scalability of the algorithm with and without prioritized replay. It is perhaps surprising that performance improved so substantially purely by increasing the number of actors, without changing the rate at which the network parameters are updated, the structure of the network, or the update rule. We hypothesize that the proposed architecture helps with a common deep reinforcement learning failure mode, in which the policy discovered is a local optimum in the parameter space, but not a global one, e.g., due to insufficient exploration. Using a large number of actors with varying amounts of exploration helps to discover promising new courses of action, and prioritized replay ensures that when this happens, the learning algorithm focuses its efforts on this important information.

首先考察性能如何随执行器数量扩展。我们在 6 款 Atari 子集上，分别用 8、16、32、64、128、256 个执行器训练 35 小时；共享回放容量固定为 100 万条转移。图 4 显示执行器增多时性能持续提升。附录给出更多游戏的学习曲线，以及有无优先回放时的可扩展性对比。或许令人惊讶的是：仅增加执行器、不改网络更新速率、结构或更新规则，性能就大幅提升。我们猜想该架构有助于缓解深度强化学习常见的失败模式——发现的策略是参数空间局部最优而非全局最优，例如因探索不足。大量带不同探索强度的执行器有助于发现有希望的新动作序列；优先回放则保证一旦发现，学习算法会把精力聚焦到这些重要信息上。

Next, we investigated varying the capacity of the replay memory (see Figure 5). We used a setup with 256 actors, for a median of $\sim$37K total environment frames per second (approximately $\sim$9K transitions). With such a large number of actors, the contents of the memory is replaced much faster than in most DQN-like agents. We observed a small benefit to using a larger replay capacity. We hypothesize this is due to the value of keeping some high priority experiences around for longer and replaying them. As above, a single learner machine trained the network with median 19 batches per second, each of 512 transitions, for a median of $\sim$9.7K transitions processed per second.

接着改变回放记忆容量（见图 5）。设定为 256 个执行器，环境帧中位数约 37K/秒（约 9K 转移）。执行器如此之多时，记忆内容被替换得远快于多数类 DQN 智能体。我们观察到更大回放容量有小幅收益，猜想是因为某些高优先级经验能留得更久并被反复回放。同上，单一学习者机器以中位每秒 19 批、每批 512 条转移训练网络，即中位约 9.7K 转移/秒。

Figure 5: Varying the capacity of the replay. Agents with larger replay memories perform better on most games. Each curve corresponds to a single run, smoothed over 20 points. The curve for Wizard Of Wor with replay size 250K is incomplete because training diverged; we did not observe this with the other replay sizes.

图 5：改变回放容量。多数游戏上，更大回放记忆的智能体表现更好。每条曲线为单次运行、经 20 点平滑。Wizard Of Wor 在回放大小 250K 时曲线不完整，因训练发散；其他回放大小未观察到该现象。

Finally, we ran additional experiments to disentangle potential effects of two confounding factors in our scalability analysis: recency of the experience data in the replay memory, and diversity of the data-generating policies. The full description of these experiments is confined to the appendix; to summarize, neither factor alone is sufficient to explain the performance we see. We therefore conclude that the results are due substantially to the positive effects of gathering more experience data; namely better exploration of the environment and better avoidance of overfitting.

最后，我们做额外实验，拆开可扩展性分析中两个可能混淆因素：回放中经验的新旧程度，以及数据生成策略的多样性。完整描述见附录；简言之，任一因素单独都不足以解释所见性能。因此我们结论是：结果很大程度上来自收集更多经验数据的正面效应，即更好的环境探索与更好的过拟合抑制。

6 Conclusion

6 结论

We have designed, implemented, and analyzed a distributed framework for prioritized replay in deep reinforcement learning. This architecture achieved state of the art results in a wide range of discrete and continuous tasks, both in terms of wall-clock learning speed and final performance.

我们设计、实现并分析了深度强化学习中优先回放的分布式框架。该架构在广泛的离散与连续任务上达到当时前沿，无论墙钟学习速度还是最终表现皆然。

In this paper we focused on applying the Ape-X framework to DQN and DPG, but it could also be combined with any other off-policy reinforcement learning update. For methods that use temporally extended sequences (Mnih et al., 2016; Wang et al., 2017, e.g., ), the Ape-X framework may be adapted to prioritize sequences of past experiences instead of individual transitions.

本文聚焦把 Ape-X 框架用于 DQN 与 DPG，但它也可与任何其他离策略强化学习更新结合。对使用时间扩展序列的方法（例如 Mnih et al., 2016；Wang et al., 2017），Ape-X 框架可改为优先排序过去经验序列而非单条转移。

Ape-X is designed for regimes in which it is possible to generate large quantities of data in parallel. This includes simulated environments but also a variety of real-world applications, such as robotic arm farms, self-driving cars, online recommender systems, or other multi-user systems in which data is generated by many instances of the same environment (Silver et al., 2013, c.f.). In applications where data is costly to obtain, our approach will not be directly applicable. With powerful function approximators, overfitting is an issue: generating more training data is the simplest way of addressing it, but may also provide guidance towards data-efficient solutions.

Ape-X 面向能够并行生成大量数据的体制，包括仿真环境，也包括机器人臂农场、自动驾驶、在线推荐或其他由同一环境多实例生成数据的多用户系统（Silver et al., 2013, c.f.）。数据获取昂贵的应用中，本方法无法直接套用。有强函数逼近器时过拟合是问题：生成更多训练数据是最简单的应对，也可能为数据高效方案提供指引。

Many deep reinforcement learning algorithms are fundamentally limited by their ability to explore effectively in large domains. Ape-X uses a naive yet effective mechanism to address this issue: generating a diverse set of experiences and then identifying and learning from the most useful events. The success of this approach suggests that simple and direct approaches to exploration may be feasible, even for synchronous agents.

许多深度强化学习算法根本上受限于在大域中有效探索的能力。Ape-X 用朴素却有效的机制应对：生成多样经验，再识别并从最有用事件中学习。其成功表明，即便对同步智能体，简单直接的探索途径也可能可行。

Our architecture illustrates that distributed systems are now practical both for research and, potentially, large-scale applications of deep reinforcement learning. We hope that the algorithms, architecture, and analysis we have presented will help to accelerate future efforts in this direction.

我们的架构说明：分布式系统现已对深度强化学习研究、乃至潜在的大规模应用都切实可用。希望本文给出的算法、架构与分析，能加速该方向的后续工作。

Acknowledgments

We would like to acknowledge the contributions of our colleagues at DeepMind, whose input and support has been vital to the success of this work. Thanks in particular to Tom Schaul, Joseph Modayil, Sriram Srinivasan, Georg Ostrovski, Josh Abramson, Todd Hester, Jean-Baptiste Lespiau, Alban Rrustemi and Dan Belov.

致谢

我们感谢 DeepMind 同事的贡献，他们的意见与支持对本工作成功至关重要。特别感谢 Tom Schaul、Joseph Modayil、Sriram Srinivasan、Georg Ostrovski、Josh Abramson、Todd Hester、Jean-Baptiste Lespiau、Alban Rrustemi 与 Dan Belov。

References

参考文献

Abadi et al. (2016) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 , 2016.

Alain et al. (2015) Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv:1511.06481 , 2015.

Babaeizadeh et al. (2017) Mohammad Babaeizadeh, Iuri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. Reinforcement learning through asynchronous advantage actor-critic on a gpu. In International Conference on Learning Representations , 2017.

Barth-Maron et al. (2018) Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributional policy gradients. In International Conference on Learning Representations , 2018.

Bellemare et al. (2013) Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res. (JAIR) , 47:253–279, 2013.

Bellemare et al. (2017) Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning , 2017.

Clemente et al. (2017) Alfredo V. Clemente, Humberto Nicolás Castejón Martínez, and Arjun Chandra. Efficient parallel methods for deep reinforcement learning. arXiv preprint arXiv:1705.04862 , 2017.

Dean et al. (2012) Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed deep networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems , NIPS’12, pp. 1223–1231, USA, 2012. Curran Associates Inc.

Deng et al. (2009) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09 , 2009.

Hastings (1970) W Keith Hastings. Monte carlo sampling methods using markov chains and their applications. Biometrika , 57(1):97–109, 1970.

Heess et al. (2017) Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin A. Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 , 2017.

Hessel et al. (2017) Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv:1710.02298 , 2017.

Hester et al. (2017) Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Learning from demonstrations for real world reinforcement learning. CoRR , abs/1704.03732, 2017.

Hinton (2007) Geoffrey E Hinton. To recognize shapes, first learn to generate images. Prog. Brain Res. , 165:535–547, 2007.

Jaderberg et al. (2017) Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations , 2017.

Kaiser et al. (2017) Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. CoRR , abs/1706.05137, 2017.

Kingma & Ba (2014) Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Krizhevsky (2014) Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997 , 2014.

Lillicrap et al. (2016) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations , 2016.

Lin (1992) Long-H Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning , 1992.

Loshchilov & Hutter (2015) Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint arXiv:1511.06343 , 2015.

Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529–533, 26 February 2015.

Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning , pp. 1928–1937, 11 June 2016.

Moore & Atkeson (1993) Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning , 13(1):103–130, 1993.

Nair et al. (2015) Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu, and David Silver. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296 , 15 July 2015.

Prokhorov & Wunsch (1997) Danil V Prokhorov and Donald C Wunsch. Adaptive critic designs. IEEE transactions on Neural Networks , 8(5):997–1007, 1997.

Riedmiller (2005) Martin Riedmiller. Neural Fitted Q Iteration – First Experiences with a Data Efficient Neural Reinforcement Learning Method , pp. 317–328. Springer Berlin Heidelberg, Berlin, Heidelberg, 2005. ISBN 978-3-540-31692-3. doi: 10.1007/11564096_32 .

Schaul et al. (2016) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations , 2016.

Silver et al. (2013) David Silver, Leonard Newnham, David Barker, Suzanne Weller, and Jason McFall. Concurrent reinforcement learning from customer interactions. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning , volume 28 of Proceedings of Machine Learning Research , pp. 924–932, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.

Silver et al. (2014) David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning (ICML-14) , pp. 387–395, 2014.

Silver et al. (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484–489, 2016.

Sutton (1988) Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3(1):9–44, 1988.

Sutton & Barto (1998) Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press Cambridge, 1998.

Sutton & Barto (2017) Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, 2nd edition . 2017.

Tassa et al. (2018) Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018.

Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on , pp. 5026–5033. IEEE, 2012.

Uhlenbeck & Ornstein (1930) George E Uhlenbeck and Leonard S Ornstein. On the theory of the brownian motion. Physical review , 36(5):823, 1930.

van Hasselt (2010) Hado van Hasselt. Double Q-learning. In Advances in Neural Information Processing Systems , pp. 2613–2621, 2010.

van Hasselt et al. (2016) Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. In Advances in Neural Information Processing Systems , 2016.

Wang et al. (2016) Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning , 2016.

Wang et al. (2017) Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. In International Conference on Learning Representations , 2017.

Werbos (1990) Paul J Werbos. A menu of designs for reinforcement learning over time. Neural networks for control , pp. 67–95, 1990.

Appendix A Recency of Experience

附录 A 经验的新旧程度

Figure 6: Testing whether improved performance is caused by recency alone: $n$ denotes the number of actors, $k$ the number of times each transition is replicated in the replay. The data in the run with $n=32$, $k=8$ is therefore as recent as the data in the run with $n=256$, $k=1$, but performance is not as good.

图 6：检验性能提升是否仅由新旧程度导致：$n$ 为执行器数，$k$ 为每条转移在回放中复制次数。因此 $n=32$、$k=8$ 的运行中数据新旧程度与 $n=256$、$k=1$ 相当，但表现不如后者。

In our main experiments we do not change the size of the replay memory in proportion to the number of actors, so by changing the number of actors we also increased the rate at which the contents of the replay memory is replaced. This means that in the experiments with more actors, transitions in the replay memory are more recent: they are generated by following policies whose parameters are closer to version of the parameters being optimized by the learner, and in this sense they are more on-policy. Could this alone be sufficient to explain the improved performance? If so, we might be able to recover the results without needing a large number of actor machines. To test this, we constructed an experiment wherein we replicate the rate at which the contents of the replay memory is replaced in the 256-actor experiments, but instead of actually using 256 actors, we use 32 actors but add each transition they generate to the replay memory 8 times over. In this setup, the contents of the replay memory is similarly generated by policies with a recent version of the network parameters: the only difference is that the data is not as diverse as in the 256-actor case. We observe (see Figure 7) that this does not recover the same performance, and therefore conclude that the recency of the experience alone is not sufficient to explain the performance of our method. Indeed, we see that adding the same data multiple times can sometimes harm performance, since although it increases recency this comes at the expense of diversity.

主实验中我们并未按执行器数成比例放大回放容量，因而增加执行器也会加快回放内容被替换的速率。执行器更多时，回放中的转移更新：它们由参数更接近学习者正在优化版本的策略生成，在此意义上更「在策略」。仅此是否足以解释性能提升？若是，或许无需大量执行器机器即可复现结果。为此我们构造实验：复制 256 执行器实验中回放内容被替换的速率，但不真用 256 个执行器，而是用 32 个，并把它们生成的每条转移向回放写入 8 次。该设定下，回放内容同样由带近期网络参数的策略生成；唯一差别是数据多样性不如 256 执行器。我们观察到（见图 7）这并不能复现同等表现，因而结论是：仅靠经验新旧不足以解释方法性能。实际上，多次写入同一数据有时还会损害表现——虽提高了新旧程度，却牺牲了多样性。

Note: in principle, duplicating the added data in this fashion has a similar effect to reducing the capacity of the replay memory, and indeed, our results with a smaller replay memory in Figure 5 do corroborate the finding. However, we test also by duplicating the data primarily in order to exclude any effects arising from the implementation. In particular, in contrast to simply reducing the replay capacity, duplicating each data point means that the computational demands on the replay server in these runs are the same as when we use the corresponding number of real actors.

注：原则上，如此复制新增数据，效果类似缩小回放容量；图 5 中较小回放记忆的结果也印证这一点。我们仍以复制数据为主来排除实现层面的效应：与单纯缩小回放不同，复制每个数据点使这些运行对回放服务器的计算需求，与使用相应数量真实执行器时相同。

Appendix B Varying the Data-Generating Policies

附录 B 改变数据生成策略

Another factor that could conceivably contribute to the scalability of our algorithm is the fact that each actor has a different $\epsilon$. To determine the extent to which this impacts upon the performance, we ran an experiment (see Figure 7) with some simple variations on the mechanism we use to choose the policies that generate the data we train on. The first alternative we tested is to choose a small fixed set of 6 values for $\epsilon$, instead of the full range that we typically use. In this test, we use prioritized replay as normal, and we find that the results with the full range of $\epsilon$ are overall slightly better. However, it is not essential for achieving good results within our distributed framework.

另一可能促成算法可扩展性的因素是：每个执行器有不同的 $\epsilon$。为衡量其影响程度，我们做实验（见图 7），对选择训练数据生成策略的机制做若干简单变体。第一种替代是选用固定的 6 个 $\epsilon$ 值，而非通常的完整范围。该测试仍用正常优先回放，发现完整 $\epsilon$ 范围整体略好，但在分布式框架内取得好结果并非必需。

Appendix C Atari: Additional Details

附录 C Atari：补充细节

The frames received from the environment are preprocessed on the actor side with the standard transformations introduced by DQN. This includes greyscaling, frame stacking, repeating actions 4 times, and clipping rewards to $[-1,1]$.

环境收到的帧在执行器侧按 DQN 引入的标准变换预处理，包括灰度化、帧堆叠、动作重复 4 次，以及把奖励裁剪到 $[-1,1]$。

The learner waits for at least 50000 transitions to be accumulated in the replay before starting learning. We use a Centered RMSProp optimizer with a learning rate of 0.00025 / 4, decay of 0.95, epsilon of 1.5e-7, and no momentum to minimize the multi-step loss (with $n=3$). Gradient norms are clipped to 40. The target network used in the loss calculation is copied from the online network every 2500 training batches. We use the same network as in the Dueling DDQN agent.

学习者等到回放至少积累 50000 条转移后才开始学习。我们用 Centered RMSProp，学习率 0.00025 / 4，衰减 0.95，epsilon 1.5e-7，无动量，最小化多步损失（$n=3$）。梯度范数裁剪到 40。损失计算中的目标网络每 2500 个训练批次从在线网络拷贝一次。网络结构与 Dueling DDQN 智能体相同。

Appendix D Continuous Control: Additional Details

附录 D 连续控制：补充细节

The critic network has a layer with 400 units, followed by a tanh activation, followed by another layer of 300 units. The actor network has a layer with 300 units, followed by a tanh activation, followed by another layer of 200 units. The gradient used to update the actor network is clipped to $[-1,1]$, element-wise. Training uses the Adam optimizer (Kingma & Ba (2014)) with learning rate of $0.0001$. The target network used in the loss calculation is copied from the online network every 100 training batches.

评论家网络：400 单元层、tanh、再 300 单元层。执行器网络：300 单元层、tanh、再 200 单元层。用于更新执行器网络的梯度按元素裁剪到 $[-1,1]$。训练用 Adam（Kingma & Ba, 2014），学习率 $0.0001$。损失计算中的目标网络每 100 个训练批次从在线网络拷贝一次。

Replay sampling priorities are set according to the absolute TD error as given by the critic, and are sampled by the learner using proportional prioritized sampling (see appendix F) with priority exponent $\alpha_{\mathrm{sample}}=0.6$. To maintain a fixed replay capacity of $10^{6}$, transitions are periodically evicted using proportional prioritized sampling, with priority exponent $\alpha_{\mathrm{evict}}=-0.4$. This is a different strategy for removing data than in the Atari experiments, which simply removed the oldest data first - it remains to be seen which is superior.

回放采样优先级按评论家给出的绝对 TD 误差设定，学习者用比例优先采样（见附录 F），优先级指数 $\alpha_{\mathrm{sample}}=0.6$。为维持 $10^{6}$ 的固定回放容量，定期用比例优先采样淘汰转移，优先级指数 $\alpha_{\mathrm{evict}}=-0.4$。这与 Atari 实验中「先删最旧数据」的策略不同——何者更优仍有待观察。

Unlike the original DPG algorithm which applies autocorrelated noise sampled from a Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein (1930)), we apply exploration noise to each action sampled from a normal distribution with $\sigma=0.3$. Evaluation is performed using the noiseless deterministic policy. Hyperparameters are otherwise as per DQN.

与原版 DPG 使用 Ornstein-Uhlenbeck 过程（Uhlenbeck & Ornstein, 1930）采样自相关噪声不同，我们对每个动作施加 $\sigma=0.3$ 的正态探索噪声。评估使用无噪声确定性策略。其余超参与 DQN 一致。

Benchmarking was performed in two continuous control domains ((a) Humanoid and (b) Manipulator, see Figure 8) implemented in the MuJoCo physics simulator (Todorov et al. (2012)). Humanoid is a humanoid walker with action, state and observation dimensionalities $|\mathcal{A}|=21$, $|\mathcal{S}|=55$ and $|\mathcal{O}|=67$ respectively. Three Humanoid tasks were considered: walk (reward for exceeding a minimum velocity), run (reward proportional to movement speed) and stand (reward proportional to standing height). Manipulator is a 2-dimensional planar arm with $|\mathcal{A}|=2$, $|\mathcal{S}|=22$ and $|\mathcal{O}|=37$, which receives reward for catching a randomly-initialized moving ball.

基准在 MuJoCo 物理仿真器（Todorov et al., 2012）实现的两个连续控制域上进行：（a）Humanoid 与（b）Manipulator，见图 8。Humanoid 是人形行走体，动作 / 状态 / 观测维数分别为 $|\mathcal{A}|=21$、$|\mathcal{S}|=55$、$|\mathcal{O}|=67$。考虑三项 Humanoid 任务：walk（超过最低速度给奖）、run（奖励与移动速度成正比）、stand（奖励与站立高度成正比）。Manipulator 是二维平面臂，$|\mathcal{A}|=2$、$|\mathcal{S}|=22$、$|\mathcal{O}|=37$，接住随机初始化运动球时给奖。

(a) Humanoid domain.

（a）Humanoid 域。

Appendix E Tuning

附录 E 调参

On Atari, we performed some limited tuning of the learning rate and batch size: we found that larger batch sizes contribute significantly to performance, when using many actors. We tried batch sizes from {32, 128, 256, 512, 1024}, seeing clear benefits up to 512. We attempted increasing the learning rate to 0.00025 with the larger batch sizes but this destabilized training on some games. We also tried a lower learning rate of 0.00025 / 8, but this did not reliably improve results.

在 Atari 上我们对学习率与批大小做了有限调参：发现执行器很多时，更大批大小对性能贡献显著。我们试了 {32, 128, 256, 512, 1024}，直至 512 都有清晰收益。试图在更大批次上把学习率提到 0.00025，但部分游戏训练不稳定。也试过更低的 0.00025 / 8，但未能可靠提升结果。

Likewise for continuous control, we experimented with batch sizes {32, 128, 256, 512, 1024} and learning rates from $10^{-3}$ to $10^{-5}$. We also experimented with the prioritization exponents $\alpha$ from $0.0$ to $1.0$, with results proving essentially consistent within the range [0.3, 0.7] (beyond 0.7, training would sometimes become unstable and diverge).

连续控制同样试了批大小 {32, 128, 256, 512, 1024} 与学习率 $10^{-3}$ 到 $10^{-5}$。优先级指数 $\alpha$ 从 $0.0$ 到 $1.0$ 也做了试验，在 $[0.3, 0.7]$ 内结果基本一致（超过 0.7 时训练有时会不稳并发散）。

For the experiments with many actors, we set the period for updating network parameters on the actors to be high enough that the learner was not overloaded with requests, and we set the number of transitions that are locally accumulated on each actor to be high enough that the replay server would not be overloaded with network traffic, but we did not otherwise tune those parameters and have not observed them to have significant impact on the learning dynamics.

多执行器实验中，我们把执行器更新网络参数的周期设得足够高，以免学习者被请求压垮；并把各执行器本地累积的转移数设得足够高，以免回放服务器被网络流量压垮。除此之外未再调这些参数，也未观察到它们对学习动态有显著影响。

Appendix F Implementation

附录 F 实现

The following section makes explicit some of the more practical details that may be of interest to anyone wishing to implement a similar system.

以下章节明确若干更实践的细节，供希望实现类似系统者参考。

Data Storage

The algorithm is implemented using TensorFlow (Abadi et al., 2016). Replay data is kept in a distributed in-memory key-value store implemented using custom TensorFlow ops, similar to the lookup ops available in core TensorFlow. The ops allow adding, reading, and removing batches of Tensor data efficiently.

数据存储

算法用 TensorFlow（Abadi et al., 2016）实现。回放数据保存在用自定义 TensorFlow op 实现的分布式内存键值存储中，类似核心 TensorFlow 中的查找 op。这些 op 允许高效地批量添加、读取与删除 Tensor 数据。

Sampling Data

We also implemented ops for efficiently maintaining and sampling from a prioritized distribution over the keys, using the algorithm for proportional prioritization described in Schaul et al. (2016). The probability of sampling a transition is $p_{k}^{\alpha}/\sum_{k}p_{k}^{\alpha}$ where $p_{k}$ is the priority of the transition with key $k$. The exponent $\alpha$ controls the amount of prioritization, and when $\alpha=0$ uniform sampling is recovered. The proportional variant sets priority $p_{k}=|\delta_{k}|$ where $\delta_{k}$ is the TD error for transition $k$. Whenever a batch of data is added to or removed from the store, or is processed by the learner, this distribution is correspondingly updated, recording any change to the set of valid keys and the priorities associated with them.

采样数据

我们还实现了用于高效维护并按键的优先分布采样的 op，采用 Schaul et al.（2016）描述的比例优先算法。采样转移的概率为 $p_{k}^{\alpha}/\sum_{k}p_{k}^{\alpha}$，其中 $p_{k}$ 为键 $k$ 对应转移的优先级。指数 $\alpha$ 控制优先程度，$\alpha=0$ 时退回均匀采样。比例变体设 $p_{k}=|\delta_{k}|$，$\delta_{k}$ 为转移 $k$ 的 TD 误差。每当一批数据加入或移出存储、或被学习者处理，该分布就相应更新，记录有效键集合及其优先级的变化。

A background thread on the learner fetches batches of sampled data from the remote replay and decompresses it using the learner’s CPU, in parallel with the gradients being computed on the GPU. The fetched data is buffered in a TensorFlow queue, so that the GPU always has data available to train on.

学习者上的后台线程从远程回放拉取采样批次，并用学习者 CPU 解压，与 GPU 上的梯度计算并行。拉取的数据缓存在 TensorFlow 队列中，保证 GPU 始终有数据可训。

Adding Data

In order to efficiently construct $n$-step transition data, each actor maintains a circular buffer of capacity $n$ containing tuples $(S_{t},A_{t},R_{t:t+B},\gamma_{t:t+B},q(S_{t},*))$, where $B$ is the current size of the buffer. With each step, the new data is appended and the accumulated per-step discounts $\gamma_{t:t+B}$ and partial returns $R_{t:t+B}$ for all entries in the buffer are updated. If the buffer has reached its capacity, $n$, then its first element may be combined with the latest state $S_{t+n}$ and value estimates $q(S_{t+n})$ to produce a valid $n$-step transition (with accompanying Q-values).

添加数据

为高效构造 $n$ 步转移数据，每个执行器维护容量为 $n$ 的环形缓冲，存放元组 $(S_{t},A_{t},R_{t:t+B},\gamma_{t:t+B},q(S_{t},*))$，其中 $B$ 为缓冲当前大小。每一步追加新数据，并更新缓冲中所有条目的累积逐步折扣 $\gamma_{t:t+B}$ 与部分回报 $R_{t:t+B}$。缓冲达到容量 $n$ 后，其首元素可与最新状态 $S_{t+n}$ 及价值估计 $q(S_{t+n})$ 组合，得到合法的 $n$ 步转移（附带 Q 值）。

However, instead of being directly added to the remote replay memory on each step, the constructed transitions $(S_{t},A_{t},R_{t:t+B},\gamma_{t:t+B},S_{t+n},q(S_{t},*),q(S_{t+n},*))$ are first stored in a local TensorFlow queue, in order to reduce the number of requests to the replay server. The queue is periodically flushed, at which stage the absolute $n$-step TD-errors (and thus the initial priorities) for the queued transitions are computed in batch, using the buffered Q-values to avoid recomputation. The Q-value estimates from which the initial priorities are derived are therefore based on the actor’s copy of the network parameters at the time the corresponding state was obtained from the environment, rather than the latest version on the learner. These Q-values need not be stored after this, since the learner does not require them, although they can be helpful for debugging.

不过，构造好的转移 $(S_{t},A_{t},R_{t:t+B},\gamma_{t:t+B},S_{t+n},q(S_{t},*),q(S_{t+n},*))$ 并不在每一步直接写入远程回放，而是先存入本地 TensorFlow 队列，以减少对回放服务器的请求。队列定期刷新；此时用缓冲的 Q 值批量计算队列中转移的绝对 $n$ 步 TD 误差（亦即初始优先级），避免重算。因此导出初始优先级的 Q 值估计，基于的是获得对应状态时执行器本地的网络参数副本，而非学习者上的最新版本。这些 Q 值此后不必再存——学习者不需要它们——但对调试仍有帮助。

A unique key is assigned to each transition, which records which actor and environment step it came from, and the dequeued transition tuples are stored in the remote replay memory. As mentioned in the previous section, the remote sampling distribution is immediately updated with the newly added keys and the corresponding initial priorities computed by the actor. Note that, since we store both the start and the end state with each transition, we are storing some data twice: this costs more RAM, but simplifies the code.

每条转移分配唯一键，记录来自哪个执行器与环境步；出队的转移元组存入远程回放。如前节所述，远程采样分布立即用新键及执行器算出的初始优先级更新。注意：由于每条转移同时存起点与终点状态，部分数据存了两份——更耗 RAM，但简化代码。

Contention

It is important that the replay server be able to handle all requests in a timely fashion, in order to avoid slowing down the whole system. Possible bottlenecks include CPU, network bandwidth, and any locks protecting the shared data. In our experiments we found CPU to be the main bottleneck, but this was resolved by ensuring all requests and responses use sufficiently large batches. Nonetheless, it is advisable to consider all of these potential performance concerns when designing such systems.

争用

回放服务器必须能及时处理全部请求，以免拖慢整个系统。可能瓶颈包括 CPU、网络带宽，以及保护共享数据的锁。实验中我们发现 CPU 是主要瓶颈，但通过确保请求与响应都使用足够大的批次即可解决。设计此类系统时，仍建议同时考虑所有这些潜在性能问题。

Asynchronicity

In our framework, since acting and learning proceed with no synchronization, and performance depends on both, it can be misleading to consider performance with reference to only one of these. For example, the results after a given total number of environment frames have been experienced are highly dependent on the number of updates the learner has performed in that time. For this reason it is important to monitor and report the speeds of all parts of the system and to consider them when analyzing results.

异步性

在我们的框架中，执行与学习不同步推进，性能依赖二者，因此只相对其中一侧谈性能可能误导。例如，经历给定总环境帧数后的结果，高度依赖学习者在这段时间做了多少次更新。因此监控并报告系统各部分速度，并在分析结果时加以考虑，十分重要。

Failure Tolerance

In distributed systems with many workers, it is inevitable that interruptions or failures will occur, either due to occasional hardware issues or because shared resources are needed by higher priority jobs. All stateful parts of the system therefore must periodically save their work and be able to resume where they left off when restarted. In our system, actors may be interrupted at any time and this will not prevent continued learning, albeit with a temporarily reduced rate of new data entering the replay memory. If the replay server is interrupted, the data it contains is discarded, and upon resuming, the memory is refilled quickly by the actors. In this event, to avoid overfitting, the learner will pause training briefly, until the minimum amount of data has once again been accumulated. If the learner is interrupted, progress will stall until it resumes.

故障容忍

在多工作进程的分布式系统中，中断或故障不可避免，或因偶发硬件问题，或因共享资源被更高优先级作业占用。因此系统中所有有状态部分都必须定期保存工作，并能在重启后从断点恢复。我们的系统中，执行器可随时中断而不阻止继续学习，只是新数据进入回放的速率会暂时降低。若回放服务器中断，其所含数据丢弃；恢复后由执行器迅速重新填满记忆。此时为避免过拟合，学习者会短暂暂停训练，直到再次积累到最少数据量。若学习者中断，进度会停滞直至其恢复。

Figure 9: Training curves for 57 Atari games (performance against wall clock time). Green: DQN baseline. Purple: Rainbow baseline. Orange: A3C baseline. Blue: Ape-X DQN with 360 actors, 1 replay server and 1 Tesla P100 GPU learner. The anomaly in Riverraid is due to an infrastructure error.

图 9：57 款 Atari 游戏的训练曲线（表现对墙钟时间）。绿：DQN 基线。紫：Rainbow 基线。橙：A3C 基线。蓝：360 执行器、1 台回放服务器与 1 块 Tesla P100 GPU 学习者的 Ape-X DQN。Riverraid 中的异常由基础设施错误引起。

Figure 10: Training curves for 57 Atari games (performance against environment frames). Only the first billion frames are shown, corresponding to 5-6 hours of training for Ape-X. Green: DQN baseline. Purple: Rainbow baseline. Blue: ApeX-DQN with 360 actors, 1 replay server and 1 Tesla P100 GPU learner.

图 10：57 款 Atari 游戏的训练曲线（表现对环境帧）。仅展示前十亿帧，对应 Ape-X 约 5–6 小时训练。绿：DQN 基线。紫：Rainbow 基线。蓝：360 执行器、1 台回放服务器与 1 块 Tesla P100 GPU 学习者的 ApeX-DQN。

Figure 11: Speed of data generation scales linearly with the number of actors.

图 11：数据生成速度随执行器数近似线性扩展。

Figure 12: Training curves showing performance against wall clock time for various numbers of actors on a selection of Atari games. Blue: prioritized replay, with learning rate 0.00025 / 4. Red: uniform replay, with learning rate 0.00025. For both prioritized and uniform, we tried both of these learning rates and selected the best. Both variants benefit from larger numbers of actors, but prioritized can better take advantage of the increased amount of data. In the 256-actor run, prioritized is equal or better in 7 of 9 games.

图 12：若干 Atari 游戏上，不同执行器数量下表现对墙钟时间的训练曲线。蓝：优先回放，学习率 0.00025 / 4。红：均匀回放，学习率 0.00025。两种设定都试过这两种学习率并取最佳。两种变体都受益于更多执行器，但优先回放更能利用增加的数据量。256 执行器运行中，优先回放在 9 款游戏里的 7 款上持平或更好。

Game	No-op starts	Human starts
alien	40,804.9	17,731.5
amidar	8,659.2	1,047.3
assault	24,559.4	24,404.6
asterix	313,305.0	283,179.5
asteroids	155,495.1	117,303.4
atlantis	944,497.5	918,714.5
bank_heist	1,716.4	1,200.8
battle_zone	98,895.0	92,275.0
beam_rider	63,305.2	72,233.7
berzerk	57,196.7	55,598.9
bowling	17.6	30.2
boxing	100.0	80.9
breakout	800.9	756.5
centipede	12,974.0	5,711.6
chopper_command	721,851.0	576,601.5
crazy_climber	320,426.0	263,953.5
defender	411,943.5	399,865.3
demon_attack	133,086.4	133,002.1
double_dunk	23.5	22.3
enduro	2,177.4	2,042.4
fishing_derby	44.4	22.4
freeway	33.7	29.0
frostbite	9,328.6	6,511.5
gopher	120,500.9	121,168.2
gravitar	1,598.5	662.0
hero	31,655.9	26,345.3
ice_hockey	33.0	24.0
jamesbond	21,322.5	18,992.3
kangaroo	1,416.0	577.5
krull	11,741.4	8,592.0
kung_fu_master	97,829.5	72,068.0
montezuma_revenge	2,500.0	1,079.0
ms_pacman	11,255.2	6,135.4
name_this_game	25,783.3	23,829.9
phoenix	224,491.1	188,788.5
pitfall	-0.6	-273.3
pong	20.9	18.7
private_eye	49.8	864.7
qbert	302,391.3	380,152.1
riverraid	63,864.4	49,982.8
road_runner	222,234.5	127,111.5
robotank	73.8	68.5
seaquest	392,952.3	377,179.8
skiing	-10,789.9	-11,359.3
solaris	2,892.9	3,115.9
space_invaders	54,681.0	50,699.3
star_gunner	434,342.5	432,958.0
surround	7.1	5.5
tennis	23.9	23.0
time_pilot	87,085.0	71,543.0
tutankham	272.6	127.7
up_n_down	401,884.3	347,912.2
venture	1,813.0	935.5
video_pinball	565,163.2	873,988.5
wizard_of_wor	46,204.0	46,897.0
yars_revenge	148,594.8	131,701.1
zaxxon	42,285.5	37,672.0

Table 2: Scores obtained by Ape-X DQN in final evaluation, under the standard no-op starts and human starts regimes. In some games the scores are higher than in the training curves: this is because the maximum episode length is shorter during training.

游戏	No-op starts	Human starts
alien	40,804.9	17,731.5
amidar	8,659.2	1,047.3
assault	24,559.4	24,404.6
asterix	313,305.0	283,179.5
asteroids	155,495.1	117,303.4
atlantis	944,497.5	918,714.5
bank_heist	1,716.4	1,200.8
battle_zone	98,895.0	92,275.0
beam_rider	63,305.2	72,233.7
berzerk	57,196.7	55,598.9
bowling	17.6	30.2
boxing	100.0	80.9
breakout	800.9	756.5
centipede	12,974.0	5,711.6
chopper_command	721,851.0	576,601.5
crazy_climber	320,426.0	263,953.5
defender	411,943.5	399,865.3
demon_attack	133,086.4	133,002.1
double_dunk	23.5	22.3
enduro	2,177.4	2,042.4
fishing_derby	44.4	22.4
freeway	33.7	29.0
frostbite	9,328.6	6,511.5
gopher	120,500.9	121,168.2
gravitar	1,598.5	662.0
hero	31,655.9	26,345.3
ice_hockey	33.0	24.0
jamesbond	21,322.5	18,992.3
kangaroo	1,416.0	577.5
krull	11,741.4	8,592.0
kung_fu_master	97,829.5	72,068.0
montezuma_revenge	2,500.0	1,079.0
ms_pacman	11,255.2	6,135.4
name_this_game	25,783.3	23,829.9
phoenix	224,491.1	188,788.5
pitfall	-0.6	-273.3
pong	20.9	18.7
private_eye	49.8	864.7
qbert	302,391.3	380,152.1
riverraid	63,864.4	49,982.8
road_runner	222,234.5	127,111.5
robotank	73.8	68.5
seaquest	392,952.3	377,179.8
skiing	-10,789.9	-11,359.3
solaris	2,892.9	3,115.9
space_invaders	54,681.0	50,699.3
star_gunner	434,342.5	432,958.0
surround	7.1	5.5
tennis	23.9	23.0
time_pilot	87,085.0	71,543.0
tutankham	272.6	127.7
up_n_down	401,884.3	347,912.2
venture	1,813.0	935.5
video_pinball	565,163.2	873,988.5
wizard_of_wor	46,204.0	46,897.0
yars_revenge	148,594.8	131,701.1
zaxxon	42,285.5	37,672.0

表 2：Ape-X DQN 最终评估分数（标准 no-op starts 与 human starts 体制）。部分游戏分数高于训练曲线：因训练时最大回合长度更短。