Proximal Policy Optimization Algorithms

近端策略优化算法（Proximal Policy Optimization Algorithms）

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov
OpenAI
{joschu, filip, prafulla, alec, oleg}@openai.com

John Schulman、Filip Wolski、Prafulla Dhariwal、Alec Radford、Oleg Klimov
OpenAI
{joschu, filip, prafulla, alec, oleg}@openai.com

Abstract

摘要

We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.

我们提出一类新的强化学习策略梯度方法：它们在与环境交互采样数据、以及用随机梯度上升优化“代理”目标函数之间交替进行。标准策略梯度方法对每批数据只做一次梯度更新，而我们提出的新目标函数允许多个 epoch 的小批量更新。我们称之为近端策略优化（proximal policy optimization，PPO）的新方法，具有信任域策略优化（trust region policy optimization，TRPO）的部分优点，但实现简单得多、更通用，并且（在经验上）具有更好的样本复杂度。我们在包括模拟机器人运动和 Atari 游戏在内的一组基准任务上测试 PPO，结果表明 PPO 优于其他在线策略梯度方法，并在样本复杂度、简洁性与墙钟时间之间取得了有利的平衡。

1 Introduction

1 引言

In recent years, several different approaches have been proposed for reinforcement learning with neural network function approximators. The leading contenders are deep Q-learning [Mni+15], "vanilla" policy gradient methods [Mni+16], and trust region / natural policy gradient methods [Sch+15b]. However, there is room for improvement in developing a method that is scalable (to large models and parallel implementations), data efficient, and robust (i.e., successful on a variety of problems without hyperparameter tuning). Q-learning (with function approximation) fails on many simple problems and is poorly understood, vanilla policy gradient methods have poor data efficiency and robustness; and trust region policy optimization (TRPO) is relatively complicated, and is not compatible with architectures that include noise (such as dropout) or parameter sharing (between the policy and value function, or with auxiliary tasks).

近年来，已有若干不同方法被提出用于带神经网络函数逼近器的强化学习。主要竞争者是深度 Q 学习 [Mni+15]、“普通”策略梯度方法 [Mni+16]，以及信任域／自然策略梯度方法 [Sch+15b]。然而，在开发一种可扩展（到大模型和并行实现）、数据高效且稳健（即无需调参就能在多种问题上成功）的方法方面，仍有改进空间。带函数逼近的 Q 学习在许多简单问题上失败，且理解不足；普通策略梯度方法的数据效率与稳健性较差；而信任域策略优化（TRPO）相对复杂，并且与包含噪声（如 dropout）或参数共享（策略与价值函数之间，或与辅助任务）的架构不兼容。

This paper seeks to improve the current state of affairs by introducing an algorithm that attains the data efficiency and reliable performance of TRPO, while using only first-order optimization. We propose a novel objective with clipped probability ratios, which forms a pessimistic estimate (i.e., lower bound) of the performance of the policy. To optimize policies, we alternate between sampling data from the policy and performing several epochs of optimization on the sampled data.

本文试图通过引入一种算法来改善现状：该算法达到 TRPO 的数据效率与可靠性能，同时只使用一阶优化。我们提出一种带裁剪概率比的新目标，它对策略性能形成悲观估计（即下界）。为优化策略，我们在从策略采样数据、以及在采样数据上执行若干轮优化之间交替。

Our experiments compare the performance of various different versions of the surrogate objective, and find that the version with the clipped probability ratios performs best. We also compare PPO to several previous algorithms from the literature. On continuous control tasks, it performs better than the algorithms we compare against. On Atari, it performs significantly better (in terms of sample complexity) than A2C and similarly to ACER though it is much simpler.

我们的实验比较了代理目标的多种不同版本，发现带裁剪概率比的版本表现最好。我们还将 PPO 与文献中若干先前算法作了比较。在连续控制任务上，它优于我们所比较的算法。在 Atari 上，它在样本复杂度上显著优于 A2C，并与 ACER 相当，尽管实现简单得多。

While DQN works well on game environments like the Arcade Learning Environment [Bel+15] with discrete action spaces, it has not been demonstrated to perform well on continuous control benchmarks such as those in OpenAI Gym [Bro+16] and described by Duan et al. [Dua+16].

虽然 DQN 在 Arcade Learning Environment [Bel+15] 这类离散动作空间的游戏环境上表现良好，但尚未被证明能在 OpenAI Gym [Bro+16] 以及 Duan 等人 [Dua+16] 所述的那些连续控制基准上表现良好。

2 Background: Policy Optimization

2 背景：策略优化

2.1 Policy Gradient Methods

2.1 策略梯度方法

Policy gradient methods work by computing an estimator of the policy gradient and plugging it into a stochastic gradient ascent algorithm. The most commonly used gradient estimator has the form

策略梯度方法通过计算策略梯度的估计量，并将其代入随机梯度上升算法来工作。最常用的梯度估计量形式为

$$\hat{g} = \hat{\mathbb{E}}_t\left[\nabla_\theta \log \pi_\theta(a_t \mid s_t)\hat{A}_t\right] \tag{1}$$

where $\pi_ heta$ is a stochastic policy and $\hat{A}_t$ is an estimator of the advantage function at timestep $t$. Here, the expectation $\hat{\mathbb{E}}_t[\ldots]$ indicates the empirical average over a finite batch of samples, in an algorithm that alternates between sampling and optimization. Implementations that use automatic differentiation software work by constructing an objective function whose gradient is the policy gradient estimator; the estimator $\hat{g}$ is obtained by differentiating the objective

其中 $\pi_ heta$ 是随机策略，$\hat{A}_t$ 是时刻 $t$ 优势函数的估计量。这里的期望 $\hat{\mathbb{E}}_t[\ldots]$ 表示在采样与优化交替的算法中，有限样本批次上的经验平均。使用自动微分软件的实现，通过构造一个梯度恰为策略梯度估计量的目标函数来工作；估计量 $\hat{g}$ 通过对目标求导得到

$$L^{PG}(\theta) = \hat{\mathbb{E}}_t\left[\log \pi_\theta(a_t \mid s_t)\hat{A}_t\right]. \tag{2}$$

While it is appealing to perform multiple steps of optimization on this loss $L^{PG}$ using the same trajectory, doing so is not well-justified, and empirically it often leads to destructively large policy updates (see Section 6.1; results are not shown but were similar or worse than the "no clipping or penalty" setting).

尽管用同一条轨迹对损失 $L^{PG}$ 做多步优化很有吸引力，但这样做缺乏充分依据，并且经验上常常导致破坏性的过大策略更新（见第 6.1 节；结果未展示，但与“无裁剪或惩罚”设置相似或更差）。

2.2 Trust Region Methods

2.2 信任域方法

In TRPO [Sch+15b], an objective function (the "surrogate" objective) is maximized subject to a constraint on the size of the policy update. Specifically,

在 TRPO [Sch+15b] 中，一个目标函数（“代理”目标）在策略更新大小的约束下被最大化。具体而言，

$$\operatorname{maximize}_\theta\ \hat{\mathbb{E}}_t\left[\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)}\hat{A}_t\right] \tag{3}$$

$$\operatorname{subject\ to}\ \hat{\mathbb{E}}_t\left[\mathrm{KL}[\pi_{\theta_{\mathrm{old}}}(\cdot \mid s_t), \pi_\theta(\cdot \mid s_t)]\right] \le \delta. \tag{4}$$

Here, $heta_{\mathrm{old}}$ is the vector of policy parameters before the update. This problem can efficiently be approximately solved using the conjugate gradient algorithm, after making a linear approximation to the objective and a quadratic approximation to the constraint.

这里，$heta_{\mathrm{old}}$ 是更新前的策略参数向量。在对目标做线性近似、对约束做二次近似之后，该问题可用共轭梯度算法高效地近似求解。

The theory justifying TRPO actually suggests using a penalty instead of a constraint, i.e., solving the unconstrained optimization problem

为 TRPO 提供依据的理论实际上建议使用惩罚而不是约束，即求解无约束优化问题

$$\operatorname{maximize}_\theta\ \hat{\mathbb{E}}_t\left[\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)}\hat{A}_t - \beta\,\mathrm{KL}[\pi_{\theta_{\mathrm{old}}}(\cdot \mid s_t), \pi_\theta(\cdot \mid s_t)]\right] \tag{5}$$

for some coefficient eta. This follows from the fact that a certain surrogate objective (which computes the max KL over states instead of the mean) forms a lower bound (i.e., a pessimistic bound) on the performance of the policy $\pi$. TRPO uses a hard constraint rather than a penalty because it is hard to choose a single value of eta that performs well across different problems—or even within a single problem, where the characteristics change over the course of learning. Hence, to achieve our goal of a first-order algorithm that emulates the monotonic improvement of TRPO, experiments show that it is not sufficient to simply choose a fixed penalty coefficient eta and optimize the penalized objective Equation (5) with SGD; additional modifications are required.

其中 eta 为某个系数。这是因为某个代理目标（在状态上计算最大 KL 而非平均 KL）对策略 $\pi$ 的性能构成下界（即悲观界）。TRPO 使用硬约束而非惩罚，是因为很难选择一个在不同问题上——甚至在单个问题内（学习过程中特性会变化）——都表现良好的单一 eta 值。因此，为实现模拟 TRPO 单调改进的一阶算法这一目标，实验表明：仅选择固定惩罚系数 eta 并用 SGD 优化惩罚目标（式 5）并不够；还需要额外的修改。

3 Clipped Surrogate Objective

3 裁剪代理目标

Let $r_t(\theta)$ denote the probability ratio $r_t(\theta)=\pi_\theta(a_t\mid s_t)/\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)$, so $r(\theta_{\mathrm{old}})=1$. TRPO maximizes a "surrogate" objective

设 $r_t(\theta)$ 表示概率比 $r_t(\theta)=\pi_\theta(a_t\mid s_t)/\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)$，因而 $r(\theta_{\mathrm{old}})=1$。TRPO 最大化一个“代理”目标

$$L^{CPI}(\theta)=\hat{\mathbb{E}}_t\left[\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)}\hat{A}_t\right]=\hat{\mathbb{E}}_t\left[r_t(\theta)\hat{A}_t\right]. \tag{6}$$

The superscript $CPI$ refers to conservative policy iteration [KL02], where this objective was proposed. Without a constraint, maximization of $L^{CPI}$ would lead to an excessively large policy update; hence, we now consider how to modify the objective, to penalize changes to the policy that move $r_t( heta)$ away from 1.

上标 $CPI$ 指保守策略迭代 [KL02]，该目标即在其中提出。若无约束，最大化 $L^{CPI}$ 会导致过大的策略更新；因此，我们现在考虑如何修改目标，以惩罚使 $r_t( heta)$ 偏离 1 的策略变化。

The main objective we propose is the following:

我们提出的主要目标如下：

$$L^{CLIP}(\theta)=\hat{\mathbb{E}}_t\left[\min\big(r_t(\theta)\hat{A}_t,\ \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t\big)\right] \tag{7}$$

where epsilon is a hyperparameter, say, $\epsilon=0.2$. The motivation for this objective is as follows. The first term inside the min is $L^{CPI}$. The second term, $\mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t$, modifies the surrogate objective by clipping the probability ratio, which removes the incentive for moving $r_t$ outside of the interval $[1-\epsilon, 1+\epsilon]$. Finally, we take the minimum of the clipped and unclipped objective, so the final objective is a lower bound (i.e., a pessimistic bound) on the unclipped objective. With this scheme, we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse. Note that $L^{CLIP}(\theta)=L^{CPI}(\theta)$ to first order around $\theta_{\mathrm{old}}$ (i.e., where $r=1$), however, they become different as $\theta$ moves away from $\theta_{\mathrm{old}}$. Figure 1 plots a single term (i.e., a single $t$) in $L^{CLIP}$; note that the probability ratio $r$ is clipped at $1-\epsilon$ or $1+\epsilon$ depending on whether the advantage is positive or negative.

其中 epsilon 是超参数，例如 $\epsilon=0.2$。该目标的动机如下。min 内的第一项是 $L^{CPI}$。第二项 $\mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t$ 通过对概率比裁剪来修改代理目标，从而消除把 $r_t$ 移出区间 $[1-\epsilon, 1+\epsilon]$ 的激励。最后，我们取裁剪与未裁剪目标的较小值，使最终目标成为未裁剪目标的下界（即悲观界）。在这一方案下，我们只在概率比变化会使目标变好时忽略该变化，而在它使目标变差时计入。注意，在 $\theta_{\mathrm{old}}$ 附近（即 $r=1$ 处）$L^{CLIP}(\theta)$ 与 $L^{CPI}(\theta)$ 一阶相同，但当 $\theta$ 离开 $\theta_{\mathrm{old}}$ 时它们变得不同。图 1 画出了 $L^{CLIP}$ 中的单项（即单个 $t$）；注意，概率比 $r$ 在优势为正或为负时分别被裁剪在 $1-\epsilon$ 或 $1+\epsilon$。

Figure 1: Plots showing one term (i.e., a single timestep) of the surrogate function $L^{CLIP}$ as a function of the probability ratio $r$, for positive advantages (left) and negative advantages (right). The red circle on each plot shows the starting point for the optimization, i.e., $r=1$. Note that $L^{CLIP}$ sums many of these terms.

图 1：代理函数 $L^{CLIP}$ 的一项（即单个时间步）作为概率比 $r$ 的函数，分别对应正优势（左）和负优势（右）。每张图上的红圈标出优化起点，即 $r=1$。注意 $L^{CLIP}$ 是许多这样的项之和。

Figure 2 provides another source of intuition about the surrogate objective $L^{CLIP}$. It shows how several objectives vary as we interpolate along the policy update direction, obtained by proximal policy optimization (the algorithm we will introduce shortly) on a continuous control problem. We can see that $L^{CLIP}$ is a lower bound on $L^{CPI}$, with a penalty for having too large of a policy update.

图 2 提供了关于代理目标 $L^{CLIP}$ 的另一直观来源。它展示了沿策略更新方向插值时若干目标如何变化，该更新方向由近端策略优化（我们稍后引入的算法）在一个连续控制问题上得到。可以看到 $L^{CLIP}$ 是 $L^{CPI}$ 的下界，并对过大的策略更新加以惩罚。

Figure 2: Surrogate objectives, as we interpolate between the initial policy parameter $heta_{\mathrm{old}}$, and the updated policy parameter, which we compute after one iteration of PPO. The updated policy has a KL divergence of about 0.02 from the initial policy, and this is the point at which $L^{CLIP}$ is maximal. This plot corresponds to the first policy update on the Hopper-v1 problem, using hyperparameters provided in Section 6.1.

图 2：在初始策略参数 $heta_{\mathrm{old}}$ 与一次 PPO 迭代后算出的更新策略参数之间插值时的代理目标。更新后的策略相对初始策略的 KL 散度约为 0.02，而这一点恰是 $L^{CLIP}$ 最大之处。该图对应 Hopper-v1 问题上的第一次策略更新，超参数见第 6.1 节。

4 Adaptive KL Penalty Coefficient

4 自适应 KL 惩罚系数

Another approach, which can be used as an alternative to the clipped surrogate objective, or in addition to it, is to use a penalty on KL divergence, and to adapt the penalty coefficient so that we achieve some target value of the KL divergence $d_{\mathrm{targ}}$ each policy update. In our experiments, we found that the KL penalty performed worse than the clipped surrogate objective, however, we've included it here because it's an important baseline.

另一种方法可作为裁剪代理目标的替代，或与之并用：对 KL 散度使用惩罚，并自适应调整惩罚系数，使每次策略更新达到某个目标 KL 散度 $d_{\mathrm{targ}}$。在我们的实验中，KL 惩罚表现不如裁剪代理目标，但我们仍将其收录于此，因为它是重要的基线。

In the simplest instantiation of this algorithm, we perform the following steps in each policy update:

• Using several epochs of minibatch SGD, optimize the KL-penalized objective

在该算法最简单的实例中，我们在每次策略更新中执行以下步骤：

• 用若干 epoch 的小批量 SGD，优化带 KL 惩罚的目标

$$L^{KLPEN}(\theta)=\hat{\mathbb{E}}_t\left[\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)}\hat{A}_t - \beta\,\mathrm{KL}[\pi_{\theta_{\mathrm{old}}}(\cdot\mid s_t), \pi_\theta(\cdot\mid s_t)]\right] \tag{8}$$

• Compute $d=\hat{\mathbb{E}}_t[\mathrm{KL}[\pi_{\theta_{\mathrm{old}}}(\cdot\mid s_t), \pi_\theta(\cdot\mid s_t)]]$
  • If $d < d_{\mathrm{targ}}/1.5$, eta \leftarrow eta/2
  • If $d > d_{\mathrm{targ}} imes 1.5$, eta \leftarrow eta imes 2

• 计算 $d=\hat{\mathbb{E}}_t[\mathrm{KL}[\pi_{\theta_{\mathrm{old}}}(\cdot\mid s_t), \pi_\theta(\cdot\mid s_t)]]$
  • 若 $d < d_{\mathrm{targ}}/1.5$，则 eta \leftarrow eta/2
  • 若 $d > d_{\mathrm{targ}} imes 1.5$，则 eta \leftarrow eta imes 2

The updated $\beta$ is used for the next policy update. With this scheme, we occasionally see policy updates where the KL divergence is significantly different from $d_{\mathrm{targ}}$, however, these are rare, and $\beta$ quickly adjusts. The parameters 1.5 and 2 above are chosen heuristically, but the algorithm is not very sensitive to them. The initial value of $\beta$ is another hyperparameter but is not important in practice because the algorithm quickly adjusts it.

更新后的 $\beta$ 用于下一次策略更新。在此方案下，我们偶尔会看到 KL 散度与 $d_{\mathrm{targ}}$ 相差显著的策略更新，但这种情况很少见，且 $\beta$ 会迅速调整。上面的参数 1.5 和 2 是启发式选取的，但算法对它们并不十分敏感。$\beta$ 的初值是另一个超参数，但在实践中不重要，因为算法会迅速调整它。

5 Algorithm

5 算法

The surrogate losses from the previous sections can be computed and differentiated with a minor change to a typical policy gradient implementation. For implementations that use automatic differentiation, one simply constructs the loss $L^{CLIP}$ or $L^{KLPEN}$ instead of $L^{PG}$, and one performs multiple steps of stochastic gradient ascent on this objective.

前面各节的代理损失只需对典型策略梯度实现做小改动即可计算并求导。对使用自动微分的实现，只需构造损失 $L^{CLIP}$ 或 $L^{KLPEN}$ 以代替 $L^{PG}$，并在该目标上执行多步随机梯度上升。

Most techniques for computing variance-reduced advantage-function estimators make use of a learned state-value function $V(s)$; for example, generalized advantage estimation [Sch+15a], or the finite-horizon estimators in [Mni+16]. If using a neural network architecture that shares parameters between the policy and value function, we must use a loss function that combines the policy surrogate and a value function error term. This objective can further be augmented by adding an entropy bonus to ensure sufficient exploration, as suggested in past work [Wil92; Mni+16]. Combining these terms, we obtain the following objective, which is (approximately) maximized each iteration:

大多数计算方差缩减优势函数估计量的技术都会用到学得的状态价值函数 $V(s)$；例如广义优势估计 [Sch+15a]，或 [Mni+16] 中的有限视野估计量。若使用在策略与价值函数之间共享参数的神经网络架构，我们必须使用把策略代理与价值函数误差项结合在一起的损失函数。该目标还可进一步加入熵奖励以保证充分探索，如既往工作所建议 [Wil92; Mni+16]。把这些项组合起来，我们得到如下目标，并在每次迭代中（近似）最大化它：

$$L_t^{CLIP+VF+S}(\theta)=\hat{\mathbb{E}}_t\left[L_t^{CLIP}(\theta) - c_1 L_t^{VF}(\theta) + c_2 S[\pi_\theta](s_t)\right], \tag{9}$$

where $c_1$, $c_2$ are coefficients, and $S$ denotes an entropy bonus, and $L_t^{VF}$ is a squared-error loss $(V_\theta(s_t) - V_t^{\mathrm{targ}})^2$.

其中 $c_1$、$c_2$ 为系数，$S$ 表示熵奖励，$L_t^{VF}$ 为平方误差损失 $(V_\theta(s_t) - V_t^{\mathrm{targ}})^2$。

One style of policy gradient implementation, popularized in [Mni+16] and well-suited for use with recurrent neural networks, runs the policy for $T$ timesteps (where $T$ is much less than the episode length), and uses the collected samples for an update. This style requires an advantage estimator that does not look beyond timestep $T$. The estimator used by [Mni+16] is

一种由 [Mni+16] 推广、并适于与循环神经网络一起使用的策略梯度实现风格，是让策略运行 $T$ 个时间步（$T$ 远小于回合长度），并用收集到的样本做一次更新。这种风格需要一个不向 $T$ 之后展望的优势估计量。[Mni+16] 使用的估计量为

$$\hat{A}_t = -V(s_t) + r_t + \gamma r_{t+1} + \cdots + \gamma^{T-t+1} r_{T-1} + \gamma^{T-t} V(s_T) \tag{10}$$

where $t$ specifies the time index in $[0, T]$, within a given length-$T$ trajectory segment. Generalizing this choice, we can use a truncated version of generalized advantage estimation, which reduces to Equation (10) when $\lambda=1$:

其中 $t$ 指定给定长度-$T$ 轨迹段内 $[0, T]$ 中的时间索引。推广这一选择，我们可以使用截断版的广义优势估计，当 $\lambda=1$ 时它退化为式 (10)：

$$\hat{A}_t = \delta_t + (\gamma\lambda)\delta_{t+1} + \cdots + (\gamma\lambda)^{T-t+1}\delta_{T-1}, \tag{11}$$

$$\text{where }\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t) \tag{12}$$

A proximal policy optimization (PPO) algorithm that uses fixed-length trajectory segments is shown below. Each iteration, each of $N$ (parallel) actors collect $T$ timesteps of data. Then we construct the surrogate loss on these $NT$ timesteps of data, and optimize it with minibatch SGD (or usually for better performance, Adam [KB14]), for $K$ epochs.

使用固定长度轨迹段的近端策略优化（PPO）算法如下所示。每次迭代，$N$ 个（并行）actor 各收集 $T$ 个时间步的数据。然后我们在这 $NT$ 个时间步的数据上构造代理损失，并用小批量 SGD（或通常为获得更好性能，用 Adam [KB14]）优化 $K$ 个 epoch。

Algorithm 1 PPO, Actor-Critic Style

for iteration=1, 2, … do
for actor=1, 2, …, $N$ do
Run policy $\pi_{\theta_{\mathrm{old}}}$ in environment for $T$ timesteps
Compute advantage estimates $\hat{A}_1,\ldots,\hat{A}_T$
end for
Optimize surrogate $L$ wrt $\theta$, with $K$ epochs and minibatch size $M \le NT$
$\theta_{\mathrm{old}} \leftarrow \theta$
end for

算法 1 PPO，Actor-Critic 风格

for iteration=1, 2, … do
for actor=1, 2, …, $N$ do
在环境中运行策略 $\pi_{\theta_{\mathrm{old}}}$，共 $T$ 个时间步
计算优势估计 $\hat{A}_1,\ldots,\hat{A}_T$
end for
对 $\theta$ 优化代理 $L$，共 $K$ 个 epoch，小批量大小 $M \le NT$
$\theta_{\mathrm{old}} \leftarrow \theta$
end for

6 Experiments

6 实验

6.1 Comparison of Surrogate Objectives

6.1 代理目标的比较

First, we compare several different surrogate objectives under different hyperparameters. Here, we compare the surrogate objective $L^{CLIP}$ to several natural variations and ablated versions.

首先，我们在不同超参数下比较若干不同的代理目标。这里，我们将代理目标 $L^{CLIP}$ 与若干自然变体和消融版本进行比较。

No clipping or penalty: $L_t(\theta)=r_t(\theta)\hat{A}_t$

Clipping: $L_t(\theta)=\min(r_t(\theta)\hat{A}_t,\ \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t)$

KL penalty (fixed or adaptive): $L_t(\theta)=r_t(\theta)\hat{A}_t - \beta\,\mathrm{KL}[\pi_{\theta_{\mathrm{old}}}, \pi_\theta]$

无裁剪或惩罚：$L_t(\theta)=r_t(\theta)\hat{A}_t$

裁剪：$L_t(\theta)=\min(r_t(\theta)\hat{A}_t,\ \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t)$

KL 惩罚（固定或自适应）：$L_t(\theta)=r_t(\theta)\hat{A}_t - \beta\,\mathrm{KL}[\pi_{\theta_{\mathrm{old}}}, \pi_\theta]$

For the KL penalty, one can either use a fixed penalty coefficient $\beta$ or an adaptive coefficient as described in Section 4 using target KL value $d_{\mathrm{targ}}$. Note that we also tried clipping in log space, but found the performance to be no better.

对 KL 惩罚，既可以使用固定惩罚系数 $\beta$，也可以使用第 4 节所述、以目标 KL 值 $d_{\mathrm{targ}}$ 为指引的自适应系数。注意我们还尝试过在对数空间裁剪，但发现性能并不更好。

Because we are searching over hyperparameters for each algorithm variant, we chose a computationally cheap benchmark to test the algorithms on. Namely, we used 7 simulated robotics tasks implemented in OpenAI Gym [Bro+16], which use the MuJoCo [TET12] physics engine. We do one million timesteps of training on each one. Besides the hyperparameters used for clipping ($\epsilon$) and the KL penalty (eta, $d_{\mathrm{targ}}$), which we search over, the other hyperparameters are provided in Table 3.

由于我们要对每种算法变体搜索超参数，我们选择了一个计算上便宜的基准来测试这些算法。具体而言，我们使用在 OpenAI Gym [Bro+16] 中实现的 7 个模拟机器人任务，它们使用 MuJoCo [TET12] 物理引擎。每个任务训练一百万时间步。除了我们搜索的裁剪超参数（$\epsilon$）和 KL 惩罚超参数（eta，$d_{\mathrm{targ}}$）外，其余超参数见表 3。

To represent the policy, we used a fully-connected MLP with two hidden layers of 64 units, and tanh nonlinearities, outputting the mean of a Gaussian distribution, with variable standard deviations, following [Sch+15b; Dua+16]. We don't share parameters between the policy and value function (so coefficient $c_1$ is irrelevant), and we don't use an entropy bonus.

为表示策略，我们使用带两个 64 单元隐层、tanh 非线性的全连接 MLP，输出高斯分布的均值，并带可变标准差，做法遵循 [Sch+15b; Dua+16]。我们不在策略与价值函数之间共享参数（因此系数 $c_1$ 无关），也不使用熵奖励。

Each algorithm was run on all 7 environments, with 3 random seeds on each. We scored each run of the algorithm by computing the average total reward of the last 100 episodes. We shifted and scaled the scores for each environment so that the random policy gave a score of 0 and the best result was set to 1, and averaged over 21 runs to produce a single scalar for each algorithm setting.

每种算法在全部 7 个环境上运行，每个环境 3 个随机种子。我们对算法的每次运行，用最后 100 个回合的平均总回报打分。我们对每个环境的分数做平移和缩放，使随机策略得分为 0、最好结果为 1，再对 21 次运行取平均，为每种算法设置得到一个标量。

The results are shown in Table 1. Note that the score is negative for the setting without clipping or penalties, because for one environment (half cheetah) it leads to a very negative score, which is worse than the initial random policy.

结果见表 1。注意，无裁剪或惩罚的设置得分为负，因为在一个环境（half cheetah）上它导致非常负的分数，比初始随机策略更差。

algorithm	avg. normalized score
No clipping or penalty	-0.39
Clipping, $\epsilon=0.1$	0.76
Clipping, $\epsilon=0.2$	0.82
Clipping, $\epsilon=0.3$	0.70
Adaptive KL $d_{\mathrm{targ}}=0.003$	0.68
Adaptive KL $d_{\mathrm{targ}}=0.01$	0.74
Adaptive KL $d_{\mathrm{targ}}=0.03$	0.71
Fixed KL, $\beta=0.3$	0.62
Fixed KL, $\beta=1.$	0.71
Fixed KL, $\beta=3.$	0.72
Fixed KL, $\beta=10.$	0.69

Table 1: Results from continuous control benchmark. Average normalized scores (over 21 runs of the algorithm, on 7 environments) for each algorithm / hyperparameter setting. $\beta$ was initialized at 1.

算法	平均归一化得分
无裁剪或惩罚	-0.39
裁剪，$\epsilon=0.1$	0.76
裁剪，$\epsilon=0.2$	0.82
裁剪，$\epsilon=0.3$	0.70
自适应 KL $d_{\mathrm{targ}}=0.003$	0.68
自适应 KL $d_{\mathrm{targ}}=0.01$	0.74
自适应 KL $d_{\mathrm{targ}}=0.03$	0.71
固定 KL，$\beta=0.3$	0.62
固定 KL，$\beta=1.$	0.71
固定 KL，$\beta=3.$	0.72
固定 KL，$\beta=10.$	0.69

表 1：连续控制基准结果。每种算法／超参数设置在 7 个环境、共 21 次运行上的平均归一化得分。$\beta$ 初始化为 1。

6.2 Comparison to Other Algorithms in the Continuous Domain

6.2 与连续域其他算法的比较

Next, we compare PPO (with the "clipped" surrogate objective from Section 3) to several other methods from the literature, which are considered to be effective for continuous problems. We compared against tuned implementations of the following algorithms: trust region policy optimization [Sch+15b], cross-entropy method (CEM) [SL06], vanilla policy gradient with adaptive stepsize, A2C [Mni+16], A2C with trust region [Wan+16]. A2C stands for advantage actor critic, and is a synchronous version of A3C, which we found to have the same or better performance than the asynchronous version. For PPO, we used the hyperparameters from the previous section, with $\epsilon=0.2$. We see that PPO outperforms the previous methods on almost all the continuous control environments.

接下来，我们将 PPO（采用第 3 节的“裁剪”代理目标）与文献中若干被认为对连续问题有效的其他方法进行比较。我们与下列算法的调优实现作了对比：信任域策略优化 [Sch+15b]、交叉熵方法（CEM）[SL06]、带自适应步长的普通策略梯度、A2C [Mni+16]、带信任域的 A2C [Wan+16]。A2C 表示优势 actor-critic，是 A3C 的同步版本，我们发现其性能与异步版本相同或更好。对 PPO，我们使用上一节的超参数，取 $\epsilon=0.2$。可以看到，PPO 在几乎所有连续控制环境上优于先前方法。

Figure 3: Comparison of several algorithms on several MuJoCo environments, training for one million timesteps.

图 3：若干算法在若干 MuJoCo 环境上的比较，训练一百万时间步。

6.3 Comparison on the Atari Domain

6.3 在 Atari 域上的比较

We also tested our algorithms on a set of Atari games from the Arcade Learning Environment [Bel+15]. We used the same architecture and hyperparameters as [Wan+16] for the algorithms we compared against. We used the same architecture and a similar hyperparameter schedule as [Mni+16] for A2C. For PPO, we used the same architecture as A2C, and the hyperparameters provided in Table 3. For ACER, we used the implementation and hyperparameters provided by the authors of [Wan+17]. As shown in Table 2, PPO outperforms A2C on all but one of the games tested, and has similar performance to ACER, despite being a much simpler algorithm.

我们还在 Arcade Learning Environment [Bel+15] 的一组 Atari 游戏上测试了算法。对所比较的算法，我们使用与 [Wan+16] 相同的架构和超参数。对 A2C，我们使用与 [Mni+16] 相同的架构和类似的超参数日程。对 PPO，我们使用与 A2C 相同的架构，以及表 3 给出的超参数。对 ACER，我们使用 [Wan+17] 作者提供的实现与超参数。如表 2 所示，PPO 在除一款以外的所有测试游戏上优于 A2C，并与 ACER 性能相近，尽管算法简单得多。

6.4 Comparison of Continuous Control Algorithms with a High-Dimensional Action Space

6.4 高维动作空间下连续控制算法的比较

In this section, we provide a comparison of continuous control algorithms on a set of more difficult, high-dimensional continuous control environments. Specifically, we used the Roboschool environments, which provide a free, open-source robotics simulation based on OpenAI Gym and Bullet Physics. We used three environments: Humanoid, HumanoidFlagrun, and HumanoidFlagrunHarder. These environments involve a high-dimensional humanoid robot, and hence provide a more challenging test of continuous control algorithms. The results are shown in Figure 4: PPO outperforms A2C and ACER, and performs similarly to ACKTR, while being simpler to implement.

本节在一组更难、更高维的连续控制环境上比较连续控制算法。具体而言，我们使用 Roboschool 环境，它基于 OpenAI Gym 与 Bullet Physics 提供免费开源的机器人仿真。我们使用三个环境：Humanoid、HumanoidFlagrun 和 HumanoidFlagrunHarder。这些环境涉及高维人形机器人，因而对连续控制算法构成更具挑战性的测试。结果见图 4：PPO 优于 A2C 和 ACER，并与 ACKTR 表现相近，同时实现更简单。

Figure 4: Comparison of several algorithms on three Roboschool environments with high-dimensional action spaces.

图 4：若干算法在三个高维动作空间 Roboschool 环境上的比较。

7 Conclusion

7 结论

We have introduced proximal policy optimization, a family of policy gradient methods that alternate between sampling data from the policy and performing several epochs of optimization on the sampled data. We showed that a clipped surrogate objective performs well in practice, and that PPO strikes a favorable balance between sample complexity, simplicity, and wall-time.

我们引入了近端策略优化，这是一类在从策略采样数据、以及在采样数据上执行若干轮优化之间交替的策略梯度方法。我们表明裁剪代理目标在实践中表现良好，并且 PPO 在样本复杂度、简洁性与墙钟时间之间取得了有利的平衡。

References

参考文献

[Bel+15] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An evaluation platform for general agents. JAIR, 2013.

[Bro+16] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. arXiv:1606.01540, 2016.

[Dua+16] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. ICML, 2016.

[KB14] D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015.

[KL02] S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. ICML, 2002.

[Mni+15] V. Mnih et al. Human-level control through deep reinforcement learning. Nature, 2015.

[Mni+16] V. Mnih et al. Asynchronous methods for deep reinforcement learning. ICML, 2016.

[Sch+15a] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. ICLR, 2016.

[Sch+15b] J. Schulman, S. Levine, P. Moritz, M. Jordan, and P. Abbeel. Trust region policy optimization. ICML, 2015.

[SL06] R. Rubinstein and D. Kroese. The Cross-Entropy Method. Springer, 2004.

[TET12] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. IROS, 2012.

[Wan+16] Z. Wang et al. Sample efficient actor-critic with experience replay. ICLR, 2017.

[Wan+17] Z. Wang et al. (ACER follow-up / related). 2017.

[Wil92] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 1992.