High-Dimensional Continuous Control Using Generalized Advantage Estimation

用广义优势估计实现高维连续控制

John Schulman    Philipp Moritz    Sergey Levine    Michael I. Jordan    Pieter Abbeel Affiliation:  Department of Electrical Engineering and Computer Science Affiliation:  University of California, Berkeley Email:  {joschu,pcmoritz,levine,jordan,pabbeel}@eecs.berkeley.edu

Abstract

摘要

Policy gradient methods are an appealing approach in reinforcement learning because they directly optimize the cumulative reward and can straightforwardly be used with nonlinear function approximators such as neural networks. The two main challenges are the large number of samples typically required, and the difficulty of obtaining stable and steady improvement despite the nonstationarity of the incoming data. We address the first challenge by using value functions to substantially reduce the variance of policy gradient estimates at the cost of some bias, with an exponentially-weighted estimator of the advantage function that is analogous to TD($\lambda$). We address the second challenge by using trust region optimization procedure for both the policy and the value function, which are represented by neural networks.

策略梯度方法在强化学习中很有吸引力，因为它们直接优化累积回报，并且可以很自然地与神经网络这类非线性函数逼近器一起使用。两个主要困难是：通常需要大量样本，以及在输入数据非平稳的情况下，难以获得稳定而持续的改进。针对第一点，我们用价值函数在引入一定偏差的代价下显著降低策略梯度估计的方差，并采用一种与 TD($\lambda$) 类似的、对优势函数做指数加权的估计器。针对第二点，我们对策略和价值函数都使用信赖域优化，二者均用神经网络表示。

Our approach yields strong empirical results on highly challenging 3D locomotion tasks, learning running gaits for bipedal and quadrupedal simulated robots, and learning a policy for getting the biped to stand up from starting out lying on the ground. In contrast to a body of prior work that uses hand-crafted policy representations, our neural network policies map directly from raw kinematics to joint torques. Our algorithm is fully model-free, and the amount of simulated experience required for the learning tasks on 3D bipeds corresponds to 1-2 weeks of real time.

我们的方法在极具挑战性的三维运动任务上取得了很强的经验结果：为双足和四足仿真机器人学到了奔跑步态，并为从仰卧起始姿态站起来的双足机器人学到了策略。与许多依赖手工设计策略表示的已有工作不同，我们的神经网络策略直接把原始运动学量映射到关节力矩。算法完全是无模型的；在三维双足任务上所需的仿真经验，大约相当于现实时间的一到两周。

1 Introduction

1 引言

The typical problem formulation in reinforcement learning is to maximize the expected total reward of a policy. A key source of difficulty is the long time delay between actions and their positive or negative effect on rewards; this issue is called the credit assignment problem in the reinforcement learning literature (Minsky, 1961; Sutton & Barto, 1998), and the distal reward problem in the behavioral literature (Hull, 1943). Value functions offer an elegant solution to the credit assignment problem—they allow us to estimate the goodness of an action before the delayed reward arrives. Reinforcement learning algorithms make use of value functions in a variety of different ways; this paper considers algorithms that optimize a parameterized policy and use value functions to help estimate how the policy should be improved.

强化学习里典型的问题设定，是最大化策略的期望总回报。一个主要难点在于动作与其带来的正负回报之间往往有很长的时间延迟；强化学习文献称之为信用分配问题（Minsky, 1961; Sutton & Barto, 1998），行为学文献则称之为远端奖赏问题（Hull, 1943）。价值函数为信用分配提供了一种优雅的解决办法——它让我们在延迟回报到来之前，就能估计某个动作的好坏。强化学习算法以多种方式使用价值函数；本文考虑的是这样一类算法：它们优化参数化策略，并用价值函数来帮助估计策略应当如何改进。

When using a parameterized stochastic policy, it is possible to obtain an unbiased estimate of the gradient of the expected total returns (Williams, 1992; Sutton et al., 1999; Baxter & Bartlett, 2000); these noisy gradient estimates can be used in a stochastic gradient ascent algorithm. Unfortunately, the variance of the gradient estimator scales unfavorably with the time horizon, since the effect of an action is confounded with the effects of past and future actions. Another class of policy gradient algorithms, called actor-critic methods, use a value function rather than the empirical returns, obtaining an estimator with lower variance at the cost of introducing bias (Konda & Tsitsiklis, 2003; Hafner & Riedmiller, 2011). But while high variance necessitates using more samples, bias is more pernicious—even with an unlimited number of samples, bias can cause the algorithm to fail to converge, or to converge to a poor solution that is not even a local optimum.

当使用参数化的随机策略时，可以获得期望总回报梯度的无偏估计（Williams, 1992; Sutton et al., 1999; Baxter & Bartlett, 2000）；这些带噪声的梯度估计可用于随机梯度上升。可惜，梯度估计器的方差会随时间视界不利地增大，因为一个动作的效果会与过去和未来动作的效果纠缠在一起。另一类称为演员–评论家的策略梯度算法用价值函数代替经验回报，以引入偏差为代价得到方差更低的估计器（Konda & Tsitsiklis, 2003; Hafner & Riedmiller, 2011）。高方差只是要求用更多样本，偏差则更为棘手——即便样本无限，偏差也可能使算法无法收敛，或收敛到甚至不是局部最优的较差解。

We propose a family of policy gradient estimators that significantly reduce variance while maintaining a tolerable level of bias. We call this estimation scheme, parameterized by $\gamma\in[0,1]$ and $\lambda\in[0,1]$, the generalized advantage estimator (GAE). Related methods have been proposed in the context of online actor-critic methods (Kimura & Kobayashi, 1998; Wawrzyński, 2009). We provide a more general analysis, which is applicable in both the online and batch settings, and discuss an interpretation of our method as an instance of reward shaping (Ng et al., 1999), where the approximate value function is used to shape the reward.

我们提出一类能显著降低方差、同时把偏差控制在可容忍水平的策略梯度估计器。这一由 $\gamma\in[0,1]$ 和 $\lambda\in[0,1]$ 参数化的估计方案，称为广义优势估计（generalized advantage estimator, GAE）。在在线演员–评论家方法中已有相关做法（Kimura & Kobayashi, 1998; Wawrzyński, 2009）。我们给出更一般的分析，同时适用于在线和批量设定，并把该方法解释为奖励塑形（reward shaping）（Ng et al., 1999）的一个实例：用近似价值函数来塑形奖励。

We present experimental results on a number of highly challenging 3D locomotion tasks, where we show that our approach can learn complex gaits using high-dimensional, general purpose neural network function approximators for both the policy and the value function, each with over $10^{4}$ parameters. The policies perform torque-level control of simulated 3D robots with up to 33 state dimensions and 10 actuators.

我们在若干极具挑战性的三维运动任务上给出实验结果，表明该方法可以用高维、通用的神经网络函数逼近器同时表示策略和价值函数（各自参数都超过 $10^{4}$），并学到复杂步态。这些策略对最多有 33 维状态和 10 个执行器的仿真三维机器人做力矩级控制。

The contributions of this paper are summarized as follows:

本文贡献可概括如下：

1. We provide justification and intuition for an effective variance reduction scheme for policy gradients, which we call generalized advantage estimation (GAE). While the formula has been proposed in prior work (Kimura & Kobayashi, 1998; Wawrzyński, 2009), our analysis is novel and enables GAE to be applied with a more general set of algorithms, including the batch trust-region algorithm we use for our experiments.

1. 我们为一套有效的策略梯度方差削减方案提供了理由和直观解释，并称之为广义优势估计（GAE）。公式本身已在先前工作中出现（Kimura & Kobayashi, 1998; Wawrzyński, 2009），但我们的分析是新的，使 GAE 能与更广泛的算法一起使用，包括实验中采用的批量信赖域算法。

2. We propose the use of a trust region optimization method for the value function, which we find is a robust and efficient way to train neural network value functions with thousands of parameters.

2. 我们提出用信赖域方法优化价值函数，发现这是训练带有数千参数的神经网络价值函数的一种稳健而高效的方式。

3. By combining (1) and (2) above, we obtain an algorithm that empirically is effective at learning neural network policies for challenging control tasks. The results extend the state of the art in using reinforcement learning for high-dimensional continuous control. Videos are available at https://sites.google.com/site/gaepapersupp.

3. 把上述（1）和（2）结合起来，我们得到一套在经验上能够为困难控制任务学习神经网络策略的算法。这些结果推进了用强化学习做高维连续控制的现有水平。视频见 https://sites.google.com/site/gaepapersupp。

2 Preliminaries

2 预备知识

We consider an undiscounted formulation of the policy optimization problem. The initial state $s_{0}$ is sampled from distribution $\rho_{0}$. A trajectory $(s_{0},a_{0},s_{1},a_{1},\dots)$ is generated by sampling actions according to the policy $a_{t}\sim\pi(a_{t}\>|\>s_{t})$ and sampling the states according to the dynamics $s_{t+1}\sim P(s_{t+1}\>|\>s_{t},a_{t})$, until a terminal (absorbing) state is reached. A reward $r_{t}=r(s_{t},a_{t},s_{t+1})$ is received at each timestep. The goal is to maximize the expected total reward $\sum_{t=0}^{\infty}r_{t}$, which is assumed to be finite for all policies. Note that we are not using a discount as part of the problem specification; it will appear below as an algorithm parameter that adjusts a bias-variance tradeoff. But the discounted problem (maximizing $\sum_{t=0}^{\infty}\gamma^{t}r_{t}$) can be handled as an instance of the undiscounted problem in which we absorb the discount factor into the reward function, making it time-dependent.

我们考虑无折扣形式的策略优化问题。初始状态 $s_{0}$ 从分布 $\rho_{0}$ 中采样。轨迹 $(s_{0},a_{0},s_{1},a_{1},\dots)$ 由按策略采样动作 $a_{t}\sim\pi(a_{t}\,|\,s_{t})$、并按动力学采样状态 $s_{t+1}\sim P(s_{t+1}\,|\,s_{t},a_{t})$ 生成，直到到达终止（吸收）状态。每个时间步收到奖励 $r_{t}=r(s_{t},a_{t},s_{t+1})$。目标是最大化期望总奖励 $\sum_{t=0}^{\infty}r_{t}$，并假定它对所有策略都有限。注意，折扣并不是问题本身的一部分；它将作为算法参数出现，用来调节偏差–方差权衡。不过，折扣问题（最大化 $\sum_{t=0}^{\infty}\gamma^{t}r_{t}$）可以看作无折扣问题的一个特例：把折扣因子吸收进奖励函数，使其依赖于时间。

Policy gradient methods maximize the expected total reward by repeatedly estimating the gradient $g\vcentcolon=\nabla_{\theta}\mathbb{E}\left[\sum_{t=0}^{\infty}r_{t}\right]$. There are several different related expressions for the policy gradient, which have the form

策略梯度方法通过反复估计梯度 $g\vcentcolon=\nabla_{\theta}\mathbb{E}\left[\sum_{t=0}^{\infty}r_{t}\right]$ 来最大化期望总奖励。策略梯度有若干相关表达式，形式均为

$$\displaystyle g=\mathbb{E}\left[\sum_{t=0}^{\infty}\Psi_{t}\nabla_{\theta}\log\pi_{\theta}(a_{t}\>|\>s_{t})\right], \tag{1}$$

where $\Psi_{t}$ may be one of the following:

其中 $\Psi_{t}$ 可以是下面之一：

1. $\sum_{t=0}^{\infty}r_{t}$: total reward of the trajectory.

1. $\sum_{t=0}^{\infty}r_{t}$：整条轨迹的总奖励。

2. $\sum_{t^{\prime}=t}^{\infty}r_{t^{\prime}}$: reward following action $a_{t}$.

2. $\sum_{t^{\prime}=t}^{\infty}r_{t^{\prime}}$：动作 $a_{t}$ 之后的奖励。

3. $\sum_{t^{\prime}=t}^{\infty}r_{t^{\prime}}-b(s_{t})$: baselined version of previous formula.

3. $\sum_{t^{\prime}=t}^{\infty}r_{t^{\prime}}-b(s_{t})$：上一式减去基线后的形式。

4. $Q^{\pi}(s_{t},a_{t})$: state-action value function.

4. $Q^{\pi}(s_{t},a_{t})$：状态–动作价值函数。

5. $A^{\pi}(s_{t},a_{t})$: advantage function.

5. $A^{\pi}(s_{t},a_{t})$：优势函数。

6. $r_{t}+V^{\pi}(s_{t+1})-V^{\pi}(s_{t})$: TD residual.

6. $r_{t}+V^{\pi}(s_{t+1})-V^{\pi}(s_{t})$：TD 残差。

The latter formulas use the definitions

后面几式用到如下定义

$$\displaystyle V^{\pi}(s_{t}) \ \displaystyle\vcentcolon=\mathbb{E}{\begin{subarray}{c}s{t+1:\infty},\
a_{t:\infty}\end{subarray}}\left[\sum_{l=0}^{\infty}r_{t+l}\right]\hskip 36.135ptQ^{\pi}(s_{t},a_{t})\vcentcolon=\mathbb{E}{\begin{subarray}{c}s{t+1:\infty},\
a_{t+1:\infty}\end{subarray}}\left[\sum_{l=0}^{\infty}r_{t+l}\right] \ \displaystyle A^{\pi}(s_{t},a_{t}) \ \displaystyle\vcentcolon=Q^{\pi}(s_{t},a_{t})-V^{\pi}(s_{t}),\quad\text{(Advantage function)}. \tag{2}$$

Here, the subscript of $\mathbb{E}$ enumerates the variables being integrated over, where states and actions are sampled sequentially from the dynamics model $P(s_{t+1}\>|\>s_{t},a_{t})$ and policy $\pi(a_{t}\>|\>s_{t})$, respectively. The colon notation $a:b$ refers to the inclusive range $(a,a+1,\dots,b)$. These formulas are well known and straightforward to obtain; they follow directly from Proposition 1, which will be stated shortly.

这里 $\mathbb{E}$ 的下标列出被积分的变量，状态和动作分别依次从动力学模型 $P(s_{t+1}\,|\,s_{t},a_{t})$ 和策略 $\pi(a_{t}\,|\,s_{t})$ 中采样。冒号记法 $a:b$ 表示闭区间 $(a,a+1,\dots,b)$。这些公式广为人知，也不难得到；它们直接由稍后陈述的命题 1 推出。

The choice $\Psi_{t}=A^{\pi}(s_{t},a_{t})$ yields almost the lowest possible variance, though in practice, the advantage function is not known and must be estimated. This statement can be intuitively justified by the following interpretation of the policy gradient: that a step in the policy gradient direction should increase the probability of better-than-average actions and decrease the probability of worse-than-average actions. The advantage function, by it’s definition $A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s)$, measures whether or not the action is better or worse than the policy’s default behavior. Hence, we should choose $\Psi_{t}$ to be the advantage function $A^{\pi}(s_{t},a_{t})$, so that the gradient term $\Psi_{t}\nabla_{\theta}\log\pi_{\theta}(a_{t}\>|\>s_{t})$ points in the direction of increased $\pi_{\theta}(a_{t}\>|\>s_{t})$ if and only if $A^{\pi}(s_{t},a_{t})>0$. See Greensmith et al. (2004) for a more rigorous analysis of the variance of policy gradient estimators and the effect of using a baseline.

取 $\Psi_{t}=A^{\pi}(s_{t},a_{t})$ 几乎能得到最低的方差，但实践中优势函数未知，必须估计。这一点可以用对策略梯度的如下直观解释来说明：沿策略梯度方向走一步，应当提高优于平均的动作的概率，并降低差于平均的动作的概率。由定义 $A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s)$，优势函数衡量的正是该动作是否好于策略的默认行为。因此应取 $\Psi_{t}$ 为优势函数 $A^{\pi}(s_{t},a_{t})$，从而使梯度项 $\Psi_{t}\nabla_{\theta}\log\pi_{\theta}(a_{t}\,|\,s_{t})$ 当且仅当 $A^{\pi}(s_{t},a_{t})>0$ 时指向增大 $\pi_{\theta}(a_{t}\,|\,s_{t})$ 的方向。关于策略梯度估计器方差以及使用基线的影响，更严格的分析见 Greensmith et al.（2004）。

We will introduce a parameter $\gamma$ that allows us to reduce variance by downweighting rewards corresponding to delayed effects, at the cost of introducing bias. This parameter corresponds to the discount factor used in discounted formulations of MDPs, but we treat it as a variance reduction parameter in an undiscounted problem; this technique was analyzed theoretically by Marbach & Tsitsiklis (2003); Kakade (2001b); Thomas (2014). The discounted value functions are given by:

我们将引入参数 $\gamma$，通过降低与延迟效应相对应的奖励权重来减小方差，代价是引入偏差。该参数对应于折扣 MDP 表述中的折扣因子，但在无折扣问题中我们把它当作方差削减参数；Marbach & Tsitsiklis（2003）、Kakade（2001b）、Thomas（2014）曾从理论上分析过这一技巧。折扣价值函数定义为：

$$\displaystyle V^{\pi,\gamma}(s_{t}) \ \displaystyle\vcentcolon=\mathbb{E}{\begin{subarray}{c}s{t+1:\infty},\
a_{t:\infty}\end{subarray}}\left[\sum_{l=0}^{\infty}\gamma^{l}r_{t+l}\right]\hskip 36.135ptQ^{\pi,\gamma}(s_{t},a_{t})\vcentcolon=\mathbb{E}{\begin{subarray}{c}s{t+1:\infty},\
a_{t+1:\infty}\end{subarray}}\left[\sum_{l=0}^{\infty}\gamma^{l}r_{t+l}\right] \ \displaystyle A^{\pi,\gamma}(s_{t},a_{t}) \ \displaystyle\vcentcolon=Q^{\pi,\gamma}(s_{t},a_{t})-V^{\pi,\gamma}(s_{t}). \tag{4}$$

The discounted approximation to the policy gradient is defined as follows:

对策略梯度的折扣近似定义如下：

$$\displaystyle g^{\gamma} \ \displaystyle\vcentcolon=\mathbb{E}{\begin{subarray}{c}s{0:\infty}\
a_{0:\infty}\end{subarray}}\left[\sum_{t=0}^{\infty}A^{\pi,\gamma}(s_{t},a_{t})\nabla_{\theta}\log\pi_{\theta}(a_{t}>|>s_{t})\right]. \tag{6}$$

The following section discusses how to obtain biased (but not too biased) estimators for $A^{\pi,\gamma}$, giving us noisy estimates of the discounted policy gradient in Equation 6.

下一节讨论如何得到 $A^{\pi,\gamma}$ 的有偏（但偏差不过大）估计，从而得到方程 6 中折扣策略梯度的带噪声估计。

Before proceeding, we will introduce the notion of a $\gamma$-just estimator of the advantage function, which is an estimator that does not introduce bias when we use it in place of $A^{\pi,\gamma}$ (which is not known and must be estimated) in Equation 6 to estimate $g^{\gamma}$.[^1] Consider an advantage estimator $\hat{A}_{t}(s_{0:\infty},a_{0:\infty})$, which may in general be a function of the entire trajectory.

[^1]: Note, that we have already introduced bias by using $A^{\pi,\gamma}$ in place of $A^{\pi}$; here we are concerned with obtaining an unbiased estimate of $g^{\gamma}$, which is a biased estimate of the policy gradient of the undiscounted MDP.

在继续之前，我们引入优势函数的 $\gamma$-正当（$\gamma$-just）估计器这一概念：若在方程 6 中用它代替未知、因而必须估计的 $A^{\pi,\gamma}$ 来估计 $g^{\gamma}$，则不会额外引入偏差。[^1] 考虑优势估计器 $\hat{A}_{t}(s_{0:\infty},a_{0:\infty})$，一般说来它可以是整条轨迹的函数。

[^1]: 需要注意，用 $A^{\pi,\gamma}$ 代替 $A^{\pi}$ 已经引入了偏差；这里关心的是得到 $g^{\gamma}$ 的无偏估计，而 $g^{\gamma}$ 本身已是无折扣 MDP 策略梯度的有偏估计。

Definition 1.

The estimator $\hat{A}_{t}$ is $\gamma$-just if

定义 1。

称估计器 $\hat{A}_{t}$ 是 $\gamma$-正当的，如果

$$\displaystyle\mathbb{E}{\begin{subarray}{c}s{0:\infty}\
a_{0:\infty}\end{subarray}}\left[\hat{A}{t}(s{0:\infty},a_{0:\infty})\nabla_{\theta}\log\pi_{\theta}(a_{t}>|>s_{t})\right]=\mathbb{E}{\begin{subarray}{c}s{0:\infty}\
a_{0:\infty}\end{subarray}}\left[A^{\pi,\gamma}(s_{t},a_{t})\nabla_{\theta}\log\pi_{\theta}(a_{t}>|>s_{t})\right]. \tag{7}$$

It follows immediately that if $\hat{A}_{t}$ is $\gamma$-just for all $t$, then

由此立即得到：若 $\hat{A}_{t}$ 对所有 $t$ 都是 $\gamma$-正当的，则

$$\displaystyle\mathbb{E}{\begin{subarray}{c}s{0:\infty}\
a_{0:\infty}\end{subarray}}\left[\sum_{t=0}^{\infty}\hat{A}{t}(s{0:\infty},a_{0:\infty})\nabla_{\theta}\log\pi_{\theta}(a_{t}>|>s_{t})\right]=g^{\gamma} \tag{8}$$

One sufficient condition for $\hat{A}_{t}$ to be $\gamma$-just is that $\hat{A}_{t}$ decomposes as the difference between two functions $Q_{t}$ and $b_{t}$, where $Q_{t}$ can depend on any trajectory variables but gives an unbiased estimator of the $\gamma$-discounted $Q$-function, and $b_{t}$ is an arbitrary function of the states and actions sampled before $a_{t}$.

$\hat{A}_{t}$ 为 $\gamma$-正当的一个充分条件是：它可分解为两个函数 $Q_{t}$ 与 $b_{t}$ 之差，其中 $Q_{t}$ 可以依赖任意轨迹变量，但是 $\gamma$-折扣 $Q$ 函数的无偏估计，而 $b_{t}$ 则是在 $a_{t}$ 之前采样的状态和动作的任意函数。

Proposition 1.

Suppose that $\hat{A}_{t}$ can be written in the form $\hat{A}_{t}(s_{0:\infty},a_{0:\infty})=Q_{t}(s_{t:\infty},a_{t:\infty})-b_{t}(s_{0:t},a_{0:t-1})$ such that for all $(s_{t},a_{t})$, $\mathbb{E}_{s_{t+1:\infty},a_{t+1:\infty}\>|\>s_{t},a_{t}}\left[Q_{t}(s_{t:\infty},a_{t:\infty})\right]=Q^{\pi,\gamma}(s_{t},a_{t})$. Then $\hat{A}$ is $\gamma$-just.

命题 1。

设 $\hat{A}_{t}$ 可写成 $\hat{A}_{t}(s_{0:\infty},a_{0:\infty})=Q_{t}(s_{t:\infty},a_{t:\infty})-b_{t}(s_{0:t},a_{0:t-1})$ 的形式，并且对所有 $(s_{t},a_{t})$ 都有 $\mathbb{E}_{s_{t+1:\infty},a_{t+1:\infty}\,|\,s_{t},a_{t}}\left[Q_{t}(s_{t:\infty},a_{t:\infty})\right]=Q^{\pi,\gamma}(s_{t},a_{t})$。则 $\hat{A}$ 是 $\gamma$-正当的。

The proof is provided in Appendix B. It is easy to verify that the following expressions are $\gamma$-just advantage estimators for $\hat{A}_{t}$:

证明见附录 B。不难验证，下列表达式都是 $\hat{A}_{t}$ 的 $\gamma$-正当优势估计器：

• • $\sum_{l=0}^{\infty}\gamma^{l}r_{t+l}$

• • $Q^{\pi,\gamma}(s_{t},a_{t})$

• • $A^{\pi,\gamma}(s_{t},a_{t})$

• • $r_{t}+\gamma V^{\pi,\gamma}(s_{t+1})-V^{\pi,\gamma}(s_{t})$.

• • $r_{t}+\gamma V^{\pi,\gamma}(s_{t+1})-V^{\pi,\gamma}(s_{t})$。

3 Advantage function estimation

3 优势函数估计

This section will be concerned with producing an accurate estimate $\hat{A}_{t}$ of the discounted advantage function $A^{\pi,\gamma}(s_{t},a_{t})$, which will then be used to construct a policy gradient estimator of the following form:

本节关心的是如何准确估计折扣优势函数 $A^{\pi,\gamma}(s_{t},a_{t})$，记为 $\hat{A}_{t}$，再用它构造如下形式的策略梯度估计器：

$$\displaystyle\hat{g}=\frac{1}{N}\sum_{n=1}^{N}\sum_{t=0}^{\infty}\hat{A}_{t}^{n}\nabla_{\theta}\log\pi_{\theta}(a_{t}^{n}\>|\>s_{t}^{n}) \tag{9}$$

where $n$ indexes over a batch of episodes.

其中 $n$ 遍历一批回合。

Let $V$ be an approximate value function. Define $\delta^{V}_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t})$, i.e., the TD residual of $V$ with discount $\gamma$ (Sutton & Barto, 1998). Note that $\delta^{V}_{t}$ can be considered as an estimate of the advantage of the action $a_{t}$. In fact, if we have the correct value function $V=V^{\pi,\gamma}$, then it is a $\gamma$-just advantage estimator, and in fact, an unbiased estimator of $A^{\pi,\gamma}$:

设 $V$ 是近似价值函数。定义 $\delta^{V}_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t})$，即 $V$ 在折扣 $\gamma$ 下的 TD 残差（Sutton & Barto, 1998）。注意 $\delta^{V}_{t}$ 可视为对动作 $a_{t}$ 优势的一种估计。事实上，若价值函数恰好正确，即 $V=V^{\pi,\gamma}$，则它是 $\gamma$-正当的优势估计器，并且还是 $A^{\pi,\gamma}$ 的无偏估计：

$$\displaystyle\mathbb{E}_{s_{t+1}}\left[\delta^{V^{\pi,\gamma}}_{t}\right] \\ \displaystyle=\mathbb{E}_{s_{t+1}}\left[r_{t}+\gamma V^{\pi,\gamma}(s_{t+1})-V^{\pi,\gamma}(s_{t})\right] \\ \displaystyle=\mathbb{E}_{s_{t+1}}\left[Q^{\pi,\gamma}(s_{t},a_{t})-V^{\pi,\gamma}(s_{t})\right]=A^{\pi,\gamma}(s_{t},a_{t}). \tag{10}$$

However, this estimator is only $\gamma$-just for $V=V^{\pi,\gamma}$, otherwise it will yield biased policy gradient estimates.

不过，该估计器只有在 $V=V^{\pi,\gamma}$ 时才是 $\gamma$-正当的，否则会得到有偏的策略梯度估计。

Next, let us consider taking the sum of $k$ of these $\delta$ terms, which we will denote by $\hat{A}_{t}^{(k)}$

接下来考虑把 $k$ 个这样的 $\delta$ 项加起来，记为 $\hat{A}_{t}^{(k)}$

$$\displaystyle\hat{A}_{t}^{(1)} \\ \displaystyle\vcentcolon=\delta^{V}_{t} \\ \displaystyle=-V(s_{t})+r_{t}+\gamma V(s_{t+1}) \\ \displaystyle\hat{A}_{t}^{(2)} \\ \displaystyle\vcentcolon=\delta^{V}_{t}+\gamma\delta^{V}_{t+1} \\ \displaystyle=-V(s_{t})+r_{t}+\gamma r_{t+1}+\gamma^{2}V(s_{t+2}) \\ \displaystyle\hat{A}_{t}^{(3)} \\ \displaystyle\vcentcolon=\delta^{V}_{t}+\gamma\delta^{V}_{t+1}+\gamma^{2}\delta^{V}_{t+2} \\ \displaystyle=-V(s_{t})+r_{t}+\gamma r_{t+1}+\gamma^{2}r_{t+2}+\gamma^{3}V(s_{t+3}) \tag{11}$$

$$\displaystyle\hat{A}_{t}^{(k)} \\ \displaystyle\vcentcolon=\sum_{l=0}^{k-1}\gamma^{l}\delta^{V}_{t+l}=-V(s_{t})+r_{t}+\gamma r_{t+1}+\dots+\gamma^{k-1}r_{t+k-1}+\gamma^{k}V(s_{t+k}) \tag{14}$$

These equations result from a telescoping sum, and we see that $\hat{A}_{t}^{(k)}$ involves a $k$-step estimate of the returns, minus a baseline term $-V(s_{t})$. Analogously to the case of $\delta^{V}_{t}=\hat{A}_{t}^{(1)}$, we can consider $\hat{A}_{t}^{(k)}$ to be an estimator of the advantage function, which is only $\gamma$-just when $V=V^{\pi,\gamma}$. However, note that the bias generally becomes smaller as $k\rightarrow\infty$, since the term $\gamma^{k}V(s_{t+k})$ becomes more heavily discounted, and the term $-V(s_{t})$ does not affect the bias. Taking $k\rightarrow\infty$, we get

这些等式来自望远镜求和。可以看到，$\hat{A}_{t}^{(k)}$ 包含对回报的 $k$ 步估计，再减去基线项 $-V(s_{t})$。与 $\delta^{V}_{t}=\hat{A}_{t}^{(1)}$ 的情形类似，可以把 $\hat{A}_{t}^{(k)}$ 看作优势函数的估计器，它同样只在 $V=V^{\pi,\gamma}$ 时才是 $\gamma$-正当的。不过请注意，随着 $k\rightarrow\infty$，偏差通常会变小，因为 $\gamma^{k}V(s_{t+k})$ 被更重地折扣，而 $-V(s_{t})$ 并不影响偏差。取 $k\rightarrow\infty$，得到

$$\displaystyle\hat{A}_{t}^{(\infty)}=\sum_{l=0}^{\infty}\gamma^{l}\delta^{V}_{t+l}=-V(s_{t})+\sum_{l=0}^{\infty}\gamma^{l}r_{t+l}, \tag{15}$$

which is simply the empirical returns minus the value function baseline.

这不过是经验回报减去价值函数基线。

The generalized advantage estimator $\mathrm{GAE(\gamma,\lambda)}$ is defined as the exponentially-weighted average of these $k$-step estimators:

广义优势估计器 $\mathrm{GAE(\gamma,\lambda)}$ 定义为这些 $k$ 步估计器的指数加权平均：

$$\displaystyle\hat{A}^{\mathrm{GAE(\gamma,\lambda)}}_{t} \\ \displaystyle\vcentcolon=(1-\lambda)\left(\hat{A}_{t}^{(1)}+\lambda\hat{A}_{t}^{(2)}+\lambda^{2}\hat{A}_{t}^{(3)}+\dots\right) \\ \displaystyle=(1-\lambda)\left(\delta^{V}_{t}+\lambda(\delta^{V}_{t}+\gamma\delta^{V}_{t+1})+\lambda^{2}(\delta^{V}_{t}+\gamma\delta^{V}_{t+1}+\gamma^{2}\delta^{V}_{t+2})+\dots\right) \\ \displaystyle=(1-\lambda)(\delta^{V}_{t}(1+\lambda+\lambda^{2}+\dots)+\gamma\delta^{V}_{t+1}(\lambda+\lambda^{2}+\lambda^{3}+\dots) \\ \displaystyle\ \ \ \ \ \ \ \ \ \ \ +\gamma^{2}\delta^{V}_{t+2}(\lambda^{2}+\lambda^{3}+\lambda^{4}+\dots)+\dots) \\ \displaystyle=(1-\lambda)\left(\delta^{V}_{t}\left(\frac{1}{1-\lambda}\right)+\gamma\delta^{V}_{t+1}\left(\frac{\lambda}{1-\lambda}\right)+\gamma^{2}\delta^{V}_{t+2}\left(\frac{\lambda^{2}}{1-\lambda}\right)+\dots\right) \\ \displaystyle=\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta^{V}_{t+l} \tag{16}$$

From Equation 16, we see that the advantage estimator has a remarkably simple formula involving a discounted sum of Bellman residual terms. Section 4 discusses an interpretation of this formula as the returns in an MDP with a modified reward function. The construction we used above is closely analogous to the one used to define TD($\lambda$) (Sutton & Barto, 1998), however TD($\lambda$) is an estimator of the value function, whereas here we are estimating the advantage function.

由方程 16 可见，该优势估计器有一个相当简洁的公式，即贝尔曼残差项的折扣和。第 4 节把这一公式解释为在修改了奖励函数的 MDP 中的回报。上面的构造与定义 TD($\lambda$)（Sutton & Barto, 1998）时所用的构造非常类似，不过 TD($\lambda$) 估计的是价值函数，而这里估计的是优势函数。

There are two notable special cases of this formula, obtained by setting $\lambda=0$ and $\lambda=1$.

令 $\lambda=0$ 和 $\lambda=1$，可以得到这一公式的两个值得注意的特例。

$$\displaystyle\mathrm{GAE}(\gamma,0):\ \ \ \hat{A}_{t} \\ \displaystyle\vcentcolon=\delta_{t} \\ \displaystyle=r_{t}+\gamma V(s_{t+1})-V(s_{t}) \\ \displaystyle\mathrm{GAE}(\gamma,1):\ \ \ \hat{A}_{t} \\ \displaystyle\vcentcolon=\sum_{l=0}^{\infty}\gamma^{l}\delta_{t+l} \\ \displaystyle=\sum_{l=0}^{\infty}\gamma^{l}r_{t+l}-V(s_{t}) \tag{17}$$

$\mathrm{GAE}(\gamma,1)$ is $\gamma$-just regardless of the accuracy of $V$, but it has high variance due to the sum of terms. $\mathrm{GAE}(\gamma,0)$ is $\gamma$-just for $V=V^{\pi,\gamma}$ and otherwise induces bias, but it typically has much lower variance. The generalized advantage estimator for $0<\lambda<1$ makes a compromise between bias and variance, controlled by parameter $\lambda$.

$\mathrm{GAE}(\gamma,1)$ 不论 $V$ 是否准确都是 $\gamma$-正当的，但由于要对多项求和，方差较高。$\mathrm{GAE}(\gamma,0)$ 只在 $V=V^{\pi,\gamma}$ 时是 $\gamma$-正当的，否则会引入偏差，但通常方差低得多。当 $0<\lambda<1$ 时，广义优势估计器在偏差与方差之间折中，由参数 $\lambda$ 控制。

We’ve described an advantage estimator with two separate parameters $\gamma$ and $\lambda$, both of which contribute to the bias-variance tradeoff when using an approximate value function. However, they serve different purposes and work best with different ranges of values. $\gamma$ most importantly determines the scale of the value function $V^{\pi,\gamma}$, which does not depend on $\lambda$. Taking $\gamma<1$ introduces bias into the policy gradient estimate, regardless of the value function’s accuracy. On the other hand, $\lambda<1$ introduces bias only when the value function is inaccurate. Empirically, we find that the best value of $\lambda$ is much lower than the best value of $\gamma$, likely because $\lambda$ introduces far less bias than $\gamma$ for a reasonably accurate value function.

到这里，我们描述了一个带有两个独立参数 $\gamma$ 和 $\lambda$ 的优势估计器，二者在使用近似价值函数时都会参与偏差–方差权衡，但用途不同，最佳取值范围也不一样。$\gamma$ 最重要的作用是决定价值函数 $V^{\pi,\gamma}$ 的尺度，而这一尺度与 $\lambda$ 无关。取 $\gamma<1$ 会给策略梯度估计引入偏差，无论价值函数多么准确。相比之下，$\lambda<1$ 只在价值函数不准确时才引入偏差。经验上，我们发现 $\lambda$ 的最佳值远低于 $\gamma$ 的最佳值，这很可能是因为在价值函数相当准确时，$\lambda$ 引入的偏差远小于 $\gamma$。

Using the generalized advantage estimator, we can construct a biased estimator of $g^{\gamma}$, the discounted policy gradient from Equation 6:

利用广义优势估计器，可以构造方程 6 中折扣策略梯度 $g^{\gamma}$ 的有偏估计：

$$\displaystyle g^{\gamma} \\ \displaystyle\approx\mathbb{E}\left[\sum_{t=0}^{\infty}\nabla_{\theta}\log\pi_{\theta}(a_{t}\>|\>s_{t})\hat{A}^{\mathrm{GAE(\gamma,\lambda)}}_{t}\right]=\mathbb{E}\left[\sum_{t=0}^{\infty}\nabla_{\theta}\log\pi_{\theta}(a_{t}\>|\>s_{t})\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta^{V}_{t+l}\right], \tag{19}$$

where equality holds when $\lambda=1$.

当 $\lambda=1$ 时等号成立。

4 Interpretation as Reward Shaping

4 作为奖励塑形的解释

In this section, we discuss how one can interpret $\lambda$ as an extra discount factor applied after performing a reward shaping transformation on the MDP. We also introduce the notion of a response function to help understand the bias introduced by $\gamma$ and $\lambda$.

本节讨论如何把 $\lambda$ 解释为：先对 MDP 做一次奖励塑形变换，再额外施加的一个折扣因子。我们还引入响应函数的概念，以帮助理解 $\gamma$ 和 $\lambda$ 所引入的偏差。

Reward shaping (Ng et al., 1999) refers to the following transformation of the reward function of an MDP: let $\Phi:\mathcal{S}\rightarrow\mathbb{R}$ be an arbitrary scalar-valued function on state space, and define the transformed reward function $\tilde{r}$ by

奖励塑形（Ng et al., 1999）指对 MDP 奖励函数做如下变换：设 $\Phi:\mathcal{S}\rightarrow\mathbb{R}$ 是状态空间上任意的标量函数，并定义变换后的奖励函数 $\tilde{r}$ 为

$$\displaystyle\tilde{r}(s,a,s^{\prime})=r(s,a,s^{\prime})+\gamma\Phi(s^{\prime})-\Phi(s), \tag{20}$$

which in turn defines a transformed MDP. This transformation leaves the discounted advantage function $A^{\pi,\gamma}$ unchanged for any policy $\pi$. To see this, consider the discounted sum of rewards of a trajectory starting with state $s_{t}$:

这一变换进而定义了一个新的 MDP。它对任意策略 $\pi$，都保持折扣优势函数 $A^{\pi,\gamma}$ 不变。为看清这一点，考虑从状态 $s_{t}$ 出发的一条轨迹的折扣奖励和：

$$\displaystyle\sum_{l=0}^{\infty}\gamma^{l}\tilde{r}(s_{t+l},a_{t},s_{t+l+1}) \\ \displaystyle=\sum_{l=0}^{\infty}\gamma^{l}r(s_{t+l},a_{t+l},s_{t+l+1})-\Phi(s_{t}). \tag{21}$$

Letting $\tilde{Q}^{\pi,\gamma},\tilde{V}^{\pi,\gamma},\tilde{A}^{\pi,\gamma}$ be the value and advantage functions of the transformed MDP, one obtains from the definitions of these quantities that

记 $\tilde{Q}^{\pi,\gamma},\tilde{V}^{\pi,\gamma},\tilde{A}^{\pi,\gamma}$ 为变换后 MDP 的价值和优势函数，由这些量的定义可得

$$\displaystyle\tilde{Q}^{\pi,\gamma}(s,a) \\ \displaystyle=Q^{\pi,\gamma}(s,a)-\Phi(s) \\ \displaystyle\tilde{V}^{\pi,\gamma}(s,a) \\ \displaystyle=V^{\pi,\gamma}(s)-\Phi(s) \\ \displaystyle\tilde{A}^{\pi,\gamma}(s,a) \\ \displaystyle=(Q^{\pi,\gamma}(s,a)-\Phi(s))-(V^{\pi,\gamma}(s)-\Phi(s))=A^{\pi,\gamma}(s,a). \tag{22}$$

Note that if $\Phi$ happens to be the state-value function $V^{\pi,\gamma}$ from the original MDP, then the transformed MDP has the interesting property that $\tilde{V}^{\pi,\gamma}(s)$ is zero at every state.

注意，若 $\Phi$ 恰好是原 MDP 的状态价值函数 $V^{\pi,\gamma}$，则变换后的 MDP 有一个有趣的性质：在每个状态上 $\tilde{V}^{\pi,\gamma}(s)$ 都为零。

Note that (Ng et al., 1999) showed that the reward shaping transformation leaves the policy gradient and optimal policy unchanged when our objective is to maximize the discounted sum of rewards $\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},s_{t+1})$. In contrast, this paper is concerned with maximizing the undiscounted sum of rewards, where the discount $\gamma$ is used as a variance-reduction parameter.

还要注意，Ng et al.（1999）证明的是：当我们的目标是最大化折扣奖励和 $\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},s_{t+1})$ 时，奖励塑形变换不改变策略梯度和最优策略。相比之下，本文关心的是最大化无折扣奖励和，折扣 $\gamma$ 只是作为方差削减参数来使用。

Having reviewed the idea of reward shaping, let us consider how we could use it to get a policy gradient estimate. The most natural approach is to construct policy gradient estimators that use discounted sums of shaped rewards $\tilde{r}$. However, Equation 21 shows that we obtain the discounted sum of the original MDP’s rewards $r$ minus a baseline term. Next, let’s consider using a “steeper” discount $\gamma\lambda$, where $0\leq\lambda\leq 1$. It’s easy to see that the shaped reward $\tilde{r}$ equals the Bellman residual term $\delta^{V}$, introduced in Section 3, where we set $\Phi=V$. Letting $\Phi=V$, we see that

回顾了奖励塑形之后，再看如何用它来估计策略梯度。最自然的做法是构造使用塑形奖励 $\tilde{r}$ 的折扣和的策略梯度估计器。但方程 21 表明，这样得到的是原 MDP 奖励 $r$ 的折扣和再减去一个基线项。接下来考虑使用更“陡”的折扣 $\gamma\lambda$，其中 $0\leq\lambda\leq 1$。不难看出，取 $\Phi=V$ 时，塑形奖励 $\tilde{r}$ 正好等于第 3 节引入的贝尔曼残差项 $\delta^{V}$。令 $\Phi=V$，我们有

$$\displaystyle\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\tilde{r}(s_{t+l},a_{t},s_{t+l+1}) \\ \displaystyle=\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta^{V}_{t+l}=\hat{A}^{\mathrm{GAE(\gamma,\lambda)}}_{t}. \tag{25}$$

Hence, by considering the $\gamma\lambda$-discounted sum of shaped rewards, we exactly obtain the generalized advantage estimators from Section 3. As shown previously, $\lambda=1$ gives an unbiased estimate of $g^{\gamma}$, whereas $\lambda<1$ gives a biased estimate.

因此，考虑塑形奖励的 $\gamma\lambda$-折扣和，恰好得到第 3 节中的广义优势估计器。如前所述，$\lambda=1$ 给出 $g^{\gamma}$ 的无偏估计，而 $\lambda<1$ 给出有偏估计。

To further analyze the effect of this shaping transformation and parameters $\gamma$ and $\lambda$, it will be useful to introduce the notion of a response function $\chi$, which we define as follows:

为进一步分析这一塑形变换以及参数 $\gamma$ 和 $\lambda$ 的作用，引入如下定义的响应函数 $\chi$ 会很有用：

$$\displaystyle\chi(l;s_{t},a_{t})=\mathbb{E}\left[r_{t+l}\>|\>s_{t},a_{t}\right]-\mathbb{E}\left[r_{t+l}\>|\>s_{t}\right]. \tag{26}$$

Note that $A^{\pi,\gamma}(s,a)=\sum_{l=0}^{\infty}\gamma^{l}\chi(l;s,a)$, hence the response function decomposes the advantage function across timesteps. The response function lets us quantify the temporal credit assignment problem: long range dependencies between actions and rewards correspond to nonzero values of the response function for $l\gg 0$.

注意 $A^{\pi,\gamma}(s,a)=\sum_{l=0}^{\infty}\gamma^{l}\chi(l;s,a)$，因此响应函数把优势函数按时间步分解开来。响应函数让我们能量化时间信用分配问题：动作与奖励之间的长程依赖，对应于 $l\gg 0$ 时响应函数的非零值。

Next, let us revisit the discount factor $\gamma$ and the approximation we are making by using $A^{\pi,\gamma}$ rather than $A^{\pi,1}$. The discounted policy gradient estimator from Equation 6 has a sum of terms of the form

接下来再看折扣因子 $\gamma$，以及用 $A^{\pi,\gamma}$ 代替 $A^{\pi,1}$ 所做的近似。方程 6 中的折扣策略梯度估计器包含如下形式的项

$$\displaystyle\nabla_{\theta}\log\pi_{\theta}(a_{t}\>|\>s_{t})A^{\pi,\gamma}(s_{t},a_{t}) \\ \displaystyle=\nabla_{\theta}\log\pi_{\theta}(a_{t}\>|\>s_{t})\sum_{l=0}^{\infty}\gamma^{l}\chi(l;s_{t},a_{t}). \tag{27}$$

Using a discount $\gamma<1$ corresponds to dropping the terms with $l\gg 1/(1-\gamma)$. Thus, the error introduced by this approximation will be small if $\chi$ rapidly decays as $l$ increases, i.e., if the effect of an action on rewards is “forgotten” after $\approx 1/(1-\gamma)$ timesteps.

使用 $\gamma<1$ 的折扣，相当于丢掉 $l\gg 1/(1-\gamma)$ 的那些项。因此，若 $\chi$ 随 $l$ 增大而迅速衰减，也就是说一个动作对奖励的影响在大约 $1/(1-\gamma)$ 步之后就被“遗忘”，则这一近似引入的误差会很小。

If the reward function $\tilde{r}$ were obtained using $\Phi=V^{\pi,\gamma}$, we would have $\mathbb{E}\left[\tilde{r}_{t+l}\>|\>s_{t},a_{t}\right]=\mathbb{E}\left[\tilde{r}_{t+l}\>|\>s_{t}\right]=0$ for $l>0$, i.e., the response function would only be nonzero at $l=0$. Therefore, this shaping transformation would turn temporally extended response into an immediate response. Given that $V^{\pi,\gamma}$ completely reduces the temporal spread of the response function, we can hope that a good approximation $V\approx V^{\pi,\gamma}$ partially reduces it. This observation suggests an interpretation of Equation 16: reshape the rewards using $V$ to shrink the temporal extent of the response function, and then introduce a “steeper” discount $\gamma\lambda$ to cut off the noise arising from long delays, i.e., ignore terms $\nabla_{\theta}\log\pi_{\theta}(a_{t}\>|\>s_{t})\delta^{V}_{t+l}$ where $l\gg 1/(1-\gamma\lambda)$.

若奖励函数 $\tilde{r}$ 是用 $\Phi=V^{\pi,\gamma}$ 得到的，则会有 $\mathbb{E}\left[\tilde{r}_{t+l}\,|\,s_{t},a_{t}\right]=\mathbb{E}\left[\tilde{r}_{t+l}\,|\,s_{t}\right]=0$（对 $l>0$），即响应函数只在 $l=0$ 时非零。因此，这一塑形变换会把时间上延展的响应变成即时响应。既然 $V^{\pi,\gamma}$ 能完全压缩响应函数的时间跨度，我们也可以希望一个好的近似 $V\approx V^{\pi,\gamma}$ 能部分地做到这一点。这一观察提示了对方程 16 的一种解释：先用 $V$ 重塑奖励以缩小响应函数的时间范围，再引入更“陡”的折扣 $\gamma\lambda$ 来截断长延迟带来的噪声，也就是忽略那些 $l\gg 1/(1-\gamma\lambda)$ 的项 $\nabla_{\theta}\log\pi_{\theta}(a_{t}\,|\,s_{t})\delta^{V}_{t+l}$。

5 Value Function Estimation

5 价值函数估计

A variety of different methods can be used to estimate the value function (see, e.g., Bertsekas (2012)). When using a nonlinear function approximator to represent the value function, the simplest approach is to solve a nonlinear regression problem:

估计价值函数可用多种方法（例如参见 Bertsekas, 2012）。当用非线性函数逼近器表示价值函数时，最简单的做法是求解一个非线性回归问题：

$$\displaystyle\minimize_{\phi}\sum_{n=1}^{N}\|V_{\phi}(s_{n})-\hat{V}_{n}\|^{2}, \tag{28}$$

where $\hat{V}_{t}=\sum_{l=0}^{\infty}\gamma^{l}r_{t+l}$ is the discounted sum of rewards, and $n$ indexes over all timesteps in a batch of trajectories. This is sometimes called the Monte Carlo or TD($1$) approach for estimating the value function (Sutton & Barto, 1998).[^2]

[^2]: Another natural choice is to compute target values with an estimator based on the TD($\lambda$) backup (Bertsekas, 2012; Sutton & Barto, 1998), mirroring the expression we use for policy gradient estimation: $\hat{V^{\lambda}_{t}}=V_{\phi_{\text{old}}}(s_{n})+\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta_{t+l}$. While we experimented with this choice, we did not notice a difference in performance from the $\lambda=1$ estimator in Equation 28.

其中 $\hat{V}_{t}=\sum_{l=0}^{\infty}\gamma^{l}r_{t+l}$ 是折扣奖励和，$n$ 遍历一批轨迹中的所有时间步。这有时称为估计价值函数的蒙特卡洛或 TD($1$) 方法（Sutton & Barto, 1998）。[^2]

[^2]: 另一个自然的选择是用基于 TD($\lambda$) 备份的估计器来计算目标值（Bertsekas, 2012; Sutton & Barto, 1998），形式与我们用于策略梯度估计的表达式相对应：$\hat{V^{\lambda}_{t}}=V_{\phi_{\text{old}}}(s_{n})+\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta_{t+l}$。我们试验过这一选择，但并未发现它与方程 28 中 $\lambda=1$ 的估计器在性能上有明显差别。

For the experiments in this work, we used a trust region method to optimize the value function in each iteration of a batch optimization procedure. The trust region helps us to avoid overfitting to the most recent batch of data. To formulate the trust region problem, we first compute $\sigma^{2}=\frac{1}{N}\sum_{n=1}^{N}\|V_{\phi_{\text{old}}}(s_{n})-\hat{V}_{n}\|^{2}$, where $\phi_{\text{old}}$ is the parameter vector before optimization. Then we solve the following constrained optimization problem:

在本工作的实验中，我们在批量优化的每次迭代里用信赖域方法优化价值函数。信赖域有助于避免对最近一批数据过拟合。为表述信赖域问题，我们先计算 $\sigma^{2}=\frac{1}{N}\sum_{n=1}^{N}\|V_{\phi_{\text{old}}}(s_{n})-\hat{V}_{n}\|^{2}$，其中 $\phi_{\text{old}}$ 是优化前的参数向量。然后求解如下约束优化问题：

$$\displaystyle\minimize_{\phi}\quad\sum_{n=1}^{N}\|V_{\phi}(s_{n})-\hat{V}_{n}\|^{2} \\ \displaystyle\text{subject to}\quad\frac{1}{N}\sum_{n=1}^{N}\frac{\|V_{\phi}(s_{n})-V_{\phi_{\text{old}}}(s_{n})\|^{2}}{2\sigma^{2}}\leq\epsilon. \tag{29}$$

This constraint is equivalent to constraining the average KL divergence between the previous value function and the new value function to be smaller than $\epsilon$, where the value function is taken to parameterize a conditional Gaussian distribution with mean $V_{\phi}(s)$ and variance $\sigma^{2}$.

这一约束等价于把新旧价值函数之间的平均 KL 散度限制在小于 $\epsilon$，这里把价值函数理解为均值 $V_{\phi}(s)$、方差 $\sigma^{2}$ 的条件高斯分布的参数。

We compute an approximate solution to the trust region problem using the conjugate gradient algorithm (Wright & Nocedal, 1999). Specifically, we are solving the quadratic program

我们用共轭梯度算法（Wright & Nocedal, 1999）求信赖域问题的近似解。具体地说，是在求解二次规划

$$\displaystyle\minimize_{\phi}\quad g^{T}(\phi-\phi_{\text{old}}) \\ \displaystyle\text{subject to}\quad\frac{1}{N}\sum_{n=1}^{N}(\phi-\phi_{\text{old}})^{T}H(\phi-\phi_{\text{old}})\leq\epsilon. \tag{30}$$

where $g$ is the gradient of the objective, and $H=\frac{1}{N}\sum_{n}j_{n}j_{n}^{T}$, where $j_{n}=\nabla_{\phi}V_{\phi}(s_{n})$. Note that $H$ is the “Gauss-Newton” approximation of the Hessian of the objective, and it is (up to a $\sigma^{2}$ factor) the Fisher information matrix when interpreting the value function as a conditional probability distribution. Using matrix-vector products $v\rightarrow Hv$ to implement the conjugate gradient algorithm, we compute a step direction $s\approx-H^{-1}g$. Then we rescale $s\rightarrow\alpha s$ such that $\frac{1}{2}(\alpha s)^{T}H(\alpha s)=\epsilon$ and take $\phi=\phi_{\text{old}}+\alpha s$. This procedure is analogous to the procedure we use for updating the policy, which is described further in Section 6 and based on Schulman et al. (2015).

其中 $g$ 是目标函数的梯度，$H=\frac{1}{N}\sum_{n}j_{n}j_{n}^{T}$，而 $j_{n}=\nabla_{\phi}V_{\phi}(s_{n})$。注意 $H$ 是目标函数 Hessian 的“高斯–牛顿”近似，并且在把价值函数解释为条件概率分布时，（相差一个 $\sigma^{2}$ 因子）它就是 Fisher 信息矩阵。我们用矩阵–向量乘积 $v\rightarrow Hv$ 来实现共轭梯度算法，算出步方向 $s\approx-H^{-1}g$。然后将 $s$ 缩放为 $\alpha s$，使得 $\frac{1}{2}(\alpha s)^{T}H(\alpha s)=\epsilon$，并取 $\phi=\phi_{\text{old}}+\alpha s$。这一流程与我们更新策略所用的流程类似，详见第 6 节，并基于 Schulman et al.（2015）。

6 Experiments

6 实验

We designed a set of experiments to investigate the following questions:

我们设计了一组实验来考察下列问题：

1. What is the empirical effect of varying $\lambda\in[0,1]$ and $\gamma\in[0,1]$ when optimizing episodic total reward using generalized advantage estimation?

1. 在用广义优势估计优化回合总奖励时，改变 $\lambda\in[0,1]$ 和 $\gamma\in[0,1]$ 的经验效果如何？

2. Can generalized advantage estimation, along with trust region algorithms for policy and value function optimization, be used to optimize large neural network policies for challenging control problems?

2. 广义优势估计，再加上用于策略和价值函数优化的信赖域算法，能否用来为困难的控制问题优化大型神经网络策略？