Deterministic Policy Gradient Algorithms
Deterministic Policy Gradient Algorithms
Abstract
摘要
In this paper we consider deterministic policy gradient algorithms for reinforcement learning with continuous actions. The deterministic policy gradient has a particularly appealing form: it is the expected gradient of the action-value function. This simple form means that the deterministic policy gradient can be estimated much more efficiently than the usual stochastic policy gradient. To ensure adequate exploration, we introduce an off-policy actor-critic algorithm that learns a deterministic target policy from an exploratory behaviour policy. We demonstrate that deterministic policy gradient algorithms can significantly outperform their stochastic counterparts in high-dimensional action spaces.
本文讨论用于连续动作强化学习的确定性策略梯度算法。确定性策略梯度有一种特别简洁的形式:它是动作值函数梯度的期望。这一简单形式意味着,确定性策略梯度可以比通常的随机策略梯度高效得多地估计。为确保充分探索,我们引入一种 off-policy actor-critic 算法,它从具有探索性的行为策略中学习确定性的目标策略。我们表明,在高维动作空间中,确定性策略梯度算法可以显著优于其随机对照方法。
Proceedings of the 31st International Conference on Machine Learning, Beijing, China, 2014. JMLR: W&CP volume 32. Copyright 2014 by the author(s).
《第 31 届国际机器学习会议论文集》,中国北京,2014。JMLR: W&CP 第 32 卷。版权归作者所有(2014)。
- Introduction
- 引言
Policy gradient algorithms are widely used in reinforcement learning problems with continuous action spaces. The basic idea is to represent the policy by a parametric probability distribution that stochastically selects action in state according to parameter vector . Policy gradient algorithms typically proceed by sampling this stochastic policy and adjusting the policy parameters in the direction of greater cumulative reward.
策略梯度算法被广泛用于具有连续动作空间的强化学习问题。基本想法是用一个参数化的概率分布 来表示策略,它根据参数向量 ,在状态 中随机选择动作 。策略梯度算法通常通过对这一随机策略采样,并沿累积回报增大的方向调整策略参数。
In this paper we instead consider deterministic policies . It is natural to wonder whether the same approach can be followed as for stochastic policies: adjusting the policy parameters in the direction of the policy gradient. It was previously believed that the deterministic policy gradient did not exist, or could only be obtained when using a model (Peters, 2010). However, we show that the deterministic policy gradient does indeed exist, and furthermore it has a simple model-free form that simply follows the gradient of the action-value function. In addition, we show that the deterministic policy gradient is the limiting case, as policy variance tends to zero, of the stochastic policy gradient.
在本文中,我们转而考虑确定性策略 。很自然会问:是否也能像随机策略那样,把策略参数沿策略梯度的方向调整。此前人们认为确定性策略梯度并不存在,或者只能在使用模型时得到(Peters, 2010)。然而我们表明,确定性策略梯度确实存在,并且具有一种简单的无模型形式,只需跟随动作值函数的梯度。此外,我们还表明,确定性策略梯度是随机策略梯度在策略方差趋于零时的极限情形。
From a practical viewpoint, there is a crucial difference between the stochastic and deterministic policy gradients. In the stochastic case, the policy gradient integrates over both state and action spaces, whereas in the deterministic case it only integrates over the state space. As a result, computing the stochastic policy gradient may require more samples, especially if the action space has many dimensions.
从实践角度看,随机策略梯度与确定性策略梯度有一个关键差别。在随机情形下,策略梯度要对状态空间和动作空间都做积分,而在确定性情形下只需对状态空间积分。因此,计算随机策略梯度可能需要更多样本,尤其当动作空间维数很高时。
In order to explore the full state and action space, a stochastic policy is often necessary. To ensure that our deterministic policy gradient algorithms continue to explore satisfactorily, we introduce an off-policy learning algorithm. The basic idea is to choose actions according to a stochastic behaviour policy (to ensure adequate exploration), but to learn about a deterministic target policy (exploiting the efficiency of the deterministic policy gradient). We use the deterministic policy gradient to derive an off-policy actor-critic algorithm that estimates the action-value function using a differentiable function approximator, and then updates the policy parameters in the direction of the approximate action-value gradient. We also introduce a notion of compatible function approximation for deterministic policy gradients, to ensure that the approximation does not bias the policy gradient.
为了探索整个状态和动作空间,往往需要随机策略。为确保我们的确定性策略梯度算法仍然能够充分探索,我们引入一种 off-policy 学习算法。基本想法是:按随机的行为策略选择动作(以保证充分探索),但学习的是确定性的目标策略(以利用确定性策略梯度的效率)。我们用确定性策略梯度导出一种 off-policy actor-critic 算法:它用可微的函数近似器估计动作值函数,然后沿近似动作值梯度的方向更新策略参数。我们还为确定性策略梯度引入兼容函数近似(compatible function approximation)的概念,以确保近似不会给策略梯度带来偏差。
We apply our deterministic actor-critic algorithms to several benchmark problems: a high-dimensional bandit; several standard benchmark reinforcement learning tasks with low dimensional action spaces; and a high-dimensional task for controlling an octopus arm. Our results demonstrate a significant performance advantage to using deterministic policy gradients over stochastic policy gradients, particularly in high dimensional tasks. Furthermore, our algorithms require no more computation than prior methods: the computational cost of each update is linear in the action dimensionality and the number of policy parameters. Finally, there are many applications (for example in robotics) where a differentiable control policy is provided, but where there is no functionality to inject noise into the controller. In these cases, the stochastic policy gradient is inapplicable, whereas our methods may still be useful.
我们把确定性 actor-critic 算法应用到若干基准问题上:一个高维赌博机;几个动作空间维数较低的标准强化学习基准任务;以及一个控制章鱼臂的高维任务。结果表明,使用确定性策略梯度相对随机策略梯度有显著的性能优势,在高维任务中尤其明显。此外,我们的算法并不比先前方法需要更多计算:每次更新的计算代价对动作维数和策略参数数目都是线性的。最后,有许多应用(例如在机器人中)会提供可微的控制策略,却没有向控制器注入噪声的功能。在这些情形下,随机策略梯度并不适用,而我们的方法可能仍然有用。
- Background
- 背景
2.1. Preliminaries
2.1. 预备知识
We study reinforcement learning and control problems in which an agent acts in a stochastic environment by sequentially choosing actions over a sequence of time steps, in order to maximise a cumulative reward. We model the problem as a Markov decision process (MDP) which comprises: a state space , an action space , an initial state distribution with density , a stationary transition dynamics distribution with conditional density satisfying the Markov property , for any trajectory in state-action space, and a reward function .
我们研究强化学习和控制问题:智能体在随机环境中,通过在一串时间步上依次选择动作,以最大化累积奖励。我们把问题建模为马尔可夫决策过程(MDP),它包括:状态空间 ,动作空间 ,具有密度 的初始状态分布,满足马尔可夫性质 的平稳转移动力学条件密度 (对状态-动作空间中的任意轨迹 ),以及奖励函数 。
A policy is used to select actions in the MDP. In general the policy is stochastic and denoted by , where is the set of probability measures on and is a vector of parameters, and is the conditional probability density at associated with the policy. The agent uses its policy to interact with the MDP to give a trajectory of states, actions and rewards, over . The return is the total discounted reward from time-step onwards, where . Value functions are defined to be the expected total discounted reward, and . The agent's goal is to obtain a policy which maximises the cumulative discounted reward from the start state, denoted by the performance objective .
策略用于在 MDP 中选择动作。一般情形下策略是随机的,记为 ,其中 是 上的概率测度集合, 是由 个参数组成的向量,而 是该策略在 处的条件概率密度。智能体用其策略与 MDP 交互,得到一条状态、动作和奖励的轨迹 ,取值于 。回报 是从时间步 起的总折扣奖励,,其中 。值函数定义为期望总折扣奖励,,。 智能体的目标是得到一个从起始状态起最大化累积折扣奖励的策略,该目标记为性能目标 。
To simplify notation, we frequently drop the random variable in the conditional density and write ; furthermore we superscript value functions by rather than .
为简化记号,我们常常在条件密度中省去随机变量,写成 ;此外,我们把值函数上标写成 而不是 。
We denote the density at state after transitioning for time steps from state by . We also denote the (improper) discounted state distribution by . We can then write the performance objective as an expectation,
我们把从状态 出发、经过 个时间步转移到状态 的密度记为 。我们还把(非正常的)折扣状态分布记为 。于是可以把性能目标写成一个期望,
where denotes the (improper) expected value with respect to discounted state distribution . In the remainder of the paper we suppose for simplicity that and that is a compact subset of .
其中 表示关于折扣状态分布 的(非正常)期望。 在本文余下部分,为简单起见,我们假定 ,并且 是 的一个紧子集。
The results in this paper may be extended to an average reward performance objective by choosing to be the stationary distribution of an ergodic MDP.
本文的结果可以推广到平均奖励性能目标,只需把 取为遍历 MDP 的平稳分布。
2.2. Stochastic Policy Gradient Theorem
2.2. 随机策略梯度定理
Policy gradient algorithms are perhaps the most popular class of continuous action reinforcement learning algorithms. The basic idea behind these algorithms is to adjust the parameters of the policy in the direction of the performance gradient . The fundamental result underlying these algorithms is the policy gradient theorem (Sutton et al., 1999),
策略梯度算法大概是连续动作强化学习算法中最流行的一类。这些算法的基本想法,是把策略参数 沿性能梯度 的方向调整。作为其基础的根本结果是策略梯度定理(Sutton et al., 1999),
The policy gradient is surprisingly simple. In particular, despite the fact that the state distribution depends on the policy parameters, the policy gradient does not depend on the gradient of the state distribution.
这一策略梯度出奇地简单。特别是,尽管状态分布 依赖于策略参数,策略梯度却并不依赖于状态分布的梯度。
This theorem has important practical value, because it reduces the computation of the performance gradient to a simple expectation. The policy gradient theorem has been used to derive a variety of policy gradient algorithms (Degris et al., 2012a), by forming a sample-based estimate of this expectation. One issue that these algorithms must address is how to estimate the action-value function . Perhaps the simplest approach is to use a sample return to estimate the value of , which leads to a variant of the REINFORCE algorithm (Williams, 1992).
这一定理有重要的实践价值,因为它把性能梯度的计算化简成一个简单的期望。人们已经用策略梯度定理,通过对这一期望做基于样本的估计,导出了多种策略梯度算法(Degris et al., 2012a)。这些算法必须面对的一个问题,是如何估计动作值函数 。也许最简单的做法,是用样本回报 来估计 的值,由此得到 REINFORCE 算法的一种变体(Williams, 1992)。
2.3. Stochastic Actor-Critic Algorithms
2.3. 随机 Actor-Critic 算法
The actor-critic is a widely used architecture based on the policy gradient theorem (Sutton et al., 1999; Peters et al., 2005; Bhatnagar et al., 2007; Degris et al., 2012a). The actor-critic consists of two eponymous components. An actor adjusts the parameters of the stochastic policy by stochastic gradient ascent of Equation 2. Instead of the unknown true action-value function in Equation 2, an action-value function is used, with parameter vector . A critic estimates the action-value function using an appropriate policy evaluation algorithm such as temporal-difference learning.
Actor-critic 是一种基于策略梯度定理、被广泛使用的架构(Sutton et al., 1999; Peters et al., 2005; Bhatnagar et al., 2007; Degris et al., 2012a)。它由名称所示的两个部分组成。Actor 通过对式 (2) 做随机梯度上升,来调整随机策略 的参数 。式 (2) 中未知的真实动作值函数 被换成带参数向量 的动作值函数 。Critic 则用合适的策略评估算法(例如时间差分学习)来估计动作值函数 。
In general, substituting a function approximator for the true action-value function may introduce bias. However, if the function approximator is compatible such that i) and ii) the parameters are chosen to minimise the mean-squared error , then there is no bias (Sutton et al., 1999),
一般而言,用函数近似器 代替真实动作值函数 可能会引入偏差。不过,如果函数近似器是兼容的,即满足 i) ,并且 ii) 参数 被选来最小化均方误差 ,则不会有偏差(Sutton et al., 1999),
More intuitively, condition i) says that compatible function approximators are linear in “features” of the stochastic policy, , and condition ii) requires that the parameters are the solution to the linear regression problem that estimates from these features. In practice, condition ii) is usually relaxed in favour of policy evaluation algorithms that estimate the value function more efficiently by temporal-difference learning (Bhatnagar et al., 2007; Degris et al., 2012b; Peters et al., 2005); indeed if both i) and ii) are satisfied then the overall algorithm is equivalent to not using a critic at all (Sutton et al., 2000), much like the REINFORCE algorithm (Williams, 1992).
更直观地说,条件 i) 表明兼容函数近似器对随机策略的“特征” 是线性的,而条件 ii) 要求参数是从这些特征估计 的线性回归问题的解。在实践中,条件 ii) 通常会被放宽,转而采用通过时间差分学习更高效地估计值函数的策略评估算法(Bhatnagar et al., 2007; Degris et al., 2012b; Peters et al., 2005);事实上,如果 i) 和 ii) 同时满足,整个算法就等价于根本不使用 critic(Sutton et al., 2000),很像 REINFORCE 算法(Williams, 1992)。
2.4. Off-Policy Actor-Critic
It is often useful to estimate the policy gradient off-policy from trajectories sampled from a distinct behaviour policy . In an off-policy setting, the performance objective is typically modified to be the value function of the target policy, averaged over the state distribution of the behaviour policy (Degris et al., 2012b),
常常有用的一种做法,是从由不同的行为策略 采样得到的轨迹中,off-policy 地估计策略梯度。在 off-policy 设定下,性能目标通常被改写成目标策略的值函数,并在行为策略的状态分布上取平均(Degris et al., 2012b),
Differentiating the performance objective and applying an approximation gives the off-policy policy-gradient (Degris et al., 2012b)
对性能目标求导并施加一个近似,就得到 off-policy 策略梯度(Degris et al., 2012b)
This approximation drops a term that depends on the action-value gradient ; Degris et al. (2012b) argue that this is a good approximation since it can preserve the set of local optima to which gradient ascent converges. The Off-Policy Actor-Critic (OffPAC) algorithm (Degris et al., 2012b) uses a behaviour policy to generate trajectories. A critic estimates a state-value function, , off-policy from these trajectories, by gradient temporal-difference learning (Sutton et al., 2009). An actor updates the policy parameters , also off-policy from these trajectories, by stochastic gradient ascent of Equation 5. Instead of the unknown action-value function in Equation 5, the temporal-difference error is used, ; this can be shown to provide an approximation to the true gradient (Bhatnagar et al., 2007). Both the actor and the critic use an importance sampling ratio to adjust for the fact that actions were selected according to rather than .
这一近似丢掉了一个依赖于动作值梯度 的项;Degris 等人(2012b)认为这是一个好的近似,因为它可以保持梯度上升所收敛到的局部最优集合。Off-Policy Actor-Critic(OffPAC)算法(Degris et al., 2012b)用行为策略 生成轨迹。Critic 借助梯度时间差分学习(Sutton et al., 2009),从这些轨迹中 off-policy 地估计状态值函数 。Actor 同样从这些轨迹中 off-policy 地对式 (5) 做随机梯度上升,以更新策略参数 。式 (5) 中未知的动作值函数 被换成时间差分误差 ,其中 ;可以证明这为真实梯度提供了一种近似(Bhatnagar et al., 2007)。Actor 和 critic 都使用重要性采样比 ,以修正动作是按 而不是按 选择的这一事实。
- Gradients of Deterministic Policies
- 确定性策略的梯度
We now consider how the policy gradient framework may be extended to deterministic policies. Our main result is a deterministic policy gradient theorem, analogous to the stochastic policy gradient theorem presented in the previous section. We provide several ways to derive and understand this result. First we provide an informal intuition behind the form of the deterministic policy gradient. We then give a formal proof of the deterministic policy gradient theorem from first principles. Finally, we show that the deterministic policy gradient theorem is in fact a limiting case of the stochastic policy gradient theorem. Details of the proofs are deferred until the appendices.
我们现在考虑如何把策略梯度框架扩展到确定性策略。我们的主要结果是一条确定性策略梯度定理,与上一节给出的随机策略梯度定理相对应。我们提供几种推导和理解这一结果的方式。首先给出关于确定性策略梯度形式的非正式直观。然后从第一原理给出确定性策略梯度定理的形式证明。最后,我们表明确定性策略梯度定理实际上是随机策略梯度定理的一种极限情形。证明细节留到附录。
3.1. Action-Value Gradients
3.1. 动作值梯度
The majority of model-free reinforcement learning algorithms are based on generalised policy iteration: interleaving policy evaluation with policy improvement (Sutton and Barto, 1998). Policy evaluation methods estimate the action-value function or , for example by Monte-Carlo evaluation or temporal-difference learning. Policy improvement methods update the policy with respect to the (estimated) action-value function. The most common approach is a greedy maximisation (or soft maximisation) of the action-value function, .
大多数无模型强化学习算法都基于广义策略迭代:把策略评估与策略改进交错进行(Sutton and Barto, 1998)。策略评估方法估计动作值函数 或 ,例如用蒙特卡洛评估或时间差分学习。策略改进方法则根据(估计的)动作值函数来更新策略。最常见的做法是对动作值函数做贪心最大化(或软最大化),。
In continuous action spaces, greedy policy improvement becomes problematic, requiring a global maximisation at every step. Instead, a simple and computationally attractive alternative is to move the policy in the direction of the gradient of , rather than globally maximising . Specifically, for each visited state , the policy parameters are updated in proportion to the gradient . Each state suggests a different direction of policy improvement; these may be averaged together by taking an expectation with respect to the state distribution ,
在连续动作空间中,贪心策略改进会变得棘手,因为每一步都需要做全局最大化。一个简单且计算上有吸引力的替代做法,是把策略沿 的梯度方向移动,而不是全局最大化 。具体而言,对每一个访问到的状态 ,策略参数 按梯度 的比例更新。每个状态都会给出一个不同的策略改进方向;这些方向可以通过对状态分布 取期望来平均,
By applying the chain rule we see that the policy improvement may be decomposed into the gradient of the action-value with respect to actions, and the gradient of the policy with respect to the policy parameters.
利用链式法则可以看到,策略改进可以分解为动作值对动作的梯度,以及策略对策略参数的梯度。
By convention is a Jacobian matrix such that each column is the gradient of the th action dimension of the policy with respect to the policy parameters . However, by changing the policy, different states are visited and the state distribution will change. As a result it is not immediately obvious that this approach guarantees improvement, without taking account of the change to distribution. However, the theory below shows that, like the stochastic policy gradient theorem, there is no need to compute the gradient of the state distribution; and that the intuitive update outlined above is following precisely the gradient of the performance objective.
按惯例, 是一个雅可比矩阵,其每一列是策略第 个动作维对策略参数 的梯度 。不过,改变策略后,访问到的状态会不同,状态分布 也会改变。因此,若不考虑分布的变化,这种做法是否保证改进并不显然。然而下面的理论表明:与随机策略梯度定理一样,并不需要计算状态分布的梯度;并且上面给出的直观更新,恰好就是沿着性能目标的梯度前进。
3.2. Deterministic Policy Gradient Theorem
3.2. 确定性策略梯度定理
We now formally consider a deterministic policy with parameter vector . We define a performance objective , and define probability distribution and discounted state distribution analogously to the stochastic case. This again lets us to write the performance objective as an expectation,
我们现在形式地考虑确定性策略 ,其参数向量为 。我们定义性能目标 ,并与随机情形类似地定义概率分布 和折扣状态分布 。这再次让我们得以把性能目标写成一个期望,
We now provide the deterministic analogue to the policy gradient theorem. The proof follows a similar scheme to (Sutton et al., 1999) and is provided in Appendix B.
我们现在给出与策略梯度定理相对应的确定性形式。证明沿用与(Sutton et al., 1999)类似的思路,并放在附录 B 中。
Theorem 1 (Deterministic Policy Gradient Theorem). Suppose that the MDP satisfies conditions A.1 (see Appendix; these imply that and exist and that the deterministic policy gradient exists. Then,
定理 1(确定性策略梯度定理)。假定 MDP 满足条件 A.1(见附录;这些条件蕴含 和 存在,并且确定性策略梯度存在)。那么,
3.3. Limit of the Stochastic Policy Gradient
3.3. 随机策略梯度的极限
The deterministic policy gradient theorem does not at first glance look like the stochastic version (Equation 2). However, we now show that, for a wide class of stochastic policies, including many bump functions, the deterministic policy gradient is indeed a special (limiting) case of the stochastic policy gradient. We parametrise stochastic policies by a deterministic policy and a variance parameter , such that for the stochastic policy is equivalent to the deterministic policy, . Then we show that as the stochastic policy gradient converges to the deterministic gradient (see Appendix C for proof and technical conditions).
确定性策略梯度定理乍看起来并不像随机版本(式 (2))。不过我们现在表明,对包括许多凸包函数(bump functions)在内的一大类随机策略,确定性策略梯度的确是随机策略梯度的一种特殊(极限)情形。我们用确定性策略 和方差参数 来参数化随机策略 ,使得当 时随机策略等价于确定性策略,。然后我们表明,当 时,随机策略梯度收敛到确定性梯度(证明和技术条件见附录 C)。
Theorem 2. Consider a stochastic policy such that , where is a parameter controlling the variance and satisfy conditions B.1 and the MDP satisfies conditions A.1 and A.2. Then,
定理 2. 考虑随机策略 ,满足 ,其中 是控制方差的参数, 满足条件 B.1,并且 MDP 满足条件 A.1 和 A.2。那么,
where on the l.h.s. the gradient is the standard stochastic policy gradient and on the r.h.s. the gradient is the deterministic policy gradient.
其中左边的梯度是标准的随机策略梯度,右边的梯度是确定性策略梯度。
This is an important result because it shows that the familiar machinery of policy gradients, for example compatible function approximation (Sutton et al., 1999), natural gradients (Kakade, 2001), actor-critic (Bhatnagar et al., 2007), or episodic/batch methods (Peters et al., 2005), is also applicable to deterministic policy gradients.
这是一个重要结果,因为它表明策略梯度中那些熟悉的工具,例如兼容函数近似(Sutton et al., 1999)、自然梯度(Kakade, 2001)、actor-critic(Bhatnagar et al., 2007),或回合/批次方法(Peters et al., 2005),同样适用于确定性策略梯度。
- Deterministic Actor-Critic Algorithms
- 确定性 Actor-Critic 算法
We now use the deterministic policy gradient theorem to derive both on-policy and off-policy actor-critic algorithms. We begin with the simplest case – on-policy updates, using a simple Sarsa critic – so as to illustrate the ideas as clearly as possible. We then consider the off-policy case, this time using a simple Q-learning critic to illustrate the key ideas. These simple algorithms may have convergence issues in practice, due both to bias introduced by the function approximator, and also the instabilities caused by off-policy learning. We then turn to a more principled approach using compatible function approximation and gradient temporal-difference learning.
我们现在用确定性策略梯度定理,导出 on-policy 和 off-policy 两类 actor-critic 算法。我们从最简单的情形开始——用简单的 Sarsa critic 做 on-policy 更新——以便尽可能清楚地说明想法。然后考虑 off-policy 情形,这次用简单的 Q-learning critic 来说明关键想法。这些简单算法在实践中可能有收敛问题,原因既包括函数近似器引入的偏差,也包括 off-policy 学习带来的不稳定。随后我们转向一种更有原则的做法,使用兼容函数近似和梯度时间差分学习。
4.1. On-Policy Deterministic Actor-Critic
4.1. On-Policy 确定性 Actor-Critic
In general, behaving according to a deterministic policy will not ensure adequate exploration and may lead to suboptimal solutions. Nevertheless, our first algorithm is an on-policy actor-critic algorithm that learns and follows a deterministic policy. Its primary purpose is didactic; however, it may be useful for environments in which there is sufficient noise in the environment to ensure adequate exploration, even with a deterministic behaviour policy.
一般而言,按确定性策略行动并不能保证充分探索,并可能导致次优解。尽管如此,我们的第一个算法仍是一种学习并跟随确定性策略的 on-policy actor-critic 算法。它的主要目的是教学性的;不过,在环境噪声已足以保证充分探索的场合,即便行为策略是确定性的,它也可能有用。
Like the stochastic actor-critic, the deterministic actor-critic consists of two components. The critic estimates the action-value function while the actor ascends the gradient of the action-value function. Specifically, an actor adjusts the parameters of the deterministic policy by stochastic gradient ascent of Equation 9. As in the stochastic actor-critic, we substitute a differentiable action-value function in place of the true action-value function . A critic estimates the action-value function , using an appropriate policy evaluation algorithm. For example, in the following deterministic actor-critic algorithm, the critic uses Sarsa updates to estimate the action-value function (Sutton and Barto, 1998),
与随机 actor-critic 一样,确定性 actor-critic 也由两个部分组成。Critic 估计动作值函数,actor 则沿动作值函数的梯度上升。具体而言,actor 通过对式 (9) 做随机梯度上升,来调整确定性策略 的参数 。与随机 actor-critic 中一样,我们用可微的动作值函数 代替真实动作值函数 。Critic 用合适的策略评估算法估计 。例如,在下面的确定性 actor-critic 算法中,critic 用 Sarsa 更新来估计动作值函数(Sutton and Barto, 1998),
4.2. Off-Policy Deterministic Actor-Critic
4.2. Off-Policy 确定性 Actor-Critic
We now consider off-policy methods that learn a deterministic target policy from trajectories generated by an arbitrary stochastic behaviour policy . As before, we modify the performance objective to be the value function of the target policy, averaged over the state distribution of the behaviour policy,
我们现在考虑 off-policy 方法:从任意随机行为策略 生成的轨迹中,学习确定性目标策略 。和前面一样,我们把性能目标改写成目标策略的值函数,并在行为策略的状态分布上取平均,
This equation gives the off-policy deterministic policy gradient. Analogous to the stochastic case (see Equation 4), we have dropped a term that depends on ; justification similar to Degris et al. (2012b) can be made in support of this approximation.
该式给出了 off-policy 确定性策略梯度。与随机情形类似(见式 (4)),我们丢掉了一个依赖于 的项;可以给出与 Degris 等人(2012b)类似的理由来支持这一近似。
We now develop an actor-critic algorithm that updates the policy in the direction of the off-policy deterministic policy gradient. We again substitute a differentiable action-value function in place of the true action-value function in Equation 15. A critic estimates the action-value function , off-policy from trajectories generated by , using an appropriate policy evaluation algorithm. In the following off-policy deterministic actor-critic (OPDAC) algorithm, the critic uses Q-learning updates to estimate the action-value function.
我们现在发展一种沿 off-policy 确定性策略梯度方向更新策略的 actor-critic 算法。我们再次在式 (15) 中用可微的动作值函数 代替真实动作值函数 。Critic 用合适的策略评估算法,从行为策略 生成的轨迹中 off-policy 地估计 。在下面的 off-policy 确定性 actor-critic(OPDAC)算法中,critic 用 Q-learning 更新来估计动作值函数。
We note that stochastic off-policy actor-critic algorithms typically use importance sampling for both actor and critic (Degris et al., 2012b). However, because the deterministic policy gradient removes the integral over actions, we can avoid importance sampling in the actor; and by using Q-learning, we can avoid importance sampling in the critic.
我们注意到,随机 off-policy actor-critic 算法通常对 actor 和 critic 都使用重要性采样(Degris et al., 2012b)。不过,由于确定性策略梯度去掉了对动作的积分,我们可以在 actor 中避免重要性采样;而通过使用 Q-learning,也可以在 critic 中避免重要性采样。
4.3. Compatible Function Approximation
4.3. 兼容函数近似
In general, substituting an approximate into the deterministic policy gradient will not necessarily follow the true gradient (nor indeed will it necessarily be an ascent direction at all). Similar to the stochastic case, we now find a class of compatible function approximators such that the true gradient is preserved. In other words, we find a critic such that the gradient can be replaced by , without affecting the deterministic policy gradient. The following theorem applies to both on-policy, , and off-policy, ,
一般而言,把近似的 代入确定性策略梯度,并不一定沿着真实梯度前进(甚至不一定是一个上升方向)。与随机情形类似,我们现在寻找一类兼容的函数近似器 ,使得真实梯度得以保持。换句话说,我们寻找一个 critic ,使得可以用 替换 ,而不影响确定性策略梯度。下面的定理同时适用于 on-policy,,和 off-policy,,
Theorem 3. A function approximator is compatible with a deterministic policy , , if
and
- minimises the mean-squared error,
where
定理 3. 函数近似器 与确定性策略 兼容,即 ,如果
并且
- 最小化均方误差
其中
Proof. If minimises the MSE then the gradient of w.r.t. must be zero. We then use the fact that, by condition 1, ,
证明. 如果 最小化 MSE,则 对 的梯度必须为零。然后我们利用条件 1 给出的事实 ,
For any deterministic policy , there always exists a compatible function approximator of the form , where may be any differentiable baseline function that is independent of the action ; for example a linear combination of state features and parameters , for parameters . A natural interpretation is that estimates the value of state , while the first term estimates the advantage of taking action over action in state . The advantage function can be viewed as a linear function approximator, with state-action features and parameters . Note that if there are action dimensions and policy parameters, then is an Jacobian matrix, so the feature vector is , and the parameter vector is also . A function approximator of this form satisfies condition 1 of Theorem 3.
对任意确定性策略 ,总存在形如 的兼容函数近似器,其中 可以是任意与动作 无关的可微基线函数;例如状态特征 与参数 的线性组合 。一种自然的解释是: 估计状态 的价值,而第一项估计在状态 中采取动作 相对于动作 的优势 。优势函数可以看作线性函数近似器 ,其状态-动作特征为 ,参数为 。注意,如果有 个动作维和 个策略参数,则 是一个 的雅可比矩阵,因而特征向量是 ,参数向量 也是 。这种形式的函数近似器满足定理 3 的条件 1。
We note that a linear function approximator is not very useful for predicting action-values globally, since the action-value diverges to for large actions. However, it can still be highly effective as a local critic. In particular, it represents the local advantage of deviating from the current policy, , where represents a small deviation from the deterministic policy. As a result, a linear function approximator is sufficient to select the direction in which the actor should adjust its policy parameters.
我们注意到,线性函数近似器并不很适合在全局上预测动作值,因为动作值会随大动作发散到 。不过,它作为局部 critic 仍然可以非常有效。具体而言,它表示偏离当前策略的局部优势 ,其中 表示对确定性策略的一个小偏离。因此,线性函数近似器足以选定 actor 应当调整其策略参数的方向。
To satisfy condition 2 we need to find the parameters that minimise the mean-squared error between the gradient of and the true gradient. This can be viewed as a linear regression problem with “features” and “targets” . In other words, features of the policy are used to predict the true gradient at state . However, acquiring unbiased samples of the true gradient is difficult. In practice, we use a linear function approximator to satisfy condition 1, but we learn by a standard policy evaluation method (for example Sarsa or Q-learning, for the on-policy or off-policy deterministic actor-critic algorithms respectively) that does not exactly satisfy condition 2. We note that a reasonable solution to the policy evaluation problem will find and will therefore approximately (for smooth function approximators) satisfy .
为满足条件 2,我们需要找到使 的梯度与真实梯度之间均方误差最小的参数 。这可以看成一个线性回归问题,其“特征”是 ,“目标”是 。换句话说,策略的特征被用来预测状态 处的真实梯度 。然而,获取真实梯度的无偏样本并不容易。在实践中,我们用线性函数近似器 来满足条件 1,但用标准的策略评估方法(例如分别用于 on-policy 或 off-policy 确定性 actor-critic 的 Sarsa 或 Q-learning)来学习 ,而这并不精确满足条件 2。我们注意到,对策略评估问题的一个合理解会得到 ,因而(对光滑函数近似器而言)会近似满足 。
To summarise, a compatible off-policy deterministic actor-critic (COPDAC) algorithm consists of two components. The critic is a linear function approximator that estimates the action-value from features . This may be learnt off-policy from samples of a behaviour policy , for example using Q-learning or gradient Q-learning. The actor then updates its parameters in the direction of the critic's action-value gradient. The following COPDAC-Q algorithm uses a simple Q-learning critic.
概括起来,兼容 off-policy 确定性 actor-critic(COPDAC)算法由两个部分组成。Critic 是一个线性函数近似器,用特征 估计动作值。它可以由行为策略 的样本 off-policy 地学习,例如使用 Q-learning 或梯度 Q-learning。Actor 然后沿 critic 的动作值梯度方向更新其参数。下面的 COPDAC-Q 算法使用简单的 Q-learning critic。
It is well-known that off-policy Q-learning may diverge when using linear function approximation. A more recent family of methods, based on gradient temporal-difference learning, are true gradient descent algorithm and are therefore sure to converge (Sutton et al., 2009). The basic idea of these methods is to minimise the mean-squared projected Bellman error (MSPBE) by stochastic gradient descent; full details are beyond the scope of this paper. Similar to the OffPAC algorithm (Degris et al., 2012b), we use gradient temporal-difference learning in the critic. Specifically, we use gradient Q-learning in the critic (Maei et al., 2010), and note that under suitable conditions on the step-sizes, , , , to ensure that the critic is updated on a faster time-scale than the actor, the critic will converge to the parameters minimising the MSPBE (Sutton et al., 2009; Degris et al., 2012b). The following COPDAC-GQ algorithm combines COPDAC with a gradient Q-learning critic,
众所周知,在使用线性函数近似时,off-policy Q-learning 可能发散。更新近的一类基于梯度时间差分学习的方法是真正的梯度下降算法,因而一定收敛(Sutton et al., 2009)。这些方法的基本想法是用随机梯度下降最小化均方投影贝尔曼误差(MSPBE);全部细节超出本文范围。与 OffPAC 算法(Degris et al., 2012b)类似,我们在 critic 中使用梯度时间差分学习。具体而言,我们在 critic 中使用梯度 Q-learning(Maei et al., 2010),并指出:在对步长 、、 的适当条件下,若保证 critic 在比 actor 更快的时间尺度上更新,则 critic 会收敛到最小化 MSPBE 的参数(Sutton et al., 2009; Degris et al., 2012b)。下面的 COPDAC-GQ 算法把 COPDAC 与梯度 Q-learning critic 结合起来,
Like stochastic actor-critic algorithms, the computational complexity of all these updates is per time-step.
与随机 actor-critic 算法一样,这些更新的计算复杂度都是每个时间步 。
Finally, we show that the natural policy gradient (Kakade, 2001; Peters et al., 2005) can be extended to deterministic policies. The steepest ascent direction of our performance objective with respect to any metric is given by (Toussaint, 2012). The natural gradient is the steepest ascent direction with respect to the Fisher information metric ; this metric is invariant to reparameterisations of the policy (Bagnell and Schneider, 2003). For deterministic policies, we use the metric which can be viewed as the limiting case of the Fisher information metric as policy variance is reduced to zero. By combining the deterministic policy gradient theorem with compatible function approximation we see that and so the steepest ascent direction is simply . This algorithm can be implemented by simplifying Equations 20 or 24 to .
最后,我们表明自然策略梯度(Kakade, 2001; Peters et al., 2005)可以推广到确定性策略。我们的性能目标相对于任意度量 的最速上升方向由 给出(Toussaint, 2012)。自然梯度是相对于 Fisher 信息度量 的最速上升方向;该度量对策略的重新参数化保持不变(Bagnell and Schneider, 2003)。对确定性策略,我们使用度量 ,它可以看作 Fisher 信息度量在策略方差减到零时的极限情形。把确定性策略梯度定理与兼容函数近似结合起来可以看到,,因此最速上升方向就是 。该算法可以通过把式 (20) 或式 (24) 简化为 来实现。
- Experiments
- 实验
5.1. Continuous Bandit
5.1. 连续赌博机
Our first experiment focuses on a direct comparison between the stochastic policy gradient and the deterministic policy gradient. The problem is a continuous bandit problem with a high-dimensional quadratic cost function, . The matrix is positive definite with eigenvalues chosen from , and . We consider action dimensions of . Although this problem could be solved analytically, given full knowledge of the quadratic, we are interested here in the relative performance of model-free stochastic and deterministic policy gradient algorithms.
我们的第一个实验集中比较随机策略梯度与确定性策略梯度。问题是一个带高维二次代价函数的连续赌博机,。矩阵 正定,特征值从 中选取,并且 。我们考虑动作维数 。尽管在完全知道这个二次型时,该问题可以解析求解,但我们这里关心的是无模型的随机与确定性策略梯度算法的相对表现。
For the stochastic actor-critic in the bandit task (SAC-B) we use an isotropic Gaussian policy, , and adapt both the mean and the variance of the policy. The deterministic actor-critic algorithm is based on COPDAC, using a target policy, and a fixed-width Gaussian behaviour policy, . The critic is simply estimated by linear regression from the compatible features to the costs: for SAC-B the compatible features are ; for COPDAC-B they are ; a bias feature is also included in both cases. For this experiment the critic is recomputed from each successive batch of steps; the actor is updated once per batch. To evaluate performance we measure the average cost per step incurred by the mean (i.e. exploration is not penalised for the on-policy algorithm). We performed a parameter sweep over all step-size parameters and variance parameters (initial for SAC; for COPDAC). Figure 1 shows the performance of the best performing parameters for each algorithm, averaged over 5 runs. The results illustrate a significant performance advantage to the deterministic update, which grows larger with increasing dimensionality.
对赌博机任务中的随机 actor-critic(SAC-B),我们使用各向同性高斯策略 ,并同时适应策略的均值和方差。确定性 actor-critic 算法基于 COPDAC,使用目标策略 以及固定宽度的高斯行为策略 。Critic 只需通过从兼容特征到代价的线性回归来估计:对 SAC-B,兼容特征是 ;对 COPDAC-B,它们是 ;两种情形都还包括一个偏置特征。在这一实验中,critic 从每连续一批 步中重新计算;actor 每批更新一次。为评估表现,我们测量由均值产生的每步平均代价(即对 on-policy 算法不惩罚探索)。我们对所有步长参数和方差参数做了参数扫描(SAC 的初始 ;COPDAC 的 )。图 1 展示了每种算法在最佳参数下的表现,对 5 次运行取平均。结果表明确定性更新有显著的性能优势,并且这一优势随维数增加而增大。
We also ran an experiment in which the stochastic actor-critic used the same fixed variance as the deterministic actor-critic, so that only the mean was adapted. This did not improve the performance of the stochastic actor-critic: COPDAC-B still outperforms SAC-B by a very wide margin that grows larger with increasing dimension.
我们还做了一项实验,让随机 actor-critic 使用与确定性 actor-critic 相同的固定方差 ,从而只适应均值。这并没有改善随机 actor-critic 的表现:COPDAC-B 仍然以很大的差距优于 SAC-B,并且这一差距随维数增加而增大。
图 1. 在连续赌博机任务上比较随机 actor-critic(SAC-B)与确定性 actor-critic(COPDAC-B)。
5.2. Continuous Reinforcement Learning
5.2. 连续强化学习
In our second experiment we consider continuous-action variants of standard reinforcement learning benchmarks: mountain car, pendulum and 2D puddle world. Our goal is to see whether stochastic or deterministic actor-critic is more efficient under Gaussian exploration. The stochastic actor-critic (SAC) algorithm was the actor-critic algorithm in Degris et al. (2012a); this algorithm performed best out of several incremental actor-critic methods in a comparison on mountain car. It uses a Gaussian policy based on a linear combination of features, , which adapts both the mean and the variance of the policy; the critic uses a linear value function approximator with the same features, updated by temporal-difference learning. The deterministic algorithm is based on COPDAC-Q, using a linear target policy, and a fixed-width Gaussian behaviour policy, . The critic again uses a linear value function , as a baseline for the compatible action-value function. In both cases the features are generated by tile-coding the state-space. We also compare to an off-policy stochastic actor-critic algorithm (OffPAC), using the same behaviour policy as just described, but learning a stochastic policy as in SAC. This algorithm also used the same critic algorithm and the update algorithm described in Degris et al. (2012b) with and .
在第二个实验中,我们考虑标准强化学习基准的连续动作变体:mountain car、pendulum 和二维 puddle world。目标是看在高斯探索下,随机还是确定性 actor-critic 更高效。随机 actor-critic(SAC)算法即 Degris 等人(2012a)中的 actor-critic 算法;在 mountain car 上的比较中,该算法在若干增量 actor-critic 方法里表现最好。它使用基于特征线性组合的高斯策略 ,同时适应策略的均值和方差;critic 使用具有相同特征的线性值函数近似器 ,并用时间差分学习更新。确定性算法基于 COPDAC-Q,使用线性目标策略 以及固定宽度的高斯行为策略 。Critic 同样使用线性值函数 ,作为兼容动作值函数的基线。两种情形下,特征 都通过对状态空间做 tile-coding 生成。我们还比较了一种 off-policy 随机 actor-critic 算法(OffPAC),它使用刚才描述的同一行为策略 ,但像 SAC 那样学习随机策略 。该算法也使用同样的 critic 算法,以及 Degris 等人(2012b)中描述的更新算法,并取 、。
For all algorithms, episodes were truncated after a maximum of 5000 steps. The discount was for mountain car and pendulum and for puddle world. Actions outside the legal range were capped. We performed a parameter sweep over step-size parameters; variance was initialised to the legal range. Figure 2 shows the performance of the best performing parameters for each algorithm, averaged over 30 runs. COPDAC-Q slightly outperformed both SAC and OffPAC in all three domains.
对所有算法,回合在最多 5000 步后截断。折扣在 mountain car 和 pendulum 上为 ,在 puddle world 上为 。超出合法范围的动作被截断。我们对步长参数做了参数扫描;方差初始化为合法范围的 。图 2 展示了每种算法在最佳参数下的表现,对 30 次运行取平均。COPDAC-Q 在所有三个领域上都略优于 SAC 和 OffPAC。
图 2. 在连续动作强化学习上比较随机 on-policy actor-critic(SAC)、随机 off-policy actor-critic(OffPAC)以及确定性 off-policy actor-critic(COPDAC)。每一点都是均值策略的平均测试表现。
5.3. Octopus Arm
5.3. 章鱼臂
Finally, we tested our algorithms on an octopus arm (Engel et al., 2005) task. The aim is to learn to control a simulated octopus arm to hit a target. The arm consists of 6 segments and is attached to a rotating base. There are 50 continuous state variables (, position/velocity of the nodes along the upper/lower side of the arm; angular position/velocity of the base) and 20 action variables that control three muscles (dorsal, transversal, central) in each segment as well as the clockwise and counter-clockwise rotation of the base. The goal is to strike the target with any part of the arm. The reward function is proportional to the change in distance between the arm and the target. An episode ends when the target is hit (with an additional reward of ) or after 300 steps. Previous work (Engel et al., 2005) simplified the high-dimensional action space using 6 “macro-actions” corresponding to particular patterns of muscle activations; or applied stochastic policy gradients to a lower dimensional octopus arm with 4 segments (Heess et al., 2012). Here, we apply deterministic policy gradients directly to a high-dimensional octopus arm with 6 segments.
最后,我们在章鱼臂(Engel et al., 2005)任务上测试了我们的算法。目标是学会控制一个模拟章鱼臂去击中目标。该臂由 6 段组成,并连接在一个可旋转的基座上。有 50 个连续状态变量(沿臂上侧/下侧各节点的 、 位置/速度;基座的角位置/角速度)以及 20 个动作变量,分别控制每一段的三块肌肉(背侧、横向、中央)以及基座的顺时针和逆时针旋转。目标是用臂的任何部分击中靶子。奖励函数与臂和目标之间距离的变化成比例。当击中目标时(额外奖励 )或在 300 步之后,回合结束。先前工作(Engel et al., 2005)用对应于特定肌肉激活模式的 6 种“宏动作”来简化高维动作空间;或者把随机策略梯度用在只有 4 段的较低维章鱼臂上(Heess et al., 2012)。在这里,我们把确定性策略梯度直接用在有 6 段的高维章鱼臂上。
We applied the COPDAC-Q algorithm, using a sigmoidal multi-layer perceptron (8 hidden units and sigmoidal output units) to represent the policy . The advantage function was represented by compatible function approximation (see Section 4.3), while the state value function was represented by a second multi-layer perceptron (40 hidden units and linear output units). The results of 10 training runs are shown in Figure 3; the octopus arm converged to a good solution in all cases. A video of an 8 segment arm, trained by COPDAC-Q, is also available.
我们应用了 COPDAC-Q 算法,用一个 sigmoid 多层感知机(8 个隐单元和 sigmoid 输出单元)来表示策略 。优势函数 由兼容函数近似表示(见第 4.3 节),而状态值函数 由第二个多层感知机表示(40 个隐单元和线性输出单元)。 10 次训练运行的结果见图 3;章鱼臂在所有情形下都收敛到了良好的解。用 COPDAC-Q 训练的 8 段臂的视频也可获得。
Recall that the compatibility criteria apply to any differentiable baseline, including non-linear state-value functions.
请注意,兼容性准则适用于任意可微基线,包括非线性状态值函数。
图 3. COPDAC 在具有 20 维动作和 50 维状态的 6 段章鱼臂上的十次运行;每一点表示每回合回报(上)以及臂到达目标所用的时间步数(下)。
- Discussion and Related Work
- 讨论与相关工作
Using a stochastic policy gradient algorithm, the policy becomes more deterministic as the algorithm homes in on a good strategy. Unfortunately this makes the stochastic policy gradient harder to estimate, because the policy gradient changes more rapidly near the mean. Indeed, the variance of the stochastic policy gradient for a Gaussian policy is proportional to (Zhao et al., 2012), which grows to infinity as the policy becomes deterministic. This problem is compounded in high dimensions, as illustrated by the continuous bandit task. The stochastic actor-critic estimates the stochastic policy gradient in Equation 2. The inner integral, , is computed by sampling a high dimensional action space. In contrast, the deterministic policy gradient can be computed immediately in closed form.
使用随机策略梯度算法时,随着算法逼近一个好策略,策略会变得更加确定性。不幸的是,这会使随机策略梯度更难估计,因为策略梯度 在均值附近变化得更快。事实上,高斯策略 的随机策略梯度方差与 成正比(Zhao et al., 2012),并在策略变得确定时趋向无穷。这一问题在高维中会加剧,连续赌博机任务就说明了这一点。随机 actor-critic 估计的是式 (2) 中的随机策略梯度。内层积分 要通过在高维动作空间中采样来计算。相比之下,确定性策略梯度可以立即以闭式计算出来。
One may view our deterministic actor-critic as analogous, in a policy gradient context, to Q-learning (Watkins and Dayan, 1992). Q-learning learns a deterministic greedy policy, off-policy, while executing a noisy version of the greedy policy. Similarly, in our experiments COPDAC-Q was used to learn a deterministic policy, off-policy, while executing a noisy version of that policy. Note that we compared on-policy and off-policy algorithms in our experiments, which may at first sight appear odd. However, it is analogous to asking whether Q-learning or Sarsa is more efficient, by measuring the greedy policy learnt by each algorithm (Sutton and Barto, 1998).
可以把我们的确定性 actor-critic 看成策略梯度语境下与 Q-learning(Watkins and Dayan, 1992)类似的东西。Q-learning 在执行贪心策略的带噪版本的同时,off-policy 地学习一个确定性贪心策略。类似地,在我们的实验中,COPDAC-Q 被用来在执行该策略的带噪版本的同时,off-policy 地学习一个确定性策略。需要指出,我们在实验中比较了 on-policy 和 off-policy 算法,这乍看起来可能有些奇怪。不过,这类似于通过测量每种算法所学到的贪心策略,来问 Q-learning 还是 Sarsa 更高效(Sutton and Barto, 1998)。
Our actor-critic algorithms are based on model-free, incremental, stochastic gradient updates; these methods are suitable when the model is unknown, data is plentiful and computation is the bottleneck. It is straightforward in principle to extend these methods to batch/episodic updates, for example by using LSTDQ (Lagoudakis and Parr, 2003) in place of the incremental Q-learning critic. There has also been a substantial literature on model-based policy gradient methods, largely focusing on deterministic and fully-known transition dynamics (Werbos, 1990). These methods are strongly related to deterministic policy gradients when the transition dynamics are also deterministic.
我们的 actor-critic 算法基于无模型、增量式的随机梯度更新;这些方法适用于模型未知、数据充足而计算才是瓶颈的情形。原则上,把这些方法扩展到批次/回合更新是直接的,例如用 LSTDQ(Lagoudakis and Parr, 2003)代替增量 Q-learning critic。也已有大量关于基于模型的策略梯度方法的文献,大多关注确定性且完全已知的转移动力学(Werbos, 1990)。当转移动力学也是确定性的时候,这些方法与确定性策略梯度密切相关。
We are not the first to notice that the action-value gradient provides a useful signal for reinforcement learning. The NFQCA algorithm (Hafner and Riedmiller, 2011) uses two neural networks to represent the actor and critic respectively. The actor adjusts the policy, represented by the first neural network, in the direction of the action-value gradient, using an update similar to Equation 7. The critic updates the action-value function, represented by the second neural network, using neural fitted-Q learning (a batch Q-learning update for approximate value iteration). However, its critic network is incompatible with the actor network; it is unclear how the local optima learnt by the critic (assuming it converges) will interact with actor updates.
我们并不是最先注意到动作值梯度能为强化学习提供有用信号的人。NFQCA 算法(Hafner and Riedmiller, 2011)分别用两个神经网络来表示 actor 和 critic。Actor 用类似于式 (7) 的更新,沿动作值梯度的方向调整由第一个神经网络表示的策略。Critic 则用 neural fitted-Q 学习(一种用于近似值迭代的批次 Q-learning 更新)来更新由第二个神经网络表示的动作值函数。不过,它的 critic 网络与 actor 网络并不兼容;即便假定 critic 收敛,它所学到的局部最优将如何与 actor 更新相互作用,也并不清楚。
- Conclusion
- 结论
We have presented a framework for deterministic policy gradient algorithms. These gradients can be estimated more efficiently than their stochastic counterparts, avoiding a problematic integral over the action space. In practice, the deterministic actor-critic significantly outperformed its stochastic counterpart by several orders of magnitude in a bandit with 50 continuous action dimensions, and solved a challenging reinforcement learning problem with 20 continuous action dimensions and 50 state dimensions.
我们给出了确定性策略梯度算法的一个框架。这些梯度可以比其随机对照更高效地估计,并避开了对动作空间的那个棘手积分。在实践中,确定性 actor-critic 在一个有 50 个连续动作维的赌博机上,以几个数量级显著优于其随机对照,并且解决了一个有 20 个连续动作维和 50 个状态维的有挑战性的强化学习问题。
Acknowledgements
致谢
This work was supported by the European Community Seventh Framework Programme (FP7/2007-2013) under grant agreement 270327 (CompLACS), the Gatsby Charitable Foundation, the Royal Society, the ANR MACSi project, INRIA Bordeaux Sud-Ouest, Mesocentre de Calcul Intensif Aquitain, and the French National Grid Infrastructure via France Grille.
本工作得到了欧洲共同体第七框架计划(FP7/2007-2013)资助协议 270327(CompLACS)、Gatsby Charitable Foundation、Royal Society、ANR MACSi 项目、INRIA Bordeaux Sud-Ouest、Mesocentre de Calcul Intensif Aquitain,以及经由 France Grille 的法国国家网格基础设施的支持。
References
参考文献
Bagnell, J. A. D. and Schneider, J. (2003). Covariant policy search. In Proceeding of the International Joint Conference on Artifical Intelligence.
Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M., and Lee, M. (2007). Incremental natural actor-critic algorithms. In Neural Information Processing Systems 21.
Degris, T., Pilarski, P. M., and Sutton, R. S. (2012a). Model-free reinforcement learning with continuous action in practice. In American Control Conference.
Degris, T., White, M., and Sutton, R. S. (2012b). Linear off-policy actor-critic. In 29th International Conference on Machine Learning.
Engel, Y., Szabó, P., and Volkinshtein, D. (2005). Learning to control an octopus arm with gaussian process temporal difference methods. In Neural Information Processing Systems 18.
Hafner, R. and Riedmiller, M. (2011). Reinforcement learning in feedback control. Machine Learning, 84(1-2):137–169.
Heess, N., Silver, D., and Teh, Y. (2012). Actor-critic reinforcement learning with energy-based policies. JMLR Workshop and Conference Proceedings: EWRL 2012, 24:43–58.
Kakade, S. (2001). A natural policy gradient. In Neural Information Processing Systems 14, pages 1531–1538.
Lagoudakis, M. G. and Parr, R. (2003). Least-squares policy iteration. Journal of Machine Learning Research, 4:1107–1149.
Maei, H. R., Szepesvári, C., Bhatnagar, S., and Sutton, R. S. (2010). Toward off-policy learning control with function approximation. In 27th International Conference on Machine Learning, pages 719–726.
Peters, J. (2010). Policy gradient methods. Scholarpedia, 5(11):3698.
Peters, J., Vijayakumar, S., and Schaal, S. (2005). Natural actor-critic. In 16th European Conference on Machine Learning, pages 280–291.
Sutton, R. and Barto, A. (1998). Reinforcement Learning: an Introduction. MIT Press.
Sutton, R. S., Maei, H. R., Precup, D., Bhatnagar, S., Silver, D., Szepesvári, C., and Wiewiora, E. (2009). Fast gradient-descent methods for temporal-difference learning with linear function approximation. In 26th International Conference on Machine Learning, page 125.
Sutton, R. S., McAllester, D. A., Singh, S. P., and Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. In Neural Information Processing Systems 12, pages 1057–1063.
Sutton, R. S., Singh, S. P., and McAllester, D. A. (2000). Comparing policy-gradient algorithms. http://webdocs.cs.ualberta.ca/~sutton/papers/SSM-unpublished.pdf.
Toussaint, M. (2012). Some notes on gradient descent. http://ipvs.informatik.uni-stuttgart.de/mlr/marc/notes/gradientDescent.pdf.
Watkins, C. and Dayan, P. (1992). Q-learning. Machine Learning, 8(3):279–292.
Werbos, P. J. (1990). A menu of designs for reinforcement learning over time. In Neural networks for control, pages 67–95. Bradford.
Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256.
Zhao, T., Hachiya, H., Niu, G., and Sugiyama, M. (2012). Analysis and improvement of policy gradient estimation. Neural Networks, 26:118–129.


