Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Soft Actor-Critic：带随机执行器的离线策略最大熵深度强化学习

Tuomas Haarnoja Affiliation:  Berkeley Artificial Intelligence Research, University of California, Berkeley, USA Correspondence to: haarnoja@berkeley.edu    Aurick Zhou Affiliation:  Berkeley Artificial Intelligence Research, University of California, Berkeley, USA    Pieter Abbeel Affiliation:  Berkeley Artificial Intelligence Research, University of California, Berkeley, USA    Sergey Levine Affiliation:  Berkeley Artificial Intelligence Research, University of California, Berkeley, USA

Tuomas Haarnoja（加州大学伯克利分校 BAIR，haarnoja@berkeley.edu）　Aurick Zhou（加州大学伯克利分校 BAIR）　Pieter Abbeel（加州大学伯克利分校 BAIR）　Sergey Levine（加州大学伯克利分校 BAIR）

Abstract

摘要

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.

无模型深度强化学习（RL）算法已在一系列具有挑战性的决策与控制任务上得到验证。然而，这些方法通常面临两大难题：一是很高的样本复杂度，二是脆弱性——即便超参数经过仔细调节，性能仍可能因随机种子而大幅波动。这两大挑战严重限制了深度强化学习算法在复杂任务与实际问题上的应用。本文提出一种基于最大熵强化学习框架的离线策略执行器–评价器（actor-critic）算法。在该框架中，执行器的目标是最大化期望回报的同时最大化熵，也即在完成任务与尽可能随机地行动之间取得成功。此前基于这一框架的深度强化学习方法要么受限于离散动作空间，要么实现复杂。我们提出一种可扩展到很大连续状态与动作空间的高效近似，并给出其收敛性的理论保证。经验结果表明，我们的方法显著优于以往的离散与连续动作空间深度强化学习算法，并在一系列基准任务上取得了显著更高的样本效率与最终性能。

1 Introduction

1 引言

Keywords:

Model-free deep reinforcement learning (RL) algorithms have been applied in a range of challenging domains, from games (Mnih et al., 2013; Silver et al., 2016) to robotic control (Schulman et al., 2015). The combination of RL and high-capacity function approximators such as neural networks holds the promise of automating a wide range of decision making and control tasks, but widespread adoption of these methods in real-world domains has been hampered by two major challenges. First, model-free deep RL methods are notoriously expensive in terms of their sample complexity. Even relatively simple tasks can require millions of steps of data collection, and complex behaviors with high-dimensional observations might need substantially more. Second, these methods are often brittle with respect to their hyperparameters: learning rates, exploration constants, and other settings must be set carefully for different problem settings to achieve good results. Both of these challenges severely limit the applicability of model-free deep RL to real-world tasks.

关键词：

无模型深度强化学习（RL）算法已应用于一系列具有挑战性的领域，从游戏 [Mnih et al., 2013; Silver et al., 2016] 到机器人控制 [Schulman et al., 2015]。强化学习与神经网络等高容量函数逼近器的结合，有望自动化多种决策与控制任务，但这些方法要在现实世界中被广泛采用，仍受样本效率低与超参数敏感等问题制约。

One cause for the poor sample efficiency of deep RL methods is on-policy learning: some of the most commonly used deep RL algorithms, such as TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017b) or A3C (Mnih et al., 2016), require new samples to be collected for each gradient step. This quickly becomes extravagantly expensive, as the number of gradient steps and samples per step needed to learn an effective policy increases with task complexity. Off-policy algorithms aim to reuse past experience. This is not directly feasible with conventional policy gradient formulations, but is relatively straightforward for Q-learning based methods (Mnih et al., 2015). Unfortunately, the combination of off-policy learning and high-dimensional, nonlinear function approximation with neural networks presents a major challenge for stability and convergence (Bhatnagar et al., 2009). This challenge is further exacerbated in continuous state and action spaces, where a separate actor network is often used to perform the maximization in Q-learning. A commonly used algorithm in such settings, deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015), provides for sample-efficient learning but is notoriously challenging to use due to its extreme brittleness and hyperparameter sensitivity (Duan et al., 2016; Henderson et al., 2017).

深度强化学习方法样本效率较差的原因之一是在策略学习：一些最常用的深度强化学习算法，例如 TRPO（Schulman 等人，2015）、PPO（Schulman 等人，2017b）或 A3C（Mnih 等人，2016），需要为每个梯度步骤收集新样本。 这很快就会变得极其昂贵，因为学习有效策略所需的梯度步骤和每步骤的样本数量随着任务复杂性的增加而增加。 离策略算法旨在重用过去的经验。 这对于传统的策略梯度公式来说并不直接可行，但对于基于 Q 学习的方法来说相对简单（Mnih 等人，2015）。 不幸的是，离策略学习和高维非线性函数逼近与神经网络的结合对稳定性和收敛性提出了重大挑战（Bhatnagar 等人，2009）。 在连续状态和动作空间中，这一挑战进一步加剧，其中通常使用单独的参与者网络来执行 Q 学习中的最大化。 在这种情况下，常用的算法是深度确定性策略梯度 (DDPG)（Lillicrap 等人，2015），它提供了样本高效学习，但由于其极端脆弱性和超参数敏感性，使用起来非常具有挑战性（Duan 等人，2016 年；Henderson 等人，2017 年）。

We explore how to design an efficient and stable model-free deep RL algorithm for continuous state and action spaces. To that end, we draw on the maximum entropy framework, which augments the standard maximum reward reinforcement learning objective with an entropy maximization term (Ziebart et al., 2008; Toussaint, 2009; Rawlik et al., 2012; Fox et al., 2016; Haarnoja et al., 2017). Maximum entropy reinforcement learning alters the RL objective, though the original objective can be recovered using a temperature parameter (Haarnoja et al., 2017). More importantly, the maximum entropy formulation provides a substantial improvement in exploration and robustness: as discussed by Ziebart (2010), maximum entropy policies are robust in the face of model and estimation errors, and as demonstrated by (Haarnoja et al., 2017), they improve exploration by acquiring diverse behaviors. Prior work has proposed model-free deep RL algorithms that perform on-policy learning with entropy maximization (O’Donoghue et al., 2016), as well as off-policy methods based on soft Q-learning and its variants (Schulman et al., 2017a; Nachum et al., 2017a; Haarnoja et al., 2017). However, the on-policy variants suffer from poor sample complexity for the reasons discussed above, while the off-policy variants require complex approximate inference procedures in continuous action spaces.

我们探索如何为连续状态与动作空间设计高效且稳定的无模型深度 RL 算法。为此，我们借助最大熵框架：在标准最大期望回报目标之外，再要求策略尽可能随机，即最大化熵。先前工作已表明该目标能带来更好的探索与鲁棒性；我们进一步证明，它还能稳定学习，并在高维任务上取得更强的最终表现。

In this paper, we demonstrate that we can devise an off-policy maximum entropy actor-critic algorithm, which we call soft actor-critic (SAC), which provides for both sample-efficient learning and stability. This algorithm extends readily to very complex, high-dimensional tasks, such as the Humanoid benchmark (Duan et al., 2016) with 21 action dimensions, where off-policy methods such as DDPG typically struggle to obtain good results (Gu et al., 2016). SAC also avoids the complexity and potential instability associated with approximate inference in prior off-policy maximum entropy algorithms based on soft Q-learning (Haarnoja et al., 2017). We present a convergence proof for policy iteration in the maximum entropy framework, and then introduce a new algorithm based on an approximation to this procedure that can be practically implemented with deep neural networks, which we call soft actor-critic. We present empirical results that show that soft actor-critic attains a substantial improvement in both performance and sample efficiency over both off-policy and on-policy prior methods. We also compare to twin delayed deep deterministic (TD3) policy gradient algorithm (Fujimoto et al., 2018), which is a concurrent work that proposes a deterministic algorithm that substantially improves on DDPG.

本文表明，我们可以设计一种离线策略最大熵 actor-critic 算法，称为 soft actor-critic（SAC），同时具备样本高效与稳定性。该算法可轻松扩展到很复杂的高维任务，例如具有 21 维动作的 Humanoid 基准 [Duan et al., 2016]；在这类任务上，DDPG 等离线策略方法通常表现不佳。

2 Related Work

2 相关工作

Our soft actor-critic algorithm incorporates three key ingredients: an actor-critic architecture with separate policy and value function networks, an off-policy formulation that enables reuse of previously collected data for efficiency, and entropy maximization to enable stability and exploration. We review prior works that draw on some of these ideas in this section. Actor-critic algorithms are typically derived starting from policy iteration, which alternates between policy evaluation—computing the value function for a policy—and policy improvement—using the value function to obtain a better policy (Barto et al., 1983; Sutton & Barto, 1998). In large-scale reinforcement learning problems, it is typically impractical to run either of these steps to convergence, and instead the value function and policy are optimized jointly. In this case, the policy is referred to as the actor, and the value function as the critic. Many actor-critic algorithms build on the standard, on-policy policy gradient formulation to update the actor (Peters & Schaal, 2008), and many of them also consider the entropy of the policy, but instead of maximizing the entropy, they use it as an regularizer (Schulman et al., 2017b; Schulman et al., 2015; Mnih et al., 2016; Gruslys et al., 2017). On-policy training tends to improve stability but results in poor sample complexity.

我们的软演员批评家算法包含三个关键要素：具有独立策略和价值函数网络的演员批评家架构、能够重用以前收集的数据以提高效率的离策略公式，以及支持稳定性和探索的熵最大化。 我们在本节中回顾了借鉴其中一些想法的先前作品。 Actor-critic 算法通常从策略迭代开始派生，策略迭代在策略评估（计算策略的价值函数）和策略改进（使用价值函数获得更好的策略）之间交替（Barto 等人，1983 年；Sutton 和 Barto，1998 年）。 在大规模强化学习问题中，运行这些步骤中的任何一个来收敛通常是不切实际的，而是联合优化价值函数和策略。 在这种情况下，策略被称为参与者，价值函数被称为批评者。 许多 actor-critic 算法都建立在标准的策略策略梯度公式的基础上来更新 actor（Peters & Schaal，2008），其中许多算法还考虑了策略的熵，但它们不是最大化熵，而是将其用作正则化器（Schulman et al., 2017b；Schulman et al., 2015；Mnih et al., 2016； Gruslys 等人，2017）。 按策略训练往往会提高稳定性，但会导致样本复杂性较差。

There have been efforts to increase the sample efficiency while retaining robustness by incorporating off-policy samples and by using higher order variance reduction techniques (O’Donoghue et al., 2016; Gu et al., 2016). However, fully off-policy algorithms still attain better efficiency. A particularly popular off-policy actor-critic method, DDPG (Lillicrap et al., 2015), which is a deep variant of the deterministic policy gradient (Silver et al., 2014) algorithm, uses a Q-function estimator to enable off-policy learning, and a deterministic actor that maximizes this Q-function. As such, this method can be viewed both as a deterministic actor-critic algorithm and an approximate Q-learning algorithm. Unfortunately, the interplay between the deterministic actor network and the Q-function typically makes DDPG extremely difficult to stabilize and brittle to hyperparameter settings (Duan et al., 2016; Henderson et al., 2017). As a consequence, it is difficult to extend DDPG to complex, high-dimensional tasks, and on-policy policy gradient methods still tend to produce the best results in such settings (Gu et al., 2016). Our method instead combines off-policy actor-critic training with a stochastic actor, and further aims to maximize the entropy of this actor with an entropy maximization objective. We find that this actually results in a considerably more stable and scalable algorithm that, in practice, exceeds both the efficiency and final performance of DDPG. A similar method can be derived as a zero-step special case of stochastic value gradients (SVG(0)) (Heess et al., 2015). However, SVG(0) differs from our method in that it optimizes the standard maximum expected return objective, and it does not make use of a separate value network, which we found to make training more stable.

人们一直在努力通过合并离策略样本和使用高阶方差减少技术来提高样本效率，同时保持稳健性（O’Donoghue 等人，2016 年；Gu 等人，2016 年）。 然而，完全离策略算法仍然可以获得更好的效率。 DDPG（Lillicrap et al., 2015）是一种特别流行的离策略 actor-critic 方法，它是确定性策略梯度（Silver et al., 2014）算法的深度变体，它使用 Q 函数估计器来实现离策略学习，并使用一个确定性 actor 来最大化该 Q 函数。 因此，该方法既可以被视为确定性行动者批评算法，也可以被视为近似 Q 学习算法。 不幸的是，确定性参与者网络和 Q 函数之间的相互作用通常使得 DDPG 极难稳定，并且对于超参数设置来说很脆弱（Duan 等人，2016 年；Henderson 等人，2017 年）。 因此，很难将 DDPG 扩展到复杂的高维任务，并且在策略策略梯度方法仍然倾向于在此类设置中产生最佳结果（Gu et al., 2016）。 相反，我们的方法将非策略演员批评家训练与“随机”演员相结合，并进一步旨在以熵最大化为目标来最大化该演员的熵。 我们发现，这实际上产生了一种更加稳定和可扩展的算法，在实践中，它超过了 DDPG 的效率和最终性能。 类似的方法可以导出为随机值梯度的零步特例 (SVG(0))（Heess 等人，2015）。 然而，SVG(0) 与我们的方法不同，因为它优化了标准最大预期回报目标，并且它没有使用单独的价值网络，我们发现这可以使训练更加稳定。

Maximum entropy reinforcement learning optimizes policies to maximize both the expected return and the expected entropy of the policy. This framework has been used in many contexts, from inverse reinforcement learning (Ziebart et al., 2008) to optimal control (Todorov, 2008; Toussaint, 2009; Rawlik et al., 2012). In guided policy search (Levine & Koltun, 2013; Levine et al., 2016), the maximum entropy distribution is used to guide policy learning towards high-reward regions. More recently, several papers have noted the connection between Q-learning and policy gradient methods in the framework of maximum entropy learning (O’Donoghue et al., 2016; Haarnoja et al., 2017; Nachum et al., 2017a; Schulman et al., 2017a). While most of the prior model-free works assume a discrete action space, Nachum et al. (2017b) approximate the maximum entropy distribution with a Gaussian and Haarnoja et al. (2017) with a sampling network trained to draw samples from the optimal policy. Although the soft Q-learning algorithm proposed by Haarnoja et al. (2017) has a value function and actor network, it is not a true actor-critic algorithm: the Q-function is estimating the optimal Q-function, and the actor does not directly affect the Q-function except through the data distribution. Hence, Haarnoja et al. (2017) motivates the actor network as an approximate sampler, rather than the actor in an actor-critic algorithm. Crucially, the convergence of this method hinges on how well this sampler approximates the true posterior. In contrast, we prove that our method converges to the optimal policy from a given policy class, regardless of the policy parameterization. Furthermore, these prior maximum entropy methods generally do not exceed the performance of state-of-the-art off-policy algorithms, such as DDPG, when learning from scratch, though they may have other benefits, such as improved exploration and ease of fine-tuning. In our experiments, we demonstrate that our soft actor-critic algorithm does in fact exceed the performance of prior state-of-the-art off-policy deep RL methods by a wide margin.

最大熵强化学习优化策略，以最大化策略的预期回报和预期熵。 该框架已在许多情况下使用，从逆强化学习（Ziebart 等人，2008）到最优控制（Todorov，2008；Toussaint，2009；Rawlik 等人，2012）。 在引导策略搜索中（Levine & Koltun，2013；Levine et al.，2016），最大熵分布用于引导策略学习朝向高回报区域。 最近，有几篇论文指出了最大熵学习框架中 Q 学习和策略梯度方法之间的联系（O’Donoghue 等人，2016；Haarnoja 等人，2017；Nachum 等人，2017a；Schulman 等人，2017a）。 虽然大多数先前的无模型作品都假设一个离散的动作空间，但 Nachum 等人。 (2017b) 使用高斯和 Haarnoja 等人近似最大熵分布。 （2017）使用经过训练的采样网络从最优策略中抽取样本。 虽然 Haarnoja 等人提出的软 Q 学习算法。 （2017）有一个价值函数和参与者网络，它不是真正的参与者批评算法：Q函数正在估计最优Q函数，并且参与者除了通过数据分布之外不会直接影响Q函数。 因此，哈诺贾等人。 （2017）激励参与者网络作为近似采样器，而不是参与者-评论家算法中的参与者。 至关重要的是，该方法的收敛性取决于该采样器对真实后验的近似程度。 相反，我们证明，无论策略参数化如何，我们的方法都会收敛到给定策略类的最优策略。 此外，这些现有的最大熵方法在从头开始学习时通常不会超过最先进的离策略算法（例如 DDPG）的性能，尽管它们可能具有其他好处，例如改进的探索和易于微调。 在我们的实验中，我们证明了我们的软 Actor-Critic 算法实际上大大超过了先前最先进的离策略深度 RL 方法的性能。

3 Preliminaries

3 预备知识

We first introduce notation and summarize the standard and maximum entropy reinforcement learning frameworks.

我们首先介绍符号并总结标准和最大熵强化学习框架。

3.1 Notation

3.1 符号

We address policy learning in continuous action spaces. We consider an infinite-horizon Markov decision process (MDP), defined by the tuple $(\mathcal{S},\mathcal{A},p,r)$, where the state space $\mathcal{S}$ and the action space $\mathcal{A}$ are continuous, and the unknown state transition probability $p:\ \mathcal{S}\times\mathcal{S}\times\mathcal{A}\rightarrow[0,\,\infty)$ represents the probability density of the next state ${\mathbf{s}_{t+1}}\in\mathcal{S}$ given the current state ${\mathbf{s}_{t}}\in\mathcal{S}$ and action ${\mathbf{a}_{t}}\in\mathcal{A}$. The environment emits a bounded reward $r:\mathcal{S}\times\mathcal{A}\rightarrow[r_{\mathrm{min}},r_{\mathrm{max}}]$ on each transition. We will use $\rho_{\pi}({\mathbf{s}_{t}})$ and $\rho_{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}})$ to denote the state and state-action marginals of the trajectory distribution induced by a policy $\pi({\mathbf{a}_{t}}|{\mathbf{s}_{t}})$.

我们在持续行动空间中解决政策学习问题。 我们考虑无限范围马尔可夫决策过程（MDP），由元组$(\mathcal{S},\mathcal{A},p,r)$定义，其中状态空间$\mathcal{S}$和动作空间$\mathcal{A}$是连续的，未知状态转移概率$p:\ \mathcal{S}\times\mathcal{S}\times\mathcal{A}\rightarrow[0,\,\infty)$表示给定当前状态${\mathbf{s}_{t}}\in\mathcal{S}$和动作${\mathbf{a}_{t}}\in\mathcal{A}$的下一个状态${\mathbf{s}_{t+1}}\in\mathcal{S}$的概率密度。 环境在每次转换时都会发出有限的奖励 $r:\mathcal{S}\times\mathcal{A}\rightarrow[r_{\mathrm{min}},r_{\mathrm{max}}]$。 我们将使用$\rho_{\pi}({\mathbf{s}_{t}})$和$\rho_{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}})$来表示由策略$\pi({\mathbf{a}_{t}}|{\mathbf{s}_{t}})$引起的轨迹分布的状态和状态动作边际。

3.2 Maximum Entropy Reinforcement Learning

3.2 最大熵强化学习

Standard RL maximizes the expected sum of rewards $\sum_{t}\operatorname{\mathbb{E}}_{({\mathbf{s}_{t}},{\mathbf{a}_{t}})\sim\rho_{\pi}}\left[r({\mathbf{s}_{t}},{\mathbf{a}_{t}})\right]$. We will consider a more general maximum entropy objective (see e.g. Ziebart (2010)), which favors stochastic policies by augmenting the objective with the expected entropy of the policy over $\rho_{\pi}({\mathbf{s}_{t}})$:

标准 RL 最大化期望回报之和 $\sum_{t}\operatorname{\mathbb{E}}_{({\mathbf{s}_{t}},{\mathbf{a}_{t}})\sim\rho_{\pi}}\left[r({\mathbf{s}_{t}},{\mathbf{a}_{t}})\right]$。我们将考虑更一般的最大熵目标，它在每一步还最大化策略的熵。

$$\displaystyle J(\pi)=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{({\mathbf{s}_{t}},{\mathbf{a}_{t}})\sim\rho_{\pi}}\left[r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\alpha\mathcal{H}(\pi({\,\cdot\,}|{\mathbf{s}_{t}}))\right]. \tag{1}$$

The temperature parameter $\alpha$ determines the relative importance of the entropy term against the reward, and thus controls the stochasticity of the optimal policy. The maximum entropy objective differs from the standard maximum expected reward objective used in conventional reinforcement learning, though the conventional objective can be recovered in the limit as $\alpha\rightarrow 0$. For the rest of this paper, we will omit writing the temperature explicitly, as it can always be subsumed into the reward by scaling it by $\alpha^{-1}$.

温度参数 $\alpha$ 决定熵项相对回报的重要性，从而控制最优策略的随机程度。最大熵目标对应于最优策略的能量基形式，其中 Q 函数充当负能量。

This objective has a number of conceptual and practical advantages. First, the policy is incentivized to explore more widely, while giving up on clearly unpromising avenues. Second, the policy can capture multiple modes of near-optimal behavior. In problem settings where multiple actions seem equally attractive, the policy will commit equal probability mass to those actions. Lastly, prior work has observed improved exploration with this objective (Haarnoja et al., 2017; Schulman et al., 2017a), and in our experiments, we observe that it considerably improves learning speed over state-of-art methods that optimize the conventional RL objective function. We can extend the objective to infinite horizon problems by introducing a discount factor $\gamma$ to ensure that the sum of expected rewards and entropies is finite. Writing down the maximum entropy objective for the infinite horizon discounted case is more involved (Thomas, 2014) and is deferred to Appendix A.

该目标有若干概念与实践上的优点。第一，策略被激励更广泛地探索，同时放弃明显无望的方向。第二，策略可以捕捉多种近优行为。第三，在机器人等领域，更随机的策略往往对模型误差与扰动更鲁棒。

Prior methods have proposed directly solving for the optimal Q-function, from which the optimal policy can be recovered (Ziebart et al., 2008; Fox et al., 2016; Haarnoja et al., 2017). We will discuss how we can devise a soft actor-critic algorithm through a policy iteration formulation, where we instead evaluate the Q-function of the current policy and update the policy through an off-policy gradient update. Though such algorithms have previously been proposed for conventional reinforcement learning, our method is, to our knowledge, the first off-policy actor-critic method in the maximum entropy reinforcement learning framework.

先前的方法提出直接求解最优 Q 函数，从中可以恢复最优策略（Ziebart 等人，2008；Fox 等人，2016；Haarnoja 等人，2017）。 我们将讨论如何通过策略迭代公式设计软演员批评算法，其中我们评估当前策略的 Q 函数并通过off-policy梯度更新来更新策略。 尽管此类算法之前已被提出用于传统的强化学习，但据我们所知，我们的方法是最大熵强化学习框架中的第一个离策略行动者批评家方法。

4 From Soft Policy Iteration to Soft Actor-Critic

4 从软策略迭代到 Soft Actor-Critic

Our off-policy soft actor-critic algorithm can be derived starting from a maximum entropy variant of the policy iteration method. We will first present this derivation, verify that the corresponding algorithm converges to the optimal policy from its density class, and then present a practical deep reinforcement learning algorithm based on this theory.

我们的离策略软演员批评算法可以从策略迭代方法的最大熵变体开始导出。 我们将首先提出这个推导，验证相应的算法从其密度类别收敛到最优策略，然后基于该理论提出一种实用的深度强化学习算法。

4.1 Derivation of Soft Policy Iteration

4.1 软策略迭代的推导

We will begin by deriving soft policy iteration, a general algorithm for learning optimal maximum entropy policies that alternates between policy evaluation and policy improvement in the maximum entropy framework. Our derivation is based on a tabular setting, to enable theoretical analysis and convergence guarantees, and we extend this method into the general continuous setting in the next section. We will show that soft policy iteration converges to the optimal policy within a set of policies which might correspond, for instance, to a set of parameterized densities.

我们将首先推导软策略迭代，这是一种用于学习最优最大熵策略的通用算法，它在最大熵框架中交替进行策略评估和策略改进。 我们的推导基于表格设置，以实现理论分析和收敛保证，并且我们在下一节中将此方法扩展到一般连续设置。 我们将展示软策略迭代收敛到一组策略中的最优策略，例如，这组策略可能对应于一组参数化密度。

In the policy evaluation step of soft policy iteration, we wish to compute the value of a policy $\pi$ according to the maximum entropy objective in Equation 1. For a fixed policy, the soft Q-value can be computed iteratively, starting from any function $Q:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ and repeatedly applying a modified Bellman backup operator $\mathcal{T}^{\pi}$ given by

在软策略迭代的策略评估步骤中，我们希望根据方程 1 中的最大熵目标来计算策略$\pi$的值。对于固定策略，可以迭代计算软 Q 值，从任何函数$Q:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$开始，并重复应用修改后的贝尔曼备份算子$\mathcal{T}^{\pi}$，该算子由下式给出

$$\displaystyle\mathcal{T}^{\pi}Q({\mathbf{s}_{t}},{\mathbf{a}_{t}})\triangleq r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p}\left[V({\mathbf{s}_{t+1}})\right], \tag{2}$$

where

在哪里

$$\displaystyle V({\mathbf{s}_{t}})=\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim\pi}\left[Q({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\log\pi({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right] \tag{3}$$

is the soft state value function. We can obtain the soft value function for any policy $\pi$ by repeatedly applying $\mathcal{T}^{\pi}$ as formalized below.

是软状态值函数。 我们可以通过重复应用$\mathcal{T}^{\pi}$来获得任何策略$\pi$的软价值函数，如下形式化所示。

Lemma 1(Soft Policy Evaluation).

Consider the soft Bellman backup operator $\mathcal{T}^{\pi}$ in Equation 2 and a mapping $Q^{0}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ with $|\mathcal{A}|<\infty$, and define $Q^{k+1}=\mathcal{T}^{\pi}Q^{k}$. Then the sequence $Q^{k}$ will converge to the soft Q-value of $\pi$ as $k\rightarrow\infty$.

引理 1（软政策评估）.

考虑等式 2 中的软 Bellman 备份运算符$\mathcal{T}^{\pi}$以及$Q^{0}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$与$|\mathcal{A}|<\infty$的映射，并定义$Q^{k+1}=\mathcal{T}^{\pi}Q^{k}$。 然后序列$Q^{k}$将收敛到$\pi$的软 Q 值，即$k\rightarrow\infty$。

Proof.

See Appendix B.1. ∎

＃＃＃ 证明。

请参阅附录 B.1。 ∎

In the policy improvement step, we update the policy towards the exponential of the new Q-function. This particular choice of update can be guaranteed to result in an improved policy in terms of its soft value. Since in practice we prefer policies that are tractable, we will additionally restrict the policy to some set of policies $\Pi$, which can correspond, for example, to a parameterized family of distributions such as Gaussians. To account for the constraint that $\pi\in\Pi$, we project the improved policy into the desired set of policies. While in principle we could choose any projection, it will turn out to be convenient to use the information projection defined in terms of the Kullback-Leibler divergence. In the other words, in the policy improvement step, for each state, we update the policy according to

在策略改进步骤中，我们将策略更新为新 Q 函数的指数。 这种特定的更新选择可以保证在其软价值方面产生改进的策略。 由于在实践中我们更喜欢易于处理的策略，因此我们还会将策略限制为某些策略$\Pi$，例如，它可以对应于参数化分布族（例如高斯分布）。 为了考虑$\pi\in\Pi$的约束，我们将改进的策略投影到所需的策略集中。 虽然原则上我们可以选择任何投影，但事实证明，使用根据 Kullback-Leibler 散度定义的信息投影会很方便。 换句话说，在策略改进步骤中，对于每个状态，我们根据

$$\displaystyle\pi_{\mathrm{new}}=\arg\underset{\pi^{\prime}\in\Pi}{\min}\mathrm{D_{KL}}\left(\pi^{\prime}({\,\cdot\,}|{\mathbf{s}_{t}})\;\middle\|\;\frac{\exp\left(Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\,\cdot\,})\right)}{Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}})}\right). \tag{4}$$

The partition function $Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}})$ normalizes the distribution, and while it is intractable in general, it does not contribute to the gradient with respect to the new policy and can thus be ignored, as noted in the next section. For this projection, we can show that the new, projected policy has a higher value than the old policy with respect to the objective in Equation 1. We formalize this result in Lemma 2.

分区函数$Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}})$对分布进行归一化，虽然它通常很棘手，但它对新策略的梯度没有贡献，因此可以被忽略，如下一节所述。 对于此预测，我们可以证明，就方程 1 中的目标而言，新的预测政策比旧政策具有更高的价值。我们在引理 2 中形式化了这一结果。

Lemma 2(Soft Policy Improvement).

Let $\pi_{\mathrm{old}}\in\Pi$ and let $\pi_{\mathrm{new}}$ be the optimizer of the minimization problem defined in Equation 4. Then $Q^{\pi_{\mathrm{new}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\geq Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})$ for all $({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A}$ with $|\mathcal{A}|<\infty$.

引理 2（软政策改进）.

令$\pi_{\mathrm{old}}\in\Pi$和$\pi_{\mathrm{new}}$为方程 4 中定义的最小化问题的优化器。然后$Q^{\pi_{\mathrm{new}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\geq Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})$对于所有$({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A}$和$|\mathcal{A}|<\infty$。

Proof.

See Appendix B.2. ∎

＃＃＃ 证明。

请参阅附录 B.2。 ∎

The full soft policy iteration algorithm alternates between the soft policy evaluation and the soft policy improvement steps, and it will provably converge to the optimal maximum entropy policy among the policies in $\Pi$ (Theorem 1). Although this algorithm will provably find the optimal solution, we can perform it in its exact form only in the tabular case. Therefore, we will next approximate the algorithm for continuous domains, where we need to rely on a function approximator to represent the Q-values, and running the two steps until convergence would be computationally too expensive. The approximation gives rise to a new practical algorithm, called soft actor-critic.

完整的软策略迭代算法在软策略评估和软策略改进步骤之间交替，并且它将证明收敛到$\Pi$中策略中的最优最大熵策略（定理 1）。 尽管该算法可以证明找到最佳解决方案，但我们只能在表格情况下以其精确形式执行它。 因此，我们接下来将近似连续域的算法，其中我们需要依靠函数逼近器来表示 Q 值，并运行这两个步骤直到收敛在计算上过于昂贵。 这种近似产生了一种新的实用算法，称为软演员批评家。

Theorem 1(Soft Policy Iteration).

Repeated application of soft policy evaluation and soft policy improvement from any $\pi\in\Pi$ converges to a policy $\pi^{}$such that$Q^{\pi^{}}({\mathbf{s}{t}},{\mathbf{a}{t}})\geq Q^{\pi}({\mathbf{s}{t}},{\mathbf{a}{t}})$for all$\pi\in\Pi$and$({\mathbf{s}{t}},{\mathbf{a}{t}})\in\mathcal{S}\times\mathcal{A}$, assuming$|\mathcal{A}|<\infty$.

定理1（软策略迭代）.

重复应用软策略评估和软策略改进，从任何$\pi\in\Pi$收敛到策略$\pi^{}$，使得$Q^{\pi^{}}({\mathbf{s}{t}},{\mathbf{a}{t}})\geq Q^{\pi}({\mathbf{s}{t}},{\mathbf{a}{t}})$对于所有$\pi\in\Pi$和$({\mathbf{s}{t}},{\mathbf{a}{t}})\in\mathcal{S}\times\mathcal{A}$，假设$|\mathcal{A}|<\infty$。

Proof.

See Appendix B.3. ∎

＃＃＃ 证明。

请参阅附录 B.3。 ∎

4.2 Soft Actor-Critic

As discussed above, large continuous domains require us to derive a practical approximation to soft policy iteration. To that end, we will use function approximators for both the Q-function and the policy, and instead of running evaluation and improvement to convergence, alternate between optimizing both networks with stochastic gradient descent. We will consider a parameterized state value function $V_{\psi}({\mathbf{s}_{t}})$, soft Q-function $Q_{\theta}({\mathbf{s}_{t}},{\mathbf{a}_{t}})$, and a tractable policy $\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})$. The parameters of these networks are ${\psi},\ \theta$, and ${\phi}$. For example, the value functions can be modeled as expressive neural networks, and the policy as a Gaussian with mean and covariance given by neural networks. We will next derive update rules for these parameter vectors.

如上所述，在大型连续域中需要对软策略迭代做实用近似。我们为 Q 函数与策略都使用函数逼近器，并且不把评估与改进跑到收敛，而是用随机梯度下降交替优化两个网络。我们将考虑参数化的状态价值函数、软 Q 函数与策略。

The state value function approximates the soft value. There is no need in principle to include a separate function approximator for the state value, since it is related to the Q-function and policy according to Equation 3. This quantity can be estimated from a single action sample from the current policy without introducing a bias, but in practice, including a separate function approximator for the soft value can stabilize training and is convenient to train simultaneously with the other networks. The soft value function is trained to minimize the squared residual error

状态值函数近似于软值。 原则上不需要为状态值包含单独的函数逼近器，因为根据方程 3，它与 Q 函数和策略相关。该量可以根据当前策略的单个动作样本进行估计，而不会引入偏差，但在实践中，为软值包含单独的函数逼近器可以稳定训练，并且方便与其他网络同时训练。 训练软值函数以最小化平方残差

$$J_{V}({\psi})=\operatorname{\mathbb{E}}_{{\mathbf{s}_{t}}\sim\mathcal{D}}\left[\frac{1}{2}\left(V_{\psi}({\mathbf{s}_{t}})-\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim\pi_{\phi}}\left[Q_{\theta}({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\log\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right]\right)^{2}\right]\, \tag{5}$$

where $\mathcal{D}$ is the distribution of previously sampled states and actions, or a replay buffer. The gradient of Equation 5 can be estimated with an unbiased estimator

其中$\mathcal{D}$是先前采样的状态和动作的分布，或重播缓冲区。 方程 5 的梯度可以使用无偏估计器进行估计

$$\hat{\nabla}_{\psi}J_{V}({\psi})=\nabla_{\psi}V_{\psi}({\mathbf{s}_{t}})\left(V_{\psi}({\mathbf{s}_{t}})-Q_{\theta}({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\log\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right), \tag{6}$$

where the actions are sampled according to the current policy, instead of the replay buffer. The soft Q-function parameters can be trained to minimize the soft Bellman residual

其中操作是根据当前策略而不是重播缓冲区进行采样的。 可以训练软 Q 函数参数以最小化软贝尔曼残差

$$\displaystyle J_{Q}(\theta)=\operatorname{\mathbb{E}}_{({\mathbf{s}_{t}},{\mathbf{a}_{t}})\sim\mathcal{D}}\left[\frac{1}{2}\left(Q_{\theta}({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\hat{Q}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\right)^{2}\right], \tag{7}$$

with

和

$$\displaystyle\hat{Q}({\mathbf{s}_{t}},{\mathbf{a}_{t}})=r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p}\left[V_{\bar{\psi}}({\mathbf{s}_{t+1}})\right], \tag{8}$$

which again can be optimized with stochastic gradients

再次可以使用随机梯度进行优化

\displaystyle\resizebox{22609920}{}{$\hat{\nabla}_{\theta}J_{Q}(\theta)=\nabla_{\theta}Q_{\theta}({\mathbf{a}_{t}},{\mathbf{s}_{t}})\left(Q_{\theta}({\mathbf{s}_{t}},{\mathbf{a}_{t}})-r({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\gamma V_{\bar{\psi}}({\mathbf{s}_{t+1}})\right)$}. \tag{9}

The update makes use of a target value network $V_{\bar{\psi}}$, where $\bar{\psi}$ can be an exponentially moving average of the value network weights, which has been shown to stabilize training (Mnih et al., 2015). Alternatively, we can update the target weights to match the current value function weights periodically (see Appendix E). Finally, the policy parameters can be learned by directly minimizing the expected KL-divergence in Equation 4:

该更新利用了目标价值网络$V_{\bar{\psi}}$，其中$\bar{\psi}$可以是价值网络权重的指数移动平均值，这已被证明可以稳定训练（Mnih 等人，2015）。 或者，我们可以定期更新目标权重以匹配当前价值函数权重（参见附录 E）。 最后，可以通过直接最小化公式 4 中的预期 KL 散度来学习策略参数：

$$\displaystyle J_{\pi}({\phi})=\operatorname{\mathbb{E}}_{{\mathbf{s}_{t}}\sim\mathcal{D}}\left[\mathrm{D_{KL}}\left(\pi_{\phi}({\,\cdot\,}|{\mathbf{s}_{t}})\;\middle\|\;\frac{\exp\left(Q_{\theta}({\mathbf{s}_{t}},{\,\cdot\,})\right)}{Z_{\theta}({\mathbf{s}_{t}})}\right)\right]. \tag{10}$$

There are several options for minimizing $J_{\pi}$. A typical solution for policy gradient methods is to use the likelihood ratio gradient estimator (Williams, 1992), which does not require backpropagating the gradient through the policy and the target density networks. However, in our case, the target density is the Q-function, which is represented by a neural network an can be differentiated, and it is thus convenient to apply the reparameterization trick instead, resulting in a lower variance estimator. To that end, we reparameterize the policy using a neural network transformation

有多种选项可用于最小化$J_{\pi}$。 策略梯度方法的典型解决方案是使用似然比梯度估计器（Williams，1992），它不需要通过策略和目标密度网络反向传播梯度。 然而，在我们的例子中，目标密度是 Q 函数，它由神经网络表示，可以微分，因此可以方便地应用重新参数化技巧，从而得到较低的方差估计量。 为此，我们使用神经网络变换重新参数化策略

$$\displaystyle{\mathbf{a}_{t}}=f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}}), \tag{11}$$

where $\epsilon_{t}$ is an input noise vector, sampled from some fixed distribution, such as a spherical Gaussian. We can now rewrite the objective in Equation 10 as

其中$\epsilon_{t}$是输入噪声向量，从某个固定分布（例如球面高斯）采样。 我们现在可以将方程 10 中的目标重写为

$$J_{\pi}({\phi})=\operatorname{\mathbb{E}}_{{\mathbf{s}_{t}}\sim\mathcal{D},\epsilon_{t}\sim\mathcal{N}}\left[\log\pi_{\phi}(f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}})|{\mathbf{s}_{t}})-Q_{\theta}({\mathbf{s}_{t}},f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}}))\right], \tag{12}$$

where $\pi_{\phi}$ is defined implicitly in terms of $f_{\phi}$, and we have noted that the partition function is independent of ${\phi}$ and can thus be omitted. We can approximate the gradient of Equation 12 with

其中$\pi_{\phi}$是根据$f_{\phi}$隐式定义的，并且我们注意到配分函数独立于${\phi}$因此可以省略。 我们可以近似方程 12 的梯度：

$$\displaystyle\hat{\nabla}_{\phi}J_{\pi}({\phi})=\nabla_{\phi}\log\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}}) \\ \displaystyle\ \ \ \ \ \ \ +(\nabla_{\mathbf{a}_{t}}\log\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})-\nabla_{\mathbf{a}_{t}}Q({\mathbf{s}_{t}},{\mathbf{a}_{t}}))\nabla_{\phi}f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}}), \tag{13}$$

where ${\mathbf{a}_{t}}$ is evaluated at $f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}})$. This unbiased gradient estimator extends the DDPG style policy gradients (Lillicrap et al., 2015) to any tractable stochastic policy.

其中${\mathbf{a}_{t}}$在$f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}})$处评估。 这种无偏梯度估计器将 DDPG 风格的策略梯度（Lillicrap 等人，2015）扩展到任何易于处理的随机策略。

Initialize parameter vectors ${\psi}$, ${\bar{\psi}}$, $\theta$, ${\phi}$.
for each iteration do
for each environment step do
${\mathbf{a}_{t}}\sim\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})$
${\mathbf{s}_{t+1}}\sim p({\mathbf{s}_{t+1}}|{\mathbf{s}_{t}},{\mathbf{a}_{t}})$
$\mathcal{D}\leftarrow\mathcal{D}\cup\left\{({\mathbf{s}_{t}},{\mathbf{a}_{t}},r({\mathbf{s}_{t}},{\mathbf{a}_{t}}),{\mathbf{s}_{t+1}})\right\}$
end for
for each gradient step do
${\psi}\leftarrow{\psi}-\lambda_{V}\hat{\nabla}_{\psi}J_{V}({\psi})$
$\theta_{i}\leftarrow\theta_{i}-\lambda_{Q}\hat{\nabla}_{\theta_{i}}J_{Q}(\theta_{i})$ for $i\in\{1,2\}$
${\phi}\leftarrow{\phi}-\lambda_{\pi}\hat{\nabla}_{\phi}J_{\pi}({\phi})$
${\bar{\psi}}\leftarrow\tau{\psi}+(1-\tau){\bar{\psi}}$
end for
end for

Algorithm 1 Soft Actor-Critic

初始化参数向量${\psi}$、${\bar{\psi}}$、$\theta$、${\phi}$。
对于每次迭代执行
对于每个环境步骤执行
数学4XX
XX数学5XX
XX数学6XX
结束 为了
对于每个梯度步骤执行
XX数学7XX
$\theta_{i}\leftarrow\theta_{i}-\lambda_{Q}\hat{\nabla}_{\theta_{i}}J_{Q}(\theta_{i})$对应 $i\in\{1,2\}$
XX数学10XX
XX数学11XX
结束 为了
结束 为了

算法 1 软演员评论家

(a) Hopper-v1

(a) 料斗-v1

Our algorithm also makes use of two Q-functions to mitigate positive bias in the policy improvement step that is known to degrade performance of value based methods (Hasselt, 2010; Fujimoto et al., 2018). In particular, we parameterize two Q-functions, with parameters $\theta_{i}$, and train them independently to optimize $J_{Q}(\theta_{i})$. We then use the minimum of the Q-functions for the value gradient in Equation 6 and policy gradient in Equation 13, as proposed by Fujimoto et al. (2018). Although our algorithm can learn challenging tasks, including a 21-dimensional Humanoid, using just a single Q-function, we found two Q-functions significantly speed up training, especially on harder tasks. The complete algorithm is described in Algorithm 1. The method alternates between collecting experience from the environment with the current policy and updating the function approximators using the stochastic gradients from batches sampled from a replay buffer. In practice, we take a single environment step followed by one or several gradient steps (see Appendix D for all hyperparameter). Using off-policy data from a replay buffer is feasible because both value estimators and the policy can be trained entirely on off-policy data. The algorithm is agnostic to the parameterization of the policy, as long as it can be evaluated for any arbitrary state-action tuple.

我们的算法还使用两个 Q 函数，以减轻策略改进步骤中已知会损害基于价值方法表现的正偏 [Hasselt, 2010; Fujimoto et al., 2018]。具体地，我们用参数 $\theta_i$ 参数化两个 Q 函数，并独立训练以优化 $J_Q(\theta_i)$，然后在价值和策略更新中取两者的较小值。

5 Experiments

5 实验

The goal of our experimental evaluation is to understand how the sample complexity and stability of our method compares with prior off-policy and on-policy deep reinforcement learning algorithms. We compare our method to prior techniques on a range of challenging continuous control tasks from the OpenAI gym benchmark suite (Brockman et al., 2016) and also on the rllab implementation of the Humanoid task (Duan et al., 2016). Although the easier tasks can be solved by a wide range of different algorithms, the more complex benchmarks, such as the 21-dimensional Humanoid (rllab), are exceptionally difficult to solve with off-policy algorithms (Duan et al., 2016). The stability of the algorithm also plays a large role in performance: easier tasks make it more practical to tune hyperparameters to achieve good results, while the already narrow basins of effective hyperparameters become prohibitively small for the more sensitive algorithms on the hardest benchmarks, leading to poor performance (Gu et al., 2016).

实验评估的目标是弄清本方法的样本复杂度与稳定性，相对先前离线策略与在线策略深度强化学习算法如何。我们在 OpenAI Gym 基准套件的一系列连续控制任务 [Brockman et al., 2016] 以及 rllab 实现的 Humanoid 任务上，与先前技术进行对比。

We compare our method to deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015), an algorithm that is regarded as one of the more efficient off-policy deep RL methods (Duan et al., 2016); proximal policy optimization (PPO) (Schulman et al., 2017b), a stable and effective on-policy policy gradient algorithm; and soft Q-learning (SQL) (Haarnoja et al., 2017), a recent off-policy algorithm for learning maximum entropy policies. Our SQL implementation also includes two Q-functions, which we found to improve its performance in most environments. We additionally compare to twin delayed deep deterministic policy gradient algorithm (TD3) (Fujimoto et al., 2018), using the author-provided implementation. This is an extension to DDPG, proposed concurrently to our method, that first applied the double Q-learning trick to continuous control along with other improvements. We have included trust region path consistency learning (Trust-PCL) (Nachum et al., 2017b) and two other variants of SAC in Appendix E. We turned off the exploration noise for evaluation for DDPG and PPO. For maximum entropy algorithms, which do not explicitly inject exploration noise, we either evaluated with the exploration noise (SQL) or use the mean action (SAC). The source code of our SAC implementation[^1] and videos[^2] are available online.

我们将我们的方法与深度确定性策略梯度（DDPG）（Lillicrap 等人，2015）进行比较，后者是一种被认为是更有效的离策略深度强化学习方法之一的算法（Duan 等人，2016）； 近端策略优化（PPO）（Schulman et al., 2017b），一种稳定有效的策略策略梯度算法； 以及软 Q 学习 (SQL)（Haarnoja 等人，2017），这是一种最近用于学习最大熵策略的离策略算法。 我们的 SQL 实现还包括两个 Q 函数，我们发现它们可以提高大多数环境中的性能。 我们还使用作者提供的实现与孪生延迟深度确定性策略梯度算法 (TD3)（Fujimoto 等人，2018）进行比较。 这是 DDPG 的扩展，与我们的方法同时提出，首先将双 Q 学习技巧应用于连续控制以及其他改进。 我们在附录 E 中纳入了信任区域路径一致性学习 (Trust-PCL)（Nachum 等人，2017b）和 SAC 的其他两个变体。我们关闭了用于评估 DDPG 和 PPO 的探索噪声。 对于不显式注入探索噪声的最大熵算法，我们要么使用探索噪声（SQL）进行评估，要么使用平均动作（SAC）。 我们的 SAC 实现 [^1] 的源代码和视频 [^2] 可在线获取。

5.1 Comparative Evaluation

5.1 对比评估

Figure 1 shows the total average return of evaluation rollouts during training for DDPG, PPO, and TD3. We train five different instances of each algorithm with different random seeds, with each performing one evaluation rollout every 1000 environment steps. The solid curves corresponds to the mean and the shaded region to the minimum and maximum returns over the five trials.

图 1 显示了 DDPG、PPO 和 TD3 训练期间评估推出的总平均回报。 我们使用不同的随机种子训练每个算法的五个不同实例，每个实例每 1000 个环境步骤执行一次评估部署。 实线对应于五次试验的平均值，阴影区域对应于最小和最大回报。

The results show that, overall, SAC performs comparably to the baseline methods on the easier tasks and outperforms them on the harder tasks with a large margin, both in terms of learning speed and the final performance. For example, DDPG fails to make any progress on Ant-v1, Humanoid-v1, and Humanoid (rllab), a result that is corroborated by prior work (Gu et al., 2016; Duan et al., 2016). SAC also learns considerably faster than PPO as a consequence of the large batch sizes PPO needs to learn stably on more high-dimensional and complex tasks. Another maximum entropy RL algorithm, SQL, can also learn all tasks, but it is slower than SAC and has worse asymptotic performance. The quantitative results attained by SAC in our experiments also compare very favorably to results reported by other methods in prior work (Duan et al., 2016; Gu et al., 2016; Henderson et al., 2017), indicating that both the sample efficiency and final performance of SAC on these benchmark tasks exceeds the state of the art. All hyperparameters used in this experiment for SAC are listed in Appendix D.

结果表明：总体而言，SAC 在较简单任务上与基线相当，在较难任务上以较大优势超过它们，无论是学习速度还是最终表现。例如，DDPG 在 Ant-v1、Humanoid-v1 与 Humanoid（rllab）上几乎没有进展，这与先前工作一致 [Gu et al., 2016; Duan et al., 2016]。SAC 还稳定地超过了 PPO 与 TD3。

5.2 Ablation Study

5.2 消融研究

The results in the previous section suggest that algorithms based on the maximum entropy principle can outperform conventional RL methods on challenging tasks such as the humanoid tasks. In this section, we further examine which particular components of SAC are important for good performance. We also examine how sensitive SAC is to some of the most important hyperparameters, namely reward scaling and target value update smoothing constant.

上一节的结果表明，基于最大熵原理的算法在处理具有挑战性的任务（例如人形任务）时可以优于传统的强化学习方法。 在本节中，我们将进一步研究 SAC 的哪些特定组件对于良好性能非常重要。 我们还研究了 SAC 对一些最重要的超参数（即奖励缩放和目标值更新平滑常数）的敏感度。

Figure 2: Comparison of SAC (blue) and a deterministic variant of SAC (red) in terms of the stability of individual random seeds on the Humanoid (rllab) benchmark. The comparison indicates that stochasticity can stabilize training as the variability between the seeds becomes much higher with a deterministic policy.

图 2：SAC（蓝色）和 SAC 的确定性变体（红色）在 Humanoid (rllab) 基准上单个随机种子的稳定性方面的比较。 比较表明，随着确定性策略下种子之间的变异性变得更高，随机性可以稳定训练。

Stochastic vs. deterministic policy.

Soft actor-critic learns stochastic policies via a maximum entropy objective. The entropy appears in both the policy and value function. In the policy, it prevents premature convergence of the policy variance (Equation 10). In the value function, it encourages exploration by increasing the value of regions of state space that lead to high-entropy behavior (Equation 5). To compare how the stochasticity of the policy and entropy maximization affects the performance, we compare to a deterministic variant of SAC that does not maximize the entropy and that closely resembles DDPG, with the exception of having two Q-functions, using hard target updates, not having a separate target actor, and using fixed rather than learned exploration noise. Figure 2 compares five individual runs with both variants, initialized with different random seeds. Soft actor-critic performs much more consistently, while the deterministic variant exhibits very high variability across seeds, indicating substantially worse stability. As evident from the figure, learning a stochastic policy with entropy maximization can drastically stabilize training. This becomes especially important with harder tasks, where tuning hyperparameters is challenging. In this comparison, we updated the target value network weights with hard updates, by periodically overwriting the target network parameters to match the current value network (see Appendix E for a comparison of average performance on all benchmark tasks).

随机政策与确定性政策。

软演员批评家通过最大熵目标学习随机策略。 熵出现在策略函数和价值函数中。 在策略中，它可以防止策略方差过早收敛（公式 10）。 在价值函数中，它通过增加导致高熵行为的状态空间区域的价值来鼓励探索（公式 5）。 为了比较策略的随机性和熵最大化如何影响性能，我们与 SAC 的确定性变体进行比较，该变体不会最大化熵，并且与 DDPG 非常相似，除了具有两个 Q 函数、使用硬目标更新、没有单独的目标参与者以及使用固定而不是学习的探索噪声。 图 2 比较了使用不同随机种子初始化的两种变体的五次单独运行。 软演员批评家的表现更加一致，而确定性变体在种子之间表现出非常高的变异性，表明稳定性要差得多。 从图中可以明显看出，学习熵最大化的随机策略可以极大地稳定训练。 这对于更困难的任务尤其重要，因为在这些任务中调整超参数具有挑战性。 在此比较中，我们通过定期覆盖目标网络参数以匹配当前价值网络，通过硬更新更新了目标价值网络权重（有关所有基准任务的平均性能的比较，请参阅附录 E）。

(a) Evaluation

(一)评价

Policy evaluation.

Since SAC converges to stochastic policies, it is often beneficial to make the final policy deterministic at the end for best performance. For evaluation, we approximate the maximum a posteriori action by choosing the mean of the policy distribution. Figure 3(a) compares training returns to evaluation returns obtained with this strategy indicating that deterministic evaluation can yield better performance. It should be noted that all of the training curves depict the sum of rewards, which is different from the objective optimized by SAC and other maximum entropy RL algorithms, including SQL and Trust-PCL, which maximize also the entropy of the policy.

政策评估。

由于 SAC 收敛于随机策略，因此最终使最终策略具有确定性通常有利于获得最佳性能。 为了进行评估，我们通过选择策略分布的平均值来近似最大后验动作。 图 3(a) 将训练回报与使用此策略获得的评估回报进行了比较，表明确定性评估可以产生更好的性能。 应该注意的是，所有训练曲线都描述了奖励的总和，这与 SAC 和其他最大熵 RL 算法（包括 SQL 和 Trust-PCL）优化的目标不同，它们也最大化了策略的熵。

Reward scale.

Soft actor-critic is particularly sensitive to the scaling of the reward signal, because it serves the role of the temperature of the energy-based optimal policy and thus controls its stochasticity. Larger reward magnitudes correspond to lower entries. Figure 3(b) shows how learning performance changes when the reward scale is varied: For small reward magnitudes, the policy becomes nearly uniform, and consequently fails to exploit the reward signal, resulting in substantial degradation of performance. For large reward magnitudes, the model learns quickly at first, but the policy then becomes nearly deterministic, leading to poor local minima due to lack of adequate exploration. With the right reward scaling, the model balances exploration and exploitation, leading to faster learning and better asymptotic performance. In practice, we found reward scale to be the only hyperparameter that requires tuning, and its natural interpretation as the inverse of the temperature in the maximum entropy framework provides good intuition for how to adjust this parameter.

奖励等级。

软演员批评家对奖励信号的缩放特别敏感，因为它起到基于能量的最优策略的温度作用，从而控制其随机性。 较大的奖励幅度对应于较低的条目。 图 3(b) 显示了奖励规模变化时学习绩效如何变化：对于较小的奖励幅度，策略变得几乎一致，因此无法利用奖励信号，导致绩效大幅下降。 对于大的奖励幅度，模型一开始学习得很快，但随后策略变得几乎是确定性的，由于缺乏足够的探索而导致局部最小值较差。 通过正确的奖励缩放，该模型可以平衡探索和利用，从而实现更快的学习和更好的渐近性能。 在实践中，我们发现奖励尺度是唯一需要调整的超参数，其自然解释为最大熵框架中温度的倒数，为如何调整该参数提供了良好的直觉。

Target network update.

It is common to use a separate target value network that slowly tracks the actual value function to improve stability. We use an exponentially moving average, with a smoothing constant $\tau$, to update the target value network weights as common in the prior work (Lillicrap et al., 2015; Mnih et al., 2015). A value of one corresponds to a hard update where the weights are copied directly at every iteration and zero to not updating the target at all. In Figure 3(c), we compare the performance of SAC when $\tau$ varies. Large $\tau$ can lead to instabilities while small $\tau$ can make training slower. However, we found the range of suitable values of $\tau$ to be relatively wide and we used the same value (0.005) across all of the tasks. In Figure 4 (Appendix E) we also compare to another variant of SAC, where instead of using exponentially moving average, we copy over the current network weights directly into the target network every 1000 gradient steps. We found this variant to benefit from taking more than one gradient step between the environment steps, which can improve performance but also increases the computational cost.

目标网络更新。

通常使用单独的目标价值网络缓慢跟踪实际价值函数以提高稳定性。 我们使用指数移动平均值和平滑常数$\tau$来更新目标值网络权重，这在之前的工作中很常见（Lillicrap 等人，2015 年；Mnih 等人，2015 年）。 值 1 对应于硬更新，其中权重在每次迭代时直接复制，值 0 表示根本不更新目标。 在图 3(c) 中，我们比较了$\tau$变化时 SAC 的性能。 大的$\tau$可能会导致不稳定，而小$\tau$可能会使训练变慢。 然而，我们发现$\tau$的合适值范围相对较宽，并且我们在所有任务中使用相同的值 (0.005)。 在图 4（附录 E）中，我们还与 SAC 的另一种变体进行了比较，其中我们不使用指数移动平均值，而是每 1000 个梯度步骤将当前网络权重直接复制到目标网络中。 我们发现这种变体受益于在环境步骤之间采取多个梯度步骤，这可以提高性能，但也会增加计算成本。

6 Conclusion

6 结论

We present soft actor-critic (SAC), an off-policy maximum entropy deep reinforcement learning algorithm that provides sample-efficient learning while retaining the benefits of entropy maximization and stability. Our theoretical results derive soft policy iteration, which we show to converge to the optimal policy. From this result, we can formulate a soft actor-critic algorithm, and we empirically show that it outperforms state-of-the-art model-free deep RL methods, including the off-policy DDPG algorithm and the on-policy PPO algorithm. In fact, the sample efficiency of this approach actually exceeds that of DDPG by a substantial margin. Our results suggest that stochastic, entropy maximizing reinforcement learning algorithms can provide a promising avenue for improved robustness and stability, and further exploration of maximum entropy methods, including methods that incorporate second order information (e.g., trust regions (Schulman et al., 2015)) or more expressive policy classes is an exciting avenue for future work.

我们提出 soft actor-critic（SAC），一种离线策略最大熵深度强化学习算法，在保留熵最大化与稳定性好处的同时实现样本高效学习。理论结果导出软策略迭代并证明其收敛到最优策略；据此我们给出实用的 soft actor-critic 算法，并在经验上表明它在一系列挑战性连续控制任务上优于先前方法。

Acknowledgments

致谢

We would like to thank Vitchyr Pong for insightful discussions and help in implementing our algorithm as well as providing the DDPG baseline code; Ofir Nachum for offering support in running Trust-PCL experiments; and George Tucker for his valuable feedback on an early version of this paper. This work was supported by Siemens and Berkeley DeepDrive.

我们要感谢 Vitchyr Pong 的富有洞察力的讨论和帮助实现我们的算法以及提供 DDPG 基线代码； Ofir Nachum 为运行 Trust-PCL 实验提供支持； 乔治·塔克 (George Tucker) 对本文早期版本的宝贵反馈。 这项工作得到了西门子和伯克利 DeepDrive 的支持。

References

参考文献

Barto et al. (1983) Barto, A. G., Sutton, R. S., and Anderson, C. W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics , pp. 834–846, 1983.

Bhatnagar et al. (2009) Bhatnagar, S., Precup, D., Silver, D., Sutton, R. S., Maei, H. R., and Szepesvári, C. Convergent temporal-difference learning with arbitrary smooth function approximation. In Advances in Neural Information Processing Systems (NIPS) , pp. 1204–1212, 2009.

Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAI gym. arXiv preprint arXiv:1606.01540 , 2016.

Duan et al. (2016) Duan, Y., Chen, X. Houthooft, R., Schulman, J., and Abbeel, P. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning (ICML) , 2016.

Fox et al. (2016) Fox, R., Pakman, A., and Tishby, N. Taming the noise in reinforcement learning via soft updates. In Conference on Uncertainty in Artificial Intelligence (UAI) , 2016.

Fujimoto et al. (2018) Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477 , 2018.

Gruslys et al. (2017) Gruslys, A., Azar, M. G., Bellemare, M. G., and Munos, R. The reactor: A sample-efficient actor-critic architecture. arXiv preprint arXiv:1704.04651 , 2017.

Gu et al. (2016) Gu, S., Lillicrap, T., Ghahramani, Z., Turner, R. E., and Levine, S. Q-prop: Sample-efficient policy gradient with an off-policy critic. arXiv preprint arXiv:1611.02247 , 2016.

Haarnoja et al. (2017) Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning (ICML) , pp. 1352–1361, 2017.

Hasselt (2010) Hasselt, H. V. Double Q-learning. In Advances in Neural Information Processing Systems (NIPS) , pp. 2613–2621, 2010.

Heess et al. (2015) Heess, N., Wayne, G., Silver, D., Lillicrap, T., Erez, T., and Tassa, Y. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems (NIPS) , pp. 2944–2952, 2015.

Henderson et al. (2017) Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560 , 2017.

Kingma & Ba (2015) Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In International Conference for Learning Presentations (ICLR) , 2015.

Levine & Koltun (2013) Levine, S. and Koltun, V. Guided policy search. In International Conference on Machine Learning (ICML) , pp. 1–9, 2013.

Levine et al. (2016) Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research , 17(39):1–40, 2016.

Lillicrap et al. (2015) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Mnih et al. (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529–533, 2015.

Mnih et al. (2016) Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML) , 2016.

Nachum et al. (2017a) Nachum, O., Norouzi, M., Xu, K., and Schuurmans, D. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems (NIPS) , pp. 2772–2782, 2017a.

Nachum et al. (2017b) Nachum, O., Norouzi, M., Xu, K., and Schuurmans, D. Trust-PCL: An off-policy trust region method for continuous control. arXiv preprint arXiv:1707.01891 , 2017b.

O’Donoghue et al. (2016) O’Donoghue, B., Munos, R., Kavukcuoglu, K., and Mnih, V. PGQ: Combining policy gradient and Q-learning. arXiv preprint arXiv:1611.01626 , 2016.

Peters & Schaal (2008) Peters, J. and Schaal, S. Reinforcement learning of motor skills with policy gradients. Neural networks , 21(4):682–697, 2008.

Rawlik et al. (2012) Rawlik, K., Toussaint, M., and Vijayakumar, S. On stochastic optimal control and reinforcement learning by approximate inference. Robotics: Science and Systems (RSS) , 2012.

Schulman et al. (2015) Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning (ICML) , pp. 1889–1897, 2015.

Schulman et al. (2017a) Schulman, J., Abbeel, P., and Chen, X. Equivalence between policy gradients and soft Q-learning. arXiv preprint arXiv:1704.06440 , 2017a.

Schulman et al. (2017b) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017b.

Silver et al. (2014) Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International Conference on Machine Learning (ICML) , 2014.

Silver et al. (2016) Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484–489, Jan 2016. ISSN 0028-0836. Article.

Sutton & Barto (1998) Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998.

Thomas (2014) Thomas, P. Bias in natural actor-critic algorithms. In International Conference on Machine Learning (ICML) , pp. 441–448, 2014.

Todorov (2008) Todorov, E. General duality between optimal control and estimation. In IEEE Conference on Decision and Control (CDC) , pp. 4286–4292. IEEE, 2008.

Toussaint (2009) Toussaint, M. Robot trajectory optimization using approximate inference. In International Conference on Machine Learning (ICML) , pp. 1049–1056. ACM, 2009.

Williams (1992) Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3-4):229–256, 1992.

Ziebart (2010) Ziebart, B. D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy . Carnegie Mellon University, 2010.

Ziebart et al. (2008) Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI) , pp. 1433–1438, 2008.

Appendix A Maximum Entropy Objective

附录 A 最大熵目标

The exact definition of the discounted maximum entropy objective is complicated by the fact that, when using a discount factor for policy gradient methods, we typically do not discount the state distribution, only the rewards. In that sense, discounted policy gradients typically do not optimize the true discounted objective. Instead, they optimize average reward, with the discount serving to reduce variance, as discussed by Thomas (2014). However, we can define the objective that is optimized under a discount factor as

折扣最大熵目标的确切定义很复杂，因为当在策略梯度方法中使用折扣因子时，我们通常不会折扣状态分布，而只会折扣奖励。 从这个意义上说，贴现政策梯度通常不会优化真正的贴现目标。 相反，正如 Thomas (2014) 所讨论的，他们优化平均奖励，通过折扣来减少方差。 但是，我们可以将折扣因子下优化的目标定义为

$$\displaystyle J(\pi)=\sum_{t=0}^{\infty}\operatorname{\mathbb{E}}_{({\mathbf{s}_{t}},{\mathbf{a}_{t}})\sim\rho_{\pi}}\left[\sum_{l=t}^{\infty}\gamma^{l-t}\operatorname{\mathbb{E}}_{\mathbf{s}_{l}\sim p,\mathbf{a}_{l}\sim\pi}\left[r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\alpha\mathcal{H}(\pi({\,\cdot\,}|{\mathbf{s}_{t}}))|{\mathbf{s}_{t}},{\mathbf{a}_{t}}\right]\right]. \tag{14}$$

This objective corresponds to maximizing the discounted expected reward and entropy for future states originating from every state-action tuple $({\mathbf{s}_{t}},{\mathbf{a}_{t}})$ weighted by its probability $\rho_{\pi}$ under the current policy.

该目标对应于最大化来自每个状态动作元组$({\mathbf{s}_{t}},{\mathbf{a}_{t}})$的未来状态的贴现预期奖励和熵，并根据当前策略的概率$\rho_{\pi}$进行加权。

Appendix B Proofs

附录 B 证明

B.1 Lemma 1

B.1 引理 1

Lemma 1(Soft Policy Evaluation).Consider the soft Bellman backup operator $\mathcal{T}^{\pi}$ in Equation 2 and a mapping $Q^{0}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ with $|\mathcal{A}|<\infty$, and define $Q^{k+1}=\mathcal{T}^{\pi}Q^{k}$. Then the sequence $Q^{k}$ will converge to the soft Q-value of $\pi$ as $k\rightarrow\infty$.

引理 1（软策略评估）。考虑方程 2 中的软 Bellman 备份运算符$\mathcal{T}^{\pi}$以及$Q^{0}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$与$|\mathcal{A}|<\infty$的映射，并定义$Q^{k+1}=\mathcal{T}^{\pi}Q^{k}$。 然后序列$Q^{k}$将收敛到$\pi$的软 Q 值，即$k\rightarrow\infty$。

Proof.

Define the entropy augmented reward as $r_{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\triangleq r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p}\left[\mathcal{H}\left(\pi({\,\cdot\,}|{\mathbf{s}_{t+1}})\right)\right]$ and rewrite the update rule as

＃＃＃ 证明。

将熵增奖励定义为$r_{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\triangleq r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p}\left[\mathcal{H}\left(\pi({\,\cdot\,}|{\mathbf{s}_{t+1}})\right)\right]$，并将更新规则重写为

$$\displaystyle Q({\mathbf{s}_{t}},{\mathbf{a}_{t}})\leftarrow r_{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p,{\mathbf{a}_{t+1}}\sim\pi}\left[Q({\mathbf{s}_{t+1}},{\mathbf{a}_{t+1}})\right] \tag{15}$$

and apply the standard convergence results for policy evaluation (Sutton & Barto, 1998). The assumption $|\mathcal{A}|<\infty$ is required to guarantee that the entropy augmented reward is bounded. ∎

并将标准收敛结果应用于政策评估（Sutton & Barto，1998）。 需要假设$|\mathcal{A}|<\infty$来保证熵增奖励是有界的。 ∎

B.2 Lemma 2

B.2 引理 2

Lemma 2(Soft Policy Improvement).Let $\pi_{\mathrm{old}}\in\Pi$ and let $\pi_{\mathrm{new}}$ be the optimizer of the minimization problem defined in Equation 4. Then $Q^{\pi_{\mathrm{new}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\geq Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})$ for all $({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A}$ with $|\mathcal{A}|<\infty$.

引理 2（软策略改进）。让$\pi_{\mathrm{old}}\in\Pi$和$\pi_{\mathrm{new}}$成为方程 4 中定义的最小化问题的优化器。然后$Q^{\pi_{\mathrm{new}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\geq Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})$对于所有$({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A}$和$|\mathcal{A}|<\infty$。

Proof.

Let ${\pi_{\mathrm{old}}}\in\Pi$ and let $Q^{\pi_{\mathrm{old}}}$ and $V^{\pi_{\mathrm{old}}}$ be the corresponding soft state-action value and soft state value, and let ${\pi_{\mathrm{new}}}$ be defined as

＃＃＃ 证明。

令${\pi_{\mathrm{old}}}\in\Pi$、$Q^{\pi_{\mathrm{old}}}$和$V^{\pi_{\mathrm{old}}}$为对应的软状态动作值和软状态值，并令${\pi_{\mathrm{new}}}$定义为

$$\displaystyle{\pi_{\mathrm{new}}}({\,\cdot\,}|{\mathbf{s}_{t}}) \\ \displaystyle=\arg\min_{\pi^{\prime}\in\Pi}\mathrm{D_{KL}}\left(\pi^{\prime}({\,\cdot\,}|{\mathbf{s}_{t}})\;\middle\|\;\exp\left(Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\,\cdot\,})-\log Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}})\right)\right) \\ \displaystyle=\arg\min_{\pi^{\prime}\in\Pi}J_{\pi_{\mathrm{old}}}(\pi^{\prime}({\,\cdot\,}|{\mathbf{s}_{t}})). \tag{16}$$

It must be the case that $J_{\pi_{\mathrm{old}}}({\pi_{\mathrm{new}}}({\,\cdot\,}|{\mathbf{s}_{t}}))\leq J_{\pi_{\mathrm{old}}}({\pi_{\mathrm{old}}}({\,\cdot\,}|{\mathbf{s}_{t}}))$, since we can always choose ${\pi_{\mathrm{new}}}={\pi_{\mathrm{old}}}\in\Pi$. Hence

肯定是$J_{\pi_{\mathrm{old}}}({\pi_{\mathrm{new}}}({\,\cdot\,}|{\mathbf{s}_{t}}))\leq J_{\pi_{\mathrm{old}}}({\pi_{\mathrm{old}}}({\,\cdot\,}|{\mathbf{s}_{t}}))$，因为我们总是可以选择${\pi_{\mathrm{new}}}={\pi_{\mathrm{old}}}\in\Pi$。 因此

\displaystyle\resizebox{22609920}{}{$\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim{\pi_{\mathrm{new}}}}\left[\log{\pi_{\mathrm{new}}}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})-Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\log Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}})\right]\leq\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim{\pi_{\mathrm{old}}}}\left[\log{\pi_{\mathrm{old}}}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})-Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\log Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}})\right]$}, \tag{17}

and since partition function $Z^{\pi_{\mathrm{old}}}$ depends only on the state, the inequality reduces to

由于配分函数$Z^{\pi_{\mathrm{old}}}$仅取决于状态，因此不等式简化为

$$\displaystyle\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim{\pi_{\mathrm{new}}}}\left[Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\log{\pi_{\mathrm{new}}}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right]\geq V^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}}). \tag{18}$$

Next, consider the soft Bellman equation:

接下来，考虑软贝尔曼方程：

$$\displaystyle Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}}) \\ \displaystyle=r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p}\left[V^{\pi_{\mathrm{old}}}({\mathbf{s}_{t+1}})\right] \\ \displaystyle\leq r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p}\left[\operatorname{\mathbb{E}}_{{\mathbf{a}_{t+1}}\sim{\pi_{\mathrm{new}}}}\left[Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t+1}},{\mathbf{a}_{t+1}})-\log{\pi_{\mathrm{new}}}({\mathbf{a}_{t+1}}|{\mathbf{s}_{t+1}})\right]\right] \\ \displaystyle\ \ \vdots \\ \displaystyle\leq Q^{\pi_{\mathrm{new}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}}), \tag{19}$$

where we have repeatedly expanded $Q^{\pi_{\mathrm{old}}}$ on the RHS by applying the soft Bellman equation and the bound in Equation 18. Convergence to $Q^{\pi_{\mathrm{new}}}$ follows from Lemma 1. ∎

其中，我们通过应用软贝尔曼方程和方程 18 中的界限，在 RHS 上反复扩展 $Q^{\pi_{\mathrm{old}}}$。从引理 1 可以得出$Q^{\pi_{\mathrm{new}}}$的收敛性。 ∎

B.3 Theorem 1

B.3 定理 1

Theorem 1(Soft Policy Iteration). Repeated application of soft policy evaluation and soft policy improvement to any $\pi\in\Pi$ converges to a policy $\pi^{}$such that$Q^{\pi^{}}({\mathbf{s}{t}},{\mathbf{a}{t}})\geq Q^{\pi}({\mathbf{s}{t}},{\mathbf{a}{t}})$for all$\pi\in\Pi$and$({\mathbf{s}{t}},{\mathbf{a}{t}})\in\mathcal{S}\times\mathcal{A}$, assuming$|\mathcal{A}|<\infty$.

定理 1（软策略迭代）。 对任何$\pi\in\Pi$重复应用软策略评估和软策略改进会收敛到策略$\pi^{}$，使得$Q^{\pi^{}}({\mathbf{s}{t}},{\mathbf{a}{t}})\geq Q^{\pi}({\mathbf{s}{t}},{\mathbf{a}{t}})$对于所有$\pi\in\Pi$和$({\mathbf{s}{t}},{\mathbf{a}{t}})\in\mathcal{S}\times\mathcal{A}$，假设$|\mathcal{A}|<\infty$。

Proof.

Let $\pi_{i}$ be the policy at iteration $i$. By Lemma 2, the sequence $Q^{\pi_{i}}$ is monotonically increasing. Since $Q^{\pi}$ is bounded above for $\pi\in\Pi$ (both the reward and entropy are bounded), the sequence converges to some $\pi^{*}$. We will still need to show that $\pi^{*}$ is indeed optimal. At convergence, it must be case that $J_{\pi^{*}}({\pi^{*}}({\,\cdot\,}|{\mathbf{s}_{t}}))<J_{\pi^{*}}(\pi({\,\cdot\,}|{\mathbf{s}_{t}}))$ for all $\pi\in\Pi$, $\pi\neq\pi^{*}$. Using the same iterative argument as in the proof of Lemma 2, we get $Q^{\pi^{*}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})>Q^{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}})$ for all $({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A}$, that is, the soft value of any other policy in $\Pi$ is lower than that of the converged policy. Hence ${\pi^{*}}$ is optimal in $\Pi$. ∎

＃＃＃ 证明。

令$\pi_{i}$为迭代$i$时的策略。 根据引理 2，序列$Q^{\pi_{i}}$是单调递增的。 由于$Q^{\pi}$在$\pi\in\Pi$上有界（奖励和熵都有界），因此序列收敛到某个$\pi^{*}$。 我们仍然需要证明$\pi^{*}$确实是最优的。 在收敛时，所有$\pi\in\Pi$、$\pi\neq\pi^{*}$都必须是$J_{\pi^{*}}({\pi^{*}}({\,\cdot\,}|{\mathbf{s}_{t}}))<J_{\pi^{*}}(\pi({\,\cdot\,}|{\mathbf{s}_{t}}))$。 使用与引理 2 证明中相同的迭代参数，我们得到所有$({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A}$的$Q^{\pi^{*}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})>Q^{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}})$，即$\Pi$中任何其他策略的软值都低于收敛策略的软值。 因此${\pi^{*}}$在$\Pi$中是最佳的。 ∎

Appendix C Enforcing Action Bounds

附录 C 执行操作界限

We use an unbounded Gaussian as the action distribution. However, in practice, the actions needs to be bounded to a finite interval. To that end, we apply an invertible squashing function ($\tanh$) to the Gaussian samples, and employ the change of variables formula to compute the likelihoods of the bounded actions. In the other words, let $\mathbf{u}\in\mathbb{R}^{D}$ be a random variable and $\mu(\mathbf{u}|\mathbf{s})$ the corresponding density with infinite support. Then $\mathbf{a}=\tanh(\mathbf{u})$, where $\tanh$ is applied elementwise, is a random variable with support in $(-1,1)$ with a density given by

我们使用无界高斯作为动作分布。 然而，在实践中，动作需要限制在有限的间隔内。 为此，我们对高斯样本应用可逆挤压函数（$\tanh$），并采用变量变化公式来计算有界动作的可能性。 换句话说，让$\mathbf{u}\in\mathbb{R}^{D}$是一个随机变量，$\mu(\mathbf{u}|\mathbf{s})$ 是对应的具有无限支持的密度。 然后$\mathbf{a}=\tanh(\mathbf{u})$，其中$\tanh$按元素应用，是一个随机变量，支持$(-1,1)$，其密度为

$$\displaystyle\pi(\mathbf{a}|\mathbf{s}) \\ \displaystyle=\mu(\mathbf{u}|\mathbf{s})\left|\det\left(\frac{\mathrm{d}\mathbf{a}}{\mathrm{d}\mathbf{u}}\right)\right|^{-1}. \tag{20}$$

Since the Jacobian $\nicefrac{{\mathrm{d}\mathbf{a}}}{{\mathrm{d}\mathbf{u}}}=\mathrm{diag}(1-\tanh^{2}(\mathbf{u}))$ is diagonal, the log-likelihood has a simple form

由于雅可比行列式$\nicefrac{{\mathrm{d}\mathbf{a}}}{{\mathrm{d}\mathbf{u}}}=\mathrm{diag}(1-\tanh^{2}(\mathbf{u}))$是对角线，因此对数似然具有简单的形式

$$\displaystyle\log\pi(\mathbf{a}|\mathbf{s}) \\ \displaystyle=\log\mu(\mathbf{u}|\mathbf{s})-\sum_{i=1}^{D}\log\left(1-\tanh^{2}(u_{i})\right), \tag{21}$$

where $u_{i}$ is the $i^{\mathrm{th}}$ element of $\mathbf{u}$.

其中$u_{i}$是$\mathbf{u}$的$i^{\mathrm{th}}$元素。

Appendix D Hyperparameters

附录 D 超参数

Table 1 lists the common SAC parameters used in the comparative evaluation in Figure 1 and Figure 4. Table 2 lists the reward scale parameter that was tuned for each environment.

表 1 列出了图 1 和图 4 中比较评估中使用的常见 SAC 参数。表 2 列出了针对每个环境调整的奖励规模参数。

Parameter	Value
Shared
	optimizer	Adam (Kingma & Ba, 2015)
	learning rate	$3\cdot 10^{-4}$
	discount ($\gamma$)	0.99
	replay buffer size	$10^{6}$
	number of hidden layers (all networks)	2
	number of hidden units per layer	256
	number of samples per minibatch	256
	nonlinearity	ReLU
SAC
	target smoothing coefficient ($\tau$)	0.005
	target update interval	1
	gradient steps	1
SAC (hard target update)
	target smoothing coefficient ($\tau$)	1
	target update interval	1000
	gradient steps (except humanoids)	4
	gradient steps (humanoids)	1

Table 1: SAC Hyperparameters

Parameter	价值
共享
	优化器	亚当 (Kingma & Ba, 2015)
	学习率	$3\cdot 10^{-4}$
	折扣 ($\gamma$)	0.99
	重播缓冲区大小	$10^{6}$
	隐藏层数（所有网络）	2
	每层隐藏单元数	256
	每个小批量的样本数	256
	非线性	热卢
* SAC *
	目标平滑系数 ($\tau$)	0.005
	目标更新间隔	1
	梯度步长	1
SAC（硬目标更新）
	目标平滑系数 ($\tau$)	1
	目标更新间隔	1000
	梯度步长（人形机器人除外）	4
	梯度步骤（人形机器人）	1

表 1：SAC 超参数

Environment	Action Dimensions	Reward Scale
Hopper-v1	3	5
Walker2d-v1	6	5
HalfCheetah-v1	6	5
Ant-v1	8	5
Humanoid-v1	17	20
Humanoid (rllab)	21	10

Table 2: SAC Environment Specific Parameters

环境	行动维度	奖励等级
料斗-v1	3	5
沃克2D-v1	6	5
半猎豹-v1	6	5
蚂蚁-v1	8	5
人形机器人-v1	17	20
人形生物（rllab）	21	10

表 2：SAC 环境特定参数

Appendix E Additional Baseline Results

附录 E 其他基线结果

Figure 4 compares SAC to Trust-PCL (Figure 4. Trust-PC fails to solve most of the task within the given number of environment steps, although it can eventually solve the easier tasks (Nachum et al., 2017b) if ran longer. The figure also includes two variants of SAC: a variant that periodically copies the target value network weights directly instead of using exponentially moving average, and a deterministic ablation which assumes a deterministic policy in the value update (Equation 6) and the policy update (Equation 13), and thus strongly resembles DDPG with the exception of having two Q-functions, using hard target updates, not having a separate target actor, and using fixed exploration noise rather than learned. Both of these methods can learn all of the tasks and they perform comparably to SAC on all but Humanoid (rllab) task, on which SAC is the fastest.

图 4 将 SAC 与 Trust-PCL 进行了比较（图 4。Trust-PC 无法在给定数量的环境步骤内解决大部分任务，尽管如果运行时间更长，它最终可以解决更简单的任务（Nachum 等人，2017b）。该图还包括 SAC 的两个变体：一种定期直接复制目标值网络权重而不是使用指数移动平均值的变体，以及一种在值更新中假设确定性策略的确定性消融 （公式 6）和策略更新（公式 13），因此与 DDPG 非常相似，除了具有两个 Q 函数、使用硬目标更新、没有单独的目标参与者以及使用固定的探索噪声而不是学习之外，这两种方法都可以学习所有任务，并且它们在除 Humanoid (rllab) 任务之外的所有任务上的表现与 SAC 相当，其中 SAC 是最快的。

(a) Hopper-v1

(a) 料斗-v1