Implicit Quantile Networks for Distributional Reinforcement Learning
Implicit Quantile Networks for Distributional Reinforcement Learning
Abstract
摘要
In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regression to approximate the full quantile function for the state-action return distribution. By reparameterizing a distribution over the sample space, this yields an implicitly defined return distribution and gives rise to a large class of risk-sensitive policies. We demonstrate improved performance on the 57 Atari 2600 games in the ALE, and use our algorithm’s implicitly defined distributions to study the effects of risk-sensitive policies in Atari games.
本文在分布强化学习的近期进展之上,给出一种通用、灵活、且当时表现领先的分布化 DQN 变体。做法是用分位数回归去逼近状态-动作回报分布的完整分位数函数。通过对样本空间上的分布做重参数化,可以得到一条隐式定义的回报分布,并由此引出一大类风险敏感策略。我们在 ALE 的 57 款 Atari 2600 游戏上展示了更好的表现,并利用算法隐式给出的分布,观察风险敏感策略在 Atari 游戏中的影响。
*Equal contribution 1DeepMind, London, UK. Correspondence to: Will Dabney wdabney@google.com, Georg Ostrovski ostrovski@google.com.
Proceedings of the 35 International Conference on Machine Learning, Stockholm, Sweden, PMLR 80, 2018. Copyright 2018 by the author(s).
- Introduction
- 引言
Distributional reinforcement learning (Jaquette, 1973; Sobel, 1982; White, 1988; Morimura et al., 2010b; Bellemare et al., 2017) focuses on the intrinsic randomness of returns within the reinforcement learning (RL) framework. As the agent interacts with the environment, irreducible randomness seeps in through the stochasticity of these interactions, the approximations in the agent’s representation, and even the inherently chaotic nature of physical interaction (Yu et al., 2016). Distributional RL aims to model the distribution over returns, whose mean is the traditional value function, and to use these distributions to evaluate and optimize a policy.
分布强化学习(distributional reinforcement learning)(Jaquette, 1973; Sobel, 1982; White, 1988; Morimura et al., 2010b; Bellemare et al., 2017)关注的是强化学习(RL)框架里回报本身的内在随机性。智能体与环境交互时,这些交互的随机性、智能体表征中的近似,乃至物理交互固有的混沌性质(Yu et al., 2016),都会引入无法消去的随机性。分布强化学习的目标,是对回报分布建模——其均值就是传统的价值函数——并用这些分布去评估和优化策略。
Any distributional RL algorithm is characterized by two aspects: the parameterization of the return distribution, and the distance metric or loss function being optimized. Together, these choices control assumptions about the random returns and how approximations will be traded off. Categorical DQN (Bellemare et al., 2017, C51) combines a categorical distribution and the cross-entropy loss with the Cramér-minimizing projection (Rowland et al., 2018). For this, it assumes returns are bounded in a known range and trades off mean-preservation at the cost of overestimating variance.
任何一种分布强化学习算法都可以从两方面来刻画:回报分布的参数化方式,以及所优化的距离度量或损失函数。这两项选择共同决定了关于随机回报的假设,以及近似将如何取舍。Categorical DQN(Bellemare et al., 2017, C51)把分类分布和交叉熵损失,与使 Cramér 距离最小的投影(Rowland et al., 2018)结合在一起。为此,它假定回报落在已知的有界区间内,并在保持均值的同时,以高估方差为代价做折中。
C51 outperformed all previous improvements to DQN on a set of 57 Atari 2600 games in the Arcade Learning Environment (Bellemare et al., 2013), which we refer to as the Atari-57 benchmark. Subsequently, several papers have built upon this successful combination to achieve significant improvements to the state-of-the-art in Atari-57 (Hessel et al., 2018; Gruslys et al., 2018), and challenging continuous control tasks (Barth-Maron et al., 2018).
在 Arcade Learning Environment(Bellemare et al., 2013)的 57 款 Atari 2600 游戏上——下文称作 Atari-57 基准——C51 超过了此前对 DQN 的所有改进。随后又有若干工作建立在这一成功组合之上,在 Atari-57(Hessel et al., 2018; Gruslys et al., 2018)以及有挑战性的连续控制任务(Barth-Maron et al., 2018)上,把当时的最好成绩又向前推进了一截。
These algorithms are restricted to assigning probabilities to an a priori fixed, discrete set of possible returns. Dabney et al. (2018) propose an alternate pair of choices, parameterizing the distribution by a uniform mixture of Diracs whose locations are adjusted using quantile regression. Their algorithm, QR-DQN, while restricted to a discrete set of quantiles, automatically adapts return quantiles to minimize the Wasserstein distance between the Bellman updated and current return distributions. This flexibility allows QR-DQN to significantly improve on C51’s Atari-57 performance.
这些算法只能把概率分配给事先固定好的一组离散可能回报。Dabney et al. (2018) 提出了另一对选择:用一组 Dirac 冲激的均匀混合来参数化分布,并用分位数回归去调整冲激的位置。他们的算法 QR-DQN 虽然仍限于离散的一组分位点,但会自动调整回报分位数,以减小贝尔曼更新后的回报分布与当前回报分布之间的 Wasserstein 距离。这种灵活性使 QR-DQN 相对 C51 在 Atari-57 上的成绩有了明显提升。
In this paper, we extend the approach of Dabney et al. (2018), from learning a discrete set of quantiles to learning the full quantile function, a continuous map from probabilities to returns. When combined with a base distribution, such as , this forms an implicit distribution capable of approximating any distribution over returns given sufficient network capacity. Our approach, implicit quantile networks (IQN), is best viewed as a simple distributional generalization of the DQN algorithm (Mnih et al., 2015), and provides several benefits over QR-DQN.
本文把 Dabney et al. (2018) 的方法从学习一组离散分位数,推进到学习完整的分位数函数,也就是从概率到回报的连续映射。再与一个基分布(例如 )结合,就构成一条隐式分布;只要网络容量足够,它就能逼近任意的回报分布。我们把这种方法称作隐式分位数网络(implicit quantile networks,IQN)。它最好被看作 DQN 算法(Mnih et al., 2015)的一种简洁的分布化推广,并且相对 QR-DQN 有若干好处。
First, the approximation error for the distribution is no longer controlled by the number of quantiles output by the network, but by the size of the network itself, and the amount of training. Second, IQN can be used with as few, or as many, samples per update as desired, providing improved data efficiency with increasing number of samples per training update. Third, the implicit representation of the return distribution allows us to expand the class of policies to more fully take advantage of the learned distribution. Specifically, by taking the base distribution to be non-uniform, we expand the class of policies to -greedy policies on arbitrary distortion risk measures (Yaari, 1987; Wang, 1996).
第一,分布的逼近误差不再由网络输出的分位数个数决定,而由网络本身的规模以及训练量决定。第二,每次更新时,IQN 可以使用任意少或任意多的样本,样本数增加会带来更好的数据效率。第三,对回报分布的隐式表示,使我们能够扩大策略类,从而更充分地利用学到的分布。具体来说,只要把基分布取成非均匀的,就可以把策略类扩展为针对任意扭曲风险度量(distortion risk measures)(Yaari, 1987; Wang, 1996)的 -greedy 策略。
We begin by reviewing distributional reinforcement learning, related work, and introducing the concepts surrounding risk-sensitive RL. In subsequent sections, we introduce our proposed algorithm, IQN, and present a series of experiments using the Atari-57 benchmark, investigating the robustness and performance of IQN. Despite being a simple distributional extension to DQN, and forgoing any other improvements, IQN significantly outperforms QR-DQN and nearly matches the performance of Rainbow, which combines many orthogonal advances. In fact, in human-starts as well as in the hardest Atari games (where current RL agents still underperform human players) IQN improves over Rainbow.
我们先回顾分布强化学习与相关工作,并介绍风险敏感强化学习的有关概念。随后各节将给出我们提出的算法 IQN,并在 Atari-57 基准上做一系列实验,考察它的稳健性与表现。尽管 IQN 只是对 DQN 做了简单的分布化扩展,并且没有叠加其他改进,它仍然明显超过 QR-DQN,并接近融合了许多正交进展的 Rainbow。事实上,在人类开局(human-starts)设定下,以及在当前强化学习智能体仍不如人类玩家的最难 Atari 游戏上,IQN 还超过了 Rainbow。
- Background / Related Work
- 背景与相关工作
We consider the standard RL setting, in which the interaction of an agent and an environment is modeled as a Markov Decision Process (Puterman, 1994), where and denote the state and action spaces, the (state- and action-dependent) reward function, the transition kernel, and a discount factor. A policy maps a state to a distribution over actions.
我们考虑标准的强化学习设定:智能体与环境的交互被建模为马尔可夫决策过程 (Puterman, 1994)。其中 和 分别表示状态空间与动作空间, 是(依赖于状态和动作的)奖励函数, 是转移核, 是折扣因子。策略 把一个状态映射为动作上的分布。
For an agent following policy , the discounted sum of future rewards is denoted by the random variable , where , , , and . The action-value function is defined as , and can be characterized by the Bellman equation
对遵循策略 的智能体,未来奖励的折扣和用随机变量 表示,其中 ,,,且 。动作价值函数定义为 ,并可由贝尔曼方程刻画:
The objective in RL is to find an optimal policy , which maximizes , i.e. for all and all . One approach is to find the unique fixed point of the Bellman optimality operator (Bellman, 1957):
强化学习的目标是找到一个最优策略 ,使 最大,也就是对所有策略 以及所有 ,都有 。一种做法是求贝尔曼最优算子(Bellman, 1957)的唯一不动点 :
To this end, Q-learning (Watkins, 1989) iteratively improves an estimate, , of the optimal action-value function, , by repeatedly applying the Bellman update:
为此,Q-learning(Watkins, 1989)通过反复施加贝尔曼更新,迭代改进对最优动作价值函数 的估计 :
The action-value function can be approximated by a parameterized function (e.g. a neural network), and trained by minimizing the squared temporal difference (TD) error,
动作价值函数可以用参数化函数 (例如神经网络)来逼近,并通过最小化平方时序差分(TD)误差来训练:
over samples observed while following an -greedy policy over . This policy acts greedily with respect to with probability and uniformly at random otherwise. DQN (Mnih et al., 2015) uses a convolutional neural network to parameterize and the Q-learning algorithm to achieve human-level play on the Atari-57 benchmark.
样本 来自在 上执行 -greedy 策略时的观察。该策略以概率 对 贪婪行动,否则均匀随机行动。DQN(Mnih et al., 2015)用卷积神经网络参数化 ,并借助 Q-learning 算法,在 Atari-57 基准上达到了人类水平的表现。
2.1. Distributional RL
2.1. 分布强化学习
In distributional RL, the distribution over returns (the law of ) is considered instead of the scalar value function that is its expectation. This change in perspective has yielded new insights into the dynamics of RL (Azar et al., 2012), and been a useful tool for analysis (Lattimore & Hutter, 2012). Empirically, distributional RL algorithms show improved sample complexity and final performance, as well as increased robustness to hyperparameter variation (Barth-Maron et al., 2018).
在分布强化学习中,关注的是回报分布(即 的分布),而不是作为其期望的标量价值函数 。这一视角上的转变,为强化学习的动态提供了新的认识(Azar et al., 2012),也成为有用的分析工具(Lattimore & Hutter, 2012)。经验上,分布强化学习算法在样本复杂度和最终表现上更好,对超参数变化也更稳健(Barth-Maron et al., 2018)。
An analogous distributional Bellman equation of the form
可以推导出形式类似的分布贝尔曼方程
can be derived, where denotes that two random variables and have equal probability laws, and the random variables and are distributed according to and , respectively.
其中 表示两个随机变量 和 具有相同的概率律,而随机变量 和 分别服从 和 。
Morimura et al. (2010a) defined the distributional Bellman operator explicitly in terms of conditional probabilities, parameterized by the mean and scale of a Gaussian or Laplace distribution, and minimized the Kullback-Leibler (KL) divergence between the Bellman target and the current estimated return distribution. However, the distributional Bellman operator is not a contraction in the KL.
Morimura et al. (2010a) 用条件概率显式定义了分布贝尔曼算子,并以高斯或拉普拉斯分布的均值与尺度作为参数,最小化贝尔曼目标与当前估计回报分布之间的 Kullback-Leibler(KL)散度。不过,分布贝尔曼算子在 KL 散度下并不是压缩映射。
As with the scalar setting, a distributional Bellman optimality operator can be defined by
与标量情形类似,也可以定义分布贝尔曼最优算子:
with distributed according to . While the distributional Bellman operator for policy evaluation is a contraction in the -Wasserstein distance (Bellemare et al., 2017), this no longer holds for the control case. Convergence to the optimal policy can still be established, but requires a more involved argument.
其中 服从 。策略评估下的分布贝尔曼算子在 -Wasserstein 距离下是压缩映射(Bellemare et al., 2017),但控制情形不再成立。仍然可以证明收敛到最优策略,不过论证要更复杂。
Bellemare et al. (2017) parameterize the return distribution as a categorical distribution over a fixed set of equidistant points and minimize the KL divergence to the projected distributional Bellman target. Their algorithm, C51, outperformed previous DQN variants on the Atari-57 benchmark. Subsequently, Hessel et al. (2018) combined C51 with enhancements such as prioritized experience replay (Schaul et al., 2016), -step updates (Sutton, 1988), and the dueling architecture (Wang et al., 2016), leading to the Rainbow agent, current state-of-the-art in Atari-57.
Bellemare et al. (2017) 把回报分布参数化为固定一组等距点上的分类分布,并最小化到投影后分布贝尔曼目标的 KL 散度。他们的算法 C51 在 Atari-57 基准上超过了此前的 DQN 变体。随后,Hessel et al. (2018) 把 C51 与优先经验回放(Schaul et al., 2016)、 步更新(Sutton, 1988)以及对决架构(Wang et al., 2016)等改进结合,得到了 Rainbow 智能体,也就是当时 Atari-57 上的最好结果。
The categorical parameterization, using the projected KL loss, has also been used in recent work to improve the critic of a policy gradient algorithm, D4PG, achieving significantly improved robustness and state-of-the-art performance across a variety of continuous control tasks (Barth-Maron et al., 2018).
这种分类参数化再配上投影 KL 损失,近期也被用来改进策略梯度算法 D4PG 的评价器,在多种连续控制任务上显著提高了稳健性,并达到了当时最好的表现(Barth-Maron et al., 2018)。
2.2. -Wasserstein Metric
2.2. -Wasserstein 度量
The -Wasserstein metric, for , plays a key role in recent results in distributional RL (Bellemare et al., 2017; Dabney et al., 2018). It has also been a topic of increasing interest in generative modeling (Arjovsky et al., 2017; Bousquet et al., 2017; Tolstikhin et al., 2017), because unlike the KL divergence, the Wasserstein metric inherently trades off approximate solutions with likelihoods.
-Wasserstein 度量()在分布强化学习的近期结果中起着关键作用(Bellemare et al., 2017; Dabney et al., 2018)。它在生成建模中也日益受到关注(Arjovsky et al., 2017; Bousquet et al., 2017; Tolstikhin et al., 2017),因为与 KL 散度不同,Wasserstein 度量本身就会在近似解与似然之间做权衡。
The -Wasserstein distance is the metric on inverse cumulative distribution functions (c.d.f.), also known as quantile functions (Müller, 1997). For random variables and with quantile functions and , respectively, the -Wasserstein distance is given by
-Wasserstein 距离就是逆累积分布函数(c.d.f.,也称作分位数函数)上的 度量(Müller, 1997)。对分别具有分位数函数 和 的随机变量 和 ,该距离为
The class of optimal transport metrics express distances between distributions in terms of the minimal cost for transporting mass to make the two distributions identical. This cost is given in terms of some metric, , on the underlying space . The -Wasserstein metric corresponds to . We are particularly interested in the Wasserstein metrics due to the predominant use of spaces in mean-value reinforcement learning.
最优传输度量这一类,是用把质量搬运到使两个分布相同所需的最小代价,来表达分布之间的距离。该代价由底空间 上的某个度量 给出。-Wasserstein 度量对应于 。我们对 Wasserstein 度量特别感兴趣,是因为均值型强化学习里普遍使用 空间。
2.3. Quantile Regression for Distributional RL
2.3. 用于分布强化学习的分位数回归
Bellemare et al. (2017) showed that the distributional Bellman operator is a contraction in the -Wasserstein metric, but as the proposed algorithm did not itself minimize the Wasserstein metric, this left a theory-practice gap for distributional RL. Recently, this gap was closed, in both directions. First and most relevant to this work, Dabney et al. (2018) proposed the use of quantile regression for distributional RL and showed that by choosing the quantile targets suitably the resulting projected distributional Bellman operator is a contraction in the -Wasserstein metric. Concurrently, Rowland et al. (2018) showed the original class of categorical algorithms are a contraction in the Cramér distance, the metric on cumulative distribution functions.
Bellemare et al. (2017) 证明了分布贝尔曼算子在 -Wasserstein 度量下是压缩映射,但他们提出的算法本身并不最小化 Wasserstein 度量,这给分布强化学习留下了理论与实践之间的空隙。这一空隙近来从两个方向都被补上了。首先,也是与本文最相关的,Dabney et al. (2018) 提出把分位数回归用于分布强化学习,并证明只要适当选择分位数目标,得到的投影分布贝尔曼算子在 -Wasserstein 度量下是压缩映射。与此同时,Rowland et al. (2018) 证明原先那一类分类算法在 Cramér 距离(累积分布函数上的 度量)下是压缩映射。
By estimating the quantile function at precisely chosen points, QR-DQN minimizes the Wasserstein distance to the distributional Bellman target (Dabney et al., 2018). This estimation uses quantile regression, which has been shown to converge to the true quantile function value when minimized using stochastic approximation (Koenker, 2005).
QR-DQN 通过在精心选定的点上估计分位数函数,来最小化到分布贝尔曼目标的 Wasserstein 距离(Dabney et al., 2018)。这一估计使用分位数回归;已有结果表明,用随机逼近最小化该损失时,会收敛到真实的分位函数值(Koenker, 2005)。
In QR-DQN, the random return is approximated by a uniform mixture of Diracs,
在 QR-DQN 中,随机回报由 个 Dirac 冲激的均匀混合来逼近:
with each assigned a fixed quantile target, for , where . These quantile estimates are trained using the Huber (1964) quantile regression loss, with threshold ,
其中每个 被赋予一个固定的分位数目标 (),且 。这些分位数估计用带阈值 的 Huber(1964)分位数回归损失来训练:
with
其中
on the pairwise TD-errors
作用在成对的 TD 误差上:
At the time of this writing, QR-DQN achieves the best performance on Atari-57, human-normalized mean and median, of all agents that do not combine distributional RL, prioritized replay, and -step updates (Dabney et al., 2018; Hessel et al., 2018; Gruslys et al., 2018).
在本文写作时,在所有没有把分布强化学习、优先回放和 步更新结合在一起的智能体中,QR-DQN 在 Atari-57 的人类归一化均值和中位数上都取得了最好成绩(Dabney et al., 2018; Hessel et al., 2018; Gruslys et al., 2018)。
2.4. Risk in Reinforcement Learning
2.4. 强化学习中的风险
Distributional RL algorithms have been theoretically justified for the Wasserstein and Cramér metrics (Bellemare et al., 2017; Rowland et al., 2018), and learning the distribution over returns, in and of itself, empirically results in significant improvements to data efficiency, final performance, and stability (Bellemare et al., 2017; Dabney et al., 2018; Gruslys et al., 2018; Barth-Maron et al., 2018). However, in each of these recent works the policy used was based entirely on the mean of the return distribution, just as in standard reinforcement learning. A natural question arises: can we expand the class of policies using information provided by the distribution over returns (i.e. to the class of risk-sensitive policies)? Furthermore, when would this larger policy class be beneficial?
分布强化学习算法已在 Wasserstein 度量和 Cramér 度量下得到理论支持(Bellemare et al., 2017; Rowland et al., 2018);而仅仅学习回报分布本身,经验上就会在数据效率、最终表现和稳定性上带来明显改进(Bellemare et al., 2017; Dabney et al., 2018; Gruslys et al., 2018; Barth-Maron et al., 2018)。不过,这些近期工作所用的策略,仍然完全基于回报分布的均值,与标准强化学习一样。一个自然的问题是:能否利用回报分布提供的信息来扩大策略类(也就是扩到风险敏感策略类)?此外,这个更大的策略类在什么时候会带来好处?
Here, ‘risk’ refers to the uncertainty over possible outcomes, and risk-sensitive policies are those which depend upon more than the mean of the outcomes. At this point, it is important to highlight the difference between intrinsic uncertainty, captured by the distribution over returns, and parametric uncertainty, the uncertainty over the value estimate typically associated with Bayesian approaches such as PSRL (Osband et al., 2013) and Kalman TD (Geist & Pietquin, 2010). Distributional RL seeks to capture the former, which classic approaches to risk are built upon.
这里的「风险」指的是对可能结果的不确定性,而风险敏感策略是那些并不只依赖于结果均值的策略。此处需要强调两种不确定性的区别:内在不确定性由回报分布刻画;参数不确定性则是对价值估计本身的不确定,通常与 PSRL(Osband et al., 2013)和 Kalman TD(Geist & Pietquin, 2010)这类贝叶斯方法联系在一起。分布强化学习要捕捉的是前者,而经典的风险处理方法也正是建立在前者之上。
One exception is the recent work (Moerland et al., 2017) towards combining both forms of uncertainty to improve exploration.
一个例外是近期工作(Moerland et al., 2017),它试图把两种不确定性结合起来以改进探索。
Expected utility theory states that if a decision policy is consistent with a particular set of four axioms regarding its choices then the decision policy behaves as though it is maximizing the expected value of some utility function (von Neumann & Morgenstern, 1947),
期望效用理论指出:若一个决策策略在选择上与某一组共四条公理相一致,那么该策略的行为就相当于在最大化某个效用函数 的期望(von Neumann & Morgenstern, 1947):
This is perhaps the most pervasive notion of risk-sensitivity. A policy maximizing a linear utility function is called risk-neutral, whereas concave or convex utility functions give rise to risk-averse or risk-seeking policies, respectively. Many previous studies on risk-sensitive RL adopt the utility function approach (Howard & Matheson, 1972; Marcus et al., 1997; Maddison et al., 2017).
这大概是最常见的风险敏感概念。最大化线性效用函数的策略称为风险中性;凹效用函数和凸效用函数则分别给出风险回避和风险寻求策略。许多关于风险敏感强化学习的既往研究采用的就是效用函数路线(Howard & Matheson, 1972; Marcus et al., 1997; Maddison et al., 2017)。
A crucial axiom of expected utility is independence: given random variables , and , such that ( preferred over ), any mixture between and is preferred to the same mixture between and (von Neumann & Morgenstern, 1947). Stated in terms of the cumulative probability functions, , . This axiom in particular has troubled many researchers because it is consistently violated by human behavior (Tversky & Kahneman, 1992). The Allais paradox is a frequently used example of a decision problem where people violate the independence axiom of expected utility theory (Allais, 1990).
期望效用理论的一条关键公理是独立性:给定随机变量 、 和 ,若 (即 优于 ),则 与 的任意混合都优于 与 的同样混合(von Neumann & Morgenstern, 1947)。用累积概率函数来写,就是 ,对任意 都成立。这条公理尤其让许多研究者感到困扰,因为人类行为会一贯地违反它(Tversky & Kahneman, 1992)。阿莱悖论(Allais paradox)就是一个常被引用的决策问题,人们会在其中违反期望效用理论的独立性公理(Allais, 1990)。
However, as Yaari (1987) showed, this axiom can be replaced by one in terms of convex combinations of outcome values, instead of mixtures of distributions. Specifically, if as before , then for any and random variable , . This leads to an alternate, dual, theory of choice than that of expected utility. Under these axioms the decision policy behaves as though it is maximizing a distorted expectation, for some continuous monotonic function :
不过,正如 Yaari (1987) 所示,这条公理可以换成用结果取值的凸组合来表述,而不是用分布的混合。具体来说,若仍有 ,则对任意 和随机变量 ,都有 。由此得到的是与期望效用不同的、对偶的另一套选择理论。在这些公理下,决策策略的行为相当于在最大化某种扭曲期望,其中 是某个连续单调函数:
Such a function is known as a distortion risk measure, as it distorts the cumulative probabilities of the random variable (Wang, 1996). That is, we have two fundamentally equivalent approaches to risk-sensitivity. Either, we choose a utility function and follow the expectation of this utility. Or, we choose a reweighting of the distribution and compute expectation under this distortion measure. Indeed, Yaari (1987) further showed that these two functions are inverses of each other. The choice between them amounts to a choice over whether the behavior should be invariant to mixing with random events or to convex combinations of outcomes.
这样的函数 被称作扭曲风险度量,因为它扭曲了随机变量的累积概率(Wang, 1996)。也就是说,对待风险敏感有两条本质上等价的路。一条是选择一个效用函数,并追随该效用的期望;另一条是选择对分布的一种重新加权,再在这种扭曲度量下计算期望。事实上,Yaari (1987) 进一步证明这两个函数互为逆。在二者之间做选择,相当于决定行为应当对与随机事件的混合保持不变,还是对结果的凸组合保持不变。
图 1:DQN 以及近期分布强化学习算法的网络结构。
Distortion risk measures include, as special cases, cumulative probability weighting used in cumulative prospect theory (Tversky & Kahneman, 1992), conditional value at risk (Chow & Ghavamzadeh, 2014), and many other methods (Morimura et al., 2010b). Recently Majumdar & Pavone (2017) argued for the use of distortion risk measures in robotics.
扭曲风险度量的特例包括累积前景理论中的累积概率加权(Tversky & Kahneman, 1992)、条件风险价值(Chow & Ghavamzadeh, 2014),以及许多其他方法(Morimura et al., 2010b)。近期 Majumdar & Pavone (2017) 还主张在机器人学中使用扭曲风险度量。
- Implicit Quantile Networks
- 隐式分位数网络
We now introduce the implicit quantile network (IQN), a deterministic parametric function trained to reparameterize samples from a base distribution, e.g. , to the respective quantile values of a target distribution. IQN provides an effective way to learn an implicit representation of the return distribution, yielding a powerful function approximator for a new DQN-like agent.
下面介绍隐式分位数网络(IQN)。它是一个确定性的参数化函数,训练目标是把来自某个基分布的样本(例如 )重参数化为目标分布相应的分位值。IQN 提供了一种有效的方式来学习回报分布的隐式表示,从而为一种新的类 DQN 智能体给出了有力的函数逼近器。
Let be the quantile function at for the random variable . For notational simplicity we write , thus for the resulting state-action return distribution sample is . We propose to model the state-action quantile function as a mapping from state-actions and samples from some base distribution, typically , to , viewed as samples from the implicitly defined return distribution.
设 为随机变量 在 处的分位数函数。为记号简便,我们写 ,于是当 时,得到的状态-动作回报分布样本满足 。我们提出把状态-动作分位数函数建模成一个映射:它以状态-动作以及来自某个基分布(通常是 )的样本为输入,输出 ,并把这些输出看作隐式定义的回报分布中的样本。
Let be a distortion risk measure, with identity corresponding to risk-neutrality. Then, the distorted expectation of under is given by
设 为扭曲风险度量,其中恒等映射对应于风险中性。那么, 在 下的扭曲期望为
Notice that the distorted expectation is equal to the expected value of weighted by , that is, . The immediate implication of this is that for any , there exists a sampling distribution for such that the mean of is equal to the distorted expectation of under , that is, any distorted expectation can be represented as a weighted sum over the quantiles (Dhaene et al., 2012).
注意,扭曲期望等于 在 加权下的期望,也就是 。一个直接推论是:对任意 ,都存在 的某种抽样分布,使得 的均值等于 在 下的扭曲期望。也就是说,任何扭曲期望都可以表示成分位数的加权和(Dhaene et al., 2012)。
Denote by the risk-sensitive greedy policy
把风险敏感的贪婪策略记为 :
For two samples , and policy , the sampled temporal difference (TD) error at step is
对两个样本 以及策略 ,第 步的抽样时序差分(TD)误差为
Then, the IQN loss function is given by
于是,IQN 的损失函数为
where and denote the respective number of iid samples used to estimate the loss. A corresponding sample-based risk-sensitive policy is obtained by approximating in Equation 1 by samples of :
其中 和 分别表示用于估计该损失的、独立同分布样本 的个数。相应的基于样本的风险敏感策略,是用 个 的样本去逼近公式 1 中的 得到的:
Implicit quantile networks differ from the approach of Dabney et al. (2018) in two ways. First, instead of approximating the quantile function at fixed values of we approximate it with for some differentiable functions , , and . If we ignore the distributional interpretation for a moment and view each as a separate action-value function, this highlights that implicit quantile networks are a type of universal value function approximator (UVFA) (Schaul et al., 2015). There may be additional benefits to implicit quantile networks beyond the obvious increase in representational fidelity. As with UVFAs, we might hope that training over many different ’s (goals in the case of the UVFA) leads to better generalization between values and improved sample complexity than attempting to train each separately.
隐式分位数网络与 Dabney et al. (2018) 的方法有两点不同。第一,我们不再只在 个固定的 值上逼近分位数函数,而是用 来逼近,其中 、 和 是一些可微函数。若暂时抛开分布上的解释,把每个 都看成单独的动作价值函数,就可以看出隐式分位数网络是一类通用价值函数逼近器(UVFA)(Schaul et al., 2015)。除了表征保真度的明显提高,隐式分位数网络还可能有其他好处。与 UVFA 类似,我们可以期望:在许多不同的 (对 UVFA 而言则是许多不同的目标)上一起训练,会比分别训练每一个带来更好的价值泛化,以及更好的样本复杂度。
Second, , , and are sampled from continuous, independent, distributions. Besides , we also explore risk-sentive policies , with non-linear . The independent sampling of each , results in the sample TD errors being decorrelated, and the estimated action-values go from being the true mean of a mixture of Diracs to a sample mean of the implicit distribution defined by reparameterizing the sampling distribution via the learned quantile function.
第二,、 和 都从连续且相互独立的分布中抽样。除了 ,我们还考察带非线性 的风险敏感策略 。对每个 、 独立抽样,会使样本 TD 误差去相关;估计的动作价值也不再是 个 Dirac 混合的真均值,而变成隐式分布的样本均值——该隐式分布是由学到的分位数函数对抽样分布做重参数化得到的。
3.1. Implementation
3.1. 实现
Consider the neural network structure used by the DQN agent (Mnih et al., 2015). Let be the function computed by the convolutional layers and the subsequent fully-connected layers mapping to the estimated action-values, such that . For our network we use the same functions and as in DQN, but include an additional function computing an embedding for the sample point . We combine these to form the approximation , where denotes the element-wise (Hadamard) product.
考虑 DQN 智能体所用的网络结构(Mnih et al., 2015)。设 为卷积层计算的函数, 为随后把 映射到估计动作价值的全连接层,从而 。我们的网络沿用 DQN 中同样的 和 ,但另外加入一个函数 ,用来计算样本点 的嵌入。把它们组合起来,得到逼近 ,其中 表示逐元素(Hadamard)乘积。
As the network for is not particularly deep, we use the multiplicative form, , to force interaction between the convolutional features and the sample embedding. Alternative functional forms, e.g. concatenation or a ‘residual’ function , are conceivable, and can be parameterized in different ways. To investigate these, we compared performance across a number of architectural variants on six Atari 2600 games (ASTERIX, ASSAULT, BREAKOUT, MS.PACMAN, QBERT, SPACE INVADERS). Full results are given in the Appendix. Despite minor variation in performance, we found the general approach to be robust to the various choices. Based upon the results we used the following function in our later experiments, for embedding dimension :
由于 的网络并不特别深,我们采用相乘形式 ,以迫使卷积特征与样本嵌入发生交互。也可以设想其他函数形式,例如拼接,或「残差」形式 ; 也可以用不同方式参数化。为考察这些选择,我们在六款 Atari 2600 游戏(ASTERIX、ASSAULT、BREAKOUT、MS.PACMAN、QBERT、SPACE INVADERS)上比较了若干结构变体的表现。完整结果见附录。尽管成绩上有些小差异,但我们发现总体方法对这些选择是稳健的。根据这些结果,后续实验采用嵌入维度 的如下函数:
After settling on a network architecture, we study the effect of the number of samples, and , used in the estimate terms of Equation 3.
在确定网络结构之后,我们再考察公式 3 的估计项中所用样本数 和 的影响。
We hypothesized that , the number of samples of , would affect the sample complexity of IQN, with larger values leading to faster learning, and that with one would potentially approach the performance of DQN. This would support the hypothesis that the improved performance of many distributional RL algorithms rests on their effect as auxiliary loss functions, which would vanish in the case of . Furthermore, we believed that , the number of samples of , would affect the variance of the gradient estimates much like a mini-batch size hyperparameter. Our prediction was that would have the greatest effect on variance of the long-term performance of the agent.
我们的假设是:(即 的样本数)会影响 IQN 的样本复杂度,更大的 会带来更快的学习;而当 时,表现有可能接近 DQN。若果真如此,就支持了这样一种看法:许多分布强化学习算法之所以更好,是因为它们起到了辅助损失的作用,而在 时这种作用会消失。此外,我们认为 (即 的样本数)会像小批量大小这一超参数那样,影响梯度估计的方差。我们的预测是: 对智能体长期表现的方差影响最大。
We used the same set of six games as before, with our chosen architecture, and varied . In Figure 2 we report the average human-normalized scores on the six games for each configuration. Figure 2 (left) shows the average performance over the first ten million frames, while (right) shows the average performance over the last ten million (from 190M to 200M).
我们沿用前面那六款游戏和已选定的结构,并让 变化。图 2 给出了每种配置在六款游戏上的平均人类归一化分数。图 2(左)是前一千万帧上的平均表现,(右)是最后一千万帧(从 1.9 亿到 2 亿帧)上的平均表现。
Figure 2. Effect of varying and , the number of samples used in the loss function in Equation 3. Figures show human-normalized agent performance, averaged over six Atari games, averaged over first 10M frames of training (left) and last 10M frames of training (right). Corresponding values for baselines: DQN (32, 253) and QR-DQN (144, 1243).
图 2:改变公式 3 损失函数中所用样本数 和 的影响。图中给出智能体的人类归一化表现,对六款 Atari 游戏取平均,并分别对训练的前 1000 万帧(左)和最后 1000 万帧(右)取平均。相应的基线数值为:DQN(32, 253)和 QR-DQN(144, 1243)。
As expected, we found that has a dramatic effect on early performance, shown by the continual improvement in score as the value increases. Additionally, we observed that affected performance very differently than expected: it had a strong effect on early performance, but minimal impact on long-term performance past .
正如所料,我们发现 对早期表现影响很大:分数随着 增大持续提高。此外, 对表现的影响与预期很不一样:它对早期表现影响很强,但一旦超过 ,对长期表现的影响就很小。
Overall, while using more samples for both distributions is generally favorable, appears to be sufficient to achieve the majority of improvements offered by IQN for long-term performance, with variation past this point largely insignificant. To our surprise we found that even for , which is comparable to DQN in the number of loss components, the longer term performance is still quite strong ( DQN).
总体来看,虽然对两个分布都使用更多样本通常更好,但就长期表现而言, 似乎已足以拿到 IQN 带来的大部分提升,再往上增加,差异大体上可以忽略。出乎我们意料的是,即使 ——此时损失项的个数已与 DQN 相当——更长期的表现仍然相当强(大约是 DQN 的 倍)。
In an informal evaluation, we did not find IQN to be sensitive to , the number of samples used for the policy, and have fixed it at for all experiments.
在非正式评估中,我们没有发现 IQN 对策略所用样本数 敏感,因此全部实验都把 固定为 。
- Risk-Sensitive Reinforcement Learning
- 风险敏感强化学习
In this section, we explore the effects of varying the distortion risk measure, , away from identity. This only affects the policy, , used both in Equation 2 and for acting in the environment. As we have argued, evaluating under different distortion risk measures is equivalent to changing the sampling distribution for , allowing us to achieve various forms of risk-sensitive policies. We focus on a handful of sampling distributions and their corresponding distortion measures.
本节考察把扭曲风险度量 从恒等映射改成其他形式会有什么影响。这只会改变策略 ,而该策略既用于公式 2,也用于在环境中行动。如前所述,在不同扭曲风险度量下做评估,等价于改变 的抽样分布,从而可以得到各种形式的风险敏感策略。我们集中讨论少数几种抽样分布及其对应的扭曲度量。
The first one is the cumulative probability weighting parameterization proposed in cumulative prospect theory (Tversky & Kahneman, 1992; Gonzalez & Wu, 1999):
第一种是累积前景理论中提出的累积概率加权参数化(Tversky & Kahneman, 1992; Gonzalez & Wu, 1999):
In particular, we use the parameter value found by Wu & Gonzalez (1996) to most closely match human subjects. This choice is interesting as, unlike the others we consider, it is neither globally convex nor concave. For small values of it is locally concave and for larger values of it becomes locally convex. Recall that concavity corresponds to risk-averse and convexity to risk-seeking policies.
具体地,我们采用 Wu & Gonzalez (1996) 发现的、与人类被试最为接近的参数值 。这一选择很有意思:与我们考虑的其他度量不同,它既不是全局凸的,也不是全局凹的。在较小的 处它局部为凹,在较大的 处则变为局部凸。回顾一下,凹对应风险回避策略,凸对应风险寻求策略。
Second, we consider the distortion risk measure proposed by Wang (2000), where and are taken to be the standard Normal cumulative distribution function and its inverse:
第二种是 Wang (2000) 提出的扭曲风险度量,其中 和 分别取标准正态累积分布函数及其逆:
For , this produces risk-averse policies and we include it due to its simple interpretation and ability to switch between risk-averse and risk-seeking distortions.
当 时,它给出风险回避策略。我们纳入这一度量,是因为它解释简单,并且能够在风险回避与风险寻求的扭曲之间切换。
Third, we consider a simple power formula for risk-averse () or risk-seeking () policies:
第三,我们考虑一个简单的幂公式,用于风险回避()或风险寻求()策略:
Finally, we consider conditional value-at-risk (CVaR):
最后,我们考虑条件风险价值(CVaR):
CVaR has been widely studied in and out of reinforcement learning (Chow & Ghavamzadeh, 2014). Its implementation as a modification to the sampling distribution of is particularly simple, as it changes to . Another interesting sampling distribution, not included in our experiments, is denoted and corresponds to sampled by averaging samples from .
CVaR 在强化学习内外都得到了广泛研究(Chow & Ghavamzadeh, 2014)。把它实现为对 抽样分布的修改尤其简单:只需把 改成 。还有一种有意思的抽样分布没有纳入我们的实验,记作 ,对应于把 的 个样本取平均来得到 。
In Figure 3 (right) we give an example of a distribution (Neutral) and how each of these distortion measures affects the implied distribution due to changing the sampling distribution of . and reduce the impact of the tails of the distribution, while Wang and CVaR heavily shift the distribution mass towards the tails, creating a risk-averse or risk-seeking preference. Additionally, while CVaR entirely ignores all values corresponding to , Wang gives these non-zero, but vanishingly small, probability.
图 3(右)给出了一个分布(Neutral)的例子,以及改变 的抽样分布后,上述各扭曲度量如何影响隐含分布。 和 会削弱分布尾部的影响,而 Wang 和 CVaR 则会把大量概率质量推向尾部,从而形成风险回避或风险寻求的偏好。此外,CVaR 完全忽略所有对应 的取值,Wang 则仍给这些取值非零、但小到几乎可以忽略的概率。
By using these sampling distributions we can induce various risk-sensitive policies in IQN. We evaluate these on the same set of six Atari 2600 games previously used. Our algorithm simply changes the policy to maximize the distorted expectations instead of the usual sample mean. Figure 3 (left) shows our results in this experiment, with average scores reported under the usual, risk-neutral, evaluation criterion.
利用这些抽样分布,可以在 IQN 中诱导出各种风险敏感策略。我们在前面用过的同一组六款 Atari 2600 游戏上评估它们。算法上只需把策略改成最大化扭曲期望,而不是通常的样本均值。图 3(左)给出了这一实验的结果,平均分数仍按通常的、风险中性的评估准则报告。
图 3:改变抽样分布,也就是采用各种累积概率加权,所带来的影响。
Intuitively, we expected to see a qualitative effect from risk-sensitive training, e.g. strengthened exploration from a risk-seeking objective. Although we did see qualitative differences, these did not always match our expectations. For two of the games, ASTERIX and ASSAULT, there is a very significant advantage to the risk-averse policies. Although CPW tends to perform almost identically to the standard risk-neutral policy, and the risk-seeking performs as well or worse than risk-neutral, we find that both risk-averse policies improve performance over standard IQN. However, we also observe that the more risk-averse of the two, , suffers some loss in performance on two other games (QBERT and SPACE INVADERS).
直观上,我们预期风险敏感训练会带来定性上的影响,例如风险寻求目标会加强探索。我们的确看到了定性差异,但并不总是符合预期。在 ASTERIX 和 ASSAULT 这两款游戏上,风险回避策略有非常明显的优势。尽管 CPW 的表现往往与标准的风险中性策略几乎相同,风险寻求的 也只是持平或更差,但两种风险回避策略都比标准 IQN 更好。不过我们也看到,两者中更回避风险的 在另外两款游戏(QBERT 和 SPACE INVADERS)上会有一些成绩损失。
Additionally, we note that the risk-seeking policy significantly underperforms the risk-neutral policy on three of the six games. It remains an open question as to exactly why we see improved performance for risk-averse policies. There are many possible explanations for this phenomenon, e.g. that risk-aversion encodes a heuristic to stay alive longer, which in many games is correlated with increased rewards.
此外,风险寻求策略在六款游戏中的三款上明显不如风险中性策略。风险回避策略究竟为何会带来更好的表现,目前仍是一个未决问题。对这一现象可以有许多解释,例如风险回避编码了一种「尽量活得更久」的启发式,而在许多游戏里这与更高的奖励是相关的。
- Full Atari-57 Results
- 完整的 Atari-57 结果
Finally, we evaluate IQN on the full Atari-57 benchmark, comparing with the state-of-the-art performance of Rainbow, a distributional RL agent that combines several advances in deep RL (Hessel et al., 2018), the closely related algorithm QR-DQN (Dabney et al., 2018), prioritized experience replay DQN (Schaul et al., 2016), and the original DQN agent (Mnih et al., 2015). Note that in this section we use the risk-neutral variant of the IQN, that is, the policy of the IQN agent is the regular -greedy policy with respect to the mean of the state-action return distribution.
最后,我们在完整的 Atari-57 基准上评估 IQN,并与以下方法比较:融合了深度强化学习多项进展的分布强化学习智能体 Rainbow 的当时最好成绩(Hessel et al., 2018)、密切相关的算法 QR-DQN(Dabney et al., 2018)、带优先经验回放的 DQN(Schaul et al., 2016),以及最初的 DQN 智能体(Mnih et al., 2015)。注意,本节使用的是 IQN 的风险中性变体,也就是说,IQN 智能体的策略是对状态-动作回报分布的均值做常规 -greedy。
It is important to remember that Rainbow builds upon the distributional RL algorithm C51 (Bellemare et al., 2017), but also includes prioritized experience replay (Schaul et al., 2016), Double DQN (van Hasselt et al., 2016), Dueling Network architecture (Wang et al., 2016), Noisy Networks (Fortunato et al., 2017), and multi-step updates (Sutton, 1988). In particular, besides the distributional update, -step updates and prioritized experience replay were found to have significant impact on the performance of Rainbow. Our other competitive baseline is QR-DQN, which is currently state-of-the-art for agents that do not combine distributional updates, -step updates, and prioritized replay.
需要记住,Rainbow 建立在分布强化学习算法 C51(Bellemare et al., 2017)之上,但还包含优先经验回放(Schaul et al., 2016)、Double DQN(van Hasselt et al., 2016)、对决网络结构(Wang et al., 2016)、Noisy Networks(Fortunato et al., 2017)以及多步更新(Sutton, 1988)。特别是,除了分布化更新外, 步更新和优先经验回放对 Rainbow 的表现也有显著影响。另一条有竞争力的基线是 QR-DQN,它是当时在不把分布化更新、 步更新和优先回放结合在一起的智能体中的最好结果。
Thus, between QR-DQN and the much more complex Rainbow we compare to the two most closely related, and best performing, agents in published work. In particular, we would expect that IQN would benefit from the additional enhancements in Rainbow, just as Rainbow improved significantly over C51.
因此,在 QR-DQN 和复杂得多的 Rainbow 之间,我们比较的是已发表工作里关系最密切、表现也最好的两个智能体。尤其可以预期,IQN 也会从 Rainbow 中那些额外改进里受益,正如 Rainbow 相对 C51 有过显著提升一样。
Figure 4 shows the mean (left) and median (right) human-normalized scores during training over the Atari-57 benchmark. IQN dramatically improves over QR-DQN, which itself improves on many previously published results. At 100 million frames IQN has reached the same level of performance as QR-DQN at 200 million frames. Table 1 gives a comparison between the same methods in terms of their best, human-normalized, scores per game under the 30 random no-op start condition. These are averages over the given number of seeds. Additionally, using human-starts, IQN achieves 162% median human-normalized score, whereas Rainbow reaches 153% (Hessel et al., 2018), see Table 2.
图 4 给出了在 Atari-57 基准上训练过程中的人类归一化分数均值(左)和中位数(右)。IQN 相对 QR-DQN 有大幅度提升,而 QR-DQN 本身已经超过许多先前发表的结果。在 1 亿帧时,IQN 已经达到 QR-DQN 在 2 亿帧时的水平。表 1 按同样这些方法,比较了在 30 次随机无操作开局条件下每款游戏的最佳人类归一化分数。这些分数是对给定个数的随机种子取平均。此外,在人类开局条件下,IQN 的中位数人类归一化分数是 162%,而 Rainbow 是 153%(Hessel et al., 2018),见表 2。
Table 1. Mean and median of scores across 57 Atari 2600 games, measured as percentages of human baseline (Nair et al., 2015). Scores are averages over number of seeds.
| Mean | Median | Human Gap | Seeds | |
|---|---|---|---|---|
| DQN | 228% | 79% | 0.334 | 1 |
| PRIOR. | 434% | 124% | 0.178 | 1 |
| C51 | 701% | 178% | 0.152 | 1 |
| RAINBOW | 1189% | 230% | 0.144 | 2 |
| QR-DQN | 864% | 193% | 0.165 | 3 |
| IQN | 1019% | 218% | 0.141 | 5 |
表 1:57 款 Atari 2600 游戏上分数的均值与中位数,以人类基线的百分比计(Nair et al., 2015)。分数是对相应个数的随机种子取平均。
| 均值 | 中位数 | 人类差距 | 种子数 | |
|---|---|---|---|---|
| DQN | 228% | 79% | 0.334 | 1 |
| PRIOR. | 434% | 124% | 0.178 | 1 |
| C51 | 701% | 178% | 0.152 | 1 |
| RAINBOW | 1189% | 230% | 0.144 | 2 |
| QR-DQN | 864% | 193% | 0.165 | 3 |
| IQN | 1019% | 218% | 0.141 | 5 |
Table 2. Median human-normalized scores for human-starts.
| Human-starts (median) | DQN | PRIOR. | A3C | C51 | RAINBOW | IQN |
|---|---|---|---|---|---|---|
| 68% | 128% | 116% | 125% | 153% | 162% |
表 2:人类开局条件下的中位数人类归一化分数。
| 人类开局(中位数) | DQN | PRIOR. | A3C | C51 | RAINBOW | IQN |
|---|---|---|---|---|---|---|
| 68% | 128% | 116% | 125% | 153% | 162% |
图 4:IQN 及其他若干算法在 Atari-57 上的人类归一化均值(左)和中位数(右)。各随机种子以轨迹画出,其中 IQN 对 5 个种子取平均,QR-DQN 对 3 个,Rainbow 对 2 个。
Finally, we took a closer look at the games in which each algorithm continues to underperform humans, and computed, on average, how far below human-level they perform. We refer to this value as the human-gap metric and give results in Table 1. Interestingly, C51 outperforms QR-DQN in this metric, and IQN outperforms all others. This shows that the remaining gap between Rainbow and IQN is entirely from games on which both algorithms are already super-human. The games where the most progress in RL is needed happen to be the games where IQN shows the greatest improvement over QR-DQN and Rainbow.
最后,我们更仔细地看了各算法仍然不如人类的那些游戏,并计算它们平均比人类水平低多少。我们把这个量称作人类差距(human-gap)指标,结果见表 1。有意思的是,在这一指标上 C51 超过了 QR-DQN,而 IQN 超过了所有其他方法。这说明 Rainbow 与 IQN 之间剩下的差距,完全来自两种算法都已经超过人类的那些游戏。强化学习最需要取得进展的那些游戏,恰好也是 IQN 相对 QR-DQN 和 Rainbow 提升最大的游戏。
Details of how this is computed can be found in the Appendix.
具体计算方式见附录。
Thanks to Joseph Modayil for proposing this metric.
感谢 Joseph Modayil 提出这一指标。
- Discussion and Conclusions
- 讨论与结论
We have proposed a generalization of recent work based around using quantile regression to learn the distribution over returns of the current policy. Our generalization leads to a simple change to the DQN agent to enable distributional RL, the natural integration of risk-sensitive policies, and significantly improved performance over existing methods. The IQN algorithm provides, for the first time, a fully integrated distributional RL agent without prior assumptions on the parameterization of the return distribution.
我们把近期用分位数回归来学习当前策略回报分布的工作做了推广。这一推广只需对 DQN 智能体做简单改动,就能实现分布强化学习,并能自然地纳入风险敏感策略,同时相对已有方法显著提高表现。IQN 算法第一次给出了一种完全整合的分布强化学习智能体,并且不对回报分布的参数化做事先假设。
IQN can be trained with as little as a single sample from each state-action value distribution, or as many as computational limits allow to improve the algorithm’s data efficiency. Furthermore, IQN allows us to expand the class of control policies to a large class of risk-sensitive policies connected to distortion risk measures. Finally, we show substantial gains on the Atari-57 benchmark over QR-DQN, and even halving the distance between QR-DQN and Rainbow.
IQN 的训练可以从每个状态-动作价值分布中只取一个样本,也可以在计算条件允许的范围内取更多样本,以提高算法的数据效率。此外,IQN 使我们能够把控制策略类扩大到与扭曲风险度量相联系的一大类风险敏感策略。最后,我们在 Atari-57 基准上相对 QR-DQN 取得了可观提升,甚至把 QR-DQN 与 Rainbow 之间的差距缩小了一半。
Despite the significant empirical successes in this paper there are many areas in need of additional theoretical analysis. We highlight a few particularly relevant open questions we were unable to address in the present work. First, sample-based convergence results have been recently shown for a class of categorical distributional RL algorithms (Rowland et al., 2018). Could existing sample-based RL convergence results be extended to the QR-based algorithms?
尽管本文有不少显著的经验成功,仍有许多地方需要进一步的理论分析。我们标出几个与本文尤其相关、但未能在此解决的未决问题。第一,最近已有工作为一类分类式分布强化学习算法给出了基于样本的收敛结果(Rowland et al., 2018)。现有的基于样本的强化学习收敛结果,能否推广到基于分位数回归的算法?
Second, can the contraction mapping results for a fixed grid of quantiles given by Dabney et al. (2018) be extended to the more general class of approximate quantile functions studied in this work? Finally, and particularly salient to our experiments with distortion risk measures, theoretical guarantees for risk-sensitive RL have been building over recent years, but have been largely limited to special cases and restricted classes of risk-sensitive policies. Can the convergence of the distribution of returns under the Bellman operator be leveraged to show convergence to a fixed-point in distorted expectations? In particular, can the control results of Bellemare et al. (2017) be expanded to cover some class of risk-sensitive policies?
第二,Dabney et al. (2018) 对固定分位数网格给出的压缩映射结果,能否推广到本文所研究的、更一般的近似分位数函数类?最后,与我们关于扭曲风险度量的实验尤其相关的是:近年来风险敏感强化学习的理论保证在不断积累,但大多仍限于特殊情形和受限的风险敏感策略类。能否借助回报分布在贝尔曼算子下的收敛,来证明扭曲期望会收敛到某个不动点?特别是,Bellemare et al. (2017) 的控制结果能否扩展到某一类风险敏感策略?
There remain many intriguing directions for future research into distributional RL, even on purely empirical fronts. Hessel et al. (2018) recently showed that distributional RL agents can be significantly improved, when combined with other techniques. Creating a Rainbow-IQN agent could yield even greater improvements on Atari-57. We also recall the surprisingly rich return distributions found by Barth-Maron et al. (2018), and hypothesize that the continuous control setting may be a particularly fruitful area for the application of distributional RL in general, and IQN in particular.
即使只看经验方面,分布强化学习仍有许多值得探索的方向。Hessel et al. (2018) 最近表明,分布强化学习智能体与其他技术结合后还能再明显提升。若构造一个 Rainbow-IQN 智能体,有可能在 Atari-57 上带来更大改进。我们也回想起 Barth-Maron et al. (2018) 所发现的、相当丰富的回报分布,并猜想连续控制设定可能尤其适合应用分布强化学习,而对 IQN 而言更是如此。
References
参考文献
Allais, M. Allais paradox. In Utility and Probability, pp. 3–9. Springer, 1990.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Azar, M. G., Munos, R., and Kappen, H. J. On the sample complexity of reinforcement learning with a generative model. In Proceedings of the International Conference on Machine Learning (ICML), 2012.
Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., TB, D., Muldal, A., Heess, N., and Lillicrap, T. Distributional policy gradients. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The Arcade Learning Environment: an evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013.
Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Bellman, R. E. Dynamic Programming. Princeton University Press, Princeton, NJ, 1957.
Bousquet, O., Gelly, S., Tolstikhin, I., Simon-Gabriel, C.-J., and Schoelkopf, B. From optimal transport to generative modeling: the vegan cookbook. arXiv preprint arXiv:1705.07642, 2017.
Chow, Y. and Ghavamzadeh, M. Algorithms for CVaR optimization in MDPs. In Advances in Neural Information Processing Systems, pp. 3509–3517, 2014.
Dabney, W., Rowland, M., Bellemare, M. G., and Munos, R. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Dhaene, J., Kukush, A., Linders, D., and Tang, Q. Remarks on quantiles and distortion risk measures. European Actuarial Journal, 2(2):319–328, 2012.
Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295, 2017.
Geist, M. and Pietquin, O. Kalman temporal differences. Journal of Artificial Intelligence Research, 39:483–532, 2010.
Gonzalez, R. and Wu, G. On the shape of the probability weighting function. Cognitive Psychology, 38(1):129–166, 1999.
Gruslys, A., Dabney, W., Azar, M. G., Piot, B., Bellemare, M. G., and Munos, R. The Reactor: a fast and sample-efficient actor-critic agent for reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Howard, R. A. and Matheson, J. E. Risk-sensitive markov decision processes. Management Science, 18(7):356–369, 1972.
Huber, P. J. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73–101, 1964.
Jaquette, S. C. Markov decision processes with a new optimality criterion: discrete time. The Annals of Statistics, 1(3):496–505, 1973.
Koenker, R. Quantile Regression. Cambridge University Press, 2005.
Lattimore, T. and Hutter, M. PAC bounds for discounted MDPs. In International Conference on Algorithmic Learning Theory, pp. 320–334. Springer, 2012.
Maddison, C. J., Lawson, D., Tucker, G., Heess, N., Doucet, A., Mnih, A., and Teh, Y. W. Particle value functions. arXiv preprint arXiv:1703.05820, 2017.
Majumdar, A. and Pavone, M. How should a robot assess risk? Towards an axiomatic theory of risk in robotics. arXiv preprint arXiv:1710.11040, 2017.
Marcus, S. I., Fernández-Gaucherand, E., Hernández-Hernandez, D., Coraluppi, S., and Fard, P. Risk sensitive markov decision processes. In Systems and Control in the Twenty-First Century, pp. 263–279. Springer, 1997.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
Moerland, T. M., Broekens, J., and Jonker, C. M. Efficient exploration with double uncertain value networks. arXiv preprint arXiv:1711.10789, 2017.
Morimura, T., Hachiya, H., Sugiyama, M., Tanaka, T., and Kashima, H. Parametric return density estimation for reinforcement learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2010a.
Morimura, T., Sugiyama, M., Kashima, H., Hachiya, H., and Tanaka, T. Nonparametric return distribution approximation for reinforcement learning. In Proceedings of the 27th International Conference on Machine Learning (ICML), pp. 799–806, 2010b.
Müller, A. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429–443, 1997.
Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., De Maria, A., Panneershelvam, V., Suleyman, M., Beattie, C., and Petersen, S. e. a. Massively parallel methods for deep reinforcement learning. In ICML Workshop on Deep Learning, 2015.
Osband, I., Russo, D., and Van Roy, B. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems, pp. 3003–3011, 2013.
Puterman, M. L. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
Rowland, M., Bellemare, M. G., Dabney, W., Munos, R., and Teh, Y. W. An analysis of categorical distributional reinforcement learning. In AISTATS, 2018.
Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In International Conference on Machine Learning, pp. 1312–1320, 2015.
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
Sobel, M. J. The variance of discounted markov decision processes. Journal of Applied Probability, 19(04):794–802, 1982.
Sutton, R. S. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9–44, 1988.
Tolstikhin, I., Bousquet, O., Gelly, S., and Schoelkopf, B. Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558, 2017.
Tversky, A. and Kahneman, D. Advances in prospect theory: cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5(4):297–323, 1992.
van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
von Neumann, J. and Morgenstern, O. Theory of Games and Economic Behavior. Princeton University Press, 1947.
Wang, S. Premium calculation by transforming the layer premium density. ASTIN Bulletin: The Journal of the IAA, 26(1):71–92, 1996.
Wang, S. S. A class of distortion operators for pricing financial and insurance risks. Journal of Risk and Insurance, pp. 15–36, 2000.
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., and de Freitas, N. Dueling network architectures for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2016.
Watkins, C. J. C. H. Learning from delayed rewards. PhD thesis, King’s College, Cambridge, 1989.
White, D. J. Mean, variance, and probabilistic criteria in finite markov decision processes: a review. Journal of Optimization Theory and Applications, 56(1):1–29, 1988.
Wu, G. and Gonzalez, R. Curvature of the probability weighting function. Management Science, 42(12):1676–1690, 1996.
Yaari, M. E. The dual theory of choice under risk. Econometrica: Journal of the Econometric Society, pp. 95–115, 1987.
Yu, K.-T., Bauza, M., Fazeli, N., and Rodriguez, A. More than a million ways to be pushed. a high-fidelity experimental dataset of planar pushing. In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on, pp. 30–37. IEEE, 2016.



