A Distributional Perspective on Reinforcement Learning

强化学习的分布视角

Marc G. Bellemare Affiliation:  DeepMind, London, UK Correspondence to: bellemare@google.com    Will Dabney Affiliation:  DeepMind, London, UK    Rémi Munos Affiliation:  DeepMind, London, UK

Abstract

摘要

In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforcement learning which models the expectation of this return, or value. Although there is an established body of literature studying the value distribution, thus far it has always been used for a specific purpose such as implementing risk-aware behaviour. We begin with theoretical results in both the policy evaluation and control settings, exposing a significant distributional instability in the latter. We then use the distributional perspective to design a new algorithm which applies Bellman’s equation to the learning of approximate value distributions. We evaluate our algorithm using the suite of games from the Arcade Learning Environment. We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning. Finally, we combine theoretical and empirical evidence to highlight the ways in which the value distribution impacts learning in the approximate setting.

本文主张价值分布具有根本重要性：它是强化学习智能体所收到的随机回报的分布。这与强化学习中的常见做法形成对照——后者建模的是这一回报的期望，也就是价值。尽管已有相当多文献研究价值分布，但迄今为止它总是被用于某个具体目的，例如实现风险敏感行为。我们从策略评估和控制两种设定下的理论结果入手，并揭示控制设定中一种显著的分布不稳定性。随后，我们利用分布视角设计了一种新算法，把贝尔曼方程用于学习近似的价值分布。我们在 Arcade Learning Environment 的游戏套件上评估该算法，既得到了当时最好的成绩，也得到了一些轶事性的证据，说明价值分布在近似强化学习中的重要性。最后，我们把理论与经验证据结合起来，说明价值分布如何影响近似设定下的学习。

1 Introduction

1 引言

One of the major tenets of reinforcement learning states that, when not otherwise constrained in its behaviour, an agent should aim to maximize its expected utility $Q$, or value (Sutton & Barto, 1998). Bellman’s equation succintly describes this value in terms of the expected reward and expected outcome of the random transition $(x,a)\hskip-1.00006pt\to\hskip-1.00006pt(X^{\prime},A^{\prime})$:

强化学习的一条主要信条是：当智能体的行为不受其他约束时，它应当致力于最大化其期望效用 $Q$，也就是价值（Sutton & Barto, 1998）。贝尔曼方程把这一价值简明地写成期望奖励，以及随机转移 $(x,a)\hskip-1.00006pt\to\hskip-1.00006pt(X^{\prime},A^{\prime})$ 的期望结果：

$$Q(x,a)=\expect R(x,a)+\gamma\expect Q(X^{\prime},A^{\prime}).$$

In this paper, we aim to go beyond the notion of value and argue in favour of a distributional perspective on reinforcement learning. Specifically, the main object of our study is the random return $Z$ whose expectation is the value $Q$. This random return is also described by a recursive equation, but one of a distributional nature:

在本文中，我们希望超越价值这一概念，并为强化学习的一种分布视角作辩护。具体来说，我们研究的主要对象是随机回报 $Z$，其期望就是价值 $Q$。这一随机回报同样由一个递归方程描述，不过是分布意义上的递归：

$$Z(x,a)\overset{D}{=}R(x,a)+\gamma Z(X^{\prime},A^{\prime}).$$

The distributional Bellman equation states that the distribution of $Z$ is characterized by the interaction of three random variables: the reward $R$, the next state-action $(X^{\prime},A^{\prime})$, and its random return $Z(X^{\prime},A^{\prime})$. By analogy with the well-known case, we call this quantity the value distribution.

分布贝尔曼方程表明，$Z$ 的分布由三个随机变量的相互作用所刻画：奖励 $R$、下一状态-动作 $(X^{\prime},A^{\prime})$，以及它的随机回报 $Z(X^{\prime},A^{\prime})$。类比熟知的情形，我们把这一量叫做价值分布。

Although the distributional perspective is almost as old as Bellman’s equation itself (Jaquette, 1973; Sobel, 1982; White, 1988), in reinforcement learning it has thus far been subordinated to specific purposes: to model parametric uncertainty (Dearden et al., 1998), to design risk-sensitive algorithms (Morimura et al., 2010b; Morimura et al., 2010a), or for theoretical analysis (Azar et al., 2012; Lattimore & Hutter, 2012). By contrast, we believe the value distribution has a central role to play in reinforcement learning.

尽管分布视角几乎与贝尔曼方程本身一样古老（Jaquette, 1973; Sobel, 1982; White, 1988），但在强化学习中，它迄今为止一直从属于某些特定目的：建模参数不确定性（Dearden et al., 1998），设计风险敏感算法（Morimura et al., 2010b; Morimura et al., 2010a），或用于理论分析（Azar et al., 2012; Lattimore & Hutter, 2012）。相比之下，我们相信价值分布在强化学习中应占据中心位置。

Contraction of the policy evaluation Bellman operator. Basing ourselves on results by Rösler (1992) we show that, for a fixed policy, the Bellman operator over value distributions is a contraction in a maximal form of the Wasserstein (also called Kantorovich or Mallows) metric. Our particular choice of metric matters: the same operator is not a contraction in total variation, Kullback-Leibler divergence, or Kolmogorov distance.

策略评估贝尔曼算子的压缩性。 基于 Rösler（1992）的结果，我们证明：对固定策略，价值分布上的贝尔曼算子在一种极大形式的 Wasserstein 度量（也称 Kantorovich 或 Mallows 度量）下是压缩映射。我们对度量的这一特定选择是要紧的：同一个算子在全变差、Kullback-Leibler 散度或 Kolmogorov 距离下并不是压缩映射。

Instability in the control setting. We will demonstrate an instability in the distributional version of Bellman’s optimality equation, in contrast to the policy evaluation case. Specifically, although the optimality operator is a contraction in expected value (matching the usual optimality result), it is not a contraction in any metric over distributions. These results provide evidence in favour of learning algorithms that model the effects of nonstationary policies.

控制设定中的不稳定性。 我们将说明，与策略评估情形不同，贝尔曼最优性方程的分布版本存在一种不稳定性。具体来说，尽管最优性算子在期望价值上是压缩映射（与通常的最优性结果一致），它在任何分布度量下都不是压缩映射。这些结果为那些会建模非平稳策略效应的学习算法提供了依据。

Better approximations. From an algorithmic standpoint, there are many benefits to learning an approximate distribution rather than its approximate expectation. The distributional Bellman operator preserves multimodality in value distributions, which we believe leads to more stable learning. Approximating the full distribution also mitigates the effects of learning from a nonstationary policy. As a whole, we argue that this approach makes approximate reinforcement learning significantly better behaved.

更好的近似。 从算法角度看，学习一个近似分布，而不是学习其近似期望，有许多好处。分布贝尔曼算子会保留价值分布中的多峰性，我们认为这会带来更稳定的学习。对完整分布做近似，也能减轻从非平稳策略中学习所带来的影响。总的来说，我们主张这一方法会让近似强化学习表现得显著更好。

We will illustrate the practical benefits of the distributional perspective in the context of the Arcade Learning Environment (Bellemare et al., 2013). By modelling the value distribution within a DQN agent (Mnih et al., 2015), we obtain considerably increased performance across the gamut of benchmark Atari 2600 games, and in fact achieve state-of-the-art performance on a number of games. Our results echo those of Veness et al. (2015), who obtained extremely fast learning by predicting Monte Carlo returns.

我们将在 Arcade Learning Environment（Bellemare et al., 2013）的语境中说明分布视角的实际好处。通过在 DQN 智能体（Mnih et al., 2015）内部对价值分布建模，我们在一系列基准 Atari 2600 游戏上获得了相当可观的性能提升，并且事实上在不少游戏上达到了当时最好的成绩。我们的结果与 Veness et al.（2015）相呼应，他们通过预测蒙特卡洛回报获得了极快的学习。

From a supervised learning perspective, learning the full value distribution might seem obvious: why restrict ourselves to the mean? The main distinction, of course, is that in our setting there are no given targets. Instead, we use Bellman’s equation to make the learning process tractable; we must, as Sutton & Barto (1998) put it, “learn a guess from a guess”. It is our belief that this guesswork ultimately carries more benefits than costs.

从监督学习的角度看，学习完整的价值分布似乎是理所当然的：为什么要把自己限制在均值上？当然，主要区别在于，我们的设定里并没有给定的目标。相反，我们用贝尔曼方程让学习过程变得可行；用 Sutton & Barto（1998）的话说，我们必须“从一个猜测中学习另一个猜测”。我们相信，这种猜测最终带来的好处多于代价。

2 Setting

2 设定

We consider an agent interacting with an environment in the standard fashion: at each step, the agent selects an action based on its current state, to which the environment responds with a reward and the next state. We model this interaction as a time-homogeneous Markov Decision Process $(\mathcal{X},\mathcal{A},R,P,\gamma)$. As usual, $\mathcal{X}$ and $\mathcal{A}$ are respectively the state and action spaces, $P$ is the transition kernel $P(\cdot\,|\,x,a)$, $\gamma\in[0,1]$ is the discount factor, and $R$ is the reward function, which in this work we explicitly treat as a random variable. A stationary policy $\pi$ maps each state $x\in\mathcal{X}$ to a probability distribution over the action space $\mathcal{A}$.

我们按标准方式考虑智能体与环境的交互：在每一步，智能体根据当前状态选择一个动作，环境则以一个奖励和下一状态作为回应。我们把这一交互建模为时间齐次的马尔可夫决策过程 $(\mathcal{X},\mathcal{A},R,P,\gamma)$。照例，$\mathcal{X}$ 和 $\mathcal{A}$ 分别是状态空间和动作空间，$P$ 是转移核 $P(\cdot\,|\,x,a)$，$\gamma\in[0,1]$ 是折扣因子，$R$ 是奖励函数，在本文中我们明确把它当作一个随机变量。平稳策略 $\pi$ 把每个状态 $x\in\mathcal{X}$ 映射为动作空间 $\mathcal{A}$ 上的一个概率分布。

2.1 Bellman’s Equations

2.1 贝尔曼方程

The return $Z^{\pi}$ is the sum of discounted rewards along the agent’s trajectory of interactions with the environment. The value function $Q^{\pi}$ of a policy $\pi$ describes the expected return from taking action $a\in\mathcal{A}$ from state $x\in\mathcal{X}$, then acting according to $\pi$:

回报 $Z^{\pi}$ 是智能体与环境交互轨迹上折扣奖励的总和。策略 $\pi$ 的价值函数 $Q^{\pi}$ 描述的是：从状态 $x\in\mathcal{X}$ 采取动作 $a\in\mathcal{A}$，然后按 $\pi$ 行动所得到的期望回报：

$$\displaystyle Q^{\pi}(x,a) \\ \displaystyle:=\expect Z^{\pi}(x,a)=\expect\left[\sum_{t=0}^{\infty}\gamma^{t}R(x_{t},a_{t})\right], \\ \displaystyle x_{t}\sim P(\cdot \\ \displaystyle\,|\,x_{t-1},a_{t-1}),a_{t}\sim\pi(\cdot\,|\,x_{t}),x_{0}=x,a_{0}=a. \tag{1}$$

Fundamental to reinforcement learning is the use of Bellman’s equation (Bellman, 1957) to describe the value function:

强化学习的一个基本做法，是用贝尔曼方程（Bellman, 1957）来描述价值函数：

$$Q^{\pi}(x,a)=\expect R(x,a)+\gamma\expect_{P,\pi}Q^{\pi}(x^{\prime},a^{\prime}).$$

In reinforcement learning we are typically interested in acting so as to maximize the return. The most common approach for doing so involves the optimality equation

在强化学习中，我们通常关心如何行动以最大化回报。最常见的做法涉及最优性方程

Q^{*}(x,a)=\expect R(x,a)+\gamma\expect\nolimits_{P}\max_{a^{\prime}\in\mathcal{A}}Q^{*}(x^{\prime},a^{\prime}).

This equation has a unique fixed point $Q^{*}$, the optimal value function, corresponding to the set of optimal policies $\Pi^{*}$ ($\pi^{*}$ is optimal if $\expect_{a\sim\pi^{*}}Q^{*}(x,a)=\max_{a}Q^{*}(x,a)$).

这一方程有唯一不动点 $Q^{*}$，即最优价值函数，它对应于最优策略集合 $\Pi^{*}$（若 $\expect_{a\sim\pi^{*}}Q^{*}(x,a)=\max_{a}Q^{*}(x,a)$，则 $\pi^{*}$ 是最优的）。

We view value functions as vectors in $\mathbb{R}^{\mathcal{X}\times\mathcal{A}}$, and the expected reward function as one such vector. In this context, the Bellman operator $\mathcal{T}^{\pi}$ and optimality operator $\mathcal{T}$ are

我们把价值函数看作 $\mathbb{R}^{\mathcal{X}\times\mathcal{A}}$ 中的向量，并把期望奖励函数也看作这样的一个向量。在这一语境下，贝尔曼算子 $\mathcal{T}^{\pi}$ 和最优性算子 $\mathcal{T}$ 是

\displaystyle\mathcal{T}^{\pi}Q(x,a) \\ \displaystyle:=\expect R(x,a)+\gamma\expect_{P,\pi}Q(x^{\prime},a^{\prime}) \\ \displaystyle\mathcal{T}Q(x,a) \\ \displaystyle:=\expect R(x,a)+\gamma\expect\nolimits_{P}\max_{a^{\prime}\in\mathcal{A}}Q(x^{\prime},a^{\prime}). \tag{2}

These operators are useful as they describe the expected behaviour of popular learning algorithms such as SARSA and Q-Learning. In particular they are both contraction mappings, and their repeated application to some initial $Q_{0}$ converges exponentially to $Q^{\pi}$ or $Q^{*}$, respectively (Bertsekas & Tsitsiklis, 1996).

这些算子是有用的，因为它们描述了 SARSA 和 Q 学习等流行学习算法的期望行为。特别地，它们都是压缩映射，反复作用于某个初始的 $Q_{0}$，会指数地收敛到 $Q^{\pi}$ 或 $Q^{*}$（Bertsekas & Tsitsiklis, 1996）。

3 The Distributional Bellman Operators

3 分布贝尔曼算子

Figure 1: A distributional Bellman operator with a deterministic reward function: (a) Next state distribution under policy $\pi$, (b) Discounting shrinks the distribution towards 0, (c) The reward shifts it, and (d) Projection step (Section 4).

图 1：带确定性奖励函数的分布贝尔曼算子：（a）策略 $\pi$ 下的下一状态分布，（b）折扣把分布向 0 收缩，（c）奖励使它平移，以及（d）投影步骤（第 4 节）。

In this paper we take away the expectations inside Bellman’s equations and consider instead the full distribution of the random variable $Z^{\pi}$. From here on, we will view $Z^{\pi}$ as a mapping from state-action pairs to distributions over returns, and call it the value distribution.

在本文中，我们去掉贝尔曼方程内部的期望，转而考虑随机变量 $Z^{\pi}$ 的完整分布。从此以后，我们将把 $Z^{\pi}$ 看作从状态-动作对到回报分布的映射，并称之为价值分布。

Our first aim is to gain an understanding of the theoretical behaviour of the distributional analogues of the Bellman operators, in particular in the less well-understood control setting. The reader strictly interested in the algorithmic contribution may choose to skip this section.

我们的第一个目标，是理解分布形式的贝尔曼算子在理论上的行为，尤其是在理解还较少的控制设定中。只关心算法贡献的读者可以选择跳过本节。

3.1 Distributional Equations

3.1 分布式方程

It will sometimes be convenient to make use of the probability space $(\Omega,\mathcal{F},\Pr)$. The reader unfamiliar with measure theory may think of $\Omega$ as the space of all possible outcomes of an experiment (Billingsley, 1995). We will write $\|\mathbf{u}\|_{p}$ to denote the $L_{p}$ norm of a vector $\mathbf{u}\in\mathbb{R}^{\mathcal{X}}$ for $1\leq p\leq\infty$; the same applies to vectors in $\mathbb{R}^{\mathcal{X}\times\mathcal{A}}$. The $L_{p}$ norm of a random vector $U:\Omega\to\mathbb{R}^{\mathcal{X}}$ (or $\mathbb{R}^{\mathcal{X}\times\mathcal{A}}$) is then $\|U\|_{p}:=\left[\expect\left[\|U(\omega)\|_{p}^{p}\right]\right]^{1/p}$, and for $p=\infty$ we have $\|U\|_{\infty}=\textrm{ess}\sup{\|U(\omega)\|_{\infty}}$ (we will omit the dependency on $\omega\in\Omega$ whenever unambiguous). We will denote the c.d.f. of a random variable $U$ by $F_{U}(y):=\Pr\{U\leq y\}$, and its inverse c.d.f. by $F^{-1}_{U}(q):=\inf\{y:F_{U}(y)\geq q\}$.

有时使用概率空间 $(\Omega,\mathcal{F},\Pr)$ 会比较方便。不熟悉测度论的读者可以把 $\Omega$ 理解为一次试验所有可能结果的空间（Billingsley, 1995）。我们将用 $\|\mathbf{u}\|_{p}$ 表示向量 $\mathbf{u}\in\mathbb{R}^{\mathcal{X}}$ 的 $L_{p}$ 范数，其中 $1\leq p\leq\infty$；这对 $\mathbb{R}^{\mathcal{X}\times\mathcal{A}}$ 中的向量同样适用。随机向量 $U:\Omega\to\mathbb{R}^{\mathcal{X}}$（或 $\mathbb{R}^{\mathcal{X}\times\mathcal{A}}$）的 $L_{p}$ 范数则为 $\|U\|_{p}:=\left[\expect\left[\|U(\omega)\|_{p}^{p}\right]\right]^{1/p}$；当 $p=\infty$ 时，有 $\|U\|_{\infty}=\textrm{ess}\sup{\|U(\omega)\|_{\infty}}$（只要不会引起歧义，我们将省略对 $\omega\in\Omega$ 的依赖）。我们将把随机变量 $U$ 的累积分布函数记为 $F_{U}(y):=\Pr\{U\leq y\}$，并把它的逆累积分布函数记为 $F^{-1}_{U}(q):=\inf\{y:F_{U}(y)\geq q\}$。

A distributional equation $U\overset{D}{:=}V$ indicates that the random variable $U$ is distributed according to the same law as $V$. Without loss of generality, the reader can understand the two sides of a distributional equation as relating the distributions of two independent random variables. Distributional equations have been used in reinforcement learning by Engel et al. (2005); Morimura et al. (2010a) among others, and in operations research by White (1988).

分布式方程 $U\overset{D}{:=}V$ 表示随机变量 $U$ 与 $V$ 服从同一规律。在不失一般性的前提下，读者可以把分布式方程的两边理解为两个独立随机变量的分布之间的关系。分布式方程已被 Engel et al.（2005）、Morimura et al.（2010a）等用于强化学习，也被 White（1988）用于运筹学。

3.2 The Wasserstein Metric

3.2 Wasserstein 度量

The main tool for our analysis is the Wasserstein metric $d_{p}$ between cumulative distribution functions (see e.g. Bickel & Freedman, 1981, where it is called the Mallows metric). For $F$, $G$ two c.d.fs over the reals, it is defined as

我们分析的主要工具是累积分布函数之间的 Wasserstein 度量 $d_{p}$（例如见 Bickel & Freedman, 1981，他们称之为 Mallows 度量）。对实数上的两个累积分布函数 $F$、$G$，它定义为

$$d_{p}(F,G):=\inf_{U,V}\|U-V\|_{p},$$

where the infimum is taken over all pairs of random variables $(U,V)$ with respective cumulative distributions $F$ and $G$. The infimum is attained by the inverse c.d.f. transform of a random variable $\mathcal{U}$ uniformly distributed on $[0,1]$:

其中下确界取遍所有分别具有累积分布 $F$ 和 $G$ 的随机变量对 $(U,V)$。该下确界由均匀分布在 $[0,1]$ 上的随机变量 $\mathcal{U}$ 的逆累积分布变换达到：

$$d_{p}(F,G)=\|F^{-1}(\mathcal{U})-G^{-1}(\mathcal{U})\|_{p}.$$

For $p<\infty$ this is more explicitly written as

当 $p<\infty$ 时，这可以更显式地写成

$$d_{p}(F,G)=\left(\int_{0}^{1}\big|F^{-1}(u)-G^{-1}(u)\big|^{p}du\right)^{1/p}.$$

Given two random variables $U$, $V$ with c.d.fs $F_{U}$, $F_{V}$, we will write $d_{p}(U,V):=d_{p}(F_{U},F_{V})$. We will find it convenient to conflate the random variables under consideration with their versions under the $\inf$, writing

给定两个分别具有累积分布 $F_{U}$、$F_{V}$ 的随机变量 $U$、$V$，我们将写 $d_{p}(U,V):=d_{p}(F_{U},F_{V})$。为方便起见，我们会把所考虑的随机变量与它们在 $\inf$ 下的版本混为一谈，写成

$$d_{p}(U,V)=\inf_{U,V}\|U-V\|_{p}.$$

whenever unambiguous; we believe the greater legibility justifies the technical inaccuracy. Finally, we extend this metric to vectors of random variables, such as value distributions, using the corresponding $L_{p}$ norm.

只要不会引起歧义；我们相信更高的可读性足以弥补这一技术上的不严谨。最后，我们用相应的 $L_{p}$ 范数，把这一度量推广到随机变量的向量上，例如价值分布。

Consider a scalar $a$ and a random variable $A$ independent of $U,V$. The metric $d_{p}$ has the following properties:

考虑标量 $a$ 以及与 $U,V$ 独立的随机变量 $A$。度量 $d_{p}$ 具有下列性质：

$$\displaystyle d_{p}(aU,aV) \\ \displaystyle\leq|a|d_{p}(U,V) \\ \displaystyle d_{p}(A+U,A+V) \\ \displaystyle\leq d_{p}(U,V) \\ \displaystyle d_{p}(AU,AV) \\ \displaystyle\leq\|A\|_{p}d_{p}(U,V). \tag{P1}$$

We will need the following additional property, which makes no independence assumptions on its variables. Its proof, and that of later results, is given in the appendix.

我们还需要下面这条额外的性质，它对其中的变量不做独立性假设。它的证明以及后面结果的证明，都放在附录中。

Lemma 1(Partition lemma).

Let $A_{1},A_{2},\dots$ be a set of random variables describing a partition of $\Omega$, i.e. $A_{i}(\omega)\in\{0,1\}$ and for any $\omega$ there is exactly one $A_{i}$ with $A_{i}(\omega)=1$. Let $U,V$ be two random variables. Then

引理 1（划分引理）。

设 $A_{1},A_{2},\dots$ 是描述 $\Omega$ 的一个划分的一组随机变量，也就是 $A_{i}(\omega)\in\{0,1\}$，并且对任意 $\omega$ 恰好有一个 $A_{i}$ 满足 $A_{i}(\omega)=1$。设 $U,V$ 是两个随机变量。则

$$d_{p}\big(U,V\big)\leq\sum\nolimits_{i}d_{p}(A_{i}U,A_{i}V).$$

Let $\mathcal{Z}$ denote the space of value distributions with bounded moments. For two value distributions $Z_{1},Z_{2}\in\mathcal{Z}$ we will make use of a maximal form of the Wasserstein metric:

设 $\mathcal{Z}$ 表示矩有界的价值分布空间。对两个价值分布 $Z_{1},Z_{2}\in\mathcal{Z}$，我们将使用一种极大形式的 Wasserstein 度量：

$$\bar{d}_{p}(Z_{1},Z_{2}):=\sup_{x,a}d_{p}(Z_{1}(x,a),Z_{2}(x,a)).$$

We will use $\bar{d}_{p}$ to establish the convergence of the distributional Bellman operators.

我们将用 $\bar{d}_{p}$ 来建立分布贝尔曼算子的收敛性。

Lemma 2.

$\bar{d}_{p}$is a metric over value distributions.

引理 2。

$\bar{d}_{p}$是价值分布上的一个度量。

3.3 Policy Evaluation

3.3 策略评估

In the policy evaluation setting (Sutton & Barto, 1998) we are interested in the value function $V^{\pi}$ associated with a given policy $\pi$. The analogue here is the value distribution $Z^{\pi}$. In this section we characterize $Z^{\pi}$ and study the behaviour of the policy evaluation operator $\mathcal{T}^{\pi}$. We emphasize that $Z^{\pi}$ describes the intrinsic randomness of the agent’s interactions with its environment, rather than some measure of uncertainty about the environment itself.

在策略评估设定中（Sutton & Barto, 1998），我们关心与给定策略 $\pi$ 相关联的价值函数 $V^{\pi}$。这里的对应物是价值分布 $Z^{\pi}$。本节我们刻画 $Z^{\pi}$，并研究策略评估算子 $\mathcal{T}^{\pi}$ 的行为。我们强调，$Z^{\pi}$ 描述的是智能体与环境交互的内在随机性，而不是关于环境本身的某种不确定性度量。

We view the reward function as a random vector $R\in\mathcal{Z}$, and define the transition operator $P^{\pi}:\mathcal{Z}\to\mathcal{Z}$

我们把奖励函数看作一个随机向量 $R\in\mathcal{Z}$，并定义转移算子 $P^{\pi}:\mathcal{Z}\to\mathcal{Z}$

$$\displaystyle P^{\pi}Z(x,a) \\ \displaystyle\overset{D}{:=}Z(X^{\prime},A^{\prime}) \\ \displaystyle X^{\prime} \\ \displaystyle\sim P(\cdot\,|\,x,a),\,A^{\prime}\sim\pi(\cdot\,|\,X^{\prime}), \tag{4}$$

where we use capital letters to emphasize the random nature of the next state-action pair $(X^{\prime},A^{\prime})$. We define the distributional Bellman operator $\mathcal{T}^{\pi}:\mathcal{Z}\to\mathcal{Z}$ as

这里我们用大写字母来强调下一状态-动作对 $(X^{\prime},A^{\prime})$ 的随机性。我们把分布贝尔曼算子 $\mathcal{T}^{\pi}:\mathcal{Z}\to\mathcal{Z}$ 定义为

$$\mathcal{T}^{\pi}Z(x,a)\overset{D}{:=}R(x,a)+\gamma P^{\pi}Z(x,a). \tag{5}$$

While $\mathcal{T}^{\pi}$ bears a surface resemblance to the usual Bellman operator (2), it is fundamentally different. In particular, three sources of randomness define the compound distribution $\mathcal{T}^{\pi}Z$:

虽然 $\mathcal{T}^{\pi}$ 表面上类似于通常的贝尔曼算子 (2)，但两者有根本不同。特别地，有三个随机性来源共同定义了复合分布 $\mathcal{T}^{\pi}Z$：

a) The randomness in the reward $R$,

a) 奖励 $R$ 中的随机性，

b) The randomness in the transition $P^{\pi}$, and

b) 转移 $P^{\pi}$ 中的随机性，以及

c) The next-state value distribution $Z(X^{\prime},A^{\prime})$.

c) 下一状态的价值分布 $Z(X^{\prime},A^{\prime})$。

In particular, we make the usual assumption that these three quantities are independent. In this section we will show that (5) is a contraction mapping whose unique fixed point is the random return $Z^{\pi}$.

特别地，我们作通常的假定：这三个量是相互独立的。在本节中我们将证明，(5) 是一个压缩映射，其唯一不动点就是随机回报 $Z^{\pi}$。

3.3.1 Contraction in $\bar{d}_{p}$

Consider the process $Z_{k+1}:=\mathcal{T}^{\pi}Z_{k}$, starting with some $Z_{0}\in\mathcal{Z}$. We may expect the limiting expectation of $\{Z_{k}\}$ to converge exponentially quickly, as usual, to $Q^{\pi}$. As we now show, the process converges in a stronger sense: $\mathcal{T}^{\pi}$ is a contraction in $\bar{d}_{p}$, which implies that all moments also converge exponentially quickly.

3.3.1 在 $\bar{d}_{p}$ 下的压缩性

考虑过程 $Z_{k+1}:=\mathcal{T}^{\pi}Z_{k}$，从某个 $Z_{0}\in\mathcal{Z}$ 开始。我们可以预期，$\{Z_{k}\}$ 的极限期望会像往常一样指数快速地收敛到 $Q^{\pi}$。正如下面将表明的，这一过程在更强的意义下收敛：$\mathcal{T}^{\pi}$ 在 $\bar{d}_{p}$ 下是压缩映射，这意味着所有矩也都指数快速地收敛。

Lemma 3.

$\mathcal{T}^{\pi}:\mathcal{Z}\to\mathcal{Z}$is a $\gamma$-contraction in $\bar{d}_{p}$.

引理 3。

$\mathcal{T}^{\pi}:\mathcal{Z}\to\mathcal{Z}$在 $\bar{d}_{p}$ 下是一个 $\gamma$-压缩映射。

Using Lemma 3, we conclude using Banach’s fixed point theorem that $\mathcal{T}^{\pi}$ has a unique fixed point. By inspection, this fixed point must be $Z^{\pi}$ as defined in (1). As we assume all moments are bounded, this is sufficient to conclude that the sequence $\{Z_{k}\}$ converges to $Z^{\pi}$ in $\bar{d}_{p}$ for $1\leq p\leq\infty$.

利用引理 3，我们借助 Banach 不动点定理得出 $\mathcal{T}^{\pi}$ 有唯一不动点。通过直接检查，这一不动点必须是 (1) 中定义的 $Z^{\pi}$。由于我们假定所有矩都有界，这足以得出：对 $1\leq p\leq\infty$，序列 $\{Z_{k}\}$ 在 $\bar{d}_{p}$ 下收敛到 $Z^{\pi}$。

To conclude, we remark that not all distributional metrics are equal; for example, Chung & Sobel (1987) have shown that $\mathcal{T}^{\pi}$ is not a contraction in total variation distance. Similar results can be derived for the Kullback-Leibler divergence and the Kolmogorov distance.

作为结束，我们指出并非所有分布度量都是等价的；例如，Chung & Sobel（1987）已经表明 $\mathcal{T}^{\pi}$ 在全变差距离下不是压缩映射。对 Kullback-Leibler 散度和 Kolmogorov 距离也可以得出类似结果。

3.3.2 Contraction in Centered Moments

Observe that $d_{2}(U,V)$ (and more generally, $d_{p}$) relates to a coupling $C(\omega):=U(\omega)-V(\omega)$, in the sense that

3.3.2 中心矩上的压缩性

注意到 $d_{2}(U,V)$（更一般地，$d_{p}$）与一个耦合 $C(\omega):=U(\omega)-V(\omega)$ 有关，意义是

$$\displaystyle d_{2}^{2}(U,V) \\ \displaystyle\leq\expect[(U-V)^{2}]=\mathbb{V}\big(C\big)+\big(\expect C\big)^{2}.$$

As a result, we cannot directly use $d_{2}$ to bound the variance difference $|\mathbb{V}(\mathcal{T}^{\pi}Z(x,a))-\mathbb{V}(Z^{\pi}(x,a))|$. However, $\mathcal{T}^{\pi}$ is in fact a contraction in variance (Sobel, 1982, see also appendix). In general, $\mathcal{T}^{\pi}$ is not a contraction in the $p^{th}$ centered moment, $p>2$, but the centered moments of the iterates $\{Z_{k}\}$ still converge exponentially quickly to those of $Z^{\pi}$; the proof extends the result of Rösler (1992).

因此，我们不能直接用 $d_{2}$ 来界定方差差 $|\mathbb{V}(\mathcal{T}^{\pi}Z(x,a))-\mathbb{V}(Z^{\pi}(x,a))|$。然而，$\mathcal{T}^{\pi}$ 实际上在方差上是压缩映射（Sobel, 1982，亦见附录）。一般而言，$\mathcal{T}^{\pi}$ 在 $p>2$ 的 $p$ 阶中心矩上不是压缩映射，但迭代 $\{Z_{k}\}$ 的中心矩仍然会指数快速地收敛到 $Z^{\pi}$ 的中心矩；证明是对 Rösler（1992）结果的推广。

3.4 Control

3.4 控制

Thus far we have considered a fixed policy $\pi$, and studied the behaviour of its associated operator $\mathcal{T}^{\pi}$. We now set out to understand the distributional operators of the control setting – where we seek a policy $\pi$ that maximizes value – and the corresponding notion of an optimal value distribution. As with the optimal value function, this notion is intimately tied to that of an optimal policy. However, while all optimal policies attain the same value $Q^{*}$, in our case a difficulty arises: in general there are many optimal value distributions.

到目前为止，我们考虑的是固定策略 $\pi$，并研究与之相关的算子 $\mathcal{T}^{\pi}$ 的行为。现在我们着手理解控制设定下的分布算子——此时我们寻求一个最大化价值的策略 $\pi$——以及相应的最优价值分布概念。与最优价值函数一样，这一概念与最优策略紧密相连。然而，虽然所有最优策略都达到同一个价值 $Q^{*}$，在我们这里却会出现一个困难：一般而言，最优价值分布有许多个。

In this section we show that the distributional analogue of the Bellman optimality operator converges, in a weak sense, to the set of optimal value distributions. However, this operator is not a contraction in any metric between distributions, and is in general much more temperamental than the policy evaluation operators. We believe the convergence issues we outline here are a symptom of the inherent instability of greedy updates, as highlighted by e.g. Tsitsiklis (2002) and most recently Harutyunyan et al. (2016).

在本节中我们表明，贝尔曼最优性算子的分布对应物会在一种弱的意义上收敛到最优价值分布的集合。不过，这一算子在任何分布之间的度量下都不是压缩映射，并且一般比策略评估算子更不稳定。我们相信，这里勾勒出的收敛问题，是贪心更新固有不稳定性的一种症状，例如 Tsitsiklis（2002）以及最近 Harutyunyan et al.（2016）所强调的那样。

Let $\Pi^{*}$ be the set of optimal policies. We begin by characterizing what we mean by an optimal value distribution.

设 $\Pi^{*}$ 为最优策略的集合。我们首先刻画我们所说的最优价值分布是什么意思。

Definition 1(Optimal value distribution).

An optimal value distribution is the v.d. of an optimal policy. The set of optimal value distributions is $\mathcal{Z}^{}:={Z^{\pi^{}}:\pi^{}\in\Pi^{}}$.

定义 1（最优价值分布）。

最优价值分布是某一最优策略的价值分布。最优价值分布的集合是 $\mathcal{Z}^{}:={Z^{\pi^{}}:\pi^{}\in\Pi^{}}$。

We emphasize that not all value distributions with expectation $Q^{*}$ are optimal: they must match the full distribution of the return under some optimal policy.

我们强调，并非所有期望为 $Q^{*}$ 的价值分布都是最优的：它们必须与某个最优策略下回报的完整分布相一致。

Definition 2.

A greedy policy $\pi$ for $Z\in\mathcal{Z}$ maximizes the expectation of $Z$. The set of greedy policies for $Z$ is

定义 2。

对 $Z\in\mathcal{Z}$，贪心策略 $\pi$ 最大化 $Z$ 的期望。$Z$ 的贪心策略集合是

$$\mathcal{G}_{Z}:=\{\pi:\sum\nolimits_{a}\pi(a\,|\,x)\expect Z(x,a)=\max_{a^{\prime}\in\mathcal{A}}\expect Z(x,a^{\prime})\}.$$

Recall that the expected Bellman optimality operator $\mathcal{T}$ is

回忆期望形式的贝尔曼最优性算子 $\mathcal{T}$ 是

\mathcal{T}Q(x,a)=\expect R(x,a)+\gamma\expect\nolimits_{P}\max_{a^{\prime}\in\mathcal{A}}Q(x^{\prime},a^{\prime}). \tag{6}

The maximization at $x^{\prime}$ corresponds to some greedy policy. Although this policy is implicit in (6), we cannot ignore it in the distributional setting. We will call a distributional Bellman optimality operator any operator $\mathcal{T}$ which implements a greedy selection rule, i.e.

在 $x^{\prime}$ 处的最大化对应于某个贪心策略。尽管这一策略在 (6) 中是隐含的，在分布设定下我们却不能忽略它。凡是实现某种贪心选择规则的算子 $\mathcal{T}$，我们都称之为分布贝尔曼最优性算子，也就是

$$\mathcal{T}Z=\mathcal{T}^{\pi}Z\text{ for some }\pi\in\mathcal{G}_{Z}.$$

As in the policy evaluation setting, we are interested in the behaviour of the iterates $Z_{k+1}:=\mathcal{T}Z_{k}$, $Z_{0}\in\mathcal{Z}$. Our first result is to assert that $\expect Z_{k}$ behaves as expected.

与策略评估设定一样，我们关心迭代 $Z_{k+1}:=\mathcal{T}Z_{k}$、$Z_{0}\in\mathcal{Z}$ 的行为。我们的第一个结果是断言 $\expect Z_{k}$ 的表现符合预期。

Lemma 4.

Let $Z_{1},Z_{2}\in\mathcal{Z}$. Then

引理 4。

设 $Z_{1},Z_{2}\in\mathcal{Z}$。则

$$\left\|\expect\mathcal{T}Z_{1}-\expect\mathcal{T}Z_{2}\right\|_{\infty}\leq\gamma\left\|\expect Z_{1}-\expect Z_{2}\right\|_{\infty},$$

and in particular $\expect Z_{k}\to Q^{}$ exponentially quickly.*

并且特别地，$\expect Z_{k}$ 会指数快速地收敛到 $Q^{}$。*

By inspecting Lemma 4, we might expect that $Z_{k}$ converges quickly in $\bar{d}_{p}$ to some fixed point in $\mathcal{Z}^{*}$. Unfortunately, convergence is neither quick nor assured to reach a fixed point. In fact, the best we can hope for is pointwise convergence, not even to the set $\mathcal{Z}^{*}$ but to the larger set of nonstationary optimal value distributions.

若只看引理 4，我们或许会期望 $Z_{k}$ 在 $\bar{d}_{p}$ 下快速收敛到 $\mathcal{Z}^{*}$ 中的某个不动点。遗憾的是，收敛既不迅速，也不能保证到达一个不动点。事实上，我们最多只能指望逐点收敛，而且甚至不是收敛到集合 $\mathcal{Z}^{*}$，而是收敛到更大的非平稳最优价值分布集合。

Definition 3.

A nonstationary optimal value distribution $Z^{**}$ is the value distribution corresponding to a sequence of optimal policies. The set of n.o.v.d. is $\mathcal{Z}^{**}$.

定义 3。

非平稳最优价值分布 $Z^{**}$ 是对应于一列最优策略的价值分布。非平稳最优价值分布的集合是 $\mathcal{Z}^{**}$。

Theorem 1(Convergence in the control setting).

Let $\mathcal{X}$ be measurable and suppose that $\mathcal{A}$ is finite. Then

定理 1（控制设定中的收敛性）。

设 $\mathcal{X}$ 是可测的，并假定 $\mathcal{A}$ 是有限的。则

$$\lim_{k\to\infty}\inf_{Z^{**}\in\mathcal{Z}^{**}}d_{p}(Z_{k}(x,a),Z^{**}(x,a))=0\quad\forall x,a.$$

If $\mathcal{X}$ is finite, then $Z_{k}$ converges to $\mathcal{Z}^{**}$ uniformly. Furthermore, if there is a total ordering $\prec$ on $\Pi^{}$, such that for any$Z^{}\in\mathcal{Z}^{}$,*

若 $\mathcal{X}$ 是有限的，则 $Z_{k}$ 一致收敛到 $\mathcal{Z}^{**}$。此外，若 $\Pi^{}$上存在一个全序$\prec$，使得对任意$Z^{}\in\mathcal{Z}^{}$，*

$$\mathcal{T}Z^{*}=\mathcal{T}^{\pi}Z^{*}\;\text{with}\;\pi\in\mathcal{G}_{Z^{*}},\;\pi\prec\pi^{\prime}\;\;\forall\pi^{\prime}\in\mathcal{G}_{Z^{*}}\setminus\{\pi\}.$$

Then $\mathcal{T}$ has a unique fixed point $Z^{}\in\mathcal{Z}^{}$.

则 $\mathcal{T}$ 有唯一不动点 $Z^{}\in\mathcal{Z}^{}$。

Comparing Theorem 1 to Lemma 4 reveals a significant difference between the distributional framework and the usual setting of expected return. While the mean of $Z_{k}$ converges exponentially quickly to $Q^{*}$, its distribution need not be as well-behaved! To emphasize this difference, we now provide a number of negative results concerning $\mathcal{T}$.

把定理 1 与引理 4 相比较，可以看出分布框架与通常的期望回报设定之间有显著差别。$Z_{k}$ 的均值会指数快速地收敛到 $Q^{*}$，但它的分布却不必同样表现良好！为了强调这一差别，我们现在给出关于 $\mathcal{T}$ 的若干否定性结果。

Proposition 1.

The operator $\mathcal{T}$ is not a contraction.

命题 1。

算子 $\mathcal{T}$ 不是压缩映射。

Consider the following example (Figure 2, left). There are two states, $x_{1}$ and $x_{2}$; a unique transition from $x_{1}$ to $x_{2}$; from $x_{2}$, action $a_{1}$ yields no reward, while the optimal action $a_{2}$ yields $1+\epsilon$ or $-1+\epsilon$ with equal probability. Both actions are terminal. There is a unique optimal policy and therefore a unique fixed point $Z^{*}$. Now consider $Z$ as given in Figure 2 (right), and its distance to $Z^{*}$:

考虑下面的例子（图 2，左）。有两个状态 $x_{1}$ 和 $x_{2}$；从 $x_{1}$ 到 $x_{2}$ 有唯一的转移；在 $x_{2}$ 处，动作 $a_{1}$ 不产生奖励，而最优动作 $a_{2}$ 以相等的概率产生 $1+\epsilon$ 或 $-1+\epsilon$。两个动作都是终止的。存在唯一的最优策略，因而有唯一的不动点 $Z^{*}$。现在考虑图 2（右）中给出的 $Z$，以及它到 $Z^{*}$ 的距离：

$$\bar{d}{1}(Z,Z^{*})=d{1}(Z(x_{2},a_{2}),Z^{*}(x_{2},a_{2}))=2\epsilon,\
$$

where we made use of the fact that $Z=Z^{*}$ everywhere except at $(x_{2},a_{2})$. When we apply $\mathcal{T}$ to $Z$, however, the greedy action $a_{1}$ is selected and $\mathcal{T}Z(x_{1})=Z(x_{2},a_{1})$. But

这里我们用到了除 $(x_{2},a_{2})$ 外，$Z=Z^{*}$ 处处成立这一事实。然而，当我们对 $Z$ 应用 $\mathcal{T}$ 时，贪心动作 $a_{1}$ 被选中，并且 $\mathcal{T}Z(x_{1})=Z(x_{2},a_{1})$。但

$$\displaystyle d_{1}(\mathcal{T}Z,\mathcal{T}Z^{*}) \\ \displaystyle=d_{1}(\mathcal{T}Z(x_{1}),Z^{*}(x_{1})) \\ \displaystyle=\tfrac{1}{2}|1-\epsilon|+\tfrac{1}{2}|1+\epsilon|>2\epsilon$$

for a sufficiently small $\epsilon$. This shows that the undiscounted update is not a nonexpansion: $\bar{d}_{1}(\mathcal{T}Z,\mathcal{T}Z^{*})>\bar{d}_{1}(Z,Z^{*})$. With $\gamma<1$, the same proof shows it is not a contraction. Using a more technically involved argument, we can extend this result to any metric which separates $Z$ and $\mathcal{T}Z$.

对足够小的 $\epsilon$ 成立。这表明无折扣更新不是非扩张的：$\bar{d}_{1}(\mathcal{T}Z,\mathcal{T}Z^{*})>\bar{d}_{1}(Z,Z^{*})$。当 $\gamma<1$ 时，同样的证明表明它不是压缩映射。用一个技术上更复杂的论证，我们可以把这一结果推广到任何能分开 $Z$ 和 $\mathcal{T}Z$ 的度量。

Figure 2: Undiscounted two-state MDP for which the optimality operator $\mathcal{T}$ is not a contraction, with example. The entries that contribute to $\bar{d}_{1}(Z,Z^{*})$ and $\bar{d}_{1}(\mathcal{T}Z,Z^{*})$ are highlighted.

图 2：最优性算子 $\mathcal{T}$ 并非压缩映射的无折扣两状态 MDP，并给出例子。对 $\bar{d}_{1}(Z,Z^{*})$ 和 $\bar{d}_{1}(\mathcal{T}Z,Z^{*})$ 有贡献的表项被标出。

Proposition 2.

Not all optimality operators have a fixed point $Z^{}=\mathcal{T}Z^{}$.

命题 2。

并非所有最优性算子都有不动点 $Z^{}=\mathcal{T}Z^{}$。

To see this, consider the same example, now with $\epsilon=0$, and a greedy operator $\mathcal{T}$ which breaks ties by picking $a_{2}$ if $Z(x_{1})=0$, and $a_{1}$ otherwise. Then the sequence $\mathcal{T}Z^{*}(x_{1}),(\mathcal{T})^{2}Z^{*}(x_{1}),\,\dots$ alternates between $Z^{*}(x_{2},a_{1})$ and $Z^{*}(x_{2},a_{2})$.

为说明这一点，考虑同一个例子，现在取 $\epsilon=0$，以及一个贪心算子 $\mathcal{T}$：它在 $Z(x_{1})=0$ 时打破平局选择 $a_{2}$，否则选择 $a_{1}$。于是序列 $\mathcal{T}Z^{*}(x_{1}),(\mathcal{T})^{2}Z^{*}(x_{1}),\,\dots$ 会在 $Z^{*}(x_{2},a_{1})$ 与 $Z^{*}(x_{2},a_{2})$ 之间交替。

Proposition 3.

That $\mathcal{T}$ has a fixed point $Z^{}=\mathcal{T}Z^{}$is insufficient to guarantee the convergence of${Z_{k}}$to$\mathcal{Z}^{}$.*

命题 3。

$\mathcal{T}$ 有不动点 $Z^{}=\mathcal{T}Z^{}$，并不足以保证${Z_{k}}$收敛到$\mathcal{Z}^{}$。*

Theorem 1 paints a rather bleak picture of the control setting. It remains to be seen whether the dynamical eccentricies highlighted here actually arise in practice. One open question is whether theoretically more stable behaviour can be derived using stochastic policies, for example from conservative policy iteration (Kakade & Langford, 2002).

定理 1 给控制设定描绘了一幅相当黯淡的图景。这里突出的这些动态异常是否真的会在实践中出现，仍有待观察。一个未决的问题是：能否通过使用随机策略，例如来自保守策略迭代（Kakade & Langford, 2002），在理论上得到更稳定的行为。

4 Approximate Distributional Learning

4 近似分布学习

In this section we propose an algorithm based on the distributional Bellman optimality operator. In particular, this will require choosing an approximating distribution. Although the Gaussian case has previously been considered (Morimura et al., 2010a; Tamar et al., 2016), to the best of our knowledge we are the first to use a rich class of parametric distributions.

在本节中，我们提出一种基于分布贝尔曼最优性算子的算法。特别地，这需要选择一种近似分布。尽管高斯情形先前已被考虑过（Morimura et al., 2010a; Tamar et al., 2016），但据我们所知，我们是最先使用一类表达力更丰富的参数分布的。

4.1 Parametric Distribution

4.1 参数化分布

We will model the value distribution using a discrete distribution parametrized by $N\in\mathbb{N}$ and $V_{\textsc{min}},V_{\textsc{max}}\in\mathbb{R}$, and whose support is the set of atoms $\{z_{i}=V_{\textsc{min}}+i\triangle z:0\leq i<N\}$, $\triangle z:=\tfrac{V_{\textsc{max}}-V_{\textsc{min}}}{N-1}$. In a sense, these atoms are the “canonical returns” of our distribution. The atom probabilities are given by a parametric model $\theta:\mathcal{X}\times\mathcal{A}\to\mathbb{R}^{N}$

我们将用一个离散分布来建模价值分布，其参数为 $N\in\mathbb{N}$ 以及 $V_{\textsc{min}},V_{\textsc{max}}\in\mathbb{R}$，支撑是原子集合 $\{z_{i}=V_{\textsc{min}}+i\triangle z:0\leq i<N\}$，其中 $\triangle z:=\tfrac{V_{\textsc{max}}-V_{\textsc{min}}}{N-1}$。在某种意义上，这些原子就是我们分布的“典型回报”。原子上的概率由参数模型 $\theta:\mathcal{X}\times\mathcal{A}\to\mathbb{R}^{N}$ 给出

$$Z_{\theta}(x,a)=z_{i}\quad\text{w.p. }\;p_{i}(x,a):=\frac{e^{\theta_{i}(x,a)}}{\sum\nolimits_{j}e^{\theta_{j}(x,a)}}.$$

The discrete distribution has the advantages of being highly expressive and computationally friendly (Van den Oord et al., 2016, see e.g.).

离散分布的优点是表达力强，并且在计算上友好（例如见 Van den Oord et al., 2016）。

4.2 Projected Bellman Update

4.2 投影贝尔曼更新

Using a discrete distribution poses a problem: the Bellman update $\mathcal{T}Z_{\theta}$ and our parametrization $Z_{\theta}$ almost always have disjoint supports. From the analysis of Section 3 it would seem natural to minimize the Wasserstein metric (viewed as a loss) between $\mathcal{T}Z_{\theta}$ and $Z_{\theta}$, which is also conveniently robust to discrepancies in support. However, a second issue prevents this: in practice we are typically restricted to learning from sample transitions, which is not possible under the Wasserstein loss (see Prop. 5 and toy results in the appendix).

使用离散分布会带来一个问题：贝尔曼更新 $\mathcal{T}Z_{\theta}$ 与我们的参数化 $Z_{\theta}$ 几乎总是具有不相交的支撑。从第 3 节的分析看，在 $\mathcal{T}Z_{\theta}$ 与 $Z_{\theta}$ 之间最小化 Wasserstein 度量（把它看作一种损失）似乎是自然的，而且它对支撑上的不一致也相当稳健。然而第二个问题阻碍了这一点：在实践中我们通常只能从样本转移中学习，而这在 Wasserstein 损失下是不可行的（见命题 5 以及附录中的玩具结果）。

Instead, we project the sample Bellman update $\hat{\mathcal{T}}Z_{\theta}$ onto the support of $Z_{\theta}$ (Figure 1, Algorithm 1), effectively reducing the Bellman update to multiclass classification. Let $\pi$ be the greedy policy w.r.t. $\expect Z_{\theta}$. Given a sample transition $(x,a,r,x^{\prime})$, we compute the Bellman update $\hat{\mathcal{T}}z_{j}:=r+\gamma z_{j}$ for each atom $z_{j}$, then distribute its probability $p_{j}(x^{\prime},\pi(x^{\prime}))$ to the immediate neighbours of $\hat{\mathcal{T}}z_{j}$. The $i^{th}$ component of the projected update $\Phi\hat{\mathcal{T}}Z_{\theta}(x,a)$ is

取而代之，我们把样本贝尔曼更新 $\hat{\mathcal{T}}Z_{\theta}$ 投影到 $Z_{\theta}$ 的支撑上（图 1，算法 1），从而有效地把贝尔曼更新化成多类分类。设 $\pi$ 是关于 $\expect Z_{\theta}$ 的贪心策略。给定一个样本转移 $(x,a,r,x^{\prime})$，我们对每个原子 $z_{j}$ 计算贝尔曼更新 $\hat{\mathcal{T}}z_{j}:=r+\gamma z_{j}$，再把它的概率 $p_{j}(x^{\prime},\pi(x^{\prime}))$ 分配给 $\hat{\mathcal{T}}z_{j}$ 的紧邻原子。投影更新 $\Phi\hat{\mathcal{T}}Z_{\theta}(x,a)$ 的第 $i$ 个分量是

$$(\Phi\hat{\mathcal{T}}Z_{\theta}(x,a))_{i}=\sum_{j=0}^{N-1}\left[1-\frac{|[\hat{\mathcal{T}}z_{j}]^{V_{\textsc{max}}}_{V_{\textsc{min}}}-z_{i}|}{\triangle z}\right]^{1}_{0}p_{j}(x^{\prime},\pi(x^{\prime})), \tag{7}$$

where $[\cdot]^{b}_{a}$ bounds its argument in the range $[a,b]$.[^1] As is usual, we view the next-state distribution as parametrized by a fixed parameter $\tilde{\theta}$. The sample loss $\mathcal{L}_{x,a}(\theta)$ is the cross-entropy term of the KL divergence

其中 $[\cdot]^{b}_{a}$ 把它的自变量限制在区间 $[a,b]$ 内。[^1] 照例，我们把下一状态分布看作由固定参数 $\tilde{\theta}$ 参数化。样本损失 $\mathcal{L}_{x,a}(\theta)$ 是 KL 散度中的交叉熵项

$$D_{\textsc{kl}}(\Phi\hat{\mathcal{T}}Z_{\tilde{\theta}}(x,a)\,\|\,Z_{\theta}(x,a)),$$

which is readily minimized e.g. using gradient descent. We call this choice of distribution and loss the categorical algorithm. When $N=2$, a simple one-parameter alternative is $\Phi\hat{\mathcal{T}}Z_{\theta}(x,a):=[\expect[\hat{\mathcal{T}}Z_{\theta}(x,a)]-V_{\textsc{min}})/\triangle z]^{1}_{0};$ we call this the Bernoulli algorithm. We note that, while these algorithms appear unrelated to the Wasserstein metric, recent work (Bellemare et al., 2017) hints at a deeper connection.

它可以方便地用例如梯度下降来最小化。我们把这一分布与损失的选择叫做分类算法。当 $N=2$ 时，一个简单的单参数替代是 $\Phi\hat{\mathcal{T}}Z_{\theta}(x,a):=[\expect[\hat{\mathcal{T}}Z_{\theta}(x,a)]-V_{\textsc{min}})/\triangle z]^{1}_{0};$ 我们称之为伯努利算法。我们指出，尽管这些算法看起来与 Wasserstein 度量无关，最近的工作（Bellemare et al., 2017）暗示它们之间有更深层的联系。

0:  A transition $x_{t},a_{t},r_{t},x_{t+1}$, $\gamma_{t}\in[0,1]$
$Q(x_{t+1},a):=\sum\nolimits_{i}z_{i}p_{i}(x_{t+1},a)$
$a^{*}\leftarrow\argmax_{a}Q(x_{t+1},a)$
$m_{i}=0,\quad i\in 0,\dots,N-1$
for $j\in 0,\dots,N-1$ do

Compute the projection of $\hat{\mathcal{T}}z_{j}$ onto the support $\{z_{i}\}$

$\hat{\mathcal{T}}z_{j}\leftarrow\left[r_{t}+\gamma_{t}z_{j}\right]^{V_{\textsc{max}}}_{V_{\textsc{min}}}$
$b_{j}\leftarrow(\hat{\mathcal{T}}z_{j}-V_{\textsc{min}})/\Delta z$   # $b_{j}\in[0,N-1]$
$l\leftarrow\lfloor b_{j}\rfloor$, $u\leftarrow\lceil b_{j}\rceil$

Distribute probability of $\hat{\mathcal{T}}z_{j}$

$m_{l}\leftarrow m_{l}+p_{j}(x_{t+1},a^{*})(u-b_{j})$
$m_{u}\leftarrow m_{u}+p_{j}(x_{t+1},a^{*})(b_{j}-l)$
end for
$-\sum_{i}m_{i}\log p_{i}(x_{t},a_{t})$   # Cross-entropy loss

Algorithm 1 Categorical Algorithm

0: 　一次转移 $x_{t},a_{t},r_{t},x_{t+1}$，$\gamma_{t}\in[0,1]$
$Q(x_{t+1},a):=\sum\nolimits_{i}z_{i}p_{i}(x_{t+1},a)$
$a^{*}\leftarrow\argmax_{a}Q(x_{t+1},a)$
$m_{i}=0,\quad i\in 0,\dots,N-1$
for $j\in 0,\dots,N-1$ do

计算 $\hat{\mathcal{T}}z_{j}$ 到支撑 $\{z_{i}\}$ 上的投影

$\hat{\mathcal{T}}z_{j}\leftarrow\left[r_{t}+\gamma_{t}z_{j}\right]^{V_{\textsc{max}}}_{V_{\textsc{min}}}$
$b_{j}\leftarrow(\hat{\mathcal{T}}z_{j}-V_{\textsc{min}})/\Delta z$   # $b_{j}\in[0,N-1]$
$l\leftarrow\lfloor b_{j}\rfloor$，$u\leftarrow\lceil b_{j}\rceil$

分配 $\hat{\mathcal{T}}z_{j}$ 的概率

$m_{l}\leftarrow m_{l}+p_{j}(x_{t+1},a^{*})(u-b_{j})$
$m_{u}\leftarrow m_{u}+p_{j}(x_{t+1},a^{*})(b_{j}-l)$
end for
$-\sum_{i}m_{i}\log p_{i}(x_{t},a_{t})$   # 交叉熵损失

算法 1 分类算法