Adam: A Method for Stochastic Optimization

Diederik P. Kingma*
University of Amsterdam, OpenAI
dpkingma@openai.com

Jimmy Lei Ba∗
University of Toronto
jimmy@psi.utoronto.ca

∗ Equal contribution. Author ordering determined by coin flip over a Google Hangout.

Abstract

摘要

We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.

我们提出 Adam，一种基于低阶矩自适应估计、用于随机目标函数的一阶梯度优化算法。该方法实现直接、计算高效、内存需求小、对梯度的对角缩放不变，并适用于数据与/或参数规模很大的问题。它也适合非平稳目标，以及噪声很大与/或梯度稀疏的问题。超参数具有直观解释，通常几乎不需调参。我们讨论了启发 Adam 的若干相关算法之间的联系。我们还分析了算法的理论收敛性质，并在在线凸优化框架下给出与当时已知最优结果可比的收敛速率遗憾界。经验结果表明 Adam 在实践中表现良好，并优于其他随机优化方法。最后，我们讨论基于无穷范数的 Adam 变体 AdaMax。

1 Introduction

1 引言

Stochastic gradient-based optimization is of core practical importance in many fields of science and engineering. Many problems in these fields can be cast as the optimization of some scalar parameterized objective function requiring maximization or minimization with respect to its parameters. If the function is differentiable w.r.t. its parameters, gradient descent is a relatively efficient optimization method, since the computation of first-order partial derivatives w.r.t. all the parameters is of the same computational complexity as just evaluating the function. Often, objective functions are stochastic. For example, many objective functions are composed of a sum of subfunctions evaluated at different subsamples of data; in this case optimization can be made more efficient by taking gradient steps w.r.t. individual subfunctions, i.e. stochastic gradient descent (SGD) or ascent. SGD proved itself as an efficient and effective optimization method that was central in many machine learning success stories, such as recent advances in deep learning (Deng et al., 2013; Krizhevsky et al., 2012; Hinton & Salakhutdinov, 2006; Hinton et al., 2012a; Graves et al., 2013). Objectives may also have other sources of noise than data subsampling, such as dropout (Hinton et al., 2012b) regularization. For all such noisy objectives, efficient stochastic optimization techniques are required. The focus of this paper is on the optimization of stochastic objectives with high-dimensional parameter spaces. In these cases, higher-order optimization methods are ill-suited, and discussion in this paper will be restricted to first-order methods.

基于随机梯度的优化在许多科学与工程领域具有核心实践重要性。这些领域中的许多问题可表述为对某个标量参数化目标函数关于其参数求最大或最小。若函数对参数可微，梯度下降是相对高效的优化方法，因为对全部参数计算一阶偏导的计算复杂度与仅求函数值相同。目标函数常常是随机的。例如，许多目标由在不同数据子样本上求值的子函数之和组成；此时可通过相对各个子函数做梯度步（即随机梯度下降 SGD 或上升）使优化更高效。SGD 已证明是高效有效的优化方法，并在许多机器学习成功故事中处于中心位置，例如深度学习的近期进展（Deng et al., 2013；Krizhevsky et al., 2012；Hinton & Salakhutdinov, 2006；Hinton et al., 2012a；Graves et al., 2013）。目标还可能有数据子采样以外的噪声来源，例如 dropout（Hinton et al., 2012b）正则。对所有这些带噪目标，都需要高效的随机优化技术。本文聚焦高维参数空间中随机目标的优化。在这些情形下，高阶优化方法不合适，因而本文讨论限于一阶方法。

We propose Adam, a method for efficient stochastic optimization that only requires first-order gradients with little memory requirement. The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients; the name Adam is derived from adaptive moment estimation. Our method is designed to combine the advantages of two recently popular methods: AdaGrad (Duchi et al., 2011), which works well with sparse gradients, and RMSProp (Tieleman & Hinton, 2012), which works well in on-line and non-stationary settings; important connections to these and other stochastic optimization methods are clarified in section 5. Some of Adam's advantages are that the magnitudes of parameter updates are invariant to rescaling of the gradient, its stepsizes are approximately bounded by the stepsize hyperparameter, it does not require a stationary objective, it works with sparse gradients, and it naturally performs a form of step size annealing.

我们提出 Adam，一种仅需一阶梯度、内存需求很小的高效随机优化方法。该方法从梯度的一阶与二阶矩估计为不同参数计算各自的自适应学习率；Adam 之名来自自适应矩估计（adaptive moment estimation）。我们的方法旨在结合两种近期流行方法的优点：对稀疏梯度表现良好的 AdaGrad（Duchi et al., 2011），以及在在线与非平稳设定下表现良好的 RMSProp（Tieleman & Hinton, 2012）；与这些及其他随机优化方法的重要联系在第 5 节阐明。Adam 的一些优点包括：参数更新幅度对梯度缩放不变，其步长近似被步长超参数上界约束，不要求目标平稳，适用于稀疏梯度，并自然地执行一种步长退火。

Algorithm 1: Adam, our proposed algorithm for stochastic optimization. See section 2 for details, and for a slightly more efficient (but less clear) order of computation. $g_t^2$ indicates the elementwise square $g_t \odot g_t$. Good default settings for the tested machine learning problems are $\alpha=0.001$, $\beta_1=0.9$, $\beta_2=0.999$ and $\epsilon=10^{-8}$. All operations on vectors are element-wise. With $\beta_1^t$ and $\beta_2^t$ we denote $\beta_1$ and $\beta_2$ to the power $t$.

Require: $\alpha$: Stepsize
Require: $\beta_1,\beta_2\in[0,1)$: Exponential decay rates for the moment estimates
Require: $f(\theta)$: Stochastic objective function with parameters $\theta$
Require: $\theta_0$: Initial parameter vector
$m_0\leftarrow 0$ (Initialize 1st moment vector)
$v_0\leftarrow 0$ (Initialize 2nd moment vector)
$t\leftarrow 0$ (Initialize timestep)
while $\theta_t$ not converged do
 $t\leftarrow t+1$
 $g_t\leftarrow\nabla_\theta f_t(\theta_{t-1})$ (Get gradients w.r.t. stochastic objective at timestep $t$)
 $m_t\leftarrow\beta_1\cdot m_{t-1}+(1-\beta_1)\cdot g_t$ (Update biased first moment estimate)
 $v_t\leftarrow\beta_2\cdot v_{t-1}+(1-\beta_2)\cdot g_t^2$ (Update biased second raw moment estimate)
 $\hat{m}_t\leftarrow m_t/(1-\beta_1^t)$ (Compute bias-corrected first moment estimate)
 $\hat{v}_t\leftarrow v_t/(1-\beta_2^t)$ (Compute bias-corrected second raw moment estimate)
 $\theta_t\leftarrow\theta_{t-1}-\alpha\cdot\hat{m}_t/(\sqrt{\hat{v}_t}+\epsilon)$ (Update parameters)
end while
return $\theta_t$ (Resulting parameters)

算法 1：Adam，我们提出的随机优化算法。细节见第 2 节，以及稍更高效（但较不清晰）的计算顺序。$g_t^2$ 表示逐元素平方 $g_t \odot g_t$。对所测机器学习问题，良好默认设置为 $\alpha=0.001$、$\beta_1=0.9$、$\beta_2=0.999$ 与 $\epsilon=10^{-8}$。对向量的所有运算均为逐元素。用 $\beta_1^t$ 与 $\beta_2^t$ 表示 $\beta_1$ 与 $\beta_2$ 的 $t$ 次幂。

要求：$\alpha$：步长
要求：$\beta_1,\beta_2\in[0,1)$：矩估计的指数衰减率
要求：$f(\theta)$：带参数 $\theta$ 的随机目标函数
要求：$\theta_0$：初始参数向量
$m_0\leftarrow 0$（初始化一阶矩向量）
$v_0\leftarrow 0$（初始化二阶矩向量）
$t\leftarrow 0$（初始化时间步）
while $\theta_t$ 未收敛 do
 $t\leftarrow t+1$
 $g_t\leftarrow\nabla_\theta f_t(\theta_{t-1})$（在时间步 $t$ 求相对随机目标的梯度）
 $m_t\leftarrow\beta_1\cdot m_{t-1}+(1-\beta_1)\cdot g_t$（更新有偏一阶矩估计）
 $v_t\leftarrow\beta_2\cdot v_{t-1}+(1-\beta_2)\cdot g_t^2$（更新有偏二阶原始矩估计）
 $\hat{m}_t\leftarrow m_t/(1-\beta_1^t)$（计算偏差校正后的一阶矩估计）
 $\hat{v}_t\leftarrow v_t/(1-\beta_2^t)$（计算偏差校正后的二阶原始矩估计）
 $\theta_t\leftarrow\theta_{t-1}-\alpha\cdot\hat{m}_t/(\sqrt{\hat{v}_t}+\epsilon)$（更新参数）
end while
return $\theta_t$（结果参数）

In section 2 we describe the algorithm and the properties of its update rule. Section 3 explains our initialization bias correction technique, and section 4 provides a theoretical analysis of Adam's convergence in online convex programming. Empirically, our method consistently outperforms other methods for a variety of models and datasets, as shown in section 6. Overall, we show that Adam is a versatile algorithm that scales to large-scale high-dimensional machine learning problems.

第 2 节描述算法及其更新规则的性质。第 3 节解释我们的初始化偏差校正技术，第 4 节给出 Adam 在在线凸规划中的收敛理论分析。经验上，如第 6 节所示，我们的方法在多种模型与数据集上一致优于其他方法。总体而言，我们表明 Adam 是一种可扩展到大规模高维机器学习问题的通用算法。

2 Algorithm

2 算法

See algorithm 1 for pseudo-code of our proposed algorithm Adam. Let $f(\theta)$ be a noisy objective function: a stochastic scalar function that is differentiable w.r.t. parameters $\theta$. We are interested in minimizing the expected value of this function, $\mathbb{E}[f(\theta)]$ w.r.t. its parameters $\theta$. With $f_1(\theta),\ldots,f_T(\theta)$ we denote the realisations of the stochastic function at subsequent timesteps $1,\ldots,T$. The stochasticity might come from the evaluation at random subsamples (minibatches) of datapoints, or arise from inherent function noise. With $g_t=\nabla_\theta f_t(\theta)$ we denote the gradient, i.e. the vector of partial derivatives of $f_t$, w.r.t $\theta$ evaluated at timestep $t$.

我们提出的算法 Adam 的伪代码见算法 1。令 $f(\theta)$ 为带噪目标函数：对参数 $\theta$ 可微的随机标量函数。我们关心相对其参数 $\theta$ 最小化该函数的期望 $\mathbb{E}[f(\theta)]$。用 $f_1(\theta),\ldots,f_T(\theta)$ 表示该随机函数在相继时间步 $1,\ldots,T$ 的实现。随机性可能来自在数据点随机子样本（小批量）上求值，或来自函数固有噪声。用 $g_t=\nabla_\theta f_t(\theta)$ 表示梯度，即 $f_t$ 相对 $\theta$ 在时间步 $t$ 求值的偏导向量。

The algorithm updates exponential moving averages of the gradient ($m_t$) and the squared gradient ($v_t$) where the hyper-parameters $\beta_1,\beta_2\in[0,1)$ control the exponential decay rates of these moving averages. The moving averages themselves are estimates of the 1st moment (the mean) and the 2nd raw moment (the uncentered variance) of the gradient. However, these moving averages are initialized as (vectors of) 0's, leading to moment estimates that are biased towards zero, especially during the initial timesteps, and especially when the decay rates are small (i.e. the $\beta$s are close to 1). The good news is that this initialization bias can be easily counteracted, resulting in bias-corrected estimates $\hat{m}_t$ and $\hat{v}_t$. See section 3 for more details.

算法更新梯度的指数滑动平均（$m_t$）与平方梯度的指数滑动平均（$v_t$），超参数 $\beta_1,\beta_2\in[0,1)$ 控制这些滑动平均的指数衰减率。滑动平均本身是梯度一阶矩（均值）与二阶原始矩（未中心化方差）的估计。然而这些滑动平均初始化为 0（向量），导致矩估计偏向零，尤其在初始时间步，且尤其当衰减率很小（即 $\beta$ 接近 1）时。好消息是这一初始化偏差可以轻易抵消，得到偏差校正估计 $\hat{m}_t$ 与 $\hat{v}_t$。详见第 3 节。

Note that the efficiency of algorithm 1 can, at the expense of clarity, be improved upon by changing the order of computation, e.g. by replacing the last three lines in the loop with the following lines: $\alpha_t=\alpha\cdot\sqrt{1-\beta_2^t}/(1-\beta_1^t)$ and $\theta_t\leftarrow\theta_{t-1}-\alpha_t\cdot m_t/(\sqrt{v_t}+\epsilon)$.

注意，以清晰度为代价，可通过改变计算顺序改进算法 1 的效率，例如把循环中最后三行替换为：$\alpha_t=\alpha\cdot\sqrt{1-\beta_2^t}/(1-\beta_1^t)$ 以及 $\theta_t\leftarrow\theta_{t-1}-\alpha_t\cdot m_t/(\sqrt{v_t}+\epsilon)$。

2.1 Adam's Update Rule

2.1 Adam 的更新规则

An important property of Adam's update rule is its careful choice of stepsizes. Assuming $\epsilon=0$, the effective step taken in parameter space at timestep $t$ is $\Delta_t=\alpha\cdot\hat{m}_t/\sqrt{\hat{v}_t}$. The effective stepsize has two upper bounds: $|\Delta_t|\leq\alpha\cdot(1-\beta_1)/\sqrt{1-\beta_2}$ in the case $(1-\beta_1)>\sqrt{1-\beta_2}$, and $|\Delta_t|\leq\alpha$ otherwise. The first case only happens in the most severe case of sparsity: when a gradient has been zero at all timesteps except at the current timestep. For less sparse cases, the effective stepsize will be smaller. When $(1-\beta_1)=\sqrt{1-\beta_2}$ we have that $|\hat{m}_t/\sqrt{\hat{v}_t}|<1$ therefore $|\Delta_t|<\alpha$. In more common scenarios, we will have that $\hat{m}_t/\sqrt{\hat{v}_t}\approx\pm 1$ since $|\mathbb{E}[g]/\sqrt{\mathbb{E}[g^2]}|\leq 1$. The effective magnitude of the steps taken in parameter space at each timestep are approximately bounded by the stepsize setting $\alpha$, i.e., $|\Delta_t|\lesssim\alpha$. This can be understood as establishing a trust region around the current parameter value, beyond which the current gradient estimate does not provide sufficient information. This typically makes it relatively easy to know the right scale of $\alpha$ in advance. For many machine learning models, for instance, we often know in advance that good optima are with high probability within some set region in parameter space; it is not uncommon, for example, to have a prior distribution over the parameters. Since $\alpha$ sets (an upper bound of) the magnitude of steps in parameter space, we can often deduce the right order of magnitude of $\alpha$ such that optima can be reached from $\theta_0$ within some number of iterations. With a slight abuse of terminology, we will call the ratio $\hat{m}_t/\sqrt{\hat{v}_t}$ the signal-to-noise ratio (SNR). With a smaller SNR the effective stepsize $\Delta_t$ will be closer to zero. This is a desirable property, since a smaller SNR means that there is greater uncertainty about whether the direction of $\hat{m}_t$ corresponds to the direction of the true gradient. For example, the SNR value typically becomes closer to 0 towards an optimum, leading to smaller effective steps in parameter space: a form of automatic annealing. The effective stepsize $\Delta_t$ is also invariant to the scale of the gradients; rescaling the gradients $g$ with factor $c$ will scale $\hat{m}_t$ with a factor $c$ and $\hat{v}_t$ with a factor $c^2$, which cancel out: $(c\cdot\hat{m}_t)/(\sqrt{c^2\cdot\hat{v}_t})=\hat{m}_t/\sqrt{\hat{v}_t}$.

Adam 更新规则的一个重要性质是其对步长的审慎选择。假定 $\epsilon=0$，时间步 $t$ 在参数空间中采取的有效步为 $\Delta_t=\alpha\cdot\hat{m}_t/\sqrt{\hat{v}_t}$。有效步长有两个上界：当 $(1-\beta_1)>\sqrt{1-\beta_2}$ 时 $|\Delta_t|\leq\alpha\cdot(1-\beta_1)/\sqrt{1-\beta_2}$，否则 $|\Delta_t|\leq\alpha$。第一种情形只发生在最严重的稀疏情况：梯度在除当前时间步外的所有时间步都为零。对不那么稀疏的情形，有效步长会更小。当 $(1-\beta_1)=\sqrt{1-\beta_2}$ 时有 $|\hat{m}_t/\sqrt{\hat{v}_t}|<1$，因而 $|\Delta_t|<\alpha$。在更常见的情形下，由于 $|\mathbb{E}[g]/\sqrt{\mathbb{E}[g^2]}|\leq 1$，会有 $\hat{m}_t/\sqrt{\hat{v}_t}\approx\pm 1$。每个时间步在参数空间中采取的有效步幅近似被步长设置 $\alpha$ 上界约束，即 $|\Delta_t|\lesssim\alpha$。这可理解为在当前参数值周围建立信任域，超出该域则当前梯度估计不再提供充分信息。这通常使事先知道 $\alpha$ 的正确量级相对容易。例如对许多机器学习模型，我们常事先知道良好最优点大概率落在参数空间的某个设定区域内；例如对参数有先验分布并不罕见。由于 $\alpha$ 设定了参数空间中步幅的（上界），我们常可推断 $\alpha$ 的正确数量级，使得能在若干迭代内从 $\theta_0$ 到达最优点。略微滥用术语，我们把比值 $\hat{m}_t/\sqrt{\hat{v}_t}$ 称为信噪比（SNR）。SNR 更小时，有效步长 $\Delta_t$ 更接近零。这是期望的性质，因为更小的 SNR 意味着对 $\hat{m}_t$ 方向是否对应真梯度方向有更大不确定性。例如，SNR 值在接近最优点时通常更接近 0，导致参数空间中有效步更小：一种自动退火。有效步长 $\Delta_t$ 也对梯度尺度不变；用因子 $c$ 缩放梯度 $g$ 会使 $\hat{m}_t$ 缩放 $c$、$\hat{v}_t$ 缩放 $c^2$，二者相消：$(c\cdot\hat{m}_t)/(\sqrt{c^2\cdot\hat{v}_t})=\hat{m}_t/\sqrt{\hat{v}_t}$。

3 Initialization Bias Correction

3 初始化偏差校正

As explained in section 2, Adam utilizes initialization bias correction terms. We will here derive the term for the second moment estimate; the derivation for the first moment estimate is completely analogous. Let $g$ be the gradient of the stochastic objective $f$, and we wish to estimate its second raw moment (uncentered variance) using an exponential moving average of the squared gradient, with decay rate $\beta_2$. Let $g_1,\ldots,g_T$ be the gradients at subsequent timesteps, each a draw from an underlying gradient distribution $g_t\sim p(g_t)$. Let us initialize the exponential moving average as $v_0=0$ (a vector of zeros). First note that the update at timestep $t$ of the exponential moving average $v_t=\beta_2\cdot v_{t-1}+(1-\beta_2)\cdot g_t^2$ (where $g_t^2$ indicates the elementwise square $g_t\odot g_t$) can be written as a function of the gradients at all previous timesteps: $v_t=(1-\beta_2)\sum_{i=1}^{t}\beta_2^{t-i}\cdot g_i^2$ (1).

如第 2 节所述，Adam 使用初始化偏差校正项。这里我们推导二阶矩估计的该项；一阶矩估计的推导完全类似。令 $g$ 为随机目标 $f$ 的梯度，我们希望用平方梯度的指数滑动平均（衰减率 $\beta_2$）估计其二阶原始矩（未中心化方差）。令 $g_1,\ldots,g_T$ 为相继时间步的梯度，各自抽自底层梯度分布 $g_t\sim p(g_t)$。将指数滑动平均初始化为 $v_0=0$（零向量）。首先注意，时间步 $t$ 的更新 $v_t=\beta_2\cdot v_{t-1}+(1-\beta_2)\cdot g_t^2$（其中 $g_t^2$ 表示逐元素平方 $g_t\odot g_t$）可写成此前所有时间步梯度的函数：$v_t=(1-\beta_2)\sum_{i=1}^{t}\beta_2^{t-i}\cdot g_i^2$ (1)。

We wish to know how $\mathbb{E}[v_t]$, the expected value of the exponential moving average at timestep $t$, relates to the true second moment $\mathbb{E}[g_t^2]$, so we can correct for the discrepancy between the two. Taking expectations of the left-hand and right-hand sides of eq. (1): $\mathbb{E}[v_t]=\mathbb{E}[g_t^2]\cdot(1-\beta_2^t)+\zeta$, where $\zeta=0$ if the true second moment $\mathbb{E}[g_i^2]$ is stationary; otherwise $\zeta$ can be kept small since the exponential decay rate $\beta_2$ can (and should) be chosen such that the exponential moving average assigns small weights to gradients too far in the past. What is left is the term $(1-\beta_2^t)$ which is caused by initializing the running average with zeros. In algorithm 1 we therefore divide by this term to correct the initialization bias. In case of sparse gradients, for a reliable estimate of the second moment one needs to average over many gradients by choosing a small value of $(1-\beta_2)$ (i.e. $\beta_2$ close to 1); however it is exactly this case where a lack of initialisation bias correction would lead to initial steps that are much larger.

我们希望知道时间步 $t$ 指数滑动平均的期望 $\mathbb{E}[v_t]$ 与真二阶矩 $\mathbb{E}[g_t^2]$ 如何相关，以便校正二者之间的差异。对式 (1) 左右两边取期望：$\mathbb{E}[v_t]=\mathbb{E}[g_t^2]\cdot(1-\beta_2^t)+\zeta$，其中若真二阶矩 $\mathbb{E}[g_i^2]$ 平稳则 $\zeta=0$；否则可使 $\zeta$ 保持很小，因为指数衰减率 $\beta_2$ 可以（也应）选得使滑动平均对过远过去的梯度赋予小权重。剩下的就是由把运行平均初始化为零所造成的项 $(1-\beta_2^t)$。因此在算法 1 中我们除以该项以校正初始化偏差。在稀疏梯度情形下，为可靠估计二阶矩，需要通过选择很小的 $(1-\beta_2)$（即 $\beta_2$ 接近 1）对许多梯度取平均；而恰恰在这种情形下，缺少初始化偏差校正会导致初始步大得多。

4 Convergence Analysis

4 收敛分析

We analyze the convergence of Adam using the online learning framework proposed in (Zinkevich, 2003). Given an arbitrary, unknown sequence of convex cost functions $f_1(\theta),f_2(\theta),\ldots,f_T(\theta)$. At each time $t$, our goal is to predict the parameter $\theta_t$ and evaluate it on a previously unknown cost function $f_t$. Since the nature of the sequence is unknown in advance, we evaluate our algorithm using the regret, that is the sum of all the previous differences between the online prediction $\theta_t$ and the best fixed parameter $\theta^*$ from a feasible set $\mathcal{X}$ for all the previous steps. Concretely, the regret is defined as: $R(T)=\sum_{t=1}^{T}[f_t(\theta_t)-f_t(\theta^*)]$ where $\theta^*=\arg\min_{\theta\in\mathcal{X}}\sum_{t=1}^{T}f_t(\theta)$. We provide a regret bound that is comparable to the best known bound for other first-order online methods under similar conditions. Our Theorem 4.1 implies when the data features are sparse and gradients are bounded, the summation terms can be much smaller than their upper bounds; adaptive methods such as Adam and AdaGrad can achieve $O(\log d\sqrt{T})$ improvement over the $O(\sqrt{dT})$ bound of non-adaptive methods for sparse features. See the appendix for the full statement and proof.

我们用 (Zinkevich, 2003) 提出的在线学习框架分析 Adam 的收敛。给定任意、未知的凸代价函数序列 $f_1(\theta),f_2(\theta),\ldots,f_T(\theta)$。在每一时刻 $t$，我们的目标是预测参数 $\theta_t$ 并在此前未知的代价函数 $f_t$ 上评估它。由于序列性质事先未知，我们用遗憾评估算法，即此前所有步上在线预测 $\theta_t$ 与可行集 $\mathcal{X}$ 中最佳固定参数 $\theta^*$ 之间差值之和。具体地，遗憾定义为：$R(T)=\sum_{t=1}^{T}[f_t(\theta_t)-f_t(\theta^*)]$，其中 $\theta^*=\arg\min_{\theta\in\mathcal{X}}\sum_{t=1}^{T}f_t(\theta)$。我们给出的遗憾界与其他一阶在线方法在类似条件下的已知最优界可比。我们的定理 4.1 意味着：当数据特征稀疏且梯度有界时，求和项可远小于其上界；Adam 与 AdaGrad 等自适应方法对稀疏特征可相对非自适应方法的 $O(\sqrt{dT})$ 界取得 $O(\log d\sqrt{T})$ 的改进。完整陈述与证明见附录。

5 Related Work

5 相关工作

Optimization methods bearing a direct relation to Adam are RMSProp (Tieleman & Hinton, 2012; Graves, 2013) and AdaGrad (Duchi et al., 2011); these relationships are discussed below. Other stochastic optimization methods include vSGD (Schaul et al., 2012), AdaDelta (Zeiler, 2012) and the natural Newton method from Roux & Fitzgibbon (2010), all setting stepsizes by estimating curvature from first-order information. The Sum-of-Functions Optimizer (SFO) (Sohl-Dickstein et al., 2014) is a quasi-Newton method based on minibatches, but (unlike Adam) has memory requirements linear in the number of minibatch partitions of a dataset, which is often infeasible on memory-constrained systems such as a GPU. Like natural gradient descent (NGD) (Amari, 1998), Adam employs a preconditioner that adapts to the geometry of the data, since $\hat{v}_t$ is an approximation to the diagonal of the Fisher information matrix (Pascanu & Bengio, 2013); however, Adam's preconditioner (like AdaGrad's) is more conservative in its adaption than vanilla NGD by preconditioning with the square root of the inverse of the diagonal Fisher information matrix approximation.

与 Adam 有直接关系的优化方法是 RMSProp（Tieleman & Hinton, 2012；Graves, 2013）与 AdaGrad（Duchi et al., 2011）；下文讨论这些关系。其他随机优化方法包括 vSGD（Schaul et al., 2012）、AdaDelta（Zeiler, 2012）以及 Roux & Fitzgibbon（2010）的自然牛顿法，它们都通过从一阶信息估计曲率来设定步长。Sum-of-Functions Optimizer（SFO）（Sohl-Dickstein et al., 2014）是基于小批量的拟牛顿方法，但（与 Adam 不同）其内存需求与数据集小批量划分数成线性，在 GPU 等内存受限系统上往往不可行。与自然梯度下降（NGD）（Amari, 1998）一样，Adam 使用适应数据几何的预条件子，因为 $\hat{v}_t$ 是 Fisher 信息矩阵对角的近似（Pascanu & Bengio, 2013）；然而 Adam 的预条件子（与 AdaGrad 一样）比朴素 NGD 更保守，它用对角 Fisher 近似之逆的平方根做预条件。

RMSProp. An optimization method closely related to Adam is RMSProp (Tieleman & Hinton, 2012). A version with momentum has sometimes been used (Graves, 2013). There are a few important differences between RMSProp with momentum and Adam: RMSProp with momentum generates its parameter updates using a momentum on the rescaled gradient, whereas Adam updates are directly estimated using a running average of first and second moment of the gradient. RMSProp also lacks a bias-correction term; this matters most in case of a value of $\beta_2$ close to 1 (required in case of sparse gradients), since in that case not correcting the bias leads to very large stepsizes and often divergence, as we also empirically demonstrate in section 6.4.

RMSProp。 与 Adam 密切相关的优化方法是 RMSProp（Tieleman & Hinton, 2012）。有时会使用带动量的版本（Graves, 2013）。带动量的 RMSProp 与 Adam 有几处重要差异：带动量的 RMSProp 通过对重缩放后的梯度施加动量生成参数更新，而 Adam 的更新直接由梯度一阶与二阶矩的运行平均估计。RMSProp 也缺少偏差校正项；在 $\beta_2$ 接近 1（稀疏梯度时需要）时这一点最要紧，因为此时不校正偏差会导致非常大的步长并经常发散，我们也在第 6.4 节经验上证明了这一点。

AdaGrad. An algorithm that works well for sparse gradients is AdaGrad (Duchi et al., 2011). Its basic version updates parameters as $\theta_{t+1}=\theta_t-\alpha\cdot g_t/\sqrt{\sum_{i=1}^{t}g_i^2}$. Note that if we choose $\beta_2$ to be infinitesimally close to 1 from below, then $\lim_{\beta_2\to 1}\hat{v}_t=t^{-1}\cdot\sum_{i=1}^{t}g_i^2$. AdaGrad corresponds to a version of Adam with $\beta_1=0$, infinitesimal $(1-\beta_2)$ and a replacement of $\alpha$ by an annealed version $\alpha_t=\alpha\cdot t^{-1/2}$. Note that this direct correspondence between Adam and AdaGrad does not hold when removing the bias-correction terms; without bias correction, like in RMSProp, a $\beta_2$ infinitesimally close to 1 would lead to infinitely large bias, and infinitely large parameter updates.

AdaGrad。 对稀疏梯度表现良好的算法是 AdaGrad（Duchi et al., 2011）。其基本版本按 $\theta_{t+1}=\theta_t-\alpha\cdot g_t/\sqrt{\sum_{i=1}^{t}g_i^2}$ 更新参数。注意若将 $\beta_2$ 从下方取无穷接近 1，则 $\lim_{\beta_2\to 1}\hat{v}_t=t^{-1}\cdot\sum_{i=1}^{t}g_i^2$。AdaGrad 对应 Adam 的一个版本：$\beta_1=0$、无穷小的 $(1-\beta_2)$，并把 $\alpha$ 换成退火版本 $\alpha_t=\alpha\cdot t^{-1/2}$。注意去掉偏差校正项后 Adam 与 AdaGrad 的这一直接对应不再成立；若无偏差校正（如 RMSProp），$\beta_2$ 无穷接近 1 会导致无穷大偏差与无穷大参数更新。

6 Experiments

6 实验

To empirically evaluate the proposed method, we investigated different popular machine learning models, including logistic regression, multilayer fully connected neural networks and deep convolutional neural networks. Using large models and datasets, we demonstrate Adam can efficiently solve practical deep learning problems. We use the same parameter initialization when comparing different optimization algorithms. The hyper-parameters, such as learning rate and momentum, are searched over a dense grid and the results are reported using the best hyper-parameter setting.

为经验评估所提方法，我们考察了多种流行机器学习模型，包括逻辑回归、多层全连接神经网络与深度卷积神经网络。借助大模型与大数据集，我们表明 Adam 能高效解决实际深度学习问题。比较不同优化算法时使用相同参数初始化。学习率与动量等超参数在稠密网格上搜索，并报告最佳超参数设定下的结果。

6.1 Experiment: Logistic Regression

6.1 实验：逻辑回归

We evaluate our proposed method on $L_2$-regularized multi-class logistic regression using the MNIST dataset. Logistic regression has a well-studied convex objective, making it suitable for comparison of different optimizers without worrying about local minimum issues. The stepsize $\alpha$ in our logistic regression experiments is adjusted by $1/\sqrt{t}$ decay, namely $\alpha_t=\alpha/\sqrt{t}$ that matches with our theoretical prediction from section 4. The logistic regression classifies the class label directly on the 784 dimension image vectors. We compare Adam to accelerated SGD with Nesterov momentum and Adagrad using minibatch size of 128. According to Figure 1, we found that the Adam yields similar convergence as SGD with momentum and both converge faster than Adagrad.

我们在 MNIST 上用 $L_2$ 正则的多类逻辑回归评估所提方法。逻辑回归具有研究充分的凸目标，适合在不必担心局部极小问题的情况下比较不同优化器。逻辑回归实验中的步长 $\alpha$ 按 $1/\sqrt{t}$ 衰减调整，即 $\alpha_t=\alpha/\sqrt{t}$，与第 4 节理论预测一致。逻辑回归直接在 784 维图像向量上分类。我们用小批量大小 128 将 Adam 与带 Nesterov 动量的加速 SGD 以及 Adagrad 比较。根据图 1，我们发现 Adam 与带动量 SGD 收敛相近，二者都快于 Adagrad。

As discussed in (Duchi et al., 2011), Adagrad can efficiently deal with sparse features and gradients as one of its main theoretical results whereas SGD is slow at learning rare features. Adam with $1/\sqrt{t}$ decay on its stepsize should theoretically match the performance of Adagrad. We examine the sparse feature problem using IMDB movie review dataset from (Maas et al., 2011). We pre-process the IMDB movie reviews into bag-of-words (BoW) feature vectors and use dropout noise as the stochastic regularization technique in logistic regression training to prevent over-fitting. In figure 1, Adagrad outperforms SGD with Nesterov momentum by a large margin both with and without dropout noise. Adam converges as fast as Adagrad. The empirical performance of Adam is consistent with our theoretical findings in sections 2 and 4. Similar to Adagrad, Adam can take advantage of sparse features and obtain faster convergence rate than normal SGD with momentum.

如 (Duchi et al., 2011) 所讨论，Adagrad 能高效处理稀疏特征与梯度，这是其主要理论结果之一，而 SGD 学习稀有特征较慢。步长带 $1/\sqrt{t}$ 衰减的 Adam 理论上应与 Adagrad 表现匹配。我们用 (Maas et al., 2011) 的 IMDB 影评数据集考察稀疏特征问题。将 IMDB 影评预处理为词袋（BoW）特征向量，并在逻辑回归训练中用 dropout 噪声作为随机正则以防止过拟合。在图 1 中，无论有无 dropout 噪声，Adagrad 都大幅优于带 Nesterov 动量的 SGD。Adam 与 Adagrad 收敛同样快。Adam 的经验表现与第 2、4 节理论发现一致。与 Adagrad 类似，Adam 能利用稀疏特征，获得比普通带动量 SGD 更快的收敛速率。

Figure 1: Logistic regression training negative log likelihood on MNIST images and IMDB movie reviews with 10,000 bag-of-words (BoW) feature vectors.

图 1： MNIST 图像与带 10,000 维词袋（BoW）特征向量的 IMDB 影评上，逻辑回归训练负对数似然。

6.2 Experiment: Multi-layer Neural Networks

6.2 实验：多层神经网络

Multi-layer neural networks are powerful models with non-convex objective functions. Although our convergence analysis does not apply to non-convex problems, we empirically found that Adam often outperforms other methods in such cases. In our experiments, we made model choices that are consistent with previous publications in the area; a neural network model with two fully connected hidden layers with 1000 hidden units each and ReLU activation are used for this experiment with minibatch size of 128. First, we study different optimizers using the standard deterministic cross-entropy objective function with $L_2$ weight decay on the parameters to prevent over-fitting. The sum-of-functions (SFO) method (Sohl-Dickstein et al., 2014) is a recently proposed quasi-Newton method that works with minibatches of data and has shown good performance on optimization of multi-layer neural networks. We used their implementation and compared with Adam to train such models. Figure 2 shows that Adam makes faster progress in terms of both the number of iterations and wall-clock time. Due to the cost of updating curvature information, SFO is 5–10× slower per iteration compared to Adam, and has a memory requirement that makes it infeasible for larger models. We also evaluate optimizers with dropout stochastic regularization; again Adam makes faster progress than AdaGrad, RMSProp, SGD with Nesterov momentum and AdaDelta.

多层神经网络是具有非凸目标函数的强力模型。尽管我们的收敛分析不适用于非凸问题，我们经验上发现 Adam 在此类情形下往往优于其他方法。实验中我们做了与该领域既往发表一致的模型选择；本实验使用各有 1000 隐单元、ReLU 激活的两层全连接隐层网络，小批量大小 128。首先，我们用带参数 $L_2$ 权重衰减以防止过拟合的标准确定性交叉熵目标函数研究不同优化器。Sum-of-functions（SFO）方法（Sohl-Dickstein et al., 2014）是近期提出的拟牛顿方法，可与数据小批量一起工作，并在多层神经网络优化上表现良好。我们使用其实现并与 Adam 比较以训练此类模型。图 2 显示 Adam 在迭代次数与挂钟时间两方面进展都更快。由于更新曲率信息的代价，SFO 每迭代比 Adam 慢 5–10 倍，且内存需求使其对更大模型不可行。我们也在带 dropout 随机正则下评估优化器；Adam 再次比 AdaGrad、RMSProp、带 Nesterov 动量的 SGD 与 AdaDelta 进展更快。

Figure 2: Training of multilayer neural networks on MNIST images. (a) Neural networks using dropout stochastic regularization. (b) Neural networks with deterministic cost function. We compare with the sum-of-functions (SFO) optimizer (Sohl-Dickstein et al., 2014).

图 2： MNIST 图像上多层神经网络的训练。(a) 使用 dropout 随机正则的神经网络。(b) 带确定性代价函数的神经网络。我们与 sum-of-functions（SFO）优化器（Sohl-Dickstein et al., 2014）比较。

6.3 Experiment: Convolutional Neural Networks

6.3 实验：卷积神经网络

Convolutional neural networks (CNNs) with several layers of convolution, pooling and non-linear units have shown considerable success in computer vision tasks. Unlike most fully connected neural nets, weight sharing in CNNs results in vastly different gradients in different layers. A smaller learning rate for the convolution layers is often used in practice when applying SGD. We show the effectiveness of Adam in deep CNNs. Our CNN architecture has three alternating stages of 5×5 convolution filters and 3×3 max pooling with stride of 2 that are followed by a fully connected layer of 1000 rectified linear hidden units (ReLU's). The input images are pre-processed by whitening, and dropout noise is applied to the input layer and fully connected layer. The minibatch size is also set to 128 similar to previous experiments.

带有若干层卷积、池化与非线性单元的卷积神经网络（CNN）在计算机视觉任务上取得了显著成功。与多数全连接神经网络不同，CNN 中的权值共享导致不同层梯度差异巨大。实践中应用 SGD 时，卷积层常使用更小的学习率。我们展示 Adam 在深度 CNN 上的有效性。我们的 CNN 架构有三个交替的 5×5 卷积滤波器与步长为 2 的 3×3 最大池化阶段，其后接 1000 个整流线性隐单元（ReLU）的全连接层。输入图像经白化预处理，并对输入层与全连接层施加 dropout 噪声。小批量大小同样设为 128。

Interestingly, although both Adam and Adagrad make rapid progress lowering the cost in the initial stage of the training, shown in Figure 3 (left), Adam and SGD eventually converge considerably faster than Adagrad for CNNs shown in Figure 3 (right). We notice the second moment estimate $\hat{v}_t$ vanishes to zeros after a few epochs and is dominated by the $\epsilon$ in algorithm 1. The second moment estimate is therefore a poor approximation to the geometry of the cost function in CNNs comparing to fully connected network from Section 6.2. Whereas, reducing the minibatch variance through the first moment is more important in CNNs and contributes to the speed-up. As a result, Adagrad converges much slower than others in this particular experiment. Though Adam shows marginal improvement over SGD with momentum, it adapts learning rate scale for different layers instead of hand picking manually as in SGD.

有趣的是，尽管 Adam 与 Adagrad 都在训练初期快速降低代价（见图 3 左），对 CNN 而言 Adam 与 SGD 最终比 Adagrad 收敛快得多（见图 3 右）。我们注意到二阶矩估计 $\hat{v}_t$ 在若干 epoch 后趋于零，并被算法 1 中的 $\epsilon$ 主导。因此相对第 6.2 节的全连接网络，该二阶矩估计是对 CNN 代价函数几何的较差近似。而通过一阶矩降低小批量方差在 CNN 中更重要，并贡献于加速。结果是在本实验中 Adagrad 比其他方法收敛慢得多。尽管 Adam 相对带动量的 SGD 仅有边际改进，它能为不同层自适应学习率尺度，而不必像 SGD 那样手工挑选。

Figure 3: Convolutional neural networks training cost. (left) Training cost for the first three epochs. (right) Training cost over 45 epochs. CIFAR-10 with c64-c64-c128-1000 architecture.

图 3： 卷积神经网络训练代价。（左）前三个 epoch 的训练代价。（右）45 个 epoch 上的训练代价。CIFAR-10，架构为 c64-c64-c128-1000。

6.4 Experiment: Bias-Correction Term

6.4 实验：偏差校正项

We also empirically evaluate the effect of the bias correction terms explained in sections 2 and 3. Discussed in section 5, removal of the bias correction terms results in a version of RMSProp (Tieleman & Hinton, 2012) with momentum. We vary the $\beta_1$ and $\beta_2$ when training a variational autoencoder (VAE) with the same architecture as in (Kingma & Welling, 2013) with a single hidden layer with 500 hidden units with softplus nonlinearities and a 50-dimensional spherical Gaussian latent variable. We iterated over a broad range of hyper-parameter choices, i.e. $\beta_1\in\{0,0.9\}$ and $\beta_2\in\{0.99,0.999,0.9999\}$, and $\log_{10}(\alpha)\in[-5,\ldots,-1]$. Values of $\beta_2$ close to 1, required for robustness to sparse gradients, results in larger initialization bias; therefore we expect the bias correction term is important in such cases of slow decay, preventing an adverse effect on optimization. In Figure 4, values $\beta_2$ close to 1 indeed lead to instabilities in training when no bias correction term was present, especially at first few epochs of the training. The best results were achieved with small values of $(1-\beta_2)$ and bias correction; this was more apparent towards the end of optimization when gradients tend to become sparser.

我们也经验评估第 2、3 节所解释的偏差校正项的效果。如第 5 节讨论，去掉偏差校正项会得到带动量的 RMSProp（Tieleman & Hinton, 2012）版本。我们在训练与 (Kingma & Welling, 2013) 相同架构的变分自编码器（VAE）时变化 $\beta_1$ 与 $\beta_2$：单隐层 500 个 softplus 非线性隐单元，以及 50 维球高斯潜变量。我们在广泛超参数选择上迭代，即 $\beta_1\in\{0,0.9\}$、$\beta_2\in\{0.99,0.999,0.9999\}$，以及 $\log_{10}(\alpha)\in[-5,\ldots,-1]$。对稀疏梯度鲁棒所需的接近 1 的 $\beta_2$ 会导致更大初始化偏差；因此我们期望在慢衰减情形下偏差校正项很重要，以防止对优化的不利影响。在图 4 中，当不存在偏差校正项时，接近 1 的 $\beta_2$ 确实导致训练不稳定，尤其在训练最初几个 epoch。最佳结果在小的 $(1-\beta_2)$ 且有偏差校正时取得；这在优化后期梯度趋于更稀疏时更为明显。

Figure 4: Effect of bias-correction terms (red line) versus no bias correction terms (green line) after 10 epochs (left) and 100 epochs (right) on the loss (y-axes) when learning a Variational Auto-Encoder (VAE) (Kingma & Welling, 2013), for different settings of stepsize $\alpha$ (x-axes) and hyperparameters $\beta_1$ and $\beta_2$.

图 4： 学习变分自编码器（VAE）（Kingma & Welling, 2013）时，偏差校正项（红线）相对无偏差校正项（绿线）在 10 个 epoch（左）与 100 个 epoch（右）后对损失（纵轴）的影响，对应不同步长 $\alpha$（横轴）与超参数 $\beta_1$、$\beta_2$ 设定。

7 Extensions

7 扩展

7.1 AdaMax

In Adam, the update rule for individual weights is to scale their gradients inversely proportional to a (scaled) $L_2$ norm of their individual current and past gradients. We can generalize the $L_2$ norm based update rule to a $L_p$ norm based update rule. Such variants become numerically unstable for large $p$. However, in the special case where we let $p\to\infty$, a surprisingly simple and stable algorithm emerges; see algorithm 2. We'll now derive the algorithm. Let, in case of the $L_p$ norm, the stepsize at time $t$ be inversely proportional to $(v_t)^{1/p}$, where $v_t=\beta_2^p\,v_{t-1}+(1-\beta_2^p)|g_t|^p=(1-\beta_2^p)\sum_{i=1}^{t}\beta_2^{p(t-i)}\cdot|g_i|^p$.

在 Adam 中，各个权值的更新规则是按其当前与过去梯度的（缩放）$L_2$ 范数反比缩放梯度。我们可以把基于 $L_2$ 范数的更新规则推广为基于 $L_p$ 范数的更新规则。这类变体在大 $p$ 时会数值不稳定。然而在令 $p\to\infty$ 的特殊情形下，会出现一个出人意料地简单且稳定的算法；见算法 2。我们现在推导该算法。在 $L_p$ 范数情形下，令时刻 $t$ 的步长反比于 $(v_t)^{1/p}$，其中 $v_t=\beta_2^p\,v_{t-1}+(1-\beta_2^p)|g_t|^p=(1-\beta_2^p)\sum_{i=1}^{t}\beta_2^{p(t-i)}\cdot|g_i|^p$。

Algorithm 2: AdaMax, a variant of Adam based on the infinity norm. See section 7.1 for details. Good default settings for the tested machine learning problems are $\alpha=0.002$, $\beta_1=0.9$ and $\beta_2=0.999$. With $\beta_1^t$ we denote $\beta_1$ to the power $t$. Here, $(\alpha/(1-\beta_1^t))$ is the learning rate with the bias-correction term for the first moment. All operations on vectors are element-wise.

Require: $\alpha$: Stepsize
Require: $\beta_1,\beta_2\in[0,1)$: Exponential decay rates
Require: $f(\theta)$: Stochastic objective function with parameters $\theta$
Require: $\theta_0$: Initial parameter vector
$m_0\leftarrow 0$ (Initialize 1st moment vector)
$u_0\leftarrow 0$ (Initialize the exponentially weighted infinity norm)
$t\leftarrow 0$ (Initialize timestep)
while $\theta_t$ not converged do
 $t\leftarrow t+1$
 $g_t\leftarrow\nabla_\theta f_t(\theta_{t-1})$ (Get gradients w.r.t. stochastic objective at timestep $t$)
 $m_t\leftarrow\beta_1\cdot m_{t-1}+(1-\beta_1)\cdot g_t$ (Update biased first moment estimate)
 $u_t\leftarrow\max(\beta_2\cdot u_{t-1},|g_t|)$ (Update the exponentially weighted infinity norm)
 $\theta_t\leftarrow\theta_{t-1}-(\alpha/(1-\beta_1^t))\cdot m_t/u_t$ (Update parameters)
end while
return $\theta_t$ (Resulting parameters)

算法 2：AdaMax，基于无穷范数的 Adam 变体。细节见第 7.1 节。对所测机器学习问题，良好默认设置为 $\alpha=0.002$、$\beta_1=0.9$ 与 $\beta_2=0.999$。用 $\beta_1^t$ 表示 $\beta_1$ 的 $t$ 次幂。这里 $(\alpha/(1-\beta_1^t))$ 是带一阶矩偏差校正项的学习率。对向量的所有运算均为逐元素。

要求：$\alpha$：步长
要求：$\beta_1,\beta_2\in[0,1)$：指数衰减率
要求：$f(\theta)$：带参数 $\theta$ 的随机目标函数
要求：$\theta_0$：初始参数向量
$m_0\leftarrow 0$（初始化一阶矩向量）
$u_0\leftarrow 0$（初始化指数加权无穷范数）
$t\leftarrow 0$（初始化时间步）
while $\theta_t$ 未收敛 do
 $t\leftarrow t+1$
 $g_t\leftarrow\nabla_\theta f_t(\theta_{t-1})$（在时间步 $t$ 求相对随机目标的梯度）
 $m_t\leftarrow\beta_1\cdot m_{t-1}+(1-\beta_1)\cdot g_t$（更新有偏一阶矩估计）
 $u_t\leftarrow\max(\beta_2\cdot u_{t-1},|g_t|)$（更新指数加权无穷范数）
 $\theta_t\leftarrow\theta_{t-1}-(\alpha/(1-\beta_1^t))\cdot m_t/u_t$（更新参数）
end while
return $\theta_t$（结果参数）

Note that the decay term is here equivalently parameterised as $\beta_2^p$ instead of $\beta_2$. Now let $p\to\infty$, and define $u_t=\lim_{p\to\infty}(v_t)^{1/p}$, then $u_t=\max(\beta_2^{t-1}|g_1|,\beta_2^{t-2}|g_2|,\ldots,\beta_2|g_{t-1}|,|g_t|)$, which corresponds to the remarkably simple recursive formula $u_t=\max(\beta_2\cdot u_{t-1},|g_t|)$ with initial value $u_0=0$. Note that, conveniently enough, we don't need to correct for initialization bias in this case. Also note that the magnitude of parameter updates has a simpler bound with AdaMax than Adam, namely $|\Delta_t|\leq\alpha$.

注意这里衰减项等价地参数化为 $\beta_2^p$ 而非 $\beta_2$。现在令 $p\to\infty$，并定义 $u_t=\lim_{p\to\infty}(v_t)^{1/p}$，则 $u_t=\max(\beta_2^{t-1}|g_1|,\beta_2^{t-2}|g_2|,\ldots,\beta_2|g_{t-1}|,|g_t|)$，这对应异常简单的递归公式 $u_t=\max(\beta_2\cdot u_{t-1},|g_t|)$，初值 $u_0=0$。注意，方便的是，此时我们不必校正初始化偏差。还要注意，AdaMax 参数更新幅度的上界比 Adam 更简单，即 $|\Delta_t|\leq\alpha$。

7.2 Temporal Averaging

7.2 时间平均

Since the last iterate is noisy due to stochastic approximation, better generalization performance is often achieved by averaging. Previously in Moulines & Bach (2011), Polyak–Ruppert averaging (Polyak & Juditsky, 1992; Ruppert, 1988) has been shown to improve the convergence of standard SGD, where $\bar{\theta}_t=\frac{1}{t}\sum_{k=1}^{t}\theta_k$. Alternatively, an exponential moving average over the parameters can be used, giving higher weight to more recent parameter values. This can be trivially implemented by adding one line to the inner loop of algorithms 1 and 2: $\bar{\theta}_t\leftarrow\beta_2\cdot\bar{\theta}_{t-1}+(1-\beta_2)\theta_t$, with $\bar{\theta}_0=0$. Initialization bias can again be corrected by the estimator $\hat{\theta}_t=\bar{\theta}_t/(1-\beta_2^t)$.

由于随机逼近，最后一次迭代有噪声，平均往往能取得更好泛化性能。此前在 Moulines & Bach（2011）中，Polyak–Ruppert 平均（Polyak & Juditsky, 1992；Ruppert, 1988）被证明可改善标准 SGD 的收敛，其中 $\bar{\theta}_t=\frac{1}{t}\sum_{k=1}^{t}\theta_k$。或者，可对参数使用指数滑动平均，给更新近的参数值更高权重。这可通过在算法 1 与 2 的内循环加一行平凡实现：$\bar{\theta}_t\leftarrow\beta_2\cdot\bar{\theta}_{t-1}+(1-\beta_2)\theta_t$，且 $\bar{\theta}_0=0$。初始化偏差同样可用估计量 $\hat{\theta}_t=\bar{\theta}_t/(1-\beta_2^t)$ 校正。

8 Conclusion

8 结论

We have introduced a simple and computationally efficient algorithm for gradient-based optimization of stochastic objective functions. Our method is aimed towards machine learning problems with large datasets and/or high-dimensional parameter spaces. The method combines the advantages of two recently popular optimization methods: the ability of AdaGrad to deal with sparse gradients, and the ability of RMSProp to deal with non-stationary objectives. The method is straightforward to implement and requires little memory. The experiments confirm the analysis on the rate of convergence in convex problems. Overall, we found Adam to be robust and well-suited to a wide range of non-convex optimization problems in the field of machine learning.

我们提出了一种简单且计算高效的、用于随机目标函数基于梯度优化的算法。我们的方法面向带大数据集与/或高维参数空间的机器学习问题。该方法结合了两种近期流行优化方法的优点：AdaGrad 处理稀疏梯度的能力，以及 RMSProp 处理非平稳目标的能力。该方法实现直接且内存需求小。实验确认了凸问题中关于收敛速率的分析。总体而言，我们发现 Adam 稳健，并很好适用于机器学习领域中广泛的非凸优化问题。

9 Acknowledgments

9 致谢

This paper would probably not have existed without the support of Google Deepmind. We would like to give special thanks to Ivo Danihelka, and Tom Schaul for coining the name Adam. Thanks to Kai Fan from Duke University for spotting an error in the original AdaMax derivation. Experiments in this work were partly carried out on the Dutch national e-infrastructure with the support of SURF Foundation. Diederik Kingma is supported by the Google European Doctorate Fellowship in Deep Learning.

若无 Google Deepmind 的支持，本文很可能不会出现。我们特别感谢 Ivo Danihelka 与 Tom Schaul 为 Adam 起名。感谢杜克大学的 Kai Fan 指出原 AdaMax 推导中的一处错误。本工作中的实验部分在 SURF Foundation 支持下于荷兰国家电子基础设施上完成。Diederik Kingma 受 Google European Doctorate Fellowship in Deep Learning 资助。

References

参考文献

Amari, Shun-Ichi. Natural gradient works efficiently in learning. Neural computation, 10(2):251–276, 1998.

Deng, Li, Li, Jinyu, Huang, Jui-Ting, Yao, Kaisheng, Yu, Dong, Seide, Frank, Seltzer, Michael, Zweig, Geoff, He, Xiaodong, Williams, Jason, et al. Recent advances in deep learning for speech research at Microsoft. ICASSP 2013, 2013.

Deng, Li 等. Recent advances in deep learning for speech research at Microsoft. ICASSP 2013, 2013.

Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011.

Graves, Alex. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.

Graves, Alex, Mohamed, Abdel-rahman, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6645–6649. IEEE, 2013.

Graves, Alex, Mohamed, Abdel-rahman, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. ICASSP 2013, pp. 6645–6649. IEEE, 2013.

Hinton, G.E. and Salakhutdinov, R.R. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006.

Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George E, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara N, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE, 29(6):82–97, 2012a.

Hinton, Geoffrey 等. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, 2012a.

Hinton, Geoffrey E, Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012b.

Hinton, Geoffrey E 等. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012b.

Kingma, Diederik P and Welling, Max. Auto-Encoding Variational Bayes. In The 2nd International Conference on Learning Representations (ICLR), 2013.

Kingma, Diederik P and Welling, Max. Auto-Encoding Variational Bayes. ICLR 2013.

Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.

Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. NeurIPS 2012, pp. 1097–1105.

Maas, Andrew L, Daly, Raymond E, Pham, Peter T, Huang, Dan, Ng, Andrew Y, and Potts, Christopher. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150, 2011.

Maas, Andrew L 等. Learning word vectors for sentiment analysis. ACL HLT 2011, pp. 142–150.

Moulines, Eric and Bach, Francis R. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In Advances in Neural Information Processing Systems, pp. 451–459, 2011.

Moulines, Eric and Bach, Francis R. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. NeurIPS 2011, pp. 451–459.

Pascanu, Razvan and Bengio, Yoshua. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013.

Polyak, Boris T and Juditsky, Anatoli B. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992.

Roux, Nicolas L and Fitzgibbon, Andrew W. A fast natural Newton method. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 623–630, 2010.

Roux, Nicolas L and Fitzgibbon, Andrew W. A fast natural Newton method. ICML 2010, pp. 623–630.

Ruppert, David. Efficient estimations from a slowly convergent Robbins-Monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988.

Ruppert, David. Efficient estimations from a slowly convergent Robbins-Monro process. Technical report, Cornell University, 1988.

Schaul, Tom, Zhang, Sixin, and LeCun, Yann. No more pesky learning rates. arXiv preprint arXiv:1206.1106, 2012.

Sohl-Dickstein, Jascha, Poole, Ben, and Ganguli, Surya. Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 604–612, 2014.

Sohl-Dickstein, Jascha, Poole, Ben, and Ganguli, Surya. Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods. ICML 2014, pp. 604–612.

Sutskever, Ilya, Martens, James, Dahl, George, and Hinton, Geoffrey. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 1139–1147, 2013.

Sutskever, Ilya, Martens, James, Dahl, George, and Hinton, Geoffrey. On the importance of initialization and momentum in deep learning. ICML 2013, pp. 1139–1147.

Tieleman, T. and Hinton, G. Lecture 6.5 - RMSProp, COURSERA: Neural Networks for Machine Learning. Technical report, 2012.

Wang, Sida and Manning, Christopher. Fast dropout training. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 118–126, 2013.

Wang, Sida and Manning, Christopher. Fast dropout training. ICML 2013, pp. 118–126.

Zeiler, Matthew D. ADADELTA: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.

Zinkevich, Martin. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the Twentieth International Conference on Machine Learning (ICML), 2003.

Zinkevich, Martin. Online convex programming and generalized infinitesimal gradient ascent. ICML 2003.

10 Appendix

10 附录

10.1 Convergence Proof

10.1 收敛证明

Definition 10.1. A function $f:\mathbb{R}^d\to\mathbb{R}$ is convex if for all $x,y\in\mathbb{R}^d$, for all $\lambda\in[0,1]$, $\lambda f(x)+(1-\lambda)f(y)\geq f(\lambda x+(1-\lambda)y)$.

定义 10.1。 函数 $f:\mathbb{R}^d\to\mathbb{R}$ 是凸的，若对所有 $x,y\in\mathbb{R}^d$、所有 $\lambda\in[0,1]$，有 $\lambda f(x)+(1-\lambda)f(y)\geq f(\lambda x+(1-\lambda)y)$。

Lemma 10.2. If a function $f:\mathbb{R}^d\to\mathbb{R}$ is convex, then for all $x,y\in\mathbb{R}^d$, $f(y)\geq f(x)+\nabla f(x)^T(y-x)$. The above lemma can be used to upper bound the regret; our proof of the main theorem is constructed by substituting the hyperplane with the Adam update rules.

引理 10.2。 若函数 $f:\mathbb{R}^d\to\mathbb{R}$ 凸，则对所有 $x,y\in\mathbb{R}^d$，有 $f(y)\geq f(x)+\nabla f(x)^T(y-x)$。上述引理可用于上界遗憾；主定理证明通过用 Adam 更新规则替换超平面来构造。

Lemma 10.3. Let $g_t=\nabla f_t(\theta_t)$ and $g_{1:T,i}=(g_{1,i},g_{2,i},\ldots,g_{T,i})$. Assume that $\|g_t\|_\infty\leq G_\infty$. Then $\sum_{t=1}^{T}g_{t,i}^2/\sqrt{t}\leq 2G_\infty\|g_{1:T,i}\|_2$.

引理 10.3。 令 $g_t=\nabla f_t(\theta_t)$ 且 $g_{1:T,i}=(g_{1,i},g_{2,i},\ldots,g_{T,i})$。假定 $\|g_t\|_\infty\leq G_\infty$。则 $\sum_{t=1}^{T}g_{t,i}^2/\sqrt{t}\leq 2G_\infty\|g_{1:T,i}\|_2$。

Lemma 10.4. Let $\gamma=\beta_1^2/\sqrt{\beta_2}$. For $\beta_1,\beta_2\in[0,1)$ that satisfy $\gamma<1$ and bounded $g_t$ with $\|g_t\|_2\leq G$, $\|g_t\|_\infty\leq G_\infty$, the following inequality holds: $\sum_{t=1}^{T}\hat{m}_{t,i}^2/\sqrt{t\hat{v}_{t,i}}\leq\frac{2}{1-\gamma}\frac{1}{\sqrt{1-\beta_2}}\|g_{1:T,i}\|_2$.

引理 10.4。 令 $\gamma=\beta_1^2/\sqrt{\beta_2}$。对满足 $\gamma<1$ 且梯度有界 $\|g_t\|_2\leq G$、$\|g_t\|_\infty\leq G_\infty$ 的 $\beta_1,\beta_2\in[0,1)$，有：$\sum_{t=1}^{T}\hat{m}_{t,i}^2/\sqrt{t\hat{v}_{t,i}}\leq\frac{2}{1-\gamma}\frac{1}{\sqrt{1-\beta_2}}\|g_{1:T,i}\|_2$。

Theorem 10.5 (Adam regret bound). Assume that the function $f_t$ has bounded gradients, $\|\nabla f_t(\theta)\|_2\leq G$, $\|\nabla f_t(\theta)\|_\infty\leq G_\infty$ for all $\theta\in\mathbb{R}^d$, and the distance between any $\theta_t$ generated by Adam is bounded, $\|\theta_n-\theta_m\|_2\leq D$, $\|\theta_m-\theta_n\|_\infty\leq D_\infty$ for any $m,n\in\{1,\ldots,T\}$, and $\beta_1,\beta_2\in[0,1)$ satisfy $\gamma=\beta_1^2/\sqrt{\beta_2}<1$. Let $\alpha_t=\alpha/\sqrt{t}$ and $\beta_{1,t}=\beta_1\lambda^{t-1}$, $\lambda\in(0,1)$. Adam achieves the following guarantee, for all $T\geq 1$:

$$R(T)\leq\frac{D^2}{2\alpha(1-\beta_1)}\sum_{i=1}^{d}\sqrt{T\hat{v}_{T,i}}+\frac{\alpha(\beta_1+1)G_\infty}{(1-\beta_1)\sqrt{1-\beta_2}(1-\gamma)^2}\sum_{i=1}^{d}\|g_{1:T,i}\|_2+\sum_{i=1}^{d}\frac{D_\infty^2 G_\infty\sqrt{1-\beta_2}}{2\alpha(1-\beta_1)(1-\lambda)^2}.$$

The proof proceeds by applying Lemma 10.2 to bound $f_t(\theta_t)-f_t(\theta^*)$ by an inner product with $g_t$, substituting the Adam update, applying Young's inequality, and summing over dimensions and timesteps with Lemmas 10.3–10.4 and an arithmetic–geometric series bound on the $\beta_{1,t}$ terms. When features are sparse and gradients bounded, the summation terms can be much smaller than their worst-case upper bounds, yielding the adaptive improvement discussed in section 4.

定理 10.5（Adam 遗憾界）。假定函数 $f_t$ 有有界梯度，$\|\nabla f_t(\theta)\|_2\leq G$、$\|\nabla f_t(\theta)\|_\infty\leq G_\infty$ 对所有 $\theta\in\mathbb{R}^d$ 成立，且 Adam 生成的任意 $\theta_t$ 之间距离有界，$\|\theta_n-\theta_m\|_2\leq D$、$\|\theta_m-\theta_n\|_\infty\leq D_\infty$ 对任意 $m,n\in\{1,\ldots,T\}$ 成立，且 $\beta_1,\beta_2\in[0,1)$ 满足 $\gamma=\beta_1^2/\sqrt{\beta_2}<1$。令 $\alpha_t=\alpha/\sqrt{t}$ 且 $\beta_{1,t}=\beta_1\lambda^{t-1}$，$\lambda\in(0,1)$。则对所有 $T\geq 1$，Adam 达到如下保证：

$$R(T)\leq\frac{D^2}{2\alpha(1-\beta_1)}\sum_{i=1}^{d}\sqrt{T\hat{v}_{T,i}}+\frac{\alpha(\beta_1+1)G_\infty}{(1-\beta_1)\sqrt{1-\beta_2}(1-\gamma)^2}\sum_{i=1}^{d}\|g_{1:T,i}\|_2+\sum_{i=1}^{d}\frac{D_\infty^2 G_\infty\sqrt{1-\beta_2}}{2\alpha(1-\beta_1)(1-\lambda)^2}.$$

证明通过应用引理 10.2 将 $f_t(\theta_t)-f_t(\theta^*)$ 用与 $g_t$ 的内积上界，代入 Adam 更新，应用 Young 不等式，并用引理 10.3–10.4 以及对 $\beta_{1,t}$ 项的算术–几何级数界对维度与时间步求和。当特征稀疏且梯度有界时，求和项可远小于最坏情形上界，从而得到第 4 节讨论的自适应改进。