Generative Adversarial NetsThanks: Sherjil Ozair is visiting Université de Montréal from Indian Institute of Technology DelhiThanks: Yoshua Bengio is a CIFAR Senior Fellow.

Generative Adversarial NetsThanks: Sherjil Ozair is visiting Université de Montréal from Indian Institute of Technology DelhiThanks: Yoshua Bengio is a CIFAR Senior Fellow.

Ian J. Goodfellow    Jean Pouget-Abadie Thanks:  Jean Pouget-Abadie is visiting Université de Montréal from Ecole Polytechnique.    Mehdi Mirza    Bing Xu    David Warde-Farley Affiliation:  Sherjil Ozair, Aaron Courville, Yoshua Bengio Affiliation:  Département d’informatique et de recherche opérationnelle Affiliation:  Université de Montréal Affiliation:  Montréal, QC H3C 3J7

Abstract

摘要

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model $G$ that captures the data distribution, and a discriminative model $D$ that estimates the probability that a sample came from the training data rather than $G$. The training procedure for $G$ is to maximize the probability of $D$ making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions $G$ and $D$, a unique solution exists, with $G$ recovering the training data distribution and $D$ equal to $\frac{1}{2}$ everywhere. In the case where $G$ and $D$ are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

我们提出一种通过对抗过程估计生成模型的新框架：同时训练两个模型——捕捉数据分布的生成模型 $G$，以及估计样本来自训练数据而非 $G$ 的概率的判别模型 $D$。$G$ 的训练目标是最大化 $D$ 出错的概率。该框架对应一个极小极大二人博弈。在任意函数空间中，$G$ 与 $D$ 存在唯一解：$G$ 恢复训练数据分布，且 $D$ 处处等于 $\frac{1}{2}$。当 $G$ 与 $D$ 由多层感知机定义时，整个系统可用反向传播训练；训练与采样都不需要马尔可夫链，也不需要展开的近似推断网络。实验从定性与定量两方面展示了该框架的潜力。

1 Introduction

1 引言

The promise of deep learning is to discover rich, hierarchical models [2] that represent probability distributions over the kinds of data encountered in artificial intelligence applications, such as natural images, audio waveforms containing speech, and symbols in natural language corpora. So far, the most striking successes in deep learning have involved discriminative models, usually those that map a high-dimensional, rich sensory input to a class label [14, 22]. These striking successes have primarily been based on the backpropagation and dropout algorithms, using piecewise linear units  [19, 9, 10] which have a particularly well-behaved gradient . Deep generative models have had less of an impact, due to the difficulty of approximating many intractable probabilistic computations that arise in maximum likelihood estimation and related strategies, and due to difficulty of leveraging the benefits of piecewise linear units in the generative context. We propose a new generative model estimation procedure that sidesteps these difficulties. [^1]

深度学习的承诺是发现丰富的分层模型 [2]，用以表示人工智能应用中常见数据上的概率分布，例如自然图像、含语音的音频波形，以及自然语言语料中的符号。迄今最引人注目的成功主要来自判别模型，通常是把高维、丰富的感官输入映射到类别标签 [14, 22]。这些成功主要依赖反向传播与 dropout，以及梯度性质良好的分段线性单元 [19, 9, 10]。深度生成模型影响较小，一是因为最大似然及相关策略中大量难解的概率计算难以近似，二是因为在生成场景中难以发挥分段线性单元的优势。我们提出一种新的生成模型估计流程，绕开这些困难。[^1]

In the proposed adversarial nets framework, the generative model is pitted against an adversary: a discriminative model that learns to determine whether a sample is from the model distribution or the data distribution. The generative model can be thought of as analogous to a team of counterfeiters, trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. Competition in this game drives both teams to improve their methods until the counterfeits are indistiguishable from the genuine articles.

在所提出的对抗网络框架中，生成模型面对一个对手：学会判断样本来自模型分布还是数据分布的判别模型。可以把生成模型比作造假团伙，试图制造并混用假币而不被察觉；判别模型则好比查伪一方，试图识别假币。双方竞争促使方法不断改进，直到假币与真币难以区分。

This framework can yield specific training algorithms for many kinds of model and optimization algorithm. In this article, we explore the special case when the generative model generates samples by passing random noise through a multilayer perceptron, and the discriminative model is also a multilayer perceptron. We refer to this special case as adversarial nets. In this case, we can train both models using only the highly successful backpropagation and dropout algorithms [17] and sample from the generative model using only forward propagation. No approximate inference or Markov chains are necessary.

该框架可为多种模型与优化算法导出具体训练过程。本文考察一种特例：生成模型把随机噪声送过多层感知机得到样本，判别模型也是多层感知机。我们称此特例为对抗网络。此时两模型只需高度成功的反向传播与 dropout [17] 即可训练，生成样本只需前向传播。不需要近似推断，也不需要马尔可夫链。

2 Related work

2 相关工作

An alternative to directed graphical models with latent variables are undirected graphical models with latent variables, such as restricted Boltzmann machines (RBMs) [27, 16], deep Boltzmann machines (DBMs) [26] and their numerous variants. The interactions within such models are represented as the product of unnormalized potential functions, normalized by a global summation/integration over all states of the random variables. This quantity (the partition function) and its gradient are intractable for all but the most trivial instances, although they can be estimated by Markov chain Monte Carlo (MCMC) methods. Mixing poses a significant problem for learning algorithms that rely on MCMC [3, 5].

带隐变量的有向图模型之外，还有带隐变量的无向图模型，例如受限玻尔兹曼机（RBM） [27, 16]、深度玻尔兹曼机（DBM） [26] 及其诸多变体。此类模型中的相互作用表示为未归一化势函数之积，再对随机变量所有状态做全局求和/积分归一化。该量（配分函数）及其梯度除最平凡情形外均难解，虽可用马尔可夫链蒙特卡洛（MCMC）估计。依赖 MCMC 的学习算法常受混合困难困扰 [3, 5]。

Deep belief networks (DBNs) [16] are hybrid models containing a single undirected layer and several directed layers. While a fast approximate layer-wise training criterion exists, DBNs incur the computational difficulties associated with both undirected and directed models.

深度信念网络（DBN） [16] 是混合模型，含一层无向层与若干有向层。虽有快速近似的逐层训练准则，DBN 仍同时承受无向与有向模型的计算困难。

Alternative criteria that do not approximate or bound the log-likelihood have also been proposed, such as score matching [18] and noise-contrastive estimation (NCE) [13]. Both of these require the learned probability density to be analytically specified up to a normalization constant. Note that in many interesting generative models with several layers of latent variables (such as DBNs and DBMs), it is not even possible to derive a tractable unnormalized probability density. Some models such as denoising auto-encoders [30] and contractive autoencoders have learning rules very similar to score matching applied to RBMs. In NCE, as in this work, a discriminative training criterion is employed to fit a generative model. However, rather than fitting a separate discriminative model, the generative model itself is used to discriminate generated data from samples a fixed noise distribution. Because NCE uses a fixed noise distribution, learning slows dramatically after the model has learned even an approximately correct distribution over a small subset of the observed variables.

也有不近似或不界住对数似然的替代准则，例如 score matching [18] 与噪声对比估计（NCE） [13]。二者都要求所学概率密度在归一化常数意义下可解析写出。注意许多含多层隐变量的有趣生成模型（如 DBN 与 DBM）甚至推不出可处理的未归一化密度。去噪自编码器 [30] 与收缩自编码器等模型的学习规则，与把 score matching 用于 RBM 非常相似。NCE 与本工作一样用判别训练准则来拟合生成模型；但并非另训一个判别模型，而是让生成模型自身把生成数据与来自固定噪声分布的样本区分开。由于 NCE 使用固定噪声分布，一旦模型在观测变量的一小部分上已学到大致正确的分布，学习会急剧变慢。

Finally, some techniques do not involve defining a probability distribution explicitly, but rather train a generative machine to draw samples from the desired distribution. This approach has the advantage that such machines can be designed to be trained by back-propagation. Prominent recent work in this area includes the generative stochastic network (GSN) framework [5], which extends generalized denoising auto-encoders [4]: both can be seen as defining a parameterized Markov chain, i.e., one learns the parameters of a machine that performs one step of a generative Markov chain. Compared to GSNs, the adversarial nets framework does not require a Markov chain for sampling. Because adversarial nets do not require feedback loops during generation, they are better able to leverage piecewise linear units [19, 9, 10], which improve the performance of backpropagation but have problems with unbounded activation when used ina feedback loop. More recent examples of training a generative machine by back-propagating into it include recent work on auto-encoding variational Bayes [20] and stochastic backpropagation [24].

最后，有些方法并不显式定义概率分布，而是训练一台生成机器从目标分布中抽样。其优点是可用反向传播训练。该方向的近期代表包括生成随机网络（GSN）框架 [5]，它推广了广义去噪自编码器 [4]：二者都可视为定义一条参数化马尔可夫链，即学习执行生成马尔可夫链一步的机器参数。相对 GSN，对抗网络框架采样时不需要马尔可夫链。由于生成阶段无需反馈环，对抗网络更能发挥分段线性单元 [19, 9, 10] 的作用——这类单元改善反向传播表现，但在反馈环中因激活无界而有问题。通过反向传播训练生成机器的更新例子还包括自编码变分贝叶斯 [20] 与随机反向传播 [24]。

3 Adversarial nets

3 对抗网络

The adversarial modeling framework is most straightforward to apply when the models are both multilayer perceptrons. To learn the generator’s distribution $p_{g}$ over data $\bm{x}$, we define a prior on input noise variables $p_{\bm{z}}(\bm{z})$, then represent a mapping to data space as $G(\bm{z};\theta_{g})$, where $G$ is a differentiable function represented by a multilayer perceptron with parameters $\theta_{g}$. We also define a second multilayer perceptron $D(\bm{x};\theta_{d})$ that outputs a single scalar. $D(\bm{x})$ represents the probability that $\bm{x}$ came from the data rather than $p_{g}$. We train $D$ to maximize the probability of assigning the correct label to both training examples and samples from $G$. We simultaneously train $G$ to minimize $\log(1-D(G(\bm{z})))$:

当两个模型都是多层感知机时，对抗建模框架最易应用。为学习生成器在数据 $\bm{x}$ 上的分布 $p_{g}$，我们在输入噪声变量上定义先验 $p_{\bm{z}}(\bm{z})$，再把到数据空间的映射写成 $G(\bm{z};\theta_{g})$，其中 $G$ 是参数为 $\theta_{g}$ 的可微多层感知机。另定义输出标量的第二多层感知机 $D(\bm{x};\theta_{d})$。$D(\bm{x})$ 表示 $\bm{x}$ 来自数据而非 $p_{g}$ 的概率。我们训练 $D$，使它对训练样本与来自 $G$ 的样本都尽量标对；同时训练 $G$，最小化 $\log(1-D(G(\bm{z})))$：

In other words, $D$ and $G$ play the following two-player minimax game with value function $V(G,D)$:

换言之，$D$ 与 $G$ 进行如下价值函数为 $V(G,D)$ 的二人极小极大博弈：

$$\min_{G}\max_{D}V(D,G)=\mathbb{E}_{\bm{x}\sim p_{\text{data}}(\bm{x})}[\log D(\bm{x})]+\mathbb{E}_{\bm{z}\sim p_{\bm{z}}(\bm{z})}[\log(1-D(G(\bm{z})))]. \tag{1}$$

In the next section, we present a theoretical analysis of adversarial nets, essentially showing that the training criterion allows one to recover the data generating distribution as $G$ and $D$ are given enough capacity, i.e., in the non-parametric limit. See Figure 1 for a less formal, more pedagogical explanation of the approach. In practice, we must implement the game using an iterative, numerical approach. Optimizing $D$ to completion in the inner loop of training is computationally prohibitive, and on finite datasets would result in overfitting. Instead, we alternate between $k$ steps of optimizing $D$ and one step of optimizing $G$. This results in $D$ being maintained near its optimal solution, so long as $G$ changes slowly enough. This strategy is analogous to the way that SML/PCD [31, 29] training maintains samples from a Markov chain from one learning step to the next in order to avoid burning in a Markov chain as part of the inner loop of learning. The procedure is formally presented in Algorithm 1.

下一节给出对抗网络的理论分析，本质上说明：在 $G$ 与 $D$ 容量足够（非参数极限）时，该训练准则可恢复数据生成分布。图 1 以更直观、教学性的方式解释该方法。实践中必须用迭代数值方法实现博弈。在内环把 $D$ 优化到极致计算代价过高，且在有限数据集上会过拟合。我们改为交替进行 $k$ 步优化 $D$ 与一步优化 $G$。只要 $G$ 变化足够慢，$D$ 就能维持在接近最优解附近。这类似于 SML/PCD [31, 29] 训练在相邻学习步之间保留马尔可夫链样本，以避免在学习内环中反复烧入。正式流程见算法 1。

In practice, equation 1 may not provide sufficient gradient for $G$ to learn well. Early in learning, when $G$ is poor, $D$ can reject samples with high confidence because they are clearly different from the training data. In this case, $\log(1-D(G(\bm{z})))$ saturates. Rather than training $G$ to minimize $\log(1-D(G(\bm{z})))$ we can train $G$ to maximize $\log D(G(\bm{z}))$. This objective function results in the same fixed point of the dynamics of $G$ and $D$ but provides much stronger gradients early in learning.

实践中，式 1 可能无法给 $G$ 提供足够梯度。学习早期 $G$ 较差时，$D$ 能高置信拒绝明显不像训练数据的样本，此时 $\log(1-D(G(\bm{z})))$ 饱和。与其让 $G$ 最小化 $\log(1-D(G(\bm{z})))$，不如让 $G$ 最大化 $\log D(G(\bm{z}))$。该目标与原动力学有相同不动点，但在学习早期提供强得多的梯度。

			…
(a)	(b)	(c)		(d)

Figure 1: Generative adversarial nets are trained by simultaneously updating the discriminative distribution ($D$, blue, dashed line) so that it discriminates between samples from the data generating distribution (black, dotted line) $p_{\bm{x}}$ from those of the generative distribution $p_{g}$ (G) (green, solid line). The lower horizontal line is the domain from which $\bm{z}$ is sampled, in this case uniformly. The horizontal line above is part of the domain of $\bm{x}$. The upward arrows show how the mapping $\bm{x}=G(\bm{z})$ imposes the non-uniform distribution $p_{g}$ on transformed samples. $G$ contracts in regions of high density and expands in regions of low density of $p_{g}$. (a) Consider an adversarial pair near convergence: $p_{g}$ is similar to $p_{\text{data}}$ and $D$ is a partially accurate classifier. (b) In the inner loop of the algorithm $D$ is trained to discriminate samples from data, converging to $D^{*}(\bm{x})=\frac{p_{\text{data}}(\bm{x})}{p_{\text{data}}(\bm{x})+p_{g}(\bm{x})}$. (c) After an update to $G$, gradient of $D$ has guided $G(\bm{z})$ to flow to regions that are more likely to be classified as data. (d) After several steps of training, if $G$ and $D$ have enough capacity, they will reach a point at which both cannot improve because $p_{g}=p_{\text{data}}$. The discriminator is unable to differentiate between the two distributions, i.e. $D(\bm{x})=\frac{1}{2}$.

			…
(a)	(b)	(c)		(d)

图 1：生成对抗网络通过同时更新判别分布（$D$，蓝色虚线）来训练，使之区分数据生成分布（黑色点线）$p_{\bm{x}}$ 与生成分布 $p_{g}$（G）（绿色实线）的样本。下方水平线是采样 $\bm{z}$ 的定义域，此处为均匀。上方水平线是 $\bm{x}$ 定义域的一部分。向上箭头显示映射 $\bm{x}=G(\bm{z})$ 如何在变换后的样本上施加非均匀分布 $p_{g}$。$G$ 在 $p_{g}$ 高密度区收缩、低密度区扩张。(a) 考虑接近收敛的对抗对：$p_{g}$ 已接近 $p_{\text{data}}$，$D$ 是部分准确的分类器。(b) 算法内环训练 $D$ 区分数据样本，收敛到 $D^{*}(\bm{x})=\frac{p_{\text{data}}(\bm{x})}{p_{\text{data}}(\bm{x})+p_{g}(\bm{x})}$。(c) 更新 $G$ 后，$D$ 的梯度引导 $G(\bm{z})$ 流向更可能被判为数据的区域。(d) 若干步训练后，若 $G$ 与 $D$ 容量足够，将到达双方都无法再改进的点，因为 $p_{g}=p_{\text{data}}$。判别器无法区分两分布，即 $D(\bm{x})=\frac{1}{2}$。

for number of training iterations do
for $k$ steps do
$\bullet$ Sample minibatch of $m$ noise samples $\{\bm{z}^{(1)},\dots,\bm{z}^{(m)}\}$ from noise prior $p_{g}(\bm{z})$.
$\bullet$ Sample minibatch of $m$ examples $\{\bm{x}^{(1)},\dots,\bm{x}^{(m)}\}$ from data generating distribution $p_{\text{data}}(\bm{x})$.
$\bullet$ Update the discriminator by ascending its stochastic gradient: $$\nabla_{\theta_{d}}\frac{1}{m}\sum_{i=1}^{m}\left[\log D\left(\bm{x}^{(i)}\right)+\log\left(1-D\left(G\left(\bm{z}^{(i)}\right)\right)\right)\right].$$
end for
$\bullet$ Sample minibatch of $m$ noise samples $\{\bm{z}^{(1)},\dots,\bm{z}^{(m)}\}$ from noise prior $p_{g}(\bm{z})$.
$\bullet$ Update the generator by descending its stochastic gradient: $$\nabla_{\theta_{g}}\frac{1}{m}\sum_{i=1}^{m}\log\left(1-D\left(G\left(\bm{z}^{(i)}\right)\right)\right).$$
end forThe gradient-based updates can use any standard gradient-based learning rule. We used momentum in our experiments.

Algorithm 1 Minibatch stochastic gradient descent training of generative adversarial nets. The number of steps to apply to the discriminator, $k$, is a hyperparameter. We used $k=1$, the least expensive option, in our experiments.

for 训练迭代次数 do
for $k$ 步 do
$\bullet$ 从噪声先验 $p_{g}(\bm{z})$ 采样大小为 $m$ 的噪声小批量 $\{\bm{z}^{(1)},\dots,\bm{z}^{(m)}\}$。
$\bullet$ 从数据生成分布 $p_{\text{data}}(\bm{x})$ 采样大小为 $m$ 的样本小批量 $\{\bm{x}^{(1)},\dots,\bm{x}^{(m)}\}$。
$\bullet$ 沿随机梯度上升更新判别器： $$\nabla_{\theta_{d}}\frac{1}{m}\sum_{i=1}^{m}\left[\log D\left(\bm{x}^{(i)}\right)+\log\left(1-D\left(G\left(\bm{z}^{(i)}\right)\right)\right)\right].$$
end for
$\bullet$ 从噪声先验 $p_{g}(\bm{z})$ 采样大小为 $m$ 的噪声小批量 $\{\bm{z}^{(1)},\dots,\bm{z}^{(m)}\}$。
$\bullet$ 沿随机梯度下降更新生成器： $$\nabla_{\theta_{g}}\frac{1}{m}\sum_{i=1}^{m}\log\left(1-D\left(G\left(\bm{z}^{(i)}\right)\right)\right).$$
end for基于梯度的更新可采用任意标准梯度学习规则。实验中我们使用了动量。

算法 1 生成对抗网络的小批量随机梯度下降训练。判别器更新步数 $k$ 为超参数。实验中取 $k=1$，这是最省计算的选择。

4 Theoretical Results

4 理论结果

The generator $G$ implicitly defines a probability distribution $p_{g}$ as the distribution of the samples $G(\bm{z})$ obtained when $\bm{z}\sim p_{\bm{z}}$. Therefore, we would like Algorithm 1 to converge to a good estimator of $p_{\text{data}}$, if given enough capacity and training time. The results of this section are done in a non-parametric setting, e.g. we represent a model with infinite capacity by studying convergence in the space of probability density functions.

生成器 $G$ 隐式定义概率分布 $p_{g}$，即当 $\bm{z}\sim p_{\bm{z}}$ 时样本 $G(\bm{z})$ 的分布。因此我们希望在容量与训练时间足够时，算法 1 收敛到 $p_{\text{data}}$ 的良好估计。本节结果在非参数设定下给出，例如通过在概率密度函数空间中研究收敛来表示无限容量模型。

We will show in section 4.1 that this minimax game has a global optimum for $p_{g}=p_{\text{data}}$. We will then show in section 4.2 that Algorithm 1 optimizes Eq 1, thus obtaining the desired result.

我们将在 4.1 节证明该极小极大博弈在 $p_{g}=p_{\text{data}}$ 时有全局最优；再在 4.2 节证明算法 1 优化式 1，从而得到所需结论。

4.1 Global Optimality of $p_{g}=p_{\text{data}}$

4.1 $p_{g}=p_{\text{data}}$ 的全局最优性

We first consider the optimal discriminator $D$ for any given generator $G$.

我们首先考虑任意给定生成器 $G$ 下的最优判别器 $D$。

Proposition 1.

For $G$ fixed, the optimal discriminator $D$ is

命题 1。

固定 $G$ 时，最优判别器 $D$ 为

$$D^{*}_{G}(\bm{x})=\frac{p_{\text{data}}(\bm{x})}{p_{\text{data}}(\bm{x})+p_{g}(\bm{x})} \tag{2}$$

Proof.

The training criterion for the discriminator D, given any generator $G$, is to maximize the quantity $V(G,D)$

证明。

给定任意生成器 $G$，判别器 $D$ 的训练准则是最大化量 $V(G,D)$

$$\displaystyle V(G,D)= \\ \displaystyle\int_{\bm{x}}p_{\text{data}}(\bm{x})\log(D(\bm{x}))dx+\int_{z}p_{\bm{z}}(\bm{z})\log(1-D(g(\bm{z})))dz \\ \displaystyle= \\ \displaystyle\int_{\bm{x}}p_{\text{data}}(\bm{x})\log(D(\bm{x}))+p_{g}(\bm{x})\log(1-D(\bm{x}))dx \tag{3}$$

For any $(a,b)\in\mathbb{R}^{2}\setminus\{0,0\}$, the function $y\rightarrow a\log(y)+b\log(1-y)$ achieves its maximum in $[0,1]$ at $\frac{a}{a+b}$. The discriminator does not need to be defined outside of $Supp(p_{\text{data}})\cup Supp(p_{g})$, concluding the proof. ∎

对任意 $(a,b)\in\mathbb{R}^{2}\setminus\{0,0\}$，函数 $y\rightarrow a\log(y)+b\log(1-y)$ 在 $[0,1]$ 上于 $\frac{a}{a+b}$ 取最大。判别器无需在 $Supp(p_{\text{data}})\cup Supp(p_{g})$ 之外定义，证毕。∎

Note that the training objective for $D$ can be interpreted as maximizing the log-likelihood for estimating the conditional probability $P(Y=y|\bm{x})$, where $Y$ indicates whether $\bm{x}$ comes from $p_{\text{data}}$ (with $y=1$) or from $p_{g}$ (with $y=0$). The minimax game in Eq. 1 can now be reformulated as:

注意 $D$ 的训练目标可解释为最大化条件概率 $P(Y=y|\bm{x})$ 的对数似然，其中 $Y$ 指示 $\bm{x}$ 来自 $p_{\text{data}}$（$y=1$）还是 $p_{g}$（$y=0$）。式 1 的极小极大博弈现可改写为：

$$\displaystyle C(G)= \\ \displaystyle\max_{D}V(G,D) \\ \displaystyle= \\ \displaystyle\mathbb{E}_{\bm{x}\sim p_{\text{data}}}[\log D^{*}_{G}(\bm{x})]+\mathbb{E}_{\bm{z}\sim p_{\bm{z}}}[\log(1-D^{*}_{G}(G(\bm{z})))] \\ \displaystyle= \\ \displaystyle\mathbb{E}_{\bm{x}\sim p_{\text{data}}}[\log D^{*}_{G}(\bm{x})]+\mathbb{E}_{\bm{x}\sim p_{g}}[\log(1-D^{*}_{G}(\bm{x}))] \\ \displaystyle= \\ \displaystyle\mathbb{E}_{\bm{x}\sim p_{\text{data}}}\left[\log\frac{p_{\text{data}}(\bm{x})}{P_{\text{data}}(\bm{x})+p_{g}(\bm{x})}\right]+\mathbb{E}_{\bm{x}\sim p_{g}}\left[\log\frac{p_{g}(\bm{x})}{p_{\text{data}}(\bm{x})+p_{g}(\bm{x})}\right] \tag{4}$$

Theorem 1.

The global minimum of the virtual training criterion $C(G)$ is achieved if and only if $p_{g}=p_{\text{data}}$. At that point, $C(G)$ achieves the value $-\log 4$.

定理 1。

*虚拟训练准则 $C(G)$ 的全局最小值当且仅当 $p_{g}=p_{\text{data}}$ 时达到。此时 $C(G)$ 取值为 $-\log 4$。

Proof.

For $p_{g}=p_{\text{data}}$, $D^{*}_{G}(\bm{x})=\frac{1}{2}$, (consider Eq. 2). Hence, by inspecting Eq. 4.1 at $D^{*}_{G}(\bm{x})=\frac{1}{2}$, we find $C(G)=\log\frac{1}{2}+\log\frac{1}{2}=-\log 4$. To see that this is the best possible value of $C(G)$, reached only for $p_{g}=p_{\text{data}}$, observe that

证明。

当 $p_{g}=p_{\text{data}}$ 时，$D^{*}_{G}(\bm{x})=\frac{1}{2}$（见式 2）。故在 $D^{*}_{G}(\bm{x})=\frac{1}{2}$ 处查看式 4.1，得 $C(G)=\log\frac{1}{2}+\log\frac{1}{2}=-\log 4$。为说明这是 $C(G)$ 的最佳可能值且仅在 $p_{g}=p_{\text{data}}$ 达到，注意

$$\mathbb{E}_{\bm{x}\sim p_{\text{data}}}\left[-\log 2\right]+\mathbb{E}_{\bm{x}\sim p_{g}}\left[-\log 2\right]=-\log 4$$

and that by subtracting this expression from $C(G)=V(D_{G}^{*},G)$, we obtain:

再从 $C(G)=V(D_{G}^{*},G)$ 减去该式，得到：

$$C(G)=-\log(4)+KL\left(p_{\text{data}}\left\|\frac{p_{\text{data}}+p_{g}}{2}\right.\right)+KL\left(p_{g}\left\|\frac{p_{\text{data}}+p_{g}}{2}\right.\right) \tag{5}$$

where KL is the Kullback–Leibler divergence. We recognize in the previous expression the Jensen–Shannon divergence between the model’s distribution and the data generating process:

其中 KL 为 Kullback–Leibler 散度。上式中可识别出模型分布与数据生成过程之间的 Jensen–Shannon 散度：

$$C(G)=-\log(4)+2\cdot JSD\left(p_{\text{data}}\left\|p_{g}\right.\right) \tag{6}$$

Since the Jensen–Shannon divergence between two distributions is always non-negative and zero only when they are equal, we have shown that $C^{*}=-\log(4)$ is the global minimum of $C(G)$ and that the only solution is $p_{g}=p_{\text{data}}$, i.e., the generative model perfectly replicating the data generating process. ∎

由于两分布间的 Jensen–Shannon 散度恒非负且仅当二者相等时为零，我们已证明 $C^{*}=-\log(4)$ 是 $C(G)$ 的全局最小，且唯一解为 $p_{g}=p_{\text{data}}$，即生成模型完美复制数据生成过程。∎

4.2 Convergence of Algorithm 1

4.2 算法 1 的收敛性

Proposition 2.

If $G$ and $D$ have enough capacity, and at each step of Algorithm 1, the discriminator is allowed to reach its optimum given $G$, and $p_{g}$ is updated so as to improve the criterion

命题 2。

若 $G$ 与 $D$ 容量足够，且算法 1 每一步都允许判别器在给定 $G$ 下达到最优，并且 $p_{g}$ 的更新使准则

$$\mathbb{E}_{\bm{x}\sim p_{\text{data}}}[\log D^{*}_{G}(\bm{x})]+\mathbb{E}_{\bm{x}\sim p_{g}}[\log(1-D^{*}_{G}(\bm{x}))]$$

then $p_{g}$ converges to $p_{\text{data}}$

得到改进，则 $p_{g}$ 收敛到 $p_{\text{data}}$

Proof.

Consider $V(G,D)=U(p_{g},D)$ as a function of $p_{g}$ as done in the above criterion. Note that $U(p_{g},D)$ is convex in $p_{g}$. The subderivatives of a supremum of convex functions include the derivative of the function at the point where the maximum is attained. In other words, if $f(x)=\sup_{\alpha\in\cal{A}}f_{\alpha}(x)$ and $f_{\alpha}(x)$ is convex in $x$ for every $\alpha$, then $\partial f_{\beta}(x)\in\partial f$ if $\beta=\arg\sup_{\alpha\in\cal{A}}f_{\alpha}(x)$. This is equivalent to computing a gradient descent update for $p_{g}$ at the optimal $D$ given the corresponding $G$. $\sup_{D}U(p_{g},D)$ is convex in $p_{g}$ with a unique global optima as proven in Thm 1, therefore with sufficiently small updates of $p_{g}$, $p_{g}$ converges to $p_{x}$, concluding the proof. ∎

证明。

把 $V(G,D)=U(p_{g},D)$ 视为上述准则中 $p_{g}$ 的函数。注意 $U(p_{g},D)$ 对 $p_{g}$ 凸。凸函数上确界的次导数包含在达到最大值处函数的导数。换言之，若 $f(x)=\sup_{\alpha\in\cal{A}}f_{\alpha}(x)$ 且每个 $\alpha$ 下 $f_{\alpha}(x)$ 对 $x$ 凸，则当 $\beta=\arg\sup_{\alpha\in\cal{A}}f_{\alpha}(x)$ 时 $\partial f_{\beta}(x)\in\partial f$。这等价于在对应 $G$ 的最优 $D$ 处对 $p_{g}$ 做梯度下降更新。$\sup_{D}U(p_{g},D)$ 对 $p_{g}$ 凸且如定理 1 所证有唯一全局最优，因此在 $p_{g}$ 更新足够小时，$p_{g}$ 收敛到 $p_{x}$，证毕。∎

In practice, adversarial nets represent a limited family of $p_{g}$ distributions via the function $G(\bm{z};\theta_{g})$, and we optimize $\theta_{g}$ rather than $p_{g}$ itself. Using a multilayer perceptron to define $G$ introduces multiple critical points in parameter space. However, the excellent performance of multilayer perceptrons in practice suggests that they are a reasonable model to use despite their lack of theoretical guarantees.

实践中，对抗网络通过函数 $G(\bm{z};\theta_{g})$ 只表示有限族 $p_{g}$，我们优化的是 $\theta_{g}$ 而非 $p_{g}$ 本身。用多层感知机定义 $G$ 会在参数空间引入多个临界点。不过多层感知机在实践中的出色表现表明，尽管缺乏理论保证，它们仍是合理的模型选择。

5 Experiments

5 实验

We trained adversarial nets an a range of datasets including MNIST[23], the Toronto Face Database (TFD) [28], and CIFAR-10 [21]. The generator nets used a mixture of rectifier linear activations [19, 9] and sigmoid activations, while the discriminator net used maxout [10] activations. Dropout [17] was applied in training the discriminator net. While our theoretical framework permits the use of dropout and other noise at intermediate layers of the generator, we used noise as the input to only the bottommost layer of the generator network.

我们在包括 MNIST[23]、Toronto Face Database（TFD） [28] 与 CIFAR-10 [21] 在内的一系列数据集上训练对抗网络。生成网络混合使用整流线性激活 [19, 9] 与 sigmoid 激活，判别网络使用 maxout [10] 激活。训练判别网络时施加 dropout [17]。虽理论框架允许在生成器中间层使用 dropout 及其他噪声，我们仅在生成网络最底层把噪声作为输入。

We estimate probability of the test set data under $p_{g}$ by fitting a Gaussian Parzen window to the samples generated with $G$ and reporting the log-likelihood under this distribution. The $\sigma$ parameter of the Gaussians was obtained by cross validation on the validation set. This procedure was introduced in Breuleux et al. 2011 and used for various generative models for which the exact likelihood is not tractable [25, 3, 5]. Results are reported in Table 1. This method of estimating the likelihood has somewhat high variance and does not perform well in high dimensional spaces but it is the best method available to our knowledge. Advances in generative models that can sample but not estimate likelihood directly motivate further research into how to evaluate such models.

我们用 $G$ 生成的样本拟合高斯 Parzen 窗，并报告该分布下的对数似然，以此估计测试集数据在 $p_{g}$ 下的概率。高斯的 $\sigma$ 参数由验证集交叉验证得到。该流程由 Breuleux 等人 2011 年引入，并用于多种精确似然不可求的生成模型 [25, 3, 5]。结果见表 1。这种似然估计方差偏高，在高维空间表现不佳，但据我们所知仍是现有最佳方法。能够采样却不能直接估计似然的生成模型进展，促使人们进一步研究如何评估此类模型。

Model	MNIST	TFD
DBN [3]	$138\pm 2$	$1909\pm 66$
Stacked CAE [3]	$121\pm 1.6$	$\mathbf{2110\pm 50}$
Deep GSN [6]	$214\pm 1.1$	$1890\pm 29$
Adversarial nets	$\mathbf{225\pm 2}$	$\mathbf{2057\pm 26}$

Table 1: Parzen window-based log-likelihood estimates. The reported numbers on MNIST are the mean log-likelihood of samples on test set, with the standard error of the mean computed across examples. On TFD, we computed the standard error across folds of the dataset, with a different $\sigma$ chosen using the validation set of each fold. On TFD, $\sigma$ was cross validated on each fold and mean log-likelihood on each fold were computed. For MNIST we compare against other models of the real-valued (rather than binary) version of dataset.

模型	MNIST	TFD
DBN [3]	$138\pm 2$	$1909\pm 66$
Stacked CAE [3]	$121\pm 1.6$	$\mathbf{2110\pm 50}$
Deep GSN [6]	$214\pm 1.1$	$1890\pm 29$
Adversarial nets	$\mathbf{225\pm 2}$	$\mathbf{2057\pm 26}$

表 1：基于 Parzen 窗的对数似然估计。MNIST 上报告的是测试集样本的平均对数似然，标准误差按样本计算。TFD 上标准误差按数据集折次计算，每折用该折验证集另选 $\sigma$。TFD 上对每折交叉验证 $\sigma$ 并计算每折平均对数似然。MNIST 上我们与实值（而非二值）版本数据集上的其他模型比较。

In Figures 2 and 3 we show samples drawn from the generator net after training. While we make no claim that these samples are better than samples generated by existing methods, we believe that these samples are at least competitive with the better generative models in the literature and highlight the potential of the adversarial framework.

图 2 与图 3 展示训练后生成网络抽出的样本。我们并不声称这些样本优于已有方法，但认为至少可与文献中较好的生成模型竞争，并凸显对抗框架的潜力。

a)	b)
c)	d)

Figure 2: Visualization of samples from the model. Rightmost column shows the nearest training example of the neighboring sample, in order to demonstrate that the model has not memorized the training set. Samples are fair random draws, not cherry-picked. Unlike most other visualizations of deep generative models, these images show actual samples from the model distributions, not conditional means given samples of hidden units. Moreover, these samples are uncorrelated because the sampling process does not depend on Markov chain mixing. a) MNIST b) TFD c) CIFAR-10 (fully connected model) d) CIFAR-10 (convolutional discriminator and “deconvolutional” generator)

a)	b)
c)	d)

图 2：模型样本可视化。最右列给出相邻样本在训练集中的最近邻，以表明模型并未记忆训练集。样本为公正随机抽取，未经挑选。与多数深度生成模型可视化不同，这些图像是模型分布的真实样本，而非给定隐单元样本的条件均值。此外，因采样过程不依赖马尔可夫链混合，这些样本互不相关。a) MNIST b) TFD c) CIFAR-10（全连接模型）d) CIFAR-10（卷积判别器与「反卷积」生成器）

Figure 3: Digits obtained by linearly interpolating between coordinates in $\bm{z}$ space of the full model.

图 3：在完整模型的 $\bm{z}$ 空间坐标之间线性插值得到的数字。

	Deep directed graphical models	Deep undirected graphical models	Generative autoencoders	Adversarial models
Training	Inference needed during training.	Inference needed during training. MCMC needed to approximate partition function gradient.	Enforced tradeoff between mixing and power of reconstruction generation	Synchronizing the discriminator with the generator. Helvetica.
Inference	Learned approximate inference	Variational inference	MCMC-based inference	Learned approximate inference
Sampling	No difficulties	Requires Markov chain	Requires Markov chain	No difficulties
Evaluating $p(x)$	Intractable, may be approximated with AIS	Intractable, may be approximated with AIS	Not explicitly represented, may be approximated with Parzen density estimation	Not explicitly represented, may be approximated with Parzen density estimation
Model design	Nearly all models incur extreme difficulty	Careful design needed to ensure multiple properties	Any differentiable function is theoretically permitted	Any differentiable function is theoretically permitted

Table 2: Challenges in generative modeling: a summary of the difficulties encountered by different approaches to deep generative modeling for each of the major operations involving a model.

	深度有向图模型	深度无向图模型	生成自编码器	对抗模型
训练	训练时需要推断。	训练时需要推断。需 MCMC 近似配分函数梯度。	在混合与重建/生成能力之间强制权衡	使判别器与生成器同步。Helvetica。
推断	学得的近似推断	变分推断	基于 MCMC 的推断	学得的近似推断
采样	无困难	需要马尔可夫链	需要马尔可夫链	无困难
评估 $p(x)$	难解，可用 AIS 近似	难解，可用 AIS 近似	不显式表示，可用 Parzen 密度估计近似	不显式表示，可用 Parzen 密度估计近似
模型设计	几乎所有模型都极困难	需仔细设计以保证多种性质	理论上允许任意可微函数	理论上允许任意可微函数

表 2：生成建模中的挑战：按涉及模型的主要操作，汇总不同深度生成建模路径遇到的困难。

6 Advantages and disadvantages

6 优点与缺点

This new framework comes with advantages and disadvantages relative to previous modeling frameworks. The disadvantages are primarily that there is no explicit representation of $p_{g}(\bm{x})$, and that $D$ must be synchronized well with $G$ during training (in particular, $G$ must not be trained too much without updating $D$, in order to avoid “the Helvetica scenario” in which $G$ collapses too many values of $\mathbf{z}$ to the same value of $\mathbf{x}$ to have enough diversity to model $p_{\text{data}}$), much as the negative chains of a Boltzmann machine must be kept up to date between learning steps. The advantages are that Markov chains are never needed, only backprop is used to obtain gradients, no inference is needed during learning, and a wide variety of functions can be incorporated into the model. Table 2 summarizes the comparison of generative adversarial nets with other generative modeling approaches.

相对先前建模框架，这一新框架有利有弊。主要缺点是没有 $p_{g}(\bm{x})$ 的显式表示，且训练中 $D$ 必须与 $G$ 良好同步（尤其不能在不更新 $D$ 的情况下把 $G$ 训得过多，以免落入「Helvetica 情景」——$G$ 把过多 $\mathbf{z}$ 塌缩到同一 $\mathbf{x}$，多样性不足以建模 $p_{\text{data}}$），这类似于玻尔兹曼机的负链须在学习步之间保持更新。优点是永不需要马尔可夫链，梯度只需反向传播，学习时无需推断，且模型可纳入多种函数。表 2 汇总了生成对抗网络与其他生成建模方法的比较。

The aforementioned advantages are primarily computational. Adversarial models may also gain some statistical advantage from the generator network not being updated directly with data examples, but only with gradients flowing through the discriminator. This means that components of the input are not copied directly into the generator’s parameters. Another advantage of adversarial networks is that they can represent very sharp, even degenerate distributions, while methods based on Markov chains require that the distribution be somewhat blurry in order for the chains to be able to mix between modes.

上述优点主要是计算上的。对抗模型还可能获得统计上的好处：生成网络不直接用数据样本更新，而只接收经判别器回传的梯度，因而输入分量不会被直接拷进生成器参数。另一优点是对抗网络可表示非常尖锐甚至退化的分布，而基于马尔可夫链的方法为使链能在模态间混合，往往要求分布带一定模糊。

7 Conclusions and future work

7 结论与未来工作

This framework admits many straightforward extensions:

该框架容许多种直接扩展：

1. A conditional generative model $p(\bm{x}\mid\bm{c})$ can be obtained by adding $\bm{c}$ as input to both $G$ and $D$.

1. 把 $\bm{c}$ 同时作为 $G$ 与 $D$ 的输入，即可得到条件生成模型 $p(\bm{x}\mid\bm{c})$。

2. Learned approximate inference can be performed by training an auxiliary network to predict $\bm{z}$ given $\bm{x}$. This is similar to the inference net trained by the wake-sleep algorithm  [15] but with the advantage that the inference net may be trained for a fixed generator net after the generator net has finished training.

2. 训练辅助网络由 $\bm{x}$ 预测 $\bm{z}$，可做学得的近似推断。这与 wake-sleep 算法 [15] 训练的推断网相似，但优点是可在生成网训练完成后再为固定生成网训练推断网。

3. One can approximately model all conditionals $p(\bm{x}_{S}\mid\bm{x}_{\not S})$ where $S$ is a subset of the indices of $\bm{x}$ by training a family of conditional models that share parameters. Essentially, one can use adversarial nets to implement a stochastic extension of the deterministic MP-DBM [11].

3. 通过训练共享参数的一族条件模型，可近似建模所有条件 $p(\bm{x}_{S}\mid\bm{x}_{\not S})$，其中 $S$ 是 $\bm{x}$ 下标的子集。本质上，可用对抗网络实现确定性 MP-DBM [11] 的随机扩展。

4. Semi-supervised learning: features from the discriminator or inference net could improve performance of classifiers when limited labeled data is available.

4. 半监督学习：标签有限时，判别器或推断网的特征可提升分类器表现。

5. Efficiency improvements: training could be accelerated greatly by divising better methods for coordinating $G$ and $D$ or determining better distributions to sample $\mathbf{z}$ from during training.

5. 效率改进：设计更好的 $G$ 与 $D$ 协调方式，或更好的训练期 $\mathbf{z}$ 采样分布，可大幅加速训练。

This paper has demonstrated the viability of the adversarial modeling framework, suggesting that these research directions could prove useful.

本文展示了对抗建模框架的可行性，表明这些研究方向可能有用。

Acknowledgments

We would like to acknowledge Patrice Marcotte, Olivier Delalleau, Kyunghyun Cho, Guillaume Alain and Jason Yosinski for helpful discussions. Yann Dauphin shared his Parzen window evaluation code with us. We would like to thank the developers of Pylearn2 [12] and Theano [7, 1], particularly Frédéric Bastien who rushed a Theano feature specifically to benefit this project. Arnaud Bergeron provided much-needed support with LaTeX typesetting. We would also like to thank CIFAR, and Canada Research Chairs for funding, and Compute Canada, and Calcul Québec for providing computational resources. Ian Goodfellow is supported by the 2013 Google Fellowship in Deep Learning. Finally, we would like to thank Les Trois Brasseurs for stimulating our creativity.

致谢

感谢 Patrice Marcotte、Olivier Delalleau、Kyunghyun Cho、Guillaume Alain 与 Jason Yosinski 的有益讨论。Yann Dauphin 与我们分享了 Parzen 窗评估代码。感谢 Pylearn2 [12] 与 Theano [7, 1] 的开发者，尤其是 Frédéric Bastien，他专门为本项目赶制了 Theano 功能。Arnaud Bergeron 在 LaTeX 排版上给予急需支持。亦感谢 CIFAR 与 Canada Research Chairs 的资助，以及 Compute Canada 与 Calcul Québec 提供的计算资源。Ian Goodfellow 获 2013 Google Fellowship in Deep Learning 支持。最后感谢 Les Trois Brasseurs 激发我们的创造力。

References

参考文献

Bastien et al. [2012] Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Bergeron, A., Bouchard, N., and Bengio, Y. (2012). Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.

Bengio [2009] Bengio, Y. (2009). Learning deep architectures for AI . Now Publishers.

Bengio et al. [2013a] Bengio, Y., Mesnil, G., Dauphin, Y., and Rifai, S. (2013a). Better mixing via deep representations. In ICML’13 .

Bengio et al. [2013b] Bengio, Y., Yao, L., Alain, G., and Vincent, P. (2013b). Generalized denoising auto-encoders as generative models. In NIPS26 . Nips Foundation.

Bengio et al. [2014a] Bengio, Y., Thibodeau-Laufer, E., and Yosinski, J. (2014a). Deep generative stochastic networks trainable by backprop. In ICML’14 .

Bengio et al. [2014b] Bengio, Y., Thibodeau-Laufer, E., Alain, G., and Yosinski, J. (2014b). Deep generative stochastic networks trainable by backprop. In Proceedings of the 30th International Conference on Machine Learning (ICML’14) .

Bergstra et al. [2010] Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy) . Oral Presentation.

Breuleux et al. [2011] Breuleux, O., Bengio, Y., and Vincent, P. (2011). Quickly generating representative samples from an RBM-derived process. Neural Computation , 23 (8), 2053–2073.

Glorot et al. [2011] Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep sparse rectifier neural networks. In AISTATS’2011 .

Goodfellow et al. [2013a] Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013a). Maxout networks. In ICML’2013 .

Goodfellow et al. [2013b] Goodfellow, I. J., Mirza, M., Courville, A., and Bengio, Y. (2013b). Multi-prediction deep Boltzmann machines. In NIPS’2013 .

Goodfellow et al. [2013c] Goodfellow, I. J., Warde-Farley, D., Lamblin, P., Dumoulin, V., Mirza, M., Pascanu, R., Bergstra, J., Bastien, F., and Bengio, Y. (2013c). Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214 .

Gutmann and Hyvarinen [2010] Gutmann, M. and Hyvarinen, A. (2010). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS’2010 .

Hinton et al. [2012a] Hinton, G., Deng, L., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B. (2012a). Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine , 29 (6), 82–97.

Hinton et al. [1995] Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995). The wake-sleep algorithm for unsupervised neural networks. Science , 268 , 1558–1161.

Hinton et al. [2006] Hinton, G. E., Osindero, S., and Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation , 18 , 1527–1554.

Hinton et al. [2012b] Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2012b). Improving neural networks by preventing co-adaptation of feature detectors. Technical report, arXiv:1207.0580.

Hyvärinen [2005] Hyvärinen, A. (2005). Estimation of non-normalized statistical models using score matching. J. Machine Learning Res. , 6 .

Jarrett et al. [2009] Jarrett, K., Kavukcuoglu, K., Ranzato, M., and LeCun, Y. (2009). What is the best multi-stage architecture for object recognition? In Proc. International Conference on Computer Vision (ICCV’09) , pages 2146–2153. IEEE.

Kingma and Welling [2014] Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR) .

Krizhevsky and Hinton [2009] Krizhevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images. Technical report, University of Toronto.

Krizhevsky et al. [2012] Krizhevsky, A., Sutskever, I., and Hinton, G. (2012). ImageNet classification with deep convolutional neural networks. In NIPS’2012 .

LeCun et al. [1998] LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86 (11), 2278–2324.

Rezende et al. [2014] Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. Technical report, arXiv:1401.4082.

Rifai et al. [2012] Rifai, S., Bengio, Y., Dauphin, Y., and Vincent, P. (2012). A generative process for sampling contractive auto-encoders. In ICML’12 .

Salakhutdinov and Hinton [2009] Salakhutdinov, R. and Hinton, G. E. (2009). Deep Boltzmann machines. In AISTATS’2009 , pages 448–455.

Smolensky [1986] Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory. In D. E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing , volume 1, chapter 6, pages 194–281. MIT Press, Cambridge.

Susskind et al. [2010] Susskind, J., Anderson, A., and Hinton, G. E. (2010). The Toronto face dataset. Technical Report UTML TR 2010-001, U. Toronto.

Tieleman [2008] Tieleman, T. (2008). Training restricted Boltzmann machines using approximations to the likelihood gradient. In W. W. Cohen, A. McCallum, and S. T. Roweis, editors, ICML 2008 , pages 1064–1071. ACM.

Vincent et al. [2008] Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. In ICML 2008 .

Younes [1999] Younes, L. (1999). On the convergence of Markovian stochastic algorithms with rapidly decreasing ergodicity rates. Stochastics and Stochastic Reports , 65 (3), 177–228.