Spectral Normalization for Generative Adversarial Networks

生成对抗网络的谱归一化

Takeru Miyato    Toshiki Kataoka    Masanori Koyama    Yuichi Yoshida {miyato, kataoka}@preferred.jpkoyama.masanori@gmail.comyyoshida@nii.ac.jp Affiliation:  Preferred Networks, Inc. Ritsumeikan University National Institute of Informatics

Abstract

摘要

One of the challenges in the study of generative adversarial networks is the instability of its training. In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator. Our new normalization technique is computationally light and easy to incorporate into existing implementations. We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques. The code with Chainer Tokui et al. 2015, generated images and pretrained models are available at https://github.com/pfnet-research/sngan_projection.

生成对抗网络研究中的一大挑战是训练不稳定。本文提出一种称为谱归一化的新型权重归一化技术，用以稳定判别器的训练。该归一化计算轻量，易于并入已有实现。我们在 CIFAR10、STL-10 与 ILSVRC2012 上检验谱归一化的效力，实验确认谱归一化 GAN（SN-GANs）相对先前训练稳定技术，能够生成质量更好或相当的图像。基于 Chainer（Tokui et al. 2015）的代码、生成图像与预训练模型见 https://github.com/pfnet-research/sngan_projection。

1 Introduction

1 引言

Generative adversarial networks (GANs) Goodfellow et al. 2014 have been enjoying considerable success as a framework of generative models in recent years, and it has been applied to numerous types of tasks and datasets Radford et al. 2016; Salimans et al. 2016; Ho & Ermon 2016; Li et al. 2017. In a nutshell, GANs are a framework to produce a model distribution that mimics a given target distribution, and it consists of a generator that produces the model distribution and a discriminator that distinguishes the model distribution from the target. The concept is to consecutively train the model distribution and the discriminator in turn, with the goal of reducing the difference between the model distribution and the target distribution measured by the best discriminator possible at each step of the training. GANs have been drawing attention in the machine learning community not only for its ability to learn highly structured probability distribution but also for its theoretically interesting aspects. For example, Nowozin et al. 2016; Uehara et al. 2016; Mohamed & Lakshminarayanan 2017 revealed that the training of the discriminator amounts to the training of a good estimator for the density ratio between the model distribution and the target. This is a perspective that opens the door to the methods of implicit models Mohamed & Lakshminarayanan 2017; Tran et al. 2017 that can be used to carry out variational optimization without the direct knowledge of the density function.

生成对抗网络（GANs，Goodfellow et al. 2014）近年来作为生成模型框架取得可观成功，并被应用于众多任务与数据集（Radford et al. 2016；Salimans et al. 2016；Ho & Ermon 2016；Li et al. 2017）。简言之，GAN 旨在产生模仿给定目标分布的模型分布：生成器产生模型分布，判别器区分模型分布与目标。概念是轮流连续训练模型分布与判别器，目标是在训练每一步用当时尽可能好的判别器，缩小模型分布与目标分布之间由其度量的差异。GAN 引起机器学习社区关注，不仅因其能学习高度结构化的概率分布，也因其 在理论上也引人关注的方面。例如 Nowozin et al. 2016；Uehara et al. 2016；Mohamed & Lakshminarayanan 2017 指出，训练判别器等价于训练模型分布与目标之间密度比的良好估计器。这一视角通向 隐式模型 方法（Mohamed & Lakshminarayanan 2017；Tran et al. 2017），可在不直接知道密度函数的情况下做变分优化。

A persisting challenge in the training of GANs is the performance control of the discriminator. In high dimensional spaces, the density ratio estimation by the discriminator is often inaccurate and unstable during the training, and generator networks fail to learn the multimodal structure of the target distribution. Even worse, when the support of the model distribution and the support of the target distribution are disjoint, there exists a discriminator that can perfectly distinguish the model distribution from the target Arjovsky & Bottou 2017. Once such discriminator is produced in this situation, the training of the generator comes to complete stop, because the derivative of the so-produced discriminator with respect to the input turns out to be 0. This motivates us to introduce some form of restriction to the choice of the discriminator.

GAN 训练中持续存在的挑战是对判别器表现的控制。在高维空间中，判别器对密度比的估计在训练过程中往往不准确且不稳定，生成器网络难以学到目标分布的多峰结构。更糟的是，当模型分布的支撑与目标分布的支撑不相交时，存在能完美区分二者的判别器（Arjovsky & Bottou 2017）。一旦在此情形下得到这样的判别器，生成器训练就会完全停滞，因为该判别器对输入的导数为 0。这促使我们在判别器的选择上引入某种限制。

In this paper, we propose a novel weight normalization method called spectral normalization that can stabilize the training of discriminator networks. Our normalization enjoys following favorable properties.

本文提出一种称为 谱归一化 的新型权重归一化方法，可稳定判别器网络的训练。该归一化具有以下有利性质。

• • Lipschitz constant is the only hyper-parameter to be tuned, and the algorithm does not require intensive tuning of the only hyper-parameter for satisfactory performance.

• • Lipschitz 常数是唯一需调的超参数，算法不必对这唯一超参数做密集调优也能取得满意表现。

• • Implementation is simple and the additional computational cost is small.

• • 实现简单，额外计算开销小。

In fact, our normalization method also functioned well even without tuning Lipschitz constant, which is the only hyper parameter. In this study, we provide explanations of the effectiveness of spectral normalization for GANs against other regularization techniques, such as weight normalization Salimans & Kingma 2016, weight clipping Arjovsky et al. 2017, and gradient penalty Gulrajani et al. 2017. We also show that, in the absence of complimentary regularization techniques (e.g., batch normalization, weight decay and feature matching on the discriminator), spectral normalization can improve the sheer quality of the generated images better than weight normalization and gradient penalty.

事实上，即便不调 Lipschitz 常数（唯一超参数），我们的归一化方法也能良好工作。本研究解释谱归一化相对权重归一化（Salimans & Kingma 2016）、权重裁剪（Arjovsky et al. 2017）与梯度惩罚（Gulrajani et al. 2017）等其他正则技术对 GAN 的有效性。我们还表明：在没有配套正则（如判别器上的批归一化、权重衰减与特征匹配）时，谱归一化相对权重归一化与梯度惩罚，更能直接提升生成图像的质量。

2 Method

2 方法

In this section, we will lay the theoretical groundwork for our proposed method. Let us consider a simple discriminator made of a neural network of the following form, with the input ${\bm{x}}$:

本节为所提方法奠定理论框架。考虑如下形式的简单神经网络判别器，输入为 ${\bm{x}}$：

$$\displaystyle f({\bm{x}},\theta)=W^{L+1}a_{L}(W^{L}(a_{L-1}(W^{L-1}(\dots a_{1}(W^{1}{\bm{x}})\dots)))), \tag{1}$$

where $\theta:=\{W^{1},\dots,W^{L},W^{L+1}\}$ is the learning parameters set, $W^{l}\in\mathbb{R}^{d_{l}\times d_{l-1}}$, $W^{L+1}\in\mathbb{R}^{1\times{d_{L}}}$, and $a_{l}$ is an element-wise non-linear activation function. We omit the bias term of each layer for simplicity. The final output of the discriminator is given by

其中 $\theta:=\{W^{1},\dots,W^{L},W^{L+1}\}$ 为学习参数集，$W^{l}\in\mathbb{R}^{d_{l}\times d_{l-1}}$，$W^{L+1}\in\mathbb{R}^{1\times{d_{L}}}$，$a_{l}$ 为逐元素非线性激活。为简洁省略各层偏置项。判别器最终输出为

$$\displaystyle D({\bm{x}},\theta)={\rm\mathcal{A}}(f({\bm{x}},\theta)), \tag{2}$$

where $\mathcal{A}$ is an activation function corresponding to the divergence of distance measure of the user’s choice. The standard formulation of GANs is given by

其中 $\mathcal{A}$ 为对应用户所选散度或距离度量的激活函数。GAN 的标准表述为

$$\min_{G}\max_{D}V(G,D)$$

where min and max of $G$ and $D$ are taken over the set of generator and discriminator functions, respectively. The conventional form of $V(G,D)$ Goodfellow et al. 2014 is given by $\mathrm{E}_{{\bm{x}}\sim q_{\rm data}}[\log D({\bm{x}})]+\mathrm{E}_{{\bm{x}}^{\prime}\sim p_{G}}[\log(1-D({\bm{x}}^{\prime}))],$ where $q_{\rm data}$ is the data distribution and $p_{G}$ is the (model) generator distribution to be learned through the adversarial min-max optimization. The activation function ${\rm\mathcal{A}}$ that is used in the $D$ of this expression is some continuous function with range $[0,1]$ (e.g, sigmoid function). It is known that, for a fixed generator $G$, the optimal discriminator for this form of $V(G,D)$ is given by $D^{*}_{G}({\bm{x}}):=q_{\rm data}({\bm{x}})/(q_{\rm data}({\bm{x}})+p_{G}({\bm{x}}))$.

其中对 $G$ 与 $D$ 的 min 与 max 分别在生成器与判别器函数集合上取。$V(G,D)$ 的常规形式（Goodfellow et al. 2014）为 $\mathrm{E}_{{\bm{x}}\sim q_{\rm data}}[\log D({\bm{x}})]+\mathrm{E}_{{\bm{x}}^{\prime}\sim p_{G}}[\log(1-D({\bm{x}}^{\prime}))]$，其中 $q_{\rm data}$ 为数据分布，$p_{G}$ 为通过对抗 min-max 优化学习的（模型）生成器分布。该表达式中 $D$ 所用的激活 ${\rm\mathcal{A}}$ 是值域为 $[0,1]$ 的某连续函数（例如 sigmoid）。已知对固定生成器 $G$，该形式 $V(G,D)$ 的最优判别器为 $D^{*}_{G}({\bm{x}}):=q_{\rm data}({\bm{x}})/(q_{\rm data}({\bm{x}})+p_{G}({\bm{x}}))$。

The machine learning community has been pointing out recently that the function space from which the discriminators are selected crucially affects the performance of GANs. A number of works Uehara et al. 2016; Qi 2017; Gulrajani et al. 2017 advocate the importance of Lipschitz continuity in assuring the boundedness of statistics. For example, the optimal discriminator of GANs on the above standard formulation takes the form

机器学习社区近来指出：选择判别器的函数空间对 GAN 表现至关重要。不少工作（Uehara et al. 2016；Qi 2017；Gulrajani et al. 2017）强调 Lipschitz 连续性对保证统计量有界的重要性。例如，上述标准表述下 GAN 的最优判别器取形式

$$\displaystyle D_{G}^{*}({\bm{x}}) \\ \displaystyle=\frac{q_{\rm data}({\bm{x}})}{q_{\rm data}({\bm{x}})+p_{G}({\bm{x}})}={\rm sigmoid}(f^{*}({\bm{x}})),\text{where }f^{*}({\bm{x}})=\log q_{\rm data}({\bm{x}})-\log p_{G}({\bm{x}}), \tag{3}$$

and its derivative

其导数

$$\displaystyle\nabla_{\bm{x}}f^{*}(x)=\frac{1}{q_{\rm data}({\bm{x}})}\nabla_{{\bm{x}}}q_{\rm data}({\bm{x}})-\frac{1}{p_{G}({\bm{x}})}\nabla_{{\bm{x}}}p_{G}({\bm{x}}) \tag{4}$$

can be unbounded or even incomputable. This prompts us to introduce some regularity condition to the derivative of $f({\bm{x}})$.

可能无界甚至不可计算。这促使我们对 $f({\bm{x}})$ 的导数引入某种正则性条件。

A particularly successful works in this array are Qi 2017; Arjovsky et al. 2017; Gulrajani et al. 2017, which proposed methods to control the Lipschitz constant of the discriminator by adding regularization terms defined on input examples ${\bm{x}}$. We would follow their footsteps and search for the discriminator $D$ from the set of $K$-Lipschitz continuous functions, that is,

这一方向上较为成功的工作包括 Qi 2017；Arjovsky et al. 2017；Gulrajani et al. 2017：它们通过对输入样本 ${\bm{x}}$ 定义的正则项来控制判别器的 Lipschitz 常数。我们沿袭其思路，从 $K$-Lipschitz 连续函数集合中搜索判别器 $D$，即

$$\displaystyle\mathop{\rm arg~max}\limits_{\|f\|_{\rm Lip}\leq K}V(G,D), \tag{5}$$

where we mean by $\|f\|_{\rm Lip}$ the smallest value $M$ such that $\|f({\bm{x}})-f({\bm{x}}^{\prime})\|/\|{\bm{x}}-{\bm{x}}^{\prime}\|\leq M$ for any ${\bm{x}},{\bm{x}}^{\prime}$, with the norm being the $\ell_{2}$ norm.

其中 $\|f\|_{\rm Lip}$ 表示使对任意 ${\bm{x}},{\bm{x}}^{\prime}$ 有 $\|f({\bm{x}})-f({\bm{x}}^{\prime})\|/\|{\bm{x}}-{\bm{x}}^{\prime}\|\leq M$ 的最小 $M$，范数为 $\ell_{2}$ 范数。

While input based regularizations allow for relatively easy formulations based on samples, they also suffer from the fact that, they cannot impose regularization on the space outside of the supports of the generator and data distributions without introducing somewhat heuristic means. A method we would introduce in this paper, called spectral normalization, is a method that aims to skirt this issue by normalizing the weight matrices using the technique devised by Yoshida & Miyato 2017.

基于输入的正则相对容易用样本表述，但也有局限：若不引入某种启发式手段，就无法对生成器与数据分布支撑之外的空间施加正则。本文引入的 谱归一化 方法，旨在用 Yoshida & Miyato 2017 提出的技术对权重矩阵做 归一化，从而绕开该问题。

2.1 Spectral Normalization

2.1 谱归一化

Our spectral normalization controls the Lipschitz constant of the discriminator function $f$ by literally constraining the spectral norm of each layer $g:{\bm{h}}_{in}\mapsto{\bm{h}}_{out}$. By definition, Lipschitz norm $\|g\|_{\rm Lip}$ is equal to $\sup_{\bm{h}}\sigma(\nabla g({\bm{h}}))$, where $\sigma(A)$ is the spectral norm of the matrix $A$ ($L_{2}$ matrix norm of $A$)

谱归一化通过约束每一层 $g:{\bm{h}}_{in}\mapsto{\bm{h}}_{out}$ 的谱范数，来控制判别器函数 $f$ 的 Lipschitz 常数。按定义，Lipschitz 范数 $\|g\|_{\rm Lip}$ 等于 $\sup_{\bm{h}}\sigma(\nabla g({\bm{h}}))$，其中 $\sigma(A)$ 为矩阵 $A$ 的谱范数（$A$ 的 $L_{2}$ 矩阵范数）

$$\displaystyle\sigma(A):=\max_{{\bm{h}}:{\bm{h}}\neq{\bm{0}}}\frac{\|A{\bm{h}}\|_{2}}{\|{\bm{h}}\|_{2}}=\max_{\|{\bm{h}}\|_{2}\leq 1}\|A{\bm{h}}\|_{2}, \tag{6}$$

which is equivalent to the largest singular value of $A$. Therefore, for a linear layer $g({\bm{h}})=W{\bm{h}}$, the norm is given by $\|g\|_{\rm Lip}=\sup_{\bm{h}}\sigma(\nabla g({\bm{h}}))=\sup_{\bm{h}}\sigma(W)=\sigma(W)$. If the Lipschitz norm of the activation function $\|a_{l}\|_{\rm Lip}$ is equal to 1 [^1], we can use the inequality $\|g_{1}\circ g_{2}\|_{\rm Lip}\leq\|g_{1}\|_{\rm Lip}\cdot\|g_{2}\|_{\rm Lip}$ to observe the following bound on $\|f\|_{\rm Lip}$:

它等于 $A$ 的最大奇异值。因此对线性层 $g({\bm{h}})=W{\bm{h}}$，范数为 $\|g\|_{\rm Lip}=\sup_{\bm{h}}\sigma(\nabla g({\bm{h}}))=\sup_{\bm{h}}\sigma(W)=\sigma(W)$。若激活函数的 Lipschitz 范数 $\|a_{l}\|_{\rm Lip}$ 等于 1[^1]，则可利用不等式 $\|g_{1}\circ g_{2}\|_{\rm Lip}\leq\|g_{1}\|_{\rm Lip}\cdot\|g_{2}\|_{\rm Lip}$，得到 $\|f\|_{\rm Lip}$ 的如下上界：

$$\displaystyle\|f\|_{\rm Lip}\leq \\ \displaystyle\|({\bm{h}}_{L}\mapsto W^{L+1}{\bm{h}}_{L})\|_{\rm Lip}\cdot\|a_{L}\|_{\rm Lip}\cdot\|({\bm{h}}_{L-1}\mapsto W^{L}{\bm{h}}_{L-1})\|_{\rm Lip} \\ \displaystyle\cdots\|a_{1}\|_{\rm Lip}\cdot\|({\bm{h}}_{0}\mapsto W^{1}{\bm{h}}_{0})\|_{\rm Lip}=\prod_{l=1}^{L+1}\|({\bm{h}}_{l-1}\mapsto W^{l}{\bm{h}}_{l-1})\|_{\rm Lip}=\prod_{l=1}^{L+1}\sigma(W^{l}). \tag{7}$$

Our spectral normalization normalizes the spectral norm of the weight matrix $W$ so that it satisfies the Lipschitz constraint $\sigma(W)=1$:

我们的 谱归一化 对权重矩阵 $W$ 做谱范数归一化，使其满足 Lipschitz 约束 $\sigma(W)=1$：

$$\displaystyle\bar{W}_{\rm SN}(W):=W/\sigma(W). \tag{8}$$

If we normalize each $W^{l}$ using (8), we can appeal to the inequality (7) and the fact that $\sigma\left(\bar{W}_{\rm SN}(W)\right)=1$ to see that $\|f\|_{\rm Lip}$ is bounded from above by $1$.

若用 (8) 归一化每个 $W^{l}$，则可借助不等式 (7) 以及 $\sigma\left(\bar{W}_{\rm SN}(W)\right)=1$，看出 $\|f\|_{\rm Lip}$ 被 1 从上界定。

Here, we would like to emphasize the difference between our spectral normalization and spectral norm ”regularization” introduced by  Yoshida & Miyato 2017. Unlike our method, spectral norm ”regularization” penalizes the spectral norm by adding explicit regularization term to the objective function. Their method is fundamentally different from our method in that they do not make an attempt to ‘set’ the spectral norm to a designated value. Moreover, when we reorganize the derivative of our normalized cost function and rewrite our objective function (12), we see that our method is augmenting the cost function with a sample data dependent regularization function. Spectral norm regularization, on the other hand, imposes sample data independent regularization on the cost function, just like L2 regularization and Lasso.

在此我们想强调本谱归一化与 Yoshida & Miyato 2017 引入的谱范数「正则」之间的差别。与我们的方法不同，谱范数「正则」通过在目标函数中加入显式正则项来惩罚谱范数。二者本质不同：他们并不试图把谱范数「设」为指定值。此外，当我们整理归一化代价函数的导数并改写目标（12）时，可见本方法是在代价函数上附加了依赖样本数据的正则函数；而谱范数正则则像 L2 正则与 Lasso 一样，对代价函数施加与样本无关的正则。

2.2 Fast Approximation of the Spectral Norm $\sigma(W)$

2.2 谱范数 $\sigma(W)$ 的快速近似

As we mentioned above, the spectral norm $\sigma(W)$ that we use to regularize each layer of the discriminator is the largest singular value of $W$. If we naively apply singular value decomposition to compute the $\sigma(W)$ at each round of the algorithm, the algorithm can become computationally heavy. Instead, we can use the power iteration method to estimate $\sigma(W)$ Golub & Van der Vorst 2000; Yoshida & Miyato 2017. With power iteration method, we can estimate the spectral norm with very small additional computational time relative to the full computational cost of the vanilla GANs. Please see Appendix A for the detail method and Algorithm  1 for the summary of the actual spectral normalization algorithm.

如上所述，用来正则判别器各层的谱范数 $\sigma(W)$ 即 $W$ 的最大奇异值。若每轮都朴素做奇异值分解算 $\sigma(W)$，算法会计算沉重。相反，可用幂迭代估计 $\sigma(W)$（Golub & Van der Vorst 2000；Yoshida & Miyato 2017）。相对原版 GAN 的总计算量，幂迭代估计谱范数的额外时间非常小。细节见附录 A，实际谱归一化算法总结见算法 1。

2.3 Gradient analysis of the spectrally normalized weights

2.3 谱归一化权重的梯度分析

The gradient[^2] of $\bar{W}_{\rm SN}(W)$ with respect to $W_{ij}$ is:

$\bar{W}_{\rm SN}(W)$ 对 $W_{ij}$ 的梯度[^2]为：

$$\displaystyle\frac{\partial\bar{W}_{\rm SN}(W)}{\partial W_{ij}}=\frac{1}{\sigma(W)}E_{ij}-\frac{1}{\sigma(W)^{2}}\frac{\partial\sigma(W)}{\partial W_{ij}}W \\ \displaystyle=\frac{1}{\sigma(W)}E_{ij}-\frac{[{\bm{u}}_{1}{\bm{v}}_{1}^{\rm T}]_{ij}}{\sigma(W)^{2}}W \\ \displaystyle=\frac{1}{\sigma(W)}\left(E_{ij}-[{\bm{u}}_{1}{\bm{v}}_{1}^{\rm T}]_{ij}\bar{W}_{\rm SN}\right), \tag{9}$$

where $E_{ij}$ is the matrix whose $(i,j)$-th entry is $1$ and zero everywhere else, and ${\bm{u}}_{1}$ and ${\bm{v}}_{1}$ are respectively the first left and right singular vectors of $W$. If ${\bm{h}}$ is the hidden layer in the network to be transformed by $\bar{W}_{SN}$, the derivative of the $V(G,D)$ calculated over the mini-batch with respect to $W$ of the discriminator $D$ is given by:

其中 $E_{ij}$ 为 $(i,j)$ 元为 1、其余为 0 的矩阵，${\bm{u}}_{1}$ 与 ${\bm{v}}_{1}$ 分别为 $W$ 的第一左、右奇异向量。若 ${\bm{h}}$ 是网络中将被 $\bar{W}_{SN}$ 变换的隐层，则在 mini-batch 上计算的 $V(G,D)$ 对判别器 $D$ 的 $W$ 的导数为：

$$\displaystyle\frac{\partial V(G,D)}{\partial W} \\ \displaystyle=\frac{1}{\sigma(W)}\left(\hat{\mathop{\mathrm{E}}}\left[{\bm{\delta}}{\bm{h}}^{\rm T}\right]-\left(\hat{\mathop{\mathrm{E}}}\left[{\bm{\delta}}^{\rm T}\bar{W}_{\rm SN}{\bm{h}}\right]\right){\bm{u}}_{1}{\bm{v}}_{1}^{\rm T}\right) \\ \displaystyle=\frac{1}{\sigma(W)}\left(\hat{\mathop{\mathrm{E}}}\left[{\bm{\delta}}{\bm{h}}^{\rm T}\right]-\lambda{\bm{u}}_{1}{\bm{v}}_{1}^{\rm T}\right) \tag{11}$$

where ${\bm{\delta}}:=\left(\partial V(G,D)/\partial\left(\bar{W}_{\rm SN}{\bm{h}}\right)\right)^{\rm T}$, $\lambda:=\hat{\mathop{\mathrm{E}}}\left[{\bm{\delta}}^{\rm T}\left(\bar{W}_{\rm SN}{\bm{h}}\right)\right]$, and $\hat{\mathop{\mathrm{E}}}[\cdot]$ represents empirical expectation over the mini-batch. $\frac{\partial V}{\partial W}=0$ when $\hat{\mathop{\mathrm{E}}}[{\bm{\delta}}{\bm{h}}^{\rm T}]=k{\bm{u}}_{1}{\bm{v}}_{1}^{T}$ for some $k\in\mathbb{R}$.

其中 ${\bm{\delta}}:=\left(\partial V(G,D)/\partial\left(\bar{W}_{\rm SN}{\bm{h}}\right)\right)^{\rm T}$，$\lambda:=\hat{\mathop{\mathrm{E}}}\left[{\bm{\delta}}^{\rm T}\left(\bar{W}_{\rm SN}{\bm{h}}\right)\right]$，$\hat{\mathop{\mathrm{E}}}[\cdot]$ 表示 mini-batch 上的经验期望。当 $\hat{\mathop{\mathrm{E}}}[{\bm{\delta}}{\bm{h}}^{\rm T}]=k{\bm{u}}_{1}{\bm{v}}_{1}^{T}$（某 $k\in\mathbb{R}$）时，$\frac{\partial V}{\partial W}=0$。

We would like to comment on the implication of (12). The first term $\hat{\mathop{\mathrm{E}}}\left[{\bm{\delta}}{\bm{h}}^{\rm T}\right]$ is the same as the derivative of the weights without normalization. In this light, the second term in the expression can be seen as the regularization term penalizing the first singular components with the adaptive regularization coefficient $\lambda$. $\lambda$ is positive when ${\bm{\delta}}$ and $\bar{W}_{\rm SN}{\bm{h}}$ are pointing in similar direction, and this prevents the column space of $W$ from concentrating into one particular direction in the course of the training. In other words, spectral normalization prevents the transformation of each layer from becoming to sensitive in one direction. We can also use spectral normalization to devise a new parametrization for the model. Namely, we can split the layer map into two separate trainable components: spectrally normalized map and the spectral norm constant. As it turns out, this parametrization has its merit on its own and promotes the performance of GANs (See Appendix E).

我们想评论 (12) 的含义。第一项 $\hat{\mathop{\mathrm{E}}}\left[{\bm{\delta}}{\bm{h}}^{\rm T}\right]$ 与未归一化时权重的导数相同。由此，表达式中的第二项可视为以 自适应 正则系数 $\lambda$ 惩罚第一奇异分量的正则项。当 ${\bm{\delta}}$ 与 $\bar{W}_{\rm SN}{\bm{h}}$ 指向相近方向时 $\lambda$ 为正，这会阻止训练过程中 $W$ 的列空间集中到某一特定方向。换言之，谱归一化防止各层变换对单一方向过于敏感。也可用谱归一化设计新的模型参数化：把层映射拆成谱归一化映射与谱范数常数两个可训练分量。事实证明该参数化本身有其价值，并能提升 GAN 表现（见附录 E）。

3 Spectral Normalization vs Other Regularization Techniques

3 谱归一化与其他正则技术

The weight normalization introduced by Salimans & Kingma 2016 is a method that normalizes the $\ell_{2}$ norm of each row vector in the weight matrix. Mathematically, this is equivalent to requiring the weight by the weight normalization $\bar{W}_{\rm WN}$:

Salimans & Kingma 2016 引入的权重归一化，是把权重矩阵每行向量的 $\ell_{2}$ 范数归一化。数学上等价于要求权重归一化后的 $\bar{W}_{\rm WN}$ 满足：

$$\displaystyle\sigma_{1}(\bar{W}_{\rm WN})^{2}+\sigma_{2}(\bar{W}_{\rm WN})^{2}+\cdots+\sigma_{T}(\bar{W}_{\rm WN})^{2}=d_{o},~{\rm where}~T=\min(d_{i},d_{o}), \tag{13}$$

where $\sigma_{t}(A)$ is a $t$-th singular value of matrix $A$. Therefore, up to a scaler, this is same as the Frobenius normalization, which requires the sum of the squared singular values to be 1. These normalizations, however, inadvertently impose much stronger constraint on the matrix than intended. If $\bar{W}_{\rm WN}$ is the weight normalized matrix of dimension $d_{i}\times d_{o}$, the norm $\|\bar{W}_{\rm WN}{\bm{h}}\|_{2}$ for a fixed unit vector ${\bm{h}}$ is maximized at $\|\bar{W}_{\rm WN}{\bm{h}}\|_{2}=\sqrt{d_{o}}$ when $\sigma_{1}(\bar{W}_{\rm WN})=\sqrt{d_{o}}$ and $\sigma_{t}(\bar{W}_{\rm WN})=0$ for $t=2,\dots,T$, which means that $\bar{W}_{\rm WN}$ is of rank one. Similar thing can be said to the Frobenius normalization (See the appendix for more details). Using such $\bar{W}_{\rm WN}$ corresponds to using only one feature to discriminate the model probability distribution from the target. In order to retain as much norm of the input as possible and hence to make the discriminator more sensitive, one would hope to make the norm of $\bar{W}_{\rm WN}{\bm{h}}$ large. For weight normalization, however, this comes at the cost of reducing the rank and hence the number of features to be used for the discriminator. Thus, there is a conflict of interests between weight normalization and our desire to use as many features as possible to distinguish the generator distribution from the target distribution. The former interest often reigns over the other in many cases, inadvertently diminishing the number of features to be used by the discriminators. Consequently, the algorithm would produce a rather arbitrary model distribution that matches the target distribution only at select few features. Weight clipping Arjovsky et al. 2017 also suffers from same pitfall.

其中 $\sigma_{t}(A)$ 为矩阵 $A$ 的第 $t$ 个奇异值。因此至多差一个标量，这与要求平方奇异值之和为 1 的 Frobenius 归一化相同。然而这些归一化无意中对矩阵施加了比预期更强的约束。若 $\bar{W}_{\rm WN}$ 为 $d_{i}\times d_{o}$ 的权重归一化矩阵，则对固定单位向量 ${\bm{h}}$，$\|\bar{W}_{\rm WN}{\bm{h}}\|_{2}$ 在 $\sigma_{1}(\bar{W}_{\rm WN})=\sqrt{d_{o}}$ 且 $\sigma_{t}(\bar{W}_{\rm WN})=0$（$t=2,\dots,T$）时取最大 $\|\bar{W}_{\rm WN}{\bm{h}}\|_{2}=\sqrt{d_{o}}$，这意味着 $\bar{W}_{\rm WN}$ 秩为一。Frobenius 归一化亦有类似结论（详见附录）。使用这样的 $\bar{W}_{\rm WN}$ 相当于只用一个特征来区分模型概率分布与目标。为尽可能保留输入范数、从而使判别器更敏感，人们希望 $\|\bar{W}_{\rm WN}{\bm{h}}\|$ 较大。但对权重归一化而言，这要以降低秩、从而减少判别可用特征数为代价。于是权重归一化与「尽量用多特征区分生成器分布与目标」的愿望发生冲突。前者常常压倒后者，无意中削弱判别器可用特征数。结果算法会产出相当任意的模型分布，只在少数选定特征上匹配目标。权重裁剪（Arjovsky et al. 2017）也落入同一陷阱。

Our spectral normalization, on the other hand, do not suffer from such a conflict in interest. Note that the Lipschitz constant of a linear operator is determined only by the maximum singular value. In other words, the spectral norm is independent of rank. Thus, unlike the weight normalization, our spectral normalization allows the parameter matrix to use as many features as possible while satisfying local $1$-Lipschitz constraint. Our spectral normalization leaves more freedom in choosing the number of singular components (features) to feed to the next layer of the discriminator.

我们的谱归一化则不会陷入此类利益冲突。注意线性算子的 Lipschitz 常数只由最大奇异值决定；换言之，谱范数与秩无关。因此与权重归一化不同，谱归一化允许参数矩阵在满足局部 1-Lipschitz 约束的同时尽可能多用特征。谱归一化在选择送入判别器下一层的奇异分量（特征）数目上留下更多自由。

Brock et al. 2016 introduced orthonormal regularization on each weight to stabilize the training of GANs. In their work, Brock et al. 2016 augmented the adversarial objective function by adding the following term:

Brock et al. 2016 对每个权重引入正交归一正则以稳定 GAN 训练。他们在对抗目标上附加如下项：

$$\displaystyle\|W^{\rm T}W-I\|_{F}^{2}. \tag{14}$$

While this seems to serve the same purpose as spectral normalization, orthonormal regularization are mathematically quite different from our spectral normalization because the orthonormal regularization destroys the information about the spectrum by setting all the singular values to one. On the other hand, spectral normalization only scales the spectrum so that the its maximum will be one.

这看似与谱归一化目的相同，但正交归一正则在数学上与谱归一化颇为不同：它把所有奇异值设为 1，从而破坏谱信息；而谱归一化只缩放谱，使最大奇异值为 1。

Gulrajani et al. 2017 used Gradient penalty method in combination with WGAN. In their work, they placed $K$-Lipschitz constant on the discriminator by augmenting the objective function with the regularizer that rewards the function for having local 1-Lipschitz constant (i.e. $\|\nabla_{\hat{{\bm{x}}}}f\|_{2}=1$) at discrete sets of points of the form $\hat{{\bm{x}}}:=\epsilon\tilde{\bm{x}}+(1-\epsilon){\bm{x}}$ generated by interpolating a sample $\tilde{\bm{x}}$ from generative distribution and a sample ${\bm{x}}$ from the data distribution. While this rather straightforward approach does not suffer from the problems we mentioned above regarding the effective dimension of the feature space, the approach has an obvious weakness of being heavily dependent on the support of the current generative distribution. As a matter of course, the generative distribution and its support gradually changes in the course of the training, and this can destabilize the effect of such regularization. In fact, we empirically observed that a high learning rate can destabilize the performance of WGAN-GP. On the contrary, our spectral normalization regularizes the function the operator space, and the effect of the regularization is more stable with respect to the choice of the batch. Training with our spectral normalization does not easily destabilize with aggressive learning rate. Moreover, WGAN-GP requires more computational cost than our spectral normalization with single-step power iteration, because the computation of $\|\nabla_{\hat{{\bm{x}}}}f\|_{2}$ requires one whole round of forward and backward propagation. In the appendix section, we compare the computational cost of the two methods for the same number of updates.

Gulrajani et al. 2017 把梯度惩罚与 WGAN 结合：通过对形如 $\hat{{\bm{x}}}:=\epsilon\tilde{\bm{x}}+(1-\epsilon){\bm{x}}$ 的离散点集（由生成分布样本 $\tilde{\bm{x}}$ 与数据样本 ${\bm{x}}$ 插值得到）奖励局部 1-Lipschitz 常数（即 $\|\nabla_{\hat{{\bm{x}}}}f\|_{2}=1$），在目标上附加正则，从而对判别器施加 $K$-Lipschitz 常数。这一相当直接的做法不会遭受上文关于特征空间有效维的问题，但明显弱点是严重依赖当前生成分布的支撑。训练过程中生成分布及其支撑逐渐变化，可能使此类正则效果不稳定。事实上我们经验上观察到高学习率会使 WGAN-GP 表现不稳。相反，谱归一化在算子空间上正则函数，正则效果相对 batch 选择更稳定；用谱归一化训练时，激进学习率也不易破坏稳定。此外，相对单步幂迭代的谱归一化，WGAN-GP 计算更贵，因为计算 $\|\nabla_{\hat{{\bm{x}}}}f\|_{2}$ 需要一整轮前向与反向。附录中我们按相同更新次数对比两种方法的计算开销。

4 Experiments

4 实验

In order to evaluate the efficacy of our approach and investigate the reason behind its efficacy, we conducted a set of extensive experiments of unsupervised image generation on CIFAR-10 Torralba et al. 2008 and STL-10 Coates et al. 2011, and compared our method against other normalization techniques. To see how our method fares against large dataset, we also applied our method on ILSVRC2012 dataset (ImageNet) Russakovsky et al. 2015 as well. This section is structured as follows. First, we will discuss the objective functions we used to train the architecture, and then we will describe the optimization settings we used in the experiments. We will then explain two performance measures on the images to evaluate the images produced by the trained generators. Finally, we will summarize our results on CIFAR-10, STL-10, and ImageNet.

为评估本方法效力并探究其背后原因，我们在 CIFAR-10（Torralba et al. 2008）与 STL-10（Coates et al. 2011）上做了大量无监督图像生成实验，并与其他归一化技术对比。为观察方法在大数据集上的表现，也在 ILSVRC2012（ImageNet，Russakovsky et al. 2015）上应用本方法。本节结构如下：先讨论训练所用目标函数，再描述优化设置；接着说明用于评估生成图像的两项性能度量；最后总结 CIFAR-10、STL-10 与 ImageNet 上的结果。

As for the architecture of the discriminator and generator, we used convolutional neural networks. Also, for the evaluation of the spectral norm for the convolutional weight $W\in\mathbb{R}^{d_{\rm out}\times d_{\rm in}\times h\times w}$, we treated the operator as a 2-D matrix of dimension $d_{\rm out}\times(d_{\rm in}hw)$[^3]. We trained the parameters of the generator with batch normalization Ioffe & Szegedy 2015. We refer the readers to Table 3 in the appendix section for more details of the architectures.

判别器与生成器结构均用卷积神经网络。对卷积权重 $W\in\mathbb{R}^{d_{\rm out}\times d_{\rm in}\times h\times w}$ 的谱范数，我们将其视为维数为 $d_{\rm out}\times(d_{\rm in}hw)$ 的二维矩阵[^3]。生成器参数用批归一化训练（Ioffe & Szegedy 2015）。结构细节见附录表 3。

For all methods other than WGAN-GP, we used the following standard objective function for the adversarial loss:

除 WGAN-GP 外，所有方法均使用如下标准对抗损失目标：

$$\displaystyle V(G,D):=\mathop{\mathrm{E}}_{x\sim q_{\rm data}({\bm{x}})}[\log D({\bm{x}})]+\mathop{\mathrm{E}}_{{\bm{z}}\sim p({\bm{z}})}[\log(1-D(G({\bm{z}})))], \tag{15}$$

where ${\bm{z}}\in\mathbb{R}^{d_{z}}$ is a latent variable, $p({\bm{z}})$ is the standard normal distribution $\mathcal{N}(0,I)$, and $G:\mathbb{R}^{d_{z}}\to\mathbb{R}^{d_{0}}$ is a deterministic generator function. We set $d_{z}$ to 128 for all of our experiments. For the updates of $G$, we used the alternate cost proposed by Goodfellow et al. 2014 $-\mathop{\mathrm{E}}_{{\bm{z}}\sim p({\bm{z}})}[\log(D(G({\bm{z}})))]$ as used in Goodfellow et al. 2014 and Warde-Farley & Bengio 2017. For the updates of $D$, we used the original cost defined in (15). We also tested the performance of the algorithm with the so-called hinge loss, which is given by

其中 ${\bm{z}}\in\mathbb{R}^{d_{z}}$ 为潜变量，$p({\bm{z}})$ 为标准正态 $\mathcal{N}(0,I)$，$G:\mathbb{R}^{d_{z}}\to\mathbb{R}^{d_{0}}$ 为确定性生成器。所有实验取 $d_{z}=128$。更新 $G$ 时使用 Goodfellow et al. 2014 提出的交替代价 $-\mathop{\mathrm{E}}_{{\bm{z}}\sim p({\bm{z}})}[\log(D(G({\bm{z}})))]$，亦见于 Warde-Farley & Bengio 2017。更新 $D$ 时使用 (15) 中的原始代价。我们也用所谓 hinge 损失测试算法表现，形式为

$$\displaystyle V_{D}(\hat{G},D) \\ \displaystyle=\mathop{\mathrm{E}}_{{\bm{x}}\sim q_{\rm data}({\bm{x}})}\left[{\rm min}\left(0,-1+D({\bm{x}})\right)\right]+\mathop{\mathrm{E}}_{{\bm{z}}\sim p({\bm{z}})}\left[{\rm min}\left(0,-1-D\left(\hat{G}({\bm{z}})\right)\right)\right] \\ \displaystyle V_{G}(G,\hat{D}) \\ \displaystyle=-\mathop{\mathrm{E}}_{{\bm{z}}\sim p({\bm{z}})}\left[\hat{D}\left(G({\bm{z}})\right)\right], \tag{16}$$

respectively for the discriminator and the generator. Optimizing these objectives is equivalent to minimizing the so-called reverse KL divergence : ${\rm KL}[p_{g}||q_{\rm data}]$. This type of loss has been already proposed and used in Lim & Ye 2017; Tran et al. 2017. The algorithm based on the hinge loss also showed good performance when evaluated with inception score and FID. For Wasserstein GANs with gradient penalty (WGAN-GP) Gulrajani et al. 2017, we used the following objective function: $V(G,D):=\mathop{\mathrm{E}}_{{\bm{x}}\sim q_{\rm data}}[D({\bm{x}})]-\mathop{\mathrm{E}}_{{\bm{z}}\sim p({\bm{z}})}[D(G({\bm{z}}))]-\lambda\mathop{\mathrm{E}}_{\hat{{\bm{x}}}\sim p_{\hat{{\bm{x}}}}}[(\|\nabla_{\hat{{\bm{x}}}}D(\hat{{\bm{x}}})\|_{2}-1)^{2}]$, where the regularization term is the one we introduced in the appendix section D.4.

分别对应判别器与生成器。优化这些目标等价于最小化所谓反向 KL 散度：${\rm KL}[p_{g}||q_{\rm data}]$。此类损失已由 Lim & Ye 2017；Tran et al. 2017 提出并使用。基于 hinge 损失的算法在 inception score 与 FID 评估下也表现良好。对带梯度惩罚的 Wasserstein GAN（WGAN-GP，Gulrajani et al. 2017），我们使用：$V(G,D):=\mathop{\mathrm{E}}_{{\bm{x}}\sim q_{\rm data}}[D({\bm{x}})]-\mathop{\mathrm{E}}_{{\bm{z}}\sim p({\bm{z}})}[D(G({\bm{z}}))]-\lambda\mathop{\mathrm{E}}_{\hat{{\bm{x}}}\sim p_{\hat{{\bm{x}}}}}[(\|\nabla_{\hat{{\bm{x}}}}D(\hat{{\bm{x}}})\|_{2}-1)^{2}]$，其中正则项即附录 D.4 所介绍者。

For quantitative assessment of generated examples, we used inception score Salimans et al. 2016 and Fréchet inception distance (FID) Heusel et al. 2017. Please see Appendix B.1 for the details of each score.

对生成样本的定量评估，我们使用 inception score（Salimans et al. 2016）与 Fréchet inception distance（FID，Heusel et al. 2017）。各项分数细节见附录 B.1。

4.1 Results on CIFAR10 and STL-10

4.1 CIFAR10 与 STL-10 上的结果

In this section, we report the accuracy of the spectral normalization (we use the abbreviation: SN-GAN for the spectrally normalized GANs) during the training, and the dependence of the algorithm’s performance on the hyperparmeters of the optimizer. We also compare the performance quality of the algorithm against those of other regularization/normalization techniques for the discriminator networks, including: Weight clipping Arjovsky et al. 2017, WGAN-GP Gulrajani et al. 2017, batch-normalization (BN) Ioffe & Szegedy 2015, layer normalization (LN) Ba et al. 2016, weight normalization (WN) Salimans & Kingma 2016 and orthonormal regularization (orthonormal) Brock et al. 2016. In order to evaluate the stand-alone efficacy of the gradient penalty, we also applied the gradient penalty term to the standard adversarial loss of GANs (15). We would refer to this method as ‘GAN-GP’. For weight clipping, we followed the original work Arjovsky et al. 2017 and set the clipping constant $c$ at 0.01 for the convolutional weight of each layer. For gradient penalty, we set $\lambda$ to 10, as suggested in Gulrajani et al. 2017. For orthonormal, we initialized the each weight of $D$ with a randomly selected orthonormal operator and trained GANs with the objective function augmented with the regularization term used in Brock et al. 2016. For all comparative studies throughout, we excluded the multiplier parameter ${\bm{\gamma}}$ in the weight normalization method, as well as in batch normalization and layer normalization method. This was done in order to prevent the methods from overtly violating the Lipschitz condition. When we experimented with different multiplier parameter, we were in fact not able to achieve any improvement.

本节报告谱归一化（简称 SN-GAN）在训练中的准确度，以及算法表现对优化器超参数的依赖。我们也把算法表现质量与判别器网络上的其他正则/归一化技术对比，包括：权重裁剪（Arjovsky et al. 2017）、WGAN-GP（Gulrajani et al. 2017）、批归一化（BN，Ioffe & Szegedy 2015）、层归一化（LN，Ba et al. 2016）、权重归一化（WN，Salimans & Kingma 2016）以及正交归一正则（orthonormal，Brock et al. 2016）。为单独评估梯度惩罚效力，我们也把梯度惩罚项加到标准 GAN 对抗损失 (15) 上，称该方法为「GAN-GP」。权重裁剪按原作 Arjovsky et al. 2017，每层卷积权重裁剪常数 $c$ 取 0.01。梯度惩罚按 Gulrajani et al. 2017 建议取 $\lambda=10$。对 orthonormal，我们用随机选取的正交算子初始化 $D$ 的每个权重，并用 Brock et al. 2016 的正则项增强目标训练 GAN。所有对比实验中，我们排除权重归一化、批归一化与层归一化中的乘子参数 ${\bm{\gamma}}$，以防方法过度违反 Lipschitz 条件。当我们试验保留不同乘子参数时，实际上未能取得任何改进。

For optimization, we used the Adam optimizer Kingma & Ba 2015 in all of our experiments. We tested with 6 settings for (1) $n_{\rm dis}$, the number of updates of the discriminator per one update of the generator and (2) learning rate $\alpha$ and the first and second order momentum parameters ($\beta_{1}$, $\beta_{2}$) of Adam. We list the details of these settings in Table 1 in the appendix section. Out of these 6 settings, A, B, and C are the settings used in previous representative works. The purpose of the settings D, E, and F is to the evaluate the performance of the algorithms implemented with more aggressive learning rates. For the details of the architectures of convolutional networks deployed in the generator and the discriminator, we refer the readers to Table 3 in the appendix section. The number of updates for GAN generator were 100K for all experiments, unless otherwise noted.

优化一律使用 Adam（Kingma & Ba 2015）。我们测试了 6 组设置，涉及 (1) $n_{\rm dis}$（生成器每更新一次时判别器更新次数）与 (2) 学习率 $\alpha$ 以及 Adam 的一、二阶动量（$\beta_{1}$，$\beta_{2}$）。细节列于附录表 1。其中 A、B、C 为先前代表性工作所用设置；D、E、F 用于评估更激进学习率下的算法表现。生成器与判别器所用卷积网络结构见附录表 3。除非另注，GAN 生成器更新次数均为 100K。

Setting	$\alpha$	$\beta_{1}$	$\beta_{2}$	$n_{\rm dis}$
A$\dagger$	0.0001	0.5	0.9	5
B$\ddagger$	0.0001	0.5	0.999	1
C$\star$	0.0002	0.5	0.999	1
D	0.001	0.5	0.9	5
E	0.001	0.5	0.999	5
F	0.001	0.9	0.999	5

Table 1: Hyper-parameter settings we tested in our experiments. $\dagger$, $\ddagger$ and $\star$ are the hyperparameter settings following Gulrajani et al. 2017, Warde-Farley & Bengio 2017 and Radford et al. 2016, respectively.

Setting	$\alpha$	$\beta_{1}$	$\beta_{2}$	$n_{\rm dis}$
A$\dagger$	0.0001	0.5	0.9	5
B$\ddagger$	0.0001	0.5	0.999	1
C$\star$	0.0002	0.5	0.999	1
D	0.001	0.5	0.9	5
E	0.001	0.5	0.999	5
F	0.001	0.9	0.999	5

表 1：实验中测试的超参数设置。$\dagger$、$\ddagger$ 与 $\star$ 分别沿用 Gulrajani et al. 2017、Warde-Farley & Bengio 2017 与 Radford et al. 2016 的超参数。

Firstly, we inspected the spectral norm of each layer during the training to make sure that our spectral normalization procedure is indeed serving its purpose. As we can see in the Figure 9 in the C.1, the spectral norms of these layers floats around $1$–$1.05$ region throughout the training. Please see Appendix C.1 for more details.

首先，我们检查训练过程中各层谱范数，以确认谱归一化程序确实在起作用。如附录 C.1 图 9 所示，这些层的谱范数在训练全程浮动于约 $1$–$1.05$。更多细节见附录 C.1。

In Figures 1 and  2 we show the inception scores of each method with the settings A–F. We can see that spectral normalization is relatively robust with aggressive learning rates and momentum parameters. WGAN-GP fails to train good GANs at high learning rates and high momentum parameters on both CIFAR-10 and STL-10. Orthonormal regularization performed poorly for the setting E on the STL-10, but performed slightly better than our method with the optimal setting. These results suggests that our method is more robust than other methods with respect to the change in the setting of the training. Also, the optimal performance of weight normalization was inferior to both WGAN-GP and spectral normalization on STL-10, which consists of more diverse examples than CIFAR-10. Best scores of spectral normalization are better than almost all other methods on both CIFAR-10 and STL-10.

图 1 与图 2 展示各方法在设置 A–F 下的 inception score。可见谱归一化对激进学习率与动量参数相对稳健。WGAN-GP 在 CIFAR-10 与 STL-10 上均未能在高学习率与高动量下训出好的 GAN。正交归一正则在 STL-10 的设置 E 上表现较差，但在最优设置下略优于本方法。这些结果表明：相对其他方法，本方法对训练设置变化更稳健。此外，权重归一化的最优表现在比 CIFAR-10 样本更多样的 STL-10 上弱于 WGAN-GP 与谱归一化。谱归一化的最佳分数在 CIFAR-10 与 STL-10 上几乎都优于其他方法。

(a) CIFAR-10

(a) CIFAR-10

In Tables 2, we show the inception scores of the different methods with optimal settings on CIFAR-10 and STL-10 dataset. We see that SN-GANs performed better than almost all contemporaries on the optimal settings. SN-GANs performed even better with hinge loss (17).[^4]. For the training with same number of iterations, SN-GANs fell behind orthonormal regularization for STL-10. For more detailed comparison between orthonormal regularization and spectral normalization, please see section 4.1.2.

表 2 给出不同方法在 CIFAR-10 与 STL-10 上最优设置下的 inception score。可见 SN-GANs 在最优设置下几乎优于所有同期方法；使用 hinge 损失 (17) 时表现更好[^4]。在相同迭代次数下，SN-GANs 在 STL-10 上落后于正交归一正则。更细致的对比见 4.1.2 节。

Method	Inception score	FID
CIFAR-10	STL-10	CIFAR-10	STL-10
Real data	11.24$\pm$.12	26.08$\pm$.26	7.8	7.9
-Standard CNN-
Weight clipping	6.41$\pm$.11	7.57$\pm$.10	42.6	64.2
GAN-GP	6.93$\pm$.08		37.7
WGAN-GP	6.68$\pm$.06	8.42$\pm$.13	40.2	55.1
Batch Norm.	6.27$\pm$.10		56.3
Layer Norm.	7.19$\pm$.12	7.61$\pm$.12	33.9	75.6
Weight Norm.	6.84$\pm$.07	7.16$\pm$.10	34.7	73.4
Orthonormal	7.40$\pm$.12	8.56$\pm$.07	29.0	46.7
(ours) SN-GANs	7.42$\pm$.08	8.28$\pm$.09	29.3	53.1
Orthonormal (2x updates)		8.67$\pm$.08		44.2
(ours) SN-GANs (2x updates)		8.69$\pm$.09		47.5
(ours) SN-GANs, Eq.(17)	7.58$\pm$.12		25.5
(ours) SN-GANs, Eq.(17) (2x updates)		8.79$\pm$.14		43.2
-ResNet-[^5]
Orthonormal, Eq.(17)	7.92$\pm$.04	8.72$\pm$.06	23.8$\pm$.58	42.4$\pm$.99
(ours) SN-GANs, Eq.(17)	8.22$\pm$.05	9.10$\pm$.04	21.7$\pm$.21	40.1$\pm$.50
DCGAN$\dagger$	6.64$\pm$.14	7.84$\pm$.07
LR-GANs$\ddagger$	7.17$\pm$.07
Warde-Farley et al.$*$	7.72$\pm$.13	8.51$\pm$.13
WGAN-GP (ResNet)$\dagger\dagger$	7.86$\pm$.08

Table 2: Inception scores and FIDs with unsupervised image generation on CIFAR-10. $\dagger$ Radford et al. 2016 (experimented by  Yang et al. 2017), $\ddagger$ Yang et al. 2017, $*$ Warde-Farley & Bengio 2017, $\dagger\dagger$ Gulrajani et al. 2017

Method	Inception score	FID
CIFAR-10	STL-10	CIFAR-10	STL-10
Real data	11.24$\pm$.12	26.08$\pm$.26	7.8	7.9
-Standard CNN-
Weight clipping	6.41$\pm$.11	7.57$\pm$.10	42.6	64.2
GAN-GP	6.93$\pm$.08		37.7
WGAN-GP	6.68$\pm$.06	8.42$\pm$.13	40.2	55.1
Batch Norm.	6.27$\pm$.10		56.3
Layer Norm.	7.19$\pm$.12	7.61$\pm$.12	33.9	75.6
Weight Norm.	6.84$\pm$.07	7.16$\pm$.10	34.7	73.4
Orthonormal	7.40$\pm$.12	8.56$\pm$.07	29.0	46.7
(ours) SN-GANs	7.42$\pm$.08	8.28$\pm$.09	29.3	53.1
Orthonormal (2x updates)		8.67$\pm$.08		44.2
(ours) SN-GANs (2x updates)		8.69$\pm$.09		47.5
(ours) SN-GANs, Eq.(17)	7.58$\pm$.12		25.5
(ours) SN-GANs, Eq.(17) (2x updates)		8.79$\pm$.14		43.2
-ResNet-[^5]
Orthonormal, Eq.(17)	7.92$\pm$.04	8.72$\pm$.06	23.8$\pm$.58	42.4$\pm$.99
(ours) SN-GANs, Eq.(17)	8.22$\pm$.05	9.10$\pm$.04	21.7$\pm$.21	40.1$\pm$.50
DCGAN$\dagger$	6.64$\pm$.14	7.84$\pm$.07
LR-GANs$\ddagger$	7.17$\pm$.07
Warde-Farley et al.$*$	7.72$\pm$.13	8.51$\pm$.13
WGAN-GP (ResNet)$\dagger\dagger$	7.86$\pm$.08

表 2：CIFAR-10 无监督图像生成的 inception score 与 FID。$\dagger$ Radford et al. 2016（由 Yang et al. 2017 实验），$\ddagger$ Yang et al. 2017，$*$ Warde-Farley & Bengio 2017，$\dagger\dagger$ Gulrajani et al. 2017。

In Figure 6 we show the images produced by the generators trained with WGAN-GP, weight normalization, and spectral normalization. SN-GANs were consistently better than GANs with weight normalization in terms of the quality of generated images. To be more precise, as we mentioned in Section 3, the set of images generated by spectral normalization was clearer and more diverse than the images produced by the weight normalization. We can also see that WGAN-GP failed to train good GANs with high learning rates and high momentums (D,E and F). The generated images with GAN-GP, batch normalization, and layer normalization is shown in Figure 12 in the appendix section.

图 6 展示用 WGAN-GP、权重归一化与谱归一化训练的生成器所产图像。就生成图像质量而言，SN-GANs 始终优于带权重归一化的 GAN。更精确地说，如第 3 节所述，谱归一化生成的图像集合比权重归一化更清晰、更多样。也可看到 WGAN-GP 在高学习率与高动量（D、E、F）下未能训出好的 GAN。GAN-GP、批归一化与层归一化生成的图像见附录图 12。

We also compared our algorithm against multiple benchmark methods ans summarized the results on the bottom half of the Table 2. We also tested the performance of our method on ResNet based GANs used in Gulrajani et al. 2017. Please note that all methods listed thereof are all different in both optimization methods and the architecture of the model. Please see Table 8 and  5 in the appendix section for the detail network architectures. Our implementation of our algorithm was able to perform better than almost all the predecessors in the performance.

我们还与多种基准方法对比，结果总结于表 2 下半部。我们也在 Gulrajani et al. 2017 所用的基于 ResNet 的 GAN 上测试本方法。请注意其中所列方法在优化方法与模型结构上均有不同。网络结构细节见附录表 8 与表 5。我们的实现几乎优于所有先前工作。

4.1.1 Analysis of SN-GANs

(a) CIFAR-10

4.1.1 对 SN-GANs 的分析

(a) CIFAR-10

Singular values analysis on the weights of the discriminator $D$

In Figure 3, we show the squared singular values of the weight matrices in the final discriminator $D$ produced by each method using the parameter that yielded the best inception score. As we predicted in Section 3, the singular values of the first to fifth layers trained with weight clipping and weight normalization concentrate on a few components. That is, the weight matrices of these layers tend to be rank deficit. On the other hand, the singular values of the weight matrices in those layers trained with spectral normalization is more broadly distributed. When the goal is to distinguish a pair of probability distributions on the low-dimensional nonlinear data manifold embedded in a high dimensional space, rank deficiencies in lower layers can be especially fatal. Outputs of lower layers have gone through only a few sets of rectified linear transformations, which means that they tend to lie on the space that is linear in most parts. Marginalizing out many features of the input distribution in such space can result in oversimplified discriminator. We can actually confirm the effect of this phenomenon on the generated images especially in Figure 6(b). The images generated with spectral normalization is more diverse and complex than those generated with weight normalization.

判别器 $D$ 权重上的奇异值分析

图 3 展示各方法用取得最佳 inception score 的参数训出的最终判别器 $D$ 中，权重矩阵的平方奇异值。如第 3 节所预测，权重裁剪与权重归一化训练的第一到第五层奇异值集中在少数分量上，即这些层的权重矩阵趋于秩亏。相反，谱归一化训练的这些层奇异值分布更广。当目标是区分嵌入在高维空间中低维非线性数据流形上的一对概率分布时，较低层的秩亏尤其致命。较低层输出只经过少数几组修正线性变换，因而大多落在局部近似线性的空间；在此空间中边缘化掉输入分布的许多特征，会导致过度简化的判别器。我们可在图 6(b) 等生成图像上实际确认该现象的影响：谱归一化生成的图像比权重归一化更多样、更复杂。

Training time

On CIFAR-10, SN-GANs is slightly slower than weight normalization (about 110 $\sim$ 120% computational time), but significantly faster than WGAN-GP. As we mentioned in Section 3, WGAN-GP is slower than other methods because WGAN-GP needs to calculate the gradient of gradient norm $\|\nabla_{{\bm{x}}}D\|_{2}$. For STL-10, the computational time of SN-GANs is almost the same as vanilla GANs, because the relative computational cost of the power iteration (18) is negligible when compared to the cost of forward and backward propagation on CIFAR-10 (images size of STL-10 is larger ($48\times 48$)). Please see Figure 10 in the appendix section for the actual computational time.

训练时间

在 CIFAR-10 上，SN-GANs 略慢于权重归一化（约 110～120% 计算时间），但显著快于 WGAN-GP。如第 3 节所述，WGAN-GP 更慢是因为需要计算梯度范数 $\|\nabla_{{\bm{x}}}D\|_{2}$ 的梯度。对 STL-10，SN-GANs 的计算时间几乎与原版 GAN 相同，因为相对 CIFAR-10 上前向与反向的开销，幂迭代 (18) 的相对成本可忽略（STL-10 图像更大，$48\times 48$）。实际计算时间见附录图 10。

4.1.2 Comparison between SN-GANs and orthonormal regularization

Figure 4: The effect on the performance on STL-10 induced by the change of the feature map dimension of the final layer. The width of the highlighted region represents standard deviation of the results over multiple seeds of weight initialization. The orthonormal regularization does not perform well with large feature map dimension, possibly because of its design that forces the discriminator to use all dimensions including the ones that are unnecessary. For the setting of the optimizers’ hyper-parameters, We used the setting C, which was optimal for “orthonormal regularization”

4.1.2 SN-GANs 与正交归一正则的对比

图 4：改变最后一层特征图维数对 STL-10 表现的影响。高亮区域宽度表示多种权重初始化种子下结果的标准差。正交归一正则 在大特征图维数下表现不佳，可能因其设计迫使判别器使用所有维，包括不必要的维。优化器超参数我们使用对「正交归一正则」最优的设置 C。

In order to highlight the difference between our spectral normalization and orthonormal regularization, we conducted an additional set of experiments. As we explained in Section 3, orthonormal regularization is different from our method in that it destroys the spectral information and puts equal emphasis on all feature dimensions, including the ones that ’shall’ be weeded out in the training process. To see the extent of its possibly detrimental effect, we experimented by increasing the dimension of the feature space [^6], especially at the final layer (7th conv) for which the training with our spectral normalization prefers relatively small feature space (dimension $<100$; see Figure 3(b)). As for the setting of the training, we selected the parameters for which the orthonormal regularization performed optimally. The figure 4 shows the result of our experiments. As we predicted, the performance of the orthonormal regularization deteriorates as we increase the dimension of the feature maps at the final layer. Our SN-GANs, on the other hand, does not falter with this modification of the architecture. Thus, at least in this perspective, we may such that our method is more robust with respect to the change of the network architecture.

为突出谱归一化与正交归一正则的差别，我们做了额外实验。如第 3 节所释，正交归一正则与本方法不同之处在于它破坏谱信息，并对所有特征维一视同仁——包括训练过程中「应当」被淘汰的维。为观察其可能的有害程度，我们增大特征空间维数[^6]，尤其是最后一层（第 7 个卷积），而谱归一化训练偏好相对较小的特征空间（维数 $<100$；见图 3(b)）。训练设置选取正交归一正则表现最优的参数。图 4 给出实验结果。如所预测，随最后一层特征图维数增大，正交归一正则表现变差；而 SN-GANs 不因这一结构修改而动摇。至少从这一角度看，可以说本方法对网络结构变化更稳健。

4.2 Image Generation on ImageNet

4.2 ImageNet 上的图像生成

Figure 5: Learning curves for conditional image generation in terms of Inception score for SN-GANs and GANs with orthonormal regularization on ImageNet.

图 5：ImageNet 上条件图像生成的学习曲线（以 Inception score 计），对比 SN-GANs 与带 正交归一正则 的 GAN。

To show that our method remains effective on a large high dimensional dataset, we also applied our method to the training of conditional GANs on ILRSVRC2012 dataset with 1000 classes, each consisting of approximately 1300 images, which we compressed to $128\times 128$ pixels. Regarding the adversarial loss for conditional GANs, we used practically the same formulation used in Mirza & Osindero 2014, except that we replaced the standard GANs loss with hinge loss (17). Please see Appendix B.3 for the details of experimental settings.

为表明本方法在大型高维数据集上仍然有效，我们还把方法用于 ILSVRC2012 上的条件 GAN 训练：1000 类，每类约 1300 张图，压缩到 $128\times 128$ 像素。条件 GAN 的对抗损失实际沿用 Mirza & Osindero 2014 的表述，只是把标准 GAN 损失换成 hinge 损失 (17)。实验设置细节见附录 B.3。

GANs without normalization and GANs with layer normalization collapsed in the beginning of training and failed to produce any meaningful images. GANs with orthonormal normalization Brock et al. 2016 and our spectral normalization, on the other hand, was able to produce images. The inception score of the orthonormal normalization however plateaued around 20Kth iterations, while SN kept improving even afterward (Figure 5.) To our knowledge, our research is the first of its kind in succeeding to produce decent images from ImageNet dataset with a single pair of a discriminator and a generator (Figure 7). To measure the degree of mode-collapse, we followed the footstep of Odena et al. 2017 and computed the intra MS-SSIM Odena et al. 2017 for pairs of independently generated GANs images of each class. We see that our SN-GANs ((intra MS-SSIM)=0.101) is suffering less from the mode-collapse than AC-GANs ((intra MS-SSIM)$\sim$0.25).

无归一化的 GAN 与带层归一化的 GAN 在训练初期即崩溃，未能产生任何有意义图像。带正交归一（Brock et al. 2016）与我们的谱归一化的 GAN 则能产生图像。正交归一的 inception score 约在第 20K 次迭代后平台化，而 SN 之后仍继续提升（图 5）。据我们所知，本研究是首次用 单对 判别器与生成器从 ImageNet 数据集成功生成像样图像（图 7）。为度量模式崩溃程度，我们沿 Odena et al. 2017，对每类独立生成的 GAN 图像对计算 intra MS-SSIM（Odena et al. 2017）。可见 SN-GANs（intra MS-SSIM=0.101）比 AC-GANs（intra MS-SSIM$\sim$0.25）更少受模式崩溃困扰。

To ensure that the superiority of our method is not limited within our specific setting, we also compared the performance of SN-GANs against orthonormal regularization on conditional GANs with projection discriminator Miyato & Koyama 2018 as well as the standard (unconditional) GANs. In our experiments, SN-GANs achieved better performance than orthonormal regularization for the both settings (See Figure 15 in the appendix section).

为确认本方法优势不限于特定设定，我们还在带 投影判别器（Miyato & Koyama 2018）的条件 GAN 以及标准（无条件）GAN 上，对比 SN-GANs 与 正交归一正则。实验中两种设定下 SN-GANs 均优于 正交归一正则（见附录图 15）。

5 Conclusion

5 结论

This paper proposes spectral normalization as a stabilizer of training of GANs. When we apply spectral normalization to the GANs on image generation tasks, the generated examples are more diverse than the conventional weight normalization and achieve better or comparative inception scores relative to previous studies. The method imposes global regularization on the discriminator as opposed to local regularization introduced by WGAN-GP, and can possibly used in combinations. In the future work, we would like to further investigate where our methods stand amongst other methods on more theoretical basis, and experiment our algorithm on larger and more complex datasets.

本文提出谱归一化作为 GAN 训练的稳定器。将谱归一化用于图像生成任务时，生成样本比传统权重归一化更多样，并相对先前研究取得更好或相当的 inception score。该方法对判别器施加全局正则，相对 WGAN-GP 引入的局部正则，并可能与之组合使用。未来工作中，我们希望在更理论的基础上进一步考察本方法相对其他方法的位置，并在更大、更复杂的数据集上实验本算法。

Acknowledgments

We would like to thank the members of Preferred Networks, Inc., particularly Shin-ichi Maeda, Eiichi Matsumoto, Masaki Watanabe and Keisuke Yahata for insightful comments and discussions. We also would like to thank anonymous reviewers and commenters on the OpenReview forum for insightful discussions.

致谢

感谢 Preferred Networks, Inc. 的成员，尤其是 Shin-ichi Maeda、Eiichi Matsumoto、Masaki Watanabe 与 Keisuke Yahata 的深刻评论与讨论。也感谢 OpenReview 论坛上的匿名审稿人与评论者的深入讨论。

References

参考文献

(a) CIFAR-10

Figure 7: 128x128 pixel images generated by SN-GANs trained on ILSVRC2012 dataset. The inception score is 21.1$\pm$.35.

图 7：在 ILSVRC2012 上训练的 SN-GANs 生成的 128×128 像素图像。inception score 为 21.1$\pm$.35。

Arjovsky & Bottou (2017) Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In ICLR , 2017.

Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML , pp. 214–223, 2017.

Arpit et al. (2016) Devansh Arpit, Yingbo Zhou, Bhargava U Kota, and Venu Govindaraju. Normalization propagation: A parametric technique for removing internal covariate shift in deep networks. In ICML , pp. 1168–1176, 2016.

Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Brock et al. (2016) Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv:1609.07093 , 2016.

Coates et al. (2011) Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS , pp. 215–223, 2011.

de Vries et al. (2017) Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In NIPS , pp. 6576–6586, 2017.

Dowson & Landau (1982) DC Dowson and BV Landau. The fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis , 12(3):450–455, 1982.

Dumoulin et al. (2017) Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In ICLR , 2017.

Glorot et al. (2011) Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS , pp. 315–323, 2011.

Golub & Van der Vorst (2000) Gene H Golub and Henk A Van der Vorst. Eigenvalue computation in the 20th century. Journal of Computational and Applied Mathematics , 123(1):35–65, 2000.

Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS , pp. 2672–2680, 2014.

Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein GANs. arXiv preprint arXiv:1704.00028 , 2017.

He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pp. 770–778, 2016.

Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a nash equilibrium. arXiv preprint arXiv:1706.08500 , 2017.

Ho & Ermon (2016) Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NIPS , pp. 4565–4573, 2016.

Ioffe & Szegedy (2015) Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML , pp. 448–456, 2015.

Jarrett et al. (2009) Kevin Jarrett, Koray Kavukcuoglu, Marc’Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In ICCV , pp. 2146–2153, 2009.

Jia et al. (2017) Kui Jia, Dacheng Tao, Shenghua Gao, and Xiangmin Xu. Improving training of deep neural networks via singular value bounding. In CVPR , 2017.

Kingma & Ba (2015) Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Li et al. (2017) Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. In EMNLP , pp. 2147–2159, 2017.

Lim & Ye (2017) Jae Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv preprint arXiv:1705.02894 , 2017.

Maas et al. (2013) Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing , 2013.

Mirza & Osindero (2014) Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 , 2014.

Miyato & Koyama (2018) Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In ICLR , 2018.

Mohamed & Lakshminarayanan (2017) Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. NIPS Workshop on Adversarial Training , 2017.

Nair & Hinton (2010) Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML , pp. 807–814, 2010.

Nowozin et al. (2016) Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. In NIPS , pp. 271–279, 2016.

Odena et al. (2017) Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier GANs. In ICML , pp. 2642–2651, 2017.

Qi (2017) Guo-Jun Qi. Loss-sensitive generative adversarial networks on lipschitz densities. arXiv preprint arXiv:1701.06264 , 2017.

Radford et al. (2016) Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR , 2016.

Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 115(3):211–252, 2015.

Saito et al. (2017) Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In ICCV , 2017.

Salimans & Kingma (2016) Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In NIPS , pp. 901–909, 2016.

Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NIPS , pp. 2226–2234, 2016.

Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR , pp. 1–9, 2015.

Tokui et al. (2015) Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: a next-generation open source framework for deep learning. In Proceedings of workshop on machine learning systems (LearningSys) in the twenty-ninth annual conference on neural information processing systems (NIPS) , 2015.

Torralba et al. (2008) Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 30(11):1958–1970, 2008.

Tran et al. (2017) Dustin Tran, Rajesh Ranganath, and David M Blei. Deep and hierarchical implicit models. arXiv preprint arXiv:1702.08896 , 2017.

Uehara et al. (2016) Masatoshi Uehara, Issei Sato, Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Generative adversarial nets from a density ratio estimation perspective. NIPS Workshop on Adversarial Training , 2016.

Warde-Farley & Bengio (2017) David Warde-Farley and Yoshua Bengio. Improving generative adversarial networks with denoising feature matching. In ICLR , 2017.

Xiang & Li (2017) Sitao Xiang and Hao Li. On the effect of batch normalization and weight normalization in generative adversarial networks. arXiv preprint arXiv:1704.03971 , 2017.

Yang et al. (2017) Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. LR-GAN: Layered recursive generative adversarial networks for image generation. ICLR , 2017.

Yoshida & Miyato (2017) Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv:1705.10941 , 2017.

Appendix A The Algorithm of Spectral Normalization

附录 A 谱归一化算法

Let us describe the shortcut in Section 2.1 in more detail. We begin with vectors $\tilde{{\bm{u}}}$ that is randomly initialized for each weight. If there is no multiplicity in the dominant singular values and if $\tilde{{\bm{u}}}$ is not orthogonal to the first left singular vectors[^7], we can appeal to the principle of the power method and produce the first left and right singular vectors through the following update rule:

让我们更详细描述 2.1 节中的捷径。对每个权重从随机初始化的向量 $\tilde{{\bm{u}}}$ 开始。若主导奇异值无重数，且 $\tilde{{\bm{u}}}$ 不与第一左奇异向量正交[^7]，则可诉诸幂法原理，通过如下更新规则得到第一左、右奇异向量：

$$\displaystyle\tilde{{\bm{v}}}\leftarrow W^{\rm T}\tilde{{\bm{u}}}/\|W^{\rm T}\tilde{{\bm{u}}}\|_{2},~\tilde{{\bm{u}}}\leftarrow W\tilde{{\bm{v}}}/\|W\tilde{{\bm{v}}}\|_{2}. \tag{18}$$

We can then approximate the spectral norm of $W$ with the pair of so-approximated singular vectors:

随后可用如此近似的奇异向量对估计 $W$ 的谱范数：

$$\displaystyle\sigma(W)\approx\tilde{{\bm{u}}}^{\rm T}W\tilde{{\bm{v}}}. \tag{19}$$

If we use SGD for updating $W$, the change in $W$ at each update would be small, and hence the change in its largest singular value. In our implementation, we took advantage of this fact and reused the $\tilde{{\bm{u}}}$ computed at each step of the algorithm as the initial vector in the subsequent step. In fact, with this ‘recycle’ procedure, one round of power iteration was sufficient in the actual experiment to achieve satisfactory performance. Algorithm 1 in the appendix summarizes the computation of the spectrally normalized weight matrix $\bar{W}$ with this approximation. Note that this procedure is very computationally cheap even in comparison to the calculation of the forward and backward propagations on neural networks. Please see Figure 10 for actual computational time with and without spectral normalization.

若用 SGD 更新 $W$，则每步 $W$ 的变化很小，从而最大奇异值变化也小。实现中我们利用这一点，把每步算出的 $\tilde{{\bm{u}}}$ 作为下一步的初始向量。事实上，借助这种「回收」手续，实际实验中一轮幂迭代就足以取得满意表现。附录算法 1 总结了用该近似计算谱归一化权重矩阵 $\bar{W}$ 的过程。注意即便与神经网络前向与反向传播相比，该手续计算也非常便宜。有无谱归一化的实际计算时间见图 10。

Initialize $\tilde{{\bm{u}}}_{l}\in\mathcal{R}^{d_{l}}~{\rm for}~l=1,\dots,L$ with a random vector (sampled from isotropic distribution).

For each update and each layer $l$:

Apply power iteration method to a unnormalized weight $W^{l}$:

Calculate $\bar{W}_{\rm SN}$ with the spectral norm:

Update $W^{l}$ with SGD on mini-batch dataset $\mathcal{D}_{M}$ with a learning rate $\alpha$:

Algorithm 1 SGD with spectral normalization

对 $l=1,\dots,L$ 用随机向量（从各向同性分布采样）初始化 $\tilde{{\bm{u}}}_{l}\in\mathcal{R}^{d_{l}}$。

对每次更新与每层 $l$：

对未归一化权重 $W^{l}$ 施加幂迭代：

用谱范数计算 $\bar{W}_{\rm SN}$：

在 mini-batch 数据集 $\mathcal{D}_{M}$ 上以学习率 $\alpha$ 用 SGD 更新 $W^{l}$：

算法 1 带谱归一化的 SGD

Appendix B Experimental Settings

附录 B 实验设置

B.1 Performance measures

B.1 性能度量

Inception score is introduced originally by Salimans et al. 2016: $I(\{x_{n}\}_{n=1}^{N}):=\exp(\mathop{\mathrm{E}}[D_{\rm KL}[p(y|{\bm{x}})||p(y)]])$, where $p(y)$ is approximated by $\frac{1}{N}\sum_{n=1}^{N}p(y|{\bm{x}}_{n})$ and $p(y|x)$ is the trained Inception convolutional neural network Szegedy et al. 2015, which we would refer to Inception model for short. In their work, Salimans et al. 2016 reported that this score is strongly correlated with subjective human judgment of image quality. Following the procedure in Salimans et al. 2016; Warde-Farley & Bengio 2017, we calculated the score for randomly generated 5000 examples from each trained generator to evaluate its ability to generate natural images. We repeated each experiment 10 times and reported the average and the standard deviation of the inception scores.

Inception score 最初由 Salimans et al. 2016 引入：$I(\{x_{n}\}_{n=1}^{N}):=\exp(\mathop{\mathrm{E}}[D_{\rm KL}[p(y|{\bm{x}})||p(y)]])$，其中 $p(y)$ 由 $\frac{1}{N}\sum_{n=1}^{N}p(y|{\bm{x}}_{n})$ 近似，$p(y|x)$ 为训练好的 Inception 卷积神经网络（Szegedy et al. 2015），下文简称 Inception 模型。Salimans et al. 2016 报告该分数与图像质量的主观人类判断强相关。沿用 Salimans et al. 2016；Warde-Farley & Bengio 2017 的手续，我们对每个训练好的生成器随机生成 5000 个样本计算分数，以评估其生成自然图像的能力。每项实验重复 10 次，报告 inception score 的均值与标准差。

Fréchet inception distance Heusel et al. 2017 is another measure for the quality of the generated examples that uses 2nd order information of the final layer of the inception model applied to the examples. On its own, the Frećhet distance Dowson & Landau 1982 is 2-Wasserstein distance between two distribution $p_{1}$ and $p_{2}$ assuming they are both multivariate Gaussian distributions:

Fréchet inception distance（Heusel et al. 2017）是另一项生成样本质量度量，使用 Inception 模型最终层作用于样本的二阶信息。就其本身而言，Fréchet 距离（Dowson & Landau 1982）是假定两分布 $p_{1}$、$p_{2}$ 均为多元高斯时的 2-Wasserstein 距离：

$$\displaystyle F(p_{1},p_{2})=\|{\bm{\mu}}_{p_{1}}-{\bm{\mu}}_{p_{2}}\|_{2}^{2}+{\rm trace}\left({C_{{p_{1}}}+C_{p_{2}}-2(C_{p_{1}}C_{p_{2}})^{1/2}}\right), \tag{24}$$

where $\{{\bm{\mu}}_{p_{1}},C_{p_{1}}\}$, $\{{\bm{\mu}}_{p_{2}},C_{p_{2}}\}$ are the mean and covariance of samples from $q$ and $p$, respectively. If $f_{\ominus}$ is the output of the final layer of the inception model before the softmax, the Fréchet inception distance (FID) between two distributions $p_{1}$ and $p_{2}$ on the images is the distance between $f_{\ominus}\circ p_{1}$ and $f_{\ominus}\circ p_{2}$. We computed the Fréchet inception distance between the true distribution and the generated distribution empirically over 10000 and 5000 samples. Multiple repetition of the experiments did not exhibit any notable variations on this score.

其中 $\{{\bm{\mu}}_{p_{1}},C_{p_{1}}\}$、$\{{\bm{\mu}}_{p_{2}},C_{p_{2}}\}$ 分别为来自 $q$ 与 $p$ 的样本均值与协方差。若 $f_{\ominus}$ 为 Inception 模型 softmax 前最终层输出，则图像上两分布 $p_{1}$、$p_{2}$ 的 Fréchet inception distance（FID）是 $f_{\ominus}\circ p_{1}$ 与 $f_{\ominus}\circ p_{2}$ 之间的距离。我们在真分布与生成分布上分别用 10000 与 5000 个样本经验计算 FID。多次重复实验未显示该分数有明显波动。

B.2 Image Generation on CIFAR-10 and STL-10

B.2 CIFAR-10 与 STL-10 上的图像生成

For the comparative study, we experimented with the recent ResNet architecture of Gulrajani et al. 2017 as well as the standard CNN. For this additional set of experiments, we used Adam again for the optimization and used the very hyper parameter used in Gulrajani et al. 2017 ($\alpha=0.0002,\beta_{1}=0,\beta_{2}=0.9,　n_{dis}=5$). For our SN-GANs, we doubled the feature map in the generator from the original, because this modification achieved better results. Note that when we doubled the dimension of the feature map for the WGAN-GP experiment, however, the performance deteriorated.

对比研究中，我们试验了 Gulrajani et al. 2017 近期的 ResNet 结构以及标准 CNN。这组额外实验仍用 Adam，并采用 Gulrajani et al. 2017 所用超参数（$\alpha=0.0002,\beta_{1}=0,\beta_{2}=0.9,\ n_{dis}=5$）。对 SN-GANs，我们把生成器特征图相对原设定加倍，因为该修改效果更好。注意对 WGAN-GP 实验加倍特征图维数时，表现反而变差。

B.3 Image Generation on ImageNet

B.3 ImageNet 上的图像生成

The images used in this set of experiments were resized to $128\times 128$ pixels. The details of the architecture are given in Table 6. For the generator network of conditional GANs, we used conditional batch normalization (CBN) Dumoulin et al. 2017; de Vries et al. 2017. Namely we replaced the standard batch normalization layer with the CBN conditional to the label information $y\in\{1,\dots,1000\}$. For the optimization, we used Adam with the same hyperparameters we used for ResNet on CIFAR-10 and STL-10 dataset. We trained the networks with 450K generator updates, and applied linear decay for the learning rate after 400K iterations so that the rate would be 0 at the end.

本组实验图像缩放至 $128\times 128$ 像素。结构细节见表 6。条件 GAN 的生成器网络使用条件批归一化（CBN，Dumoulin et al. 2017；de Vries et al. 2017）：把标准批归一化层换成以标签信息 $y\in\{1,\dots,1000\}$ 为条件的 CBN。优化使用与 CIFAR-10、STL-10 上 ResNet 相同的 Adam 超参数。网络训练 450K 次生成器更新，并在 400K 次迭代后对学习率线性衰减，使结束时为 0。

B.4 Network Architectures

B.4 网络结构

$z\in\mathbb{R}^{128}\sim\mathcal{N}(0,I)$
dense $\rightarrow$ $M_{g}$ $\times$ $M_{g}$ $\times$ 512
4$\times$4, stride=2 deconv. BN 256 ReLU
4$\times$4, stride=2 deconv. BN 128 ReLU
4$\times$4, stride=2 deconv. BN 64 ReLU
3$\times$3, stride=1 conv. 3 Tanh

RGB image $x\in\mathbb{R}^{M\times M\times 3}$
3$\times$3, stride=1 conv 64 lReLU
4$\times$4, stride=2 conv 64 lReLU
3$\times$3, stride=1 conv 128 lReLU
4$\times$4, stride=2 conv 128 lReLU
3$\times$3, stride=1 conv 256 lReLU
4$\times$4, stride=2 conv 256 lReLU
3$\times$3, stride=1 conv. 512 lReLU
dense $\rightarrow$ 1

Table 3: Standard CNN models for CIFAR-10 and STL-10 used in our experiments on image Generation. The slopes of all lReLU functions in the networks are set to $0.1$.

$z\in\mathbb{R}^{128}\sim\mathcal{N}(0,I)$
dense $\rightarrow$ $M_{g}$ $\times$ $M_{g}$ $\times$ 512
4$\times$4, stride=2 deconv. BN 256 ReLU
4$\times$4, stride=2 deconv. BN 128 ReLU
4$\times$4, stride=2 deconv. BN 64 ReLU
3$\times$3, stride=1 conv. 3 Tanh

RGB image $x\in\mathbb{R}^{M\times M\times 3}$
3$\times$3, stride=1 conv 64 lReLU
4$\times$4, stride=2 conv 64 lReLU
3$\times$3, stride=1 conv 128 lReLU
4$\times$4, stride=2 conv 128 lReLU
3$\times$3, stride=1 conv 256 lReLU
4$\times$4, stride=2 conv 256 lReLU
3$\times$3, stride=1 conv. 512 lReLU
dense $\rightarrow$ 1

表 3：图像生成实验中用于 CIFAR-10 与 STL-10 的标准 CNN 模型。网络中所有 lReLU 斜率设为 $0.1$。