ImageNet Classification with Deep Convolutional Neural Networks

Alex Krizhevsky 1 , Ilya Sutskever 1 , Geoffrey E. Hinton 1

ImageNet Classification with Deep Convolutional Neural Networks

Alex Krizhevsky Ilya Sutskever Geoffrey E. Hinton University of Toronto University of Toronto University of Toronto kriz@cs.utoronto.ca ilya@cs.utoronto.ca hinton@cs.utoronto.ca

Abstract

摘要

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 dif- ferent classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make train- ing faster, we used non-saturating neurons and a very efficient GPU implemen- tation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.

我们训练了一座大型深度卷积神经网络，把 ImageNet LSVRC-2010 竞赛中的 120 万张高分辨率图像分到 1000 个类别。在测试集上，top-1 与 top-5 错误率分别达到 37.5% 与 17.0%，明显好于此前最优结果。该网络约有 6000 万参数、65 万个神经元，含五层卷积（部分后接最大池化）与三层全连接，末端为 1000 路 softmax。为加快训练，我们使用非饱和神经元，并对卷积运算做了高效的 GPU 实现。为减轻全连接层过拟合，我们采用了当时新提出、效果很好的正则化方法「dropout」。该模型的一个变体还参加了 ILSVRC-2012，以 15.3% 的 top-5 测试错误率夺冠，第二名是 26.2%。

1 Introduction

1 引言

Current approaches to object recognition make essential use of machine learning methods. To im- prove their performance, we can collect larger datasets, learn more powerful models, and use bet- ter techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the current- best error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to col- lect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories. To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this prob- lem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be con- trolled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

当前物体识别方法本质上依赖机器学习。要再往上推，可以收集更大的数据集、学习更强的模型，并采用更好的防过拟合手段。直到不久前，带标注的图像集还相对较小——通常只有数万张（例如 NORB [16]、Caltech-101/256 [8, 9]、CIFAR-10/100 [12]）。对简单识别任务，这种规模往往够用，尤其再配合保持标签的变换扩增。例如 MNIST 数字识别当前最优错误率已低于 0.3%，接近人类水平 [4]。但真实场景中物体变化极大，要学会识别就必须用大得多的训练集。小图像集的不足早已被广泛指出（如 Pinto 等 [21]），而收集百万级带标注图像也只是最近才变得可行。新的大型数据集包括含数十万张全分割图像的 LabelMe [23]，以及拥有逾 1500 万张高分辨率标注图像、覆盖两万二千余类别的 ImageNet [6]。要从数百万图像中学习数千种物体，模型需要很大的学习容量。但物体识别任务本身极其复杂，即便 ImageNet 这样的规模也无法把问题完全「写死」，因此模型还应带有大量先验，以弥补我们没有的数据。卷积神经网络（CNN）正是这类模型 [16, 11, 13, 18, 15, 22, 26]。其容量可通过深度与宽度调节，并对图像性质做了强且大体正确的假设（统计平稳性与像素依赖的局部性）。因此，与同等规模的标准前馈网络相比，CNN 的连接与参数少得多，更易训练，而理论上的最优性能通常只会略差一点。

1

Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luck- ily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting. The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly1 . Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3. The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting, which are described in Section 4. Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than 1% of the model’s parameters) resulted in inferior performance. In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.

尽管 CNN 优点突出、局部结构也相对高效，把它们大规模用到高分辨率图像上仍长期代价过高。幸运的是，当前 GPU 配合高度优化的二维卷积实现，已足以训练有趣规模的大 CNN；而 ImageNet 这类新数据集又提供了足够多的标注样本，使这类模型可以在不过度过拟合的情况下训练。本文具体贡献如下：我们在 ILSVRC-2010 与 ILSVRC-2012 所用的 ImageNet 子集上 [2] 训练了当时最大的卷积网络之一，并在这些数据集上取得了迄今报告过的最好结果。我们编写了高度优化的二维卷积及训练 CNN 所需全部运算的 GPU 实现，并公开提供¹。网络包含若干新的、不寻常的设计，用以提升性能并缩短训练时间，详见第 3 节。即便有 120 万标注训练样本，网络规模仍使过拟合成为显著问题，因此我们采用了若干有效的防过拟合技术，见第 4 节。最终网络含五层卷积与三层全连接；深度似乎很关键：去掉任意一层卷积（每层参数不超过模型的 1%）都会变差。最终，网络规模主要受限于当前 GPU 显存，以及我们愿意承受的训练时间。在两块 GTX 580 3GB GPU 上训练约需五到六天。全部实验都表明：只要等待更快的 GPU 与更大的数据集，结果还能继续提升。

2 The Dataset

2 数据集

ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Ama- zon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images. ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model. ImageNet consists of variable-resolution images, while our system requires a constant input dimen- sionality. Therefore, we down-sampled the images to a fixed resolution of 256 × 256. Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central 256×256 patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels.

ImageNet 含逾 1500 万张带标注的高分辨率图像，约属 22,000 个类别。图像来自网络，由人工标注者借助亚马逊 Mechanical Turk 众包完成标注。自 2010 年起，作为 Pascal Visual Object Challenge 的一部分，每年举办 ImageNet 大规模视觉识别挑战赛（ILSVRC）。ILSVRC 使用 ImageNet 的一个子集：约 1000 个类别、每类约 1000 张图。合计约 120 万训练图、5 万验证图、15 万测试图。ILSVRC-2010 是唯一公开测试集标签的版本，因此我们大部分实验在其上完成。我们也参加了 ILSVRC-2012，第 6 节一并报告该版本结果（其测试标签未公开）。在 ImageNet 上习惯报告两种错误率：top-1 与 top-5；后者指正确标签不在模型给出的五个最可能标签之中的测试图像比例。ImageNet 图像分辨率不一，而我们的系统需要固定输入维。于是把图像下采样到固定的 256×256：对方形图先把短边缩到 256，再裁中心 256×256 块。除了从每个像素减去训练集上的均值活动外，不做其他预处理。网络直接在（居中后的）原始 RGB 像素值上训练。

3 The Architecture

3 网络结构

The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.

网络结构概览见图 2。共八个带权层——五层卷积、三层全连接。下文介绍若干新颖或不寻常的结构特点。第 3.1–3.4 节按我们估计的重要性排序，最重要的在前。

1 http://code.google.com/p/cuda-convnet/

2

3.1 ReLU Nonlinearity

3.1 ReLU 非线性

The standard way to model a neuron’s output f as a function of its input x is with f (x) = tanh(x) or f (x) = (1 + e−x )−1 . In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity f (x) = max(0, x). Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep convolutional neural net- works with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in

把神经元输出 $f$ 写成输入 $x$ 的函数，标准做法是 $f(x)=\tanh(x)$ 或 $f(x)=(1+e^{-x})^{-1}$。就用梯度下降训练所需时间而言，这些饱和非线性远慢于非饱和的 $f(x)=\max(0,x)$。沿用 Nair 与 Hinton [20]，我们把带这种非线性的神经元称为修正线性单元（ReLU）。使用 ReLU 的深度卷积网络比等价的 tanh 网络快数倍，这在

Figure 1, which shows the number of iterations re-

图 1 中可见：它给出某四层卷积网络在 CIFAR-10 上达到 25% 训练错误所需的迭代次数。

quired to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional net- work. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron Figure 1: A four-layer convolutional neural models. network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster We are not the first to consider alternatives to tradi- than an equivalent network with tanh neurons tional neuron models in CNNs. For example, Jarrett (dashed line). The learning rates for each net- et al. [11] claim that the nonlinearity f (x) = |tanh(x)| work were chosen independently to make train- works particularly well with their type of contrast nor- ing as fast as possible. No regularization of malization followed by local average pooling on the any kind was employed. The magnitude of the Caltech-101 dataset. However, on this dataset the pri- effect demonstrated here varies with network mary concern is preventing overfitting, so the effect architecture, but networks with ReLUs consis- they are observing is different from the accelerated tently learn several times faster than equivalents ability to fit the training set which we report when us- with saturating neurons. ing ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets.

该图表明，若仍用传统饱和神经元，我们根本无法在本工作中试验如此大的网络。图 1：带 ReLU（实线）的四层卷积网络在 CIFAR-10 上达到 25% 训练错误率的速度，约为等价 tanh 网络（虚线）的六倍。各网络的学习率独立选取，以使训练尽可能快。未使用任何正则化。效应大小随结构而变，但带 ReLU 的网络一贯比饱和神经元等价物快数倍。我们并非最早在 CNN 中考虑替代神经元模型的人。例如 Jarrett 等 [11] 声称非线性 $f(x)=|\tanh(x)|$ 配合其对比度归一化与局部平均池化，在 Caltech-101 上表现很好。但该数据集的首要关切是防过拟合，因此他们观察到的效应，与我们报告的「用 ReLU 加速拟合训练集」并不相同。更快的学习对在大型数据集上训练大型模型的性能有重大影响。

3.2 Training on Multiple GPUs

3.2 多 GPU 训练

A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation. The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Cireşan et al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net2 .

单块 GTX 580 只有 3GB 显存，限制了可训练网络的最大规模。事实证明，120 万训练样本足以训练出单卡装不下的网络，因此我们把网络摊到两块 GPU 上。当前 GPU 很适合跨卡并行：它们可以直接读写彼此显存，无需经过主机内存。我们采用的并行方案本质上是把一半卷积核（或神经元）放在每块 GPU 上，并加一条技巧：GPU 只在某些层通信。例如，第 3 层的核从第 2 层的全部核映射取输入；而第 4 层的核只从同卡上第 3 层的核映射取输入。连通模式需交叉验证选择，但这使我们能精确调节通信量，直到它只占计算量的可接受比例。最终结构有点像 Cireşan 等 [5] 的「柱状」CNN，只是我们的柱并不独立（见图 2）。与在单卡上训练、每层卷积核减半的网络相比，该方案分别把 top-1 与 top-5 错误率降低 1.7% 与 1.2%。双卡网训练时间略短于单卡网²。

2 The one-GPU net actually has the same number of kernels as the two-GPU net in the final convolutional layer. This is because most of the net’s parameters are in the first fully-connected layer, which takes the last convolutional layer as input. So to make the two nets have approximately the same number of parameters, we did not halve the size of the final convolutional layer (nor the fully-conneced layers which follow). Therefore this comparison is biased in favor of the one-GPU net, since it is bigger than “half the size” of the two-GPU net.

² 单卡网在最后一层卷积上其实与双卡网有相同数量的核。因为网络大部分参数在第一层全连接，它以最后一层卷积为输入。为使两网参数量大致相同，我们没有把最后一层卷积（以及其后的全连接）减半。因此该对比偏向单卡网，因为它比「双卡网一半大小」更大。

3

3.3 Local Response Normalization

3.3 局部响应归一化

ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by aix,y the activity of a neuron computed by applying kernel i at position (x, y) and then applying the ReLU nonlinearity, the response-normalized activity bix,y is given by the expression  β min(N −1,i+n/2) X bix,y = aix,y /k + α (ajx,y )2  j=max(0,i−n/2) where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants k, n, α, and β are hyper-parameters whose values are determined using a validation set; we used k = 2, n = 5, α = 10−4 , and β = 0.75. We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5). This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and 11% with normalization3 .

ReLU 有个好处：不必做输入归一化来防止饱和。只要部分训练样本给某个 ReLU 正输入，该神经元就会学习。不过我们仍发现下列局部归一化有助于泛化。记 $a^i_{x,y}$ 为在位置 $(x,y)$ 用第 $i$ 个核并经 ReLU 后的活动，响应归一化后的活动 $b^i_{x,y}$ 由下式给出：$b^i_{x,y} = a^i_{x,y} / \big(k + \alpha \sum_{j=\max(0,i-n/2)}^{\min(N-1,i+n/2)} (a^j_{x,y})^2\big)^\beta$，其中求和覆盖同一空间位置上 $n$ 个「相邻」核映射，$N$ 为该层核总数。核映射顺序任意，在训练前固定。这种响应归一化实现了一种受真实神经元侧抑制启发的竞争：用不同核算出的大响应会互相压制。常数 $k,n,\alpha,\beta$ 是超参，由验证集确定；我们取 $k=2$，$n=5$，$\alpha=10^{-4}$，$\beta=0.75$。在若干层的 ReLU 之后施加该归一化（见 3.5 节）。它与 Jarrett 等 [11] 的局部对比度归一化有相似之处，但我们更宜称作「亮度归一化」，因为没有减均值活动。响应归一化把 top-1 / top-5 错误率分别再降 1.4% / 1.2%。在 CIFAR-10 上也验证了有效性：四层 CNN 无归一化测试错误 13%，有归一化则 11%³。

3.4 Overlapping Pooling

3.4 重叠池化

Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size z × z centered at the location of the pooling unit. If we set s = z, we obtain traditional local pooling as commonly employed in CNNs. If we set s < z, we obtain overlapping pooling. This is what we use throughout our network, with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2, z = 2, which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.

CNN 中的池化层汇总同一核映射上相邻神经元组的输出。传统上，相邻池化单元汇总的邻域互不重叠（如 [17, 11, 4]）。更精确地说，池化层可看成间距为 $s$ 像素的池化单元网格，每个单元汇总以自身为中心、大小 $z\times z$ 的邻域。若 $s=z$，即得 CNN 中常见的传统局部池化；若 $s<z$，则得到重叠池化。我们全程使用后者，取 $s=2$、$z=3$。与产生同等输出尺寸的非重叠方案 $s=2,z=2$ 相比，该方案分别把 top-1 / top-5 错误率再降 0.4% / 0.3%。训练中我们一般观察到：带重叠池化的模型略难过拟合。

3.5 Overall Architecture

3.5 整体结构

Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fully- connected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution. The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fully- connected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer. The first convolutional layer filters the 224 × 224 × 3 input image with 96 kernels of size 11 × 11 × 3 with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring 3 We cannot describe this network in detail due to space constraints, but it is specified precisely by the code and parameter files provided here: http://code.google.com/p/cuda-convnet/.

现在可以描述 CNN 的整体结构。如图 2 所示，网络含八个带权层：前五层卷积，后三层全连接。最后一层全连接输出送入 1000 路 softmax，得到 1000 类标签上的分布。网络最大化多项逻辑回归目标，等价于最大化训练样本上「正确标签在预测分布下的对数概率」的平均。第 2、4、5 层卷积的核只连接到前一层中位于同一 GPU 上的核映射（见图 2）；第 3 层卷积的核连接到第 2 层全部核映射。全连接层的神经元与前一层全部神经元相连。第 1、2 层卷积后接响应归一化；第 3.4 节所述的最大池化接在两处响应归一化之后以及第 5 层卷积之后。ReLU 施加于每个卷积与全连接层的输出。第 1 层卷积用 96 个大小为 $11\times 11\times 3$、步幅 4 的核对 $224\times 224\times 3$ 输入滤波（步幅即相邻感受野中心的间距）。³ 因篇幅限制无法详述该网络，但其代码与参数文件精确给出：http://code.google.com/p/cuda-convnet/。

4

Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities

图 2：CNN 结构示意，明确标出两块 GPU 各自负责的部分。

between the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264– 4096–4096–1000.

一块 GPU 跑图上方的层片段，另一块跑下方。GPU 只在某些层通信。网络输入为 150,528 维，其余各层神经元数为 253,440–186,624–64,896–64,896–43,264–4096–4096–1000。

neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5 × 5 × 48. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3 × 3 × 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3 × 3 × 192 , and the fifth convolutional layer has 256 kernels of size 3 × 3 × 192. The fully-connected layers have 4096 neurons each.

第 2 层卷积以第 1 层（经响应归一化与池化）的输出为输入，用 256 个大小 $5\times 5\times 48$ 的核滤波。第 3、4、5 层卷积彼此直接相连，中间无池化或归一化。第 3 层有 384 个 $3\times 3\times 256$ 的核，接到第 2 层（归一化并池化）的输出；第 4 层有 384 个 $3\times 3\times 192$ 的核；第 5 层有 256 个 $3\times 3\times 192$ 的核。全连接层各有 4096 个神经元。

4 Reducing Overfitting

4 减轻过拟合

Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting.

我们的网络结构有 6000 万参数。尽管 ILSVRC 的 1000 类使每个训练样本对「图像→标签」映射施加约 10 bit 约束，这仍不足以在无明显过拟合的情况下学到这么多参数。下面介绍我们应对过拟合的两种主要手段。

4.1 Data Augmentation

4.1 数据扩增

The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free. The first form of data augmentation consists of generating image translations and horizontal reflec- tions. We do this by extracting random 224 × 224 patches (and their horizontal reflections) from the 256×256 images and training our network on these extracted patches4 . This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly inter- dependent. Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches. The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components, 4 This is the reason why the input images in Figure 2 are 224 × 224 × 3-dimensional.

减轻图像数据过拟合最简单也最常用的办法，是用保持标签的变换人为扩大数据集（如 [25, 4, 5]）。我们采用两种扩增，都能从原图以极少计算生成变换图，因此不必落盘。实现上，CPU 用 Python 生成变换图的同时，GPU 在训上一批，因而这两种方案实际上几乎不增加算力。第一种是平移与水平翻转：从 256×256 图像中随机裁 224×224 块（及其水平翻转）并在其上训练⁴。训练集名义上扩大约 2048 倍，但所得样本高度相关。没有这一方案，网络会严重过拟合，只能被迫用小得多的网络。测试时，网络抽取五个 224×224 块（四角与中心）及其水平翻转（共十块），对 softmax 在十块上的预测取平均。第二种是改变训练图像 RGB 通道强度：在整个 ImageNet 训练集的 RGB 像素上做 PCA，再向每张训练图加入主成分的倍数。⁴ 这也是图 2 中输入图像为 224×224×3 维的原因。

5

with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel Ixy = R G B T [Ixy , Ixy , Ixy ] we add the following quantity: [p1 , p2 , p3 ][α1 λ1 , α2 λ2 , α3 λ3 ]T where pi and λi are ith eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values, respectively, and αi is the aforementioned random variable. Each αi is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn. This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination. This scheme reduces the top-1 error rate by over 1%.

倍数大小正比于对应特征值乘以一个均值为 0、标准差为 0.1 的高斯随机变量。因此对每个 RGB 像素 $I_{xy}=[I^R_{xy}, I^G_{xy}, I^B_{xy}]^T$，加入 $[p_1,p_2,p_3][\alpha_1\lambda_1,\alpha_2\lambda_2,\alpha_3\lambda_3]^T$，其中 $p_i,\lambda_i$ 是 RGB 像素 3×3 协方差矩阵的第 $i$ 个特征向量与特征值，$\alpha_i$ 即上述随机变量。每张训练图的全部像素共用一次抽到的 $\alpha_i$，直到该图再次被用于训练才重抽。该方案近似抓住了自然图像的一个重要性质：物体身份对光照强度与颜色变化保持不变。它使 top-1 错误率再降逾 1%。

4.2 Dropout

Combining the predictions of many different models is a very successful way to reduce test errors [1, 3], but it appears to be too expensive for big neural networks that already take several days to train. There is, however, a very efficient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called “dropout” [10], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in back- propagation. So every time an input is presented, the neural network samples a different architecture, but all these architectures share weights. This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks. We use dropout in the first two fully-connected layers of Figure 2. Without dropout, our network ex- hibits substantial overfitting. Dropout roughly doubles the number of iterations required to converge.

融合许多不同模型的预测是降低测试错误的成功手段 [1, 3]，但对已经要训好几天的大网络来说似乎太贵。不过有一种很高效的模型组合，训练代价大约只翻一倍。新近提出的「dropout」[10] 以 0.5 概率把每个隐神经元输出置零。被「丢掉」的神经元不参与前向，也不参与反向传播。于是每次输入到来，网络都采样一个不同结构，但这些结构共享权重。该技术减少神经元之间复杂的共适应：一个神经元不能依赖另一些特定神经元一定存在，因而被迫学习在许多随机子集上仍有用的更稳健特征。测试时使用全部神经元，但把输出乘以 0.5，这是对指数多个 dropout 网络预测分布几何平均的合理近似。我们在图 2 的前两层全连接上使用 dropout。没有 dropout，网络会严重过拟合。Dropout 大约使收敛所需迭代数翻倍。

5 Details of learning

5 学习细节

We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight decay was important for the model to learn. In Figure 3: 96 convolutional kernels of size other words, weight decay here is not merely a regularizer: 11×11×3 learned by the first convolutional it reduces the model’s training error. The update rule for layer on the 224×224×3 input images. The weight w was top 48 kernels were learned on GPU 1 while

我们用批量大小 128、动量 0.9、权重衰减 0.0005 的随机梯度下降训练。发现这一小量权重衰减对模型学习很重要——换句话说，这里的权重衰减不只是正则项：它还降低了训练错误。权重 $w$ 的更新规则为图 3：第一层卷积在 224×224×3 输入上学到的 96 个大小 11×11×3 的卷积核。上方 48 个核在 GPU 1 上学习，

∂L

$\partial L$

the bottom 48 kernels were learned on GPU vi+1 := 0.9 · vi − 0.0005 ·  · wi −  · 2. See Section 6.1 for details. ∂w wi Di wi+1 := wi + vi+1 D E ∂L where i is the iteration index, v is the momentum variable,  is the learning rate, and ∂w wi is Di the average over the ith batch Di of the derivative of the objective with respect to w, evaluated at wi . We initialized the weights in each layer from a zero-mean Gaussian distribution with standard de- viation 0.01. We initialized the neuron biases in the second, fourth, and fifth convolutional layers, as well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates the early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron biases in the remaining layers with the constant 0. We used an equal learning rate for all layers, which we adjusted manually throughout training. The heuristic which we followed was to divide the learning rate by 10 when the validation error rate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and

下方 48 个核在 GPU 2 上学习。细节见 6.1 节。$v_{i+1} := 0.9\cdot v_i - 0.0005\cdot \epsilon \cdot w_i - \epsilon \cdot \langle \partial L/\partial w \rangle_{w_i}\rangle_{D_i}$，$w_{i+1} := w_i + v_{i+1}$，其中 $i$ 为迭代下标，$v$ 为动量变量，$\epsilon$ 为学习率，$\langle \partial L/\partial w \rangle_{w_i}\rangle_{D_i}$ 是目标对 $w$ 的导数在第 $i$ 批 $D_i$ 上、于 $w_i$ 处的平均。每层权重由均值为 0、标准差 0.01 的高斯初始化。第 2、4、5 层卷积以及全连接隐层的偏置初始化为常数 1，以便给 ReLU 正输入、加速早期学习；其余层偏置初始化为 0。各层使用相同学习率，训练中手动调节：验证错误率在当前学习率下不再改善时，就把学习率除以 10。学习率初值为 0.01，并

6

reduced three times prior to termination. We trained the network for roughly 90 cycles through the training set of 1.2 million images, which took five to six days on two NVIDIA GTX 580 3GB GPUs.

在终止前降低三次。我们在 120 万张训练图像上大约跑了 90 个周期，在两块 NVIDIA GTX 580 3GB GPU 上耗时五到六天。

6 Results

6 结果

Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of 37.5% and 17.0%5 . The best performance achieved during the ILSVRC- 2010 competition was 47.1% and 28.2% with an approach that averages the predictions produced from six sparse-coding models trained on different features [2], and since then the best pub- lished results are 45.7% and 25.7% with an approach that averages the predictions of two classi- fiers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24].

ILSVRC-2010 结果总结于表 1。我们的网络在测试集上达到 top-1 / top-5 错误率 37.5% / 17.0%⁵。ILSVRC-2010 竞赛期间最佳成绩是 47.1% / 28.2%，来自对六种稀疏编码模型在不同特征上预测的平均 [2]；此后已发表的最好结果是 45.7% / 25.7%，来自对两种稠密采样特征上 Fisher 向量（FV）训练的两个分类器取平均 [24]。

We also entered our model in the ILSVRC-2012 com- Model Top-1 Top-5 petition and report our results in Table 2. Since the Sparse coding [2] 47.1% 28.2% ILSVRC-2012 test set labels are not publicly available, SIFT + FVs [24] 45.7% 25.7% we cannot report test error rates for all the models that CNN 37.5% 17.0% we tried. In the remainder of this paragraph, we use validation and test error rates interchangeably because Table 1: Comparison of results on ILSVRC- in our experience they do not differ by more than 0.1% 2010 test set. In italics are best results (see Table 2). The CNN described in this paper achieves achieved by others. a top-5 error rate of 18.2%. Averaging the predictions of five similar CNNs gives an error rate of 16.4%. Training one CNN, with an extra sixth con- volutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release (15M images, 22K categories), and then “fine-tuning” it on ILSVRC-2012 gives an error rate of 16.6%. Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 re- lease with the aforementioned five CNNs gives an error rate of 15.3%. The second-best con- test entry achieved an error rate of 26.2% with an approach that averages the predictions of sev- eral classifiers trained on FVs computed from different types of densely-sampled features [7]. Finally, we also report our error rates on the Fall 2009 version of Model Top-1 (val) Top-5 (val) Top-5 (test) ImageNet with 10,184 categories SIFT + FVs [7] — — 26.2% and 8.9 million images. On this

我们也参加了 ILSVRC-2012，结果见表 2。由于 ILSVRC-2012 测试标签未公开，无法报告我们试过的全部模型的测试错误率。本段余下把验证与测试错误率互换使用，因为经验上二者相差不超过 0.1%（见表 2）。本文所述 CNN 的 top-5 错误率为 18.2%。平均五个相似 CNN 的预测得到 16.4%。再训练一个在最后池化层之上多加第六层卷积的 CNN，先在整个 ImageNet Fall 2011（1500 万图、2.2 万类）上分类，再在 ILSVRC-2012 上微调，错误率为 16.6%。把两个在 Fall 2011 全集上预训练的 CNN 与上述五个 CNN 的预测平均，得到 15.3%。竞赛第二名以 26.2% 的错误率取得，方法是平均若干在不同稠密采样特征 FV 上训练的分类器 [7]。最后，我们也报告 Fall 2009 版 ImageNet（10,184 类、890 万图）上的错误率。表 1：ILSVRC-2010 测试集结果对比。斜体为他人最佳。稀疏编码 [2] 47.1%/28.2%；SIFT+FVs [24] 45.7%/25.7%；CNN 37.5%/17.0%。

1 CNN 40.7% 18.2% —

dataset we follow the convention in the literature of using half of 5 CNNs 38.1% 16.4% 16.4% the images for training and half 1 CNN* 39.0% 16.6% — for testing. Since there is no es- 7 CNNs* 36.7% 15.4% 15.3% tablished test set, our split neces- sarily differs from the splits used Table 2: Comparison of error rates on ILSVRC-2012 validation and by previous authors, but this does test sets. In italics are best results achieved by others. Models with an not affect the results appreciably. asterisk* were “pre-trained” to classify the entire ImageNet 2011 Fall Our top-1 and top-5 error rates release. See Section 6 for details. on this dataset are 67.4% and 40.9%, attained by the net described above but with an additional, sixth convolutional layer over the last pooling layer. The best published results on this dataset are 78.1% and 60.9% [19].

在该数据集上我们遵循文献惯例，一半图像训练、一半测试。因无既定测试集，我们的划分必然不同于以往作者，但对结果影响不大。上述网络再加第六层卷积后，top-1 / top-5 错误率为 67.4% / 40.9%。已发表最好结果为 78.1% / 60.9% [19]。表 2：ILSVRC-2012 验证与测试错误率对比。斜体为他人最佳。带星号的模型曾在整个 ImageNet 2011 Fall 上「预训练」。详见第 6 节。5 CNNs 38.1% / 16.4% / 16.4%；1 CNN 39.0% / 16.6% / —；7 CNNs* 36.7% / 15.4% / 15.3%。

6.1 Qualitative Evaluations

6.1 定性评估

Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers. The

图 3 展示了网络两个与数据直接相连的层所学到的卷积核。

network has learned a variety of frequency- and orientation-selective kernels, as well as various col- ored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connec- tivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels on on GPU 2 are largely color-specific. This kind of specialization occurs during every run and is independent of any particular random weight initialization (modulo a renumbering of the GPUs).

网络学到了多种频率与方向选择性核，以及各种彩色色块。注意两块 GPU 表现出的专化，这是第 3.5 节所述受限连通的结果：GPU 1 上的核大多对颜色不敏感，GPU 2 上的核大多对颜色敏感。这种专化每次运行都会出现，与具体的随机权重初始化无关（最多只是 GPU 编号对调）。

5 The error rates without averaging predictions over ten patches as described in Section 4.1 are 39.0% and 18.3%.

⁵ 若不按第 4.1 节所述对十个图像块预测取平均，错误率为 39.0% 与 18.3%。

7

Figure 4: (Left) Eight ILSVRC-2010 test images and the five labels considered most probable by our model.

图 4：（左）八张 ILSVRC-2010 测试图像及模型认为最可能的五个标签。

The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar (if it happens to be in the top 5). (Right) Five ILSVRC-2010 test images in the first column. The remaining columns show the six training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image.

正确标签写在每张图下方；若正确标签落在 top-5 内，其概率以红条标出。（右）第一列为五张 ILSVRC-2010 测试图；其余列给出在最后隐层特征上与该测试图欧氏距离最小的六张训练图。

In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example, only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph. Another way to probe the network’s visual knowledge is to consider the feature activations induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar. Figure 4 shows five images from the test set and the six images from the training set that are most similar to each of them according to this measure. Notice that at the pixel level, the retrieved training images are generally not close in L2 to the query images in the first column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the results for many more test images in the supplementary material. Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vec- tors is inefficient, but it could be made efficient by training an auto-encoder to compress these vectors to short binary codes. This should produce a much better image retrieval method than applying auto- encoders to the raw pixels [14], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar.

在图 4 左栏，我们通过八张测试图的 top-5 预测定性评估网络所学。注意即便偏心物体（如左上角的螨）也能被认出。多数 top-5 标签看起来合理：例如对豹子，模型只把其他猫科当作可信标签。有些情形（汽车进气格栅、樱桃）照片本意焦点本身就有歧义。另一种探查网络视觉知识的方式，是看图像在最后 4096 维隐层诱发的特征激活。若两张图的特征向量欧氏距离很小，可以说网络高层认为它们相似。图 4 给出五张测试图及其按该度量最相似的六张训练图。注意在像素级，$L_2$ 意义下检索到的训练图通常并不靠近查询图：例如检索到的狗与大象姿态各异。补充材料中还有更多测试图结果。用两个 4096 维实向量的欧氏距离算相似度并不高效，但可训练自编码器把这些向量压成短二进制码以提高效率。这应比直接对原始像素做自编码器 [14] 好得多——后者不用图像标签，往往检索到边缘模式相似、语义却不一定相近的图。

7 Discussion

7 讨论

Our results show that a large, deep convolutional neural network is capable of achieving record- breaking results on a highly challenging dataset using purely supervised learning. It is notable that our network’s performance degrades if a single convolutional layer is removed. For example, removing any of the middle layers results in a loss of about 2% for the top-1 performance of the network. So the depth really is important for achieving our results. To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data. Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system. Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images.

结果表明，大型深度卷积网络仅用纯监督学习就能在极具挑战的数据集上打破纪录。值得注意的是，去掉任意一层卷积，网络性能都会下降：例如去掉任一层中间卷积，top-1 大约掉 2%。因此深度对取得我们的结果确实重要。为简化实验，我们未使用无监督预训练，尽管我们预期它会有帮助——尤其当我们有足够算力显著增大网络、却得不到相应增长的标注数据时。迄今为止，网络越大、训得越久，结果越好；但要匹配人类视觉系统的下颞通路，我们还有许多数量级的差距。最终我们希望把非常大且深的卷积网络用到视频序列上：时间结构能提供静态图像中缺失或远不那么明显的有用信息。

8

References

参考文献

2004 IEEE Computer Society Conference on, volume 2, pages II–97. IEEE, 2004.

9