Very Deep Convolutional Networks for Large-Scale Image Recognition | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表
摘要1 引言2 ConvNet 配置2.1 结构2.2 配置2.3 讨论3 分类框架3.1 训练训练图像尺寸3.2 测试3.3 实现细节4 分类实验数据集4.1 单尺度评估4.2 多尺度评估4.3 多裁剪评估4.4 ConvNet 融合4.5 与当时最优的比较5 结论致谢参考文献附录 A 定位A.1 定位 ConvNet训练测试A.2 定位实验设定比较完整评估与当时最优的比较附录 B 极深特征的泛化VOC-2007 与 VOC-2012 上的图像分类Caltech-101 与 Caltech-256 上的图像分类VOC-2012 上的动作分类其他识别任务附录 C 论文修订
返回论文列表
目录

Very Deep Convolutional Networks for Large-Scale Image Recognition

收录于 ICLR 2015；arXiv:1409.1556
打开论文来源

Very Deep Convolutional Networks for Large-Scale Image Recognition

用于大规模图像识别的极深卷积网络

Karen Simonyan   & Andrew ZissermanVisual Geometry Group, Department of Engineering Science, University of Oxford {karen,az}@robots.ox.ac.uk

Abstract

摘要

In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3×33\times 33×3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16–19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

本文研究卷积网络深度对大规模图像识别精度的影响。主要贡献是：在统一采用极小（3×33\times 33×3）卷积滤波器的结构下，系统评估不断加深的网络，表明把深度推到 16–19 个权重层，相对此前配置可取得显著提升。这些发现构成我们 ImageNet Challenge 2014 参赛的基础——本队在定位赛道获第一、分类赛道获第二。我们还表明所学表征能良好泛化到其他数据集并达到当时最优。两套表现最好的 ConvNet 模型已公开发布，以便进一步研究深度视觉表征在计算机视觉中的用法。

1 Introduction

1 引言

Convolutional networks (ConvNets) have recently enjoyed a great success in large-scale image and video recognition (Krizhevsky et al. 2012; Zeiler & Fergus 2013; Sermanet et al. 2014; Simonyan & Zisserman 2014) which has become possible due to the large public image repositories, such as ImageNet (Deng et al. 2009), and high-performance computing systems, such as GPUs or large-scale distributed clusters (Dean et al. 2012). In particular, an important role in the advance of deep visual recognition architectures has been played by the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) (Russakovsky et al. 2014), which has served as a testbed for a few generations of large-scale image classification systems, from high-dimensional shallow feature encodings (Perronnin et al. 2010) (the winner of ILSVRC-2011) to deep ConvNets (Krizhevsky et al. 2012) (the winner of ILSVRC-2012).

卷积网络（ConvNet）近来在大规模图像与视频识别上取得很大成功（Krizhevsky et al. 2012；Zeiler & Fergus 2013；Sermanet et al. 2014；Simonyan & Zisserman 2014），这得益于 ImageNet（Deng et al. 2009）一类公开大规模图像库，以及 GPU 或大规模分布式集群（Dean et al. 2012）等高性能计算。尤其是 ImageNet 大规模视觉识别挑战（ILSVRC）（Russakovsky et al. 2014）扮演了试验场：从高维浅层特征编码（Perronnin et al. 2010，ILSVRC-2011 冠军）到深度 ConvNet（Krizhevsky et al. 2012，ILSVRC-2012 冠军），几代大规模图像分类系统都在此较量。

With ConvNets becoming more of a commodity in the computer vision field, a number of attempts have been made to improve the original architecture of Krizhevsky et al. 2012 in a bid to achieve better accuracy. For instance, the best-performing submissions to the ILSVRC-2013 (Zeiler & Fergus 2013; Sermanet et al. 2014) utilised smaller receptive window size and smaller stride of the first convolutional layer. Another line of improvements dealt with training and testing the networks densely over the whole image and over multiple scales (Sermanet et al. 2014; Howard 2014). In this paper, we address another important aspect of ConvNet architecture design – its depth. To this end, we fix other parameters of the architecture, and steadily increase the depth of the network by adding more convolutional layers, which is feasible due to the use of very small (3×33\times 33×3) convolution filters in all layers.

随着 ConvNet 在计算机视觉中日渐普及，不少工作试图改进 Krizhevsky et al. 2012 的原始结构以换更高精度。例如 ILSVRC-2013 表现最好的提交（Zeiler & Fergus 2013；Sermanet et al. 2014）采用了更小的第一层感受野与更小步长。另一条改进线是在整图与多尺度上稠密训练与测试（Sermanet et al. 2014；Howard 2014）。本文关注结构的另一关键方面——深度。为此我们固定结构其余参数，通过不断加入卷积层稳步加深网络；由于各层都用极小（3×33\times 33×3）卷积滤波器，这样做是可行的。

As a result, we come up with significantly more accurate ConvNet architectures, which not only achieve the state-of-the-art accuracy on ILSVRC classification and localisation tasks, but are also applicable to other image recognition datasets, where they achieve excellent performance even when used as a part of a relatively simple pipelines (e.g. deep features classified by a linear SVM without fine-tuning). We have released our two best-performing models[^1] to facilitate further research.

结果是显著更准的 ConvNet 结构：不仅在 ILSVRC 分类与定位上达到当时最优，也能用于其他图像识别数据集——即便嵌在相对简单的流水线里（例如深度特征加线性 SVM、不做微调）表现也很好。我们已发布两套表现最好的模型[^1]，方便后续研究。

The rest of the paper is organised as follows. In Sect. 2, we describe our ConvNet configurations. The details of the image classification training and evaluation are then presented in Sect. 3, and the configurations are compared on the ILSVRC classification task in Sect. 4. Sect. 5 concludes the paper. For completeness, we also describe and assess our ILSVRC-2014 object localisation system in Appendix A, and discuss the generalisation of very deep features to other datasets in Appendix B. Finally, Appendix C contains the list of major paper revisions.

全文组织如下。第 2 节描述 ConvNet 配置；第 3 节给出图像分类训练与评估细节；第 4 节在 ILSVRC 分类任务上比较各配置；第 5 节作结。为完整起见，附录 A 描述并评估我们的 ILSVRC-2014 物体定位系统，附录 B 讨论极深特征在其他数据集上的泛化，附录 C 列出主要修订。

2 ConvNet Configurations

2 ConvNet 配置

To measure the improvement brought by the increased ConvNet depth in a fair setting, all our ConvNet layer configurations are designed using the same principles, inspired by Ciresan et al. 2011; Krizhevsky et al. 2012. In this section, we first describe a generic layout of our ConvNet configurations (Sect. 2.1) and then detail the specific configurations used in the evaluation (Sect. 2.2). Our design choices are then discussed and compared to the prior art in Sect. 2.3.

为在公平设定下衡量加深带来的提升，所有层配置遵循同一设计原则，灵感来自 Ciresan et al. 2011；Krizhevsky et al. 2012。本节先给出通用布局（第 2.1 节），再详述评估所用具体配置（第 2.2 节），并在第 2.3 节讨论设计选择、与先前工作对比。

2.1 Architecture

2.1 结构

During training, the input to our ConvNets is a fixed-size 224×224224\times 224224×224 RGB image. The only pre-processing we do is subtracting the mean RGB value, computed on the training set, from each pixel. The image is passed through a stack of convolutional (conv.) layers, where we use filters with a very small receptive field: 3×33\times 33×3 (which is the smallest size to capture the notion of left/right, up/down, center). In one of the configurations we also utilise 1×11\times 11×1 convolution filters, which can be seen as a linear transformation of the input channels (followed by non-linearity). The convolution stride is fixed to 111 pixel; the spatial padding of conv. layer input is such that the spatial resolution is preserved after convolution, i.e. the padding is 111 pixel for 3×33\times 33×3 conv. layers. Spatial pooling is carried out by five max-pooling layers, which follow some of the conv. layers (not all the conv. layers are followed by max-pooling). Max-pooling is performed over a 2×22\times 22×2 pixel window, with stride 222.

训练时，输入为固定尺寸 224×224224\times 224224×224 RGB 图像。唯一预处理是减去训练集上算得的均值 RGB。图像经过一叠卷积层，滤波器感受野极小：3×33\times 33×3（能捕捉左/右、上/下、中心概念的最小尺寸）。某一配置还使用 1×11\times 11×1 卷积，可视作对输入通道的线性变换（后再接非线性）。卷积步长固定为 111 像素；空间填充使卷积后分辨率不变，即 3×33\times 33×3 卷积填充 111 像素。空间池化由五个最大池化层完成，跟在部分（非全部）卷积层之后；池化窗口 2×22\times 22×2、步长 222。

A stack of convolutional layers (which has a different depth in different architectures) is followed by three Fully-Connected (FC) layers: the first two have 4096 channels each, the third performs 1000-way ILSVRC classification and thus contains 1000 channels (one for each class). The final layer is the soft-max layer. The configuration of the fully connected layers is the same in all networks.

一叠卷积层（不同结构深度不同）之后是三层全连接（FC）：前两层各 4096 通道，第三层做 1000 类 ILSVRC 分类因而含 1000 通道。最后是 soft-max。各网络的全连接配置相同。

All hidden layers are equipped with the rectification (ReLU (Krizhevsky et al. 2012)) non-linearity. We note that none of our networks (except for one) contain Local Response Normalisation (LRN) normalisation (Krizhevsky et al. 2012): as will be shown in Sect. 4, such normalisation does not improve the performance on the ILSVRC dataset, but leads to increased memory consumption and computation time. Where applicable, the parameters for the LRN layer are those of (Krizhevsky et al. 2012).

所有隐层配备 ReLU（Krizhevsky et al. 2012）非线性。除一个网络外，我们都不使用局部响应归一化（LRN）（Krizhevsky et al. 2012）：第 4 节将表明，该归一化不提升 ILSVRC 表现，却增加显存与耗时。若使用 LRN，参数取自 Krizhevsky et al. 2012。

2.2 Configurations

2.2 配置

The ConvNet configurations, evaluated in this paper, are outlined in Table 1, one per column. In the following we will refer to the nets by their names (A–E). All configurations follow the generic design presented in Sect. 2.1, and differ only in the depth: from 11 weight layers in the network A (8 conv. and 3 FC layers) to 19 weight layers in the network E (16 conv. and 3 FC layers). The width of conv. layers (the number of channels) is rather small, starting from 646464 in the first layer and then increasing by a factor of 222 after each max-pooling layer, until it reaches 512512512.

本文评估的 ConvNet 配置见表 1，每列一个。下文以名称（A–E）指称。全部遵循第 2.1 节通用设计，仅深度不同：从网络 A 的 11 个权重层（8 卷积 + 3 FC）到网络 E 的 19 个权重层（16 卷积 + 3 FC）。卷积层宽度（通道数）偏小，第一层 646464，每经一次最大池化乘 222，直至 512512512。

In Table 2 we report the number of parameters for each configuration. In spite of a large depth, the number of weights in our nets is not greater than the number of weights in a more shallow net with larger conv. layer widths and receptive fields (144M weights in (Sermanet et al. 2014)).

表 2 给出各配置参数量。尽管很深，权重量并不大于感受野更大、通道更宽的较浅网络（Sermanet et al. 2014 为 1.44 亿权重）。

ConvNet Configuration
A A-LRN B C D E
11 weight 11 weight 13 weight 16 weight 16 weight 19 weight
layers layers layers layers layers layers
input (224×224224\times 224224×224 RGB image)
conv3-64 conv3-64 conv3-64 conv3-64 conv3-64 conv3-64
LRN conv3-64 conv3-64 conv3-64 conv3-64
maxpool
conv3-128 conv3-128 conv3-128 conv3-128 conv3-128 conv3-128
conv3-128 conv3-128 conv3-128 conv3-128
maxpool
conv3-256 conv3-256 conv3-256 conv3-256 conv3-256 conv3-256
conv3-256 conv3-256 conv3-256 conv3-256 conv3-256 conv3-256
conv1-256 conv3-256 conv3-256
conv3-256
maxpool
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv1-512 conv3-512 conv3-512
conv3-512
maxpool
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv1-512 conv3-512 conv3-512
conv3-512
maxpool
FC-4096
FC-4096
FC-1000
soft-max

Table 1: ConvNet configurations (shown in columns). The depth of the configurations increases from the left (A) to the right (E), as more layers are added (the added layers are shown in bold). The convolutional layer parameters are denoted as “conv⟨\langle⟨receptive field size⟩\rangle⟩-⟨\langle⟨number of channels⟩\rangle⟩”. The ReLU activation function is not shown for brevity.

ConvNet Configuration
A A-LRN B C D E
11 weight 11 weight 13 weight 16 weight 16 weight 19 weight
layers layers layers layers layers layers
input (224×224224\times 224224×224 RGB image)
conv3-64 conv3-64 conv3-64 conv3-64 conv3-64 conv3-64
LRN conv3-64 conv3-64 conv3-64 conv3-64
maxpool
conv3-128 conv3-128 conv3-128 conv3-128 conv3-128 conv3-128
conv3-128 conv3-128 conv3-128 conv3-128
maxpool
conv3-256 conv3-256 conv3-256 conv3-256 conv3-256 conv3-256
conv3-256 conv3-256 conv3-256 conv3-256 conv3-256 conv3-256
conv1-256 conv3-256 conv3-256
conv3-256
maxpool
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv1-512 conv3-512 conv3-512
conv3-512
maxpool
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv1-512 conv3-512 conv3-512
conv3-512
maxpool
FC-4096
FC-4096
FC-1000
soft-max

表 1：ConvNet 配置（按列展示）。深度自左（A）至右（E）递增，新增层以粗体标出。卷积层参数记为 “conv⟨\langle⟨感受野尺寸⟩\rangle⟩-⟨\langle⟨通道数⟩\rangle⟩”。为简洁省略 ReLU。

Network A,A-LRN B C D E
Number of parameters 133 133 134 138 144

Table 2: Number of parameters (in millions).

Network A,A-LRN B C D E
Number of parameters 133 133 134 138 144

表 2：参数量（百万）。

2.3 Discussion

2.3 讨论

Our ConvNet configurations are quite different from the ones used in the top-performing entries of the ILSVRC-2012 (Krizhevsky et al. 2012) and ILSVRC-2013 competitions (Zeiler & Fergus 2013; Sermanet et al. 2014). Rather than using relatively large receptive fields in the first conv. layers (e.g. 11×1111\times 1111×11 with stride 444 in (Krizhevsky et al. 2012), or 7×77\times 77×7 with stride 222 in (Zeiler & Fergus 2013; Sermanet et al. 2014)), we use very small 3×33\times 33×3 receptive fields throughout the whole net, which are convolved with the input at every pixel (with stride 111). It is easy to see that a stack of two 3×33\times 33×3 conv. layers (without spatial pooling in between) has an effective receptive field of 5×55\times 55×5; three such layers have a 7×77\times 77×7 effective receptive field. So what have we gained by using, for instance, a stack of three 3×33\times 33×3 conv. layers instead of a single 7×77\times 77×7 layer? First, we incorporate three non-linear rectification layers instead of a single one, which makes the decision function more discriminative. Second, we decrease the number of parameters: assuming that both the input and the output of a three-layer 3×33\times 33×3 convolution stack has CCC channels, the stack is parametrised by 3(32C2)=27C23\left(3^{2}C^{2}\right)=27C^{2}3(32C2)=27C2 weights; at the same time, a single 7×77\times 77×7 conv. layer would require 72C2=49C27^{2}C^{2}=49C^{2}72C2=49C2 parameters, i.e. 81%81\%81% more. This can be seen as imposing a regularisation on the 7×77\times 77×7 conv. filters, forcing them to have a decomposition through the 3×33\times 33×3 filters (with non-linearity injected in between).

我们的 ConvNet 配置与 ILSVRC-2012（Krizhevsky et al. 2012）及 ILSVRC-2013（Zeiler & Fergus 2013；Sermanet et al. 2014）顶尖参赛结构颇为不同。不是在前几层用较大感受野（如 Krizhevsky et al. 2012 的 11×1111\times 1111×11、步长 444，或 Zeiler & Fergus 2013；Sermanet et al. 2014 的 7×77\times 77×7、步长 222），我们全程使用极小的 3×33\times 33×3 感受野，并在每个像素处以步长 111 卷积。易见：两层 3×33\times 33×3 卷积（中间无空间池化）有效感受野为 5×55\times 55×5；三层则为 7×77\times 77×7。那么，用三层 3×33\times 33×3 代替单层 7×77\times 77×7 得到了什么？第一，引入三次非线性整流而非一次，决策函数更有判别力；第二，参数更少：若三层 3×33\times 33×3 卷积栈的输入输出都是 CCC 通道，参数为 3(32C2)=27C23(3^{2}C^{2})=27C^{2}3(32C2)=27C2；单层 7×77\times 77×7 则需 72C2=49C27^{2}C^{2}=49C^{2}72C2=49C2，多出 81%81\%81%。这可看作对 7×77\times 77×7 滤波器施加正则，迫使它们经带非线性的 3×33\times 33×3 分解。

The incorporation of 1×11\times 11×1 conv. layers (configuration C, Table 1) is a way to increase the non-linearity of the decision function without affecting the receptive fields of the conv. layers. Even though in our case the 1×11\times 11×1 convolution is essentially a linear projection onto the space of the same dimensionality (the number of input and output channels is the same), an additional non-linearity is introduced by the rectification function. It should be noted that 1×11\times 11×1 conv. layers have recently been utilised in the “Network in Network” architecture of Lin et al. 2014.

引入 1×11\times 11×1 卷积（配置 C，表 1）可在不影响卷积层感受野的前提下增加决策函数非线性。即便此处 1×11\times 11×1 本质上是同维线性投影（输入输出通道数相同），整流函数仍额外引入非线性。需指出，1×11\times 11×1 卷积近期也出现在 Lin et al. 2014 的 “Network in Network” 结构中。

Small-size convolution filters have been previously used by Ciresan et al. 2011, but their nets are significantly less deep than ours, and they did not evaluate on the large-scale ILSVRC dataset. Goodfellow et al. 2014 applied deep ConvNets (111111 weight layers) to the task of street number recognition, and showed that the increased depth led to better performance. GoogLeNet (Szegedy et al. 2014), a top-performing entry of the ILSVRC-2014 classification task, was developed independently of our work, but is similar in that it is based on very deep ConvNets (22 weight layers) and small convolution filters (apart from 3×33\times 33×3, they also use 1×11\times 11×1 and 5×55\times 55×5 convolutions). Their network topology is, however, more complex than ours, and the spatial resolution of the feature maps is reduced more aggressively in the first layers to decrease the amount of computation. As will be shown in Sect. 4.5, our model is outperforming that of Szegedy et al. 2014 in terms of the single-network classification accuracy.

Ciresan et al. 2011 也曾用小尺寸卷积滤波器，但其网络远不如我们深，且未在大规模 ILSVRC 上评估。Goodfellow et al. 2014 把深度 ConvNet（11 个权重层）用于门牌号识别，并表明加深带来更好表现。GoogLeNet（Szegedy et al. 2014）是 ILSVRC-2014 分类顶尖参赛之一，与我们独立发展，但同样基于极深 ConvNet（22 个权重层）与小卷积滤波器（除 3×33\times 33×3 外还有 1×11\times 11×1 与 5×55\times 55×5）。其拓扑更复杂，且在前几层更激进地降低特征图空间分辨率以减计算。第 4.5 节将表明，就单网络分类精度而言，我们的模型优于 Szegedy et al. 2014。

3 Classification Framework

3 分类框架

In the previous section we presented the details of our network configurations. In this section, we describe the details of classification ConvNet training and evaluation.

上一节给出了网络配置细节。本节描述分类 ConvNet 的训练与评估细节。

3.1 Training

3.1 训练

The ConvNet training procedure generally follows Krizhevsky et al. 2012 (except for sampling the input crops from multi-scale training images, as explained later). Namely, the training is carried out by optimising the multinomial logistic regression objective using mini-batch gradient descent (based on back-propagation (LeCun et al. 1989)) with momentum. The batch size was set to 256256256, momentum to 0.90.90.9. The training was regularised by weight decay (the L2L_{2}L2​ penalty multiplier set to 5⋅10−45\cdot 10^{-4}5⋅10−4) and dropout regularisation for the first two fully-connected layers (dropout ratio set to 0.50.50.5). The learning rate was initially set to 10−210^{-2}10−2, and then decreased by a factor of 101010 when the validation set accuracy stopped improving. In total, the learning rate was decreased 3 times, and the learning was stopped after 370370370K iterations (74 epochs). We conjecture that in spite of the larger number of parameters and the greater depth of our nets compared to (Krizhevsky et al. 2012), the nets required less epochs to converge due to (a) implicit regularisation imposed by greater depth and smaller conv. filter sizes; (b) pre-initialisation of certain layers.

ConvNet 训练流程大体遵循 Krizhevsky et al. 2012（稍后说明的从多尺度训练图像采样输入裁剪除外）。即用基于反向传播（LeCun et al. 1989）的小批量梯度下降加动量，优化多项逻辑回归目标。批大小 256256256，动量 0.90.90.9。正则包括权重衰减（L2L_{2}L2​ 惩罚系数 5⋅10−45\cdot 10^{-4}5⋅10−4）以及前两层全连接的 dropout（比例 0.50.50.5）。学习率初值 10−210^{-2}10−2，验证集精度不再提升时除以 101010。共降三次学习率，在 370370370K 次迭代（74 个 epoch）后停止。我们推测：尽管参数更多、网络更深，收敛所需 epoch 却少于 Krizhevsky et al. 2012，原因可能是 (a) 更大深度与更小卷积滤波器带来的隐式正则；(b) 对部分层的预初始化。

The initialisation of the network weights is important, since bad initialisation can stall learning due to the instability of gradient in deep nets. To circumvent this problem, we began with training the configuration A (Table 1), shallow enough to be trained with random initialisation. Then, when training deeper architectures, we initialised the first four convolutional layers and the last three fully-connected layers with the layers of net A (the intermediate layers were initialised randomly). We did not decrease the learning rate for the pre-initialised layers, allowing them to change during learning. For random initialisation (where applicable), we sampled the weights from a normal distribution with the zero mean and 10−210^{-2}10−2 variance. The biases were initialised with zero. It is worth noting that after the paper submission we found that it is possible to initialise the weights without pre-training by using the random initialisation procedure of Glorot & Bengio 2010.

网络权重初始化很重要：坏的初始化会因深层梯度不稳而卡住学习。为规避，我们先训练可用随机初始化训起来的较浅配置 A（表 1）。再训更深结构时，用网络 A 的层初始化前四个卷积层与后三个全连接层（中间层随机初始化）。对预初始化层不降低学习率，允许其在学习中继续变化。随机初始化（适用处）从均值 000、方差 10−210^{-2}10−2 的正态分布采样；偏置置零。值得一提：投稿后我们发现，用 Glorot & Bengio 2010 的随机初始化程序，可不经预训练直接初始化权重。

To obtain the fixed-size 224×224224\times 224224×224 ConvNet input images, they were randomly cropped from rescaled training images (one crop per image per SGD iteration). To further augment the training set, the crops underwent random horizontal flipping and random RGB colour shift (Krizhevsky et al. 2012). Training image rescaling is explained below.

为得到固定尺寸 224×224224\times 224224×224 的 ConvNet 输入，从缩放后的训练图像随机裁剪（每次 SGD 迭代每图一裁）。为进一步增广，裁剪再经随机水平翻转与随机 RGB 颜色偏移（Krizhevsky et al. 2012）。训练图像缩放见下。

Training image size.

Let SSS be the smallest side of an isotropically-rescaled training image, from which the ConvNet input is cropped (we also refer to SSS as the training scale). While the crop size is fixed to 224×224224\times 224224×224, in principle SSS can take on any value not less than 224224224: for S=224S=224S=224 the crop will capture whole-image statistics, completely spanning the smallest side of a training image; for S≫224S\gg 224S≫224 the crop will correspond to a small part of the image, containing a small object or an object part.

训练图像尺寸

令 SSS 为各向同性缩放后训练图像的最短边，ConvNet 输入自此裁出（也称训练尺度）。裁剪尺寸固定为 224×224224\times 224224×224，原则上 SSS 可取不小于 224224224 的任意值：S=224S=224S=224 时裁剪覆盖整图统计、完全跨过最短边；S≫224S\gg 224S≫224 时裁剪只对应图像一小部分，含小物体或物体局部。

We consider two approaches for setting the training scale SSS. The first is to fix SSS, which corresponds to single-scale training (note that image content within the sampled crops can still represent multi-scale image statistics). In our experiments, we evaluated models trained at two fixed scales: S=256S=256S=256 (which has been widely used in the prior art (Krizhevsky et al. 2012; Zeiler & Fergus 2013; Sermanet et al. 2014)) and S=384S=384S=384. Given a ConvNet configuration, we first trained the network using S=256S=256S=256. To speed-up training of the S=384S=384S=384 network, it was initialised with the weights pre-trained with S=256S=256S=256, and we used a smaller initial learning rate of 10−310^{-3}10−3.

设定训练尺度 SSS 有两种做法。其一是固定 SSS，即单尺度训练（采样裁剪内仍可呈现多尺度图像统计）。实验中评估了两个固定尺度：S=256S=256S=256（先前广泛使用，Krizhevsky et al. 2012；Zeiler & Fergus 2013；Sermanet et al. 2014）与 S=384S=384S=384。给定配置，先以 S=256S=256S=256 训练；为加速 S=384S=384S=384 网络，用 S=256S=256S=256 预训练权重初始化，并取更小初学习率 10−310^{-3}10−3。

The second approach to setting SSS is multi-scale training, where each training image is individually rescaled by randomly sampling SSS from a certain range [Smin,Smax]\left[S_{min},S_{max}\right][Smin​,Smax​] (we used Smin=256S_{min}=256Smin​=256 and Smax=512S_{max}=512Smax​=512). Since objects in images can be of different size, it is beneficial to take this into account during training. This can also be seen as training set augmentation by scale jittering, where a single model is trained to recognise objects over a wide range of scales. For speed reasons, we trained multi-scale models by fine-tuning all layers of a single-scale model with the same configuration, pre-trained with fixed S=384S=384S=384.

其二是多尺度训练：每张训练图像单独缩放，从区间 [Smin,Smax]\left[S_{min},S_{max}\right][Smin​,Smax​] 随机采样 SSS（取 Smin=256S_{min}=256Smin​=256，Smax=512S_{max}=512Smax​=512）。图像中物体尺寸各异，训练时加以考虑有益；也可看成用尺度抖动做训练集增广，使单模型学会识别宽尺度范围的物体。为求速度，多尺度模型由同配置、固定 S=384S=384S=384 预训练的单尺度模型微调全部层得到。

3.2 Testing

3.2 测试

At test time, given a trained ConvNet and an input image, it is classified in the following way. First, it is isotropically rescaled to a pre-defined smallest image side, denoted as QQQ (we also refer to it as the test scale). We note that QQQ is not necessarily equal to the training scale SSS (as we will show in Sect. 4, using several values of QQQ for each SSS leads to improved performance). Then, the network is applied densely over the rescaled test image in a way similar to (Sermanet et al. 2014). Namely, the fully-connected layers are first converted to convolutional layers (the first FC layer to a 7×77\times 77×7 conv. layer, the last two FC layers to 1×11\times 11×1 conv. layers). The resulting fully-convolutional net is then applied to the whole (uncropped) image. The result is a class score map with the number of channels equal to the number of classes, and a variable spatial resolution, dependent on the input image size. Finally, to obtain a fixed-size vector of class scores for the image, the class score map is spatially averaged (sum-pooled). We also augment the test set by horizontal flipping of the images; the soft-max class posteriors of the original and flipped images are averaged to obtain the final scores for the image.

测试时，给定训好的 ConvNet 与输入图像，分类如下。先各向同性缩放到预定义最短边 QQQ（也称测试尺度）。注意 QQQ 不必等于训练尺度 SSS（第 4 节将表明，对每个 SSS 用多个 QQQ 可提升表现）。随后类似 Sermanet et al. 2014，在缩放后的测试图上稠密应用网络：先把全连接层转成卷积层（第一 FC 变为 7×77\times 77×7 卷积，后两 FC 变为 1×11\times 11×1 卷积），得到的全卷积网再作用于整张（未裁剪）图像，输出通道数等于类别数、空间分辨率随输入变化的类别得分图；最后对得分图做空间平均（求和池化）得到固定长度类别得分向量。测试集还用水平翻转增广：原图与翻转图的 soft-max 后验取平均作为最终得分。

Since the fully-convolutional network is applied over the whole image, there is no need to sample multiple crops at test time (Krizhevsky et al. 2012), which is less efficient as it requires network re-computation for each crop. At the same time, using a large set of crops, as done by Szegedy et al. 2014, can lead to improved accuracy, as it results in a finer sampling of the input image compared to the fully-convolutional net. Also, multi-crop evaluation is complementary to dense evaluation due to different convolution boundary conditions: when applying a ConvNet to a crop, the convolved feature maps are padded with zeros, while in the case of dense evaluation the padding for the same crop naturally comes from the neighbouring parts of an image (due to both the convolutions and spatial pooling), which substantially increases the overall network receptive field, so more context is captured. While we believe that in practice the increased computation time of multiple crops does not justify the potential gains in accuracy, for reference we also evaluate our networks using 505050 crops per scale (5×55\times 55×5 regular grid with 222 flips), for a total of 150150150 crops over 333 scales, which is comparable to 144144144 crops over 444 scales used by Szegedy et al. 2014.

由于全卷积网络作用于整图，测试时不必像 Krizhevsky et al. 2012 那样采样多裁剪——那需要对每裁剪重算网络，效率较低。同时，像 Szegedy et al. 2014 那样用大批裁剪可提高精度，因为相对全卷积网，对输入图像的采样更细。多裁剪评估与稠密评估也互补，因卷积边界条件不同：对裁剪应用 ConvNet 时，卷积特征图用零填充；稠密评估下，同一裁剪的填充自然来自图像邻域（卷积与空间池化皆然），从而显著扩大整体感受野、捕获更多上下文。我们认为实践中多裁剪增加的计算往往不值潜在精度收益；为对照，我们也按每尺度 505050 裁剪（5×55\times 55×5 规则网格加 222 次翻转）、三尺度共 150150150 裁剪评估，与 Szegedy et al. 2014 四尺度 144144144 裁剪可比。

3.3 Implementation Details

3.3 实现细节

Our implementation is derived from the publicly available C++ Caffe toolbox (Jia 2013) (branched out in December 2013), but contains a number of significant modifications, allowing us to perform training and evaluation on multiple GPUs installed in a single system, as well as train and evaluate on full-size (uncropped) images at multiple scales (as described above). Multi-GPU training exploits data parallelism, and is carried out by splitting each batch of training images into several GPU batches, processed in parallel on each GPU. After the GPU batch gradients are computed, they are averaged to obtain the gradient of the full batch. Gradient computation is synchronous across the GPUs, so the result is exactly the same as when training on a single GPU.

实现基于公开的 C++ Caffe 工具箱（Jia 2013）（2013 年 12 月分支），但做了多项重要修改，以支持单机多 GPU 训练与评估，以及在多尺度全尺寸（未裁剪）图像上训练与评估（如上所述）。多 GPU 训练利用数据并行：把每批训练图拆成若干 GPU 子批并行处理，算完梯度后平均得到全批梯度。各 GPU 同步计算梯度，结果与单 GPU 训练完全一致。

While more sophisticated methods of speeding up ConvNet training have been recently proposed (Krizhevsky 2014), which employ model and data parallelism for different layers of the net, we have found that our conceptually much simpler scheme already provides a speedup of 3.753.753.75 times on an off-the-shelf 4-GPU system, as compared to using a single GPU. On a system equipped with four NVIDIA Titan Black GPUs, training a single net took 2–3 weeks depending on the architecture.

尽管近期有更复杂的加速方案（Krizhevsky 2014）对不同层混用模型并行与数据并行，我们发现这一概念上简单得多的方案，在现成的 4-GPU 系统上相对单 GPU 已有 3.753.753.75 倍加速。配备四块 NVIDIA Titan Black 的机器上，训一个网络视结构需 2–3 周。

4 Classification Experiments

4 分类实验

Dataset.

In this section, we present the image classification results achieved by the described ConvNet architectures on the ILSVRC-2012 dataset (which was used for ILSVRC 2012–2014 challenges). The dataset includes images of 1000 classes, and is split into three sets: training (1.31.31.3M images), validation (505050K images), and testing (100100100K images with held-out class labels). The classification performance is evaluated using two measures: the top-1 and top-5 error. The former is a multi-class classification error, i.e. the proportion of incorrectly classified images; the latter is the main evaluation criterion used in ILSVRC, and is computed as the proportion of images such that the ground-truth category is outside the top-5 predicted categories.

数据集

本节给出上述 ConvNet 结构在 ILSVRC-2012 数据集（用于 ILSVRC 2012–2014 挑战）上的图像分类结果。数据集含 1000 类，分为三部分：训练（1.31.31.3M 图）、验证（505050K 图）、测试（100100100K 图，类别标签保留）。分类表现用两项指标：top-1 与 top-5 错误率。前者即多类分类错误率（错分图像比例）；后者是 ILSVRC 主评测标准，定义为真实类别不在前五预测中的图像比例。

For the majority of experiments, we used the validation set as the test set. Certain experiments were also carried out on the test set and submitted to the official ILSVRC server as a “VGG” team entry to the ILSVRC-2014 competition (Russakovsky et al. 2014).

多数实验以验证集当测试集。部分实验也在测试集上跑，并以 “VGG” 队名义提交到官方 ILSVRC 服务器，参加 ILSVRC-2014（Russakovsky et al. 2014）。

4.1 Single Scale Evaluation

4.1 单尺度评估

We begin with evaluating the performance of individual ConvNet models at a single scale with the layer configurations described in Sect. 2.2. The test image size was set as follows: Q=SQ=SQ=S for fixed SSS, and Q=0.5(Smin+Smax)Q=0.5(S_{min}+S_{max})Q=0.5(Smin​+Smax​) for jittered S∈[Smin,Smax]S\in\left[S_{min},S_{max}\right]S∈[Smin​,Smax​]. The results of are shown in Table 3.

我们先按第 2.2 节层配置，在单尺度上评估各 ConvNet。测试图像尺寸设定为：固定 SSS 时 Q=SQ=SQ=S；抖动 S∈[Smin,Smax]S\in\left[S_{min},S_{max}\right]S∈[Smin​,Smax​] 时 Q=0.5(Smin+Smax)Q=0.5(S_{min}+S_{max})Q=0.5(Smin​+Smax​)。结果见表 3。

First, we note that using local response normalisation (A-LRN network) does not improve on the model A without any normalisation layers. We thus do not employ normalisation in the deeper architectures (B–E).

首先，局部响应归一化（A-LRN）并不优于无归一化层的模型 A。因此更深结构（B–E）不再使用归一化。

Second, we observe that the classification error decreases with the increased ConvNet depth: from 11 layers in A to 19 layers in E. Notably, in spite of the same depth, the configuration C (which contains three 1×11\times 11×1 conv. layers), performs worse than the configuration D, which uses 3×33\times 33×3 conv. layers throughout the network. This indicates that while the additional non-linearity does help (C is better than B), it is also important to capture spatial context by using conv. filters with non-trivial receptive fields (D is better than C). The error rate of our architecture saturates when the depth reaches 191919 layers, but even deeper models might be beneficial for larger datasets. We also compared the net B with a shallow net with five 5×55\times 55×5 conv. layers, which was derived from B by replacing each pair of 3×33\times 33×3 conv. layers with a single 5×55\times 55×5 conv. layer (which has the same receptive field as explained in Sect. 2.3). The top-1 error of the shallow net was measured to be 7%7\%7% higher than that of B (on a center crop), which confirms that a deep net with small filters outperforms a shallow net with larger filters.

其次，分类错误随 ConvNet 加深而下降：从 A 的 11 层到 E 的 19 层。值得注意：尽管深度相同，含三个 1×11\times 11×1 卷积的配置 C 不如全程 3×33\times 33×3 的配置 D。说明额外非线性确有帮助（C 优于 B），但用非平凡感受野的卷积滤波器捕获空间上下文同样重要（D 优于 C）。错误率在深度达 191919 层时趋于饱和，但对更大的数据集，更深模型仍可能有益。我们还将网络 B 与一个由 B 派生的浅网比较：把每对 3×33\times 33×3 换成单个 5×55\times 55×5（感受野相同，见第 2.3 节）。该浅网 top-1 错误比 B 高 7%7\%7%（中心裁剪），证实小滤波器深网优于大滤波器浅网。

Finally, scale jittering at training time (S∈[256;512]S\in[256;512]S∈[256;512]) leads to significantly better results than training on images with fixed smallest side (S=256S=256S=256 or S=384S=384S=384), even though a single scale is used at test time. This confirms that training set augmentation by scale jittering is indeed helpful for capturing multi-scale image statistics.

最后，训练时尺度抖动（S∈[256;512]S\in[256;512]S∈[256;512]）显著优于固定最短边训练（S=256S=256S=256 或 S=384S=384S=384），即便测试只用单尺度。这证实用尺度抖动增广训练集确实有助于捕获多尺度图像统计。

ConvNet config. (Table 1) smallest image side top-1 val. error (%) top-5 val. error (%)
train (SSS) test (QQQ)
A 256 256 29.6 10.4
A-LRN 256 256 29.7 10.5
B 256 256 28.7 9.9
C 256 256 28.1 9.4
384 384 28.1 9.3
[256;512] 384 27.3 8.8
D 256 256 27.0 8.8
384 384 26.8 8.7
[256;512] 384 25.6 8.1
E 256 256 27.3 9.0
384 384 26.9 8.7
[256;512] 384 25.5 8.0

Table 3: ConvNet performance at a single test scale.

ConvNet config. (Table 1) smallest image side top-1 val. error (%) top-5 val. error (%)
train (SSS) test (QQQ)
A 256 256 29.6 10.4
A-LRN 256 256 29.7 10.5
B 256 256 28.7 9.9
C 256 256 28.1 9.4
384 384 28.1 9.3
[256;512] 384 27.3 8.8
D 256 256 27.0 8.8
384 384 26.8 8.7
[256;512] 384 25.6 8.1
E 256 256 27.3 9.0
384 384 26.9 8.7
[256;512] 384 25.5 8.0

表 3：单测试尺度下的 ConvNet 表现。

4.2 Multi-Scale Evaluation

4.2 多尺度评估

Having evaluated the ConvNet models at a single scale, we now assess the effect of scale jittering at test time. It consists of running a model over several rescaled versions of a test image (corresponding to different values of QQQ), followed by averaging the resulting class posteriors. Considering that a large discrepancy between training and testing scales leads to a drop in performance, the models trained with fixed SSS were evaluated over three test image sizes, close to the training one: Q={S−32,S,S+32}Q=\{S-32,S,S+32\}Q={S−32,S,S+32}. At the same time, scale jittering at training time allows the network to be applied to a wider range of scales at test time, so the model trained with variable S∈[Smin;Smax]S\in[S_{min};S_{max}]S∈[Smin​;Smax​] was evaluated over a larger range of sizes Q={Smin,0.5(Smin+Smax),Smax}Q=\{S_{min},0.5(S_{min}+S_{max}),S_{max}\}Q={Smin​,0.5(Smin​+Smax​),Smax​}.

评估完单尺度后，我们考察测试时尺度抖动的影响：在同一测试图的多个缩放版本（不同 QQQ）上跑模型，再平均类别后验。训练与测试尺度差距过大性能会掉，故固定 SSS 训练的模型在接近训练尺度的三个测试尺寸上评估：Q={S−32,S,S+32}Q=\{S-32,S,S+32\}Q={S−32,S,S+32}。训练时做尺度抖动则允许测试用更宽尺度范围，故 S∈[Smin;Smax]S\in[S_{min};S_{max}]S∈[Smin​;Smax​] 的模型在 Q={Smin,0.5(Smin+Smax),Smax}Q=\{S_{min},0.5(S_{min}+S_{max}),S_{max}\}Q={Smin​,0.5(Smin​+Smax​),Smax​} 上评估。

The results, presented in Table 4, indicate that scale jittering at test time leads to better performance (as compared to evaluating the same model at a single scale, shown in Table 3). As before, the deepest configurations (D and E) perform the best, and scale jittering is better than training with a fixed smallest side SSS. Our best single-network performance on the validation set is 24.8%/7.5%24.8\%/7.5\%24.8%/7.5% top-1/top-5 error (highlighted in bold in Table 4). On the test set, the configuration E achieves 7.3%7.3\%7.3% top-5 error.

表 4 结果表明，测试时尺度抖动优于同模型单尺度评估（表 3）。如前，最深配置（D 与 E）最好，尺度抖动优于固定最短边 SSS 训练。验证集上最佳单网络为 24.8%/7.5%24.8\%/7.5\%24.8%/7.5% top-1/top-5 错误（表 4 粗体）。测试集上配置 E 达 7.3%7.3\%7.3% top-5 错误。

ConvNet config. (Table 1) smallest image side top-1 val. error (%) top-5 val. error (%)
train (SSS) test (QQQ)
B 256 224,256,288 28.2 9.6
C 256 224,256,288 27.7 9.2
384 352,384,416 27.8 9.2
[256;512]\left[256;512\right][256;512] 256,384,512 26.3 8.2
D 256 224,256,288 26.6 8.6
384 352,384,416 26.5 8.6
[256;512]\left[256;512\right][256;512] 256,384,512 24.8 7.5
E 256 224,256,288 26.9 8.7
384 352,384,416 26.7 8.6
[256;512]\left[256;512\right][256;512] 256,384,512 24.8 7.5

Table 4: ConvNet performance at multiple test scales.

ConvNet config. (Table 1) smallest image side top-1 val. error (%) top-5 val. error (%)
train (SSS) test (QQQ)
B 256 224,256,288 28.2 9.6
C 256 224,256,288 27.7 9.2
384 352,384,416 27.8 9.2
[256;512]\left[256;512\right][256;512] 256,384,512 26.3 8.2
D 256 224,256,288 26.6 8.6
384 352,384,416 26.5 8.6
[256;512]\left[256;512\right][256;512] 256,384,512 24.8 7.5
E 256 224,256,288 26.9 8.7
384 352,384,416 26.7 8.6
[256;512]\left[256;512\right][256;512] 256,384,512 24.8 7.5

表 4：多测试尺度下的 ConvNet 表现。

4.3 Multi-crop evaluation

4.3 多裁剪评估

In Table 5 we compare dense ConvNet evaluation with mult-crop evaluation (see Sect. 3.2 for details). We also assess the complementarity of the two evaluation techniques by averaging their soft-max outputs. As can be seen, using multiple crops performs slightly better than dense evaluation, and the two approaches are indeed complementary, as their combination outperforms each of them. As noted above, we hypothesize that this is due to a different treatment of convolution boundary conditions.

表 5 比较稠密 ConvNet 评估与多裁剪评估（细节见第 3.2 节）。我们还通过平均二者 soft-max 输出，考察两种技术是否互补。可见多裁剪略优于稠密评估，且二者确互补——组合优于各自。如前所述，我们推测这源于卷积边界条件处理不同。

ConvNet config. (Table 1) Evaluation method top-1 val. error (%) top-5 val. error (%)
D dense 24.8 7.5
multi-crop 24.6 7.5
multi-crop & dense 24.4 7.2
E dense 24.8 7.5
multi-crop 24.6 7.4
multi-crop & dense 24.4 7.1

Table 5: ConvNet evaluation techniques comparison. In all experiments the training scale SSS was sampled from [256;512]\left[256;512\right][256;512], and three test scales QQQ were considered: {256,384,512}\left\{256,384,512\right\}{256,384,512}.

ConvNet config. (Table 1) Evaluation method top-1 val. error (%) top-5 val. error (%)
D dense 24.8 7.5
multi-crop 24.6 7.5
multi-crop & dense 24.4 7.2
E dense 24.8 7.5
multi-crop 24.6 7.4
multi-crop & dense 24.4 7.1

表 5：ConvNet 评估技术比较。 各实验训练尺度 SSS 从 [256;512]\left[256;512\right][256;512] 采样，测试尺度 QQQ 取 {256,384,512}\left\{256,384,512\right\}{256,384,512}。

4.4 ConvNet Fusion

4.4 ConvNet 融合

Up until now, we evaluated the performance of individual ConvNet models. In this part of the experiments, we combine the outputs of several models by averaging their soft-max class posteriors. This improves the performance due to complementarity of the models, and was used in the top ILSVRC submissions in 2012 (Krizhevsky et al. 2012) and 2013 (Zeiler & Fergus 2013; Sermanet et al. 2014).

至此我们评估的是单个 ConvNet。本部分通过平均多个模型的 soft-max 类别后验来组合输出。因模型互补，融合可提升表现；2012（Krizhevsky et al. 2012）与 2013（Zeiler & Fergus 2013；Sermanet et al. 2014）顶尖 ILSVRC 提交也使用此法。

The results are shown in Table 6. By the time of ILSVRC submission we had only trained the single-scale networks, as well as a multi-scale model D (by fine-tuning only the fully-connected layers rather than all layers). The resulting ensemble of 7 networks has 7.3%7.3\%7.3% ILSVRC test error. After the submission, we considered an ensemble of only two best-performing multi-scale models (configurations D and E), which reduced the test error to 7.0%7.0\%7.0% using dense evaluation and 6.8%6.8\%6.8% using combined dense and multi-crop evaluation. For reference, our best-performing single model achieves 7.1%7.1\%7.1% error (model E, Table 5).

结果见表 6。提交 ILSVRC 时我们只训了单尺度网络，以及一个多尺度模型 D（仅微调全连接层而非全部层）。7 网络集成测试错误为 7.3%7.3\%7.3%。提交后，仅用两个最佳多尺度模型（配置 D 与 E）的集成，稠密评估把测试错误降到 7.0%7.0\%7.0%，稠密加多裁剪组合为 6.8%6.8\%6.8%。对照：最佳单模型错误为 7.1%7.1\%7.1%（模型 E，表 5）。

Combined ConvNet models Error
top-1 val top-5 val top-5 test
ILSVRC submission
(D/256/224,256,288), (D/384/352,384,416), (D/[256;512]/256,384,512) (C/256/224,256,288), (C/384/352,384,416) (E/256/224,256,288), (E/384/352,384,416) 24.7 7.5 7.3
post-submission
(D/[256;512]/256,384,512), (E/[256;512]/256,384,512) , dense eval. 24.0 7.1 7.0
(D/[256;512]/256,384,512), (E/[256;512]/256,384,512) , multi-crop 23.9 7.2 -
(D/[256;512]/256,384,512), (E/[256;512]/256,384,512) , multi-crop & dense eval. 23.7 6.8 6.8

Table 6: Multiple ConvNet fusion results.

Combined ConvNet models Error
top-1 val top-5 val top-5 test
ILSVRC submission
(D/256/224,256,288), (D/384/352,384,416), (D/[256;512]/256,384,512) (C/256/224,256,288), (C/384/352,384,416) (E/256/224,256,288), (E/384/352,384,416) 24.7 7.5 7.3
post-submission
(D/[256;512]/256,384,512), (E/[256;512]/256,384,512) , dense eval. 24.0 7.1 7.0
(D/[256;512]/256,384,512), (E/[256;512]/256,384,512) , multi-crop 23.9 7.2 -
(D/[256;512]/256,384,512), (E/[256;512]/256,384,512) , multi-crop & dense eval. 23.7 6.8 6.8

表 6：多 ConvNet 融合结果。

4.5 Comparison with the State of the Art

4.5 与当时最优的比较

Finally, we compare our results with the state of the art in Table 7. In the classification task of ILSVRC-2014 challenge (Russakovsky et al. 2014), our “VGG” team secured the 2nd place with 7.3%7.3\%7.3% test error using an ensemble of 7 models. After the submission, we decreased the error rate to 6.8%6.8\%6.8% using an ensemble of 2 models.

最后，表 7 与当时最优比较。在 ILSVRC-2014 分类赛道（Russakovsky et al. 2014），我们的 “VGG” 队以 7 模型集成 7.3%7.3\%7.3% 测试错误获第二；提交后用 2 模型集成把错误率降到 6.8%6.8\%6.8%。

As can be seen from Table 7, our very deep ConvNets significantly outperform the previous generation of models, which achieved the best results in the ILSVRC-2012 and ILSVRC-2013 competitions. Our result is also competitive with respect to the classification task winner (GoogLeNet with 6.7%6.7\%6.7% error) and substantially outperforms the ILSVRC-2013 winning submission Clarifai, which achieved 11.2%11.2\%11.2% with outside training data and 11.7%11.7\%11.7% without it. This is remarkable, considering that our best result is achieved by combining just two models – significantly less than used in most ILSVRC submissions. In terms of the single-net performance, our architecture achieves the best result (7.0%7.0\%7.0% test error), outperforming a single GoogLeNet by 0.9%0.9\%0.9%. Notably, we did not depart from the classical ConvNet architecture of LeCun et al. 1989, but improved it by substantially increasing the depth.

由表 7 可见，极深 ConvNet 显著优于 ILSVRC-2012 与 ILSVRC-2013 的上一代最佳模型。相对分类冠军（GoogLeNet，6.7%6.7\%6.7% 错误）我们的结果也具竞争力，并大幅优于 ILSVRC-2013 获胜提交 Clarifai（有外部训练数据 11.2%11.2\%11.2%、无外部 11.7%11.7\%11.7%）。考虑到最佳结果仅组合两个模型——远少于多数 ILSVRC 提交所用数量——这一点尤为突出。就单网表现，我们的结构以 7.0%7.0\%7.0% 测试错误居首，比单网 GoogLeNet 优 0.9%0.9\%0.9%。值得注意的是，我们并未离开 LeCun et al. 1989 的经典 ConvNet 架构，而是靠大幅加深改进它。

Method top-1 val. error (%) top-5 val. error (%) top-5 test error (%)
VGG (2 nets, multi-crop & dense eval.) 23.7 6.8 6.8
VGG (1 net, multi-crop & dense eval.) 24.4 7.1 7.0
VGG (ILSVRC submission, 7 nets, dense eval.) 24.7 7.5 7.3
GoogLeNet (Szegedy et al. 2014) (1 net) - 7.9
GoogLeNet (Szegedy et al. 2014) (7 nets) - 6.7
MSRA (He et al. 2014) (11 nets) - - 8.1
MSRA (He et al. 2014) (1 net) 27.9 9.1 9.1
Clarifai (Russakovsky et al. 2014) (multiple nets) - - 11.7
Clarifai (Russakovsky et al. 2014) (1 net) - - 12.5
Zeiler & Fergus (Zeiler & Fergus 2013) (6 nets) 36.0 14.7 14.8
Zeiler & Fergus (Zeiler & Fergus 2013) (1 net) 37.5 16.0 16.1
OverFeat (Sermanet et al. 2014) (7 nets) 34.0 13.2 13.6
OverFeat (Sermanet et al. 2014) (1 net) 35.7 14.2 -
Krizhevsky et al. (Krizhevsky et al. 2012) (5 nets) 38.1 16.4 16.4
Krizhevsky et al. (Krizhevsky et al. 2012) (1 net) 40.7 18.2 -

Table 7: Comparison with the state of the art in ILSVRC classification. Our method is denoted as “VGG”. Only the results obtained without outside training data are reported.

Method top-1 val. error (%) top-5 val. error (%) top-5 test error (%)
VGG (2 nets, multi-crop & dense eval.) 23.7 6.8 6.8
VGG (1 net, multi-crop & dense eval.) 24.4 7.1 7.0
VGG (ILSVRC submission, 7 nets, dense eval.) 24.7 7.5 7.3
GoogLeNet (Szegedy et al. 2014) (1 net) - 7.9
GoogLeNet (Szegedy et al. 2014) (7 nets) - 6.7
MSRA (He et al. 2014) (11 nets) - - 8.1
MSRA (He et al. 2014) (1 net) 27.9 9.1 9.1
Clarifai (Russakovsky et al. 2014) (multiple nets) - - 11.7
Clarifai (Russakovsky et al. 2014) (1 net) - - 12.5
Zeiler & Fergus (Zeiler & Fergus 2013) (6 nets) 36.0 14.7 14.8
Zeiler & Fergus (Zeiler & Fergus 2013) (1 net) 37.5 16.0 16.1
OverFeat (Sermanet et al. 2014) (7 nets) 34.0 13.2 13.6
OverFeat (Sermanet et al. 2014) (1 net) 35.7 14.2 -
Krizhevsky et al. (Krizhevsky et al. 2012) (5 nets) 38.1 16.4 16.4
Krizhevsky et al. (Krizhevsky et al. 2012) (1 net) 40.7 18.2 -

表 7：与 ILSVRC 分类当时最优的比较。我们的方法记为 “VGG”。仅报告不使用外部训练数据的结果。

5 Conclusion

5 结论

In this work we evaluated very deep convolutional networks (up to 19 weight layers) for large-scale image classification. It was demonstrated that the representation depth is beneficial for the classification accuracy, and that state-of-the-art performance on the ImageNet challenge dataset can be achieved using a conventional ConvNet architecture (LeCun et al. 1989; Krizhevsky et al. 2012) with substantially increased depth. In the appendix, we also show that our models generalise well to a wide range of tasks and datasets, matching or outperforming more complex recognition pipelines built around less deep image representations. Our results yet again confirm the importance of depth in visual representations.

本文评估了用于大规模图像分类的极深卷积网络（最多 19 个权重层）。结果表明表征深度有利于分类精度；用大幅加深的常规 ConvNet 结构（LeCun et al. 1989；Krizhevsky et al. 2012）即可在 ImageNet 挑战数据集上达到当时最优。附录还表明我们的模型能良好泛化到多种任务与数据集，匹配或优于围绕较浅图像表征构建的更复杂识别流水线。结果再次确认了视觉表征中深度的重要性。

Acknowledgements

This work was supported by ERC grant VisRec no. 228180. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GPUs used for this research.

致谢

本工作获 ERC 资助 VisRec no. 228180。感谢 NVIDIA 公司捐赠本研究所用 GPU。

References

参考文献

Bell et al. (2014) Bell, S., Upchurch, P., Snavely, N., and Bala, K. Material recognition in the wild with the materials in context database. CoRR , abs/1412.0623, 2014.

Chatfield et al. (2014) Chatfield, K., Simonyan, K., Vedaldi, A., and Zisserman, A. Return of the devil in the details: Delving deep into convolutional nets. In Proc. BMVC. , 2014.

Cimpoi et al. (2014) Cimpoi, M., Maji, S., and Vedaldi, A. Deep convolutional filter banks for texture recognition and segmentation. CoRR , abs/1411.6836, 2014.

Ciresan et al. (2011) Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., and Schmidhuber, J. Flexible, high performance convolutional neural networks for image classification. In IJCAI , pp. 1237–1242, 2011.

Dean et al. (2012) Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., and Ng, A. Y. Large scale distributed deep networks. In NIPS , pp. 1232–1240, 2012.

Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proc. CVPR , 2009.

Donahue et al. (2013) Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. Decaf: A deep convolutional activation feature for generic visual recognition. CoRR , abs/1310.1531, 2013.

Everingham et al. (2015) Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C., Winn, J., and Zisserman, A. The Pascal visual object classes challenge: A retrospective. IJCV , 111(1):98–136, 2015.

Fei-Fei et al. (2004) Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE CVPR Workshop of Generative Model Based Vision , 2004.

Girshick et al. (2014) Girshick, R. B., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. CoRR , abs/1311.2524v5, 2014. Published in Proc. CVPR, 2014.

Gkioxari et al. (2014) Gkioxari, G., Girshick, R., and Malik, J. Actions and attributes from wholes and parts. CoRR , abs/1412.2604, 2014.

Glorot & Bengio (2010) Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. AISTATS , volume 9, pp. 249–256, 2010.

Goodfellow et al. (2014) Goodfellow, I. J., Bulatov, Y., Ibarz, J., Arnoud, S., and Shet, V. Multi-digit number recognition from street view imagery using deep convolutional neural networks. In Proc. ICLR , 2014.

Griffin et al. (2007) Griffin, G., Holub, A., and Perona, P. Caltech-256 object category dataset. Technical Report 7694, California Institute of Technology, 2007.

He et al. (2014) He, K., Zhang, X., Ren, S., and Sun, J. Spatial pyramid pooling in deep convolutional networks for visual recognition. CoRR , abs/1406.4729v2, 2014.

Hoai (2014) Hoai, M. Regularized max pooling for image categorization. In Proc. BMVC. , 2014.

Howard (2014) Howard, A. G. Some improvements on deep convolutional neural network based image classification. In Proc. ICLR , 2014.

Jia (2013) Jia, Y. Caffe: An open source convolutional architecture for fast feature embedding. http://caffe.berkeleyvision.org/ , 2013.

Karpathy & Fei-Fei (2014) Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. CoRR , abs/1412.2306, 2014.

Kiros et al. (2014) Kiros, R., Salakhutdinov, R., and Zemel, R. S. Unifying visual-semantic embeddings with multimodal neural language models. CoRR , abs/1411.2539, 2014.

Krizhevsky (2014) Krizhevsky, A. One weird trick for parallelizing convolutional neural networks. CoRR , abs/1404.5997, 2014.

Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet classification with deep convolutional neural networks. In NIPS , pp. 1106–1114, 2012.

LeCun et al. (1989) LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. Backpropagation applied to handwritten zip code recognition. Neural Computation , 1(4):541–551, 1989.

Lin et al. (2014) Lin, M., Chen, Q., and Yan, S. Network in network. In Proc. ICLR , 2014.

Long et al. (2014) Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. CoRR , abs/1411.4038, 2014.

Oquab et al. (2014) Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks. In Proc. CVPR , 2014.

Perronnin et al. (2010) Perronnin, F., Sánchez, J., and Mensink, T. Improving the Fisher kernel for large-scale image classification. In Proc. ECCV , 2010.

Razavian et al. (2014) Razavian, A., Azizpour, H., Sullivan, J., and Carlsson, S. CNN Features off-the-shelf: an Astounding Baseline for Recognition. CoRR , abs/1403.6382, 2014.

Russakovsky et al. (2014) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet large scale visual recognition challenge. CoRR , abs/1409.0575, 2014.

Sermanet et al. (2014) Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks. In Proc. ICLR , 2014.

Simonyan & Zisserman (2014) Simonyan, K. and Zisserman, A. Two-stream convolutional networks for action recognition in videos. CoRR , abs/1406.2199, 2014. Published in Proc. NIPS, 2014.

Szegedy et al. (2014) Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. CoRR , abs/1409.4842, 2014.

Wei et al. (2014) Wei, Y., Xia, W., Huang, J., Ni, B., Dong, J., Zhao, Y., and Yan, S. CNN: Single-label to multi-label. CoRR , abs/1406.5726, 2014.

Zeiler & Fergus (2013) Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. CoRR , abs/1311.2901, 2013. Published in Proc. ECCV, 2014.

Appendix A Localisation

附录 A 定位

In the main body of the paper we have considered the classification task of the ILSVRC challenge, and performed a thorough evaluation of ConvNet architectures of different depth. In this section, we turn to the localisation task of the challenge, which we have won in 2014 with 25.3%25.3\%25.3% error. It can be seen as a special case of object detection, where a single object bounding box should be predicted for each of the top-5 classes, irrespective of the actual number of objects of the class. For this we adopt the approach of Sermanet et al. 2014, the winners of the ILSVRC-2013 localisation challenge, with a few modifications. Our method is described in Sect. A.1 and evaluated in Sect. A.2.

正文已讨论 ILSVRC 分类任务，并系统评估不同深度的 ConvNet。本节转向我们在 2014 年以 25.3%25.3\%25.3% 错误率夺冠的定位任务。它可看作目标检测的特例：对 top-5 各类各预测一个物体包围框，而不论该类实际物体个数。我们采用 ILSVRC-2013 定位冠军 Sermanet et al. 2014 的思路并做少量修改。方法见第 A.1 节，评估见第 A.2 节。

A.1 Localisation ConvNet

A.1 定位 ConvNet

To perform object localisation, we use a very deep ConvNet, where the last fully connected layer predicts the bounding box location instead of the class scores. A bounding box is represented by a 4-D vector storing its center coordinates, width, and height. There is a choice of whether the bounding box prediction is shared across all classes (single-class regression, SCR (Sermanet et al. 2014)) or is class-specific (per-class regression, PCR). In the former case, the last layer is 4-D, while in the latter it is 4000-D (since there are 1000 classes in the dataset). Apart from the last bounding box prediction layer, we use the ConvNet architecture D (Table 1), which contains 16 weight layers and was found to be the best-performing in the classification task (Sect. 4).

为做物体定位，我们使用极深 ConvNet，最后全连接层预测包围框位置而非类别得分。包围框用 4 维向量表示中心坐标、宽与高。可选择框预测跨类共享（单类回归 SCR，Sermanet et al. 2014）或按类专用（每类回归 PCR）。前者最后一层 4 维，后者 4000 维（数据集 1000 类）。除最后包围框预测层外，采用配置 D（表 1）——含 16 个权重层，分类任务中表现最佳（第 4 节）。

Training.

Training of localisation ConvNets is similar to that of the classification ConvNets (Sect. 3.1). The main difference is that we replace the logistic regression objective with a Euclidean loss, which penalises the deviation of the predicted bounding box parameters from the ground-truth. We trained two localisation models, each on a single scale: S=256S=256S=256 and S=384S=384S=384 (due to the time constraints, we did not use training scale jittering for our ILSVRC-2014 submission). Training was initialised with the corresponding classification models (trained on the same scales), and the initial learning rate was set to 10−310^{-3}10−3. We explored both fine-tuning all layers and fine-tuning only the first two fully-connected layers, as done in (Sermanet et al. 2014). The last fully-connected layer was initialised randomly and trained from scratch.

训练

定位 ConvNet 训练类似分类（第 3.1 节）。主要差别是把逻辑回归目标换成欧氏损失，惩罚预测框参数相对真值的偏差。我们训了两个定位模型，各用单尺度：S=256S=256S=256 与 S=384S=384S=384（时间所限，ILSVRC-2014 提交未用训练尺度抖动）。训练用对应分类模型（同尺度）初始化，初学习率 10−310^{-3}10−3。我们探索了微调全部层，以及像 Sermanet et al. 2014 那样只微调前两层全连接。最后全连接层随机初始化、从零训练。

Testing.

We consider two testing protocols. The first is used for comparing different network modifications on the validation set, and considers only the bounding box prediction for the ground truth class (to factor out the classification errors). The bounding box is obtained by applying the network only to the central crop of the image.

测试

考虑两种测试协议。第一种用于在验证集上比较不同网络改动，只看真实类别的包围框预测（以剥离分类错误）；框由仅对图像中心裁剪应用网络得到。

The second, fully-fledged, testing procedure is based on the dense application of the localisation ConvNet to the whole image, similarly to the classification task (Sect. 3.2). The difference is that instead of the class score map, the output of the last fully-connected layer is a set of bounding box predictions. To come up with the final prediction, we utilise the greedy merging procedure of Sermanet et al. 2014, which first merges spatially close predictions (by averaging their coordinates), and then rates them based on the class scores, obtained from the classification ConvNet. When several localisation ConvNets are used, we first take the union of their sets of bounding box predictions, and then run the merging procedure on the union. We did not use the multiple pooling offsets technique of Sermanet et al. 2014, which increases the spatial resolution of the bounding box predictions and can further improve the results.

第二种完整测试流程类似分类（第 3.2 节），把定位 ConvNet 稠密作用于整图。差别是最后全连接层输出的是一组包围框预测而非类别得分图。最终预测采用 Sermanet et al. 2014 的贪心合并：先合并空间接近的预测（坐标取平均），再按分类 ConvNet 的类别得分排序。多个定位 ConvNet 时，先取框预测集合的并，再对并集跑合并。我们未使用 Sermanet et al. 2014 的多池化偏移技术——该技术提高框预测空间分辨率，可进一步提升结果。

A.2 Localisation Experiments

A.2 定位实验

In this section we first determine the best-performing localisation setting (using the first test protocol), and then evaluate it in a fully-fledged scenario (the second protocol). The localisation error is measured according to the ILSVRC criterion (Russakovsky et al. 2014), i.e. the bounding box prediction is deemed correct if its intersection over union ratio with the ground-truth bounding box is above 0.50.50.5.

本节先用第一测试协议确定最佳定位设定，再在完整场景（第二协议）下评估。定位错误按 ILSVRC 准则（Russakovsky et al. 2014）计量：预测框与真值框交并比高于 0.50.50.5 则判对。

Settings comparison.

As can be seen from Table 8, per-class regression (PCR) outperforms the class-agnostic single-class regression (SCR), which differs from the findings of Sermanet et al. 2014, where PCR was outperformed by SCR. We also note that fine-tuning all layers for the localisation task leads to noticeably better results than fine-tuning only the fully-connected layers (as done in (Sermanet et al. 2014)). In these experiments, the smallest images side was set to S=384S=384S=384; the results with S=256S=256S=256 exhibit the same behaviour and are not shown for brevity.

设定比较

由表 8，每类回归（PCR）优于类无关的单类回归（SCR），这与 Sermanet et al. 2014 中 PCR 不如 SCR 的发现不同。我们还注意到：为定位任务微调全部层，明显优于只微调全连接层（如 Sermanet et al. 2014）。这些实验最短边取 S=384S=384S=384；S=256S=256S=256 表现趋势相同，从略。

Fine-tuned layers regression type GT class localisation error
1st and 2nd FC SCR 36.4
PCR 34.3
all PCR 33.1

Table 8: Localisation error for different modifications with the simplified testing protocol: the bounding box is predicted from a single central image crop, and the ground-truth class is used. All ConvNet layers (except for the last one) have the configuration D (Table 1), while the last layer performs either single-class regression (SCR) or per-class regression (PCR).

Fine-tuned layers regression type GT class localisation error
1st and 2nd FC SCR 36.4
PCR 34.3
all PCR 33.1

Table 8: Localisation error for different modifications with the simplified testing protocol: the bounding box is predicted from a single central image crop, and the ground-truth class is used. All ConvNet layers (except for the last one) have the configuration D (Table 1), while the last layer performs either single-class regression (SCR) or per-class regression (PCR).

表 8：不同改动的定位错误（简化测试协议：单中心裁剪预测框，并用真实类别）。除最后一层外均为配置 D（表 1）；最后一层为单类回归（SCR）或每类回归（PCR）。

Fully-fledged evaluation.

Having determined the best localisation setting (PCR, fine-tuning of all layers), we now apply it in the fully-fledged scenario, where the top-5 class labels are predicted using our best-performing classification system (Sect. 4.5), and multiple densely-computed bounding box predictions are merged using the method of Sermanet et al. 2014. As can be seen from Table 9, application of the localisation ConvNet to the whole image substantially improves the results compared to using a center crop (Table 8), despite using the top-5 predicted class labels instead of the ground truth. Similarly to the classification task (Sect. 4), testing at several scales and combining the predictions of multiple networks further improves the performance.

完整评估

确定最佳定位设定（PCR、微调全部层）后，我们在完整场景下应用：用最佳分类系统（第 4.5 节）预测 top-5 类别标签，并用 Sermanet et al. 2014 的方法合并多个稠密计算的包围框预测。由表 9，尽管用的是 top-5 预测类而非真值，把定位 ConvNet 作用于整图相对中心裁剪（表 8）仍大幅提升。与分类（第 4 节）类似，多尺度测试与多网络融合可进一步提升。

smallest image side top-5 localisation error (%)
train (SSS) test (QQQ) val. test.
256 256 29.5 -
384 384 28.2 26.7
384 352,384 27.5 -
fusion: 256/256 and 384/352,384 26.9 25.3

Table 9: Localisation error

smallest image side top-5 localisation error (%)
train (SSS) test (QQQ) val. test.
256 256 29.5 -
384 384 28.2 26.7
384 352,384 27.5 -
fusion: 256/256 and 384/352,384 26.9 25.3

表 9：定位错误

Comparison with the state of the art.

We compare our best localisation result with the state of the art in Table 10. With 25.3%25.3\%25.3% test error, our “VGG” team won the localisation challenge of ILSVRC-2014 (Russakovsky et al. 2014). Notably, our results are considerably better than those of the ILSVRC-2013 winner Overfeat (Sermanet et al. 2014), even though we used less scales and did not employ their resolution enhancement technique. We envisage that better localisation performance can be achieved if this technique is incorporated into our method. This indicates the performance advancement brought by our very deep ConvNets – we got better results with a simpler localisation method, but a more powerful representation.

与当时最优的比较

表 10 比较我们的最佳定位结果与当时最优。以 25.3%25.3\%25.3% 测试错误，“VGG” 队赢得 ILSVRC-2014 定位挑战（Russakovsky et al. 2014）。相对 ILSVRC-2013 冠军 Overfeat（Sermanet et al. 2014），我们用更少尺度且未用其分辨率增强技术，结果仍明显更好。若纳入该技术，定位表现或可再升。这表明极深 ConvNet 带来的表征进步——用更简单的定位方法、更强的表征，就能得到更好结果。

Method top-5 val. error (%) top-5 test error (%)
VGG 26.9 25.3
GoogLeNet (Szegedy et al. 2014) - 26.7
OverFeat (Sermanet et al. 2014) 30.0 29.9
Krizhevsky et al. (Krizhevsky et al. 2012) - 34.2

Table 10: Comparison with the state of the art in ILSVRC localisation. Our method is denoted as “VGG”.

Method top-5 val. error (%) top-5 test error (%)
VGG 26.9 25.3
GoogLeNet (Szegedy et al. 2014) - 26.7
OverFeat (Sermanet et al. 2014) 30.0 29.9
Krizhevsky et al. (Krizhevsky et al. 2012) - 34.2

表 10：与 ILSVRC 定位当时最优的比较。我们的方法记为 “VGG”。

Appendix B Generalisation of Very Deep Features

附录 B 极深特征的泛化

In the previous sections we have discussed training and evaluation of very deep ConvNets on the ILSVRC dataset. In this section, we evaluate our ConvNets, pre-trained on ILSVRC, as feature extractors on other, smaller, datasets, where training large models from scratch is not feasible due to over-fitting. Recently, there has been a lot of interest in such a use case (Zeiler & Fergus 2013; Donahue et al. 2013; Razavian et al. 2014; Chatfield et al. 2014), as it turns out that deep image representations, learnt on ILSVRC, generalise well to other datasets, where they have outperformed hand-crafted representations by a large margin. Following that line of work, we investigate if our models lead to better performance than more shallow models utilised in the state-of-the-art methods. In this evaluation, we consider two models with the best classification performance on ILSVRC (Sect. 4) – configurations “Net-D” and “Net-E” (which we made publicly available).

前几节讨论了在 ILSVRC 上训练与评估极深 ConvNet。本节把在 ILSVRC 上预训练的 ConvNet 当作特征提取器，评估到其他较小数据集——因过拟合风险，这些数据集上不宜从零训大模型。此类用法近来很受关注（Zeiler & Fergus 2013；Donahue et al. 2013；Razavian et al. 2014；Chatfield et al. 2014）：在 ILSVRC 上学到的深度图像表征能良好泛化到其他数据集，并大幅超过手工特征。沿此路线，我们考察相对当时最优方法所用的较浅模型，我们的模型是否带来更好表现。评估采用 ILSVRC 分类表现最好的两个模型（第 4 节）——配置 “Net-D” 与 “Net-E”（已公开发布）。

To utilise the ConvNets, pre-trained on ILSVRC, for image classification on other datasets, we remove the last fully-connected layer (which performs 1000-way ILSVRC classification), and use 4096-D activations of the penultimate layer as image features, which are aggregated across multiple locations and scales. The resulting image descriptor is L2L_{2}L2​-normalised and combined with a linear SVM classifier, trained on the target dataset. For simplicity, pre-trained ConvNet weights are kept fixed (no fine-tuning is performed).

为把 ILSVRC 预训练 ConvNet 用于其他数据集的图像分类，去掉最后做 1000 类 ILSVRC 分类的全连接层，用倒数第二层的 4096 维激活作为图像特征，并在多位置、多尺度上聚合。所得图像描述子经 L2L_{2}L2​ 归一化，再与在目标数据集上训练的线性 SVM 结合。为简单起见，预训练 ConvNet 权重固定（不做微调）。

Aggregation of features is carried out in a similar manner to our ILSVRC evaluation procedure (Sect. 3.2). Namely, an image is first rescaled so that its smallest side equals QQQ, and then the network is densely applied over the image plane (which is possible when all weight layers are treated as convolutional). We then perform global average pooling on the resulting feature map, which produces a 4096-D image descriptor. The descriptor is then averaged with the descriptor of a horizontally flipped image. As was shown in Sect. 4.2, evaluation over multiple scales is beneficial, so we extract features over several scales QQQ. The resulting multi-scale features can be either stacked or pooled across scales. Stacking allows a subsequent classifier to learn how to optimally combine image statistics over a range of scales; this, however, comes at the cost of the increased descriptor dimensionality. We return to the discussion of this design choice in the experiments below. We also assess late fusion of features, computed using two networks, which is performed by stacking their respective image descriptors.

特征聚合方式类似我们的 ILSVRC 评估（第 3.2 节）。即先把图像缩放到最短边等于 QQQ，再在图像平面稠密应用网络（把所有权重层当作卷积即可）。随后对所得特征图做全局平均池化，得到 4096 维图像描述子；再与水平翻转图像的描述子取平均。第 4.2 节表明多尺度评估有益，故在多个尺度 QQQ 上提取特征。多尺度特征可跨尺度堆叠或池化：堆叠让后续分类器学习如何最优组合各尺度统计，但描述子维度升高。下文实验回到这一设计选择。我们还评估两个网络特征的后期融合——堆叠各自图像描述子。

Method VOC-2007 VOC-2012 Caltech-101 Caltech-256
(mean AP) (mean AP) (mean class recall) (mean class recall)
Zeiler & Fergus (Zeiler & Fergus 2013) - 79.0 86.5 ±\pm± 0.5 74.2 ±\pm± 0.3
Chatfield et al. (Chatfield et al. 2014) 82.4 83.2 88.4 ±\pm± 0.6 77.6 ±\pm± 0.1
He et al. (He et al. 2014) 82.4 - 93.4 ±\pm± 0.5 -
Wei et al. (Wei et al. 2014) 81.5 (85.2∗) 81.7 (90.3∗) - -
VGG Net-D (16 layers) 89.3 89.0 91.8 ±\pm± 1.0 85.0 ±\pm± 0.2
VGG Net-E (19 layers) 89.3 89.0 92.3 ±\pm± 0.5 85.1 ±\pm± 0.3
VGG Net-D & Net-E 89.7 89.3 92.7 ±\pm± 0.5 86.2 ±\pm± 0.3

Table 11: Comparison with the state of the art in image classification on VOC-2007, VOC-2012, Caltech-101, and Caltech-256. Our models are denoted as “VGG”. Results marked with * were achieved using ConvNets pre-trained on the extended ILSVRC dataset (2000 classes).

Method VOC-2007 VOC-2012 Caltech-101 Caltech-256
(mean AP) (mean AP) (mean class recall) (mean class recall)
Zeiler & Fergus (Zeiler & Fergus 2013) - 79.0 86.5 ±\pm± 0.5 74.2 ±\pm± 0.3
Chatfield et al. (Chatfield et al. 2014) 82.4 83.2 88.4 ±\pm± 0.6 77.6 ±\pm± 0.1
He et al. (He et al. 2014) 82.4 - 93.4 ±\pm± 0.5 -
Wei et al. (Wei et al. 2014) 81.5 (85.2∗) 81.7 (90.3∗) - -
VGG Net-D (16 layers) 89.3 89.0 91.8 ±\pm± 1.0 85.0 ±\pm± 0.2
VGG Net-E (19 layers) 89.3 89.0 92.3 ±\pm± 0.5 85.1 ±\pm± 0.3
VGG Net-D & Net-E 89.7 89.3 92.7 ±\pm± 0.5 86.2 ±\pm± 0.3

表 11：在 VOC-2007、VOC-2012、Caltech-101 与 Caltech-256 图像分类上与当时最优的比较。我们的模型记为 “VGG”。标 * 的结果使用在扩展 ILSVRC（2000 类）上预训练的 ConvNet。

Image Classification on VOC-2007 and VOC-2012.

We begin with the evaluation on the image classification task of PASCAL VOC-2007 and VOC-2012 benchmarks (Everingham et al. 2015). These datasets contain 10K and 22.5K images respectively, and each image is annotated with one or several labels, corresponding to 20 object categories. The VOC organisers provide a pre-defined split into training, validation, and test data (the test data for VOC-2012 is not publicly available; instead, an official evaluation server is provided). Recognition performance is measured using mean average precision (mAP) across classes.

VOC-2007 与 VOC-2012 上的图像分类

我们先在 PASCAL VOC-2007 与 VOC-2012 基准（Everingham et al. 2015）的图像分类任务上评估。二者分别约 10K 与 22.5K 图，每图标有对应 20 个物体类别的一个或多个标签。主办方提供训练/验证/测试预定义划分（VOC-2012 测试数据不公开，提供官方评测服务器）。识别表现用各类平均精度均值（mAP）衡量。

Notably, by examining the performance on the validation sets of VOC-2007 and VOC-2012, we found that aggregating image descriptors, computed at multiple scales, by averaging performs similarly to the aggregation by stacking. We hypothesize that this is due to the fact that in the VOC dataset the objects appear over a variety of scales, so there is no particular scale-specific semantics which a classifier could exploit. Since averaging has a benefit of not inflating the descriptor dimensionality, we were able to aggregated image descriptors over a wide range of scales: Q∈{256,384,512,640,768}Q\in\{256,384,512,640,768\}Q∈{256,384,512,640,768}. It is worth noting though that the improvement over a smaller range of {256,384,512}\{256,384,512\}{256,384,512} was rather marginal (0.3%0.3\%0.3%).

在 VOC-2007 与 VOC-2012 验证集上我们发现：对多尺度计算的图像描述子做平均聚合，与堆叠聚合表现相近。我们推测是因为 VOC 中物体尺度变化大，分类器难以利用特定尺度语义。平均不抬升描述子维度，因而我们能在宽尺度范围聚合：Q∈{256,384,512,640,768}Q\in\{256,384,512,640,768\}Q∈{256,384,512,640,768}。相对较小范围 {256,384,512}\{256,384,512\}{256,384,512} 的提升其实有限（0.3%0.3\%0.3%）。

The test set performance is reported and compared with other approaches in Table 11. Our networks “Net-D” and “Net-E” exhibit identical performance on VOC datasets, and their combination slightly improves the results. Our methods set the new state of the art across image representations, pre-trained on the ILSVRC dataset, outperforming the previous best result of Chatfield et al. 2014 by more than 6%6\%6%. It should be noted that the method of Wei et al. 2014, which achieves 1%1\%1% better mAP on VOC-2012, is pre-trained on an extended 2000-class ILSVRC dataset, which includes additional 1000 categories, semantically close to those in VOC datasets. It also benefits from the fusion with an object detection-assisted classification pipeline.

测试集表现及与其他方法比较见表 11。“Net-D” 与 “Net-E” 在 VOC 上表现相同，组合略有提升。在 ILSVRC 预训练的图像表征中，我们的方法刷新当时最优，相对 Chatfield et al. 2014 先前最佳高出逾 6%6\%6%。需指出：Wei et al. 2014 在 VOC-2012 上 mAP 高约 1%1\%1%，但其预训练用的是扩展的 2000 类 ILSVRC（含与 VOC 语义接近的额外 1000 类），并融合了物体检测辅助的分类流水线。

Image Classification on Caltech-101 and Caltech-256.

In this section we evaluate very deep features on Caltech-101 (Fei-Fei et al. 2004) and Caltech-256 (Griffin et al. 2007) image classification benchmarks. Caltech-101 contains 9K images labelled into 102 classes (101 object categories and a background class), while Caltech-256 is larger with 31K images and 257 classes. A standard evaluation protocol on these datasets is to generate several random splits into training and test data and report the average recognition performance across the splits, which is measured by the mean class recall (which compensates for a different number of test images per class). Following Chatfield et al. 2014; Zeiler & Fergus 2013; He et al. 2014, on Caltech-101 we generated 3 random splits into training and test data, so that each split contains 30 training images per class, and up to 50 test images per class. On Caltech-256 we also generated 3 splits, each of which contains 60 training images per class (and the rest is used for testing). In each split, 20% of training images were used as a validation set for hyper-parameter selection.

Caltech-101 与 Caltech-256 上的图像分类

本节在 Caltech-101（Fei-Fei et al. 2004）与 Caltech-256（Griffin et al. 2007）图像分类基准上评估极深特征。Caltech-101 约 9K 图、102 类（101 物体类加背景）；Caltech-256 更大，约 31K 图、257 类。标准协议是生成若干随机训练/测试划分，报告跨划分平均识别表现，指标为各类召回均值（补偿每类测试图数量差异）。遵循 Chatfield et al. 2014；Zeiler & Fergus 2013；He et al. 2014：在 Caltech-101 上生成 3 个随机划分，每划分每类 30 张训练、至多 50 张测试；Caltech-256 也生成 3 个划分，每划分每类 60 张训练（其余测试）。每个划分中 20%20\%20% 训练图作验证集选超参。

We found that unlike VOC, on Caltech datasets the stacking of descriptors, computed over multiple scales, performs better than averaging or max-pooling. This can be explained by the fact that in Caltech images objects typically occupy the whole image, so multi-scale image features are semantically different (capturing the whole object vs. object parts), and stacking allows a classifier to exploit such scale-specific representations. We used three scales Q∈{256,384,512}Q\in\{256,384,512\}Q∈{256,384,512}.

我们发现与 VOC 不同：在 Caltech 上，多尺度描述子堆叠优于平均或最大池化。可解释为 Caltech 图像中物体通常占满整图，多尺度特征语义不同（整物体 vs. 部件），堆叠让分类器利用这种尺度特定表征。我们使用三尺度 Q∈{256,384,512}Q\in\{256,384,512\}Q∈{256,384,512}。

Our models are compared to each other and the state of the art in Table 11. As can be seen, the deeper 19-layer Net-E performs better than the 16-layer Net-D, and their combination further improves the performance. On Caltech-101, our representations are competitive with the approach of He et al. 2014, which, however, performs significantly worse than our nets on VOC-2007. On Caltech-256, our features outperform the state of the art (Chatfield et al. 2014) by a large margin (8.6%8.6\%8.6%).

表 11 比较各模型与当时最优。更深的 19 层 Net-E 优于 16 层 Net-D，组合进一步提升。在 Caltech-101 上，我们的表征与 He et al. 2014 有竞争力——但后者在 VOC-2007 上明显弱于我们的网络。在 Caltech-256 上，我们的特征大幅超过当时最优（Chatfield et al. 2014）（高出 8.6%8.6\%8.6%）。

Action Classification on VOC-2012.

We also evaluated our best-performing image representation (the stacking of Net-D and Net-E features) on the PASCAL VOC-2012 action classification task (Everingham et al. 2015), which consists in predicting an action class from a single image, given a bounding box of the person performing the action. The dataset contains 4.6K training images, labelled into 11 classes. Similarly to the VOC-2012 object classification task, the performance is measured using the mAP. We considered two training settings: (i) computing the ConvNet features on the whole image and ignoring the provided bounding box; (ii) computing the features on the whole image and on the provided bounding box, and stacking them to obtain the final representation. The results are compared to other approaches in Table 12.

VOC-2012 上的动作分类

我们还把最佳图像表征（Net-D 与 Net-E 特征堆叠）用于 PASCAL VOC-2012 动作分类任务（Everingham et al. 2015）：给定单张图像与执行动作的人的包围框，预测动作类。数据集约 4.6K 训练图、11 类。与 VOC-2012 物体分类类似，用 mAP 衡量。考虑两种训练设定：(i) 在整图上算 ConvNet 特征、忽略给定包围框；(ii) 分别在整图与给定包围框上算特征再堆叠得到最终表征。结果与其他方法比较见表 12。

Method VOC-2012 (mean AP)
(Oquab et al. 2014) 70.2∗
(Gkioxari et al. 2014) 73.6
(Hoai 2014) 76.3
VGG Net-D & Net-E, image-only 79.2
VGG Net-D & Net-E, image and bounding box 84.0

Table 12: Comparison with the state of the art in single-image action classification on VOC-2012. Our models are denoted as “VGG”. Results marked with * were achieved using ConvNets pre-trained on the extended ILSVRC dataset (1512 classes).

Method VOC-2012 (mean AP)
(Oquab et al. 2014) 70.2∗
(Gkioxari et al. 2014) 73.6
(Hoai 2014) 76.3
VGG Net-D & Net-E, image-only 79.2
VGG Net-D & Net-E, image and bounding box 84.0

表 12：VOC-2012 单图动作分类与当时最优的比较。我们的模型记为 “VGG”。标 * 的结果使用在扩展 ILSVRC（1512 类）上预训练的 ConvNet。

Our representation achieves the state of art on the VOC action classification task even without using the provided bounding boxes, and the results are further improved when using both images and bounding boxes. Unlike other approaches, we did not incorporate any task-specific heuristics, but relied on the representation power of very deep convolutional features.

即便不使用给定包围框，我们的表征也在 VOC 动作分类上达到当时最优；同时使用图像与包围框时结果进一步提升。与其他方法不同，我们未加入任何任务特定启发式，只依赖极深卷积特征的表征能力。

Other Recognition Tasks.

Since the public release of our models, they have been actively used by the research community for a wide range of image recognition tasks, consistently outperforming more shallow representations. For instance, Girshick et al. 2014 achieve the state of the object detection results by replacing the ConvNet of Krizhevsky et al. 2012 with our 16-layer model. Similar gains over a more shallow architecture of Krizhevsky et al. 2012 have been observed in semantic segmentation (Long et al. 2014), image caption generation (Kiros et al. 2014; Karpathy & Fei-Fei 2014), texture and material recognition (Cimpoi et al. 2014; Bell et al. 2014).

其他识别任务

模型公开发布后，研究社区已积极将其用于多种图像识别任务，并持续优于较浅表征。例如 Girshick et al. 2014 把 Krizhevsky et al. 2012 的 ConvNet 换成我们的 16 层模型，取得当时最优物体检测结果。语义分割（Long et al. 2014）、图像描述生成（Kiros et al. 2014；Karpathy & Fei-Fei 2014）、纹理与材料识别（Cimpoi et al. 2014；Bell et al. 2014）等任务上，相对 Krizhevsky et al. 2012 的较浅结构也观察到类似增益。

Appendix C Paper Revisions

附录 C 论文修订

Here we present the list of major paper revisions, outlining the substantial changes for the convenience of the reader.

此处列出主要论文修订，概述实质改动，方便读者对照。

v1 Initial version. Presents the experiments carried out before the ILSVRC submission.

v1 初始版本。给出提交 ILSVRC 前完成的实验。

v2 Adds post-submission ILSVRC experiments with training set augmentation using scale jittering, which improves the performance.

v2 加入提交后的 ILSVRC 实验：用尺度抖动做训练集增广，提升了表现。

v3 Adds generalisation experiments (Appendix B) on PASCAL VOC and Caltech image classification datasets. The models used for these experiments are publicly available.

v3 加入附录 B 在 PASCAL VOC 与 Caltech 图像分类数据集上的泛化实验。所用模型已公开发布。

v4 The paper is converted to ICLR-2015 submission format. Also adds experiments with multiple crops for classification.

v4 转为 ICLR-2015 投稿格式。并加入分类的多裁剪实验。

v6 Camera-ready ICLR-2015 conference paper. Adds a comparison of the net B with a shallow net and the results on PASCAL VOC action classification benchmark.

v6 ICLR-2015 会议终稿。加入网络 B 与浅网的比较，以及 PASCAL VOC 动作分类基准结果。