Feature Pyramid Networks for Object Detection

Tsung-Yi Lin Affiliation: Facebook AI Research (FAIR) Affiliation: Cornell University and Cornell Tech    Piotr Dollár Affiliation: Facebook AI Research (FAIR)    Ross Girshick Affiliation: Facebook AI Research (FAIR)    Kaiming He Affiliation: Facebook AI Research (FAIR)    Bharath Hariharan Affiliation: Facebook AI Research (FAIR)    Serge Belongie Affiliation: Cornell University and Cornell Tech

Abstract

摘要

Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But recent deep learning object detectors have avoided pyramid representations, in part because they are compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using FPN in a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 6 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

特征金字塔是识别系统中检测不同尺度物体的基本组件。但近年来的深度学习目标检测器往往回避金字塔表示，部分原因是它们在计算与内存上开销很大。本文利用深度卷积网络固有的多尺度金字塔层级结构，以很小的额外代价构建特征金字塔。我们设计了一种带横向连接的自顶向下架构，在所有尺度上生成具有高层语义的特征图。这一架构称为特征金字塔网络（Feature Pyramid Network，FPN），作为通用特征提取器在多种应用中带来显著提升。把 FPN 接到基本的 Faster R-CNN 系统中，我们的方法在 COCO 检测基准上取得了当时最优的单模型结果，无需花哨技巧，超过包括 COCO 2016 挑战赛优胜者在内的所有既有单模型条目。此外，该方法在 GPU 上可达约 6 FPS，因而是多尺度目标检测的一种实用且准确的方案。代码将公开。

1 Introduction

1 引言

Recognizing objects at vastly different scales is a fundamental challenge in computer vision. Feature pyramids built upon image pyramids (for short we call these featurized image pyramids) form the basis of a standard solution [1] (Fig. 1(a)). These pyramids are scale-invariant in the sense that an object’s scale change is offset by shifting its level in the pyramid. Intuitively, this property enables a model to detect objects across a large range of scales by scanning the model over both positions and pyramid levels.

在尺度相差极大的情况下识别物体，是计算机视觉的基本难题。建立在图像金字塔之上的特征金字塔（简称特征化图像金字塔）构成了标准解法的基础 [1]（图 1(a)）。这类金字塔具有尺度不变性：物体尺度的变化可由其在金字塔中层级的移动来抵消。直观上，这一性质使模型既能扫过位置、也能扫过金字塔层级，从而在很大尺度范围内检测物体。

Featurized image pyramids were heavily used in the era of hand-engineered features [5, 25]. They were so critical that object detectors like DPM [7] required dense scale sampling to achieve good results (e.g., 10 scales per octave). For recognition tasks, engineered features have largely been replaced with features computed by deep convolutional networks (ConvNets) [19, 20]. Aside from being capable of representing higher-level semantics, ConvNets are also more robust to variance in scale and thus facilitate recognition from features computed on a single input scale [15, 11, 29] (Fig. 1(b)). But even with this robustness, pyramids are still needed to get the most accurate results. All recent top entries in the ImageNet [33] and COCO [21] detection challenges use multi-scale testing on featurized image pyramids (e.g., [16, 35]). The principle advantage of featurizing each level of an image pyramid is that it produces a multi-scale feature representation in which all levels are semantically strong, including the high-resolution levels.

在手工特征时代，特征化图像金字塔被大量使用 [5, 25]。它们如此关键，以至于像 DPM [7] 这样的检测器需要稠密的尺度采样才能取得好结果（例如每倍频程 10 个尺度）。在识别任务中，工程特征已大多被深度卷积网络（ConvNet）计算的特征所取代 [19, 20]。ConvNet 不仅能表达更高层语义，对尺度变化也更稳健，因而便于在单一输入尺度上计算特征并完成识别 [15, 11, 29]（图 1(b)）。即便如此，要拿到最准的结果仍然需要金字塔。ImageNet [33] 与 COCO [21] 检测挑战中近期的顶尖条目都在特征化图像金字塔上做多尺度测试（例如 [16, 35]）。对图像金字塔每一层做特征化的主要优点是：得到的多尺度特征表示中，所有层级在语义上都足够强，包括高分辨率层级。

Figure 1: (a) Using an image pyramid to build a feature pyramid. Features are computed on each of the image scales independently, which is slow. (b) Recent detection systems have opted to use only single scale features for faster detection. (c) An alternative is to reuse the pyramidal feature hierarchy computed by a ConvNet as if it were a featurized image pyramid. (d) Our proposed Feature Pyramid Network (FPN) is fast like (b) and (c), but more accurate. In this figure, feature maps are indicate by blue outlines and thicker outlines denote semantically stronger features.

图 1：(a) 用图像金字塔构建特征金字塔。在每个图像尺度上独立计算特征，速度慢。(b) 近期检测系统为求更快，往往只用单尺度特征。(c) 另一种做法是复用 ConvNet 算出的金字塔特征层级，把它当作特征化图像金字塔。(d) 我们提出的特征金字塔网络（FPN）像 (b)(c) 一样快，但更准。图中特征图用蓝色轮廓表示，更粗的轮廓表示语义更强的特征。

Nevertheless, featurizing each level of an image pyramid has obvious limitations. Inference time increases considerably (e.g., by four times [11]), making this approach impractical for real applications. Moreover, training deep networks end-to-end on an image pyramid is infeasible in terms of memory, and so, if exploited, image pyramids are used only at test time [15, 11, 16, 35], which creates an inconsistency between train/test-time inference. For these reasons, Fast and Faster R-CNN [11, 29] opt to not use featurized image pyramids under default settings.

然而，对图像金字塔每一层做特征化有明显局限。推理时间显著增加（例如约四倍 [11]），使该方法难以用于实际应用。此外，在图像金字塔上端到端训练深度网络在内存上不可行，因此即便使用，图像金字塔也往往只在测试时启用 [15, 11, 16, 35]，造成训练/测试推理不一致。基于这些原因，Fast 与 Faster R-CNN [11, 29] 在默认设置下选择不使用特征化图像金字塔。

However, image pyramids are not the only way to compute a multi-scale feature representation. A deep ConvNet computes a feature hierarchy layer by layer, and with subsampling layers the feature hierarchy has an inherent multi-scale, pyramidal shape. This in-network feature hierarchy produces feature maps of different spatial resolutions, but introduces large semantic gaps caused by different depths. The high-resolution maps have low-level features that harm their representational capacity for object recognition.

不过，图像金字塔并非计算多尺度特征表示的唯一途径。深度 ConvNet 逐层计算特征层级，再经下采样层，该层级天然具有多尺度的金字塔形状。这种网络内特征层级产生不同空间分辨率的特征图，但因深度不同引入了很大的语义鸿沟。高分辨率图携带的是低层特征，损害了它们用于物体识别的表征能力。

The Single Shot Detector (SSD) [22] is one of the first attempts at using a ConvNet’s pyramidal feature hierarchy as if it were a featurized image pyramid (Fig. 1(c)). Ideally, the SSD-style pyramid would reuse the multi-scale feature maps from different layers computed in the forward pass and thus come free of cost. But to avoid using low-level features SSD foregoes reusing already computed layers and instead builds the pyramid starting from high up in the network (e.g., conv4_3 of VGG nets [36]) and then by adding several new layers. Thus it misses the opportunity to reuse the higher-resolution maps of the feature hierarchy. We show that these are important for detecting small objects.

单次检测器 SSD [22] 是最早尝试把 ConvNet 的金字塔特征层级当作特征化图像金字塔来用的方法之一（图 1(c)）。理想情况下，SSD 风格的金字塔应复用前向传播中不同层已算出的多尺度特征图，从而几乎零成本。但为避免使用低层特征，SSD 放弃复用已计算层，而是从网络较高处开始建金字塔（例如 VGG 的 conv4_3 [36]），再额外添加若干新层。于是它错失了复用特征层级中更高分辨率图的机会。我们表明，这些高分辨率图对检测小物体很重要。

The goal of this paper is to naturally leverage the pyramidal shape of a ConvNet’s feature hierarchy while creating a feature pyramid that has strong semantics at all scales. To achieve this goal, we rely on an architecture that combines low-resolution, semantically strong features with high-resolution, semantically weak features via a top-down pathway and lateral connections (Fig. 1(d)). The result is a feature pyramid that has rich semantics at all levels and is built quickly from a single input image scale. In other words, we show how to create in-network feature pyramids that can be used to replace featurized image pyramids without sacrificing representational power, speed, or memory.

本文的目标是自然利用 ConvNet 特征层级的金字塔形状，同时构建在所有尺度上都具有强语义的特征金字塔。为此，我们依赖一种架构：通过自顶向下通路与横向连接，把低分辨率但语义强的特征与高分辨率但语义弱的特征结合起来（图 1(d)）。结果是一个在所有层级都具有丰富语义、且可由单一输入图像尺度快速建成的特征金字塔。换言之，我们展示如何创建可替代特征化图像金字塔的网络内特征金字塔，而不牺牲表征能力、速度或内存。

Similar architectures adopting top-down and skip connections are popular in recent research [28, 17, 8, 26]. Their goals are to produce a single high-level feature map of a fine resolution on which the predictions are to be made (Fig. 2 top). On the contrary, our method leverages the architecture as a feature pyramid where predictions (e.g., object detections) are independently made on each level (Fig. 2 bottom). Our model echoes a featurized image pyramid, which has not been explored in these works.

采用自顶向下与跳跃连接的类似架构在近期研究中很常见 [28, 17, 8, 26]。它们的目标是生成一张精细分辨率的高层特征图，并在其上做预测（图 2 上）。相反，我们的方法把该架构用作特征金字塔，预测（例如目标检测）在每一层上独立进行（图 2 下）。我们的模型呼应了特征化图像金字塔，而这在上述工作中尚未被探索。

We evaluate our method, called a Feature Pyramid Network (FPN), in various systems for detection and segmentation [11, 29, 27]. Without bells and whistles, we report a state-of-the-art single-model result on the challenging COCO detection benchmark [21] simply based on FPN and a basic Faster R-CNN detector [29], surpassing all existing heavily-engineered single-model entries of competition winners. In ablation experiments, we find that for bounding box proposals, FPN significantly increases the Average Recall (AR) by 8.0 points; for object detection, it improves the COCO-style Average Precision (AP) by 2.3 points and PASCAL-style AP by 3.8 points, over a strong single-scale baseline of Faster R-CNN on ResNets [16]. Our method is also easily extended to mask proposals and improves both instance segmentation AR and speed over state-of-the-art methods that heavily depend on image pyramids.

我们在多种检测与分割系统中评估称为特征金字塔网络（FPN）的方法 [11, 29, 27]。无需花哨技巧，仅基于 FPN 与基本的 Faster R-CNN 检测器 [29]，我们就在具有挑战性的 COCO 检测基准 [21] 上报告了当时最优的单模型结果，超过竞赛优胜者中所有重度工程化的单模型条目。消融实验中，我们发现：对边界框提案，FPN 把平均召回（AR）显著提高 8.0 点；对目标检测，相对基于 ResNet [16] 的强单尺度 Faster R-CNN 基线，COCO 风格平均精度（AP）提高 2.3 点，PASCAL 风格 AP 提高 3.8 点。我们的方法也易于扩展到掩码提案，并在实例分割 AR 与速度上超过严重依赖图像金字塔的既有最优方法。

In addition, our pyramid structure can be trained end-to-end with all scales and is used consistently at train/test time, which would be memory-infeasible using image pyramids. As a result, FPNs are able to achieve higher accuracy than all existing state-of-the-art methods. Moreover, this improvement is achieved without increasing testing time over the single-scale baseline. We believe these advances will facilitate future research and applications. Our code will be made publicly available.

此外，我们的金字塔结构可在所有尺度上端到端训练，并在训练/测试时一致使用；若换用图像金字塔，这在内存上不可行。因此 FPN 能够取得高于所有既有最优方法的精度。而且，这一提升并未相对单尺度基线增加测试时间。我们相信这些进展将便利未来的研究与应用。代码将公开。

Figure 2: Top: a top-down architecture with skip connections, where predictions are made on the finest level (e.g., [28]). Bottom: our model that has a similar structure but leverages it as a feature pyramid, with predictions made independently at all levels.

图 2：上：带跳跃连接的自顶向下架构，预测在最精细层级上做出（例如 [28]）。下：我们的模型结构类似，但把它用作特征金字塔，在所有层级上独立做预测。

2 Related Work

2 相关工作

Hand-engineered features and early neural networks.

SIFT features [25] were originally extracted at scale-space extrema and used for feature point matching. HOG features [5], and later SIFT features as well, were computed densely over entire image pyramids. These HOG and SIFT pyramids have been used in numerous works for image classification, object detection, human pose estimation, and more. There has also been significant interest in computing featurized image pyramids quickly. Dollár et al. [6] demonstrated fast pyramid computation by first computing a sparsely sampled (in scale) pyramid and then interpolating missing levels. Before HOG and SIFT, early work on face detection with ConvNets [38, 32] computed shallow networks over image pyramids to detect faces across scales.

手工特征与早期神经网络。

SIFT 特征 [25] 最初在尺度空间极值处提取，用于特征点匹配。HOG 特征 [5]，以及后来的 SIFT，都在整幅图像金字塔上稠密计算。这些 HOG 与 SIFT 金字塔被大量用于图像分类、目标检测、人体姿态估计等。快速计算特征化图像金字塔也受到关注。Dollár 等人 [6] 先计算尺度上稀疏采样的金字塔，再插值缺失层级，从而加速。在 HOG 与 SIFT 之前，早期基于 ConvNet 的人脸检测工作 [38, 32] 在图像金字塔上运行浅层网络，以跨尺度检测人脸。

Deep ConvNet object detectors.

With the development of modern deep ConvNets [19], object detectors like OverFeat [34] and R-CNN [12] showed dramatic improvements in accuracy. OverFeat adopted a strategy similar to early neural network face detectors by applying a ConvNet as a sliding window detector on an image pyramid. R-CNN adopted a region proposal-based strategy [37] in which each proposal was scale-normalized before classifying with a ConvNet. SPPnet [15] demonstrated that such region-based detectors could be applied much more efficiently on feature maps extracted on a single image scale. Recent and more accurate detection methods like Fast R-CNN [11] and Faster R-CNN [29] advocate using features computed from a single scale, because it offers a good trade-off between accuracy and speed. Multi-scale detection, however, still performs better, especially for small objects.

深度 ConvNet 目标检测器。

随着现代深度 ConvNet [19] 的发展，OverFeat [34] 与 R-CNN [12] 等检测器在精度上大幅提升。OverFeat 采取与早期神经网络人脸检测器类似的策略，把 ConvNet 作为滑动窗口检测器作用在图像金字塔上。R-CNN 采用基于区域提案的策略 [37]，每个提案先尺度归一化再用 ConvNet 分类。SPPnet [15] 表明，这类基于区域的检测器可以更高效地作用在单尺度图像提取的特征图上。近期更准确的检测方法如 Fast R-CNN [11] 与 Faster R-CNN [29] 主张使用单尺度计算的特征，因为这在精度与速度之间提供了良好折中。然而多尺度检测仍然更好，尤其对小物体。

Methods using multiple layers.

A number of recent approaches improve detection and segmentation by using different layers in a ConvNet. FCN [24] sums partial scores for each category over multiple scales to compute semantic segmentations. Hypercolumns [13] uses a similar method for object instance segmentation. Several other approaches (HyperNet [18], ParseNet [23], and ION [2]) concatenate features of multiple layers before computing predictions, which is equivalent to summing transformed features. SSD [22] and MS-CNN [3] predict objects at multiple layers of the feature hierarchy without combining features or scores.

使用多层的方法。

一些近期方法通过使用 ConvNet 中的不同层来改进检测与分割。FCN [24] 在多个尺度上对各类别的部分分数求和以计算语义分割。Hypercolumns [13] 对物体实例分割使用类似方法。另一些方法（HyperNet [18]、ParseNet [23] 与 ION [2]）在预测前拼接多层特征，等价于对变换后的特征求和。SSD [22] 与 MS-CNN [3] 在特征层级的多层上预测物体，而不融合特征或分数。

There are recent methods exploiting lateral/skip connections that associate low-level feature maps across resolutions and semantic levels, including U-Net [31] and SharpMask [28] for segmentation, Recombinator networks [17] for face detection, and Stacked Hourglass networks [26] for keypoint estimation. Ghiasi et al. [8] present a Laplacian pyramid presentation for FCNs to progressively refine segmentation. Although these methods adopt architectures with pyramidal shapes, they are unlike featurized image pyramids [5, 7, 34] where predictions are made independently at all levels, see Fig. 2. In fact, for the pyramidal architecture in Fig. 2 (top), image pyramids are still needed to recognize objects across multiple scales [28].

也有近期方法利用横向/跳跃连接，把低层特征图跨分辨率与语义层级关联起来，包括用于分割的 U-Net [31] 与 SharpMask [28]、用于人脸检测的 Recombinator 网络 [17]，以及用于关键点估计的 Stacked Hourglass 网络 [26]。Ghiasi 等人 [8] 为 FCN 提出拉普拉斯金字塔表示以逐步细化分割。尽管这些方法采用金字塔形状的架构，它们不同于在所有层级上独立做预测的特征化图像金字塔 [5, 7, 34]，见图 2。事实上，对图 2（上）的金字塔架构而言，仍需要图像金字塔才能跨多尺度识别物体 [28]。

3 Feature Pyramid Networks

3 特征金字塔网络

Our goal is to leverage a ConvNet’s pyramidal feature hierarchy, which has semantics from low to high levels, and build a feature pyramid with high-level semantics throughout. The resulting Feature Pyramid Network is general-purpose and in this paper we focus on sliding window proposers (Region Proposal Network, RPN for short) [29] and region-based detectors (Fast R-CNN) [11]. We also generalize FPNs to instance segmentation proposals in Sec. 6.

我们的目标是利用 ConvNet 从低到高具有不同语义的金字塔特征层级，构建贯穿各层都具有高层语义的特征金字塔。由此得到的特征金字塔网络是通用的；本文聚焦于滑动窗口提案器（区域提案网络，简称 RPN）[29] 与基于区域的检测器（Fast R-CNN）[11]。我们还在第 6 节把 FPN 推广到实例分割提案。

Our method takes a single-scale image of an arbitrary size as input, and outputs proportionally sized feature maps at multiple levels, in a fully convolutional fashion. This process is independent of the backbone convolutional architectures (e.g., [19, 36, 16]), and in this paper we present results using ResNets [16]. The construction of our pyramid involves a bottom-up pathway, a top-down pathway, and lateral connections, as introduced in the following.

我们的方法以任意尺寸的单尺度图像为输入，以全卷积方式输出多个层级、尺寸成比例的特征图。该过程与骨干卷积架构无关（例如 [19, 36, 16]），本文用 ResNet [16] 给出结果。金字塔的构建包含自底向上通路、自顶向下通路与横向连接，如下所述。

Bottom-up pathway.

The bottom-up pathway is the feedforward computation of the backbone ConvNet, which computes a feature hierarchy consisting of feature maps at several scales with a scaling step of 2. There are often many layers producing output maps of the same size and we say these layers are in the same network stage. For our feature pyramid, we define one pyramid level for each stage. We choose the output of the last layer of each stage as our reference set of feature maps, which we will enrich to create our pyramid. This choice is natural since the deepest layer of each stage should have the strongest features.

自底向上通路。

自底向上通路是骨干 ConvNet 的前馈计算，它计算由若干尺度特征图组成的特征层级，尺度步长为 2。往往有许多层产生相同尺寸的输出图，我们称这些层属于同一网络阶段。对我们的特征金字塔，我们为每个阶段定义一个金字塔层级。我们选择每个阶段最后一层的输出作为特征图参考集，再加以丰富以形成金字塔。这一选择很自然，因为每个阶段最深的层应具有最强的特征。

Specifically, for ResNets [16] we use the feature activations output by each stage’s last residual block. We denote the output of these last residual blocks as $\{C_{2},C_{3},C_{4},C_{5}\}$ for conv2, conv3, conv4, and conv5 outputs, and note that they have strides of {4, 8, 16, 32} pixels with respect to the input image. We do not include conv1 into the pyramid due to its large memory footprint.

具体而言，对 ResNet [16]，我们使用每个阶段最后一个残差块输出的特征激活。将这些最后残差块的输出记为 $\{C_{2},C_{3},C_{4},C_{5}\}$，分别对应 conv2、conv3、conv4 与 conv5，相对输入图像的步长为 {4, 8, 16, 32} 像素。由于内存占用大，我们不把 conv1 纳入金字塔。

Top-down pathway and lateral connections.

The top-down pathway hallucinates higher resolution features by upsampling spatially coarser, but semantically stronger, feature maps from higher pyramid levels. These features are then enhanced with features from the bottom-up pathway via lateral connections. Each lateral connection merges feature maps of the same spatial size from the bottom-up pathway and the top-down pathway. The bottom-up feature map is of lower-level semantics, but its activations are more accurately localized as it was subsampled fewer times.

自顶向下通路与横向连接。

自顶向下通路通过对空间上更粗、但语义更强的高层特征图上采样，生成更高分辨率的特征。这些特征再经横向连接，用自底向上通路的特征加以增强。每条横向连接合并来自自底向上与自顶向下、空间尺寸相同的特征图。自底向上特征图语义层次较低，但由于下采样次数更少，其激活的定位更准确。

Figure 3: A building block illustrating the lateral connection and the top-down pathway, merged by addition.

图 3：展示横向连接与自顶向下通路、并通过加法合并的基本构建块。

Fig. 3 shows the building block that constructs our top-down feature maps. With a coarser-resolution feature map, we upsample the spatial resolution by a factor of 2 (using nearest neighbor upsampling for simplicity). The upsampled map is then merged with the corresponding bottom-up map (which undergoes a 1$\times$1 convolutional layer to reduce channel dimensions) by element-wise addition. This process is iterated until the finest resolution map is generated. To start the iteration, we simply attach a 1$\times$1 convolutional layer on $C_{5}$ to produce the coarsest resolution map. Finally, we append a $3\times 3$ convolution on each merged map to generate the final feature map, which is to reduce the aliasing effect of upsampling. This final set of feature maps is called $\{P_{2},P_{3},P_{4},P_{5}\}$, corresponding to $\{C_{2},C_{3},C_{4},C_{5}\}$ that are respectively of the same spatial sizes.

图 3 给出构建自顶向下特征图的基本块。对较粗分辨率的特征图，我们将空间分辨率上采样 2 倍（为简单起见使用最近邻上采样）。上采样后的图再与对应的自底向上图（先经 $1\times1$ 卷积降维）逐元素相加合并。该过程迭代直至生成最精细分辨率图。迭代起点是：在 $C_{5}$ 上接一个 $1\times1$ 卷积以产生最粗分辨率图。最后，在每个合并图上接一个 $3\times3$ 卷积生成最终特征图，以减轻上采样的混叠效应。这一最终特征图集合称为 $\{P_{2},P_{3},P_{4},P_{5}\}$，分别与 $\{C_{2},C_{3},C_{4},C_{5}\}$ 空间尺寸相同。

Because all levels of the pyramid use shared classifiers/regressors as in a traditional featurized image pyramid, we fix the feature dimension (numbers of channels, denoted as $d$) in all the feature maps. We set $d=256$ in this paper and thus all extra convolutional layers have 256-channel outputs. There are no non-linearities in these extra layers, which we have empirically found to have minor impacts.

因为金字塔所有层级像传统特征化图像金字塔一样共享分类器/回归器，我们在所有特征图上固定特征维度（通道数，记为 $d$）。本文设 $d=256$，因而所有额外卷积层输出均为 256 通道。这些额外层中没有非线性，我们经验上发现其影响很小。

Simplicity is central to our design and we have found that our model is robust to many design choices. We have experimented with more sophisticated blocks (e.g., using multi-layer residual blocks [16] as the connections) and observed marginally better results. Designing better connection modules is not the focus of this paper, so we opt for the simple design described above.

简洁是我们设计的核心，并且我们发现模型对许多设计选择具有稳健性。我们试验过更复杂的块（例如用多层残差块 [16] 作为连接），只观察到边际更好的结果。设计更好的连接模块并非本文重点，因此我们选择上述简单设计。

4 Applications

4 应用

Our method is a generic solution for building feature pyramids inside deep ConvNets. In the following we adopt our method in RPN [29] for bounding box proposal generation and in Fast R-CNN [11] for object detection. To demonstrate the simplicity and effectiveness of our method, we make minimal modifications to the original systems of [29, 11] when adapting them to our feature pyramid.

我们的方法是在深度 ConvNet 内部构建特征金字塔的通用方案。下面我们把它接到 RPN [29] 以生成边界框提案，并接到 Fast R-CNN [11] 做目标检测。为展示方法的简洁与有效，在适配到我们的特征金字塔时，我们对 [29, 11] 的原系统只做最少修改。

4.1 Feature Pyramid Networks for RPN

4.1 用于 RPN 的特征金字塔网络

RPN [29] is a sliding-window class-agnostic object detector. In the original RPN design, a small subnetwork is evaluated on dense 3$\times$3 sliding windows, on top of a single-scale convolutional feature map, performing object/non-object binary classification and bounding box regression. This is realized by a 3$\times$3 convolutional layer followed by two sibling 1$\times$1 convolutions for classification and regression, which we refer to as a network head. The object/non-object criterion and bounding box regression target are defined with respect to a set of reference boxes called anchors [29]. The anchors are of multiple pre-defined scales and aspect ratios in order to cover objects of different shapes.

RPN [29] 是一种滑动窗口、类别无关的目标检测器。在原始 RPN 设计中，一个小型子网络在单尺度卷积特征图上、对稠密的 $3\times3$ 滑动窗口求值，完成物体/非物体二分类与边界框回归。具体由一个 $3\times3$ 卷积层，再接两个兄弟 $1\times1$ 卷积分别做分类与回归实现，我们称之为网络头。物体/非物体判据与边界框回归目标相对于一组称为锚的参考框定义 [29]。锚具有多种预定义尺度与长宽比，以覆盖不同形状的物体。

We adapt RPN by replacing the single-scale feature map with our FPN. We attach a head of the same design (3$\times$3 conv and two sibling 1$\times$1 convs) to each level on our feature pyramid. Because the head slides densely over all locations in all pyramid levels, it is not necessary to have multi-scale anchors on a specific level. Instead, we assign anchors of a single scale to each level. Formally, we define the anchors to have areas of $\{32^{2},64^{2},128^{2},256^{2},512^{2}\}$ pixels on $\{P_{2},P_{3},P_{4},P_{5},P_{6}\}$ respectively.[^1] As in [29] we also use anchors of multiple aspect ratios $\{1$:$2,$ $1$:$1$, $2$:$1\}$ at each level. So in total there are 15 anchors over the pyramid.

我们用 FPN 替换单尺度特征图来适配 RPN。在特征金字塔的每一层上接相同设计的头（$3\times3$ 卷积与两个兄弟 $1\times1$ 卷积）。由于头在所有金字塔层级的所有位置上稠密滑动，不必在某一层上再放多尺度锚。相反，我们给每层分配单一尺度的锚。形式上，我们定义锚在 $\{P_{2},P_{3},P_{4},P_{5},P_{6}\}$ 上的面积分别为 $\{32^{2},64^{2},128^{2},256^{2},512^{2}\}$ 像素。[^1] 与 [29] 一样，每层还使用多种长宽比 $\{1$:$2,$ $1$:$1$, $2$:$1\}$。因此整个金字塔上共有 15 种锚。

We assign training labels to the anchors based on their Intersection-over-Union (IoU) ratios with ground-truth bounding boxes as in [29]. Formally, an anchor is assigned a positive label if it has the highest IoU for a given ground-truth box or an IoU over 0.7 with any ground-truth box, and a negative label if it has IoU lower than 0.3 for all ground-truth boxes. Note that scales of ground-truth boxes are not explicitly used to assign them to the levels of the pyramid; instead, ground-truth boxes are associated with anchors, which have been assigned to pyramid levels. As such, we introduce no extra rules in addition to those in [29].

我们按锚与真值框的交并比（IoU）分配训练标签，做法同 [29]。形式上，若某锚对给定真值框具有最高 IoU，或与任一真值框 IoU 超过 0.7，则标为正；若对所有真值框 IoU 都低于 0.3，则标为负。注意，真值框的尺度并不被显式用于分配到金字塔层级；相反，真值框与已分配到各层的锚相关联。因此，我们除 [29] 中的规则外不引入额外规则。

We note that the parameters of the heads are shared across all feature pyramid levels; we have also evaluated the alternative without sharing parameters and observed similar accuracy. The good performance of sharing parameters indicates that all levels of our pyramid share similar semantic levels. This advantage is analogous to that of using a featurized image pyramid, where a common head classifier can be applied to features computed at any image scale.

我们指出，头的参数在所有特征金字塔层级之间共享；我们也评估了不共享参数的替代方案，观察到相近的精度。参数共享表现良好，表明我们金字塔的所有层级具有相似的语义水平。这一优势类似于使用特征化图像金字塔：同一头部分类器可作用在任意图像尺度上算出的特征。

With the above adaptations, RPN can be naturally trained and tested with our FPN, in the same fashion as in [29]. We elaborate on the implementation details in the experiments.

经过上述适配，RPN 可以自然地与我们的 FPN 一起训练与测试，方式与 [29] 相同。实现细节在实验部分详述。

4.2 Feature Pyramid Networks for Fast R-CNN

4.2 用于 Fast R-CNN 的特征金字塔网络

Fast R-CNN [11] is a region-based object detector in which Region-of-Interest (RoI) pooling is used to extract features. Fast R-CNN is most commonly performed on a single-scale feature map. To use it with our FPN, we need to assign RoIs of different scales to the pyramid levels.

Fast R-CNN [11] 是基于区域的目标检测器，用感兴趣区域（RoI）池化提取特征。Fast R-CNN 通常在单尺度特征图上运行。要与我们的 FPN 一起使用，需要把不同尺度的 RoI 分配到金字塔各层。

We view our feature pyramid as if it were produced from an image pyramid. Thus we can adapt the assignment strategy of region-based detectors [15, 11] in the case when they are run on image pyramids. Formally, we assign an RoI of width $w$ and height $h$ (on the input image to the network) to the level $P_{k}$ of our feature pyramid by:

我们把特征金字塔视为如同由图像金字塔产生。因此可以沿用基于区域的检测器在图像金字塔上运行时的分配策略 [15, 11]。形式上，把宽为 $w$、高为 $h$（相对网络输入图像）的 RoI 分配到特征金字塔的 $P_{k}$ 层：

$$k=\lfloor k_{0}+\log_{2}(\sqrt{wh}/224)\rfloor. \tag{1}$$

Here 224 is the canonical ImageNet pre-training size, and $k_{0}$ is the target level on which an RoI with $w\times h=224^{2}$ should be mapped into. Analogous to the ResNet-based Faster R-CNN system [16] that uses $C_{4}$ as the single-scale feature map, we set $k_{0}$ to 4. Intuitively, Eqn. (1) means that if the RoI’s scale becomes smaller (say, 1/2 of 224), it should be mapped into a finer-resolution level (say, $k=3$).

这里 224 是规范的 ImageNet 预训练尺寸，$k_{0}$ 是面积 $w\times h=224^{2}$ 的 RoI 应映射到的目标层级。类比于以 $C_{4}$ 为单尺度特征图的基于 ResNet 的 Faster R-CNN 系统 [16]，我们设 $k_{0}$ 为 4。直观上，式 (1) 表示：若 RoI 尺度变小（例如变为 224 的 1/2），则应映射到更精细分辨率的层级（例如 $k=3$）。

We attach predictor heads (in Fast R-CNN the heads are class-specific classifiers and bounding box regressors) to all RoIs of all levels. Again, the heads all share parameters, regardless of their levels. In [16], a ResNet’s conv5 layers (a 9-layer deep subnetwork) are adopted as the head on top of the conv4 features, but our method has already harnessed conv5 to construct the feature pyramid. So unlike [16], we simply adopt RoI pooling to extract 7$\times$7 features, and attach two hidden 1,024-d fully-connected (fc) layers (each followed by ReLU) before the final classification and bounding box regression layers. These layers are randomly initialized, as there are no pre-trained fc layers available in ResNets. Note that compared to the standard conv5 head, our 2-fc MLP head is lighter weight and faster.

我们把预测头（在 Fast R-CNN 中头是类别特定的分类器与边界框回归器）接到所有层级的所有 RoI 上。同样，头无论层级如何都共享参数。在 [16] 中，ResNet 的 conv5 层（一个 9 层深的子网络）被用作 conv4 特征之上的头，但我们的方法已利用 conv5 来构建特征金字塔。因此与 [16] 不同，我们仅用 RoI 池化提取 $7\times7$ 特征，再接两个隐层为 1024 维的全连接（fc）层（各后接 ReLU），然后才是最终的分类与边界框回归层。这些层随机初始化，因为 ResNet 中没有预训练的 fc 层可用。注意，相对标准的 conv5 头，我们的 2-fc MLP 头更轻、更快。

Based on these adaptations, we can train and test Fast R-CNN on top of the feature pyramid. Implementation details are given in the experimental section.

基于这些适配，我们可以在特征金字塔上训练与测试 Fast R-CNN。实现细节见实验部分。

5 Experiments on Object Detection

5 目标检测实验

We perform experiments on the 80 category COCO detection dataset [21]. We train using the union of 80k train images and a 35k subset of val images (trainval35k [2]), and report ablations on a 5k subset of val images (minival). We also report final results on the standard test set (test-std) [21] which has no disclosed labels.

我们在 80 类 COCO 检测数据集 [21] 上实验。训练使用 80k 训练图像与 val 的 35k 子集之并（trainval35k [2]），在 val 的 5k 子集（minival）上报告消融。我们也在标准测试集（test-std）[21] 上报告最终结果，该集标签未公开。

As is common practice [12], all network backbones are pre-trained on the ImageNet1k classification set [33] and then fine-tuned on the detection dataset. We use the pre-trained ResNet-50 and ResNet-101 models that are publicly available.[^2] Our code is a reimplementation of py-faster-rcnn[^3] using Caffe2.[^4]

按常见做法 [12]，所有网络骨干先在 ImageNet1k 分类集 [33] 上预训练，再在检测数据集上微调。我们使用公开的预训练 ResNet-50 与 ResNet-101 模型。[^2] 代码是用 Caffe2 对 py-faster-rcnn[^3] 的重实现。[^4]

RPN	feature	# anchors	lateral?	top-down?	AR100	AR1k	AR${}^{1k}_{s}$	AR${}^{1k}_{m}$	AR${}^{1k}_{l}$
(a) baseline on conv4	$C_{4}$	47k			36.1	48.3	32.0	58.7	62.2
(b) baseline on conv5	$C_{5}$	12k			36.3	44.9	25.3	55.5	64.2
(c) FPN	$\{P_{k}\}$	200k	✓	✓	44.0	56.3	44.9	63.4	66.2
Ablation experiments follow:
(d) bottom-up pyramid	$\{P_{k}\}$	200k	✓		37.4	49.5	30.5	59.9	68.0
(e) top-down pyramid, w/o lateral	$\{P_{k}\}$	200k		✓	34.5	46.1	26.5	57.4	64.7
(f) only finest level	$P_{2}$	750k	✓	✓	38.4	51.3	35.1	59.7	67.6

Fast R-CNN	proposals	feature	head	lateral?	top-down?	AP@0.5	AP	APs	APm	APl
(a) baseline on conv4	RPN, $\{P_{k}\}$	$C_{4}$	conv5			54.7	31.9	15.7	36.5	45.5
(b) baseline on conv5	RPN, $\{P_{k}\}$	$C_{5}$	2fc			52.9	28.8	11.9	32.4	43.4
(c) FPN	RPN, $\{P_{k}\}$	$\{P_{k}\}$	2fc	✓	✓	56.9	33.9	17.8	37.7	45.8
Ablation experiments follow:
(d) bottom-up pyramid	RPN, $\{P_{k}\}$	$\{P_{k}\}$	2fc	✓		44.9	24.9	10.9	24.4	38.5
(e) top-down pyramid, w/o lateral	RPN, $\{P_{k}\}$	$\{P_{k}\}$	2fc		✓	54.0	31.3	13.3	35.2	45.3
(f) only finest level	RPN, $\{P_{k}\}$	$P_{2}$	2fc	✓	✓	56.3	33.4	17.3	37.3	45.6

Faster R-CNN	proposals	feature	head	lateral?	top-down?	AP@0.5	AP	APs	APm	APl
(*) baseline from He et al. [16]†	RPN, $C_{4}$	$C_{4}$	conv5			47.3	26.3	-	-	-
(a) baseline on conv4	RPN, $C_{4}$	$C_{4}$	conv5			53.1	31.6	13.2	35.6	47.1
(b) baseline on conv5	RPN, $C_{5}$	$C_{5}$	2fc			51.7	28.0	9.6	31.9	43.1
(c) FPN	RPN, $\{P_{k}\}$	$\{P_{k}\}$	2fc	✓	✓	56.9	33.9	17.8	37.7	45.8

Table 1: Bounding box proposal results using RPN [29], evaluated on the COCO minival set. All models are trained on trainval35k. The columns “lateral” and “top-down” denote the presence of lateral and top-down connections, respectively. The column “feature” denotes the feature maps on which the heads are attached. All results are based on ResNet-50 and share the same hyper-parameters.

RPN	特征	# 锚	横向?	自顶向下?	AR100	AR1k	AR${}^{1k}_{s}$	AR${}^{1k}_{m}$	AR${}^{1k}_{l}$
(a) conv4 基线	$C_{4}$	47k			36.1	48.3	32.0	58.7	62.2
(b) conv5 基线	$C_{5}$	12k			36.3	44.9	25.3	55.5	64.2
(c) FPN	$\{P_{k}\}$	200k	✓	✓	44.0	56.3	44.9	63.4	66.2
后续为消融实验：
(d) 自底向上金字塔	$\{P_{k}\}$	200k	✓		37.4	49.5	30.5	59.9	68.0
(e) 自顶向下金字塔，无横向	$\{P_{k}\}$	200k		✓	34.5	46.1	26.5	57.4	64.7
(f) 仅最精细层	$P_{2}$	750k	✓	✓	38.4	51.3	35.1	59.7	67.6

Fast R-CNN	提案	特征	头	横向?	自顶向下?	AP@0.5	AP	APs	APm	APl
(a) conv4 基线	RPN, $\{P_{k}\}$	$C_{4}$	conv5			54.7	31.9	15.7	36.5	45.5
(b) conv5 基线	RPN, $\{P_{k}\}$	$C_{5}$	2fc			52.9	28.8	11.9	32.4	43.4
(c) FPN	RPN, $\{P_{k}\}$	$\{P_{k}\}$	2fc	✓	✓	56.9	33.9	17.8	37.7	45.8
后续为消融实验：
(d) 自底向上金字塔	RPN, $\{P_{k}\}$	$\{P_{k}\}$	2fc	✓		44.9	24.9	10.9	24.4	38.5
(e) 自顶向下金字塔，无横向	RPN, $\{P_{k}\}$	$\{P_{k}\}$	2fc		✓	54.0	31.3	13.3	35.2	45.3
(f) 仅最精细层	RPN, $\{P_{k}\}$	$P_{2}$	2fc	✓	✓	56.3	33.4	17.3	37.3	45.6

Faster R-CNN	提案	特征	头	横向?	自顶向下?	AP@0.5	AP	APs	APm	APl
(*) He 等人基线 [16]†	RPN, $C_{4}$	$C_{4}$	conv5			47.3	26.3	-	-	-
(a) conv4 基线	RPN, $C_{4}$	$C_{4}$	conv5			53.1	31.6	13.2	35.6	47.1
(b) conv5 基线	RPN, $C_{5}$	$C_{5}$	2fc			51.7	28.0	9.6	31.9	43.1
(c) FPN	RPN, $\{P_{k}\}$	$\{P_{k}\}$	2fc	✓	✓	56.9	33.9	17.8	37.7	45.8

表 1：使用 RPN [29] 的边界框提案结果，在 COCO minival 上评估。所有模型在 trainval35k 上训练。“横向”与“自顶向下”列分别表示是否存在横向与自顶向下连接。“特征”列表示头所接的特征图。所有结果基于 ResNet-50，并共享相同超参数。

5.1 Region Proposal with RPN

5.1 用 RPN 做区域提案

We evaluate the COCO-style Average Recall (AR) and AR on small, medium, and large objects (ARs, ARm, and ARl) following the definitions in [21]. We report results for 100 and 1000 proposals per images (AR100 and AR1k).

我们按 [21] 的定义评估 COCO 风格平均召回（AR）以及小、中、大物体上的 AR（ARs、ARm 与 ARl）。报告每图 100 与 1000 个提案的结果（AR100 与 AR1k）。

Implementation details.

All architectures in Table 3 are trained end-to-end. The input image is resized such that its shorter side has 800 pixels. We adopt synchronized SGD training on 8 GPUs. A mini-batch involves 2 images per GPU and 256 anchors per image. We use a weight decay of 0.0001 and a momentum of 0.9. The learning rate is 0.02 for the first 30k mini-batches and 0.002 for the next 10k. For all RPN experiments (including baselines), we include the anchor boxes that are outside the image for training, which is unlike [29] where these anchor boxes are ignored. Other implementation details are as in [29]. Training RPN with FPN on 8 GPUs takes about 8 hours on COCO.

实现细节。

表 3 中所有架构均端到端训练。输入图像缩放使其短边为 800 像素。我们在 8 块 GPU 上采用同步 SGD 训练。每个小批量每 GPU 2 张图、每图 256 个锚。权重衰减 0.0001，动量 0.9。学习率前 30k 小批量为 0.02，随后 10k 为 0.002。对所有 RPN 实验（含基线），训练时包含位于图像外的锚框，这与 [29] 忽略这些锚框不同。其余实现细节同 [29]。在 8 GPU 上用 FPN 训练 RPN 在 COCO 上约需 8 小时。

Comparisons with baselines.

For fair comparisons with original RPNs [29], we run two baselines (Table 3(a, b)) using the single-scale map of $C_{4}$ (the same as [16]) or $C_{5}$, both using the same hyper-parameters as ours, including using 5 scale anchors of $\{32^{2},64^{2},128^{2},256^{2},512^{2}\}$. Table 3 (b) shows no advantage over (a), indicating that a single higher-level feature map is not enough because there is a trade-off between coarser resolutions and stronger semantics.

与基线比较。

为与原始 RPN [29] 公平比较，我们运行两个基线（表 3(a, b)），分别使用 $C_{4}$（与 [16] 相同）或 $C_{5}$ 的单尺度特征图，并采用与我们相同的超参数，包括使用面积为 $\{32^{2},64^{2},128^{2},256^{2},512^{2}\}$ 的 5 种尺度锚。表 3(b) 相对 (a) 无优势，表明单一更高层特征图不够，因为更粗分辨率与更强语义之间存在折中。

Placing FPN in RPN improves AR1k to 56.3 (Table 3 (c)), which is 8.0 points increase over the single-scale RPN baseline (Table 3 (a)). In addition, the performance on small objects (AR${}^{1k}_{s}$) is boosted by a large margin of 12.9 points. Our pyramid representation greatly improves RPN’s robustness to object scale variation.

把 FPN 放入 RPN 后，AR1k 提升到 56.3（表 3(c)），相对单尺度 RPN 基线（表 3(a)）提高 8.0 点。此外，小物体上的表现（AR${}^{1k}_{s}$）大幅提升 12.9 点。我们的金字塔表示显著增强了 RPN 对物体尺度变化的稳健性。

How important is top-down enrichment?

Table 3(d) shows the results of our feature pyramid without the top-down pathway. With this modification, the 1$\times$1 lateral connections followed by 3$\times$3 convolutions are attached to the bottom-up pyramid. This architecture simulates the effect of reusing the pyramidal feature hierarchy (Fig. 1(b)).

自顶向下增强有多重要？

表 3(d) 给出去掉自顶向下通路后的特征金字塔结果。在此修改下，$1\times1$ 横向连接再接 $3\times3$ 卷积被接到自底向上金字塔上。该架构模拟了复用金字塔特征层级的效果（图 1(b)）。

The results in Table 3(d) are just on par with the RPN baseline and lag far behind ours. We conjecture that this is because there are large semantic gaps between different levels on the bottom-up pyramid (Fig. 1(b)), especially for very deep ResNets. We have also evaluated a variant of Table 3(d) without sharing the parameters of the heads, but observed similarly degraded performance. This issue cannot be simply remedied by level-specific heads.

表 3(d) 的结果仅与 RPN 基线持平，远落后于我们的方法。我们推测这是因为自底向上金字塔不同层级之间存在很大的语义鸿沟（图 1(b)），对很深的 ResNet 尤其如此。我们也评估了表 3(d) 的一个变体：头参数不共享，但观察到同样变差的表现。这一问题不能简单靠层级特定的头来补救。

How important are lateral connections?

Table 3(e) shows the ablation results of a top-down feature pyramid without the 1$\times$1 lateral connections. This top-down pyramid has strong semantic features and fine resolutions. But we argue that the locations of these features are not precise, because these maps have been downsampled and upsampled several times. More precise locations of features can be directly passed from the finer levels of the bottom-up maps via the lateral connections to the top-down maps. As a results, FPN has an AR1k score 10 points higher than Table 3(e).

横向连接有多重要？

表 3(e) 给出没有 $1\times1$ 横向连接的自顶向下特征金字塔消融结果。该自顶向下金字塔具有强语义特征与精细分辨率。但我们认为这些特征的位置并不精确，因为这些图已经过多次下采样与上采样。更精确的特征位置可通过横向连接，直接从自底向上图的更精细层级传到自顶向下图。因此，FPN 的 AR1k 比表 3(e) 高 10 点。

How important are pyramid representations?

Instead of resorting to pyramid representations, one can attach the head to the highest-resolution, strongly semantic feature maps of $P_{2}$ (i.e., the finest level in our pyramids). Similar to the single-scale baselines, we assign all anchors to the $P_{2}$ feature map. This variant (Table 3(f)) is better than the baseline but inferior to our approach. RPN is a sliding window detector with a fixed window size, so scanning over pyramid levels can increase its robustness to scale variance.

金字塔表示有多重要？

若不诉诸金字塔表示，可以把头接到 $P_{2}$ 这一最高分辨率、强语义的特征图上（即我们金字塔中最精细的层）。与单尺度基线类似，我们把所有锚都分配到 $P_{2}$ 特征图。该变体（表 3(f)）优于基线，但不如我们的方法。RPN 是固定窗口大小的滑动窗口检测器，因此在金字塔各层上扫描可增强其对尺度变化的稳健性。

In addition, we note that using $P_{2}$ alone leads to more anchors (750k, Table 3(f)) caused by its large spatial resolution. This result suggests that a larger number of anchors is not sufficient in itself to improve accuracy.

此外我们指出，仅使用 $P_{2}$ 会因其大空间分辨率导致更多锚（750k，表 3(f)）。这一结果表明，单纯增加锚的数量并不足以提升精度。

5.2 Object Detection with Fast/Faster R-CNN

5.2 用 Fast/Faster R-CNN 做目标检测

Next we investigate FPN for region-based (non-sliding window) detectors. We evaluate object detection by the COCO-style Average Precision (AP) and PASCAL-style AP (at a single IoU threshold of 0.5). We also report COCO AP on objects of small, medium, and large sizes (namely, APs, APm, and APl) following the definitions in [21].

接下来我们考察 FPN 用于基于区域（非滑动窗口）的检测器。我们用 COCO 风格平均精度（AP）与 PASCAL 风格 AP（单一 IoU 阈值 0.5）评估目标检测。我们也按 [21] 的定义报告小、中、大物体上的 COCO AP（即 APs、APm 与 APl）。

			image	test-dev	test-std
method	backbone	competition	pyramid	AP*@.5*	AP	APs	APm	APl	AP*@.5*	AP	APs	APm	APl
ours, Faster R-CNN on FPN	ResNet-101	-		59.1	36.2	18.2	39.0	48.2	58.5	35.8	17.5	38.7	47.8
Competition-winning single-model results follow:
G-RMI†	Inception-ResNet	2016		-	34.7	-	-	-	-	-	-	-	-
AttractioNet*‡* [10]	VGG16 + Wide ResNet*§*	2016	✓	53.4	35.7	15.6	38.0	52.7	52.9	35.3	14.7	37.6	51.9
Faster R-CNN +++ [16]	ResNet-101	2015	✓	55.7	34.9	15.6	38.7	50.9	-	-	-	-	-
Multipath [40] (on minival)	VGG-16	2015		49.6	31.5	-	-	-	-	-	-	-	-
ION‡ [2]	VGG-16	2015		53.4	31.2	12.8	32.9	45.2	52.9	30.7	11.8	32.8	44.8

Table 4: Comparisons of single-model results on the COCO detection benchmark. Some results were not available on the test-std set, so we also include the test-dev results (and for Multipath [40] on minival). †: http://image-net.org/challenges/talks/2016/GRMI-COCO-slidedeck.pdf. ‡: http://mscoco.org/dataset/#detections-leaderboard. §: This entry of AttractioNet [10] adopts VGG-16 for proposals and Wide ResNet [39] for object detection, so is not strictly a single-model result.

			图像	test-dev	test-std
方法	骨干	竞赛	金字塔	AP*@.5*	AP	APs	APm	APl	AP*@.5*	AP	APs	APm	APl
本文，Faster R-CNN on FPN	ResNet-101	-		59.1	36.2	18.2	39.0	48.2	58.5	35.8	17.5	38.7	47.8
后续为竞赛优胜单模型结果：
G-RMI†	Inception-ResNet	2016		-	34.7	-	-	-	-	-	-	-	-
AttractioNet*‡* [10]	VGG16 + Wide ResNet*§*	2016	✓	53.4	35.7	15.6	38.0	52.7	52.9	35.3	14.7	37.6	51.9
Faster R-CNN +++ [16]	ResNet-101	2015	✓	55.7	34.9	15.6	38.7	50.9	-	-	-	-	-
Multipath [40]（在 minival 上）	VGG-16	2015		49.6	31.5	-	-	-	-	-	-	-	-
ION‡ [2]	VGG-16	2015		53.4	31.2	12.8	32.9	45.2	52.9	30.7	11.8	32.8	44.8

表 4：COCO 检测基准上的单模型结果比较。部分结果在 test-std 上不可得，故同时列出 test-dev 结果（Multipath [40] 则在 minival 上）。†: http://image-net.org/challenges/talks/2016/GRMI-COCO-slidedeck.pdf。‡: http://mscoco.org/dataset/#detections-leaderboard。§: AttractioNet [10] 该条目用 VGG-16 做提案、Wide ResNet [39] 做检测，因而并非严格的单模型结果。

Implementation details.

The input image is resized such that its shorter side has 800 pixels. Synchronized SGD is used to train the model on 8 GPUs. Each mini-batch involves 2 image per GPU and 512 RoIs per image. We use a weight decay of 0.0001 and a momentum of 0.9. The learning rate is 0.02 for the first 60k mini-batches and 0.002 for the next 20k. We use 2000 RoIs per image for training and 1000 for testing. Training Fast R-CNN with FPN takes about 10 hours on the COCO dataset.

实现细节。

输入图像缩放使其短边为 800 像素。在 8 块 GPU 上用同步 SGD 训练。每个小批量每 GPU 2 张图、每图 512 个 RoI。权重衰减 0.0001，动量 0.9。学习率前 60k 小批量为 0.02，随后 20k 为 0.002。训练每图用 2000 个 RoI，测试用 1000 个。在 COCO 上用 FPN 训练 Fast R-CNN 约需 10 小时。

5.2.1 Fast R-CNN (on fixed proposals)

To better investigate FPN’s effects on the region-based detector alone, we conduct ablations of Fast R-CNN on a fixed set of proposals. We choose to freeze the proposals as computed by RPN on FPN (Table 3(c)), because it has good performance on small objects that are to be recognized by the detector. For simplicity we do not share features between Fast R-CNN and RPN, except when specified.

5.2.1 Fast R-CNN（在固定提案上）

为更好单独考察 FPN 对基于区域的检测器的影响，我们在固定提案集上对 Fast R-CNN 做消融。我们选择冻结由 FPN 上的 RPN 算出的提案（表 3(c)），因为它在小物体上表现良好，而这些小物体正是检测器需要识别的。为简单起见，除特别说明外，我们不在 Fast R-CNN 与 RPN 之间共享特征。

As a ResNet-based Fast R-CNN baseline, following [16], we adopt RoI pooling with an output size of 14$\times$14 and attach all conv5 layers as the hidden layers of the head. This gives an AP of 31.9 in Table 3(a). Table 3(b) is a baseline exploiting an MLP head with 2 hidden fc layers, similar to the head in our architecture. It gets an AP of 28.8, indicating that the 2-fc head does not give us any orthogonal advantage over the baseline in Table 3(a).

作为基于 ResNet 的 Fast R-CNN 基线，我们按 [16] 采用输出尺寸为 $14\times14$ 的 RoI 池化，并把全部 conv5 层作为头的隐层。这在表 3(a) 中得到 AP 31.9。表 3(b) 是采用带 2 个隐 fc 层的 MLP 头的基线，类似我们架构中的头。它得到 AP 28.8，表明 2-fc 头相对表 3(a) 的基线并未带来额外正交优势。

Table 3(c) shows the results of our FPN in Fast R-CNN. Comparing with the baseline in Table 3(a), our method improves AP by 2.0 points and small object AP by 2.1 points. Comparing with the baseline that also adopts a 2fc head (Table 3(b)), our method improves AP by 5.1 points.[^5] These comparisons indicate that our feature pyramid is superior to single-scale features for a region-based object detector.

表 3(c) 给出我们的 FPN 在 Fast R-CNN 中的结果。相对表 3(a) 的基线，我们的方法把 AP 提高 2.0 点，小物体 AP 提高 2.1 点。相对同样采用 2fc 头的基线（表 3(b)），我们的方法把 AP 提高 5.1 点。[^5] 这些比较表明，对我们的基于区域的目标检测器而言，特征金字塔优于单尺度特征。

Table 3(d) and (e) show that removing top-down connections or removing lateral connections leads to inferior results, similar to what we have observed in the above subsection for RPN. It is noteworthy that removing top-down connections (Table 3(d)) significantly degrades the accuracy, suggesting that Fast R-CNN suffers from using the low-level features at the high-resolution maps.

表 3(d) 与 (e) 表明，去掉自顶向下连接或去掉横向连接都会导致更差的结果，与上一小节对 RPN 的观察类似。值得注意的是，去掉自顶向下连接（表 3(d)）显著降低精度，说明 Fast R-CNN 在使用高分辨率图上的低层特征时会受损。

In Table 3(f), we adopt Fast R-CNN on the single finest scale feature map of $P_{2}$. Its result (33.4 AP) is marginally worse than that of using all pyramid levels (33.9 AP, Table 3(c)). We argue that this is because RoI pooling is a warping-like operation, which is less sensitive to the region’s scales. Despite the good accuracy of this variant, it is based on the RPN proposals of $\{P_{k}\}$ and has thus already benefited from the pyramid representation.

在表 3(f) 中，我们在单一最精细尺度特征图 $P_{2}$ 上采用 Fast R-CNN。其结果（33.4 AP）略差于使用全部金字塔层级（33.9 AP，表 3(c)）。我们认为这是因为 RoI 池化是一种类似变形的操作，对区域尺度不那么敏感。尽管该变体精度不错，它基于 $\{P_{k}\}$ 的 RPN 提案，因而已经受益于金字塔表示。

5.2.2 Faster R-CNN (on consistent proposals)

In the above we used a fixed set of proposals to investigate the detectors. But in a Faster R-CNN system [29], the RPN and Fast R-CNN must use the same network backbone in order to make feature sharing possible. Table 3 shows the comparisons between our method and two baselines, all using consistent backbone architectures for RPN and Fast R-CNN. Table 3(a) shows our reproduction of the baseline Faster R-CNN system as described in [16]. Under controlled settings, our FPN (Table 3(c)) is better than this strong baseline by 2.3 points AP and 3.8 points AP@0.5.

5.2.2 Faster R-CNN（在一致提案上）

上文用固定提案集考察检测器。但在 Faster R-CNN 系统 [29] 中，RPN 与 Fast R-CNN 必须使用同一网络骨干，才能实现特征共享。表 3 比较了我们的方法与两个基线，它们都对 RPN 与 Fast R-CNN 使用一致的骨干架构。表 3(a) 是我们按 [16] 复现的基线 Faster R-CNN 系统。在受控设置下，我们的 FPN（表 3(c)）相对这一强基线高出 2.3 点 AP 与 3.8 点 AP@0.5。

Note that Table 3(a) and (b) are baselines that are much stronger than the baseline provided by He et al. [16] in Table 3(). We find the following implementations contribute to the gap: (i) We use an image scale of 800 pixels instead of 600 in [11, 16]; (ii) We train with 512 RoIs per image which accelerate convergence, in contrast to 64 RoIs in [11, 16]; (iii) We use 5 scale anchors instead of 4 in [16] (adding $32^{2}$); (iv) At test time we use 1000 proposals per image instead of 300 in [16]. So comparing with He et al.’s ResNet-50 Faster R-CNN baseline in Table 3(), our method improves AP by 7.6 points and AP@0.5 by 9.6 points.

注意表 3(a) 与 (b) 是远强于 He 等人 [16] 在表 3() 中所给基线的基线。我们发现以下实现贡献了差距：(i) 我们使用 800 像素的图像尺度而非 [11, 16] 中的 600；(ii) 我们每图用 512 个 RoI 训练以加速收敛，而 [11, 16] 为 64 个 RoI；(iii) 我们使用 5 种尺度锚而非 [16] 中的 4 种（增加了 $32^{2}$）；(iv) 测试时每图用 1000 个提案而非 [16] 中的 300。因此相对 He 等人在表 3() 中的 ResNet-50 Faster R-CNN 基线，我们的方法把 AP 提高 7.6 点，AP@0.5 提高 9.6 点。

	ResNet-50	ResNet-101
share features?	AP*@0.5*	AP	AP*@0.5*	AP
no	56.9	33.9	58.0	35.0
yes	57.2	34.3	58.2	35.2

Table 5: More object detection results using Faster R-CNN and our FPNs, evaluated on minival. Sharing features increases train time by 1.5$\times$ (using 4-step training [29]), but reduces test time.

	ResNet-50	ResNet-101
共享特征?	AP*@0.5*	AP	AP*@0.5*	AP
否	56.9	33.9	58.0	35.0
是	57.2	34.3	58.2	35.2

表 5：使用 Faster R-CNN 与我们的 FPN 的更多目标检测结果，在 minival 上评估。共享特征使训练时间增加 1.5$\times$（采用 4 步训练 [29]），但减少测试时间。

Sharing features.

In the above, for simplicity we do not share the features between RPN and Fast R-CNN. In Table 5, we evaluate sharing features following the 4-step training described in [29]. Similar to [29], we find that sharing features improves accuracy by a small margin. Feature sharing also reduces the testing time.

共享特征。

上文为简单起见，我们不在 RPN 与 Fast R-CNN 之间共享特征。在表 5 中，我们按 [29] 所述的 4 步训练评估共享特征。与 [29] 类似，我们发现共享特征小幅提升精度。特征共享也减少测试时间。

Running time.

With feature sharing, our FPN-based Faster R-CNN system has inference time of 0.148 seconds per image on a single NVIDIA M40 GPU for ResNet-50, and 0.172 seconds for ResNet-101.[^6] As a comparison, the single-scale ResNet-50 baseline in Table 3(a) runs at 0.32 seconds. Our method introduces small extra cost by the extra layers in the FPN, but has a lighter weight head. Overall our system is faster than the ResNet-based Faster R-CNN counterpart. We believe the efficiency and simplicity of our method will benefit future research and applications.

运行时间。

在共享特征时，我们基于 FPN 的 Faster R-CNN 系统在单块 NVIDIA M40 GPU 上，ResNet-50 每图推理 0.148 秒，ResNet-101 为 0.172 秒。[^6] 作为对比，表 3(a) 中的单尺度 ResNet-50 基线为 0.32 秒。我们的方法因 FPN 中的额外层引入少量额外开销，但头更轻。总体而言，我们的系统快于基于 ResNet 的 Faster R-CNN 对应方案。我们相信方法的效率与简洁将惠及未来研究与应用。

5.2.3 Comparing with COCO Competition Winners

We find that our ResNet-101 model in Table 5 is not sufficiently trained with the default learning rate schedule. So we increase the number of mini-batches by 2$\times$ at each learning rate when training the Fast R-CNN step. This increases AP on minival to 35.6, without sharing features. This model is the one we submitted to the COCO detection leaderboard, shown in Table 4. We have not evaluated its feature-sharing version due to limited time, which should be slightly better as implied by Table 5.

5.2.3 与 COCO 竞赛优胜者比较

我们发现表 5 中的 ResNet-101 模型在默认学习率日程下训练不够充分。因此在训练 Fast R-CNN 步骤时，把每个学习率下的小批量数增加 $2\times$。这使 minival 上的 AP 提高到 35.6（不共享特征）。该模型即我们提交到 COCO 检测排行榜、并在表 4 中展示的模型。因时间有限，我们尚未评估其共享特征版本；由表 5 可推断应略好。

Table 4 compares our method with the single-model results of the COCO competition winners, including the 2016 winner G-RMI and the 2015 winner Faster R-CNN+++. Without adding bells and whistles, our single-model entry has surpassed these strong, heavily engineered competitors. On the test-dev set, our method increases over the existing best results by 0.5 points of AP (36.2 vs. 35.7) and 3.4 points of AP@0.5 (59.1 vs. 55.7). It is worth noting that our method does not rely on image pyramids and only uses a single input image scale, but still has outstanding AP on small-scale objects. This could only be achieved by high-resolution image inputs with previous methods.

表 4 将我们的方法与 COCO 竞赛优胜者的单模型结果比较，包括 2016 年优胜者 G-RMI 与 2015 年优胜者 Faster R-CNN+++。无需添加花哨技巧，我们的单模型条目已超过这些强大、重度工程化的竞争者。在 test-dev 集上，我们的方法相对既有最佳结果提高 0.5 点 AP（36.2 对 35.7）与 3.4 点 AP@0.5（59.1 对 55.7）。值得注意的是，我们的方法不依赖图像金字塔，只使用单一输入图像尺度，但在小尺度物体上仍有出色的 AP。以往方法通常只有在高分辨率图像输入下才能做到这一点。

Moreover, our method does not exploit many popular improvements, such as iterative regression [9], hard negative mining [35], context modeling [16], stronger data augmentation [22], etc. These improvements are complementary to FPNs and should boost accuracy further.

此外，我们的方法没有利用许多流行改进，例如迭代回归 [9]、难负例挖掘 [35]、上下文建模 [16]、更强的数据增强 [22] 等。这些改进与 FPN 互补，应能进一步提升精度。

Recently, FPN has enabled new top results in all tracks of the COCO competition, including detection, instance segmentation, and keypoint estimation. See [14] for details.

最近，FPN 已在 COCO 竞赛的全部赛道上促成了新的顶尖结果，包括检测、实例分割与关键点估计。详见 [14]。

6 Extensions: Segmentation Proposals

6 扩展：分割提案

Figure 4: FPN for object segment proposals. The feature pyramid is constructed with identical structure as for object detection. We apply a small MLP on 5$\times$5 windows to generate dense object segments with output dimension of 14$\times$14. Shown in orange are the size of the image regions the mask corresponds to for each pyramid level (levels $P_{3-5}$ are shown here). Both the corresponding image region size (light orange) and canonical object size (dark orange) are shown. Half octaves are handled by an MLP on 7x7 windows ($7\approx 5\sqrt{2}$), not shown here. Details are in the appendix.

图 4：用于物体分割提案的 FPN。特征金字塔的构建结构与目标检测相同。我们在 $5\times5$ 窗口上应用小型 MLP，以输出维度 $14\times14$ 生成稠密物体分割。橙色所示为每个金字塔层级上掩码对应的图像区域大小（此处显示 $P_{3-5}$）。同时给出对应图像区域大小（浅橙）与规范物体大小（深橙）。半倍频程由 $7\times7$ 窗口上的 MLP 处理（$7\approx 5\sqrt{2}$），图中未画出。细节见附录。

Our method is a generic pyramid representation and can be used in applications other than object detection. In this section we use FPNs to generate segmentation proposals, following the DeepMask/SharpMask framework [27, 28].

我们的方法是一种通用的金字塔表示，可用于目标检测以外的应用。本节我们按 DeepMask/SharpMask 框架 [27, 28]，用 FPN 生成分割提案。

DeepMask/SharpMask were trained on image crops for predicting instance segments and object/non-object scores. At inference time, these models are run convolutionally to generate dense proposals in an image. To generate segments at multiple scales, image pyramids are necessary [27, 28].

DeepMask/SharpMask 在图像裁剪上训练，以预测实例分割与物体/非物体分数。推理时，这些模型以卷积方式运行，在图像中生成稠密提案。为在多个尺度上生成分割，需要图像金字塔 [27, 28]。

It is easy to adapt FPN to generate mask proposals. We use a fully convolutional setup for both training and inference. We construct our feature pyramid as in Sec. 5.1 and set $d=128$. On top of each level of the feature pyramid, we apply a small 5$\times$5 MLP to predict 14$\times$14 masks and object scores in a fully convolutional fashion, see Fig. 4. Additionally, motivated by the use of 2 scales per octave in the image pyramid of [27, 28], we use a second MLP of input size 7$\times$7 to handle half octaves. The two MLPs play a similar role as anchors in RPN. The architecture is trained end-to-end; full implementation details are given in the appendix.

把 FPN 适配到生成掩码提案很容易。训练与推理都使用全卷积设置。我们按第 5.1 节构建特征金字塔并设 $d=128$。在特征金字塔每一层之上，我们应用小型 $5\times5$ MLP，以全卷积方式预测 $14\times14$ 掩码与物体分数，见图 4。此外，受 [27, 28] 图像金字塔中每倍频程 2 个尺度的启发，我们使用第二个输入尺寸为 $7\times7$ 的 MLP 来处理半倍频程。两个 MLP 扮演与 RPN 中锚类似的角色。该架构端到端训练；完整实现细节见附录。

	image pyramid	AR	ARs	ARm	ARl	time (s)
DeepMask [27]	✓	37.1	15.8	50.1	54.9	0.49
SharpMask [28]	✓	39.8	17.4	53.1	59.1	0.77
InstanceFCN [4]	✓	39.2	–	–	–	1.50†
FPN Mask Results:
single MLP [5$\times$5]		43.4	32.5	49.2	53.7	0.15
single MLP [7$\times$7]		43.5	30.0	49.6	57.8	0.19
dual MLP [5$\times$5, 7$\times$7]		45.7	31.9	51.5	60.8	0.24
+ 2x mask resolution		46.7	31.7	53.1	63.2	0.25
+ 2x train schedule		48.1	32.6	54.2	65.6	0.25

Table 6: Instance segmentation proposals evaluated on the first 5k COCO val images. All models are trained on the train set. DeepMask, SharpMask, and FPN use ResNet-50 while InstanceFCN uses VGG-16. DeepMask and SharpMask performance is computed with models available from https://github.com/facebookresearch/deepmask (both are the ‘zoom’ variants). †Runtimes are measured on an NVIDIA M40 GPU, except the InstanceFCN timing which is based on the slower K40.

	图像金字塔	AR	ARs	ARm	ARl	时间 (s)
DeepMask [27]	✓	37.1	15.8	50.1	54.9	0.49
SharpMask [28]	✓	39.8	17.4	53.1	59.1	0.77
InstanceFCN [4]	✓	39.2	–	–	–	1.50†
FPN 掩码结果：
单 MLP [$5\times5$]		43.4	32.5	49.2	53.7	0.15
单 MLP [$7\times7$]		43.5	30.0	49.6	57.8	0.19
双 MLP [$5\times5$, $7\times7$]		45.7	31.9	51.5	60.8	0.24
+ 2x 掩码分辨率		46.7	31.7	53.1	63.2	0.25
+ 2x 训练日程		48.1	32.6	54.2	65.6	0.25

表 6：实例分割提案，在前 5k 张 COCO val 图像上评估。所有模型在训练集上训练。DeepMask、SharpMask 与 FPN 使用 ResNet-50，InstanceFCN 使用 VGG-16。DeepMask 与 SharpMask 的表现用 https://github.com/facebookresearch/deepmask 上可用的模型计算（均为 'zoom' 变体）。†运行时间在 NVIDIA M40 GPU 上测量，InstanceFCN 计时基于更慢的 K40。

6.1 Segmentation Proposal Results

6.1 分割提案结果

Results are shown in Table 6. We report segment AR and segment AR on small, medium, and large objects, always for 1000 proposals. Our baseline FPN model with a single 5$\times$5 MLP achieves an AR of 43.4. Switching to a slightly larger 7$\times$7 MLP leaves accuracy largely unchanged. Using both MLPs together increases accuracy to 45.7 AR. Increasing mask output size from 14$\times$14 to 28$\times$28 increases AR another point (larger sizes begin to degrade accuracy). Finally, doubling the training iterations increases AR to 48.1.

结果见表 6。我们报告分割 AR 以及小、中、大物体上的分割 AR，提案数始终为 1000。带单个 $5\times5$ MLP 的基线 FPN 模型达到 AR 43.4。换成稍大的 $7\times7$ MLP，精度基本不变。两个 MLP 一起使用把精度提到 45.7 AR。把掩码输出尺寸从 $14\times14$ 增到 $28\times28$ 再提高约 1 点 AR（更大尺寸开始损害精度）。最后，把训练迭代加倍把 AR 提到 48.1。

We also report comparisons to DeepMask [27], SharpMask [28], and InstanceFCN [4], the previous state of the art methods in mask proposal generation. We outperform the accuracy of these approaches by over 8.3 points AR. In particular, we nearly double the accuracy on small objects.

我们也与 DeepMask [27]、SharpMask [28] 和 InstanceFCN [4] 这些掩码提案生成的既有最优方法做了比较。我们的精度超过这些方法 8.3 点以上 AR。尤其在小物体上，精度几乎翻倍。

Existing mask proposal methods [27, 28, 4] are based on densely sampled image pyramids (e.g., scaled by $2^{\{-2:0.5:1\}}$ in [27, 28]), making them computationally expensive. Our approach, based on FPNs, is substantially faster (our models run at 6 to 7 FPS). These results demonstrate that our model is a generic feature extractor and can replace image pyramids for other multi-scale detection problems.

既有掩码提案方法 [27, 28, 4] 基于稠密采样的图像金字塔（例如 [27, 28] 中按 $2^{\{-2:0.5:1\}}$ 缩放），计算昂贵。我们基于 FPN 的方法显著更快（模型运行在 6 到 7 FPS）。这些结果表明，我们的模型是通用特征提取器，可在其他多尺度检测问题中替代图像金字塔。

7 Conclusion

7 结论

We have presented a clean and simple framework for building feature pyramids inside ConvNets. Our method shows significant improvements over several strong baselines and competition winners. Thus, it provides a practical solution for research and applications of feature pyramids, without the need of computing image pyramids. Finally, our study suggests that despite the strong representational power of deep ConvNets and their implicit robustness to scale variation, it is still critical to explicitly address multi-scale problems using pyramid representations.

我们提出了一种在 ConvNet 内部构建特征金字塔的干净简洁框架。我们的方法相对若干强基线与竞赛优胜者都有显著提升。因此，它为特征金字塔的研究与应用提供了实用方案，而无需计算图像金字塔。最后，我们的研究提示：尽管深度 ConvNet 表征能力强、对尺度变化有隐含稳健性，用金字塔表示显式处理多尺度问题仍然至关重要。

Appendix A Implementation of Segmentation Proposals

附录 A 分割提案的实现

We use our feature pyramid networks to efficiently generate object segment proposals, adopting an image-centric training strategy popular for object detection [11, 29]. Our FPN mask generation model inherits many of the ideas and motivations from DeepMask/SharpMask [27, 28]. However, in contrast to these models, which were trained on image crops and used a densely sampled image pyramid for inference, we perform fully-convolutional training for mask prediction on a feature pyramid. While this requires changing many of the specifics, our implementation remains similar in spirit to DeepMask. Specifically, to define the label of a mask instance at each sliding window, we think of this window as being a crop on the input image, allowing us to inherit definitions of positives/negatives from DeepMask. We give more details next, see also Fig. 4 for a visualization.

我们用特征金字塔网络高效生成物体分割提案，并采用目标检测中流行的以图像为中心的训练策略 [11, 29]。我们的 FPN 掩码生成模型继承了 DeepMask/SharpMask [27, 28] 的许多想法与动机。但与这些在图像裁剪上训练、推理时使用稠密采样图像金字塔的模型不同，我们在特征金字塔上对掩码预测做全卷积训练。这虽然需要改动许多细节，实现精神仍与 DeepMask 相近。具体而言，为定义每个滑动窗口处掩码实例的标签，我们把该窗口视为输入图像上的裁剪，从而继承 DeepMask 对正/负例的定义。下文给出更多细节，可视化亦见图 4。

We construct the feature pyramid with $P_{2-6}$ using the same architecture as described in Sec. 5.1. We set $d=128$. Each level of our feature pyramid is used for predicting masks at a different scale. As in DeepMask, we define the scale of a mask as the max of its width and height. Masks with scales of $\{32,64,128,256,512\}$ pixels map to $\{P_{2},P_{3},P_{4},P_{5},P_{6}\}$, respectively, and are handled by a 5$\times$5 MLP. As DeepMask uses a pyramid with half octaves, we use a second slightly larger MLP of size 7$\times$7 ($7\approx 5\sqrt{2}$) to handle half-octaves in our model (e.g., a $128\sqrt{2}$ scale mask is predicted by the 7$\times$7 MLP on $P_{4}$). Objects at intermediate scales are mapped to the nearest scale in log space.

我们用与第 5.1 节相同的架构构建 $P_{2-6}$ 特征金字塔，并设 $d=128$。特征金字塔的每一层用于在不同尺度上预测掩码。与 DeepMask 一样，我们把掩码尺度定义为其宽与高的最大值。尺度为 $\{32,64,128,256,512\}$ 像素的掩码分别映射到 $\{P_{2},P_{3},P_{4},P_{5},P_{6}\}$，并由 $5\times5$ MLP 处理。由于 DeepMask 使用带半倍频程的金字塔，我们使用第二个稍大的 $7\times7$ MLP（$7\approx 5\sqrt{2}$）来处理模型中的半倍频程（例如，尺度为 $128\sqrt{2}$ 的掩码由 $P_{4}$ 上的 $7\times7$ MLP 预测）。中间尺度的物体在对数空间映射到最近尺度。

As the MLP must predict objects at a range of scales for each pyramid level (specifically a half octave range), some padding must be given around the canonical object size. We use 25% padding. This means that the mask output over $\{P_{2},P_{3},P_{4},P_{5},P_{6}\}$ maps to $\{40,80,160,320,640\}$ sized image regions for the 5$\times$5 MLP (and to $\sqrt{2}$ larger corresponding sizes for the 7$\times$7 MLP).

由于 MLP 必须在每个金字塔层级上预测一定尺度范围（具体为一个半倍频程）内的物体，规范物体尺寸周围需要一些填充。我们使用 25% 填充。这意味着 $5\times5$ MLP 在 $\{P_{2},P_{3},P_{4},P_{5},P_{6}\}$ 上的掩码输出分别映射到大小为 $\{40,80,160,320,640\}$ 的图像区域（$7\times7$ MLP 对应尺寸再大 $\sqrt{2}$ 倍）。

Each spatial position in the feature map is used to predict a mask at a different location. Specifically, at scale $P_{k}$, each spatial position in the feature map is used to predict the mask whose center falls within $2^{k}$ pixels of that location (corresponding to $\pm 1$ cell offset in the feature map). If no object center falls within this range, the location is considered a negative, and, as in DeepMask, is used only for training the score branch and not the mask branch.

特征图中的每个空间位置用于在不同位置预测掩码。具体而言，在尺度 $P_{k}$ 上，特征图中每个空间位置用于预测中心落在该位置 $2^{k}$ 像素范围内的掩码（对应特征图中 $\pm 1$ 个单元的偏移）。若没有物体中心落在此范围内，则该位置视为负例，并与 DeepMask 一样，仅用于训练分数分支而非掩码分支。

The MLP we use for predicting the mask and score is fairly simple. We apply a 5$\times$5 kernel with 512 outputs, followed by sibling fully connected layers to predict a 14$\times$14 mask ($14^{2}$ outputs) and object score (1 output). The model is implemented in a fully convolutional manner (using 1$\times$1 convolutions in place of fully connected layers). The 7$\times$7 MLP for handling objects at half octave scales is identical to the 5$\times$5 MLP except for its larger input region.

我们用于预测掩码与分数的 MLP 相当简单。我们应用输出 512 维的 $5\times5$ 核，再接兄弟全连接层以预测 $14\times14$ 掩码（$14^{2}$ 个输出）与物体分数（1 个输出）。模型以全卷积方式实现（用 $1\times1$ 卷积替代全连接层）。用于处理半倍频程尺度物体的 $7\times7$ MLP 与 $5\times5$ MLP 相同，只是输入区域更大。

During training, we randomly sample 2048 examples per mini-batch (128 examples per image from 16 images) with a positive/negative sampling ratio of 1:3. The mask loss is given 10$\times$ higher weight than the score loss. This model is trained end-to-end on 8 GPUs using synchronized SGD (2 images per GPU). We start with a learning rate of 0.03 and train for 80k mini-batches, dividing the learning rate by 10 after 60k mini-batches. The image scale is set to 800 pixels during training and testing (we do not use scale jitter). During inference our fully-convolutional model predicts scores at all positions and scales and masks at the 1000 highest scoring locations. We do not perform any non-maximum suppression or post-processing.

训练时，我们每个小批量随机采样 2048 个样本（16 张图、每图 128 个样本），正负采样比为 1:3。掩码损失的权重设为分数损失的 $10\times$。该模型在 8 块 GPU 上用同步 SGD（每 GPU 2 张图）端到端训练。学习率从 0.03 开始，训练 80k 小批量，并在 60k 后将学习率除以 10。训练与测试时图像尺度设为 800 像素（不使用尺度抖动）。推理时，我们的全卷积模型在所有位置与尺度上预测分数，并在得分最高的 1000 个位置预测掩码。我们不做任何非极大值抑制或后处理。

References

参考文献

(1) E. H. Adelson, C. H. Anderson, J. R. Bergen, P. J. Burt, and J. M. Ogden. Pyramid methods in image processing. RCA engineer , 1984.

(2) S. Bell, C. L. Zitnick, K. Bala, and R. Girshick. Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. In CVPR , 2016.

(3) Z. Cai, Q. Fan, R. S. Feris, and N. Vasconcelos. A unified multi-scale deep convolutional neural network for fast object detection. In ECCV , 2016.

(4) J. Dai, K. He, Y. Li, S. Ren, and J. Sun. Instance-sensitive fully convolutional networks. In ECCV , 2016.

(5) N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR , 2005.

(6) P. Dollár, R. Appel, S. Belongie, and P. Perona. Fast feature pyramids for object detection. TPAMI , 2014.

(7) P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. TPAMI , 2010.

(8) G. Ghiasi and C. C. Fowlkes. Laplacian pyramid reconstruction and refinement for semantic segmentation. In ECCV , 2016.

(9) S. Gidaris and N. Komodakis. Object detection via a multi-region & semantic segmentation-aware CNN model. In ICCV , 2015.

(10) S. Gidaris and N. Komodakis. Attend refine repeat: Active box proposal generation via in-out localization. In BMVC , 2016.

(11) R. Girshick. Fast R-CNN. In ICCV , 2015.

(12) R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , 2014.

(13) B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and fine-grained localization. In CVPR , 2015.

(14) K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. arXiv:1703.06870 , 2017.

(15) K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In ECCV . 2014.

(16) K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , 2016.

(17) S. Honari, J. Yosinski, P. Vincent, and C. Pal. Recombinator networks: Learning coarse-to-fine feature aggregation. In CVPR , 2016.

(18) T. Kong, A. Yao, Y. Chen, and F. Sun. Hypernet: Towards accurate region proposal generation and joint object detection. In CVPR , 2016.

(19) A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS , 2012.

(20) Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation , 1989.

(21) T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV , 2014.

(22) W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. Reed. SSD: Single shot multibox detector. In ECCV , 2016.

(23) W. Liu, A. Rabinovich, and A. C. Berg. ParseNet: Looking wider to see better. In ICLR workshop , 2016.

(24) J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR , 2015.

(25) D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV , 2004.

(26) A. Newell, K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In ECCV , 2016.

(27) P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In NIPS , 2015.

(28) P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollár. Learning to refine object segments. In ECCV , 2016.

(29) S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS , 2015.

(30) S. Ren, K. He, R. Girshick, X. Zhang, and J. Sun. Object detection networks on convolutional feature maps. PAMI , 2016.

(31) O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI , 2015.

(32) H. Rowley, S. Baluja, and T. Kanade. Human face detection in visual scenes. Technical Report CMU-CS-95-158R, Carnegie Mellon University, 1995.

(33) O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV , 2015.

(34) P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. In ICLR , 2014.

(35) A. Shrivastava, A. Gupta, and R. Girshick. Training region-based object detectors with online hard example mining. In CVPR , 2016.

(36) K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR , 2015.

(37) J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W. Smeulders. Selective search for object recognition. IJCV , 2013.

(38) R. Vaillant, C. Monrocq, and Y. LeCun. Original approach for the localisation of objects in images. IEE Proc. on Vision, Image, and Signal Processing , 1994.

(39) S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC , 2016.

(40) S. Zagoruyko, A. Lerer, T.-Y. Lin, P. O. Pinheiro, S. Gross, S. Chintala, and P. Dollár. A multipath network for object detection. In BMVC , 2016.