MobileNetV2: Inverted Residuals and Linear Bottlenecks

MobileNetV2：倒残差与线性瓶颈

Mark Sandler Andrew Howard Menglong Zhu Andrey Zhmoginov Liang-Chieh Chen Google Inc. {sandler, howarda, menglong, azhmogin, lcchen}@google.com

Abstract

摘要

In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3.

本文描述一种新的移动端架构 MobileNetV2：它在多项任务和基准上提升了移动模型的当时最优表现，并且覆盖一系列不同的模型规模。我们还说明如何在一种称为 SSDLite 的新框架里，把这些移动模型高效用于目标检测。此外，我们展示如何通过精简版 DeepLabv3（称为 Mobile DeepLabv3）构建移动端语义分割模型。

is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design.

其基础是倒残差结构：捷径连接加在较窄的瓶颈层之间。中间的扩展层用轻量深度卷积过滤特征，作为非线性的来源。我们还发现，为了保持表征能力，有必要去掉窄层中的非线性。我们证明这一点能提升性能，并给出促成这一设计的直观理由。

Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on ImageNet Russakovsky:2015:ILS:2846547.2846559 classification, COCO object detection COCO, VOC image segmentation PASCAL. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as actual latency, and the number of parameters.

最后，我们的方法把输入/输出域与变换的表达能力分开，为进一步分析提供了方便的框架。我们在 ImageNet 分类、COCO 目标检测和 VOC 图像分割上衡量表现，并评估精度与乘法–加法次数（MAdd）、实际延迟以及参数量之间的权衡。

1 Introduction

1 引言

Neural networks have revolutionized many areas of machine intelligence, enabling superhuman accuracy for challenging image recognition tasks. However, the drive to improve accuracy often comes at a cost: modern state of the art networks require high computational resources beyond the capabilities of many mobile and embedded applications.

神经网络已经改变了机器智能的许多领域，使具有挑战性的图像识别任务达到超人类精度。然而，追求更高精度往往要付出代价：现代最优网络所需的计算资源，超出了许多移动和嵌入式应用的能力。

This paper introduces a new neural network architecture that is specifically tailored for mobile and resource constrained environments. Our network pushes the state of the art for mobile tailored computer vision models, by significantly decreasing the number of operations and memory needed while retaining the same accuracy.

本文提出一种专门面向移动和资源受限环境的神经网络架构。我们的网络显著减少了所需运算量和内存，同时保持相同精度，从而推进了面向移动端的计算机视觉模型。

Our main contribution is a novel layer module: the inverted residual with linear bottleneck. This module takes as an input a low-dimensional compressed representation which is first expanded to high dimension and filtered with a lightweight depthwise convolution. Features are subsequently projected back to a low-dimensional representation with a linear convolution. The official implementation is available as part of TensorFlow-Slim model library in MobilenetV2-implementation.

我们的主要贡献是一种新的层模块：带线性瓶颈的倒残差。该模块接收低维压缩表征，先扩展到高维，再用轻量深度卷积过滤，随后用线性卷积投影回低维表征。官方实现作为 TensorFlow-Slim 模型库的一部分发布，见 MobilenetV2-implementation。

This module can be efficiently implemented using standard operations in any modern framework and allows our models to beat state of the art along multiple performance points using standard benchmarks. Furthermore, this convolutional module is particularly suitable for mobile designs, because it allows to significantly reduce the memory footprint needed during inference by never fully materializing large intermediate tensors. This reduces the need for main memory access in many embedded hardware designs, that provide small amounts of very fast software controlled cache memory.

该模块可以用任何现代框架中的标准运算高效实现，并使我们的模型在多个性能点上、按标准基准超过当时最优。此外，这一卷积模块特别适合移动端设计，因为它在推理时不必完整物化大型中间张量，从而显著降低内存占用。许多嵌入式硬件只提供少量由软件控制的高速缓存，减少对主存的访问因此很重要。

2 Related Work

2 相关工作

Tuning deep neural architectures to strike an optimal balance between accuracy and performance has been an area of active research for the last several years. Both manual architecture search and improvements in training algorithms, carried out by numerous teams has lead to dramatic improvements over early designs such as AlexNet AlexNet, VGGNet VGGNet, GoogLeNet GoogleNet. , and ResNet ResNet. Recently there has been lots of progress in algorithmic architecture exploration included hyper-parameter optimization BergstraRandomSearch; BayesianOptimizationML; BayesianOptimizationDNN as well as various methods of network pruning BrainSurgeon; BrainDamage; Han2015; Han2016; Guo2016; LiPruning and connectivity learning ConnectivityLearning; BudgetedSuperNetworks. A substantial amount of work has also been dedicated to changing the connectivity structure of the internal convolutional blocks such as in ShuffleNet ShuffleNet2017 or introducing sparsity PowerOfSparsity and others SpatialBottleneck2016.

过去几年里，调节深度神经架构以在精度与性能之间取得最佳平衡，一直是活跃的研究方向。众多团队进行的手工架构搜索和训练算法改进，相对 AlexNet、VGGNet、GoogLeNet 和 ResNet 等早期设计带来了显著提升。最近，算法化的架构探索进展很多，包括超参数优化，以及各种网络剪枝和连接学习方法。也有大量工作改变内部卷积块的连接结构，例如 ShuffleNet，或引入稀疏性等。

Recently, LearningToLearnScale; GeneticCNN; EvolutionImageClassifiers; NAS_reinforcement, opened up a new direction of bringing optimization methods including genetic algorithms and reinforcement learning to architectural search. However one drawback is that the resulting networks end up very complex. In this paper, we pursue the goal of developing better intuition about how neural networks operate and use that to guide the simplest possible network design. Our approach should be seen as complimentary to the one described in LearningToLearnScale and related work. In this vein our approach is similar to those taken by ShuffleNet2017; SpatialBottleneck2016 and allows to further improve the performance, while providing a glimpse on its internal operation. Our network design is based on MobileNetV1 MobilenetV1. It retains its simplicity and does not require any special operators while significantly improves its accuracy, achieving state of the art on multiple image classification and detection tasks for mobile applications.

近来，把遗传算法和强化学习等优化方法引入架构搜索开辟了新方向。缺点是得到的网络往往非常复杂。本文希望更好地理解神经网络如何工作，并用这种直觉指导尽可能简单的网络设计。我们的方法应看作对 LearningToLearnScale 及相关工作的补充。在这一思路上，它与 ShuffleNet、SpatialBottleneck 等相近，既能进一步提升性能，也能让人看到内部运作的一斑。我们的网络设计基于 MobileNetV1，保留了它的简洁性，不需要任何特殊算子，同时显著提高精度，在多项面向移动应用的图像分类和检测任务上达到当时最优。

3 Preliminaries, discussion and intuition

3 预备知识、讨论与直觉

3.1 Depthwise Separable Convolutions

3.1 深度可分离卷积

Depthwise Separable Convolutions are a key building block for many efficient neural network architectures MobilenetV1; Chollet_2017_CVPR; ShuffleNet2017 and we use them in the present work as well. The basic idea is to replace a full convolutional operator with a factorized version that splits convolution into two separate layers. The first layer is called a depthwise convolution, it performs lightweight filtering by applying a single convolutional filter per input channel. The second layer is a $1\times 1$ convolution, called a pointwise convolution, which is responsible for building new features through computing linear combinations of the input channels.

深度可分离卷积是许多高效神经网络架构的关键构件，本文同样使用它们。基本想法是用分解版本替换完整卷积算子，把卷积拆成两层。第一层称为深度卷积，对每个输入通道施加一个卷积滤波器，做轻量过滤。第二层是 $1\times 1$ 卷积，称为逐点卷积，通过计算输入通道的线性组合来构建新特征。

Standard convolution takes an $h_{i}\times w_{i}\times d_{i}$ input tensor $L_{i}$, and applies convolutional kernel $K\in{\cal R}^{k\times k\times d_{i}\times d_{j}}$ to produce an $h_{i}\times w_{i}\times d_{j}$ output tensor $L_{j}$. Standard convolutional layers have the computational cost of $h_{i}\cdot w_{i}\cdot d_{i}\cdot d_{j}\cdot k\cdot k$.

标准卷积接收 $h_{i}\times w_{i}\times d_{i}$ 的输入张量 $L_{i}$，施加卷积核 $K\in{\cal R}^{k\times k\times d_{i}\times d_{j}}$，得到 $h_{i}\times w_{i}\times d_{j}$ 的输出张量 $L_{j}$。标准卷积层的计算代价是 $h_{i}\cdot w_{i}\cdot d_{i}\cdot d_{j}\cdot k\cdot k$。

Depthwise separable convolutions are a drop-in replacement for standard convolutional layers. Empirically they work almost as well as regular convolutions but only cost:

深度可分离卷积可以替换标准卷积层。经验上它们几乎与常规卷积一样好，但代价仅为：

$$h_{i}\cdot w_{i}\cdot d_{i}(k^{2}+d_{j}) \tag{1}$$

which is the sum of the depthwise and $1\times 1$ pointwise convolutions. Effectively depthwise separable convolution reduces computation compared to traditional layers by almost a factor of $k^{2}$[^1]. MobileNetV2 uses $k=3$ ($3\times 3$ depthwise separable convolutions) so the computational cost is $8$ to $9$ times smaller than that of standard convolutions at only a small reduction in accuracy MobilenetV1.

也就是深度卷积与 $1\times 1$ 逐点卷积之和。相对于传统层，深度可分离卷积几乎把计算量降低了 $k^{2}$ 倍[^1]。MobileNetV2 使用 $k=3$（$3\times 3$ 深度可分离卷积），因此计算代价比标准卷积小 8 到 9 倍，精度只略有下降。

3.2 Linear Bottlenecks

3.2 线性瓶颈

Consider a deep neural network consisting of $n$ layers $L_{i}$ each of which has an activation tensor of dimensions $h_{i}\times w_{i}\times d_{i}$. Throughout this section we will be discussing the basic properties of these activation tensors, which we will treat as containers of $h_{i}\times w_{i}$ “pixels” with $d_{i}$ dimensions. Informally, for an input set of real images, we say that the set of layer activations (for any layer $L_{i}$) forms a “manifold of interest”. It has been long assumed that manifolds of interest in neural networks could be embedded in low-dimensional subspaces. In other words, when we look at all individual $d$-channel pixels of a deep convolutional layer, the information encoded in those values actually lie in some manifold, which in turn is embeddable into a low-dimensional subspace[^2].

考虑一个由 $n$ 层 $L_{i}$ 组成的深度网络，每层有维度为 $h_{i}\times w_{i}\times d_{i}$ 的激活张量。本节将讨论这些激活张量的基本性质，并把它们看作 $h_{i}\times w_{i}$ 个“像素”、每个像素有 $d_{i}$ 维的容器。非正式地说，对于一组真实输入图像，我们称（任意层 $L_{i}$ 的）层激活集合构成“感兴趣流形”。人们长期以来假定，神经网络中的感兴趣流形可以嵌入低维子空间。换句话说，当我们观察深度卷积层中所有单个 $d$ 通道像素时，这些值所编码的信息实际上位于某个流形上，而该流形又能嵌入一个低维子空间[^2]。

At a first glance, such a fact could then be captured and exploited by simply reducing the dimensionality of a layer thus reducing the dimensionality of the operating space. This has been successfully exploited by MobileNetV1 MobilenetV1 to effectively trade off between computation and accuracy via a width multiplier parameter, and has been incorporated into efficient model designs of other networks as well ShuffleNet2017. Following that intuition, the width multiplier approach allows one to reduce the dimensionality of the activation space until the manifold of interest spans this entire space. However, this intuition breaks down when we recall that deep convolutional neural networks actually have non-linear per coordinate transformations, such as $\reluop$. For example, $\reluop$ applied to a line in 1D space produces a ’ray’, where as in ${\cal R}^{n}$ space, it generally results in a piece-wise linear curve with $n$-joints.

乍看之下，这一事实似乎只需降低层的维度、从而降低工作空间的维度就能捕捉并利用。MobileNetV1 已成功用宽度乘数在计算与精度之间权衡，其他高效模型设计也吸收了这一点。按这种直觉，宽度乘数可以把激活空间的维度降到感兴趣流形刚好张满整个空间为止。然而，一旦记起深度卷积网络实际上有逐坐标的非线性变换（例如 $\reluop$），这一直觉就站不住了。例如，把 $\reluop$ 作用到一维空间中的一条直线会得到一条“射线”；而在 ${\cal R}^{n}$ 中，它一般会得到带 $n$ 个关节的分段线性曲线。

It is easy to see that in general if a result of a layer transformation $\reluop(Bx)$ has a non-zero volume $S$, the points mapped to $\operatorname{interior}{S}$ are obtained via a linear transformation $B$ of the input, thus indicating that the part of the input space corresponding to the full dimensional output, is limited to a linear transformation. In other words, deep networks only have the power of a linear classifier on the non-zero volume part of the output domain. We refer to supplemental material for a more formal statement.

不难看出，一般情形下，若层变换 $\reluop(Bx)$ 的结果具有非零体积 $S$，则映射到 $\operatorname{interior}{S}$ 的点是通过输入的线性变换 $B$ 得到的，这表明对应于满维输出的那部分输入空间，仅限于一次线性变换。换句话说，深度网络在输出域的非零体积部分上，只有线性分类器的能力。更形式的陈述见补充材料。

On the other hand, when $\reluop$ collapses the channel, it inevitably loses information in that channel. However if we have lots of channels, and there is a a structure in the activation manifold that information might still be preserved in the other channels. In supplemental materials, we show that if the input manifold can be embedded into a significantly lower-dimensional subspace of the activation space then the $\reluop$ transformation preserves the information while introducing the needed complexity into the set of expressible functions.

另一方面，当 $\reluop$ 把某个通道压塌时，它不可避免地丢失该通道中的信息。但如果通道很多，且激活流形具有结构，信息仍可能保存在其他通道里。补充材料表明：若输入流形可以嵌入激活空间中一个显著更低维的子空间，则 $\reluop$ 变换在引入所需复杂性的同时仍能保留信息。

Figure 1: Examples of $\reluop$ transformations of low-dimensional manifolds embedded in higher-dimensional spaces. In these examples the initial spiral is embedded into an $n$-dimensional space using random matrix $T$ followed by $\reluop$, and then projected back to the 2D space using $T^{-1}$. In examples above $n=2,3$ result in information loss where certain points of the manifold collapse into each other, while for $n=15$ to $30$ the transformation is highly non-convex.

图 1：$\reluop$ 对嵌入高维空间的低维流形所做变换的例子。这些例子里，初始螺旋线用随机矩阵 $T$ 嵌入 $n$ 维空间，再经过 $\reluop$，然后用 $T^{-1}$ 投影回二维。上面的例子中，$n=2,3$ 会造成信息损失，流形上某些点会互相塌缩；而 $n=15$ 到 $30$ 时，变换高度非凸。

Figure 2: Evolution of separable convolution blocks. The diagonally hatched texture indicates layers that do not contain non-linearities. The last (lightly colored) layer indicates the beginning of the next block. Note: 2d and 2c are equivalent blocks when stacked. Best viewed in color.

图 2：可分离卷积块的演变。对角阴影纹理表示不含非线性的层。最后一层（浅色）表示下一块的开始。注意：堆叠时 2d 与 2c 是等价块。请看彩色图。

To summarize, we have highlighted two properties that are indicative of the requirement that the manifold of interest should lie in a low-dimensional subspace of the higher-dimensional activation space:

概括起来，我们强调了两点，它们表明感兴趣流形应当位于更高维激活空间的一个低维子空间中：

1. If the manifold of interest remains non-zero volume after $\reluop$ transformation, it corresponds to a linear transformation.

1. 若感兴趣流形在 $\reluop$ 变换后仍保持非零体积，则它对应于一次线性变换。

2. $\reluop$ is capable of preserving complete information about the input manifold, but only if the input manifold lies in a low-dimensional subspace of the input space.

2. $\reluop$ 能够保留关于输入流形的完整信息，但仅当输入流形位于输入空间的一个低维子空间时。

These two insights provide us with an empirical hint for optimizing existing neural architectures: assuming the manifold of interest is low-dimensional we can capture this by inserting linear bottleneck layers into the convolutional blocks. Experimental evidence suggests that using linear layers is crucial as it prevents non-linearities from destroying too much information. In Section 6, we show empirically that using non-linear layers in bottlenecks indeed hurts the performance by several percent, further validating our hypothesis[^3]. We note that similar reports where non-linearity was helped were reported in DeepPyramidalNetworks where non-linearity was removed from the input of the traditional residual block and that lead to improved performance on CIFAR dataset.

这两点洞察为我们优化现有神经架构提供了经验提示：假定感兴趣流形是低维的，就可以通过在卷积块中插入线性瓶颈层来捕捉这一点。实验证据表明，使用线性层至关重要，因为它能防止非线性破坏过多信息。第 6 节我们经验上表明，在瓶颈中使用非线性层确实会使性能下降几个百分点，进一步验证了这一假设[^3]。我们注意到，DeepPyramidalNetworks 也报告过类似情况：从传统残差块的输入去掉非线性，在 CIFAR 上带来了更好的表现。

For the remainder of this paper we will be utilizing bottleneck convolutions. We will refer to the ratio between the size of the input bottleneck and the inner size as the expansion ratio.

本文余下部分将使用瓶颈卷积。我们把输入瓶颈大小与内部大小之比称为扩展比。

3.3 Inverted residuals

3.3 倒残差

Figure 3: The difference between residual block [8, 30] and inverted residual. Diagonally hatched layers do not use non-linearities. We use thickness of each block to indicate its relative number of channels. Note how classical residuals connects the layers with high number of channels, whereas the inverted residuals connect the bottlenecks. Best viewed in color.

图 3：残差块与倒残差的差别。对角阴影的层不使用非线性。我们用每块的厚度表示相对通道数。注意，经典残差连接的是通道数多的层，而倒残差连接的是瓶颈。请看彩色图。

The bottleneck blocks appear similar to residual block where each block contains an input followed by several bottlenecks then followed by expansion ResNet. However, inspired by the intuition that the bottlenecks actually contain all the necessary information, while an expansion layer acts merely as an implementation detail that accompanies a non-linear transformation of the tensor, we use shortcuts directly between the bottlenecks. Figure 3 provides a schematic visualization of the difference in the designs. The motivation for inserting shortcuts is similar to that of classical residual connections: we want to improve the ability of a gradient to propagate across multiplier layers. However, the inverted design is considerably more memory efficient (see Section 4 for details), as well as works slightly better in our experiments.

瓶颈块看起来类似于残差块：每个块先接收输入，再经过若干瓶颈，最后再扩展。不过，受“必要信息其实都在瓶颈里、扩展层只是伴随张量非线性变换的实现细节”这一直觉启发，我们把捷径直接加在瓶颈之间。图 3 示意了两种设计的差别。插入捷径的动机与经典残差连接类似：我们希望改善梯度穿过多层的传播能力。然而倒过来的设计内存效率明显更高（细节见第 4 节），在我们的实验中也略好一些。

Running time and parameter count for bottleneck convolution

The basic implementation structure is illustrated in Table 1. For a block of size $h\times w$, expansion factor $t$ and kernel size $k$ with $d^{\prime}$ input channels and $d^{\prime\prime}$ output channels, the total number of multiply add required is $h\cdot w\cdot d^{\prime}\cdot t(d^{\prime}+k^{2}+d^{\prime\prime})$. Compared with (1) this expression has an extra term, as indeed we have an extra $1\times 1$ convolution, however the nature of our networks allows us to utilize much smaller input and output dimensions. In Table 3 we compare the needed sizes for each resolution between MobileNetV1, MobileNetV2 and ShuffleNet.

瓶颈卷积的运行时间与参数量

基本实现结构见表 1。对于大小为 $h\times w$、扩展因子为 $t$、核大小为 $k$、输入通道为 $d^{\prime}$、输出通道为 $d^{\prime\prime}$ 的块，所需乘法–加法总数为 $h\cdot w\cdot d^{\prime}\cdot t(d^{\prime}+k^{2}+d^{\prime\prime})$。与 (1) 相比，该式多了一项，因为确实多了一次 $1\times 1$ 卷积；不过我们这类网络可以用小得多的输入和输出维度。表 3 比较了 MobileNetV1、MobileNetV2 和 ShuffleNet 在各分辨率下所需的大小。

3.4 Information flow interpretation

3.4 信息流解释

One interesting property of our architecture is that it provides a natural separation between the input/output domains of the building blocks (bottleneck layers), and the layer transformation – that is a non-linear function that converts input to the output. The former can be seen as the capacity of the network at each layer, whereas the latter as the expressiveness. This is in contrast with traditional convolutional blocks, both regular and separable, where both expressiveness and capacity are tangled together and are functions of the output layer depth.

我们架构的一个有趣性质是：它自然地把构建块的输入/输出域（瓶颈层）与层变换分开——后者是把输入转换成输出的非线性函数。前者可以看作网络在每一层的容量，后者则是表达能力。这与传统卷积块（无论常规还是可分离）不同：在那些块里，表达能力和容量缠在一起，都是输出层深度的函数。

In particular, in our case, when inner layer depth is $0$ the underlying convolution is the identity function thanks to the shortcut connection. When the expansion ratio is smaller than $1$, this is a classical residual convolutional block ResNet; ResNext2016. However, for our purposes we show that expansion ratio greater than $1$ is the most useful.

特别地，在我们这里，当内部层深度为 $0$ 时，由于捷径连接，底层卷积就是恒等函数。当扩展比小于 $1$ 时，这就是经典的残差卷积块。不过就我们的目的而言，扩展比大于 $1$ 最为有用。

This interpretation allows us to study the expressiveness of the network separately from its capacity and we believe that further exploration of this separation is warranted to provide a better understanding of the network properties.

这一解释让我们可以把网络的表达能力与容量分开研究。我们相信，进一步探索这种分离，有助于更好地理解网络性质。

4 Model Architecture

4 模型架构

Now we describe our architecture in detail. As discussed in the previous section the basic building block is a bottleneck depth-separable convolution with residuals. The detailed structure of this block is shown in Table 1. The architecture of MobileNetV2 contains the initial fully convolution layer with $32$ filters, followed by $19$ residual bottleneck layers described in the Table 2. We use ${\operatorname{\mathop{ReLU6}\,}}$ as the non-linearity because of its robustness when used with low-precision computation MobilenetV1. We always use kernel size $3\times 3$ as is standard for modern networks, and utilize dropout and batch normalization during training.

现在详细描述我们的架构。如前一节所述，基本构建块是带残差的瓶颈深度可分离卷积。该块的详细结构见表 1。MobileNetV2 的架构包含一个带 $32$ 个滤波器的初始全卷积层，随后是表 2 中描述的 19 个残差瓶颈层。非线性采用 ${\operatorname{\mathop{ReLU6}\,}}$，因为它在低精度计算中更稳健。我们始终使用 $3\times 3$ 核（这是现代网络的标准做法），并在训练时使用 dropout 和批归一化。

With the exception of the first layer, we use constant expansion rate throughout the network. In our experiments we find that expansion rates between $5$ and $10$ result in nearly identical performance curves, with smaller networks being better off with slightly smaller expansion rates and larger networks having slightly better performance with larger expansion rates.

除第一层外，我们在整个网络中使用恒定的扩展率。实验发现，扩展率在 $5$ 到 $10$ 之间时性能曲线几乎相同：较小的网络略受益于稍小的扩展率，较大的网络则略受益于稍大的扩展率。

For all our main experiments we use expansion factor of $6$ applied to the size of the input tensor. For example, for a bottleneck layer that takes $64$-channel input tensor and produces a tensor with $128$ channels, the intermediate expansion layer is then $64\cdot 6=384$ channels.

在所有主要实验中，我们把扩展因子取为输入张量大小的 $6$ 倍。例如，一个接收 $64$ 通道输入、产生 $128$ 通道输出的瓶颈层，其中间扩展层就是 $64\cdot 6=384$ 通道。

Input	Operator	Output
$h\times w\times k$	1x1 $\operatorname{\mathop{conv2d}\,}$, $\operatorname{\mathop{ReLU6}\,}$	$h\times w\times(tk)$
$h\times w\times tk$	3x3 $\operatorname{\mathop{dwise}\,}$s=$s$, $\operatorname{\mathop{ReLU6}\,}$	$\frac{h}{s}\times\frac{w}{s}\times(tk)$
$\frac{h}{s}\times\frac{w}{s}\times tk$	linear 1x1 $\operatorname{\mathop{conv2d}\,}$	$\frac{h}{s}\times\frac{w}{s}\times k^{\prime}$

Table 1: Bottleneck residual block transforming from $k$ to $k^{\prime}$ channels, with stride $s$, and expansion factor $t$.

输入	算子	输出
$h\times w\times k$	1x1 ${\operatorname{\mathop{conv2d}\,}}$，${\operatorname{\mathop{ReLU6}\,}}$	$h\times w\times(tk)$
$h\times w\times tk$	3x3 ${\operatorname{\mathop{dwise}\,}}$s=$s$，${\operatorname{\mathop{ReLU6}\,}}$	$\frac{h}{s}\times\frac{w}{s}\times(tk)$
$\frac{h}{s}\times\frac{w}{s}\times tk$	线性 1x1 ${\operatorname{\mathop{conv2d}\,}}$	$\frac{h}{s}\times\frac{w}{s}\times k^{\prime}$

表 1：把通道数从 $k$ 变到 $k^{\prime}$、步幅为 $s$、扩展因子为 $t$ 的瓶颈残差块。

Input	Operator	$t$	$c$	$n$	$s$
$224^{2}\times 3$	conv2d	-	32	1	2
$112^{2}\times 32$	bottleneck	1	16	1	1
$112^{2}\times 16$	bottleneck	6	24	2	2
$56^{2}\times 24$	bottleneck	6	32	3	2
$28^{2}\times 32$	bottleneck	6	64	4	2
$14^{2}\times 64$	bottleneck	6	96	3	1
$14^{2}\times 96$	bottleneck	6	160	3	2
$7^{2}\times 160$	bottleneck	6	320	1	1
$7^{2}\times 320$	conv2d 1x1	-	1280	1	1
$7^{2}\times 1280$	avgpool 7x7	-	-	1	-
$1\times 1\times 1280$	conv2d 1x1	-	k	-

Table 2: MobileNetV2 : Each line describes a sequence of 1 or more identical (modulo stride) layers, repeated $n$ times. All layers in the same sequence have the same number $c$ of output channels. The first layer of each sequence has a stride $s$ and all others use stride $1$. All spatial convolutions use $3\times 3$ kernels. The expansion factor $t$ is always applied to the input size as described in Table 1.

输入	算子	$t$	$c$	$n$	$s$
$224^{2}\times 3$	conv2d	-	32	1	2
$112^{2}\times 32$	bottleneck	1	16	1	1
$112^{2}\times 16$	bottleneck	6	24	2	2
$56^{2}\times 24$	bottleneck	6	32	3	2
$28^{2}\times 32$	bottleneck	6	64	4	2
$14^{2}\times 64$	bottleneck	6	96	3	1
$14^{2}\times 96$	bottleneck	6	160	3	2
$7^{2}\times 160$	bottleneck	6	320	1	1
$7^{2}\times 320$	conv2d 1x1	-	1280	1	1
$7^{2}\times 1280$	avgpool 7x7	-	-	1	-
$1\times 1\times 1280$	conv2d 1x1	-	k	-

表 2：MobileNetV2：每一行描述 1 个或更多相同（步幅除外）的层，重复 $n$ 次。同一序列中的所有层具有相同的输出通道数 $c$。每个序列的第一层步幅为 $s$，其余层步幅为 $1$。所有空间卷积使用 $3\times 3$ 核。扩展因子 $t$ 始终按表 1 所述作用于输入大小。

Size	MobileNetV1	MobileNetV2	ShuffleNet
			(2x,g=3)
112x112	64/1600	16/400	32/800
56x56	128/800	32/200	48/300
28x28	256/400	64/100	400/600K
14x14	512/200	160/62	800/310
7x7	1024/199	320/32	1600/156
1x1	1024/2	1280/2	1600/3
max	1600K	400K	600K

Table 3: The max number of channels/memory (in Kb) that needs to be materialized at each spatial resolution for different architectures. We assume 16-bit floats for activations. For ShuffleNet, we use $2x,g=3$ that matches the performance of MobileNetV1 and MobileNetV2. For the first layer of MobileNetV2 and ShuffleNet we can employ the trick described in Section 4 to reduce memory requirement. Even though ShuffleNet employs bottlenecks elsewhere, the non-bottleneck tensors still need to be materialized due to the presence of shortcuts between the non-bottleneck tensors.

大小	MobileNetV1	MobileNetV2	ShuffleNet
			(2x,g=3)
112x112	64/1600	16/400	32/800
56x56	128/800	32/200	48/300
28x28	256/400	64/100	400/600K
14x14	512/200	160/62	800/310
7x7	1024/199	320/32	1600/156
1x1	1024/2	1280/2	1600/3
最大	1600K	400K	600K

表 3：不同架构在各空间分辨率下需要物化的最大通道数/内存（Kb）。激活值按 16 位浮点计算。对 ShuffleNet，我们使用与 MobileNetV1 和 MobileNetV2 性能相当的 $2x,g=3$。对 MobileNetV2 和 ShuffleNet 的第一层，可以采用第 4 节描述的技巧降低内存需求。尽管 ShuffleNet 在其他地方也使用瓶颈，但由于捷径连在非瓶颈张量之间，非瓶颈张量仍需物化。

Trade-off hyper parameters

As in MobilenetV1 we tailor our architecture to different performance points, by using the input image resolution and width multiplier as tunable hyper parameters, that can be adjusted depending on desired accuracy/performance trade-offs. Our primary network (width multiplier $1$, $224\times 224$), has a computational cost of 300 million multiply-adds and uses 3.4 million parameters. We explore the performance trade offs, for input resolutions from $96$ to $224$, and width multipliers of $0.35$ to $1.4$. The network computational cost ranges from $7$ multiply adds to 585M MAdds, while the model size vary between 1.7M and 6.9M parameters.

权衡超参数

与 MobileNetV1 一样，我们用输入图像分辨率和宽度乘数作为可调超参数，按所需的精度/性能权衡来裁剪架构。主网络（宽度乘数 $1$，$224\times 224$）的计算代价为 3 亿次乘法–加法，参数量 340 万。我们探索输入分辨率从 $96$ 到 $224$、宽度乘数从 $0.35$ 到 $1.4$ 的性能权衡。网络计算代价从 7 次乘法–加法到 5.85 亿 MAdd，模型大小则在 170 万到 690 万参数之间。

One minor implementation difference, with MobilenetV1 is that for multipliers less than one, we apply width multiplier to all layers except the very last convolutional layer. This improves performance for smaller models.

与 MobileNetV1 的一处细微实现差异是：当乘数小于 1 时，宽度乘数作用于除最后一层卷积外的所有层。这对较小模型有帮助。

5 Implementation Notes

5 实现说明

Figure 4: Comparison of convolutional blocks for different architectures. ShuffleNet uses Group Convolutions [20] and shuffling, it also uses conventional residual approach where inner blocks are narrower than output. ShuffleNet and NasNet illustrations are from respective papers.

图 4：不同架构卷积块的比较。ShuffleNet 使用分组卷积和通道混洗，并沿用内部块比输出更窄的传统残差做法。ShuffleNet 与 NasNet 的示意图来自各自论文。

5.1 Memory efficient inference

5.1 内存高效的推理

The inverted residual bottleneck layers allow a particularly memory efficient implementation which is very important for mobile applications. A standard efficient implementation of inference that uses for instance TensorFlowtensorflow2015-whitepaper or Caffe caffe, builds a directed acyclic compute hypergraph $G$, consisting of edges representing the operations and nodes representing tensors of intermediate computation. The computation is scheduled in order to minimize the total number of tensors that needs to be stored in memory. In the most general case, it searches over all plausible computation orders $\Sigma(G)$ and picks the one that minimizes

倒残差瓶颈层允许一种对移动应用非常重要的内存高效实现。例如 TensorFlow 或 Caffe 中常见的高效推理实现，会构建有向无环计算超图 $G$，边表示运算，节点表示中间计算的张量。调度计算是为了尽量减少需要存进内存的张量总数。最一般的情形是搜索所有合理的计算顺序 $\Sigma(G)$，并选取使下式最小的那个：

$$M(G)=\min_{\pi\in\Sigma(G)}\max_{i\in 1..n}\left[\sum_{A\in R(i,\pi,G)}|A|\right]+\text{size}(\pi_{i}).$$

where $R(i,\pi,G)$ is the list of intermediate tensors that are connected to any of $\pi_{i}\dots\pi_{n}$ nodes, $|A|$ represents the size of the tensor $A$ and $size(i)$ is the total amount of memory needed for internal storage during operation $i$.

其中 $R(i,\pi,G)$ 是连接到 $\pi_{i}\dots\pi_{n}$ 中任一节点的中间张量列表，$|A|$ 表示张量 $A$ 的大小，$size(i)$ 是运算 $i$ 期间内部存储所需的总内存。

For graphs that have only trivial parallel structure (such as residual connection), there is only one non-trivial feasible computation order, and thus the total amount and a bound on the memory needed for inference on compute graph $G$ can be simplified:

对于只有平凡并行结构（例如残差连接）的图，只有一种非平凡的可行计算顺序，因此在计算图 $G$ 上推理所需的内存总量及其上界可以简化为：

$$M(G)=\max_{op\in G}\left[\sum_{A\in\text{op}_{inp}}|A|+\sum_{B\in\text{op}_{out}}|B|+|op|\right] \tag{2}$$

Or to restate, the amount of memory is simply the maximum total size of combined inputs and outputs across all operations. In what follows we show that if we treat a bottleneck residual block as a single operation (and treat inner convolution as a disposable tensor), the total amount of memory would be dominated by the size of bottleneck tensors, rather than the size of tensors that are internal to bottleneck (and much larger).

换句话说，内存量就是所有运算中输入与输出合计大小的最大值。下面我们说明：若把一个瓶颈残差块当作单一运算（并把内部卷积当作可丢弃张量），则总内存将由瓶颈张量的大小主导，而不是由瓶颈内部（大得多的）张量大小主导。

Bottleneck Residual Block

A bottleneck block operator ${\cal F}(x)$ shown in Figure 3(b) can be expressed as a composition of three operators ${\cal F}(x)=[A\circ{\cal N}\circ B]x$, where $A$ is a linear transformation $A:{\cal R}^{s\times s\times k}\rightarrow{\cal R}^{s\times s\times n}$, ${\cal N}$ is a non-linear per-channel transformation: ${\cal N}:{\cal R}^{s\times s\times n}\rightarrow{\cal R}^{s^{\prime}\times s^{\prime}\times n}$, and $B$ is again a linear transformation to the output domain: $B:{\cal R}^{s^{\prime}\times s^{\prime}\times n}\rightarrow{\cal R}^{s^{\prime}\times s^{\prime}\times k^{\prime}}$.

瓶颈残差块

图 3(b) 所示的瓶颈块算子 ${\cal F}(x)$ 可以写成三个算子的复合 ${\cal F}(x)=[A\circ{\cal N}\circ B]x$，其中 $A$ 是线性变换 $A:{\cal R}^{s\times s\times k}\rightarrow{\cal R}^{s\times s\times n}$，${\cal N}$ 是逐通道的非线性变换：${\cal N}:{\cal R}^{s\times s\times n}\rightarrow{\cal R}^{s^{\prime}\times s^{\prime}\times n}$，$B$ 再次是到输出域的线性变换：$B:{\cal R}^{s^{\prime}\times s^{\prime}\times n}\rightarrow{\cal R}^{s^{\prime}\times s^{\prime}\times k^{\prime}}$。

For our networks ${\cal N}={\operatorname{\mathop{ReLU6}\,}}\circ{\operatorname{\mathop{dwise}\,}}\circ{\operatorname{\mathop{ReLU6}\,}}$, but the results apply to any per-channel transformation. Suppose the size of the input domain is $|x|$ and the size of the output domain is $|y|$, then the memory required to compute $F(X)$ can be as low as $|s^{2}k|+|s^{\prime 2}k^{\prime}|+O(\max(s^{2},s^{\prime 2}))$.

对我们的网络，${\cal N}={\operatorname{\mathop{ReLU6}\,}}\circ{\operatorname{\mathop{dwise}\,}}\circ{\operatorname{\mathop{ReLU6}\,}}$，但结论适用于任何逐通道变换。设输入域大小为 $|x|$、输出域大小为 $|y|$，则计算 $F(X)$ 所需内存可以低至 $|s^{2}k|+|s^{\prime 2}k^{\prime}|+O(\max(s^{2},s^{\prime 2}))$。

The algorithm is based on the fact that the inner tensor $\cal I$ can be represented as concatenation of $t$ tensors, of size $n/t$ each and our function can then be represented as

该算法基于如下事实：内部张量 $\cal I$ 可以表示成 $t$ 个大小各为 $n/t$ 的张量的拼接，于是我们的函数可以写成

$${\cal F}(x)=\sum_{i=1}^{t}(A_{i}\circ N\circ B_{i})(x)$$

by accumulating the sum, we only require one intermediate block of size $n/t$ to be kept in memory at all times. Using $n=t$ we end up having to keep only a single channel of the intermediate representation at all times. The two constraints that enabled us to use this trick is (a) the fact that the inner transformation (which includes non-linearity and depthwise) is per-channel, and (b) the consecutive non-per-channel operators have significant ratio of the input size to the output. For most of the traditional neural networks, such trick would not produce a significant improvement.

通过累加求和，我们始终只需在内存中保留一块大小为 $n/t$ 的中间块。取 $n=t$ 时，就始终只需保留中间表征的单个通道。使我们能使用这一技巧的两个约束是：(a) 内部变换（包含非线性和深度卷积）是逐通道的；(b) 相邻的非逐通道算子在输入大小与输出大小之间有显著比例。对大多数传统神经网络，这一技巧不会带来明显改进。

We note that, the number of multiply-adds operators needed to compute $F(X)$ using $t$-way split is independent of $t$, however in existing implementations we find that replacing one matrix multiplication with several smaller ones hurts runtime performance due to increased cache misses. We find that this approach is the most helpful to be used with $t$ being a small constant between $2$ and $5$. It significantly reduces the memory requirement, but still allows one to utilize most of the efficiencies gained by using highly optimized matrix multiplication and convolution operators provided by deep learning frameworks. It remains to be seen if special framework level optimization may lead to further runtime improvements.

我们注意到，用 $t$ 路拆分计算 $F(X)$ 所需的乘法–加法次数与 $t$ 无关；但在现有实现中，用若干较小的矩阵乘法替换一次大的矩阵乘法，会因缓存未命中增加而损害运行时性能。我们发现，把 $t$ 取成 2 到 5 之间的小常数时这一方法最有用。它显著降低内存需求，同时仍能利用深度学习框架提供的高度优化的矩阵乘法和卷积算子。特殊的框架级优化是否还能进一步改善运行时，仍有待观察。

6 Experiments

6 实验

Figure 5: Performance curve of MobileNetV2 vs MobileNetV1, ShuffleNet, NAS. For our networks we use multipliers $0.35$, $0.5$, $0.75$, $1.0$ for all resolutions, and additional $1.4$ for for $224$. Best viewed in color.

图 5：MobileNetV2 相对 MobileNetV1、ShuffleNet、NAS 的性能曲线。对我们的网络，所有分辨率使用乘数 $0.35$、$0.5$、$0.75$、$1.0$，对 $224$ 额外使用 $1.4$。请看彩色图。

Figure 6: The impact of non-linearities and various types of shortcut (residual) connections.

图 6：非线性以及不同捷径（残差）连接方式的影响。

6.1 ImageNet Classification

6.1 ImageNet 分类

Training setup

We train our models using TensorFlowtensorflow2015-whitepaper. We use the standard RMSPropOptimizer with both decay and momentum set to $0.9$. We use batch normalization after every layer, and the standard weight decay is set to $0.00004$. Following MobileNetV1MobilenetV1 setup we use initial learning rate of $0.045$, and learning rate decay rate of $0.98$ per epoch. We use 16 GPU asynchronous workers, and a batch size of $96$.

训练设置

我们用 TensorFlow 训练模型。优化器为标准 RMSPropOptimizer，衰减和动量都设为 $0.9$。每一层之后使用批归一化，标准权重衰减设为 $0.00004$。沿用 MobileNetV1 的设置，初始学习率为 $0.045$，每轮衰减率为 $0.98$。我们使用 16 个 GPU 异步 worker，批次大小为 $96$。

Results

We compare our networks against MobileNetV1, ShuffleNet and NASNet-A models. The statistics of a few selected models is shown in Table 4 with the full performance graph shown in Figure 5.

结果

我们把网络与 MobileNetV1、ShuffleNet 和 NASNet-A 模型比较。若干选定模型的统计见表 4，完整性能图见图 5。

Network	Top 1	Params	MAdds	CPU
MobileNetV1	70.6	4.2M	575M	113ms
ShuffleNet (1.5)	71.5	3.4M	292M	-
ShuffleNet (x2)	73.7	5.4M	524M	-
NasNet-A	74.0	5.3M	564M	183ms
MobileNetV2	72.0	3.4M	300M	75ms
MobileNetV2 (1.4)	74.7	6.9M	585M	143ms

Table 4: Performance on ImageNet, comparison for different networks. As is common practice for ops, we count the total number of Multiply-Adds. In the last column we report running time in milliseconds (ms) for a single large core of the Google Pixel 1 phone (using TF-Lite). We do not report ShuffleNet numbers as efficient group convolutions and shuffling are not yet supported.

网络	Top 1	参数量	MAdds	CPU
MobileNetV1	70.6	4.2M	575M	113ms
ShuffleNet (1.5)	71.5	3.4M	292M	-
ShuffleNet (x2)	73.7	5.4M	524M	-
NasNet-A	74.0	5.3M	564M	183ms
MobileNetV2	72.0	3.4M	300M	75ms
MobileNetV2 (1.4)	74.7	6.9M	585M	143ms

表 4：ImageNet 上不同网络的性能比较。按惯例，运算量统计乘法–加法总数。最后一列是在 Google Pixel 1 手机的单个大核上（使用 TF-Lite）的运行时间（毫秒）。我们未报告 ShuffleNet 的时间，因为高效分组卷积和通道混洗尚未得到支持。

6.2 Object Detection

6.2 目标检测

We evaluate and compare the performance of MobileNetV2 and MobileNetV1 as feature extractors huang2016speed for object detection with a modified version of the Single Shot Detector (SSD) liu2016ssd on COCO dataset COCO. We also compare to YOLOv2 redmon2016yolo9000 and original SSD (with VGG-16 VGGNet as base network) as baselines. We do not compare performance with other architectures such as Faster-RCNN ren2015faster and RFCN dai2016rfcn since our focus is on mobile/real-time models.

我们把 MobileNetV2 和 MobileNetV1 作为特征提取器，用修改版 Single Shot Detector（SSD）在 COCO 数据集上评估并比较目标检测表现，同时以 YOLOv2 和原始 SSD（以 VGG-16 为骨干）作为基线。我们不与 Faster-RCNN 和 RFCN 等其他架构比较性能，因为关注点是移动/实时模型。

SSDLite: In this paper, we introduce a mobile friendly variant of regular SSD. We replace all the regular convolutions with separable convolutions (depthwise followed by $1\times 1$ projection) in SSD prediction layers. This design is in line with the overall design of MobileNets and is seen to be much more computationally efficient. We call this modified version SSDLite. Compared to regular SSD, SSDLite dramatically reduces both parameter count and computational cost as shown in Table 5.

SSDLite：本文引入一种更适合移动端的常规 SSD 变体。我们在 SSD 预测层中把所有常规卷积替换成可分离卷积（深度卷积后接 $1\times 1$ 投影）。这一设计与 MobileNets 的整体思路一致，计算上也高效得多。我们把这一修改版称为 SSDLite。如表 5 所示，相对常规 SSD，SSDLite 显著减少了参数量和计算代价。

	Params	MAdds
SSDliu2016ssd	14.8M	1.25B
SSDLite	2.1M	0.35B

Table 5: Comparison of the size and the computational cost between SSD and SSDLite configured with MobileNetV2 and making predictions for $80$ classes.

	参数量	MAdds
SSD	14.8M	1.25B
SSDLite	2.1M	0.35B

表 5：与 MobileNetV2 搭配、预测 $80$ 类时，SSD 与 SSDLite 在规模和计算代价上的比较。

For MobileNetV1, we follow the setup in huang2016speed. For MobileNetV2, the first layer of SSDLite is attached to the expansion of layer 15 (with output stride of 16). The second and the rest of SSDLite layers are attached on top of the last layer (with output stride of $32$). This setup is consistent with MobileNetV1 as all layers are attached to the feature map of the same output strides.

对 MobileNetV1，我们沿用已有设置。对 MobileNetV2，SSDLite 的第一层接到第 15 层的扩展输出（输出步幅为 16）。第二层及之后的 SSDLite 层接在最后一层之上（输出步幅为 $32$）。这一设置与 MobileNetV1 一致，因为所有层都接到相同输出步幅的特征图上。

Both MobileNet models are trained and evaluated with Open Source TensorFlow Object Detection API detection-api. The input resolution of both models is $320\times 320$. We benchmark and compare both mAP (COCO challenge metrics), number of parameters and number of Multiply-Adds. The results are shown in Table 6. MobileNetV2 SSDLite is not only the most efficient model, but also the most accurate of the three. Notably, MobileNetV2 SSDLite is $20\times$ more efficient and $10\times$ smaller while still outperforms YOLOv2 on COCO dataset.

两个 MobileNet 模型都用开源 TensorFlow Object Detection API 训练和评估，输入分辨率为 $320\times 320$。我们比较 mAP（COCO 挑战指标）、参数量和乘法–加法次数，结果见表 6。MobileNetV2 SSDLite 不仅是三者中最高效的模型，也是其中最准确的。值得注意的是，MobileNetV2 SSDLite 效率约为 YOLOv2 的 $20\times$、体积约为其 $1/10$，在 COCO 上仍超过 YOLOv2。

Network	mAP	Params	MAdd	CPU
SSD300liu2016ssd	23.2	36.1M	35.2B	-
SSD512liu2016ssd	26.8	36.1M	99.5B	-
YOLOv2redmon2016yolo9000	21.6	50.7M	17.5B	-
MNet V1 + SSDLite	22.2	5.1M	1.3B	270ms
MNet V2 + SSDLite	22.1	4.3M	0.8B	200ms

Table 6: Performance comparison of MobileNetV2 + SSDLite and other realtime detectors on the COCO dataset object detection task. MobileNetV2 + SSDLite achieves competitive accuracy with significantly fewer parameters and smaller computational complexity. All models are trained on trainval35k and evaluated on test-dev. SSD/YOLOv2 numbers are from redmon2016yolo9000. The running time is reported for the large core of the Google Pixel 1 phone, using an internal version of the TF-Lite engine.

网络	mAP	参数量	MAdd	CPU
SSD300	23.2	36.1M	35.2B	-
SSD512	26.8	36.1M	99.5B	-
YOLOv2	21.6	50.7M	17.5B	-
MNet V1 + SSDLite	22.2	5.1M	1.3B	270ms
MNet V2 + SSDLite	22.1	4.3M	0.8B	200ms

表 6：MobileNetV2 + SSDLite 与其他实时检测器在 COCO 目标检测任务上的性能比较。MobileNetV2 + SSDLite 以显著更少的参数和更小的计算复杂度达到有竞争力的精度。所有模型在 trainval35k 上训练，在 test-dev 上评估。SSD/YOLOv2 数字来自相应原文。运行时间是在 Google Pixel 1 手机的大核上、使用内部版 TF-Lite 引擎测得。

6.3 Semantic Segmentation

6.3 语义分割

In this section, we compare MobileNetV1 and MobileNetV2 models used as feature extractors with DeepLabv3 DeepLabV3 for the task of mobile semantic segmentation. DeepLabv3 adopts atrous convolution Holschneider1989real; Sermanet2013Overfeat; Papandreou2014untangling, a powerful tool to explicitly control the resolution of computed feature maps, and builds five parallel heads including (a) Atrous Spatial Pyramid Pooling module (ASPP) DeepLabV2 containing three $3\times 3$ convolutions with different atrous rates, (b) $1\times 1$ convolution head, and (c) Image-level features ParseNet. We denote by $\emph{output\_stride}$ the ratio of input image spatial resolution to final output resolution, which is controlled by applying the atrous convolution properly. For semantic segmentation, we usually employ $\emph{output\_stride}=16$ or $8$ for denser feature maps. We conduct the experiments on the PASCAL VOC 2012 dataset PASCAL, with extra annotated images from Hariharan2011semantic and evaluation metric mIOU.

本节比较把 MobileNetV1 和 MobileNetV2 作为特征提取器、与 DeepLabv3 搭配做移动端语义分割的表现。DeepLabv3 采用空洞卷积来显式控制计算特征图的分辨率，并构建五个并行头，包括：(a) 含三个不同空洞率 $3\times 3$ 卷积的空洞空间金字塔池化模块（ASPP）；(b) $1\times 1$ 卷积头；(c) 图像级特征。我们用 $\emph{output\_stride}$ 表示输入图像空间分辨率与最终输出分辨率之比，通过恰当使用空洞卷积来控制。语义分割通常采用 $\emph{output\_stride}=16$ 或 $8$，以得到更密的特征图。实验在 PASCAL VOC 2012 数据集上进行，并使用额外标注图像，评价指标为 mIOU。

To build a mobile model, we experimented with three design variations: (1) different feature extractors, (2) simplifying the DeepLabv3 heads for faster computation, and (3) different inference strategies for boosting the performance. Our results are summarized in Table 7. We have observed that: (a) the inference strategies, including multi-scale inputs and adding left-right flipped images, significantly increase the MAdds and thus are not suitable for on-device applications, (b) using $\emph{output\_stride}=16$ is more efficient than $\emph{output\_stride}=8$, (c) MobileNetV1 is already a powerful feature extractor and only requires about $4.9-5.7$ times fewer MAdds than ResNet-101 ResNet (e.g., mIOU: 78.56 $\emph{vs}$ 82.70, and MAdds: 941.9B $\emph{vs}$ 4870.6B), (d) it is more efficient to build DeepLabv3 heads on top of the second last feature map of MobileNetV2 than on the original last-layer feature map, since the second to last feature map contains $320$ channels instead of $1280$, and by doing so, we attain similar performance, but require about $2.5$ times fewer operations than the MobileNetV1 counterparts, and (e) DeepLabv3 heads are computationally expensive and removing the ASPP module significantly reduces the MAdds with only a slight performance degradation. In the end of the Table 7, we identify a potential candidate for on-device applications (in bold face), which attains $75.32\%$ mIOU and only requires $2.75$B MAdds.

为构建移动模型，我们试验了三种设计变化：(1) 不同的特征提取器，(2) 简化 DeepLabv3 的头以加快计算，(3) 不同的推理策略以提升表现。结果汇总于表 7。我们观察到：(a) 包括多尺度输入和左右翻转图像在内的推理策略会显著增加 MAdds，因而不适合端侧应用；(b) 使用 $\emph{output\_stride}=16$ 比 $\emph{output\_stride}=8$ 更高效；(c) MobileNetV1 已经是很强的特征提取器，MAdds 大约只需 ResNet-101 的 $1/4.9$ 到 $1/5.7$（例如 mIOU：78.56 对 82.70，MAdds：941.9B 对 4870.6B）；(d) 把 DeepLabv3 的头建在 MobileNetV2 倒数第二张特征图上，比建在原来的最后一层特征图上更高效，因为倒数第二张特征图有 $320$ 个通道而不是 $1280$，这样做能达到相近表现，运算量大约只需 MobileNetV1 对应设置的 $1/2.5$；(e) DeepLabv3 的头计算昂贵，去掉 ASPP 模块会显著减少 MAdds，性能只略有下降。表 7 末尾我们标出一个适合端侧应用的候选（加粗），它达到 $75.32\%$ mIOU，只需 $2.75$B MAdds。

Network	OS	ASPP	MF	mIOU	Params	MAdds
MNet V1	16	✓		75.29	11.15M	14.25B
	8	✓	✓	78.56	11.15M	941.9B
MNet V2*	16	✓		75.70	4.52M	5.8B
	8	✓	✓	78.42	4.52M	387B
MNet V2*	16			75.32	2.11M	2.75B
	8		✓	77.33	2.11M	152.6B
ResNet-101	16	✓		80.49	58.16M	81.0B
	8	✓	✓	82.70	58.16M	4870.6B

Table 7: MobileNet + DeepLabv3 inference strategy on the PASCAL VOC 2012 validation set. MNet V2*: Second last feature map is used for DeepLabv3 heads, which includes (1) Atrous Spatial Pyramid Pooling (ASPP) module, and (2) $1\times 1$ convolution as well as image-pooling feature. OS: output_stride that controls the output resolution of the segmentation map. MF: Multi-scale and left-right flipped inputs during test. All of the models have been pretrained on COCO. The potential candidate for on-device applications is shown in bold face. PASCAL images have dimension $512\times 512$ and atrous convolution allows us to control output feature resolution without increasing the number of parameters.

网络	OS	ASPP	MF	mIOU	参数量	MAdds
MNet V1	16	✓		75.29	11.15M	14.25B
	8	✓	✓	78.56	11.15M	941.9B
MNet V2*	16	✓		75.70	4.52M	5.8B
	8	✓	✓	78.42	4.52M	387B
MNet V2*	16			75.32	2.11M	2.75B
	8		✓	77.33	2.11M	152.6B
ResNet-101	16	✓		80.49	58.16M	81.0B
	8	✓	✓	82.70	58.16M	4870.6B

表 7：MobileNet + DeepLabv3 在 PASCAL VOC 2012 验证集上的推理策略。MNet V2*：DeepLabv3 的头使用倒数第二张特征图，包括 (1) 空洞空间金字塔池化（ASPP）模块，以及 (2) $1\times 1$ 卷积和图像池化特征。OS：控制分割图输出分辨率的 output_stride。MF：测试时使用多尺度和左右翻转输入。所有模型都在 COCO 上预训练。适合端侧应用的候选以粗体标出。PASCAL 图像尺寸为 $512\times 512$，空洞卷积使我们能在不增加参数量的情况下控制输出特征分辨率。

6.4 Ablation study

6.4 消融实验

Inverted residual connections. The importance of residual connection has been studied extensively ResNet; ResNext2016; InceptionV4. The new result reported in this paper is that the shortcut connecting bottleneck perform better than shortcuts connecting the expanded layers (see Figure 6(b) for comparison).

倒残差连接。 残差连接的重要性已有大量研究。本文报告的新结果是：连接瓶颈的捷径比连接扩展层的捷径表现更好（比较见图 6(b)）。

Importance of linear bottlenecks. The linear bottleneck models are strictly less powerful than models with non-linearities, because the activations can always operate in linear regime with appropriate changes to biases and scaling. However our experiments shown in Figure 6(a) indicate that linear bottlenecks improve performance, providing support that non-linearity destroys information in low-dimensional space.

线性瓶颈的重要性。 线性瓶颈模型在能力上严格弱于带非线性的模型，因为只要适当改变偏置和缩放，激活总可以工作在线性区间。然而图 6(a) 的实验表明，线性瓶颈能提升性能，这支持了“非线性会破坏低维空间中的信息”这一判断。

7 Conclusions and future work

7 结论与未来工作

We described a very simple network architecture that allowed us to build a family of highly efficient mobile models. Our basic building unit, has several properties that make it particularly suitable for mobile applications. It allows very memory-efficient inference and relies utilize standard operations present in all neural frameworks.

我们描述了一种非常简单的网络架构，用以构建一族高效的移动模型。基本构建单元有若干特别适合移动应用的性质：它允许非常节省内存的推理，并依赖所有神经框架中都有的标准运算。

For the ImageNet dataset, our architecture improves the state of the art for wide range of performance points.

在 ImageNet 数据集上，我们的架构在很宽的性能区间上提升了当时最优水平。

For object detection task, our network outperforms state-of-art realtime detectors on COCO dataset both in terms of accuracy and model complexity. Notably, our architecture combined with the SSDLite detection module is $20\times$ less computation and $10\times$ less parameters than YOLOv2.

在目标检测任务上，我们的网络在 COCO 数据集上同时在精度和模型复杂度上超过当时最优的实时检测器。值得注意的是，该架构与 SSDLite 检测模块结合后，计算量约为 YOLOv2 的 $1/20$，参数量约为其 $1/10$。

On the theoretical side: the proposed convolutional block has a unique property that allows to separate the network expressiveness (encoded by expansion layers) from its capacity (encoded by bottleneck inputs). Exploring this is an important direction for future research.

在理论方面：所提出的卷积块有一个独特性质，可以把网络的表达能力（由扩展层编码）与容量（由瓶颈输入编码）分开。探索这一点是未来研究的重要方向。

Acknowledgments

We would like to thank Matt Streeter and Sergey Ioffe for their helpful feedback and discussion.

致谢

感谢 Matt Streeter 和 Sergey Ioffe 提供的有益反馈与讨论。

References

参考文献

[1] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vision , 115(3):211–252, December 2015.

[2] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV , 2014.

[3] Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserma. The pascal visual object classes challenge – a retrospective. IJCV , 2014.

[4] Mobilenetv2 source code. Available from https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet .

[5] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Bartlett et al. [ 48 ] , pages 1106–1114.

[6] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR , abs/1409.1556, 2014.

[7] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015 , pages 1–9. IEEE Computer Society, 2015.

[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR , abs/1512.03385, 2015.

[9] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research , 13:281–305, 2012.

[10] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Bartlett et al. [ 48 ] , pages 2960–2968.

[11] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 , volume 37 of JMLR Workshop and Conference Proceedings , pages 2171–2180. JMLR.org, 2015.

[12] Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Stephen Jose Hanson, Jack D. Cowan, and C. Lee Giles, editors, Advances in Neural Information Processing Systems 5, [NIPS Conference, Denver, Colorado, USA, November 30 - December 3, 1992] , pages 164–171. Morgan Kaufmann, 1992.

[13] Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In David S. Touretzky, editor, Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989] , pages 598–605. Morgan Kaufmann, 1989.

[14] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 1135–1143, 2015.

[15] Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Shijian Tang, Erich Elsen, Bryan Catanzaro, John Tran, and William J. Dally. DSD: regularizing deep neural networks with dense-sparse-dense training flow. CoRR , abs/1607.04381, 2016.

[16] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 1379–1387, 2016.

[17] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. CoRR , abs/1608.08710, 2016.

[18] Karim Ahmed and Lorenzo Torresani. Connectivity learning in multi-branch networks. CoRR , abs/1709.09582, 2017.

[19] Tom Veniat and Ludovic Denoyer. Learning time-efficient deep architectures with budgeted super networks. CoRR , abs/1706.00046, 2017.

[20] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. CoRR , abs/1707.01083, 2017.

[21] Soravit Changpinyo, Mark Sandler, and Andrey Zhmoginov. The power of sparsity in convolutional neural networks. CoRR , abs/1702.06257, 2017.

[22] Min Wang, Baoyuan Liu, and Hassan Foroosh. Design of efficient convolutional layers using single intra-channel convolution, topological subdivisioning and spatial ”bottleneck” structure. CoRR , abs/1608.04337, 2016.

[23] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. CoRR , abs/1707.07012, 2017.

[24] Lingxi Xie and Alan L. Yuille. Genetic CNN. CoRR , abs/1703.01513, 2017.

[25] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 2902–2911. PMLR, 2017.

[26] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. CoRR , abs/1611.01578, 2016.

[27] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR , abs/1704.04861, 2017.

[28] Francois Chollet. Xception: Deep learning with depthwise separable convolutions. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.

[29] Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. CoRR , abs/1610.02915, 2016.

[30] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. CoRR , abs/1611.05431, 2016.

[31] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.

[32] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 , 2014.

[33] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In CVPR , 2017.

[34] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV , 2016.

[35] Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. arXiv preprint arXiv:1612.08242 , 2016.

[36] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems , pages 91–99, 2015.

[37] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fcn: Object detection via region-based fully convolutional networks. In Advances in neural information processing systems , pages 379–387, 2016.

[38] Jonathan Huang, Vivek Rathod, Derek Chow, Chen Sun, and Menglong Zhu. Tensorflow object detection api, 2017.

[39] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR , abs/1706.05587, 2017.

[40] Matthias Holschneider, Richard Kronland-Martinet, Jean Morlet, and Ph Tchamitchian. A real-time algorithm for signal analysis with the help of the wavelet transform. In Wavelets: Time-Frequency Methods and Phase Space , pages 289–297. 1989.

[41] Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv:1312.6229 , 2013.

[42] George Papandreou, Iasonas Kokkinos, and Pierre-Andre Savalle. Modeling local and global deformations in deep learning: Epitomic convolution, multiple instance learning, and sliding window detection. In CVPR , 2015.

[43] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI , 2017.

[44] Wei Liu, Andrew Rabinovich, and Alexander C. Berg. Parsenet: Looking wider to see better. CoRR , abs/1506.04579, 2015.

[45] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV , 2011.

[46] Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR , abs/1602.07261, 2016.

[47] Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems , NIPS’14, pages 2924–2932, Cambridge, MA, USA, 2014. MIT Press.

[48] Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors. Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States , 2012.

Appendix A Bottleneck transformation

附录 A 瓶颈变换

In this section we study the properties of an operator $A\reluop(Bx)$, where $x\in{\cal R}^{n}$ represents an $n$-channel pixel, $B$ is an $m\times n$ matrix and $A$ is an $n\times m$ matrix. We argue that if $m\leq n$, transformations of this form can only exploit non-linearity at the cost of losing information. In contrast, if $n\ll m$, such transforms can be highly non-linear but still invertible with high probability (for the initial random weights).

本节研究算子 $A\reluop(Bx)$ 的性质，其中 $x\in{\cal R}^{n}$ 表示一个 $n$ 通道像素，$B$ 是 $m\times n$ 矩阵，$A$ 是 $n\times m$ 矩阵。我们指出：若 $m\leq n$，这种形式的变换只能以丢失信息为代价来利用非线性。相反，若 $n\ll m$，这类变换可以高度非线性，但在随机初始权重下仍以高概率可逆。

First we show that $\reluop$ is an identity transformation for any point that lies in the interior of its image.

我们首先证明：$\reluop$ 对其像的内部中的任何点都是恒等变换。

Lemma 1

Let $S(X)=\{\reluop(x)|x\in X\}$. If a volume of $S(X)$ is non-zero, then $\operatorname{interior}S(X)\subseteq X$.

引理 1

*令 $S(X)=\{\reluop(x)|x\in X\}$。若 $S(X)$ 的体积非零，则 $\operatorname{interior}S(X)\subseteq X$。

Proof:

Let $S^{\prime}=\operatorname{interior}\reluop(S)$. First we note that if $x\in S^{\prime}$, then $x_{i}>0$ for all $i$. Indeed, image of $\reluop$ does not contain points with negative coordinates, and points with zero-valued coordinates can not be interior points. Therefore for each $x\in S^{\prime}$, $x=\reluop(x)$ as desired. $\blacksquare$

证明：

令 $S^{\prime}=\operatorname{interior}\reluop(S)$。首先注意到，若 $x\in S^{\prime}$，则对所有 $i$ 都有 $x_{i}>0$。的确，$\reluop$ 的像不包含带负坐标的点，而带零值坐标的点不可能是内点。因此对每个 $x\in S^{\prime}$，都有 $x=\reluop(x)$。$\blacksquare$

It follows that for an arbitrary composition of interleaved linear transformation and $\reluop$ operators, if it preserves non-zero volume, that part of the input space $X$ that is preserved over such a composition is a linear transformation, and thus is likely to be a minor contributor to the power of deep networks. However, this is a fairly weak statement. Indeed, if the input manifold can be embedded into $(n-1)$-dimensional manifold (out of $n$ dimensions total), the lemma is trivially true, since the starting volume is $0$. In what follows we show that when the dimensionality of input manifold is significantly lower we can ensure that there will be no information loss.

由此可知，对任意交错的线性变换与 $\reluop$ 算子的复合，若它保持非零体积，则输入空间 $X$ 中被这种复合保留下来的那部分是一次线性变换，因而多半不是深度网络能力的主要来源。不过这是一个相当弱的陈述。的确，若输入流形可以嵌入（总共 $n$ 维中的）$(n-1)$ 维流形，引理就平凡成立，因为起始体积为 $0$。下面我们说明：当输入流形的维数显著更低时，可以保证没有信息损失。

Since the $\reluop(x)$ nonlinearity is a surjective function mapping the entire ray $x\leq 0$ to $0$, using this nonlinearity in a neural network can result in information loss. Once $\reluop$ collapses a subset of the input manifold to a smaller-dimensional output, the following network layers can no longer distinguish between collapsed input samples. In the following, we show that bottlenecks with sufficiently large expansion layers are resistant to information loss caused by the presence of $\reluop$ activation functions.

由于 $\reluop(x)$ 非线性是把整条射线 $x\leq 0$ 映射到 $0$ 的满射，在神经网络中使用这一非线性可能导致信息损失。一旦 $\reluop$ 把输入流形的一个子集塌缩到更低维的输出，后续网络层就无法再区分这些被塌缩的输入样本。下面我们说明，带有足够大扩展层的瓶颈能够抵抗由 $\reluop$ 激活函数造成的信息损失。

Figure 7: Distribution of activation patterns. The x-axis is the layer index, and we show minimum/maximum/average number of positive channels after each convolution with ReLU. y-axis is either absolute or relative number of channels. The "threshold" line indicates the ReLU invertibility threshold - that is the number of positive dimensions is higher than the input space. In our case this is 1/6 fraction of the channels. Note how at the beginning of the training on Figure 7a the distribution is much more tightly concentrated around the mean. After the training has finished (Figure 7b), the average hasn't changed but the standard deviation grew dramatically. Best viewed in color.

图 7：激活模式的分布。横轴是层序号，我们展示每层带 ReLU 的卷积之后正通道的最小/最大/平均值。纵轴是通道数的绝对值或相对值。“threshold”线表示 ReLU 可逆的阈值，也就是正维度数目高于输入空间的维数；在我们这里，这相当于通道数的 1/6。注意，训练开始时（图 7a）分布更紧密地集中在均值附近；训练结束后（图 7b），平均值几乎没变，但标准差显著增大。请看彩色图。

Lemma 2(Invertibility of ReLU)

Consider an operator $\reluop(Bx)$, where $B$ is an $m\times n$ matrix and $x\in{\cal R}^{n}$. Let $y_{0}=\reluop(Bx_{0})$ for some $x_{0}\in{\cal R}^{n}$, then equation $y_{0}=\reluop(Bx)$ has a unique solution with respect to $x$ if and only if $y_{0}$ has at least $n$ non-zero values and there are $n$ linearly independent rows of $B$ that correspond to non-zero coordinates of $y_{0}$.

引理 2（ReLU 的可逆性）

考虑算子 $\reluop(Bx)$，其中 $B$ 是 $m\times n$ 矩阵，$x\in{\cal R}^{n}$。设对某个 $x_{0}\in{\cal R}^{n}$ 有 $y_{0}=\reluop(Bx_{0})$，则方程 $y_{0}=\reluop(Bx)$ 关于 $x$ 有唯一解，当且仅当 $y_{0}$ 至少有 $n$ 个非零值，并且 $B$ 中有 $n$ 个线性无关的行对应于 $y_{0}$ 的非零坐标。

Proof:

Denote the set of non-zero coordinates of $y_{0}$ as $T$ and let $y_{T}$ and $B_{T}$ be restrictions of $y$ and $B$ to the subspace defined by $T$. If $|T|<n$, we have $y_{T}=B_{T}x_{0}$ where $B_{T}$ is under-determined with at least one solution $x_{0}$, thus there are infinitely many solutions. Now consider the case of $|T|\geq n$ and let the rank of $B_{T}$ be $n$. Suppose there is an additional solution $x_{1}\neq x_{0}$ such that $y_{0}=\reluop(Bx_{1})$, then we have $y_{T}=B_{T}x_{0}=B_{T}x_{1}$, which cannot be satisfied unless $x_{0}=x_{1}$. $\blacksquare$

证明：

把 $y_{0}$ 的非零坐标集合记为 $T$，并令 $y_{T}$ 和 $B_{T}$ 为 $y$ 和 $B$ 在 $T$ 所定义子空间上的限制。若 $|T|<n$，则有 $y_{T}=B_{T}x_{0}$，其中 $B_{T}$ 欠定且至少有一个解 $x_{0}$，因此有无穷多解。现在考虑 $|T|\geq n$ 且 $B_{T}$ 的秩为 $n$ 的情形。假设另有解 $x_{1}\neq x_{0}$ 使得 $y_{0}=\reluop(Bx_{1})$，则 $y_{T}=B_{T}x_{0}=B_{T}x_{1}$，除非 $x_{0}=x_{1}$，否则无法成立。$\blacksquare$

One of the corollaries of this lemma says that if $m\gg n$, we only need a small fraction of values of $Bx$ to be positive for $\reluop(Bx)$ to be invertible.

该引理的一个推论是：若 $m\gg n$，只需 $Bx$ 中一小部分值为正，$\reluop(Bx)$ 就可以可逆。

The constraints of the lemma 2 can be empirically validated for real networks and real inputs and hence we can be assured that information is indeed preserved. We further show that with respect to initialization, we can be sure that these constraints are satisfied with high probability. Note that for random initialization the conditions of lemma 2 are satisfied due to initialization symmetries. However even for trained graphs these constraints can be empirically validated by running the network over valid inputs and verifying that all or most inputs are above the threshold. On Figure 7 we show how this distribution looks for different MobileNetV2 layers. At step $0$ the activation patterns concentrate around having half of the positive channel (as predicted by initialization symmetries). For fully trained network, while the standard deviation grew significantly, all but the two layers are still above the invertibility thresholds. We believe further study of this is warranted and might lead to helpful insights on network design.

引理 2 的约束可以在真实网络和真实输入上经验验证，因而我们可以确信信息确实被保留。我们进一步说明，就初始化而言，这些约束会以高概率得到满足。注意，随机初始化时，由于初始化对称性，引理 2 的条件是满足的。即便对训练后的图，也可以通过在有效输入上运行网络、验证全部或大多数输入高于阈值，来经验地验证这些约束。图 7 展示了这一分布在不同 MobileNetV2 层上的样子。第 0 步时，激活模式集中在大约一半通道为正（与初始化对称性的预测一致）。对完全训练的网络，尽管标准差显著增大，除两层外所有层仍高于可逆性阈值。我们相信这一点值得进一步研究，并可能对网络设计带来有益洞见。

Theorem 1

Let $S$ be a compact $n$-dimensional submanifold of ${\cal R}^{n}$. Consider a family of functions $f_{B}(x)=\reluop(Bx)$ from ${\cal R}^{n}$ to ${\cal R}^{m}$ parameterized by $m\times n$ matrices $B\in\mathcal{B}$. Let $p(B)$ be a probability density on the space of all matrices $\mathcal{B}$ that satisfies:

定理 1

设 $S$ 是 ${\cal R}^{n}$ 的紧致 $n$ 维子流形。考虑由 $m\times n$ 矩阵 $B\in\mathcal{B}$ 参数化的一族函数 $f_{B}(x)=\reluop(Bx)$，它们从 ${\cal R}^{n}$ 映到 ${\cal R}^{m}$。设 $p(B)$ 是所有矩阵空间 $\mathcal{B}$ 上的概率密度，并满足：

• • $P(Z)=0$for any measure-zero subset$Z\subset\mathcal{B}$;

• • 对任何测度为零的子集 $Z\subset\mathcal{B}$，有 $P(Z)=0$；

• • (a symmetry condition)$p(DB)=p(B)$for any$B\in\mathcal{B}$and any$m\times m$diagonal matrix$D$with all diagonal elements being either$+1$or$-1$.

• • （对称性条件）对任何 $B\in\mathcal{B}$ 以及任何对角元全为 $+1$ 或 $-1$ 的 $m\times m$ 对角矩阵 $D$，有 $p(DB)=p(B)$。

Then, the average $n$-volume of the subset of $S$ that is collapsed by $f_{B}$ to a lower-dimensional manifold is

那么，被 $f_{B}$ 塌缩到更低维流形的 $S$ 的子集的平均 $n$ 维体积为

$$V-\frac{N_{m,n}V}{2^{m}},$$

where $V=\vol S$ and

其中 $V=\vol S$，并且

$$N_{m,n}\equiv\sum_{k=0}^{m-n}\binom{m}{k}.$$

Proof:

For any $\sigma=(s_{1},\dots,s_{m})$ with $s_{k}\in\{-1,+1\}$, let $Q_{\sigma}=\{x\in{\cal R}^{m}|x_{i}s_{i}>0\}$ be a corresponding quadrant in ${\cal R}^{m}$. For any $n$-dimensional submanifold $\Gamma\subset{\cal R}^{m}$, $\reluop$ acts as a bijection on $\Gamma\cap Q_{\sigma}$ if $\sigma$ has at least $n$ positive values[^4] and contracts $\Gamma\cap Q_{\sigma}$ otherwise. Also notice that the intersection of $BS$ with ${\cal R}^{m}\backslash(\cup_{\sigma}Q_{\sigma})$ is almost surely $(n-1)$-dimensional. The average $n$-volume of $S$ that is not collapsed by applying $\reluop$ to $BS$ is therefore given by:

证明：

对任意 $\sigma=(s_{1},\dots,s_{m})$ 且 $s_{k}\in\{-1,+1\}$，令 $Q_{\sigma}=\{x\in{\cal R}^{m}|x_{i}s_{i}>0\}$ 为 ${\cal R}^{m}$ 中相应的卦限。对任何 $n$ 维子流形 $\Gamma\subset{\cal R}^{m}$，若 $\sigma$ 至少有 $n$ 个正值，则 $\reluop$ 在 $\Gamma\cap Q_{\sigma}$ 上是双射[^4]，否则会收缩 $\Gamma\cap Q_{\sigma}$。还要注意，$BS$ 与 ${\cal R}^{m}\backslash(\cup_{\sigma}Q_{\sigma})$ 的交几乎必然是 $(n-1)$ 维的。因此，对 $BS$ 施加 $\reluop$ 后不被塌缩的 $S$ 的平均 $n$ 维体积由下式给出：

$$\sum_{\sigma\in\Sigma_{n}}\mathbb{E}_{B}[V_{\sigma}(B)], \tag{3}$$

where $\Sigma_{n}=\left\{(s_{1},\dots,s_{m})|\sum_{k}\theta(s_{k})\geq n\right\}$, $\theta$ is a step function and $V_{\sigma}(B)$ is a volume of the largest subset of $S$ that is mapped by $B$ to $Q_{\sigma}$. Now let us calculate $\mathbb{E}_{B}[V_{\sigma}(B)]$. Recalling that $p(DB)=p(B)$ for any $D=\diag(s_{1},\dots,s_{m})$ with $s_{k}\in\{-1,+1\}$, this average can be rewritten as $\mathbb{E}_{B}\mathbb{E}_{D}[V_{\sigma}(DB)]$. Noticing that the subset of $S$ mapped by $DB$ to $Q_{\sigma}$ is also mapped by $B$ to $D^{-1}Q_{\sigma}$, we immediately obtain $\sum_{\sigma^{\prime}}V_{\sigma}[\diag(\sigma^{\prime})B]=\sum_{\sigma^{\prime}}V_{\sigma^{\prime}}[B]=\vol S$ and therefore $\mathbb{E}_{B}[V_{\sigma}(B)]=2^{-m}\,\vol S$. Substituting this and $|\Sigma_{n}|=\sum_{k=0}^{m-n}\binom{m}{k}$ into Eq. 3 concludes the proof. $\blacksquare$

其中 $\Sigma_{n}=\left\{(s_{1},\dots,s_{m})|\sum_{k}\theta(s_{k})\geq n\right\}$，$\theta$ 是阶跃函数，$V_{\sigma}(B)$ 是 $S$ 中被 $B$ 映射到 $Q_{\sigma}$ 的最大子集的体积。现在计算 $\mathbb{E}_{B}[V_{\sigma}(B)]$。回忆对任何 $D=\diag(s_{1},\dots,s_{m})$ 且 $s_{k}\in\{-1,+1\}$ 都有 $p(DB)=p(B)$，这一平均可以改写为 $\mathbb{E}_{B}\mathbb{E}_{D}[V_{\sigma}(DB)]$。注意到被 $DB$ 映射到 $Q_{\sigma}$ 的 $S$ 的子集，也被 $B$ 映射到 $D^{-1}Q_{\sigma}$，我们立即得到 $\sum_{\sigma^{\prime}}V_{\sigma}[\diag(\sigma^{\prime})B]=\sum_{\sigma^{\prime}}V_{\sigma^{\prime}}[B]=\vol S$，因此 $\mathbb{E}_{B}[V_{\sigma}(B)]=2^{-m}\,\vol S$。把这一点以及 $|\Sigma_{n}|=\sum_{k=0}^{m-n}\binom{m}{k}$ 代入公式 3，即完成证明。$\blacksquare$