Momentum Contrast for Unsupervised Visual Representation Learning

动量对比：用于无监督视觉表示学习

Kaiming He    Haoqi Fan    Yuxin Wu    Saining Xie    Ross Girshick Affiliation: Facebook AI Research (FAIR)

Abstract

摘要

We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning [29] as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.

我们提出动量对比（Momentum Contrast, MoCo），用于无监督视觉表示学习。从把对比学习 [29] 看作词典查找的视角出发，我们用队列与动量平均编码器构建动态词典，从而能在线维护一个又大又一致的词典，便于对比式无监督学习。MoCo 在 ImageNet 线性分类协议下表现有竞争力；更重要的是，学到的表示能较好迁移到下游任务。在 PASCAL VOC、COCO 等数据集上的 7 项检测/分割任务中，MoCo 可超过其有监督预训练对照，有时领先幅度很大。这表明在许多视觉任务上，无监督与有监督表示学习之间的差距已被大幅缩小。

Code: https://github.com/facebookresearch/moco

代码：https://github.com/facebookresearch/moco

1 Introduction

1 引言

Unsupervised representation learning is highly successful in natural language processing, e.g., as shown by GPT [50, 51] and BERT [12]. But supervised pre-training is still dominant in computer vision, where unsupervised methods generally lag behind. The reason may stem from differences in their respective signal spaces. Language tasks have discrete signal spaces (words, sub-word units, etc.) for building tokenized dictionaries, on which unsupervised learning can be based. Computer vision, in contrast, further concerns dictionary building [54, 9, 5], as the raw signal is in a continuous, high-dimensional space and is not structured for human communication (e.g., unlike words).

无监督表示学习在自然语言处理中非常成功，例如 GPT [50, 51] 与 BERT [12]。但在计算机视觉里，有监督预训练仍占主导，无监督方法普遍落后。原因可能来自信号空间的差异：语言任务有离散信号空间（词、子词等），便于构建分词词典并据此做无监督学习；视觉则还要面对词典构建本身 [54, 9, 5]，因为原始信号处于连续、高维空间，且并非为人际交流而结构化（例如不像词）。

Several recent studies [61, 46, 36, 66, 35, 56, 2] present promising results on unsupervised visual representation learning using approaches related to the contrastive loss [29]. Though driven by various motivations, these methods can be thought of as building dynamic dictionaries. The “keys” (tokens) in the dictionary are sampled from data (e.g., images or patches) and are represented by an encoder network. Unsupervised learning trains encoders to perform dictionary look-up: an encoded “query” should be similar to its matching key and dissimilar to others. Learning is formulated as minimizing a contrastive loss [29].

若干近期研究 [61, 46, 36, 66, 35, 56, 2] 用与对比损失 [29] 相关的方法，在无监督视觉表示学习上给出了有希望的结果。动机各异，但这些方法都可看作在构建动态词典：词典中的「键」（token）从数据（如图像或图块）采样，并由编码器网络表示。无监督学习训练编码器去做词典查找：编码后的「查询」应与匹配键相似、与其他键不相似。学习被表述为最小化对比损失 [29]。

From this perspective, we hypothesize that it is desirable to build dictionaries that are: (i) large and (ii) consistent as they evolve during training. Intuitively, a larger dictionary may better sample the underlying continuous, high-dimensional visual space, while the keys in the dictionary should be represented by the same or similar encoder so that their comparisons to the query are consistent. However, existing methods that use contrastive losses can be limited in one of these two aspects (discussed later in context).

由此我们假设，理想词典应同时满足：(i) 规模大，以及 (ii) 随训练演化时保持一致。直观上，更大的词典能更好采样底层连续高维视觉空间；词典中的键应由相同或相近的编码器表示，以便与查询的比较一致。然而，现有使用对比损失的方法往往在这两点之一上受限（后文再讨论）。

Figure 1: Momentum Contrast (MoCo) trains a visual representation encoder by matching an encoded query $q$ to a dictionary of encoded keys using a contrastive loss. The dictionary keys $\{k_{0},k_{1},k_{2},...\}$ are defined on-the-fly by a set of data samples. The dictionary is built as a queue, with the current mini-batch enqueued and the oldest mini-batch dequeued, decoupling it from the mini-batch size. The keys are encoded by a slowly progressing encoder, driven by a momentum update with the query encoder. This method enables a large and consistent dictionary for learning visual representations.

图 1：动量对比（MoCo）用对比损失训练视觉表示编码器，使编码查询 $q$ 匹配由编码键构成的词典。词典键 $\{k_{0},k_{1},k_{2},...\}$ 由一组数据样本在线定义。词典以队列形式维护：当前 mini-batch 入队、最旧 mini-batch 出队，从而与 mini-batch 大小解耦。键由缓慢推进的编码器编码，该编码器相对查询编码器做动量更新。该方法使学习视觉表示时能拥有又大又一致的词典。

We present Momentum Contrast (MoCo) as a way of building large and consistent dictionaries for unsupervised learning with a contrastive loss (Figure 1). We maintain the dictionary as a queue of data samples: the encoded representations of the current mini-batch are enqueued, and the oldest are dequeued. The queue decouples the dictionary size from the mini-batch size, allowing it to be large. Moreover, as the dictionary keys come from the preceding several mini-batches, a slowly progressing key encoder, implemented as a momentum-based moving average of the query encoder, is proposed to maintain consistency.

我们提出动量对比（MoCo），作为用对比损失做无监督学习时构建又大又一致词典的一种方式（图 1）。词典维护为数据样本的队列：当前 mini-batch 的编码表示入队，最旧的出队。队列使词典大小与 mini-batch 大小解耦，从而可以很大。此外，由于词典键来自此前若干 mini-batch，我们提出用查询编码器的动量滑动平均实现缓慢推进的键编码器，以保持一致性。

MoCo is a mechanism for building dynamic dictionaries for contrastive learning, and can be used with various pretext tasks. In this paper, we follow a simple instance discrimination task [61, 63, 2]: a query matches a key if they are encoded views (e.g., different crops) of the same image. Using this pretext task, MoCo shows competitive results under the common protocol of linear classification in the ImageNet dataset [11].

MoCo 是为对比学习构建动态词典的机制，可与多种 pretext 任务配合。本文沿用简单的实例判别任务 [61, 63, 2]：若查询与键是同一图像的不同编码视图（例如不同裁剪），则匹配。在此 pretext 下，MoCo 在 ImageNet [11] 常见线性分类协议上表现有竞争力。

A main purpose of unsupervised learning is to pre-train representations (i.e., features) that can be transferred to downstream tasks by fine-tuning. We show that in 7 downstream tasks related to detection or segmentation, MoCo unsupervised pre-training can surpass its ImageNet supervised counterpart, in some cases by nontrivial margins. In these experiments, we explore MoCo pre-trained on ImageNet or on a one-billion Instagram image set, demonstrating that MoCo can work well in a more real-world, billion-image scale, and relatively uncurated scenario. These results show that MoCo largely closes the gap between unsupervised and supervised representation learning in many computer vision tasks, and can serve as an alternative to ImageNet supervised pre-training in several applications.

无监督学习的一个主要目的是预训练可迁移到下游、经微调使用的表示（即特征）。我们表明：在 7 项与检测或分割相关的下游任务中，MoCo 无监督预训练可超过其 ImageNet 有监督对照，有时差距不小。实验中我们探索在 ImageNet 或约十亿张 Instagram 图像集上预训练 MoCo，说明它能在更接近真实世界、十亿量级且相对未精选的场景下工作良好。这些结果表明 MoCo 在许多计算机视觉任务上大幅缩小了无监督与有监督表示学习的差距，并可在若干应用中作为 ImageNet 有监督预训练的替代。

2 Related Work

2 相关工作

Unsupervised/self-supervised[^1] learning methods generally involve two aspects: pretext tasks and loss functions. The term “pretext” implies that the task being solved is not of genuine interest, but is solved only for the true purpose of learning a good data representation. Loss functions can often be investigated independently of pretext tasks. MoCo focuses on the loss function aspect. Next we discuss related studies with respect to these two aspects.

无监督/自监督[^1]学习方法通常涉及两方面：pretext 任务与损失函数。「pretext」意味着所解任务本身并非真正兴趣所在，求解它只是为了学到好的数据表示。损失函数常可与 pretext 任务分开研究。MoCo 聚焦损失层面。下面按这两方面讨论相关工作。

Loss functions.

A common way of defining a loss function is to measure the difference between a model’s prediction and a fixed target, such as reconstructing the input pixels (e.g., auto-encoders) by L1 or L2 losses, or classifying the input into pre-defined categories (e.g., eight positions [13], color bins [64]) by cross-entropy or margin-based losses. Other alternatives, as described next, are also possible.

损失函数

定义损失的常见方式是度量模型预测与固定目标的差异，例如用 L1/L2 重建输入像素（如自编码器），或用交叉熵/间隔损失把输入分到预定义类别（如八个位置 [13]、颜色箱 [64]）。如下所述，也有其他选择。

Contrastive losses [29] measure the similarities of sample pairs in a representation space. Instead of matching an input to a fixed target, in contrastive loss formulations the target can vary on-the-fly during training and can be defined in terms of the data representation computed by a network [29]. Contrastive learning is at the core of several recent works on unsupervised learning [61, 46, 36, 66, 35, 56, 2], which we elaborate on later in context (Sec. 3.1).

对比损失 [29] 度量表示空间中样本对的相似度。与把输入匹配到固定目标不同，对比损失表述里目标可在训练中动态变化，并可用网络算出的数据表示来定义 [29]。对比学习是若干近期无监督学习工作 [61, 46, 36, 66, 35, 56, 2] 的核心，后文（第 3.1 节）再展开。

Adversarial losses [24] measure the difference between probability distributions. It is a widely successful technique for unsupervised data generation. Adversarial methods for representation learning are explored in [15, 16]. There are relations (see [24]) between generative adversarial networks and noise-contrastive estimation (NCE) [28].

对抗损失 [24] 度量概率分布差异，是无监督数据生成中广泛成功的技术。[15, 16] 探索了用于表示学习的对抗方法。生成对抗网络与噪声对比估计（NCE）[28] 之间存在联系（见 [24]）。

Pretext tasks.

A wide range of pretext tasks have been proposed. Examples include recovering the input under some corruption, e.g., denoising auto-encoders [58], context auto-encoders [48], or cross-channel auto-encoders (colorization) [64, 65]. Some pretext tasks form pseudo-labels by, e.g., transformations of a single (“exemplar”) image [17], patch orderings [13, 45], tracking [59] or segmenting objects [47] in videos, or clustering features [3, 4].

Pretext 任务

已有大量 pretext 任务被提出。例子包括在某种损坏下恢复输入，如去噪自编码器 [58]、上下文自编码器 [48]，或跨通道自编码器（着色）[64, 65]。另一些用伪标签，例如单张（「exemplar」）图像的变换 [17]、图块排序 [13, 45]、视频中的跟踪 [59] 或分割物体 [47]，或对特征聚类 [3, 4]。

Contrastive learning vs. pretext tasks.

Various pretext tasks can be based on some form of contrastive loss functions. The instance discrimination method [61] is related to the exemplar-based task [17] and NCE [28]. The pretext task in contrastive predictive coding (CPC) [46] is a form of context auto-encoding [48], and in contrastive multiview coding (CMC) [56] it is related to colorization [64].

对比学习 与 pretext 任务

多种 pretext 可用某种形式的对比损失。实例判别方法 [61] 与基于 exemplar 的任务 [17] 及 NCE [28] 相关。对比预测编码（CPC）[46] 的 pretext 是一种上下文自编码 [48]，对比多视图编码（CMC）[56] 则与着色 [64] 相关。

3 Method

3 方法

3.1 Contrastive Learning as Dictionary Look-up

3.1 作为词典查找的对比学习

Contrastive learning [29], and its recent developments, can be thought of as training an encoder for a dictionary look-up task, as described next.

对比学习 [29] 及其近期发展，可看作训练编码器去完成词典查找任务，如下所述。

Consider an encoded query $q$ and a set of encoded samples $\{k_{0},k_{1},k_{2},...\}$ that are the keys of a dictionary. Assume that there is a single key (denoted as $k_{+}$) in the dictionary that $q$ matches. A contrastive loss [29] is a function whose value is low when $q$ is similar to its positive key $k_{+}$ and dissimilar to all other keys (considered negative keys for $q$). With similarity measured by dot product, a form of a contrastive loss function, called InfoNCE [46], is considered in this paper:

考虑编码查询 $q$ 以及作为词典键的一组编码样本 $\{k_{0},k_{1},k_{2},...\}$。假设词典中有唯一键（记为 $k_{+}$）与 $q$ 匹配。对比损失 [29] 在 $q$ 与正键 $k_{+}$ 相似、与其他键（相对 $q$ 的负键）不相似时取值较低。用点积度量相似度时，本文采用一种称为 InfoNCE [46] 的对比损失形式：

$$\small\mathcal{L}_{q}=-\log\frac{\exp(q{\cdot}k_{+}/\tau)}{\sum_{i=0}^{K}\exp(q{\cdot}k_{i}/\tau)} \tag{1}$$

where $\tau$ is a temperature hyper-parameter per [61]. The sum is over one positive and $K$ negative samples. Intuitively, this loss is the log loss of a $({K}{+}1)$-way softmax-based classifier that tries to classify $q$ as $k_{+}$. Contrastive loss functions can also be based on other forms [29, 59, 61, 36], such as margin-based losses and variants of NCE losses.

其中 $\tau$ 为温度超参，取自 [61]。求和覆盖一个正样本与 $K$ 个负样本。直观上，该损失是 $(K{+}1)$ 类 softmax 分类器的对数损失，试图把 $q$ 分到 $k_{+}$。对比损失也可基于其他形式 [29, 59, 61, 36]，如间隔损失与 NCE 变体。

Figure 2: Conceptual comparison of three contrastive loss mechanisms (empirical comparisons are in Figure 3 and Table 3). Here we illustrate one pair of query and key. The three mechanisms differ in how the keys are maintained and how the key encoder is updated. (a): The encoders for computing the query and key representations are updated end-to-end by back-propagation (the two encoders can be different). (b): The key representations are sampled from a memory bank [61]. (c): MoCo encodes the new keys on-the-fly by a momentum-updated encoder, and maintains a queue (not illustrated in this figure) of keys.

图 2：三种对比损失机制的概念比较（经验比较见图 3 与表 3）。这里示意一对查询与键。三种机制的差别在于如何维护键、如何更新键编码器。(a)：计算查询与键表示的编码器由反向传播端到端更新（两编码器可以不同）。(b)：键表示从记忆库 [61] 采样。(c)：MoCo 用动量更新的编码器在线编码新键，并维护键的队列（本图未画出）。

The contrastive loss serves as an unsupervised objective function for training the encoder networks that represent the queries and keys [29]. In general, the query representation is $q=f_{\textrm{q}}(x^{q})$ where $f_{\textrm{q}}$ is an encoder network and $x^{q}$ is a query sample (likewise, $k=f_{\textrm{k}}(x^{k})$). Their instantiations depend on the specific pretext task. The input $x^{q}$ and $x^{k}$ can be images [29, 61, 63], patches [46], or context consisting a set of patches [46]. The networks $f_{\textrm{q}}$ and $f_{\textrm{k}}$ can be identical [29, 59, 63], partially shared [46, 36, 2], or different [56].

对比损失作为训练表示查询与键的编码器网络的无监督目标 [29]。一般地，查询表示为 $q=f_{\textrm{q}}(x^{q})$，其中 $f_{\textrm{q}}$ 是编码器、$x^{q}$ 是查询样本（同理 $k=f_{\textrm{k}}(x^{k})$）。具体实例取决于 pretext 任务。输入 $x^{q}$、$x^{k}$ 可以是图像 [29, 61, 63]、图块 [46]，或由一组图块构成的上下文 [46]。网络 $f_{\textrm{q}}$ 与 $f_{\textrm{k}}$ 可以相同 [29, 59, 63]、部分共享 [46, 36, 2]，或不同 [56]。

3.2 Momentum Contrast

3.2 动量对比

From the above perspective, contrastive learning is a way of building a discrete dictionary on high-dimensional continuous inputs such as images. The dictionary is dynamic in the sense that the keys are randomly sampled, and that the key encoder evolves during training. Our hypothesis is that good features can be learned by a large dictionary that covers a rich set of negative samples, while the encoder for the dictionary keys is kept as consistent as possible despite its evolution. Based on this motivation, we present Momentum Contrast as described next.

从上述视角看，对比学习是在图像这类高维连续输入上构建离散词典的一种方式。词典是动态的：键随机采样，键编码器随训练演化。我们的假设是：用大词典覆盖丰富负样本，同时尽管编码器在演化仍尽量保持键编码器一致，就能学到好特征。基于此动机，我们提出动量对比，如下。

Dictionary as a queue.

At the core of our approach is maintaining the dictionary as a queue of data samples. This allows us to reuse the encoded keys from the immediate preceding mini-batches. The introduction of a queue decouples the dictionary size from the mini-batch size. Our dictionary size can be much larger than a typical mini-batch size, and can be flexibly and independently set as a hyper-parameter.

词典作为队列

方法核心是把词典维护为数据样本的队列。这样可以复用紧邻前几个 mini-batch 编码出的键。引入队列使词典大小与 mini-batch 大小解耦：词典可以远大于典型 mini-batch，并可作为超参灵活独立设定。

The samples in the dictionary are progressively replaced. The current mini-batch is enqueued to the dictionary, and the oldest mini-batch in the queue is removed. The dictionary always represents a sampled subset of all data, while the extra computation of maintaining this dictionary is manageable. Moreover, removing the oldest mini-batch can be beneficial, because its encoded keys are the most outdated and thus the least consistent with the newest ones.

词典中的样本逐步被替换：当前 mini-batch 入队，队列中最旧的 mini-batch 移除。词典始终代表全体数据的一个采样子集，维护开销可控。移除最旧 batch 也有好处，因其编码键最过时，与最新键的一致性也最差。

Momentum update.

Using a queue can make the dictionary large, but it also makes it intractable to update the key encoder by back-propagation (the gradient should propagate to all samples in the queue). A naïve solution is to copy the key encoder $f_{\textrm{k}}$ from the query encoder $f_{\textrm{q}}$, ignoring this gradient. But this solution yields poor results in experiments (Sec. 4.1). We hypothesize that such failure is caused by the rapidly changing encoder that reduces the key representations’ consistency. We propose a momentum update to address this issue.

动量更新

队列能让词典变大，但也使通过反向传播更新键编码器变得不可行（梯度需传到队列中所有样本）。朴素做法是把键编码器 $f_{\textrm{k}}$ 直接复制自查询编码器 $f_{\textrm{q}}$、忽略该梯度；实验中效果很差（第 4.1 节）。我们假设失败原因是编码器变化过快，降低了键表示一致性。为此提出动量更新。

Formally, denoting the parameters of $f_{\textrm{k}}$ as $\theta_{\textrm{k}}$ and those of $f_{\textrm{q}}$ as $\theta_{\textrm{q}}$, we update $\theta_{\textrm{k}}$ by:

形式地，记 $f_{\textrm{k}}$ 的参数为 $\theta_{\textrm{k}}$、$f_{\textrm{q}}$ 的参数为 $\theta_{\textrm{q}}$，按下式更新 $\theta_{\textrm{k}}$：

$$\small\theta_{\textrm{k}}\leftarrow m\theta_{\textrm{k}}+(1-m)\theta_{\textrm{q}}. \tag{2}$$

Here $m\in[0,1)$ is a momentum coefficient. Only the parameters $\theta_{\textrm{q}}$ are updated by back-propagation. The momentum update in Eqn.(2) makes $\theta_{\textrm{k}}$ evolve more smoothly than $\theta_{\textrm{q}}$. As a result, though the keys in the queue are encoded by different encoders (in different mini-batches), the difference among these encoders can be made small. In experiments, a relatively large momentum (e.g., $m$ $=$ 0.999, our default) works much better than a smaller value (e.g., $m$ $=$ $0.9$), suggesting that a slowly evolving key encoder is a core to making use of a queue.

其中 $m\in[0,1)$ 为动量系数。只有 $\theta_{\textrm{q}}$ 由反向传播更新。式 (2) 的动量更新使 $\theta_{\textrm{k}}$ 比 $\theta_{\textrm{q}}$ 演化更平滑。因此，尽管队列中的键由不同（不同 mini-batch 的）编码器编码，这些编码器之间的差异可以很小。实验中，相对较大的动量（例如默认 $m=0.999$）远好于较小值（例如 $m=0.9$），说明缓慢演化的键编码器是用好队列的核心。

Relations to previous mechanisms.

MoCo is a general mechanism for using contrastive losses. We compare it with two existing general mechanisms in Figure 2. They exhibit different properties on the dictionary size and consistency.

与先前机制的关系

MoCo 是使用对比损失的通用机制。我们在图 2 中与两种已有通用机制比较。它们在词典大小与一致性上呈现不同性质。

The end-to-end update by back-propagation is a natural mechanism (e.g., [29, 46, 36, 63, 2, 35], Figure 2a). It uses samples in the current mini-batch as the dictionary, so the keys are consistently encoded (by the same set of encoder parameters). But the dictionary size is coupled with the mini-batch size, limited by the GPU memory size. It is also challenged by large mini-batch optimization [25]. Some recent methods [46, 36, 2] are based on pretext tasks driven by local positions, where the dictionary size can be made larger by multiple positions. But these pretext tasks may require special network designs such as patchifying the input [46] or customizing the receptive field size [2], which may complicate the transfer of these networks to downstream tasks.

由反向传播做端到端更新是自然机制（例如 [29, 46, 36, 63, 2, 35]，图 2a）。它用当前 mini-batch 中的样本作词典，因此键编码一致（同一组编码器参数）。但词典大小与 mini-batch 大小耦合，受 GPU 显存限制，也受大 batch 优化难题 [25] 制约。一些近期方法 [46, 36, 2] 基于由局部位置驱动的 pretext，可用多位置把词典做大；但这些任务可能需要特殊网络设计，如把输入切块 [46] 或定制感受野 [2]，可能使网络向下游迁移更复杂。

Another mechanism is the memory bank approach proposed by [61] (Figure 2b). A memory bank consists of the representations of all samples in the dataset. The dictionary for each mini-batch is randomly sampled from the memory bank with no back-propagation, so it can support a large dictionary size. However, the representation of a sample in the memory bank was updated when it was last seen, so the sampled keys are essentially about the encoders at multiple different steps all over the past epoch and thus are less consistent. A momentum update is adopted on the memory bank in [61]. Its momentum update is on the representations of the same sample, not the encoder. This momentum update is irrelevant to our method, because MoCo does not keep track of every sample. Moreover, our method is more memory-efficient and can be trained on billion-scale data, which can be intractable for a memory bank.

另一种机制是 [61] 提出的记忆库（图 2b）。记忆库包含数据集中所有样本的表示；每个 mini-batch 的词典从记忆库随机采样且无反向传播，因此可支持大词典。但某样本在记忆库中的表示是在上次见到时更新的，采样出的键本质上来自过去整个 epoch 中多个不同步骤的编码器，一致性较差。[61] 也对记忆库采用动量更新，但其动量作用在同一样本的表示上，不是编码器上。该动量与我们的方法无关，因为 MoCo 不跟踪每个样本。此外，我们的方法更省内存，可在十亿级数据上训练，这对记忆库可能不可行。

Sec. 4 empirically compares these three mechanisms.

第 4 节经验比较这三种机制。

f_q, f_k: encoder networks for query and key

queue: dictionary as a queue of K keys (CxK)

m: momentum

t: temperature

f_k.params = f_q.params # initialize
for x in loader: # load a minibatch x with N samples
x_q = aug(x) # a randomly augmented version
x_k = aug(x) # another randomly augmented version

q = f_q.forward(x_q) # queries: NxC
k = f_k.forward(x_k) # keys: NxC
k = k.detach() # no gradient to keys

positive logits: Nx1

l_pos = bmm(q.view(N,1,C), k.view(N,C,1))

negative logits: NxK

l_neg = mm(q.view(N,C), queue.view(C,K))

logits: Nx(1+K)

logits = cat([l_pos, l_neg], dim=1)

contrastive loss, Eqn.(1)

labels = zeros(N) # positives are the 0-th
loss = CrossEntropyLoss(logits/t, labels)

SGD update: query network

loss.backward()
update(f_q.params)

momentum update: key network

f_k.params = m*f_k.params+(1-m)*f_q.params

update dictionary

enqueue(queue, k) # enqueue the current minibatch
dequeue(queue) # dequeue the earliest minibatch

Algorithm 1 Pseudocode of MoCo in a PyTorch-like style.

f_q, f_k: encoder networks for query and key

queue: dictionary as a queue of K keys (CxK)

m: momentum

t: temperature

f_k.params = f_q.params # initialize
for x in loader: # load a minibatch x with N samples
x_q = aug(x) # a randomly augmented version
x_k = aug(x) # another randomly augmented version

q = f_q.forward(x_q) # queries: NxC
k = f_k.forward(x_k) # keys: NxC
k = k.detach() # no gradient to keys

positive logits: Nx1

l_pos = bmm(q.view(N,1,C), k.view(N,C,1))

negative logits: NxK

l_neg = mm(q.view(N,C), queue.view(C,K))

logits: Nx(1+K)

logits = cat([l_pos, l_neg], dim=1)

contrastive loss, Eqn.(1)

labels = zeros(N) # positives are the 0-th
loss = CrossEntropyLoss(logits/t, labels)

SGD update: query network

loss.backward()
update(f_q.params)

momentum update: key network

f_k.params = m*f_k.params+(1-m)*f_q.params

update dictionary

enqueue(queue, k) # enqueue the current minibatch
dequeue(queue) # dequeue the earliest minibatch

算法 1 类 PyTorch 风格的 MoCo 伪代码。

3.3 Pretext Task

3.3 Pretext 任务

Contrastive learning can drive a variety of pretext tasks. As the focus of this paper is not on designing a new pretext task, we use a simple one mainly following the instance discrimination task in [61], to which some recent works [63, 2] are related.

对比学习可驱动多种 pretext 任务。本文重点不在设计新的 pretext，因而主要沿用 [61] 中的实例判别任务，近期若干工作 [63, 2] 也与之相关。

Following [61], we consider a query and a key as a positive pair if they originate from the same image, and otherwise as a negative sample pair. Following [63, 2], we take two random “views” of the same image under random data augmentation to form a positive pair. The queries and keys are respectively encoded by their encoders, $f_{\textrm{q}}$ and $f_{\textrm{k}}$. The encoder can be any convolutional neural network [39].

沿用 [61]，若查询与键来自同一图像则视为正对，否则为负对。沿用 [63, 2]，对同一图像在随机数据增强下取两个随机「视图」构成正对。查询与键分别由其编码器 $f_{\textrm{q}}$、$f_{\textrm{k}}$ 编码。编码器可以是任意卷积神经网络 [39]。

Algorithm 1 provides the pseudo-code of MoCo for this pretext task. For the current mini-batch, we encode the queries and their corresponding keys, which form the positive sample pairs. The negative samples are from the queue.

算法 1 给出了该 pretext 下 MoCo 的伪代码。对当前 mini-batch，我们编码查询及其对应键，构成正样本对；负样本来自队列。

Technical details.

We adopt a ResNet [33] as the encoder, whose last fully-connected layer (after global average pooling) has a fixed-dimensional output (128-D [61]). This output vector is normalized by its L2-norm [61]. This is the representation of the query or key. The temperature $\tau$ in Eqn.(1) is set as 0.07 [61]. The data augmentation setting follows [61]: a 224$\times$224-pixel crop is taken from a randomly resized image, and then undergoes random color jittering, random horizontal flip, and random grayscale conversion, all available in PyTorch’s torchvision package.

技术细节

编码器采用 ResNet [33]，全局平均池化后的最后全连接层输出固定维（128 维 [61]）。该输出向量按 L2 范数归一化 [61]，即查询或键的表示。式 (1) 中温度 $\tau$ 取 0.07 [61]。数据增强设定沿用 [61]：从随机缩放图像取 224×224 裁剪，再经随机颜色抖动、随机水平翻转与随机灰度转换，均可在 PyTorch 的 torchvision 中获得。

Shuffling BN.

Our encoders $f_{\textrm{q}}$ and $f_{\textrm{k}}$ both have Batch Normalization (BN) [37] as in the standard ResNet [33]. In experiments, we found that using BN prevents the model from learning good representations, as similarly reported in [35] (which avoids using BN). The model appears to “cheat” the pretext task and easily finds a low-loss solution. This is possibly because the intra-batch communication among samples (caused by BN) leaks information.

打乱 BN

编码器 $f_{\textrm{q}}$ 与 $f_{\textrm{k}}$ 都像标准 ResNet [33] 一样含 Batch Normalization（BN）[37]。实验中发现使用 BN 会妨碍学到好表示，与 [35] 的报告类似（后者干脆不用 BN）。模型似乎会「作弊」完成 pretext，轻易找到低损失解。可能原因是 BN 造成的 batch 内样本通信泄漏了信息。

We resolve this problem by shuffling BN. We train with multiple GPUs and perform BN on the samples independently for each GPU (as done in common practice). For the key encoder $f_{\textrm{k}}$, we shuffle the sample order in the current mini-batch before distributing it among GPUs (and shuffle back after encoding); the sample order of the mini-batch for the query encoder $f_{\textrm{q}}$ is not altered. This ensures the batch statistics used to compute a query and its positive key come from two different subsets. This effectively tackles the cheating issue and allows training to benefit from BN.

我们用打乱 BN 解决该问题。训练使用多 GPU，并在每个 GPU 上独立对样本做 BN（常见做法）。对键编码器 $f_{\textrm{k}}$，在把当前 mini-batch 分到各 GPU 之前打乱样本顺序（编码后再打乱回来）；查询编码器 $f_{\textrm{q}}$ 的 mini-batch 顺序不变。这样，计算查询及其正键所用的 batch 统计来自两个不同子集，有效抑制作弊，并让训练仍能受益于 BN。

We use shuffled BN in both our method and its end-to-end ablation counterpart (Figure 2a). It is irrelevant to the memory bank counterpart (Figure 2b), which does not suffer from this issue because the positive keys are from different mini-batches in the past.

打乱 BN 用于我们的方法及其端到端消融对照（图 2a）。与记忆库对照（图 2b）无关，因后者正键来自过去不同 mini-batch，不存在该问题。

4 Experiments

4 实验

We study unsupervised training performed in:

我们研究在以下数据上的无监督训练：

ImageNet-1M(IN-1M): This is the ImageNet [11] training set that has $\scriptstyle\sim$1.28 million images in 1000 classes (often called ImageNet-1K; we count the image number instead, as classes are not exploited by unsupervised learning). This dataset is well-balanced in its class distribution, and its images generally contain iconic view of objects.

ImageNet-1M**（IN-1M）：即 ImageNet [11] 训练集，约 $\scriptstyle\sim$128 万张图、1000 类（常称 ImageNet-1K；我们按图像数计，因无监督学习不利用类别）。该集类别分布较均衡，图像多为物体的标志性视角。

Instagram-1B(IG-1B): Following [44], this is a dataset of $\scriptstyle\sim$1 billion (940M) public images from Instagram. The images are from $\scriptstyle\sim$1500 hashtags [44] that are related to the ImageNet categories. This dataset is relatively uncurated comparing to IN-1M, and has a long-tailed, unbalanced distribution of real-world data. This dataset contains both iconic objects and scene-level images.

Instagram-1B（IG-1B）：沿用 [44]，约 10 亿（940M）张来自 Instagram 的公开图像，来自与 ImageNet 类别相关的约 1500 个 hashtag [44]。相对 IN-1M 更未精选*，呈真实世界数据的*长尾、不平衡分布，既含标志性物体也含场景级图像。

Training.

We use SGD as our optimizer. The SGD weight decay is 0.0001 and the SGD momentum is 0.9. For IN-1M, we use a mini-batch size of 256 ($N$ in Algorithm 1) in 8 GPUs, and an initial learning rate of 0.03. We train for 200 epochs with the learning rate multiplied by 0.1 at 120 and 160 epochs [61], taking $\scriptstyle\sim$53 hours training ResNet-50. For IG-1B, we use a mini-batch size of 1024 in 64 GPUs, and a learning rate of 0.12 which is exponentially decayed by 0.9$\times$ after every 62.5k iterations (64M images). We train for 1.25M iterations ($\scriptstyle\sim$1.4 epochs of IG-1B), taking $\scriptstyle\sim$6 days for ResNet-50.

训练

优化器用 SGD，权重衰减 0.0001，动量 0.9。IN-1M 上用 8 GPU、mini-batch 256（算法 1 中的 $N$），初始学习率 0.03；训 200 epoch，在第 120、160 epoch 将学习率乘 0.1 [61]，ResNet-50 约需 53 小时。IG-1B 上用 64 GPU、mini-batch 1024，学习率 0.12，每 62.5k 次迭代（约 64M 图）指数衰减 0.9×；训 1.25M 次迭代（约 IG-1B 的 1.4 epoch），ResNet-50 约需 6 天。

4.1 Linear Classification Protocol

4.1 线性分类协议

We first verify our method by linear classification on frozen features, following a common protocol. In this subsection we perform unsupervised pre-training on IN-1M. Then we freeze the features and train a supervised linear classifier (a fully-connected layer followed by softmax). We train this classifier on the global average pooling features of a ResNet, for 100 epochs. We report 1-crop, top-1 classification accuracy on the ImageNet validation set.

我们先在冻结特征上用线性分类验证方法，沿用常见协议。本小节在 IN-1M 上做无监督预训练，然后冻结特征，训练有监督线性分类器（全连接层加 softmax）。在 ResNet 的全局平均池化特征上训 100 epoch，报告 ImageNet 验证集上 1-crop top-1 分类准确率。

For this classifier, we perform a grid search and find the optimal initial learning rate is 30 and weight decay is 0 (similarly reported in [56]). These hyper-parameters perform consistently well for all ablation entries presented in this subsection. These hyper-parameter values imply that the feature distributions (e.g., magnitudes) can be substantially different from those of ImageNet supervised training, an issue we will revisit in Sec. 4.2.

对该分类器做网格搜索，最优初始学习率为 30、权重衰减为 0（[56] 也有类似报告）。这些超参对本小节所有消融条目表现一致。它们也暗示特征分布（例如量级）可能与 ImageNet 有监督训练明显不同，第 4.2 节会再讨论。

Ablation: contrastive loss mechanisms.

We compare the three mechanisms that are illustrated in Figure 2. To focus on the effect of contrastive loss mechanisms, we implement all of them in the same pretext task as described in Sec. 3.3. We also use the same form of InfoNCE as the contrastive loss function, Eqn.(1). As such, the comparison is solely on the three mechanisms.

消融：对比损失机制

比较图 2 所示三种机制。为聚焦机制本身，三者都实现为第 3.3 节同一 pretext，并用同一形式的 InfoNCE（式 1）作对比损失。因此比较仅针对三种机制。

The results are in Figure 3. Overall, all three mechanisms benefit from a larger $K$. A similar trend has been observed in [61, 56] under the memory bank mechanism, while here we show that this trend is more general and can be seen in all mechanisms. These results support our motivation of building a large dictionary.

结果见图 3。总体而言，三种机制都受益于更大的 $K$。记忆库机制下 [61, 56] 已观察到类似趋势；这里表明该趋势更一般，在所有机制中都能看到。这些结果支持我们构建大词典的动机。

The end-to-end mechanism performs similarly to MoCo when $K$ is small. However, the dictionary size is limited by the mini-batch size due to the end-to-end requirement. Here the largest mini-batch a high-end machine (8 Volta 32GB GPUs) can afford is 1024. More essentially, large mini-batch training is an open problem [25]: we found it necessary to use the linear learning rate scaling rule [25] here, without which the accuracy drops (by $\scriptstyle\sim$2% with a 1024 mini-batch). But optimizing with a larger mini-batch is harder [25], and it is questionable whether the trend can be extrapolated into a larger $K$ even if memory is sufficient.

端到端机制在 $K$ 较小时与 MoCo 相近。但因端到端要求，词典大小受 mini-batch 限制。高端机器（8 块 Volta 32GB GPU）能承受的最大 mini-batch 约为 1024。更本质的是，大 batch 训练仍是开放问题 [25]：此处必须使用线性学习率缩放规则 [25]，否则准确率下降（1024 batch 时约 2%）。但更大 batch 更难优化 [25]，即便显存够，趋势能否外推到更大 $K$ 也存疑。

The memory bank [61] mechanism can support a larger dictionary size. But it is 2.6% worse than MoCo. This is inline with our hypothesis: the keys in the memory bank are from very different encoders all over the past epoch and they are not consistent. Note the memory bank result of 58.0% reflects our improved implementation of [61].[^2]

记忆库 [61] 机制可支持更大词典，但比 MoCo 低 2.6%。这与我们的假设一致：记忆库中的键来自过去整个 epoch 中差异很大的编码器，不够一致。注意 58.0% 的记忆库结果反映的是我们对 [61] 的改进实现。[^2]

Figure 3: Comparison of three contrastive loss mechanisms under the ImageNet linear classification protocol. We adopt the same pretext task (Sec. 3.3) and only vary the contrastive loss mechanism (Figure 2). The number of negatives is $K$ in memory bank and MoCo, and is $K{-}1$ in end-to-end (offset by one because the positive key is in the same mini-batch). The network is ResNet-50.

图 3：在 ImageNet 线性分类协议下比较三种对比损失机制。采用同一 pretext（第 3.3 节），仅改变对比损失机制（图 2）。负样本数在记忆库与 MoCo 中为 $K$，在端到端中为 $K{-}1$（因正键同在当前 mini-batch，故偏移 1）。网络为 ResNet-50。

Ablation: momentum.

The table below shows ResNet-50 accuracy with different MoCo momentum values ($m$ in Eqn.(2)) used in pre-training ($K$ $=$ 4096 here) :

消融：动量

下表给出预训练中使用不同 MoCo 动量值（式 2 中的 $m$）时 ResNet-50 的准确率（此处 $K=4096$）：

It performs reasonably well when $m$ is in 0.99 $\scriptstyle\sim$ 0.9999, showing that a slowly progressing (i.e., relatively large momentum) key encoder is beneficial. When $m$ is too small (e.g., 0.9), the accuracy drops considerably; at the extreme of no momentum ($m$ is 0), the training loss oscillates and fails to converge. These results support our motivation of building a consistent dictionary.

当 $m$ 在 0.99 $\scriptstyle\sim$ 0.9999 时表现合理，说明缓慢推进（即相对较大动量）的键编码器有益。$m$ 过小（例如 0.9）时准确率明显下降；极端情况无动量（$m=0$）时训练损失振荡且无法收敛。这些结果支持构建一致词典的动机。

method	architecture	#params (M)	accuracy (%)
Exemplar [17]	R50w${3}{\times}$	211	46.0 [38]
RelativePosition [13]	R50w${2}{\times}$	94	51.4 [38]
Jigsaw [45]	R50w${2}{\times}$	94	44.6 [38]
Rotation [19]	Rv50w${4}{\times}$	86	55.4 [38]
Colorization [64]	R101∗	28	39.6 [14]
DeepCluster [3]	VGG [53]	15	48.4 [4]
BigBiGAN [16]	R50	24	56.6
	Rv50w${4}{\times}$	86	61.3
methods based on contrastive learning follow:
InstDisc [61]	R50	24	54.0
LocalAgg [66]	R50	24	58.8
CPC v1 [46]	R101∗	28	48.7
CPC v2 [35]	R170${}^{*}_{\text{wider}}$	303	65.9
CMC [56]	R50${}_{\text{L+ab}}$	47	64.1†
	R50w${2}{\times}$${}_{\text{L+ab}}$	188	68.4†
AMDIM [2]	AMDIM${}_{\text{small}}$	194	63.5†
	AMDIM${}_{\text{large}}$	626	68.1†
MoCo	R50	24	60.6
	RX50	46	63.9
	R50w${2}{\times}$	94	65.4
	R50w${4}{\times}$	375	68.6

Table 1: **Comparison under the linear classification protocol on ImageNet. The figure visualizes the table. All are reported as unsupervised pre-training on the ImageNet-1M training set, followed by supervised linear classification trained on frozen features, evaluated on the validation set. The parameter counts are those of the feature extractors. We compare with improved re-implementations if available (referenced after the numbers).**Notations: R101∗/R170∗ is ResNet-101/170 with the last residual stage removed [14, 46, 35], and R170 is made wider [35]; Rv50 is a reversible net [23], RX50 is ResNeXt-50-32${\times}$8d [62]. †: Pre-training uses FastAutoAugment [40] that is supervised by ImageNet labels.

method	architecture	#params (M)	accuracy (%)
Exemplar [17]	R50w${3}{\times}$	211	46.0 [38]
RelativePosition [13]	R50w${2}{\times}$	94	51.4 [38]
Jigsaw [45]	R50w${2}{\times}$	94	44.6 [38]
Rotation [19]	Rv50w${4}{\times}$	86	55.4 [38]
Colorization [64]	R101∗	28	39.6 [14]
DeepCluster [3]	VGG [53]	15	48.4 [4]
BigBiGAN [16]	R50	24	56.6
	Rv50w${4}{\times}$	86	61.3
methods based on contrastive learning follow:
InstDisc [61]	R50	24	54.0
LocalAgg [66]	R50	24	58.8
CPC v1 [46]	R101∗	28	48.7
CPC v2 [35]	R170${}^{*}_{\text{wider}}$	303	65.9
CMC [56]	R50${}_{\text{L+ab}}$	47	64.1†
	R50w${2}{\times}$${}_{\text{L+ab}}$	188	68.4†
AMDIM [2]	AMDIM${}_{\text{small}}$	194	63.5†
	AMDIM${}_{\text{large}}$	626	68.1†
MoCo	R50	24	60.6
	RX50	46	63.9
	R50w${2}{\times}$	94	65.4
	R50w${4}{\times}$	375	68.6

表 1：ImageNet 线性分类协议下的比较。图可视化该表。均为在 ImageNet-1M 训练集上无监督预训练，再在冻结特征上训有监督线性分类，于验证集评估。参数量为特征提取器。若有改进复现则报告之（数字后附引用）。记号：R101∗/R170∗ 为去掉最后残差阶段的 ResNet-101/170 [14, 46, 35]，R170 加宽 [35]；Rv50 为可逆网络 [23]，RX50 为 ResNeXt-50-32×8d [62]。†：预训练使用由 ImageNet 标签监督的 FastAutoAugment [40]。

Comparison with previous results.

Previous unsupervised learning methods can differ substantially in model sizes. For a fair and comprehensive comparison, we report accuracyvs. #parameters[^3] trade-offs. Besides ResNet-50 (R50) [33], we also report its variants that are ${2}{\times}$ and ${4}{\times}$ wider (more channels), following [38].[^4] We set $K$ $=$ 65536 and $m$ $=$ 0.999. Table 1 is the comparison.

与先前结果比较

先前无监督方法在模型规模上差异很大。为公平全面比较，我们报告准确率与参数量[^3]的权衡。除 ResNet-50（R50）[33] 外，还报告按 [38] 加宽 ${2}{\times}$、${4}{\times}$（更多通道）的变体。[^4]设定 $K=65536$、$m=0.999$。表 1 为比较。

MoCo with R50 performs competitively and achieves 60.6% accuracy, better than all competitors of similar model sizes ($\scriptstyle\sim$24M). MoCo benefits from larger models and achieves 68.6% accuracy with R50w${4}{\times}$.

R50 上的 MoCo 有竞争力，达 60.6% 准确率，优于相近规模（约 24M）的所有竞争者。MoCo 受益于更大模型，R50w${4}{\times}$ 达 68.6%。

Notably, we achieve competitive results using a standard ResNet-50 and require no specific architecture designs, e.g., patchified inputs [46, 35], carefully tailored receptive fields [2], or combining two networks [56]. By using an architecture that is not customized for the pretext task, it is easier to transfer features to a variety of visual tasks and make comparisons, studied in the next subsection.

值得一提的是，我们用标准 ResNet-50 即取得有竞争力的结果，无需为 pretext 定制结构，例如切块输入 [46, 35]、精心裁剪感受野 [2]，或组合两个网络 [56]。使用不为 pretext 定制的结构，更易于把特征迁移到多种视觉任务并做比较，见下一小节。

This paper’s focus is on a mechanism for general contrastive learning; we do not explore orthogonal factors (such as specific pretext tasks) that may further improve accuracy. As an example, “MoCo v2” [8], an extension of a preliminary version of this manuscript, achieves 71.1% accuracy with R50 (up from 60.6%), given small changes on the data augmentation and output projection head [7]. We believe that this additional result shows the generality and robustness of the MoCo framework.

本文焦点是通用对比学习机制；我们不探索可能进一步抬高准确率的正交因素（如特定 pretext）。例如，「MoCo v2」[8] 作为本稿初稿的扩展，在数据增强与输出投影头上做小改动 [7] 后，R50 准确率从 60.6% 升至 71.1%。我们认为这一额外结果说明了 MoCo 框架的通用性与稳健性。

pre-train	AP${}_{\text{50}}$	AP	AP${}_{\text{75}}$
random init.	random init.	64.4	64.4		37.9	37.9		38.6	38.6
random init.
64.4
37.9
38.6
super. IN-1M	super.	IN-1M	81.4	81.4		54.0	54.0		59.1	59.1
super.	IN-1M
81.4
54.0
59.1
MoCo IN-1M	MoCo	IN-1M	81.1 (${-}$0.3)	81.1	(${-}$0.3)	54.6 (${+}$0.6)	54.6	(${+}$0.6)	59.9 (${+}$0.8)	59.9	(${+}$0.8)
MoCo	IN-1M
81.1	(${-}$0.3)
54.6	(${+}$0.6)
59.9	(${+}$0.8)
MoCo IG-1B	MoCo	IG-1B	81.6 (${+}$0.2)	81.6	(${+}$0.2)	55.5 (${+}$1.5)	55.5	(${+}$1.5)	61.2 (${+}$2.1)	61.2	(${+}$2.1)
MoCo	IG-1B
81.6	(${+}$0.2)
55.5	(${+}$1.5)
61.2	(${+}$2.1)

random init.

64.4

37.9

38.6

super.	IN-1M

81.4

54.0

59.1

MoCo	IN-1M

81.1	(${-}$0.3)

54.6	(${+}$0.6)

59.9	(${+}$0.8)

MoCo	IG-1B

81.6	(${+}$0.2)

55.5	(${+}$1.5)

61.2	(${+}$2.1)

pre-train	AP${}_{\text{50}}$	AP	AP${}_{\text{75}}$
random init.	random init.	60.2	60.2		33.8	33.8		33.1	33.1
random init.
60.2
33.8
33.1
super. IN-1M	super.	IN-1M	81.3	81.3		53.5	53.5		58.8	58.8
super.	IN-1M
81.3
53.5
58.8
MoCo IN-1M	MoCo	IN-1M	81.5 (${+}$0.2)	81.5	(${+}$0.2)	55.9 (${+}$2.4)	55.9	(${+}$2.4)	62.6 (${+}$3.8)	62.6	(${+}$3.8)
MoCo	IN-1M
81.5	(${+}$0.2)
55.9	(${+}$2.4)
62.6	(${+}$3.8)
MoCo IG-1B	MoCo	IG-1B	82.2 (${+}$0.9)	82.2	(${+}$0.9)	57.2 (${+}$3.7)	57.2	(${+}$3.7)	63.7 (${+}$4.9)	63.7	(${+}$4.9)
MoCo	IG-1B
82.2	(${+}$0.9)
57.2	(${+}$3.7)
63.7	(${+}$4.9)

random init.

60.2

33.8

33.1

super.	IN-1M

81.3

53.5

58.8

MoCo	IN-1M

81.5	(${+}$0.2)

55.9	(${+}$2.4)

62.6	(${+}$3.8)

MoCo	IG-1B

82.2	(${+}$0.9)

57.2	(${+}$3.7)

63.7	(${+}$4.9)

(a) Faster R-CNN, R50-dilated-C5

表 2a：Faster R-CNN，R50-dilated-C5（及 C4 等）在 VOC 上的结果摘要；数字与括号内相对有监督 IN-1M 的差值保持原样。

pre-train	AP${}_{\text{50}}$	AP	AP${}_{\text{75}}$
random init.	random init.	64.4	64.4		37.9	37.9		38.6	38.6
random init.
64.4
37.9
38.6
super. IN-1M	super.	IN-1M	81.4	81.4		54.0	54.0		59.1	59.1
super.	IN-1M
81.4
54.0
59.1
MoCo IN-1M	MoCo	IN-1M	81.1 (${-}$0.3)	81.1	(${-}$0.3)	54.6 (${+}$0.6)	54.6	(${+}$0.6)	59.9 (${+}$0.8)	59.9	(${+}$0.8)
MoCo	IN-1M
81.1	(${-}$0.3)
54.6	(${+}$0.6)
59.9	(${+}$0.8)
MoCo IG-1B	MoCo	IG-1B	81.6 (${+}$0.2)	81.6	(${+}$0.2)	55.5 (${+}$1.5)	55.5	(${+}$1.5)	61.2 (${+}$2.1)	61.2	(${+}$2.1)
MoCo	IG-1B
81.6	(${+}$0.2)
55.5	(${+}$1.5)
61.2	(${+}$2.1)

random init.

64.4

37.9

38.6

super.	IN-1M

81.4

54.0

59.1

MoCo	IN-1M

81.1	(${-}$0.3)

54.6	(${+}$0.6)

59.9	(${+}$0.8)

MoCo	IG-1B

81.6	(${+}$0.2)

55.5	(${+}$1.5)

61.2	(${+}$2.1)

pre-train	AP${}_{\text{50}}$	AP	AP${}_{\text{75}}$
random init.	random init.	60.2	60.2		33.8	33.8		33.1	33.1
random init.
60.2
33.8
33.1
super. IN-1M	super.	IN-1M	81.3	81.3		53.5	53.5		58.8	58.8
super.	IN-1M
81.3
53.5
58.8
MoCo IN-1M	MoCo	IN-1M	81.5 (${+}$0.2)	81.5	(${+}$0.2)	55.9 (${+}$2.4)	55.9	(${+}$2.4)	62.6 (${+}$3.8)	62.6	(${+}$3.8)
MoCo	IN-1M
81.5	(${+}$0.2)
55.9	(${+}$2.4)
62.6	(${+}$3.8)
MoCo IG-1B	MoCo	IG-1B	82.2 (${+}$0.9)	82.2	(${+}$0.9)	57.2 (${+}$3.7)	57.2	(${+}$3.7)	63.7 (${+}$4.9)	63.7	(${+}$4.9)
MoCo	IG-1B
82.2	(${+}$0.9)
57.2	(${+}$3.7)
63.7	(${+}$4.9)

random init.

60.2

33.8

33.1

super.	IN-1M

81.3

53.5

58.8

MoCo	IN-1M

81.5	(${+}$0.2)

55.9	(${+}$2.4)

62.6	(${+}$3.8)

MoCo	IG-1B

82.2	(${+}$0.9)

57.2	(${+}$3.7)

63.7	(${+}$4.9)

(a) Faster R-CNN, R50-dilated-C5

	R50-dilated-C5	R50-C4
pre-train	AP${}_{\text{50}}$	AP	AP${}_{\text{75}}$	AP${}_{\text{50}}$	AP	AP${}_{\text{75}}$
end-to-end	79.2	52.0	56.6	80.4	54.6	60.3
memory bank	79.8	52.9	57.9	80.6	54.9	60.6
MoCo	81.1	54.6	59.9	81.5	55.9	62.6

Table 3: Comparison of three contrastive loss mechanisms on PASCAL VOC object detection, fine-tuned on trainval07+12 and evaluated on test2007 (averages over 5 trials). All models are implemented by us (Figure 3), pre-trained on IN-1M, and fine-tuned using the same settings as in Table 2.

	R50-dilated-C5	R50-C4
pre-train	AP${}_{\text{50}}$	AP	AP${}_{\text{75}}$	AP${}_{\text{50}}$	AP	AP${}_{\text{75}}$
end-to-end	79.2	52.0	56.6	80.4	54.6	60.3
memory bank	79.8	52.9	57.9	80.6	54.9	60.6
MoCo	81.1	54.6	59.9	81.5	55.9	62.6

表 3：三种对比损失机制在 PASCAL VOC 目标检测上的比较，于 trainval07+12 微调、test2007 评估（5 次试验平均）。模型均由我们实现（图 3），在 IN-1M 预训练，微调设定同表 2。

		AP${}_{\text{50}}$	AP	AP${}_{\text{75}}$
pre-train	RelPos, by [14]	Multi-task [14]	Jigsaw, by [26]	LocalAgg [66]	MoCo	MoCo	Multi-task [14]	MoCo
super.	IN-1M	74.2	74.2		74.2	74.2		70.5	70.5		74.6	74.6		74.4	74.4		42.4	42.4		44.3	44.3		42.7	42.7
74.2
74.2
70.5
74.6
74.4
42.4
44.3
42.7
unsup.	IN-1M	66.8 (${-}$7.4)	66.8	(${-}$7.4)	70.5 (${-}$3.7)	70.5	(${-}$3.7)	61.4 (${-}$9.1)	61.4	(${-}$9.1)	69.1 (${-}$5.5)	69.1	(${-}$5.5)	74.9 (${+}$0.5)	74.9	(${+}$0.5)	46.6 (${+}$4.2)	46.6	(${+}$4.2)	43.9 (${-}$0.4)	43.9	(${-}$0.4)	50.1 (${+}$7.4)	50.1	(${+}$7.4)
66.8	(${-}$7.4)
70.5	(${-}$3.7)
61.4	(${-}$9.1)
69.1	(${-}$5.5)
74.9	(${+}$0.5)
46.6	(${+}$4.2)
43.9	(${-}$0.4)
50.1	(${+}$7.4)
unsup.	IN-14M	-	-		-	-		69.2 (${-}$1.3)	69.2	(${-}$1.3)	-	-		75.2 (${+}$0.8)	75.2	(${+}$0.8)	46.9 (${+}$4.5)	46.9	(${+}$4.5)	-	-		50.2 (${+}$7.5)	50.2	(${+}$7.5)
-
-
69.2	(${-}$1.3)
-
75.2	(${+}$0.8)
46.9	(${+}$4.5)
-
50.2	(${+}$7.5)
unsup.	YFCC-100M	-	-		-	-		66.6 (${-}$3.9)	66.6	(${-}$3.9)	-	-		74.7 (${+}$0.3)	74.7	(${+}$0.3)	45.9 (${+}$3.5)	45.9	(${+}$3.5)	-	-		49.0 (${+}$6.3)	49.0	(${+}$6.3)
-
-
66.6	(${-}$3.9)
-
74.7	(${+}$0.3)
45.9	(${+}$3.5)
-
49.0	(${+}$6.3)
unsup.	IG-1B	-	-		-	-		-	-		-	-		75.6 (${+}$1.2)	75.6	(${+}$1.2)	47.6 (${+}$5.2)	47.6	(${+}$5.2)	-	-		51.7 (${+}$9.0)	51.7	(${+}$9.0)
-
-
-
-
75.6	(${+}$1.2)
47.6	(${+}$5.2)
-
51.7	(${+}$9.0)

74.2

74.2

70.5

74.6

74.4

42.4

44.3

42.7

66.8	(${-}$7.4)

70.5	(${-}$3.7)

61.4	(${-}$9.1)

69.1	(${-}$5.5)

74.9	(${+}$0.5)

46.6	(${+}$4.2)

43.9	(${-}$0.4)

50.1	(${+}$7.4)

-

-

69.2	(${-}$1.3)

-

75.2	(${+}$0.8)

46.9	(${+}$4.5)

-

50.2	(${+}$7.5)

-

-

66.6	(${-}$3.9)

-

74.7	(${+}$0.3)

45.9	(${+}$3.5)

-

49.0	(${+}$6.3)

-

-

-

-

75.6	(${+}$1.2)

47.6	(${+}$5.2)

-

51.7	(${+}$9.0)

Table 4: Comparison with previous methods on object detection fine-tuned on PASCAL VOC trainval2007. Evaluation is on test2007. The ImageNet supervised counterparts are from the respective papers, and are reported as having the same structure as the respective unsupervised pre-training counterparts. All entries are based on the C4 backbone. The models in [14] are R101 v2 [34], and others are R50. The RelPos (relative position) [13] result is the best single-task case in the Multi-task paper [14]. The Jigsaw [45] result is from the ResNet-based implementation in [26]. Our results are with 9k-iteration fine-tuning, averaged over 5 trials. In the brackets are the gaps to the ImageNet supervised pre-training counterpart. In green are the gaps of at least ${+}$0.5 point.

		AP${}_{\text{50}}$	AP	AP${}_{\text{75}}$
pre-train	RelPos, by [14]	Multi-task [14]	Jigsaw, by [26]	LocalAgg [66]	MoCo	MoCo	Multi-task [14]	MoCo
super.	IN-1M	74.2	74.2		74.2	74.2		70.5	70.5		74.6	74.6		74.4	74.4		42.4	42.4		44.3	44.3		42.7	42.7
74.2
74.2
70.5
74.6
74.4
42.4
44.3
42.7
unsup.	IN-1M	66.8 (${-}$7.4)	66.8	(${-}$7.4)	70.5 (${-}$3.7)	70.5	(${-}$3.7)	61.4 (${-}$9.1)	61.4	(${-}$9.1)	69.1 (${-}$5.5)	69.1	(${-}$5.5)	74.9 (${+}$0.5)	74.9	(${+}$0.5)	46.6 (${+}$4.2)	46.6	(${+}$4.2)	43.9 (${-}$0.4)	43.9	(${-}$0.4)	50.1 (${+}$7.4)	50.1	(${+}$7.4)
66.8	(${-}$7.4)
70.5	(${-}$3.7)
61.4	(${-}$9.1)
69.1	(${-}$5.5)
74.9	(${+}$0.5)
46.6	(${+}$4.2)
43.9	(${-}$0.4)
50.1	(${+}$7.4)
unsup.	IN-14M	-	-		-	-		69.2 (${-}$1.3)	69.2	(${-}$1.3)	-	-		75.2 (${+}$0.8)	75.2	(${+}$0.8)	46.9 (${+}$4.5)	46.9	(${+}$4.5)	-	-		50.2 (${+}$7.5)	50.2	(${+}$7.5)
-
-
69.2	(${-}$1.3)
-
75.2	(${+}$0.8)
46.9	(${+}$4.5)
-
50.2	(${+}$7.5)
unsup.	YFCC-100M	-	-		-	-		66.6 (${-}$3.9)	66.6	(${-}$3.9)	-	-		74.7 (${+}$0.3)	74.7	(${+}$0.3)	45.9 (${+}$3.5)	45.9	(${+}$3.5)	-	-		49.0 (${+}$6.3)	49.0	(${+}$6.3)
-
-
66.6	(${-}$3.9)
-
74.7	(${+}$0.3)
45.9	(${+}$3.5)
-
49.0	(${+}$6.3)
unsup.	IG-1B	-	-		-	-		-	-		-	-		75.6 (${+}$1.2)	75.6	(${+}$1.2)	47.6 (${+}$5.2)	47.6	(${+}$5.2)	-	-		51.7 (${+}$9.0)	51.7	(${+}$9.0)
-
-
-
-
75.6	(${+}$1.2)
47.6	(${+}$5.2)
-
51.7	(${+}$9.0)

74.2

74.2

70.5

74.6

74.4

42.4

44.3

42.7

66.8	(${-}$7.4)

70.5	(${-}$3.7)

61.4	(${-}$9.1)

69.1	(${-}$5.5)

74.9	(${+}$0.5)

46.6	(${+}$4.2)

43.9	(${-}$0.4)

50.1	(${+}$7.4)

-

-

69.2	(${-}$1.3)

-

75.2	(${+}$0.8)

46.9	(${+}$4.5)

-

50.2	(${+}$7.5)

-

-

66.6	(${-}$3.9)

-

74.7	(${+}$0.3)

45.9	(${+}$3.5)

-

49.0	(${+}$6.3)

-

-

-

-

75.6	(${+}$1.2)

47.6	(${+}$5.2)

-

51.7	(${+}$9.0)

表 4：与先前方法在 PASCAL VOC 目标检测上的比较，于 trainval2007 微调、test2007 评估。有监督对照取自各文，并报告为与相应无监督预训练同结构。条目均基于 C4 骨干。[14] 中模型为 R101 v2 [34]，其余为 R50。RelPos（相对位置）[13] 结果为多任务文 [14] 中最佳单任务情形。Jigsaw [45] 结果取自 [26] 的 ResNet 实现。我们的结果为 9k 次迭代微调、5 次平均。括号内为相对 ImageNet 有监督预训练对照的差距。绿色为至少 +0.5 点的差距。