Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Bootstrap Your Own Latent A New Approach to Self-Supervised Learning

Bootstrap Your Own Latent：一种新的自监督学习方法

Jean-Bastien Grill Florian Strub 1 1 footnotemark: 1 Florent Altché 1 1 footnotemark: 1 Corentin Tallec 1 1 footnotemark: 1 Pierre H. Richemond 1 1 footnotemark: 1 † † thanks: Equal contribution; the order of first authors was randomly selected. Affiliation: DeepMind Affiliation: Imperial College Elena Buchatskaya Carl Doersch Bernardo Avila Pires Zhaohan Daniel Guo Affiliation: DeepMind Mohammad Gheshlaghi Azar Bilal Piot Koray Kavukcuoglu Rémi Munos Michal Valko Email: [jbgrill,fstrub,altche,corentint,richemond]@google.com Affiliation: DeepMind

Abstract

摘要

74.3\%

74.3\%

1 Introduction

1 引言

[1, 2, 3, 4, 5, 6, 7]

2\times

2\times

Learning good image representations is a key challenge in computer vision [1, 2, 3] as it allows for efficient training on downstream tasks [4, 5, 6, 7]. Many different training approaches have been proposed to learn such representations, usually relying on visual pretext tasks. Among them, state-of-the-art contrastive methods [8, 9, 10, 11, 12] are trained by reducing the distance between representations of different augmented views of the same image (‘positive pairs’), and increasing the distance between representations of augmented views from different images (‘negative pairs’). These methods need careful treatment of negative pairs [13] by either relying on large batch sizes [8, 12], memory banks [9] or customized mining strategies [14, 15] to retrieve the negative pairs. In addition, their performance critically depends on the choice of image augmentations [8, 12].

学习良好的图像表征是计算机视觉的关键挑战 [1, 2, 3]，因为它能让下游任务更高效地训练 [4, 5, 6, 7]。人们提出了许多训练此类表征的方法，通常依赖视觉代理任务。其中，当前最优的对比方法 [8, 9, 10, 11, 12] 通过缩小同一图像不同增强视图表征之间的距离（「正样本对」），并增大不同图像增强视图表征之间的距离（「负样本对」）来训练。这些方法需要谨慎处理负样本对 [13]，要么依赖大批次 [8, 12]、记忆库 [9]，要么定制挖掘策略 [14, 15] 来获取负样本。此外，其表现严重依赖图像增强的选择 [8, 12]。

In this paper, we introduce B ootstrap Y our O wn L atent (BYOL), a new algorithm for self-supervised learning of image representations. BYOL achieves higher performance than state-of-the-art contrastive methods without using negative pairs. It iteratively bootstraps[^4] the outputs of a network to serve as targets for an enhanced representation. Moreover, BYOL is more robust to the choice of image augmentations than contrastive methods; we suspect that not relying on negative pairs is one of the leading reasons for its improved robustness. While previous methods based on bootstrapping have used pseudo-labels [16], cluster indices [17] or a handful of labels [18, 19, 20], we propose to directly bootstrap the representations . In particular, BYOL uses two neural networks, referred to as online and target networks, that interact and learn from each other. Starting from an augmented view of an image, BYOL trains its online network to predict the target network’s representation of another augmented view of the same image. While this objective admits collapsed solutions, e.g., outputting the same vector for all images, we empirically show that BYOL does not converge to such solutions. We hypothesize (see Section 3.2) that the combination of (i) the addition of a predictor to the online network and (ii) the use of a slow-moving average of the online parameters as the target network encourages encoding more and more information within the online projection and avoids collapsed solutions.

本文提出 B ootstrap Y our O wn L atent（BYOL），一种新的图像表征自监督学习算法。BYOL 在不使用负样本对的情况下，取得高于当前最优对比方法的表现。它迭代地自举网络输出，作为增强表征的目标。此外，BYOL 对图像增强选择比对比方法更稳健；我们怀疑不依赖负样本对是稳健性提升的主因之一。先前基于自举的方法使用伪标签 [16]、聚类索引 [17] 或少量标签 [18, 19, 20]，我们则提出 直接自举表征 。具体地，BYOL 使用在线与目标两个神经网络彼此交互学习。从一幅图像的增强视图出发，BYOL 训练在线网络去预测目标网络对同一图像另一增强视图的表征。该目标虽允许崩塌解（例如对所有图像输出同一向量），但我们经验上表明 BYOL 不会收敛到这类解。我们猜想（见第 3.2 节）：(i) 在在线网络上增加预测器，以及 (ii) 用在线参数的慢速滑动平均作为目标网络，二者结合会鼓励在线投影编码越来越多的信息，并避免崩塌解。

74.3\%

74.3\%

2 Related work

2 相关工作

Most unsupervised methods for representation learning can be categorized as either generative or discriminative [23, 8]. Generative approaches to representation learning build a distribution over data and latent embedding and use the learned embeddings as image representations. Many of these approaches rely either on auto-encoding of images [24, 25, 26] or on adversarial learning [27], jointly modelling data and representation [28, 29, 30, 31]. Generative methods typically operate directly in pixel space. This however is computationally expensive, and the high level of detail required for image generation may not be necessary for representation learning.

多数无监督表征学习方法可归为生成式或判别式 [23, 8]。生成式方法在数据与隐嵌入上建立分布，并把学到的嵌入当作图像表征。其中许多依赖图像自编码 [24, 25, 26] 或对抗学习 [27]，联合建模数据与表征 [28, 29, 30, 31]。生成式方法通常直接在像素空间操作，计算昂贵，且图像生成所需的高细节对表征学习未必必要。

Among discriminative methods, contrastive methods [9, 10, 32, 33, 34, 11, 35, 36] currently achieve state-of-the-art performance in self-supervised learning [37, 8, 38, 12]. Contrastive approaches avoid a costly generation step in pixel space by bringing representation of different views of the same image closer (‘positive pairs’), and spreading representations of views from different images (‘negative pairs’) apart [39, 40]. Contrastive methods often require comparing each example with many other examples to work well [9, 8] prompting the question of whether using negative pairs is necessary.

在判别式方法中，对比方法 [9, 10, 32, 33, 34, 11, 35, 36] 目前在自监督学习中表现最优 [37, 8, 38, 12]。对比方法通过拉近同一图像不同视图的表征（「正样本对」）、推远不同图像视图的表征（「负样本对」）[39, 40]，避免在像素空间做昂贵生成。对比方法往往需要把每个样本与许多其他样本比较才能工作得好 [9, 8]，于是引出负样本对是否必要的问题。

DeepCluster [17] partially answers this question. It uses bootstrapping on previous versions of its representation to produce targets for the next representation; it clusters data points using the prior representation, and uses the cluster index of each sample as a classification target for the new representation. While avoiding the use of negative pairs, this requires a costly clustering phase and specific precautions to avoid collapsing to trivial solutions.

DeepCluster [17] 部分回答了该问题。它对自己先前版本的表征做自举，为下一表征产生目标：用先前表征对数据点聚类，并把每个样本的聚类索引当作新表征的分类目标。虽然避免了负样本对，但这需要昂贵的聚类阶段，并需采取特定防范以避免崩塌到平凡解。

Some self-supervised methods are not contrastive but rely on using auxiliary handcrafted prediction tasks to learn their representation. In particular, relative patch prediction [23, 40], colorizing gray-scale images [41, 42], image inpainting [43], image jigsaw puzzle [44], image super-resolution [45], and geometric transformations [46, 47] have been shown to be useful. Yet, even with suitable architectures [48], these methods are being outperformed by contrastive methods [37, 8, 12].

一些自监督方法不是对比式的，而是依赖手工设计的辅助预测任务来学习表征。相对图块预测 [23, 40]、灰度图上色 [41, 42]、图像修复 [43]、拼图 [44]、超分辨率 [45] 以及几何变换 [46, 47] 都被证明有用。然而即便有合适架构 [48]，这些方法也正被对比方法超越 [37, 8, 12]。

Our approach has some similarities with Predictions of Bootstrapped Latents (PBL, [49]), a self-supervised representation learning technique for reinforcement learning (RL). PBL jointly trains the agent’s history representation and an encoding of future observations. The observation encoding is used as a target to train the agent’s representation, and the agent’s representation as a target to train the observation encoding. Unlike PBL, BYOL uses a slow-moving average of its representation to provide its targets, and does not require a second network.

我们的方法与强化学习中的自监督表征学习技术 Predictions of Bootstrapped Latents （PBL，[49]）有一些相似。PBL 联合训练智能体历史表征与未来观测编码。观测编码用作训练智能体表征的目标，智能体表征又用作训练观测编码的目标。与 PBL 不同，BYOL 用自身表征的慢速滑动平均提供目标，且不需要第二个网络。

The idea of using a slow-moving average target network to produce stable targets for the online network was inspired by deep RL [50, 51, 52, 53]. Target networks stabilize the bootstrapping updates provided by the Bellman equation, making them appealing to stabilize the bootstrap mechanism in BYOL. While most RL methods use fixed target networks, BYOL uses a weighted moving average of previous networks (as in [54]) in order to provide smoother changes in the target representation.

用慢速滑动平均目标网络为在线网络产生稳定目标的想法，受深度强化学习启发 [50, 51, 52, 53]。目标网络稳定贝尔曼方程提供的自举更新，因而也适合稳定 BYOL 中的自举机制。多数 RL 方法使用固定目标网络，而 BYOL 使用先前网络的加权滑动平均（如 [54]），以使目标表征变化更平滑。

\ell_{2}

\ell_{2}

Finally, in self-supervised learning, MoCo [9] uses a slow-moving average network (momentum encoder) to maintain consistent representations of negative pairs drawn from a memory bank. Instead, BYOL uses a moving average network to produce prediction targets as a means of stabilizing the bootstrap step. We show in Section 5 that this mere stabilizing effect can also improve existing contrastive methods.

最后，在自监督学习中，MoCo [9] 使用慢速滑动平均网络（ 动量编码器 ）来维护从记忆库抽取的负样本对的一致表征。BYOL 则用滑动平均网络产生预测目标，作为稳定自举步骤的手段。我们在第 5 节表明，这一单纯的稳定效果也能改进现有对比方法。

3 Method

3 方法

We start by motivating our method before explaining its details in Section 3.1. Many successful self-supervised learning approaches build upon the cross-view prediction framework introduced in [63]. Typically, these approaches learn representations by predicting different views (e.g., different random crops) of the same image from one another. Many such approaches cast the prediction problem directly in representation space: the representation of an augmented view of an image should be predictive of the representation of another augmented view of the same image. However, predicting directly in representation space can lead to collapsed representations: for instance, a representation that is constant across views is always fully predictive of itself. Contrastive methods circumvent this problem by reformulating the prediction problem into one of discrimination: from the representation of an augmented view, they learn to discriminate between the representation of another augmented view of the same image, and the representations of augmented views of different images. In the vast majority of cases, this prevents the training from finding collapsed representations. Yet, this discriminative approach typically requires comparing each representation of an augmented view with many negative examples, to find ones sufficiently close to make the discrimination task challenging. In this work, we thus tasked ourselves to find out whether these negative examples are indispensable to prevent collapsing while preserving high performance.

我们先说明动机，再在第 3.1 节给出细节。许多成功的自监督学习方法建立在 [63] 提出的跨视图预测框架之上。这些方法通常通过从一个视图预测同一图像的另一视图（例如不同随机裁剪）来学习表征。许多方法把预测问题直接放在表征空间：一幅图像增强视图的表征应能预测同一图像另一增强视图的表征。然而直接在表征空间预测可能导致崩塌表征：例如跨视图恒定的表征总是能完全预测自身。对比方法把预测改写成判别：从一幅增强视图的表征出发，学会把同一图像另一增强视图的表征，与不同图像增强视图的表征区分开。绝大多数情况下，这能防止训练找到崩塌表征。但这种判别做法通常需要把每个增强视图的表征与大量负例比较，才能找到足够接近、使判别任务具有挑战性的样本。因此，本工作的任务是弄清：负例对于防止崩塌同时保持高性能是否必不可少。

18.8\%

18.8\%

3.1 Description of BYOL

3.1 BYOL 的描述

q_{\theta}(z_{\theta})

q_{\theta}(z_{\theta})

y_{\theta}

y_{\theta}

\xi\leftarrow\tau\xi+(1-\tau){\theta}. \tag{1}

\mathcal{D}

\mathcal{D}

\mathcal{L}_{{\theta},\xi}\mathrel{\hbox{\hskip 3.8889pt\hskip-3.8889pt\hbox{$=$}\hskip-3.8889pt\hskip-2.7431pt\raisebox{4.66875pt}{\hbox{$\scriptscriptstyle\Delta$}}\hskip-2.7431pt\hskip 3.8889pt}}\mathopen{}\mathclose{{\left\|\overline{q_{\theta}}(z_{\theta})-\overline{z}^{\prime}_{\xi}}}\right\|_{2}^{2}=2-2\cdot\frac{\langle q_{\theta}(z_{\theta}),z^{\prime}_{\xi}\rangle}{\big\|q_{\theta}(z_{\theta})\big\|_{2}\cdot\big\|z^{\prime}_{\xi}\big\|_{2}}\cdot \tag{2}

\mathcal{L}_{{\theta},\xi}

v^{\prime}

\displaystyle{\theta}\leftarrow\mathrm{optimizer}\mathopen{}\mathclose{{\left({\theta},\nabla_{\theta}\mathcal{L}^{\texttt{BYOL}}_{{\theta},\xi},\eta}}\right), \\ \displaystyle\xi\leftarrow\tau\xi+(1-\tau){\theta}, \tag{3}

\mathrm{optimizer}

\mathrm{optimizer}

f_{\theta}

f_{\theta}

3.2 Intuitions on BYOL’s behavior

3.2 关于 BYOL 行为的直觉

\mathcal{L}_{{\theta},\xi}^{\texttt{BYOL}}

{\theta}

q_{\theta}=q^{\star}

q_{\theta}=q^{\star}

q^{\star}\mathrel{\hbox{\hskip 3.8889pt\hskip-3.8889pt\hbox{$=$}\hskip-3.8889pt\hskip-2.7431pt\raisebox{4.66875pt}{\hbox{$\scriptscriptstyle\Delta$}}\hskip-2.7431pt\hskip 3.8889pt}}\argmin_{q}\mathbb{E}\mathopen{}\mathclose{{\left[\mathopen{}\mathclose{{\left\|q(z_{\theta})-z^{\prime}_{\xi}}}\right\|_{2}^{2}}}\right],\quad\text{where}\quad q^{\star}(z_{\theta})=\mathbb{E}\mathopen{}\mathclose{{\left[z^{\prime}_{\xi}|z_{\theta}}}\right], \tag{4}

{\theta}

{\theta}

\nabla_{\theta}\mathbb{E}\mathopen{}\mathclose{{\left[\mathopen{}\mathclose{{\left\|q^{\star}(z_{\theta})-z^{\prime}_{\xi}}}\right\|_{2}^{2}}}\right]=\nabla_{\theta}\mathbb{E}\mathopen{}\mathclose{{\left[\mathopen{}\mathclose{{\left\|\mathbb{E}\mathopen{}\mathclose{{\left[z^{\prime}_{\xi}|z_{\theta}}}\right]-z^{\prime}_{\xi}}}\right\|_{2}^{2}}}\right]=\nabla_{\theta}\mathbb{E}\mathopen{}\mathclose{{\left[\sum_{i}\var(z^{\prime}_{\xi,i}|z_{\theta})}}\right], \tag{5}

z^{\prime}_{\xi,i}

z^{\prime}_{\xi,i}

X,

X

z_{\theta}

z_{\theta}

{\theta}

{\theta}

3.3 Implementation details

3.3 实现细节

224\times 224

224\times 224

50(1\times)

50(1\times)

1000

1000

4 Experimental evaluation

4 实验评估

We assess the performance of BYOL’s representation after self-supervised pretraining on the training set of the ImageNet ILSVRC-2012 dataset [21]. We first evaluate it on ImageNet (IN) in both linear evaluation and semi-supervised setups. We then measure its transfer capabilities on other datasets and tasks, including classification, segmentation, object detection and depth estimation. For comparison, we also report scores for a representation trained using labels from the train ImageNet subset, referred to as Supervised-IN. In Appendix E, we assess the generality of BYOL by pretraining a representation on the Places365-Standard dataset [73] before reproducing this evaluation protocol.

我们在 ImageNet ILSVRC-2012 数据集 [21] 的训练集上做自监督预训练后，评估 BYOL 表征的表现。先在 ImageNet（IN）上分别做线性评估与半监督设置；再度量其在其他数据集与任务上的迁移能力，包括分类、分割、目标检测与深度估计。为对照，我们也报告用 ImageNet 训练子集标签训练的表征分数，记为 Supervised-IN。附录 E 中，我们在 Places365-Standard 数据集 [73] 上预训练表征，并复现该评估协议，以检验 BYOL 的通用性。

1

1

1

Method	Top- $1$	Top- $5$
Local Agg.	$60.2$	-
PIRL [35]	$63.6$	-
CPC v2 [32]	$63.8$	$85.3$
CMC [11]	$66.2$	$87.0$
SimCLR [8]	$69.3$	$89.0$
MoCo v2 [37]	$71.1$	-
InfoMin Aug. [12]	$73.0$	$91.1$
BYOL (ours)	$\pagecolor{pearDark!20}\bf{74.3}$	$\pagecolor{pearDark!20}\bf{91.6}$

Method	Architecture	Param.	Top- $1$	Top- $5$
SimCLR [8]	ResNet- $50$ ( $2\times$ )	$94$ M	$74.2$	$92.0$
CMC [11]	ResNet- $50$ ( $2\times$ )	$94$ M	$70.6$	$89.7$
BYOL (ours)	ResNet- $50$ ( $2\times$ )	$94$ M	$77.4$	$93.6$
CPC v2 [32]	ResNet- $161$	$305$ M	$71.5$	$90.1$
MoCo [9]	ResNet- $50$ ( $4\times$ )	$375$ M	$68.6$	-
SimCLR [8]	ResNet- $50$ ( $4\times$ )	$375$ M	$76.5$	$93.2$
BYOL (ours)	ResNet- $50$ ( $4\times$ )	$375$ M	$\pagecolor{pearDark!20}78.6$	$94.2$
BYOL (ours)	ResNet- $200$ ( $2\times$ )	$250$ M	$\pagecolor{pearDark!20}\bf{79.6}$	$\bf{94.8}$

1

方法	Top- $1$	Top- $5$
Local Agg.	$60.2$	-
PIRL [35]	$63.6$	-
CPC v2 [32]	$63.8$	$85.3$
CMC [11]	$66.2$	$87.0$
SimCLR [8]	$69.3$	$89.0$
MoCo v2 [37]	$71.1$	-
InfoMin Aug. [12]	$73.0$	$91.1$
BYOL（本文）	$\mathbf{74.3}$	$\mathbf{91.6}$

方法	架构	参数量	Top- $1$	Top- $5$
SimCLR [8]	ResNet- $50$ （ $2\times$ ）	$94$ M	$74.2$	$92.0$
CMC [11]	ResNet- $50$ （ $2\times$ ）	$94$ M	$70.6$	$89.7$
BYOL（本文）	ResNet- $50$ （ $2\times$ ）	$94$ M	$77.4$	$93.6$
CPC v2 [32]	ResNet- $161$	$305$ M	$71.5$	$90.1$
MoCo [9]	ResNet- $50$ （ $4\times$ ）	$375$ M	$68.6$	-
SimCLR [8]	ResNet- $50$ （ $4\times$ ）	$375$ M	$76.5$	$93.2$
BYOL（本文）	ResNet- $50$ （ $4\times$ ）	$375$ M	$78.6$	$94.2$
BYOL（本文）	ResNet- $200$ （ $2\times$ ）	$250$ M	$\mathbf{79.6}$	$\mathbf{94.8}$

1\%

1\%

1

Method	Top- $1$	Top- $5$
	$1\%$	$10\%$	$1\%$	$10\%$
Supervised [77]	$25.4$	$56.4$	$48.4$	$80.4$
InstDisc	-	-	$39.2$	$77.4$
PIRL [35]	-	-	$57.2$	$83.8$
SimCLR [8]	$48.3$	$65.6$	$75.5$	$87.8$
BYOL (ours)	$\bf{53.2}$	$\bf{68.8}$	$\bf{78.4}$	$\bf{89.0}$

Method	Architecture	Param.	Top- $1$	Top- $5$
			$1\%$	$10\%$	$1\%$	$10\%$
CPC v2 [32]	ResNet- $161$	$305$ M	-	-	$77.9$	$91.2$
SimCLR [8]	ResNet- $50$ ( $2\times$ )	$94$ M	$58.5$	$71.7$	${83.0}$	$91.2$
BYOL (ours)	ResNet- $50$ ( $2\times$ )	$94$ M	${62.2}$	${73.5}$	${84.1}$	${91.7}$
SimCLR [8]	ResNet- $50$ ( $4\times$ )	$375$ M	$63.0$	$74.4$	$85.8$	${92.6}$
BYOL (ours)	ResNet- $50$ ( $4\times$ )	$375$ M	${69.1}$	${75.7}$	${87.9}$	$92.5$
BYOL (ours)	ResNet- $200$ ( $2\times$ )	$250$ M	$\bf{71.2}$	$\pagecolor{pearDark!20}\bf{77.7}$	$\bf{89.5}$	$\bf{93.7}$

1

方法	Top- $1$	Top- $5$
	$1\%$	$10\%$	$1\%$	$10\%$
Supervised [77]	$25.4$	$56.4$	$48.4$	$80.4$
InstDisc	-	-	$39.2$	$77.4$
PIRL [35]	-	-	$57.2$	$83.8$
SimCLR [8]	$48.3$	$65.6$	$75.5$	$87.8$
BYOL（本文）	$\mathbf{53.2}$	$\mathbf{68.8}$	$\mathbf{78.4}$	$\mathbf{89.0}$

方法	架构	参数量	Top- $1$	Top- $5$
			$1\%$	$10\%$	$1\%$	$10\%$
CPC v2 [32]	ResNet- $161$	$305$ M	-	-	$77.9$	$91.2$
SimCLR [8]	ResNet- $50$ （ $2\times$ ）	$94$ M	$58.5$	$71.7$	$83.0$	$91.2$
BYOL（本文）	ResNet- $50$ （ $2\times$ ）	$94$ M	$62.2$	$73.5$	$84.1$	$91.7$
SimCLR [8]	ResNet- $50$ （ $4\times$ ）	$375$ M	$63.0$	$74.4$	$85.8$	$92.6$
BYOL（本文）	ResNet- $50$ （ $4\times$ ）	$375$ M	$69.1$	$75.7$	$87.9$	$92.5$
BYOL（本文）	ResNet- $200$ （ $2\times$ ）	$250$ M	$\mathbf{71.2}$	$\mathbf{77.7}$	$\mathbf{89.5}$	$\mathbf{93.7}$

7

12

\bf{75.3}

Method	Food101	CIFAR10	CIFAR100	Birdsnap	SUN397	Cars	Aircraft	VOC2007	DTD	Pets	Caltech-101	Flowers
Linear evaluation:
BYOL (ours)	$\bf{75.3}$	$91.3$	$\bf{78.4}$	$\bf{57.2}$	$\bf{62.2}$	$\bf{67.8}$	$60.6$	$82.5$	$75.5$	$90.4$	$94.2$	$\bf{96.1}$
SimCLR (repro)	$72.8$	$90.5$	$74.4$	$42.4$	$60.6$	$49.3$	$49.8$	$81.4$	$\bf{75.7}$	$84.6$	$89.3$	$92.6$
SimCLR [8]	$68.4$	$90.6$	$71.6$	$37.4$	$58.8$	$50.3$	$50.3$	$80.5$	$74.5$	$83.6$	$90.3$	$91.2$
Supervised-IN [8]	$72.3$	$\bf{93.6}$	$78.3$	$53.7$	$61.9$	$66.7$	$\bf{61.0}$	$\bf{82.8}$	$74.9$	$\bf{91.5}$	$\bf{94.5}$	$94.7$
Fine-tuned:
BYOL (ours)	$\bf{88.5}$	$\bf{97.8}$	$86.1$	$\bf{76.3}$	$63.7$	$91.6$	$\bf{88.1}$	$\bf{85.4}$	$\bf{76.2}$	$91.7$	$\bf{93.8}$	$97.0$
SimCLR (repro)	$87.5$	$97.4$	$85.3$	$75.0$	$63.9$	$91.4$	$87.6$	$84.5$	$75.4$	$89.4$	$91.7$	$96.6$
SimCLR [8]	$88.2$	$97.7$	$85.9$	$75.9$	$63.5$	$91.3$	$88.1$	$84.1$	$73.2$	$89.2$	$92.1$	$97.0$
Supervised-IN [8]	$88.3$	$97.5$	$\bf{86.4}$	$75.8$	$\bf{64.3}$	$\bf{92.1}$	$86.0$	$85.0$	$74.6$	$\bf{92.1}$	$93.3$	$\bf{97.6}$
Random init [8]	$86.9$	$95.9$	$80.2$	$76.1$	$53.6$	$91.4$	$85.9$	$67.3$	$64.8$	$81.5$	$72.6$	$92.0$

\mathbf{75.3}

方法	Food101	CIFAR10	CIFAR100	Birdsnap	SUN397	Cars	Aircraft	VOC2007	DTD	Pets	Caltech-101	Flowers
线性评估：
BYOL（本文）	$\mathbf{75.3}$	$91.3$	$\mathbf{78.4}$	$\mathbf{57.2}$	$\mathbf{62.2}$	$\mathbf{67.8}$	$60.6$	$82.5$	$75.5$	$90.4$	$94.2$	$\mathbf{96.1}$
SimCLR（复现）	$72.8$	$90.5$	$74.4$	$42.4$	$60.6$	$49.3$	$49.8$	$81.4$	$\mathbf{75.7}$	$84.6$	$89.3$	$92.6$
SimCLR [8]	$68.4$	$90.6$	$71.6$	$37.4$	$58.8$	$50.3$	$50.3$	$80.5$	$74.5$	$83.6$	$90.3$	$91.2$
Supervised-IN [8]	$72.3$	$\mathbf{93.6}$	$78.3$	$53.7$	$61.9$	$66.7$	$\mathbf{61.0}$	$\mathbf{82.8}$	$74.9$	$\mathbf{91.5}$	$\mathbf{94.5}$	$94.7$
微调：
BYOL（本文）	$\mathbf{88.5}$	$\mathbf{97.8}$	$86.1$	$\mathbf{76.3}$	$63.7$	$91.6$	$\mathbf{88.1}$	$\mathbf{85.4}$	$\mathbf{76.2}$	$91.7$	$\mathbf{93.8}$	$97.0$
SimCLR（复现）	$87.5$	$97.4$	$85.3$	$75.0$	$63.9$	$91.4$	$87.6$	$84.5$	$75.4$	$89.4$	$91.7$	$96.6$
SimCLR [8]	$88.2$	$97.7$	$85.9$	$75.9$	$63.5$	$91.3$	$88.1$	$84.1$	$73.2$	$89.2$	$92.1$	$97.0$
Supervised-IN [8]	$88.3$	$97.5$	$\mathbf{86.4}$	$75.8$	$\mathbf{64.3}$	$\mathbf{92.1}$	$86.0$	$85.0$	$74.6$	$\mathbf{92.1}$	$93.3$	$\mathbf{97.6}$
随机初始化 [8]	$86.9$	$95.9$	$80.2$	$76.1$	$53.6$	$91.4$	$85.9$	$67.3$	$64.8$	$81.5$	$72.6$	$92.0$

Transfer to other vision tasks We evaluate our representation on different tasks relevant to computer vision practitioners, namely semantic segmentation, object detection and depth estimation. With this evaluation, we assess whether BYOL’s representation generalizes beyond classification tasks.

迁移到其他视觉任务 我们在对计算机视觉实践者相关的不同任务上评估表征，即语义分割、目标检测与深度估计。通过该评估，判断 BYOL 的表征是否能推广到分类以外的任务。

+1.9

1.9

+3.1

3.1

654

654

74.4

Method	AP50	mIoU
Supervised-IN [9]	$74.4$	$74.4$
MoCo [9]	$74.9$	$72.5$
SimCLR (repro)	$75.2$	$75.2$
BYOL (ours)	$\bf{77.5}$	$\bf{76.3}$

	Higher better	Lower better
Method	pct. $\!<\!1.25$	pct. $\!<\!1.25^{2}$	pct. $\!<\!1.25^{3}$	rms	rel
Supervised-IN [83]	$81.1$	$95.3$	$98.8$	$0.573$	$\bf{0.127}$
SimCLR (repro)	$83.3$	$96.5$	$99.1$	$0.557$	$0.134$
BYOL (ours)	$\bf{84.6}$	$\bf{96.7}$	$\bf{99.1}$	$\bf{0.541}$	$0.129$

74.4

方法	AP50	mIoU
Supervised-IN [9]	$74.4$	$74.4$
MoCo [9]	$74.9$	$72.5$
SimCLR（复现）	$75.2$	$75.2$
BYOL（本文）	$\mathbf{77.5}$	$\mathbf{76.3}$

	越高越好	越低越好
方法	pct. $<1.25$	pct. $<1.25^{2}$	pct. $<1.25^{3}$	rms	rel
Supervised-IN [83]	$81.1$	$95.3$	$98.8$	$0.573$	$\mathbf{0.127}$
SimCLR（复现）	$83.3$	$96.5$	$99.1$	$0.557$	$0.134$
BYOL（本文）	$\mathbf{84.6}$	$\mathbf{96.7}$	$\mathbf{99.1}$	$\mathbf{0.541}$	$0.129$

5 Building intuitions with ablations

5 用消融建立直觉

0.25

0.25

128

128

256

256

(a) Impact of batch size

(a) 批次大小的影响

Image augmentations Contrastive methods are sensitive to the choice of image augmentations. For instance, SimCLR does not work well when removing color distortion from its image augmentations. As an explanation, SimCLR shows that crops of the same image mostly share their color histograms. At the same time, color histograms vary across images. Therefore, when a contrastive task only relies on random crops as image augmentations, it can be mostly solved by focusing on color histograms alone. As a result the representation is not incentivized to retain information beyond color histograms. To prevent that, SimCLR adds color distortion to its set of image augmentations. Instead, BYOL is incentivized to keep any information captured by the target representation into its online network, to improve its predictions. Therefore, even if augmented views of a same image share the same color histogram, BYOL is still incentivized to retain additional features in its representation. For that reason, we believe that BYOL is more robust to the choice of image augmentations than contrastive methods.

图像增强 对比方法对图像增强选择敏感。例如，SimCLR 在去掉颜色扰动后表现不佳。一种解释是：同一图像的裁剪大多共享颜色直方图，而颜色直方图在不同图像间变化。因此，当对比任务只依赖随机裁剪作为增强时，主要关注颜色直方图就能大体完成任务，表征便没有动力保留颜色直方图之外的信息。为防止这一点，SimCLR 在增强集合中加入颜色扰动。相反，BYOL 被激励把目标表征捕获的任何信息保留到在线网络中，以改进预测。因此，即便同一图像的增强视图共享颜色直方图，BYOL 仍被激励在表征中保留额外特征。基于此，我们相信 BYOL 对图像增强选择比对比方法更稳健。

-9.1

9.1

1

1

\tau_{\text{base}}

Target	$\tau_{\text{base}}$	Top- $1$
Constant random network	$1$	$18.8{\scriptstyle\pm 0.7}$
Moving average of online	$0.999$	$69.8$
Moving average of online	$0.99$	$\bf{72.5}$
Moving average of online	$0.9$	$68.4$
Stop gradient of online†	$0$	$0.3$

Method	Predictor	Target network	$\beta$	Top- $1$
BYOL	✓	✓	0	$\bf{72.5}$
$-$	✓	✓	1	$70.9$
$-$		✓	1	$70.7$
SimCLR			1	$69.4$
$-$	✓		1	$69.1$
$-$	✓		0	$0.3$
$-$		✓	0	$0.2$
$-$			0	$0.1$

\tau_{\mathrm{base}}

目标	$\tau_{\mathrm{base}}$	Top- $1$
恒定随机网络	$1$	$18.8{\scriptstyle\pm 0.7}$
在线网络的滑动平均	$0.999$	$69.8$
在线网络的滑动平均	$0.99$	$\mathbf{72.5}$
在线网络的滑动平均	$0.9$	$68.4$
在线网络的停止梯度	$0$	$0.3$

方法	预测器	目标网络	$\beta$	Top- $1$
BYOL	✓	✓	0	$\mathbf{72.5}$
$-$	✓	✓	1	$70.9$
$-$		✓	1	$70.7$
SimCLR			1	$69.4$
$-$	✓		1	$69.1$
$-$	✓		0	$0.3$
$-$		✓	0	$0.2$
$-$			0	$0.1$

Ablation to contrastive methods In this subsection, we recast SimCLR and BYOL using the same formalism to better understand where the improvement of BYOL over SimCLR comes from. Let us consider the following objective that extends the InfoNCE objective [10, 84] (see Section F.4),

相对对比方法的消融 本小节用同一形式体系重写 SimCLR 与 BYOL，以更好理解 BYOL 相对 SimCLR 的改进来自何处。考虑如下扩展 InfoNCE 目标 [10, 84] 的目标（见第 F.4 节），

\displaystyle\text{InfoNCE}^{\alpha,\beta}_{{\theta}}\mathrel{\hbox{\hskip 3.8889pt\hskip-3.8889pt\hbox{$=$}\hskip-3.8889pt\hskip-2.7431pt\raisebox{4.66875pt}{\hbox{$\scriptscriptstyle\Delta$}}\hskip-2.7431pt\hskip 3.8889pt}}\frac{2}{B}\sum_{i=1}^{B}S_{\theta}(v_{i},v^{\prime}_{i})\!-\beta\cdot\frac{2\alpha}{B}\sum_{i=1}^{B}\ln\mathopen{}\mathclose{{\left(\sum\limits_{j\neq i}\exp\frac{S_{\theta}(v_{i},v_{j})}{\alpha}+\sum\limits_{j}\exp\frac{S_{\theta}(v_{i},v^{\prime}_{j})}{\alpha}}}\right)\mathbin{\raisebox{2.15277pt}{,}} \tag{6}

\alpha>0

\alpha>0

S_{\theta}(u_{1},u_{2})\mathrel{\hbox{\hskip 3.8889pt\hskip-3.8889pt\hbox{$=$}\hskip-3.8889pt\hskip-2.7431pt\raisebox{4.66875pt}{\hbox{$\scriptscriptstyle\Delta$}}\hskip-2.7431pt\hskip 3.8889pt}}\frac{\langle\phi(u_{1}),\psi(u_{2})\rangle}{\|\phi(u_{1})\|_{2}\cdot\|\psi(u_{2})\|_{2}}\cdot \tag{7}

\phi(u_{1})=z_{\theta}(u_{1})

\phi(u_{1})=z_{\theta}(u_{1})

\beta=0

\beta=0

+1.6

1.6

72\%

72\%

\ell_{2}

\ell_{2}

52.5\%

52.5\%

6 Conclusion

6 结论

50

50

Nevertheless, BYOL remains dependent on existing sets of augmentations that are specific to vision applications. To generalize BYOL to other modalities (e.g., audio, video, text, \dots) it is necessary to obtain similarly suitable augmentations for each of them. Designing such augmentations may require significant effort and expertise. Therefore, automating the search for these augmentations would be an important next step to generalize BYOL to other modalities.

尽管如此，BYOL 仍依赖现有、专用于视觉应用的增强集合。要把 BYOL 推广到其他模态（例如音频、视频、文本等），需要为每种模态获得同样合适的增强。设计此类增强可能需要大量精力与专长。因此，自动化搜索这些增强，将是把 BYOL 推广到其他模态的重要下一步。

Broader impact

更广泛的影响

The presented research should be categorized as research in the field of unsupervised learning. This work may inspire new algorithms, theoretical, and experimental investigation. The algorithm presented here can be used for many different vision applications and a particular use may have both positive or negative impacts, which is known as the dual use problem. Besides, as vision datasets could be biased, the representation learned by BYOL could be susceptible to replicate these biases.

本项研究应归入无监督学习领域的研究。本工作可能启发新的算法、理论与实验探索。此处给出的算法可用于许多不同的视觉应用，特定用途可能同时带来正面或负面影响，即所谓两用问题。此外，由于视觉数据集可能有偏，BYOL 学到的表征也可能复制这些偏置。

Acknowledgements

致谢

The authors would like to thank the following people for their help throughout the process of writing this paper, in alphabetical order: Aaron van den Oord, Andrew Brock, Jason Ramapuram, Jeffrey De Fauw, Karen Simonyan, Katrina McKinney, Nathalie Beauguerlange, Olivier Henaff, Oriol Vinyals, Pauline Luc, Razvan Pascanu, Sander Dieleman, and the DeepMind team. We especially thank Jason Ramapuram and Jeffrey De Fauw, who provided the JAX SimCLR reproduction used throughout the paper.

作者按字母顺序列出并感谢以下人士在撰写本文过程中的帮助：Aaron van den Oord、Andrew Brock、Jason Ramapuram、Jeffrey De Fauw、Karen Simonyan、Katrina McKinney、Nathalie Beauguerlange、Olivier Henaff、Oriol Vinyals、Pauline Luc、Razvan Pascanu、Sander Dieleman，以及 DeepMind 团队。我们特别感谢 Jason Ramapuram 与 Jeffrey De Fauw，他们提供了本文通篇使用的 JAX SimCLR 复现。

References

参考文献

[1] Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics , 36(4):193-202, 1980. [2] Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural Computation , 14(4), 2002. [3] Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation , 18(7):1527-1554, 2006. [4] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Computer Vision and Pattern Recognition , 2014. [5] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. [6] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition , 2014. [7] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Computer Vision and Pattern Recognition , 2015. [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 , 2020. [9] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 , 2019. [10] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

[11] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849v4 , 2019. [12] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005.10243 , 2020. [13] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning , 2019. [14] R. Manmatha, Chao-Yuan Wu, Alexander J. Smola, and Philipp Krähenbühl. Sampling matters in deep embedding learning. In International Conference on Computer Vision , 2017. [15] Ben Harwood, Vijay B. G. Kumar, Gustavo Carneiro, Ian Reid, and Tom Drummond. Smart mining for deep metric learning. In International Conference on Computer Vision , 2017. [16] Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In International Conference on Machine Learning , 2013. [17] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In European Conference on Computer Vision , 2018. [18] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances in neural information processing systems , pages 3365-3373, 2014. [19] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 , 2016. [20] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in neural information processing systems , pages 1195-1204, 2017.

[21] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision , 115(3):211-252, 2015. [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition , 2016. [23] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Computer Vision and Pattern Recognition , 2015. [24] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning , 2008. [25] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. [26] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic back-propagation and variational inference in deep latent gaussian models. arXiv preprint arXiv:1401.4082 , 2014. [27] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Processing Systems , 2014. [28] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.09782 , 2016. [29] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martín Arjovsky, Olivier Mastropietro, and Aaron C. Courville. Adversarially learned inference. arXiv preprint arXiv:1606.00704 , 2017. [30] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Neural Information Processing Systems , 2019.

[31] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018. [32] Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning , 2019. [33] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2019. [34] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Neural Information Processing Systems , 2019. [35] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. arXiv preprint arXiv:1912.01991 , 2019. [36] Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966 , 2020. [37] Rishabh Jain, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020. [38] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029 , 2020. [39] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Computer Vision and Pattern Recognition , 2018. [40] Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In International Conference on Computer Vision , 2017.

[61] David Berthelot, N. Carlini, E. D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In ICLR , 2020. [62] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685 , 2020. [63] Suzanna Becker and Geoffrey E. Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature , 355(6356):161-163, 1992. [64] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne. JAX: composable transformations of Python+NumPy programs, 2018. [65] Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020. [66] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems , pages 2672-2680, 2014. [67] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016. [68] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning , 2015. [69] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning , 2010. [70] Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888 , 2017.

[81] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Computer Vision and Pattern Recognition , 2014. [82] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Neural Information Processing Systems , 2015. [83] I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab. Deeper depth prediction with fully convolutional residual networks. In International Conference on 3D Vision , 2016. [84] Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On variational bounds of mutual information. arXiv preprint arXiv:1905.06922 , 2019. [85] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , 2015. [86] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning , 2013. [87] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. arXiv preprint arXiv:1909.13719 , 2019. [88] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Computer Vision and Pattern Recognition , pages 2818-2826, 2016. [89] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision , 2014. [90] Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L. Alexander, David W. Jacobs, and Peter N. Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Computer Vision and Pattern Recognition , 2014.

[91] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In Workshop on 3D Representation and Recognition , Sydney, Australia, 2013. [92] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013. [93] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In Computer Vision and Pattern Recognition , 2012. [94] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop , 2004. [95] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing , 2008. [96] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International Conference on Machine Learning , 2014. [97] Art B Owen. A robust hybrid of lasso and ridge regression. Contemporary Mathematics , 443(7):59-72, 2007. [98] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In International Conference on Computer Vision , 2019. [99] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Conference on Computer Vision and Pattern Recognition , 2017. [100] Yuxin Wu and Kaiming He. Group normalization. In European Conference on Computer Vision , 2018.

Appendix A Algorithm

附录 A 算法

\mathcal{D}

$\mathcal{D}$ , $\mathcal{T},$ and $\mathcal{T}^{\prime}$	set of images and distributions of transformations
${\theta}$ , $f_{\theta}$ , $g_{\theta},$ and $q_{\theta}$	initial online parameters, encoder, projector, and predictor
$\xi$ , $f_{\xi}$ , $g_{\xi}$	initial target parameters, target encoder, and target projector
$\mathrm{optimizer}$	optimizer, updates online parameters using the loss gradient
$K$ and $N$	total number of optimization steps and batch size
$\{\tau_{k}\}_{k=1}^{K}$ and $\{\eta_{k}\}_{k=1}^{K}$	target network update schedule and learning rate schedule

\mathcal{D}

$\mathcal{D}$ 、 $\mathcal{T}$ 与 $\mathcal{T}^{\prime}$	图像集合与变换分布
${\theta}$ 、 $f_{\theta}$ 、 $g_{\theta}$ 与 $q_{\theta}$	初始在线参数、编码器、投影器与预测器
$\xi$ 、 $f_{\xi}$ 、 $g_{\xi}$	初始目标参数、目标编码器与目标投影器
$\mathrm{optimizer}$	优化器，用损失梯度更新在线参数
$K$ 与 $N$	总优化步数与批次大小
$\{\tau_{k}\}_{k=1}^{K}$ 与 $\{\eta_{k}\}_{k=1}^{K}$	目标网络更新日程与学习率日程

Appendix B Image augmentations

附录 B 图像增强

During self-supervised training, BYOL uses the following image augmentations (which are a subset of the ones presented in [8]):

在自监督训练期间，BYOL 使用如下图像增强（它们是 [8] 中给出增强的子集）：

3/4

3/4

• optional left-right flip;

• 可选的左右翻转；

• color jittering: the brightness, contrast, saturation and hue of the image are shifted by a uniformly random offset applied on all the pixels of the same image. The order in which these shifts are performed is randomly selected for each patch;

• 颜色抖动：对亮度、对比度、饱和度与色调施加均匀随机偏移，同一图像的所有像素使用同一偏移；这些偏移的施加顺序对每个图块随机选定；

(r,g,b)

(r,g,b)

224\times 224

224\times 224

x\mapsto x\cdot{\bf 1}_{\{x<0.5\}}+(1-x)\cdot{\bf 1}_{\{x\geq 0.5\}}

x\mapsto x\cdot{\bf 1}_{\{x<0.5\}}+(1-x)\cdot{\bf 1}_{\{x\geq 0.5\}}

\mathcal{T}

\mathcal{T}

256

256

\mathcal{T}

Parameter	$\mathcal{T}$	$\mathcal{T}^{\prime}$
Random crop probability	$1.0$	$1.0$
Flip probability	$0.5$	$0.5$
Color jittering probability	$0.8$	$0.8$
Brightness adjustment max intensity	$0.4$	$0.4$
Contrast adjustment max intensity	$0.4$	$0.4$
Saturation adjustment max intensity	$0.2$	$0.2$
Hue adjustment max intensity	$0.1$	$0.1$
Color dropping probability	$0.2$	$0.2$
Gaussian blurring probability	$1.0$	$0.1$
Solarization probability	$0.0$	$0.2$

\mathcal{T}

参数	$\mathcal{T}$	$\mathcal{T}^{\prime}$
随机裁剪概率	$1.0$	$1.0$
翻转概率	$0.5$	$0.5$
颜色抖动概率	$0.8$	$0.8$
亮度调整最大强度	$0.4$	$0.4$
对比度调整最大强度	$0.4$	$0.4$
饱和度调整最大强度	$0.2$	$0.2$
色调调整最大强度	$0.1$	$0.1$
去色概率	$0.2$	$0.2$
高斯模糊概率	$1.0$	$0.1$
曝光过度概率	$0.0$	$0.2$

Appendix C Evaluation on ImageNet training

附录 C 在 ImageNet 训练上的评估

C.1 Self-supervised learning evaluation on ImageNet

C.1 ImageNet 上的自监督学习评估

224\times 224

224\times 224

Variant on linear evaluation on ImageNet In this paragraph only, we deviate from the protocol of [8, 37] and propose another way of performing linear evaluation on top of a frozen representation. This method achieves better performance both in top-1 and top-5 accuracy.

ImageNet 线性评估的变体 仅在本段中，我们偏离 [8, 37] 的协议，并提出另一种在冻结表征上做线性评估的方式。该方法在 top-1 与 top-5 精度上都能取得更好表现。

224\times 224

224\times 224

{\rm tclip}(x)\triangleq\alpha\cdot\tanh(x/\alpha),

{\rm tclip}(x)\triangleq\alpha\cdot\tanh(x/\alpha),

1

50

1

Architecture	Pre-train augmentations	Logits regularization	Top- $1$	Top- $5$
ResNet- $50$ ( $1\times$ )			$74.3$	$91.6$
✓		$74.4$	$91.8$
	✓	$74.7$	$91.8$
✓	✓	$\bf{74.8}$	$\bf{91.8}$
ResNet- $50$ ( $4\times$ )			$78.6$	$94.2$
✓		$78.6$	$94.3$
	✓	$78.9$	$94.3$
✓	✓	$\bf{79.0}$	$\bf{94.5}$
ResNet- $200$ ( $2\times$ )			$79.6$	$94.8$
✓		$79.6$	$94.8$
	✓	$79.8$	$95.0$
✓	✓	$\bf{80.0}$	$\bf{95.0}$

224\times 224

224\times 224

(a) Top-1 accuracy

(a) Top-1 精度

100\%

Supervised:	Semi-supervised ( $100\%$ ):
Method	Top- $1$	Top- $5$	Method	Top- $1$	Top- $5$
Supervised[8]	$76.5$	$-$	SimCLR [8]	$76.0$	$93.1$
AutoAugment [87]	$77.6$	$93.8$	SimCLR (repro)	$76.5$	$93.5$
MaxUp [75]	$\bf{78.9}$	$\bf{94.2}$	BYOL	$\bf{77.7}$	$\bf{93.9}$

1

1\%

1

50

76.5

76.5

Figure 5: Semi-supervised training with a fraction of ImageNet labels on multiple ResNets architecture pretrained with BYOL. Note that large networks are facing overfitting problems.

图 5：在多种用 BYOL 预训练的 ResNet 架构上，用一部分 ImageNet 标签做半监督训练。注意大型网络面临过拟合问题。

C.2 Linear evaluation on larger architectures and supervised baselines

C.2 更大架构上的线性评估与监督基线

1\times

			BYOL	Supervised (ours)	Supervised [8]
Architecture	Multiplier	Weights	Top-1	Top-5	Top-1	Top-5	Top-1
ResNet-50	$1\times$	24M	$74.3$	$91.6$	$76.4$	$92.9$	$76.5$
ResNet-101	$1\times$	43M	$76.4$	$93.0$	$78.0$	$94.0$	-
ResNet-152	$1\times$	58M	$77.3$	$93.7$	$79.1$	$94.5$	-
ResNet-200	$1\times$	63M	$77.8$	$93.9$	$79.3$	$94.6$	-
ResNet-50	$2\times$	94M	$77.4$	$93.6$	$79.9$	$95.0$	$77.8$
ResNet-101	$2\times$	170M	$78.7$	$94.3$	$80.3$	$95.0$	-
ResNet-50	$3\times$	211M	$78.2$	$93.9$	$80.2$	$95.0$	-
ResNet-152	$2\times$	232M	$79.0$	$94.6$	$80.6$	$95.3$	-
ResNet-200	$2\times$	250M	$\bf{79.6}$	$\bf{94.9}$	$80.1$	$95.2$	-
ResNet-50	$4\times$	375M	$78.6$	$94.2$	$80.7$	$\bf{95.3}$	$78.9$
ResNet-101	$3\times$	382M	$78.4$	$94.2$	$80.7$	$\bf{95.3}$	-
ResNet-152	$3\times$	522M	$79.5$	$94.6$	$\bf{80.9}$	$95.2$	-

50

50

Figure 6: Results for linear evaluation of BYOL compared to fully supervised baselines with various ResNet architectures. Our supervised baselines are ran with RandAugment [87] augmentations.

图 6：多种 ResNet 架构下，BYOL 线性评估相对完全监督基线的结果。我们的监督基线使用 RandAugment [87] 增强。

Appendix D Transfer to other datasets

附录 D 迁移到其他数据集

D.1 Datasets

D.1 数据集

1000

Dataset	Classes	Original train examples	Train examples	Valid. examples	Test examples	Accuracy measure	Test provided
ImageNet [21]	$1000$	$1281167$	$1271158$	$10009$	$50000$	Top-1 accuracy	-
Food101 [89]	$101$	$75750$	$68175$	$7575$	$25250$	Top-1 accuracy	-
CIFAR-10 [78]	$10$	$50000$	$45000$	$5000$	$10000$	Top-1 accuracy	-
CIFAR-100 [78]	$100$	$50000$	$44933$	$5067$	$10000$	Top-1 accuracy	-
Birdsnap [90]	$500$	$47386$	$42405$	$4981$	$2443$	Top-1 accuracy	-
Sun397 (split 1) [79]	$397$	$19850$	$15880$	$3970$	$19850$	Top-1 accuracy	-
Cars [91]	$196$	$8144$	$6494$	$1650$	$8041$	Top-1 accuracy	-
Aircraft [92]	$100$	$3334$	$3334$	$3333$	$3333$	Mean per-class accuracy	Yes
PASCAL-VOC2007 [80]	$20$	$5011$	$2501$	$2510$	$4952$	11-point mAP / AP50	-
PASCAL-VOC2012 [80]	$21$	$10582$	$-$	$2119$	$1449$	Mean IoU	-
DTD (split 1) [81]	$47$	$1880$	$1880$	$1880$	$1880$	Top-1 accuracy	Yes
Pets [93]	$37$	$3680$	$2940$	$740$	$3669$	Mean per-class accuracy	-
Caltech-101 [94]	$101$	$3060$	$2550$	$510$	$6084$	Mean per-class accuracy	-
Places365 [73]	$365$	$1803460$	$1803460$	$-$	$36500$	Top-1 accuracy	-
Flowers [95]	$102$	$1020$	$1020$	$1020$	$6149$	Mean per-class accuracy	Yes

1000

数据集	类别数	原始训练样本	训练样本	验证样本	测试样本	精度指标	是否提供测试集
ImageNet [21]	$1000$	$1281167$	$1271158$	$10009$	$50000$	Top-1 精度	-
Food101 [89]	$101$	$75750$	$68175$	$7575$	$25250$	Top-1 精度	-
CIFAR-10 [78]	$10$	$50000$	$45000$	$5000$	$10000$	Top-1 精度	-
CIFAR-100 [78]	$100$	$50000$	$44933$	$5067$	$10000$	Top-1 精度	-
Birdsnap [90]	$500$	$47386$	$42405$	$4981$	$2443$	Top-1 精度	-
Sun397 (split 1) [79]	$397$	$19850$	$15880$	$3970$	$19850$	Top-1 精度	-
Cars [91]	$196$	$8144$	$6494$	$1650$	$8041$	Top-1 精度	-
Aircraft [92]	$100$	$3334$	$3334$	$3333$	$3333$	每类平均精度	Yes
PASCAL-VOC2007 [80]	$20$	$5011$	$2501$	$2510$	$4952$	11-point mAP / AP50	-
PASCAL-VOC2012 [80]	$21$	$10582$	$-$	$2119$	$1449$	Mean IoU	-
DTD (split 1) [81]	$47$	$1880$	$1880$	$1880$	$1880$	Top-1 精度	Yes
Pets [93]	$37$	$3680$	$2940$	$740$	$3669$	每类平均精度	-
Caltech-101 [94]	$101$	$3060$	$2550$	$510$	$6084$	每类平均精度	-
Places365 [73]	$365$	$1803460$	$1803460$	$-$	$36500$	Top-1 精度	-
Flowers [95]	$102$	$1020$	$1020$	$1020$	$6149$	每类平均精度	Yes

101

101

1

1

1

1

• Point 11-mAP : We compute the empirical mean average precision as defined in [80].

• Point 11-mAP ：我们计算 [80] 所定义的经验平均精度均值。

• Mean IoU: We compute the empirical mean Intersection-Over-Union as defined in [80].

• *Mean IoU：*我们计算 [80] 所定义的经验交并比均值。

• AP50: We compute the Average Precision as defined in [80].

• *AP50：*我们计算 [80] 所定义的平均精度。

We detail the validation procedures for some specific datasets:

我们对若干特定数据集的验证流程说明如下：

• For Sun397 [79], the original dataset specifies 10 train/test splits, all of which contain 50 examples/images of 397 different classes. We use the first train/test split. The original dataset specifies no validation split and therefore, the training images have been further subdivided into 40 images per class for the train split and 10 images per class for the valid split.

• 对 Sun397 [79]，原数据集指定 10 个训练/测试划分，每个划分含 397 类、每类 50 张样本/图像。我们使用第一个训练/测试划分。原数据集未指定验证划分，因此训练图像被进一步划分为：训练划分每类 40 张，验证划分每类 10 张。

• For Birdsnap [90], we use a random selection of valid images with the same number of images per category as the test split.

• 对 Birdsnap [90]，我们随机选取与测试划分每类图像数相同的验证图像。

• For DTD [81], the original dataset specifies 10 train/validation/test splits, we only use the first split.

• 对 DTD [81]，原数据集指定 10 个训练/验证/测试划分，我们只使用第一个划分。

• For Caltech-101 [94], the original does not dataset specifies any train/test splits. We have followed the approach used in [96]: This file defines datasets for 5 random splits of 25 training images per category, with 5 validation images per category and the remaining images used for testing.

• 对 Caltech-101 [94]，原数据集未指定任何训练/测试划分。我们遵循 [96] 的做法：该文件定义了 5 个随机划分，每类 25 张训练图像、5 张验证图像，其余用于测试。

10009

10009

• For Oxford-IIIT Pets, the valid set consists of 20 randomly selected images per class.

• 对 Oxford-IIIT Pets，验证集由每类随机选取的 20 张图像组成。

Information about the dataset are summarized in Table 10.

关于数据集的信息总结于表 10。

D.2 Transfer via linear classification

D.2 经线性分类的迁移

224

224

D.3 Transfer via fine-tuning

D.3 经微调的迁移

224\times 224

224\times 224

D.4 Implementation details for semantic segmentation

D.4 语义分割实现细节

50

50

2119

2119

D.5 Implementation details for object detection

D.5 目标检测实现细节

50

50

D.6 Implementation details for depth estimation

D.6 深度估计实现细节

50

50

2

2

0.3

0.3

0.1255

0.1255

• Saturation with a saturation factor randomly picked in the interval \mathopen{}\mathclose{{\left[0.5,1.5}}\right].

[0.5,1.5]

• Hue with a hue adjustment factor randomly picked in the interval \mathopen{}\mathclose{{\left[-0.2,0.2}}\right].

[-0.2,0.2]

7500

7500

D.7 Further comparisons on PASCAL and NYU v2 Depth

D.7 PASCAL 与 NYU v2 Depth 上的进一步比较

For completeness, Table 11 and 12 extends Table 4 with other published baselines which use comparable networks. We see that in almost all settings, BYOL outperforms these baselines, even when those baselines use more data or deeper models. One notable exception is RMS error for NYU Depth prediction, which is a metric that’s sensitive to outliers. The reason for this is unclear, but one possibility is that the network is producing higher-variance predictions due to being more confident about a test-set scene’s similarities with those in the training set.

为完整起见，表 11 与表 12 在表 4 之外补充了使用可比网络的其他已发表基线。几乎在所有设定下 BYOL 都优于这些基线，即便对方用了更多数据或更深模型。一个明显例外是 NYU 深度预测的 RMS 误差——该指标对异常值敏感。原因尚不清楚，一种可能是网络对测试场景与训练集相似性更有把握，从而给出方差更高的预测。

74.4

Method	AP50	mIoU
Supervised-IN [9]	$74.4$	$74.4$
RelPos [23], by [40]∗	$66.8$	-
Multi-task [40]∗	$70.5$	-
LocalAgg [98]	$69.1$	-
MoCo [9]	$74.9$	$72.5$
MoCo + IG-1B [9]	$75.6$	$73.6$
CPC[32]∗∗	$76.6$	-
SimCLR (repro)	$75.2$	$75.2$
BYOL (ours)	$\bf{77.5}$	$\bf{76.3}$

74.4

方法	AP50	mIoU
Supervised-IN [9]	$74.4$	$74.4$
RelPos [23], by [40]∗	$66.8$	-
Multi-task [40]∗	$70.5$	-
LocalAgg [98]	$69.1$	-
MoCo [9]	$74.9$	$72.5$
MoCo + IG-1B [9]	$75.6$	$73.6$
CPC[32]∗∗	$76.6$	-
SimCLR（复现）	$75.2$	$75.2$
BYOL（本文）	$\bf{77.5}$	$\bf{76.3}$

\!<\!1.25

	Higher better	Lower better
Method	pct. $\!<\!1.25$	pct. $\!<\!1.25^{2}$	pct. $\!<\!1.25^{3}$	rms	rel
Supervised-IN [83]	$81.1$	$95.3$	$98.8$	$0.573$	$\bf{0.127}$
RelPos [23], by [40]∗	$80.6$	$94.7$	$98.3$	$\bf{0.399}$	$0.146$
Color [41], by [40]∗	$76.8$	$93.5$	$97.7$	$0.444$	$0.164$
Exemplar [46, 40]∗	$71.3$	$90.6$	$96.5$	$0.513$	$0.191$
Mot. Seg. [99], by [40]∗	$74.2$	$92.4$	$97.4$	$0.473$	$0.177$
Multi-task [40]∗	$79.3$	$94.2$	$98.1$	$0.422$	$0.152$
SimCLR (repro)	$83.3$	$96.5$	$99.1$	$0.557$	$0.134$
BYOL (ours)	$\bf{84.6}$	$\bf{96.7}$	$\bf{99.1}$	$0.541$	$0.129$

\!<\!1.25

	越高越好	越低越好
方法	pct. $\!<\!1.25$	pct. $\!<\!1.25^{2}$	pct. $\!<\!1.25^{3}$	rms	rel
Supervised-IN [83]	$81.1$	$95.3$	$98.8$	$0.573$	$\bf{0.127}$
RelPos [23], by [40]∗	$80.6$	$94.7$	$98.3$	$\bf{0.399}$	$0.146$
Color [41], by [40]∗	$76.8$	$93.5$	$97.7$	$0.444$	$0.164$
Exemplar [46, 40]∗	$71.3$	$90.6$	$96.5$	$0.513$	$0.191$
Mot. Seg. [99], by [40]∗	$74.2$	$92.4$	$97.4$	$0.473$	$0.177$
Multi-task [40]∗	$79.3$	$94.2$	$98.1$	$0.422$	$0.152$
SimCLR（复现）	$83.3$	$96.5$	$99.1$	$0.557$	$0.134$
BYOL（本文）	$\bf{84.6}$	$\bf{96.7}$	$\bf{99.1}$	$0.541$	$0.129$

Appendix E Pretraining on Places 365

附录 E 在 Places 365 上预训练

365

365

55.2\%

50

51.0

Method	Places365	ImageNet	Food101	CIFAR10	CIFAR100	Birdsnap	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers
BYOL-IN	$51.0$	$74.3$	$75.3$	$91.3$	$78.4$	$57.3$	$62.6$	$67.2$	$60.6$	$76.5$	$90.4$	$94.3$	$96.1$
BYOL-PL	$\bf{53.2}$	$\bf{58.5}$	$\bf{64.7}$	$\bf{84.5}$	$\bf{66.1}$	$\bf{28.8}$	$\bf{64.2}$	$\bf{55.6}$	$\bf{55.9}$	$\bf{68.5}$	$\bf{66.1}$	$\bf{84.3}$	$\bf{90.0}$
SimCLR-PL	$53.0$	$56.5$	$61.7$	$80.8$	$61.1$	$21.2$	$62.5$	$40.1$	$44.3$	$64.3$	$59.4$	$77.1$	$85.9$

51.0

方法	Places365	ImageNet	Food101	CIFAR10	CIFAR100	Birdsnap	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers
BYOL-IN	$51.0$	$74.3$	$75.3$	$91.3$	$78.4$	$57.3$	$62.6$	$67.2$	$60.6$	$76.5$	$90.4$	$94.3$	$96.1$
BYOL-PL	$\bf{53.2}$	$\bf{58.5}$	$\bf{64.7}$	$\bf{84.5}$	$\bf{66.1}$	$\bf{28.8}$	$\bf{64.2}$	$\bf{55.6}$	$\bf{55.9}$	$\bf{68.5}$	$\bf{66.1}$	$\bf{84.3}$	$\bf{90.0}$
SimCLR-PL	$53.0$	$56.5$	$61.7$	$80.8$	$61.1$	$21.2$	$62.5$	$40.1$	$44.3$	$64.3$	$59.4$	$77.1$	$85.9$

Appendix F Additional ablation results

附录 F 额外消融结果

3

300

F.1 Architecture settings

F.1 架构设置

1

1

g_{\theta}

Proj. $g_{\theta}$ depth	Pred. $q_{\theta}$ depth	Top- $1$	Top- $5$
$1$	$1$	$61.9$	$86.0$
	$2$	$65.0$	$86.8$
	$3$	$65.7$	$86.8$
$2$	$1$	$71.5$	$90.7$
	$2$	$\bf{72.5}$	$\bf{90.8}$
	$3$	$71.4$	$90.4$
$3$	$1$	$71.4$	$90.4$
	$2$	$72.1$	$90.5$
	$3$	$72.1$	$90.5$

g_{\theta}

投影器 $g_{\theta}$ 深度	预测器 $q_{\theta}$ 深度	Top- $1$	Top- $5$
$1$	$1$	$61.9$	$86.0$
	$2$	$65.0$	$86.8$
	$3$	$65.7$	$86.8$
$2$	$1$	$71.5$	$90.7$
	$2$	$\bf{72.5}$	$\bf{90.8}$
	$3$	$71.4$	$90.4$
$3$	$1$	$71.4$	$90.4$
	$2$	$72.1$	$90.5$
	$3$	$72.1$	$90.5$

Table 15(a) shows the influence of the initial learning rate on BYOL. Note that the optimal value depends on the number of training epochs. Table 15(b) displays the influence of the weight decay on BYOL.

表 15(a) 显示初始学习率对 BYOL 的影响。注意最优值依赖训练 epoch 数。表 15(b) 显示权重衰减对 BYOL 的影响。

1

Learning
rate	Top- $1$	Top- $5$
$0.01$	$34.8{\scriptstyle\pm 3.0}$	$60.8{\scriptstyle\pm 3.2}$
$0.1$	$65.0$	$87.0$
$0.2$	$71.7$	$90.6$
$0.3$	$\bf{72.5}$	$\bf{90.8}$
$0.4$	$72.3$	$90.6$
$0.5$	$71.5$	$90.1$
$1$	$69.4$	$89.2$

Weight decay coefficient	Top- $1$	Top- $5$
$1\cdot 10^{-7}$	$72.1$	$90.4$
$5\cdot 10^{-7}$	$\bf{72.6}$	$\bf{91.0}$
$1\cdot 10^{-6}$	$72.5$	$90.8$
$5\cdot 10^{-6}$	$71.0{\scriptstyle\pm 0.3}$	$90.0$
$1\cdot 10^{-5}$	$69.6{\scriptstyle\pm 0.4}$	$89.3$

1

学习
率	Top- $1$	Top- $5$
$0.01$	$34.8{\scriptstyle\pm 3.0}$	$60.8{\scriptstyle\pm 3.2}$
$0.1$	$65.0$	$87.0$
$0.2$	$71.7$	$90.6$
$0.3$	$\bf{72.5}$	$\bf{90.8}$
$0.4$	$72.3$	$90.6$
$0.5$	$71.5$	$90.1$
$1$	$69.4$	$89.2$

权重衰减系数	Top- $1$	Top- $5$
$1\cdot 10^{-7}$	$72.1$	$90.4$
$5\cdot 10^{-7}$	$\bf{72.6}$	$\bf{91.0}$
$1\cdot 10^{-6}$	$72.5$	$90.8$
$5\cdot 10^{-6}$	$71.0{\scriptstyle\pm 0.3}$	$90.0$
$1\cdot 10^{-5}$	$69.6{\scriptstyle\pm 0.4}$	$89.3$

F.2 Batch size

F.2 批次大小

N

N

1

Batch	Top- $1$	Top- $5$
size	BYOL (ours)	SimCLR (repro)	BYOL (ours)	SimCLR (repro)
$4096$	$\bf{72.5}$	$67.9$	$\bf{90.8}$	$88.5$
$2048$	$72.4$	$67.8$	$90.7$	$88.5$
$1024$	$72.2$	$67.4$	$90.7$	$88.1$
$512$	$72.2$	$66.5$	$90.8$	$87.6$
$256$	$71.8$	$64.3{\scriptstyle\pm 2.1}$	$90.7$	$86.3{\scriptstyle\pm 1.0}$
$128$	$69.6{\scriptstyle\pm 0.5}$	$63.6$	$89.6$	$85.9$
$64$	$59.7{\scriptstyle\pm 1.5}$	$59.2{\scriptstyle\pm 2.9}$	$83.2{\scriptstyle\pm 1.2}$	$83.0{\scriptstyle\pm 1.9}$

1

Batch	Top- $1$	Top- $5$
size	BYOL（本文）	SimCLR（复现）	BYOL（本文）	SimCLR（复现）
$4096$	$\bf{72.5}$	$67.9$	$\bf{90.8}$	$88.5$
$2048$	$72.4$	$67.8$	$90.7$	$88.5$
$1024$	$72.2$	$67.4$	$90.7$	$88.1$
$512$	$72.2$	$66.5$	$90.8$	$87.6$
$256$	$71.8$	$64.3{\scriptstyle\pm 2.1}$	$90.7$	$86.3{\scriptstyle\pm 1.0}$
$128$	$69.6{\scriptstyle\pm 0.5}$	$63.6$	$89.6$	$85.9$
$64$	$59.7{\scriptstyle\pm 1.5}$	$59.2{\scriptstyle\pm 2.9}$	$83.2{\scriptstyle\pm 1.2}$	$83.0{\scriptstyle\pm 1.9}$

F.3 Image augmentations

F.3 图像增强

0.5

\mathcal{T}

1

	Top- $1$	Top- $5$
Image augmentation	BYOL (ours)	SimCLR (repro)	BYOL (ours)	SimCLR (repro)
Baseline	$\bf{72.5}$	$67.9$	$\bf{90.8}$	$88.5$
Remove flip	$71.9$	$67.3$	$90.6$	$88.2$
Remove blur	$71.2$	$65.2$	$90.3$	$86.6$
Remove color (jittering and grayscale)	$63.4{\scriptstyle\pm 0.7}$	$45.7$	$85.3{\scriptstyle\pm 0.5}$	$70.6$
Remove color jittering	$71.8$	$63.7$	$90.7$	$85.9$
Remove grayscale	$70.3$	$61.9$	$89.8$	$84.1$
Remove blur in $\mathcal{T}^{\prime}$	$72.4$	$67.5$	$90.8$	$88.4$
Remove solarize in $\mathcal{T}^{\prime}$	$72.3$	$67.7$	$90.8$	$88.2$
Remove blur and solarize in $\mathcal{T}^{\prime}$	$72.2$	$67.4$	$90.8$	$88.1$
Symmetric blurring/solarization	$72.5$	$68.1$	$90.8$	$88.4$
Crop only	$59.4{\scriptstyle\pm 0.3}$	$40.3{\scriptstyle\pm 0.3}$	$82.4$	$64.8{\scriptstyle\pm 0.4}$
Crop and flip only	$60.1{\scriptstyle\pm 0.3}$	$40.2$	$83.0{\scriptstyle\pm 0.3}$	$64.8$
Crop and color only	$70.7$	$64.2$	$90.0$	$86.2$
Crop and blur only	$61.1{\scriptstyle\pm 0.3}$	$41.7$	$83.9$	$66.4$

1

	Top- $1$	Top- $5$
图像增强	BYOL（本文）	SimCLR（复现）	BYOL（本文）	SimCLR（复现）
基线	$\bf{72.5}$	$67.9$	$\bf{90.8}$	$88.5$
去掉翻转	$71.9$	$67.3$	$90.6$	$88.2$
去掉模糊	$71.2$	$65.2$	$90.3$	$86.6$
去掉颜色（抖动与灰度）	$63.4{\scriptstyle\pm 0.7}$	$45.7$	$85.3{\scriptstyle\pm 0.5}$	$70.6$
去掉颜色抖动	$71.8$	$63.7$	$90.7$	$85.9$
去掉灰度	$70.3$	$61.9$	$89.8$	$84.1$
在 $\mathcal{T}^{\prime}$ 中去掉模糊	$72.4$	$67.5$	$90.8$	$88.4$
在 $\mathcal{T}^{\prime}$ 中去掉日晒化	$72.3$	$67.7$	$90.8$	$88.2$
在 $\mathcal{T}^{\prime}$ 中去掉模糊与日晒化	$72.2$	$67.4$	$90.8$	$88.1$
对称模糊/日晒化	$72.5$	$68.1$	$90.8$	$88.4$
仅裁剪	$59.4{\scriptstyle\pm 0.3}$	$40.3{\scriptstyle\pm 0.3}$	$82.4$	$64.8{\scriptstyle\pm 0.4}$
仅裁剪与翻转	$60.1{\scriptstyle\pm 0.3}$	$40.2$	$83.0{\scriptstyle\pm 0.3}$	$64.8$
仅裁剪与颜色	$70.7$	$64.2$	$90.0$	$86.2$
仅裁剪与模糊	$61.1{\scriptstyle\pm 0.3}$	$41.7$	$83.9$	$66.4$

\beta

Loss weight $\beta$	Temperature $\alpha$	Top- $1$	Top- $5$
$0$	$0.1$	$72.5$	$90.8$
$0.1$	$0.01$	$72.2$	$90.7$
$0.1$	$72.4$	$90.9$
$0.3$	$\bf{72.7}$	$91.0$
$1$	$72.6$	$90.9$
$3$	$72.5$	$90.9$
$10$	$72.5$	$90.9$
$0.5$	$0.01$	$70.9$	$90.2$
$0.1$	$72.0$	$90.8$
$0.3$	$\bf{72.7}$	$\bf{91.2}$
$1$	$72.7$	$91.1$
$3$	$72.6$	$91.1$
$10$	$72.5$	$91.0$
$1$	$0.01$	$53.9{\scriptstyle\pm 0.5}$	$77.5{\scriptstyle\pm 0.5}$
$0.1$	$70.9$	$90.3$
$0.3$	$\bf{72.7}$	$91.1$
$1$	$\bf{72.7}$	$91.1$
$3$	$72.6$	$91.0$
$10$	$72.6$	$91.1$

\beta

损失权重 $\beta$	温度 $\alpha$	Top- $1$	Top- $5$
$0$	$0.1$	$72.5$	$90.8$
$0.1$	$0.01$	$72.2$	$90.7$
$0.1$	$72.4$	$90.9$
$0.3$	$\bf{72.7}$	$91.0$
$1$	$72.6$	$90.9$
$3$	$72.5$	$90.9$
$10$	$72.5$	$90.9$
$0.5$	$0.01$	$70.9$	$90.2$
$0.1$	$72.0$	$90.8$
$0.3$	$\bf{72.7}$	$\bf{91.2}$
$1$	$72.7$	$91.1$
$3$	$72.6$	$91.1$
$10$	$72.5$	$91.0$
$1$	$0.01$	$53.9{\scriptstyle\pm 0.5}$	$77.5{\scriptstyle\pm 0.5}$
$0.1$	$70.9$	$90.3$
$0.3$	$\bf{72.7}$	$91.1$
$1$	$\bf{72.7}$	$91.1$
$3$	$72.6$	$91.0$
$10$	$72.6$	$91.1$

F.4 Details on the relation to contrastive methods

F.4 与对比方法关系的细节

As mentioned in Section 5, the BYOL loss Equation 2 can be derived from the InfoNCE loss

如第 5 节所述，BYOL 损失（方程 2）可由 InfoNCE 损失导出

\text{InfoNCE}^{\alpha,\beta}_{{\theta}}\mathrel{\hbox{\hskip 3.8889pt\hskip-3.8889pt\hbox{$=$}\hskip-3.8889pt\hskip-2.7431pt\raisebox{4.66875pt}{\hbox{$\scriptscriptstyle\Delta$}}\hskip-2.7431pt\hskip 3.8889pt}}\frac{2}{B}\sum_{i=1}^{B}S_{\theta}(v_{i},v^{\prime}_{i})\!-\!\frac{2\alpha\cdot\beta}{B}\sum_{i=1}^{B}\ln\mathopen{}\mathclose{{\left(\sum\limits_{j\neq i}\exp\frac{S_{\theta}(v_{i},v_{j})}{\alpha}+\sum\limits_{j}\exp\frac{S_{\theta}(v_{i},v^{\prime}_{j})}{\alpha}}}\right)\mathbin{\raisebox{2.15277pt}{,}} \tag{8}

with

其中

S_{\theta}(u_{1},u_{2})\mathrel{\hbox{\hskip 3.8889pt\hskip-3.8889pt\hbox{$=$}\hskip-3.8889pt\hskip-2.7431pt\raisebox{4.66875pt}{\hbox{$\scriptscriptstyle\Delta$}}\hskip-2.7431pt\hskip 3.8889pt}}\frac{\langle\phi(u_{1}),\psi(u_{2})\rangle}{\|\phi(u_{1})\|_{2}\cdot\|\psi(u_{2})\|_{2}}\cdot \tag{9}

The InfoNCE loss, introduced in [10], can be found in factored form in [84] as

InfoNCE 损失由 [10] 引入，可在 [84] 中找到因式形式

\text{InfoNCE}_{{\theta}}\mathrel{\hbox{\hskip 3.8889pt\hskip-3.8889pt\hbox{$=$}\hskip-3.8889pt\hskip-2.7431pt\raisebox{4.66875pt}{\hbox{$\scriptscriptstyle\Delta$}}\hskip-2.7431pt\hskip 3.8889pt}}\frac{1}{B}\sum_{i=1}^{B}\ln\cfrac{f(v_{i},v^{\prime}_{i})}{\frac{1}{B}\sum\limits_{j}\exp f(v_{i},v^{\prime}_{j})}\,\cdot \tag{10}

(v_{i},v_{j})_{j\neq i}

(v_{i},v_{j})_{j\neq i}

\displaystyle\frac{1}{B}\sum_{i=1}^{B}\ln\cfrac{\exp f(v_{i},v^{\prime}_{i})}{\frac{1}{B}\sum\limits_{j\neq i}\exp f(v_{i},v_{j})+\frac{1}{B}\sum\limits_{j}\exp f(v_{i},v^{\prime}_{j})} \\ \displaystyle\qquad=\ln B+\frac{1}{B}\sum_{i=1}^{B}f(v_{i},v^{\prime}_{i})-\frac{1}{B}\sum_{i=1}^{B}\ln\mathopen{}\mathclose{{\left(\sum\limits_{j\neq i}\exp f(v_{i},v_{j})+\sum\limits_{j}\exp f(v_{i},v^{\prime}_{j})}}\right). \tag{11}

\ln B

\ln B

\alpha

\alpha

Using a target network in the loss has two effects: stopping the gradient through the prediction targets and stabilizing the targets with averaging. Stopping the gradient through the target change the objective while averaging makes the target stable and stale. In Table 5(b) we only shows results of the ablation when either using the online network as the prediction target (and flowing the gradient through it) or with a target network (both stopping the gradient into the prediction targets and computing the prediction targets with a moving average of the online network). We shown in Table 5(b) that using a target network is beneficial but it has two distinct effects we would like to understand from which effect the improvement comes from. We report in Table 19 the results already in Table 5(b) but also when the prediction target is computed with a stop gradient of the online network (the gradient does not flow into the prediction targets). This shows that making the prediction targets stable and stale is the main cause of the improvement rather than the change in the objective due to the stop gradient.

在损失里使用目标网络有两重效果：截断经预测目标的梯度，以及用平均使目标稳定。截断梯度会改变目标函数，平均则让目标稳定且“陈旧”。表 5(b) 只给出两类消融：要么用在线网络作预测目标（梯度穿过去），要么用目标网络（既截断进入预测目标的梯度，又用在线网络的滑动平均算预测目标）。表 5(b) 表明使用目标网络有益，但这两种效果混在一起，我们想弄清增益主要来自哪一种。表 19 在表 5(b) 已有结果之外，补充了用在线网络的停梯度作为预测目标（梯度不流入预测目标）的情形。结果表明：让预测目标稳定且陈旧，才是主要增益来源，而不是停梯度带来的目标函数变化。

\beta

Method	Predictor	Target parameters	$\beta$	Top-1
BYOL	✓	$\xi$	0	$\bf{72.5}$
	✓	$\xi$	1	$70.9$
		$\xi$	1	$70.7$
	✓	$\mathrm{sg}({\theta})$	1	$70.2$
SimCLR		${\theta}$	1	$69.4$
	✓	$\mathrm{sg}({\theta})$	1	$70.1$
		$\mathrm{sg}({\theta})$	1	$69.2$
	✓	${\theta}$	1	$69.0$
	✓	$\mathrm{sg}({\theta})$	0	$5.5$
	✓	${\theta}$	0	$0.3$
		$\xi$	0	$0.2$
		$\mathrm{sg}({\theta})$	0	$0.1$
		${\theta}$	0	$0.1$

\beta

方法	预测器	目标参数	$\beta$	Top-1
BYOL	✓	$\xi$	0	$\bf{72.5}$
	✓	$\xi$	1	$70.9$
		$\xi$	1	$70.7$
	✓	$\mathrm{sg}({\theta})$	1	$70.2$
SimCLR		${\theta}$	1	$69.4$
	✓	$\mathrm{sg}({\theta})$	1	$70.1$
		$\mathrm{sg}({\theta})$	1	$69.2$
	✓	${\theta}$	1	$69.0$
	✓	$\mathrm{sg}({\theta})$	0	$5.5$
	✓	${\theta}$	0	$0.3$
		$\xi$	0	$0.2$
		$\mathrm{sg}({\theta})$	0	$0.1$
		${\theta}$	0	$0.1$

F.5 SimCLR baseline of Section 5

F.5 第 5 节的 SimCLR 基线

\beta=1

\beta=1

\displaystyle\text{InfoNCE}_{{\theta}}\mathrel{\hbox{\hskip 3.8889pt\hskip-3.8889pt\hbox{$=$}\hskip-3.8889pt\hskip-2.7431pt\raisebox{4.66875pt}{\hbox{$\scriptscriptstyle\Delta$}}\hskip-2.7431pt\hskip 3.8889pt}}\frac{1}{B}\sum_{i=1}^{B}\frac{S_{\theta}(v_{i},v^{\prime}_{i})}{\alpha}\!-\!\frac{1}{B}\sum_{i=1}^{B}\ln\mathopen{}\mathclose{{\left(\sum\limits_{j\neq i}\exp\frac{S_{\theta}(v_{i},v_{j})}{\alpha}+\sum\limits_{j}\exp\frac{S_{\theta}(v_{i},v^{\prime}_{j})}{\alpha}}}\right)\cdot \tag{13}

2\alpha

2\alpha

67.9\%

67.9\%

F.6 Ablation on the normalization in the loss function

F.6 损失函数中归一化的消融

\ell_{2}

\ell_{2}

\ell_{2}

Normalization	Top-1	Top-5
$\ell_{2}$ -norm	$\bf{72.5}$	$\bf{90.8}$
LayerNorm	$72.5{\scriptstyle\pm 0.4}$	$90.1$
No normalization	$67.4$	$87.1$
BatchNorm	$65.3$	$85.3$

\ell_{2}

归一化	Top-1	Top-5
$\ell_{2}$ -范数	$\mathbf{72.5}$	$\mathbf{90.8}$
LayerNorm	$72.5{\scriptstyle\pm 0.4}$	$90.1$
无归一化	$67.4$	$87.1$
BatchNorm	$65.3$	$85.3$

\ell_{2}

\ell_{2}

\displaystyle{n_{\text{{B\hskip-0.70004ptN} }}^{j}}_{i} \\ \displaystyle:x\rightarrow\frac{x_{i}^{j}-\mu_{\text{{B\hskip-0.70004ptN} }}^{j}(x)}{\sigma_{\text{BN}}^{j}(x)\cdot\sqrt{d}}\mathbin{\raisebox{2.15277pt}{,}}\quad{n_{\text{{L\hskip-0.70004ptN} }}^{j}}_{i}:x\rightarrow\frac{x_{i}^{j}-{\mu_{\text{{L\hskip-0.70004ptN} }}}_{i}(x)}{{\sigma_{\text{LN}}}_{i}(x)\cdot\sqrt{d}}\mathbin{\raisebox{2.15277pt}{,}}\quad n_{\text{{I\hskip-0.70004ptd} }}:x\rightarrow x, \\ \displaystyle\mu_{\text{{B\hskip-0.70004ptN} }}^{j} \\ \displaystyle:x\rightarrow\frac{1}{B}\sum_{i=1}^{B}x_{i}^{j},\quad\sigma_{\text{{B\hskip-0.70004ptN} }}^{j}:x\rightarrow\sqrt{\frac{1}{B}\sum_{i=1}^{B}\mathopen{}\mathclose{{\left({x_{i}^{j}}}}\right)^{2}-\mu_{\text{{B\hskip-0.70004ptN} }}^{j}(x)^{2}},\quad \\ \displaystyle{\mu_{\text{{L\hskip-0.70004ptN} }}}_{i} \\ \displaystyle:x\rightarrow\frac{1}{d}\sum_{j=1}^{d}x_{i}^{j},\quad{\sigma_{\text{{L\hskip-0.70004ptN} }}}_{i}:x\rightarrow\frac{\|x_{i}-{\mu_{\textsc{L\hskip-0.70004ptN}}}_{i}(x)\|_{2}}{\sqrt{d}}

\ell_{2}

\ell_{2}

Appendix G Training with smaller batch sizes

附录 G 用更小批次训练

4096

4096

Appendix H Details on Equation 5 in Section 3.2

附录 H 第 3.2 节方程 5 的细节

In this section we clarify why BYOL’s update is related to Equation 5 from Section 3.2,

本节澄清为何 BYOL 的更新与第 3.2 节方程 5 相关，

\displaystyle\nabla_{\theta}\mathbb{E}\mathopen{}\mathclose{{\left[\mathopen{}\mathclose{{\left\|q^{\star}(z_{\theta})-z^{\prime}_{\xi}}}\right\|_{2}^{2}}}\right]=\nabla_{\theta}\mathbb{E}\mathopen{}\mathclose{{\left[\mathopen{}\mathclose{{\left\|\mathbb{E}\mathopen{}\mathclose{{\left[z^{\prime}_{\xi}|z_{\theta}}}\right]-z^{\prime}_{\xi}}}\right\|_{2}^{2}}}\right]=\nabla_{\theta}\mathbb{E}\mathopen{}\mathclose{{\left[\sum_{i}\var(z^{\prime}_{\xi,i}|z_{\theta})}}\right]. \tag{5}

q^{\star}

q^{\star}

\displaystyle q^{\star}\mathrel{\hbox{\hskip 3.8889pt\hskip-3.8889pt\hbox{$=$}\hskip-3.8889pt\hskip-2.7431pt\raisebox{4.66875pt}{\hbox{$\scriptscriptstyle\Delta$}}\hskip-2.7431pt\hskip 3.8889pt}}\argmin_{q}\mathbb{E}\mathopen{}\mathclose{{\left[\mathopen{}\mathclose{{\left\|q(z_{\theta})-z^{\prime}_{\xi}}}\right\|_{2}^{2}}}\right],\quad\text{where}\quad q^{\star}(z_{\theta})=\mathbb{E}\mathopen{}\mathclose{{\left[z^{\prime}_{\xi}|z_{\theta}}}\right], \tag{4}

{\theta}

{\theta}

q

q

\displaystyle\mathbb{E}\mathopen{}\mathclose{{\left[\mathopen{}\mathclose{{\left\|q^{\star}({\theta},\xi,z_{\theta})-z^{\prime}_{\xi}}}\right\|_{2}^{2}}}\right]=\mathbb{E}\mathopen{}\mathclose{{\left[L(q^{\star}({\theta},\xi,z_{\theta}),z^{\prime}_{\xi})}}\right], \tag{14}

{\theta}

{\theta}

\displaystyle\frac{\partial}{\partial{\theta}}\mathbb{E}\mathopen{}\mathclose{{\left[L(q^{\star}({\theta},\xi,z_{\theta}),z^{\prime}_{\xi})}}\right]=\mathbb{E}\mathopen{}\mathclose{{\left[\frac{\partial L}{\partial q}\cdot\frac{\partial q^{\star}}{\partial{\theta}}+\frac{\partial L}{\partial q}\cdot\frac{\partial q^{\star}}{\partial z}\cdot\frac{\partial z_{\theta}}{\partial{\theta}}}}\right], \tag{15}

\frac{\partial q^{\star}}{\partial{\theta}}

\frac{\partial q^{\star}}{\partial{\theta}}

Appendix I Importance of a near-optimal predictor

附录 I 接近最优的预测器的重要性

In this part we build upon the intuitions of Section 3.2 on the importance of keeping the predictor near-optimal. Specifically, we show that it is possible to remove the exponential moving average in BYOL’s target network (i.e., simply copy weights of the online network into the target) without causing the representation to collapse, provided the predictor remains sufficiently good.

本部分建立在第 3.2 节关于保持预测器接近最优之重要性的直觉之上。具体地，我们表明：只要预测器保持足够好，就有可能去掉 BYOL 目标网络中的指数滑动平均（即简单地把在线网络权重复制到目标网络），而不导致表征崩塌。

I.1 Predictor learning rate

I.1 预测器学习率

\tau=0

\tau=0

\lambda

\lambda

I.2 Optimal linear predictor in closed form

I.2 闭式最优线性预测器

Similarly, we can get rid of the slowly moving target network if we use a closed form optimal predictor on the batch, instead of a learned, non-optimal one. In this case we restrict ourselves to a linear predictor,

类似地，若在批次上使用闭式最优预测器，而不是学到的、非最优预测器，我们也可以去掉慢速滑动的目标网络。此时我们把自身限制为线性预测器，

\displaystyle q^{\star}=\argmin_{Q}\mathopen{}\mathclose{{\left\|Z_{\theta}Q-Z^{\prime}_{\xi}}}\right\|_{2}^{2}=\mathopen{}\mathclose{{\left(Z_{\theta}^{\mathsf{\scriptscriptstyle T}}Z_{\theta}}}\right)^{-1}Z_{\theta}^{\mathsf{\scriptscriptstyle T}}Z^{\prime}_{\xi}

Z_{\theta}

\|\cdot\|_{2}

300

300

\lambda

$\lambda$	Top- $1$
$0$	$0.01$
$1$	$5.5$
$2$	$62.8{\scriptstyle\pm 1.5}$
$10$	$66.6$
$20$	$66.3{\scriptstyle\pm 0.3}$
Baseline	$72.5$

\lambda

\lambda_{pred}

\lambda_{pred}

Figure 8: BYOL sketch summarizing the method by emphasizing the neural architecture.

图 8：通过强调神经网络架构来概括方法的 BYOL 示意图。

See pages 1-3 of lalala.pdf

见 lalala.pdf 第 1-3 页。

		$\lambda_{pred}$
		$1$	$1.5$	$2$	$5$	$10$
$\mu_{proj}$	$1$	$3.2{\scriptstyle\pm 2.9}$	$25.7{\scriptstyle\pm 6.6}$	$60.8{\scriptstyle\pm 2.9}$	$66.7{\scriptstyle\pm 0.4}$	$66.9$
$1.5$	$1.4{\scriptstyle\pm 2.0}$	$9.2{\scriptstyle\pm 7.0}$	$55.2{\scriptstyle\pm 5.8}$	$61.5{\scriptstyle\pm 0.6}$	$66.0{\scriptstyle\pm 0.3}$
$2$	$2.0{\scriptstyle\pm 2.8}$	$5.3{\scriptstyle\pm 1.9}$	$15.8{\scriptstyle\pm 13.4}$	$60.9{\scriptstyle\pm 0.8}$	$66.3$
$5$	$1.5{\scriptstyle\pm 0.9}$	$2.5{\scriptstyle\pm 1.5}$	$2.5{\scriptstyle\pm 1.4}$	$20.5{\scriptstyle\pm 2.0}$	$60.5{\scriptstyle\pm 0.6}$
$10$	$0.1$	$2.1{\scriptstyle\pm 0.3}$	$1.9{\scriptstyle\pm 0.8}$	$2.8{\scriptstyle\pm 0.4}$	$8.3{\scriptstyle\pm 6.8}$

Bootstrap Your Own Latent A New Approach to Self-Supervised Learning

Bootstrap Your Own Latent：一种新的自监督学习方法

Abstract

摘要

1 Introduction

1 引言

2 Related work

2 相关工作

3 Method

3 方法

3.1 Description of BYOL

3.1 BYOL 的描述

3.2 Intuitions on BYOL’s behavior

3.2 关于 BYOL 行为的直觉

3.3 Implementation details

3.3 实现细节

Image augmentations

图像增强

Architecture

架构

Optimization

优化

4 Experimental evaluation

4 实验评估

Linear evaluation on ImageNet

ImageNet 上的线性评估

Semi-supervised training on ImageNet

ImageNet 上的半监督训练

Transfer to other classification tasks

迁移到其他分类任务

Transfer to other vision tasks

迁移到其他视觉任务

5 Building intuitions with ablations

5 用消融建立直觉

Batch size

批次大小

Image augmentations

图像增强

Bootstrapping

自举

Ablation to contrastive methods

相对对比方法的消融

Network hyperparameters

网络超参数

Relationship with Mean Teacher

与 Mean Teacher 的关系

Importance of a near-optimal predictor

接近最优的预测器的重要性

6 Conclusion

6 结论

Broader impact

更广泛的影响

Acknowledgements

致谢

References

参考文献

Appendix A Algorithm

附录 A 算法

Appendix B Image augmentations

附录 B 图像增强

Appendix C Evaluation on ImageNet training

附录 C 在 ImageNet 训练上的评估

C.1 Self-supervised learning evaluation on ImageNet

C.1 ImageNet 上的自监督学习评估

Linear evaluation protocol on ImageNet

ImageNet 上的线性评估协议

Variant on linear evaluation on ImageNet

ImageNet 线性评估的变体

Semi-supervised learning on ImageNet

ImageNet 上的半监督学习

C.2 Linear evaluation on larger architectures and supervised baselines

C.2 更大架构上的线性评估与监督基线

Appendix D Transfer to other datasets

附录 D 迁移到其他数据集

D.1 Datasets

D.1 数据集

D.2 Transfer via linear classification

D.2 经线性分类的迁移

D.3 Transfer via fine-tuning

D.3 经微调的迁移