An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
一张图抵得上 16×16 个词：规模化图像识别中的 Transformer（An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale）
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Affiliation: equal technical contribution, equal advising Google Research, Brain Team {adosovitskiy, neilhoulsby}@google.com Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby Affiliation: equal technical contribution, equal advising Google Research, Brain Team {adosovitskiy, neilhoulsby}@google.com
Alexey Dosovitskiy*、Lucas Beyer*、Alexander Kolesnikov*、Dirk Weissenborn*、Xiaohua Zhai*、Thomas Unterthiner、Mostafa Dehghani、Matthias Minderer、Georg Heigold、Sylvain Gelly、Jakob Uszkoreit、Neil Houlsby*
Google Research, Brain Team
{adosovitskiy, neilhoulsby}@google.com
- 同等技术贡献 / 同等指导
Abstract
摘要
While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.[^1]
Transformer 架构已成为自然语言处理任务的事实标准，但在计算机视觉中的应用仍有限。视觉里注意力要么与卷积网络联用，要么只替换卷积网络的部分组件而保留整体结构。我们表明对 CNN 的这种依赖并非必要：把标准 Transformer 直接作用在图像块序列上，就能在图像分类上表现很好。在大规模数据上预训练并迁移到多个中等或小型识别基准（ImageNet、CIFAR-100、VTAB 等）时，视觉 Transformer（Vision Transformer，ViT）相对当时最优卷积网络取得优异结果，而训练所需计算资源明显更少。
1 Introduction
1 引言
Self-attention-based architectures, in particular Transformers (Vaswani et al. 2017), have become the model of choice in natural language processing (NLP). The dominant approach is to pre-train on a large text corpus and then fine-tune on a smaller task-specific dataset (Devlin et al. 2019). Thanks to Transformers’ computational efficiency and scalability, it has become possible to train models of unprecedented size, with over 100B parameters (Brown et al. 2020; Lepikhin et al. 2020). With the models and datasets growing, there is still no sign of saturating performance.
自注意力建筑,特别是Transformer(Vaswani et al. 2017),已经成为自然语言处理(NLP)中的选择模式. 主要的方法是先对大字集进行预训,然后对较小的任务特定数据集进行微调(Devlin等人,2019年)。 由于Transformer的计算效率和可伸缩性,培训规模空前的模型成为可能,其参数超过100B(Brown等人,2020年;Lepikhin等人,2020年)。 随着模型和数据集的增长,仍然没有饱和性能的迹象.
In computer vision, however, convolutional architectures remain dominant (LeCun et al. 1989; Krizhevsky et al. 2012; He et al. 2016). Inspired by NLP successes, multiple works try combining CNN-like architectures with self-attention (Wang et al. 2018; Carion et al. 2020), some replacing the convolutions entirely (Ramachandran et al. 2019; Wang et al. 2020a). The latter models, while theoretically efficient, have not yet been scaled effectively on modern hardware accelerators due to the use of specialized attention patterns. Therefore, in large-scale image recognition, classic ResNet-like architectures are still state of the art (Mahajan et al. 2018; Xie et al. 2020; Kolesnikov et al. 2020).
然而,在计算机视野中,卷积结构仍然占主导地位(LeCun等人,1989年;Krizhevsky等人,2012年;He等人,2016年)。 在NLP成功的启发下,多部作品尝试将CNN类建筑与自觉相结合(Wang等人,2018年;Carion等人,2020年),一些作品完全取代了演化(Ramachandran等人,2019年;Wang等人,2020年a). 后一种模式虽然在理论上是有效的,但由于采用了专门的关注模式,尚未在现代硬件加速器上有效推广. 因此,在大规模图像识别中,经典的ResNet类建筑仍然处于艺术状态(Mahajan等人,2018年;Xie等人,2020年;Kolesnikov等人,2020年).
Inspired by the Transformer scaling successes in NLP, we experiment with applying a standard Transformer directly to images, with the fewest possible modifications. To do so, we split an image into patches and provide the sequence of linear embeddings of these patches as an input to a Transformer. Image patches are treated the same way as tokens (words) in an NLP application. We train the model on image classification in supervised fashion.
在NLP的Transformer缩放成功激励下,我们尝试将标准Transformer直接应用到图像上,进行尽可能少的修改. 为此,我们将一个图像分割成补丁,提供这些补丁的线性嵌入序列,作为Transformer的输入. 图像补丁在NLP应用程序中与符号(字)一样处理. 我们以监督的方式培训图像分类模型。
When trained on mid-sized datasets such as ImageNet without strong regularization, these models yield modest accuracies of a few percentage points below ResNets of comparable size. This seemingly discouraging outcome may be expected: Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data.
如果在ImageNet等中型数据集的培训没有强有力的正规化,这些模型的速率就略微低于可比规模的ResNet。 Transformer缺乏CNN固有的一些诱导性偏差, 如翻译等同性和地点性, 因此,
However, the picture changes if the models are trained on larger datasets (14M-300M images). We find that large scale training trumps inductive bias. Our Vision Transformer (ViT) attains excellent results when pre-trained at sufficient scale and transferred to tasks with fewer datapoints. When pre-trained on the public ImageNet-21k dataset or the in-house JFT-300M dataset, ViT approaches or beats state of the art on multiple image recognition benchmarks. In particular, the best model reaches the accuracy of on ImageNet, on ImageNet-ReaL, on CIFAR-100, and on the VTAB suite of 19 tasks.
然而,如果这些模型在更大的数据集(14M-300M图像)上得到训练,图片就会发生变化. 我们认为,大规模训练超越了诱导性偏见。 我们的ViTTransformer(ViT)在经过足够规模的预先训练并转移到数据点较少的任务时,会取得优异的成果. 当在公共ImageNet-21k数据集或内部JFT-300M数据集进行预训时,ViT在多个图像识别基准上接近或击败了最新技术. 特别是,最佳模型在ImageNet上达到,在ImageNet-ReaL上达到,在CIFAR-100上达到,在VTAB套件上达到的精度,共19项任务.
2 Related Work
2 相关工作
Transformers were proposed by Vaswani et al. 2017 for machine translation, and have since become the state of the art method in many NLP tasks. Large Transformer-based models are often pre-trained on large corpora and then fine-tuned for the task at hand: BERT (Devlin et al. 2019) uses a denoising self-supervised pre-training task, while the GPT line of work uses language modeling as its pre-training task (Radford et al. 2018; Radford et al. 2019; Brown et al. 2020).
Transformer由Vaswani等人2017年提出用于机器翻译,从此成为许多NLP任务中最先进的艺术方法. 基于大型变压器的模型往往在大型公司上进行预训,然后为手头的任务进行微调:BERT(Devlin等人,2019年)采用自授式的预训任务,而GPT系列的工作则使用语言模型作为预训任务(Radford等人,2018年;Radford等人,2019年;Brown等人,2020年).
Naive application of self-attention to images would require that each pixel attends to every other pixel. With quadratic cost in the number of pixels, this does not scale to realistic input sizes. Thus, to apply Transformers in the context of image processing, several approximations have been tried in the past. Parmar et al. 2018 applied the self-attention only in local neighborhoods for each query pixel instead of globally. Such local multi-head dot-product self attention blocks can completely replace convolutions (Hu et al. 2019; Ramachandran et al. 2019; Zhao et al. 2020). In a different line of work, Sparse Transformers (Child et al. 2019) employ scalable approximations to global self-attention in order to be applicable to images. An alternative way to scale attention is to apply it in blocks of varying sizes (Weissenborn et al. 2019), in the extreme case only along individual axes (Ho et al. 2019; Wang et al. 2020a). Many of these specialized attention architectures demonstrate promising results on computer vision tasks, but require complex engineering to be implemented efficiently on hardware accelerators.
对图像进行自觉的自然应用,需要每个像素关注其他像素. 以像素数的四面体成本来表示,这并不能达到现实的输入大小. 因此,为了在图像处理中应用Transformer,过去曾尝试过几种近似. Parmar 等人 2018 只对每个查询像素的本地社区应用了自我关注,而不是全球范围. 这样的本地多头点产品自我关注块可以完全取代转录(Hu等人,2019年;Ramachandran等人,2019年;Zhao等人,2020年). 在另一行工作中,SparseTransformer(Child等人,2019年)采用可伸缩近似全球自觉,以适用于图像. 扩大注意力的另一种方法是将其应用到大小不同的区块中(Weissenborn等人,2019年),极端情况下只沿着单个轴线(Ho等人,2019年;Wang等人,2020年a). 这些专业关注架构中有许多显示出计算机视觉任务上有希望的成果,但需要复杂的工程在硬件加速器上高效实施.
Most related to ours is the model of Cordonnier et al. 2020, which extracts patches of size from the input image and applies full self-attention on top. This model is very similar to ViT, but our work goes further to demonstrate that large scale pre-training makes vanilla transformers competitive with (or even better than) state-of-the-art CNNs. Moreover, Cordonnier et al. 2020 use a small patch size of pixels, which makes the model applicable only to small-resolution images, while we handle medium-resolution images as well.
与我们最相关的是Cordonnier等人2020年的模型,该模型从输入图像中提取大小为的补丁,并在顶部应用充分的自注意力. 这个模式与ViT非常相似,但我们的工作更进一步证明,大规模的预训使得香草变压器与最先进的CNN具有竞争力(甚至更好). 此外,Cordonnier等人2020使用一个小补丁尺寸的像素,这使得模型仅适用于小分辨率图像,而我们也处理中分辨率图像.
There has also been a lot of interest in combining convolutional neural networks (CNNs) with forms of self-attention, e.g. by augmenting feature maps for image classification (Bello et al. 2019) or by further processing the output of a CNN using self-attention, e.g. for object detection (Hu et al. 2018; Carion et al. 2020), video processing (Wang et al. 2018; Sun et al. 2019), image classification (Wu et al. 2020), unsupervised object discovery (Locatello et al. 2020), or unified text-vision tasks (Chen et al. 2020c; Lu et al. 2019; Li et al. 2019).
也有很多兴趣将卷积神经网络(CNNs)与自觉形式相结合,例如通过增加图像分类的特征图(Bello等人,2019年),或利用自觉性进一步处理CNN的输出,如物体探测(Hu等人,2018年;Carion等人,2020年),视频处理(Wang等人,2018年;Sun等人,2019年),图像分类(Wu等人,2020年),无人监督的物体发现(Locatello等人,2020年),或统一文本视觉任务(Chen等人,2020年c;Lu等人,2019年;Li等人,2019年).
Another recent related model is image GPT (iGPT) (Chen et al. 2020a), which applies Transformers to image pixels after reducing image resolution and color space. The model is trained in an unsupervised fashion as a generative model, and the resulting representation can then be fine-tuned or probed linearly for classification performance, achieving a maximal accuracy of 72% on ImageNet.
最近另一个相关的模型是图像GPT(iGPT)(Chen等人,2020年a),在缩小图像分辨率和色彩空间后,将Transformer应用于图像像素. 该模型被作为基因模型进行无监督的训练,由此形成的表示可以进行线性调整或探究分类性能,在ImageNet上达到最大精度72%.
Our work adds to the increasing collection of papers that explore image recognition at larger scales than the standard ImageNet dataset. The use of additional data sources allows to achieve state-of-the-art results on standard benchmarks (Mahajan et al. 2018; Touvron et al. 2019; Xie et al. 2020). Moreover, Sun et al. 2017 study how CNN performance scales with dataset size, and Kolesnikov et al. 2020; Djolonga et al. 2020 perform an empirical exploration of CNN transfer learning from large scale datasets such as ImageNet-21k and JFT-300M. We focus on these two latter datasets as well, but train Transformers instead of ResNet-based models used in prior works.
我们的工作增加了越来越多的论文,这些论文比标准图像网数据集更大规模地探索图像识别. 利用额外数据来源可以实现标准基准的最新结果(Mahajan等人,2018年;Touvron等人,2019年;Xie等人,2020年)。 此外,Sun等人2017研究了CNN的性能如何以数据集大小衡量,Kolesnikov等人2020年;Djolonga等人2020年对CNN从ImageNet-21k和JFT-300M等大尺度数据集的转移学习进行了实证探索. 我们关注的是后两个数据集,但培训Transformer而不是以往作品中使用的基于ResNet的模型。
3 Method
3 方法
Figure 1: Model overview. We split an image into fixed-size patches, linearly embed each of them, add position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder. In order to perform classification, we use the standard approach of adding an extra learnable “classification token” to the sequence. The illustration of the Transformer encoder was inspired by Vaswani et al. 2017.
图1:模式概述。 我们将一个图像分割成固定大小的补丁,线性嵌入每个补丁,添加位置嵌入,并将由此产生的矢量序列反馈给标准的Transformer编码器. 为了进行分类,我们采用标准方法,在序列中增加一个可学习的“分类符号”。 "Transformer编码器"的插图灵感来自瓦斯瓦尼等2017年.
In model design we follow the original Transformer (Vaswani et al. 2017) as closely as possible. An advantage of this intentionally simple setup is that scalable NLP Transformer architectures – and their efficient implementations – can be used almost out of the box.
在模型设计中,我们尽可能紧跟原Transformer(Vaswani等人,2017年). 这种有意的简单设置的一个优点是,可以扩展的NLPTransformer架构 — — 及其高效的实现 — — 几乎可以在盒子外使用.
3.1 Vision Transformer (ViT)
3.1 视觉 Transformer（ViT）
An overview of the model is depicted in Figure 1. The standard Transformer receives as input a 1D sequence of token embeddings. To handle 2D images, we reshape the image into a sequence of flattened 2D patches , where is the resolution of the original image, is the number of channels, is the resolution of each image patch, and is the resulting number of patches, which also serves as the effective input sequence length for the Transformer. The Transformer uses constant latent vector size through all of its layers, so we flatten the patches and map to dimensions with a trainable linear projection (Eq. 1). We refer to the output of this projection as the patch embeddings.
图1概述了该模型。 标准Transformer作为输入接收一个1D序列的符号嵌入. 为了处理2D图像,我们将图像重塑成平整的2D补丁的序列,其中是原始图像的解析度,是通道数,是每个图像补丁的解析度,是由此产生的补丁数,这也成为Transformer的有效输入序列长度. Transformer通过它的所有层使用恒定潜在矢量大小,所以我们用可导线性投影(Eq. 1)将补丁和映射到维度. 我们将此投影的输出称为补丁嵌入.
Similar to BERT’s [class] token, we prepend a learnable embedding to the sequence of embedded patches (), whose state at the output of the Transformer encoder () serves as the image representation (Eq. 4). Both during pre-training and fine-tuning, a classification head is attached to . The classification head is implemented by a MLP with one hidden layer at pre-training time and by a single linear layer at fine-tuning time.
与BERT的[类]符号类似,我们预先将一个可学习的嵌入到嵌入的补丁()的序列上,在Transformer编码器()输出时的状态充当图像代表 (Eq. 4). 在训练前和微调期间,都附有一个分类头. 分类头由一个MLP执行,在预训时间有一个隐藏的层,在微调时间有一个单线性层.
Position embeddings are added to the patch embeddings to retain positional information. We use standard learnable 1D position embeddings, since we have not observed significant performance gains from using more advanced 2D-aware position embeddings (Appendix D.4). The resulting sequence of embedding vectors serves as input to the encoder.
位置嵌入被添加到补丁嵌入中以保留位置信息. 我们使用标准的可学习的1D位置嵌入,因为我们没有观察到使用更高级的2D-意识位置嵌入器(附录D.4)而取得显著的性能收益. 由此产生的嵌入向量的序列充当编码器的输入.
The Transformer encoder (Vaswani et al. 2017) consists of alternating layers of multiheaded self-attention (MSA, see Appendix A) and MLP blocks (Eq. 2, 3). Layernorm (LN) is applied before every block, and residual connections after every block (Wang et al. 2019; Baevski & Auli 2019). The MLP contains two layers with a GELU non-linearity.
Transformer编码器(Vaswani等人,2017年)由多头自留(MSA,见附录A)和MLP区块(Eq.2,3)的交替层组成. Layernorm(LN)在每个区块之前应用,在每个区块之后应用残余连接(Wang等人,2019年;Baevski & Auli 2019年). MLP包含两层,具有GELU非线性.
Inductive bias.
We note that Vision Transformer has much less image-specific inductive bias than CNNs. In CNNs, locality, two-dimensional neighborhood structure, and translation equivariance are baked into each layer throughout the whole model. In ViT, only MLP layers are local and translationally equivariant, while the self-attention layers are global. The two-dimensional neighborhood structure is used very sparingly: in the beginning of the model by cutting the image into patches and at fine-tuning time for adjusting the position embeddings for images of different resolution (as described below). Other than that, the position embeddings at initialization time carry no information about the 2D positions of the patches and all spatial relations between the patches have to be learned from scratch.
- 感官偏执
与CNN相比, 在CNN,地点,二维邻里结构, 和翻译等效 被烤制成整个模型的每层。 在ViT中,只有MLP层是局部和翻译等效的,而自注层则是全球性的. 二维邻接结构被非常节制地使用:在模型开始时,通过将图像切成补丁,并在微调时调整不同分辨率图像嵌入位置(如下所述). 除此之外,在初始化时嵌入的位置不包含补丁的2D位置信息,补丁之间的所有空间关系都必须从零开始学习.
Hybrid Architecture.
As an alternative to raw image patches, the input sequence can be formed from feature maps of a CNN (LeCun et al. 1989). In this hybrid model, the patch embedding projection (Eq. 1) is applied to patches extracted from a CNN feature map. As a special case, the patches can have spatial size 1x1, which means that the input sequence is obtained by simply flattening the spatial dimensions of the feature map and projecting to the Transformer dimension. The classification input embedding and position embeddings are added as described above.
混合建筑.
作为原始图像补丁的替代,输入序列可以从CNN的特征地图中形成(LeCun等人,1989年). 在这个混合模型中,补丁嵌入投影(Eq. 1)被应用于从CNN特征图中提取的补丁. 作为特例,补丁可以有1x1的空间尺寸,这意味着输入序列是通过简单地平整特征图的空间维度和投影到Transformer维度来获得的. 如上文所述,增加了分类输入嵌入和位置嵌入。
3.2 Fine-tuning and Higher Resolution
3.2 微调与更高分辨率
Typically, we pre-train ViT on large datasets, and fine-tune to (smaller) downstream tasks. For this, we remove the pre-trained prediction head and attach a zero-initialized feedforward layer, where is the number of downstream classes. It is often beneficial to fine-tune at higher resolution than pre-training (Touvron et al. 2019; Kolesnikov et al. 2020). When feeding images of higher resolution, we keep the patch size the same, which results in a larger effective sequence length. The Vision Transformer can handle arbitrary sequence lengths (up to memory constraints), however, the pre-trained position embeddings may no longer be meaningful. We therefore perform 2D interpolation of the pre-trained position embeddings, according to their location in the original image. Note that this resolution adjustment and patch extraction are the only points at which an inductive bias about the 2D structure of the images is manually injected into the Vision Transformer.
通常情况下,我们先在大型数据集上进行虚拟技术培训,然后对下游任务(较小)进行微调。 为此,我们移除了预先训练的预测头,并附加一个零初始化的 feedforward 层,其中是下游类的数量. 微调的分辨率往往高于预培训(Touvron等人,2019年;Kolesnikov等人,2020年)。 在输入更高分辨率的图像时,我们保持补丁大小不变,从而产生更大的有效序列长度. "视觉Transformer"可以处理任意的序列长度(最高到内存限制),然而,预训位置嵌入可能不再有意义. 因此,我们根据在原始图像中的位置,对训练前位置嵌入进行2D插值。 注意,此分辨率调整和补丁提取是将图像2D结构的诱导偏差手动注入视觉Transformer的唯一点.
4 Experiments
4 实验
We evaluate the representation learning capabilities of ResNet, Vision Transformer (ViT), and the hybrid. To understand the data requirements of each model, we pre-train on datasets of varying size and evaluate many benchmark tasks. When considering the computational cost of pre-training the model, ViT performs very favourably, attaining state of the art on most recognition benchmarks at a lower pre-training cost. Lastly, we perform a small experiment using self-supervision, and show that self-supervised ViT holds promise for the future.
我们评价ResNet、Visual Transformer(ViT)和混合体的代表性学习能力。 为了了解每个模型的数据要求,我们对大小不同的数据集进行预训,并评价许多基准任务. 在考虑培训前模式的计算成本时,VIT表现非常出色,以较低的培训前成本达到大多数识别基准的先进水平。 最后,我们利用自监督进行一个小型实验,并表明自监督的Vit对未来有希望。
4.1 Setup
4.1 设置
Datasets. To explore model scalability, we use the ILSVRC-2012 ImageNet dataset with 1k classes and 1.3M images (we refer to it as ImageNet in what follows), its superset ImageNet-21k with 21k classes and 14M images (Deng et al. 2009), and JFT (Sun et al. 2017) with 18k classes and 303M high-resolution images. We de-duplicate the pre-training datasets w.r.t. the test sets of the downstream tasks following Kolesnikov et al. 2020. We transfer the models trained on these dataset to several benchmark tasks: ImageNet on the original validation labels and the cleaned-up ReaL labels (Beyer et al. 2020), CIFAR-10/100 (Krizhevsky 2009), Oxford-IIIT Pets (Parkhi et al. 2012), and Oxford Flowers-102 (Nilsback & Zisserman 2008). For these datasets, pre-processing follows Kolesnikov et al. 2020.
页:1 为了探索模型的可扩展性,我们使用ILSVRC-2012影像网数据集,包含1k类和1.3M影像(我们在下文中将其称为ImageNet),其超设的ImageNet-21k包含21k类和14M影像(Deng等人,2009年),JFT(Sun等人,2017年)包含18k类和303M高分辨率影像. 我们不再复制科列斯尼科夫等人2020年之后下游任务的预训数据集 w.r.t. 我们将这些数据集培训的模型转移到几个基准任务:原始验证标签上的图像网和清理的ReaL标签(Beyer等人,2020年)、CIFAR-10/100(Krizhevsky,2009年)、牛津-IIIT Pets(Parkhi等人,2012年)和牛津花卉102(Nilsback & Zisserman,2008年)。 对于这些数据集,预处理遵循科莱斯尼科夫等人2020年.
We also evaluate on the 19-task VTAB classification suite (Zhai et al. 2019b). VTAB evaluates low-data transfer to diverse tasks, using 1 000 training examples per task. The tasks are divided into three groups: Natural – tasks like the above, Pets, CIFAR, etc. Specialized – medical and satellite imagery, and Structured – tasks that require geometric understanding like localization.
我们还评价了19任务VTAB分类套件(Zhai等人,2019年b). VTAB评价低数据转移到多种任务,每个任务使用1000个培训实例. 任务分为三组: 自然 – 任务如上述,宠物,CIFAR等. 专业 – 医疗和卫星图像, *Structured * – 任务如本地化需要几何理解.
Model Variants. We base ViT configurations on those used for BERT (Devlin et al. 2019), as summarized in Table 1. The “Base” and “Large” models are directly adopted from BERT and we add the larger “Huge” model. In what follows we use brief notation to indicate the model size and the input patch size: for instance, ViT-L/16 means the “Large” variant with input patch size. Note that the Transformer’s sequence length is inversely proportional to the square of the patch size, thus models with smaller patch size are computationally more expensive.
** 示范备选案文。 ** 如表1所示,我们以BERT(Devlin等人,2019年)使用的VIT配置为基础。 “Base”和“Large”模型直接取自BERT,我们添加更大的“Huge”模型。 在下文中,我们使用简短的标记来表示模型大小和输入补丁大小:例如,Vit-L/16是指具有输入补丁大小的“大”变体。 注意Transformer的序列长度与补丁大小的平方成反比,因此,补丁尺寸较小的模型在计算上更为昂贵.
For the baseline CNNs, we use ResNet (He et al. 2016), but replace the Batch Normalization layers (Ioffe & Szegedy 2015) with Group Normalization (Wu & He 2018), and used standardized convolutions (Qiao et al. 2019). These modifications improve transfer (Kolesnikov et al. 2020), and we denote the modified model “ResNet (BiT)”. For the hybrids, we feed the intermediate feature maps into ViT with patch size of one “pixel”. To experiment with different sequence lengths, we either (i) take the output of stage 4 of a regular ResNet50 or (ii) remove stage 4, place the same number of layers in stage 3 (keeping the total number of layers), and take the output of this extended stage 3. Option (ii) results in a 4x longer sequence length, and a more expensive ViT model.
对于基线编写关于CNN的节目,我们使用ResNet(He等人,2016年),但将Batch Normalization层(Ioffe & Szegedy,2015年)替换为Group Normalization(Wu & He,2018年),并使用了标准化的演化(Qiao等人,2019年). 这些修改改进了转移(Kolesnikov等人,2020年),我们表示经过修改的模型“ResNet(BIT)”。 对于混合体,我们用一个“像素”的补丁大小将中间特征地图输入VIT。 为了进行不同序列长度的实验,我们要么(一)接受常规ResNet50第4阶段的输出,要么(二)去除第4阶段的输出,将同样数量的层放在第3阶段(保持第3阶段的总数),并承担这一扩展阶段第3阶段的输出. 选项(二)导致4x更长的序列长度,以及更昂贵的VIT模型.
| Model | Layers | Hidden size | MLP size | Heads | Params |
|---|---|---|---|---|---|
| ViT-Base | 12 | 768 | 3072 | 12 | 86M |
| ViT-Large | 24 | 1024 | 4096 | 16 | 307M |
| ViT-Huge | 32 | 1280 | 5120 | 16 | 632M |
Table 1: Details of Vision Transformer model variants.
| 型号 | 层 | 隐藏大小 | MLP 大小 | 头 | Params |
|---|---|---|---|---|---|
| 维T-贝斯 | 12 | 768 | 3072 | 12 | 86M (中文(简体) ). |
| VIT 宽度 | 24 | 1024 | 4096 | 16 | 307海里 |
| ViT - 厚度 | 32 | 1280 | 5120 | 16 | 632M (英语). |
表1:远景变换器模型变体的细节.
Training & Fine-tuning. We train all models, including ResNets, using Adam (Kingma & Ba 2015) with , , a batch size of 4096 and apply a high weight decay of , which we found to be useful for transfer of all models (Appendix D.1 shows that, in contrast to common practices, Adam works slightly better than SGD for ResNets in our setting). We use a linear learning rate warmup and decay, see Appendix B.1 for details. For fine-tuning we use SGD with momentum, batch size 512, for all models, see Appendix B.1.1. For ImageNet results in Table 2, we fine-tuned at higher resolution: for ViT-L/16 and for ViT-H/14, and also used Polyak & Juditsky 1992 averaging with a factor of (Ramachandran et al. 2019; Wang et al. 2020b).
** 培训和精细调整。 ** 我们使用 Adam (Kingma & Ba 2015) 来训练所有模型, 包括 ResNet, 使用 , , 批量大小为 4096, 并应用的高重量衰变, 我们发现它对于所有模型的转移是有用的( 附录 D.1 显示, 相对于常见的做法, Adam 对我们的 ResNet 来说, 工作略优于 SGD ) 。 我们使用线性学习热速率和衰变,详情见附录B.1. 关于微调,我们使用带有动力的SGD,批量尺寸512,所有型号,见附录B.1.1。 对于ImageNet的表2结果,我们以更高的分辨率进行了微调:VIT-L/16的和VIT-H/14的,还使用了Polyak & Juditsky 1992年平均系数(Ramakhandran等人,2019年;Wang等人,2020年b)。
Metrics. We report results on downstream datasets either through few-shot or fine-tuning accuracy. Fine-tuning accuracies capture the performance of each model after fine-tuning it on the respective dataset. Few-shot accuracies are obtained by solving a regularized least-squares regression problem that maps the (frozen) representation of a subset of training images to target vectors. This formulation allows us to recover the exact solution in closed form. Though we mainly focus on fine-tuning performance, we sometimes use linear few-shot accuracies for fast on-the-fly evaluation where fine-tuning would be too costly.
** 计量。 ** 我们报告下游数据集的结果,要么是少拍,要么是微调精度。 精细调整精度捕捉每个模型在相应的数据集上精度调整后的表现. 通过解决一个常规化的最小平方回归问题,将一组训练图像的(被冻结的)代表图映射到目标矢量上,得到的射速是很少的。 这种提法使我们能够以封闭的形式恢复确切的解决方案。 虽然我们主要侧重于微调性能,但有时我们使用线性几发速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速射速
4.2 Comparison to State of the Art
4.2 与当时最优方法的对比
We first compare our largest models – ViT-H/14 and ViT-L/16 – to state-of-the-art CNNs from the literature. The first comparison point is Big Transfer (BiT) (Kolesnikov et al. 2020), which performs supervised transfer learning with large ResNets. The second is Noisy Student (Xie et al. 2020), which is a large EfficientNet trained using semi-supervised learning on ImageNet and JFT-300M with the labels removed. Currently, Noisy Student is the state of the art on ImageNet and BiT-L on the other datasets reported here. All models were trained on TPUv3 hardware, and we report the number of TPUv3-core-days taken to pre-train each of them, that is, the number of TPU v3 cores (2 per chip) used for training multiplied by the training time in days.
我们首先将我们最大的模型——Vit-H/14和Vit-L/16——与文献中最先进的CNN进行比较. 第一个比较点是大转移(Bit)(Kolesnikov等人,2020年),它用大型ResNet进行监督转移学习。 第二个是Noisy Student(Xie等人,2020年),这是一个大型的高效网络,通过在ImageNet和JFT-300M上进行半监督学习,并删除了标签。 目前,Noisy Student是ImageNet和BiT-L上关于本文报道的其他数据集的艺术状态. 所有型号都接受了TPUv3硬件的培训,我们报告每个型号进行预训所需的TPUv3核心日数,即用于训练的TPU v3核数(每芯片2个)乘以数日训练时间.
| Ours-JFT | Ours-JFT | Ours-I21k | BiT-L | Noisy Student | |
|---|---|---|---|---|---|
| (ViT-H/14) | (ViT-L/16) | (ViT-L/16) | (ResNet152x4) | (EfficientNet-L2) | |
| ImageNet | |||||
| ImageNet ReaL | |||||
| CIFAR-10 | |||||
| CIFAR-100 | |||||
| Oxford-IIIT Pets | |||||
| Oxford Flowers-102 | |||||
| VTAB (19 tasks) | |||||
| TPUv3-core-days | k | k | k | k | k |
Table 2: Comparison with state of the art on popular image classification benchmarks. We report mean and standard deviation of the accuracies, averaged over three fine-tuning runs. Vision Transformer models pre-trained on the JFT-300M dataset outperform ResNet-based baselines on all datasets, while taking substantially less computational resources to pre-train. ViT pre-trained on the smaller public ImageNet-21k dataset performs well too. ∗Slightly improved result reported in Touvron et al. 2020.
| 我们的JFT | 我们的JFT | 我们的I21k | 双T - L级 | 噪音学生 | |
|---|---|---|---|---|---|
| (维塔-H/14). | (维特-L/16) | (维特-L/16) | (ResNet152x4) (中文(简体) ). | (效能网-L2) | |
| 图像网 | 第二十届 | ||||
| 图像网络ReaL | 二十世纪 | ||||
| 科伊法尔-10 | 第二十届 | ||||
| 你想说什么? | 第二十届 | ||||
| 牛津-IIIT 宠物 | 第二十届 | ||||
| 牛津花报-102 | 第二十届 | ||||
| VTAB(19项任务) | 第二十届 | ||||
| TPUv3 核心日数 | k | k | k | k | k |
表2:与流行图像分类基准的最新技术比较。 平均三次微调 远景变换器模型预先在JFT-300M数据集上训练,在所有数据集上超越基于ResNet的基线,同时将大量较少的计算资源用于预训. ViT在较小的公用ImageNet-21k数据集上预训也表现良好. * 图夫龙等人2020年报告的成果略有改进。
Figure 2: Breakdown of VTAB performance in Natural, Specialized, and Structured task groups.
图2:VTAB在自然、专业和结构任务组中的绩效细目。
Table 2 shows the results. The smaller ViT-L/16 model pre-trained on JFT-300M outperforms BiT-L (which is pre-trained on the same dataset) on all tasks, while requiring substantially less computational resources to train. The larger model, ViT-H/14, further improves the performance, especially on the more challenging datasets – ImageNet, CIFAR-100, and the VTAB suite. Interestingly, this model still took substantially less compute to pre-train than prior state of the art. However, we note that pre-training efficiency may be affected not only by the architecture choice, but also other parameters, such as training schedule, optimizer, weight decay, etc. We provide a controlled study of performance vs. compute for different architectures in Section 4.4. Finally, the ViT-L/16 model pre-trained on the public ImageNet-21k dataset performs well on most datasets too, while taking fewer resources to pre-train: it could be trained using a standard cloud TPUv3 with 8 cores in approximately 30 days.
表2显示结果。 在JFT-300M上的更小的VIT-L/16模型在所有任务上预训BiT-L(在同一个数据集上预训),同时训练所需的计算资源要少得多. 更大的型号VIT-H/14进一步提高了性能,特别是在更具挑战性的数据集—ImageNet,CIFAR-100以及VTAB套件上. 有趣的是,这种型号在预训时的计算仍然大大低于以前的先进水平. 然而,我们注意到,培训前的效率不仅可能受到架构选择的影响,而且还可能受到其他参数的影响,如培训时间表,优化器,体重衰减等. 我们对4.4节不同建筑的性能和计算进行了有控制的研究。 最后,在公共ImageNet-21k数据集上预先训练的VIT-L/16模型在大多数数据集上也表现良好,同时使用较少的资源进行预训练:可以在大约30天内使用标准云TPUv3与8个核心进行训练.
Figure 2 decomposes the VTAB tasks into their respective groups, and compares to previous SOTA methods on this benchmark: BiT, VIVI – a ResNet co-trained on ImageNet and Youtube (Tschannen et al. 2020), and S4L – supervised plus semi-supervised learning on ImageNet (Zhai et al. 2019a). ViT-H/14 outperforms BiT-R152x4, and other methods, on the Natural and Structured tasks. On the Specialized the performance of the top two models is similar.
图2将VTAB任务分解为各自的组别,并与本基准上以往的SOTA方法比较:BiT,VIVI – a ResNet 共同训练于ImageNet和Youtube(Tschannn等人,2020年),S4L – 监督+半监督学习于ImageNet(Zhai等人,2019年a). VIT-H/14在自然和结构任务上超越了BIT-R152x4和其他方法. 在专业上,前两个模型的性能相似.
4.3 Pre-training Data Requirements
4.3 预训练数据需求
Figure 3: Transfer to ImageNet. While large ViT models perform worse than BiT ResNets (shaded area) when pre-trained on small datasets, they shine when pre-trained on larger datasets. Similarly, larger ViT variants overtake smaller ones as the dataset grows.
图3:转移到ImageNet。 虽然大型VIT模型在小数据集上预训时表现比BiT ResNets(阴影区)更差,但在大数据集上预训时却闪耀. 同样,随着数据集的增长,较大的VIT变体也超过了较小的变体。
The Vision Transformer performs well when pre-trained on a large JFT-300M dataset. With fewer inductive biases for vision than ResNets, how crucial is the dataset size? We perform two series of experiments.
在JFT-300M大型数据集上预训时,VisionTransformer表现良好. 在视觉的诱导偏差比ResNets少的情况下,数据集的大小有多重要? 我们做了两个实验
First, we pre-train ViT models on datasets of increasing size: ImageNet, ImageNet-21k, and JFT-300M. To boost the performance on the smaller datasets, we optimize three basic regularization parameters – weight decay, dropout, and label smoothing. Figure 4 shows the results after fine-tuning to ImageNet (results on other datasets are shown in Table 5)[^2]. When pre-trained on the smallest dataset, ImageNet, ViT-Large models underperform compared to ViT-Base models, despite (moderate) regularization. With ImageNet-21k pre-training, their performances are similar. Only with JFT-300M, do we see the full benefit of larger models. Figure 4 also shows the performance region spanned by BiT models of different sizes. The BiT CNNs outperform ViT on ImageNet, but with the larger datasets, ViT overtakes.
首先,我们先在大小越来越大的数据集上预训VIT模型:ImageNet,ImageNet-21k,以及JFT-300M. 为了提升小数据集的性能,我们优化了三个基本的规范化参数 — — 重量衰减、辍学和标签平滑。 图4显示微调到ImageNet后的结果(其他数据集的结果见表5)[^2]。 当在最小的数据集ImageNet,Vit-Large模型上预训时,与Vit-Base模型相比表现不佳,尽管(中度)规范化. 随着ImageNet-21k的预训,他们的表演相似. 只有JFT-300M,我们才能看到更大的模型的全部好处. 图4还显示了不同大小的BiT模型所跨越的性能区域. BiT CNNs在ImageNet上的表现超过了Vit,但随着更大的数据集,ViT超过了.
Second, we train our models on random subsets of 9M, 30M, and 90M as well as the full JFT-300M dataset. We do not perform additional regularization on the smaller subsets and use the same hyper-parameters for all settings. This way, we assess the intrinsic model properties, and not the effect of regularization. We do, however, use early-stopping, and report the best validation accuracy achieved during training. To save compute, we report few-shot linear accuracy instead of full fine-tuning accuracy. Figure 4 contains the results. Vision Transformers overfit more than ResNets with comparable computational cost on smaller datasets. For example, ViT-B/32 is slightly faster than ResNet50; it performs much worse on the 9M subset, but better on 90M+ subsets. The same is true for ResNet152x2 and ViT-L/16. This result reinforces the intuition that the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns directly from data is sufficient, even beneficial.
其次,我们用9M,30M,90M的随机子集以及完整的JFT-300M数据集来训练我们的模型. 我们并不对较小的子集进行额外的正规化,并且对所有设置使用相同的超参数。 这样,我们评估的是内在的模型属性,而不是正规化的效果. 然而,我们确实采用早期制止方法,并报告在培训期间所取得的最佳验证准确性。 为了节省计算,我们报告几发线性精确度,而不是完全微调精度. 图4载有结果。 视野变换器比ResNet更适合在较小的数据集上具有可比较的计算成本. 例如,Vit-B/32略快于ResNet50;它在9M子集上表现得更差,但在90M+子集上表现得更好. ResNet152x2和Vit-L/16也一样. 这一结果加强了直觉,即酥毛导体偏差对较小的数据集有用,但对于较大的数据集,直接从数据中学习相关的规律就足够了,甚至有益了.
Overall, the few-shot results on ImageNet (Figure 4), as well as the low-data results on VTAB (Table 2) seem promising for very low-data transfer. Further analysis of few-shot properties of ViT is an exciting direction of future work.
总体而言,在ImageNet(图4)上的少数镜头结果,以及VTAB(表2)上的低数据结果,对于极低数据传输来说似乎很有希望. 对ViT的几弹特性的进一步分析是未来工作的令人振奋的方向.
Figure 5: Performance versus pre-training compute for different architectures: Vision Transformers, ResNets, and hybrids. Vision Transformers generally outperform ResNets with the same computational budget. Hybrids improve upon pure Transformers for smaller model sizes, but the gap vanishes for larger models.
图5:不同架构的性能与培训前计算:视觉Transformer,ResNet,和混合型. 远景变换器一般在相同的计算预算下,优于ResNet. 混合型改进了小型模型的纯Transformer,但大模型的缺口消失。
4.4 Scaling Study
4.4 规模扩展研究
We perform a controlled scaling study of different models by evaluating transfer performance from JFT-300M. In this setting data size does not bottleneck the models’ performances, and we assess performance versus pre-training cost of each model. The model set includes: 7 ResNets, R50x1, R50x2 R101x1, R152x1, R152x2, pre-trained for 7 epochs, plus R152x2 and R200x3 pre-trained for 14 epochs; 6 Vision Transformers, ViT-B/32, B/16, L/32, L/16, pre-trained for 7 epochs, plus L/16 and H/14 pre-trained for 14 epochs; and 5 hybrids, R50+ViT-B/32, B/16, L/32, L/16 pre-trained for 7 epochs, plus R50+ViT-L/16 pre-trained for 14 epochs (for hybrids, the number at the end of the model name stands not for the patch size, but for the total dowsampling ratio in the ResNet backbone).
我们通过评价JFT-300M的转移性能,对不同的模型进行有控制的缩放研究. 在这个设定中,数据大小并不阻碍模型的性能,我们评估每个模型的性能与培训前成本。 该型号集包括:7 ResNet,R50x1,R50x2 R101x1,R152x1,R152x2,预训7个纪元,外加R152x2和R200x3,预训14个纪元;6 远景变换器,VIT-B/32,B/16,L/32,L/16,预训7个纪元,外加L/16,预训14个纪元;5 混合型,R50+ViT-B/32,B/16,L/32,L/16,预训7个纪元,外加R50+ViT-L/16,预训14个纪元(混合型,模式名称末尾的编号不代表补丁大小,而代表ResNet主干线的总剂量比).
Figure 5 contains the transfer performance versus total pre-training compute (see Appendix D.5 for details on computational costs). Detailed results per model are provided in Table 6 in the Appendix. A few patterns can be observed. First, Vision Transformers dominate ResNets on the performance/compute trade-off. ViT uses approximately less compute to attain the same performance (average over 5 datasets). Second, hybrids slightly outperform ViT at small computational budgets, but the difference vanishes for larger models. This result is somewhat surprising, since one might expect convolutional local feature processing to assist ViT at any size. Third, Vision Transformers appear not to saturate within the range tried, motivating future scaling efforts.
图5载有在其各自培训前的计算中转帐业绩(关于计算费用的详细情况,见附录D.5)。 附录表6提供了每个模型的详细结果。 可以观察到一些模式。 首先,视觉变换器在性能/计算权衡上主导ResNet. VIT使用大约较少的计算来达到相同的性能(平均超过5个数据集). 第二,混合体在小计算预算上略高于VIT,但对于较大的模型来说,差异消失了. 这一结果有些令人惊讶,因为人们可能期望在任何规模上都会对本地特性进行不断演变的处理,以协助VIT。 第三, " 愿景变换器 " 似乎不会在所尝试的范围内饱和,从而推动今后的推广努力。
4.5 Inspecting Vision Transformer
4.5 检视视觉 Transformer
Figure 6: Representative examples of attention from the output token to the input space. See Appendix D.7 for details.
图的6:从产出符号到投入空间的注意的代表性例子。 ,详见附录D.7。
To begin to understand how the Vision Transformer processes image data, we analyze its internal representations. The first layer of the Vision Transformer linearly projects the flattened patches into a lower-dimensional space (Eq. 1). Figure 7 (left) shows the top principal components of the the learned embedding filters. The components resemble plausible basis functions for a low-dimensional representation of the fine structure within each patch.
为了开始理解视觉Transformer如何处理图像数据,我们分析其内部表现. 视觉Transformer的第一层线性地将扁平的补丁投射到下维空间(Eq. 图7(左)显示所学嵌入滤镜的顶部主要部件。 这些组件类似于每个补丁内微细结构低维表示的可信基础功能。
After the projection, a learned position embedding is added to the patch representations. Figure 7 (center) shows that the model learns to encode distance within the image in the similarity of position embeddings, i.e. closer patches tend to have more similar position embeddings. Further, the row-column structure appears; patches in the same row/column have similar embeddings. Finally, a sinusoidal structure is sometimes apparent for larger grids (Appendix D). That the position embeddings learn to represent 2D image topology explains why hand-crafted 2D-aware embedding variants do not yield improvements (Appendix D.4).
在投影后,在补丁表示中加入一个已学习到的位置嵌入. 图7(中心)显示模型在位置嵌入的相似性中学习了在图像内编码距离,即更接近的补丁往往具有更相似的位置嵌入. 此外,行柱结构出现;同一行/列的补丁也有类似的嵌入. 最后,对于更大的网格(Appendix D)来说,一个鼻音结构有时是明显的. 嵌入位置学会代表2D图像地形解释为什么手工制作的2D-意识嵌入变体不会产生改进(附录D.4).
Figure 7: Left: Filters of the initial linear embedding of RGB values of ViT-L/32. Center: Similarity of position embeddings of ViT-L/32. Tiles show the cosine similarity between the position embedding of the patch with the indicated row and column and the position embeddings of all other patches. Right: Size of attended area by head and network depth. Each dot shows the mean attention distance across images for one of 16 heads at one layer. See Appendix D.7 for details.
图7:左: ViT-L/32. RGB值初始线性嵌入的过滤器。** Center**: ViT-L/32. 位置嵌入的相似性。 平面显示与指定行和列嵌入补丁的位置与所有其他补丁嵌入的位置之间的余弦相似性. ** 权利**: 按头部和网络深度划分的参加区域大小。 每个点都显示一个层上16个头中的一个的图像之间的平均关注距离. 详情见附录D.7。
Self-attention allows ViT to integrate information across the entire image even in the lowest layers. We investigate to what degree the network makes use of this capability. Specifically, we compute the average distance in image space across which information is integrated, based on the attention weights (Figure 7, right). This “attention distance” is analogous to receptive field size in CNNs. We find that some heads attend to most of the image already in the lowest layers, showing that the ability to integrate information globally is indeed used by the model. Other attention heads have consistently small attention distances in the low layers. This highly localized attention is less pronounced in hybrid models that apply a ResNet before the Transformer (Figure 7, right), suggesting that it may serve a similar function as early convolutional layers in CNNs. Further, the attention distance increases with network depth. Globally, we find that the model attends to image regions that are semantically relevant for classification (Figure 6).
自觉可以让VIT整合整个图像中的信息,即使是最低层. 我们调查该网络在多大程度上利用了这种能力。 具体地说,我们根据注意的权重(图7,右方)来计算信息集成在图像空间中的平均距离. 这种“注意距离”类似于有线电视新闻网所接受的实地规模。 我们发现,一些头目关注着已经处于最低层的大部分图像,这表明该模型确实利用了在全球整合信息的能力。 其他注意头在低层的注意距离一直很小. 这种高度本地化的注意在混合模型中不太明显,这些模型在Transformer之前应用了ResNet(图7,右),说明它可能起到类似于CNNs早期的演化层的作用. 此外,关注距离随着网络深度的增加而增加. 在全球范围内,我们发现该模型涉及与分类密切相关的图像区域(图6)。
4.6 Self-supervision
4.6 自监督
Transformers show impressive performance on NLP tasks. However, much of their success stems not only from their excellent scalability but also from large scale self-supervised pre-training (Devlin et al. 2019; Radford et al. 2018). We also perform a preliminary exploration on masked patch prediction for self-supervision, mimicking the masked language modeling task used in BERT. With self-supervised pre-training, our smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, a significant improvement of 2% to training from scratch, but still 4% behind supervised pre-training. Appendix B.1.2 contains further details. We leave exploration of contrastive pre-training (Chen et al. 2020b; He et al. 2020; Bachman et al. 2019; Hénaff et al. 2020) to future work.
Transformer在NLP任务上表现出令人印象深刻的性能. 然而,他们的大部分成功不仅来源于其优秀的可伸缩性,也来源于大规模自监督的预训(Devlin等人,2019年;拉德福德等人,2018年). 我们还对自监督的“假补丁预测”进行了初步探索,模仿了BERT中使用的蒙面语言模型工作。 通过自监督的预训,我们较小的ViT-B/16模式在ImageNet上实现了79.9%的精度,从头到尾大幅提升到2%的训练,但仍有4%落后于监督的预训. 附录B.1.2载有进一步的细节。 我们把对比性预训(Chen等人,2020年b;He等人,2020年;Bachman等人,2019年;Hénaff等人,2020年)的探索留给今后的工作.
5 Conclusion
5 结论
We have explored the direct application of Transformers to image recognition. Unlike prior works using self-attention in computer vision, we do not introduce image-specific inductive biases into the architecture apart from the initial patch extraction step. Instead, we interpret an image as a sequence of patches and process it by a standard Transformer encoder as used in NLP. This simple, yet scalable, strategy works surprisingly well when coupled with pre-training on large datasets. Thus, Vision Transformer matches or exceeds the state of the art on many image classification datasets, whilst being relatively cheap to pre-train.
我们探索了将Transformer直接应用于图像识别. 与之前的作品在计算机视觉中使用自觉性不同,除了最初的补丁提取步骤外,我们不会在架构中引入针对图像的诱导偏差. 相反,我们将一个图像解释为一个补丁的序列,用NLP中使用的标准Transformer编码器处理. 这一简单但可扩展的战略,如果与大型数据集的预训相结合,效果就惊人。 因此,Visual Transformer在许多图像分类数据集上匹配或超过最新水平,同时对预训来说相对便宜.
While these initial results are encouraging, many challenges remain. One is to apply ViT to other computer vision tasks, such as detection and segmentation. Our results, coupled with those in Carion et al. 2020, indicate the promise of this approach. Another challenge is to continue exploring self-supervised pre-training methods. Our initial experiments show improvement from self-supervised pre-training, but there is still large gap between self-supervised and large-scale supervised pre-training. Finally, further scaling of ViT would likely lead to improved performance.
虽然这些初步成果令人鼓舞,但仍然存在许多挑战。 一是将VIT应用于其他计算机视觉任务,如检测和分割. 我们的成果,加上Carion等人2020年的成果,都表明了这一办法的希望。 另一个挑战是继续探索自监督的培训前方法。 我们的初始实验表明,比起自监督的预训有所改进,但自监督的预训和大规模监督的预训之间仍然有很大的差距. 最后,进一步扩大虚拟技术的规模,很可能导致业绩的改善。
Acknowledgements
致谢
The work was performed in Berlin, Zürich, and Amsterdam. We thank many colleagues at Google for their help, in particular Andreas Steiner for crucial help with the infrastructure and the open-source release of the code; Joan Puigcerver and Maxim Neumann for help with the large-scale training infrastructure; Dmitry Lepikhin, Aravindh Mahendran, Daniel Keysers, Mario Lučić, Noam Shazeer, Ashish Vaswani, and Colin Raffel for useful discussions.
这项工作在柏林、苏黎世和阿姆斯特丹进行。 我们感谢Google的许多同事的帮助,特别是Andreas Steiner在基础设施和公开源码发布方面提供了关键帮助;Joan Puigcerver和Maxim Neumann在大规模培训基础设施方面提供了帮助;Dmitry Lepikhin、Aravendh Mahendran、Daniel Keysers、Mario Lučić、Noam Shazeer、Ashish Vaswani和Colin Raffel进行了有益的讨论。
References
参考文献
Abnar & Zuidema (2020) Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In ACL , 2020.
Bachman et al. (2019) Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS , 2019.
Baevski & Auli (2019) Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In ICLR , 2019.
Bello et al. (2019) I. Bello, B. Zoph, Q. Le, A. Vaswani, and J. Shlens. Attention augmented convolutional networks. In ICCV , 2019.
Beyer et al. (2020) Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv , 2020.
Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv , 2020.
Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020.
Chen et al. (2020a) Mark Chen, Alec Radford, Rewon Child, Jeff Wu, and Heewoo Jun. Generative pretraining from pixels. In ICML , 2020a.
Chen et al. (2020b) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020b.
Chen et al. (2020c) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: UNiversal Image-TExt Representation Learning. In ECCV , 2020c.
Child et al. (2019) Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv , 2019.
Cordonnier et al. (2020) Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. In ICLR , 2020.
Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.
Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL , 2019.
Djolonga et al. (2020) Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan Moldovan, Sylvan Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucic. On robustness and transferability of convolutional neural networks. arXiv , 2020.
He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.
He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020.
Ho et al. (2019) Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv , 2019.
Hu et al. (2018) Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In CVPR , 2018.
Hu et al. (2019) Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In ICCV , 2019.
Huang et al. (2020) Zilong Huang, Xinggang Wang, Yunchao Wei, Lichao Huang, Humphrey Shi, Wenyu Liu, and Thomas S. Huang. Ccnet: Criss-cross attention for semantic segmentation. In ICCV , 2020.
Hénaff et al. (2020) Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In ICML , 2020.
Ioffe & Szegedy (2015) Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015.
Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.
Kolesnikov et al. (2020) Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. In ECCV , 2020.
Krizhevsky (2009) Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS , 2012.
LeCun et al. (1989) Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation , 1:541–551, 1989.
Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv , 2020.
Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. VisualBERT: A Simple and Performant Baseline for Vision and Language. In Arxiv , 2019.
Locatello et al. (2020) Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv , 2020.
Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In NeurIPS . 2019.
Mahajan et al. (2018) Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV , 2018.
Nilsback & Zisserman (2008) M. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In ICVGIP , 2008.
Parkhi et al. (2012) Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR , 2012.
Parmar et al. (2018) Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML , 2018.
Polyak & Juditsky (1992) B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization , 30(4):838–855, 1992. doi: 10.1137/0330046 . URL https://doi.org/10.1137/0330046 .
Qiao et al. (2019) Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. arXiv preprint arXiv:1903.10520 , 2019.
Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding with unsupervised learning. Technical Report , 2018.
Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical Report , 2019.
Ramachandran et al. (2019) Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In NeurIPS , 2019.
Sun et al. (2017) Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV , 2017.
Sun et al. (2019) Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV , 2019.
Touvron et al. (2019) Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy. In NeurIPS . 2019.
Touvron et al. (2020) Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy: Fixefficientnet. arXiv preprint arXiv:2003.08237 , 2020.
Tschannen et al. (2020) Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucic. Self-supervised learning of video-induced visual invariances. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.
Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , 2017.
Wang et al. (2020a) Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In ECCV , 2020a.
Wang et al. (2020b) Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. arXiv preprint arXiv:2003.07853 , 2020b.
Wang et al. (2019) Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In ACL , 2019.
Wang et al. (2018) Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR , 2018.
Weissenborn et al. (2019) Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In ICLR , 2019.
Wu et al. (2020) Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Masayoshi Tomizuka, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision. arxiv , 2020.
Wu & He (2018) Yuxin Wu and Kaiming He. Group normalization. In ECCV , 2018.
Xie et al. (2020) Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In CVPR , 2020.
Zhai et al. (2019a) Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S 4 L: Self-Supervised Semi-Supervised Learning. In ICCV , 2019a.
Zhai et al. (2019b) Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867 , 2019b.
Zhao et al. (2020) Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In CVPR , 2020.
Appendix
附录
Appendix A Multihead Self-attention
附录 A 多头自注意力
Standard self-attention (SA, Vaswani et al. 2017) is a popular building block for neural architectures. For each element in an input sequence , we compute a weighted sum over all values in the sequence. The attention weights are based on the pairwise similarity between two elements of the sequence and their respective query and key representations.
标准自注(SA,Vaswani et al. 2017)是神经建筑的热门构件. 对于输入序列中的每个元素,我们计算序列中所有值的加权和数. 注意权重是基于序列的两个要素及其各自的查询和关键表示的对等相似性.
Multihead self-attention (MSA) is an extension of SA in which we run self-attention operations, called “heads”, in parallel, and project their concatenated outputs. To keep compute and number of parameters constant when changing , (Eq. 5) is typically set to .
多头自留(MSA)是SA的延伸,我们在其中同时进行称为“头”的自留业务,并预测其连带产出。 要在改变时保持计算和参数数常数,(Eq. 5)通常被设定为.
Appendix B Experiment details
附录 B 实验细节
B.1 Training
B.1 训练
| Models | Dataset | Epochs | Base LR | LR decay | Weight decay | Dropout |
|---|---|---|---|---|---|---|
| ViT-B/{16,32} | JFT-300M | 7 | linear | 0.1 | 0.0 | |
| ViT-L/32 | JFT-300M | 7 | linear | 0.1 | 0.0 | |
| ViT-L/16 | JFT-300M | 7/14 | linear | 0.1 | 0.0 | |
| ViT-H/14 | JFT-300M | 14 | linear | 0.1 | 0.0 | |
| R50x{1,2} | JFT-300M | 7 | linear | 0.1 | 0.0 | |
| R101x1 | JFT-300M | 7 | linear | 0.1 | 0.0 | |
| R152x{1,2} | JFT-300M | 7 | linear | 0.1 | 0.0 | |
| R50+ViT-B/{16,32} | JFT-300M | 7 | linear | 0.1 | 0.0 | |
| R50+ViT-L/32 | JFT-300M | 7 | linear | 0.1 | 0.0 | |
| R50+ViT-L/16 | JFT-300M | 7/14 | linear | 0.1 | 0.0 | |
| ViT-B/{16,32} | ImageNet-21k | 90 | linear | 0.03 | 0.1 | |
| ViT-L/{16,32} | ImageNet-21k | 30/90 | linear | 0.03 | 0.1 | |
| ViT- | ImageNet | 300 | cosine | 0.3 | 0.1 |
Table 3: Hyperparameters for training. All models are trained with a batch size of 4096 and learning rate warmup of 10k steps. For ImageNet we found it beneficial to additionally apply gradient clipping at global norm 1. Training resolution is 224.
| 模型 | Data | 埃波希斯语Name | 基础 LR | LR 衰变 | 重量衰减 | 辍学 |
|---|---|---|---|---|---|---|
| 维T-B/{16,32} | JFT - 300M (韩语) | 7 | 线性 | 0.1 | 0.0 | |
| VIT-L/32号卫星 | JFT - 300M (韩语) | 7 | 线性 | 0.1 | 0.0 | |
| 维T-L/16 | JFT - 300M (韩语) | 7/14 | 线性 | 0.1 | 0.0 | |
| 维特-H/14 | JFT - 300M (韩语) | 14 | 线性 | 0.1 | 0.0 | |
| R50x{1,2} (英语). | JFT - 300M (韩语) | 7 | 诸位 | 0.1 | 0.0 | |
| R101x1 坐标 | JFT - 300M (韩语) | 7 | 线性 | 0.1 | 0.0 | |
| R152x{1,2} (英语). | JFT - 300M (韩语) | 7 | 线性 | 0.1 | 0.0 | |
| R50+ViT-B/{16,32} (英语). | JFT - 300M (韩语) | 7 | 线性 | 0.1 | 0.0 | |
| R50+ViT-L/32 导弹 | JFT - 300M (韩语) | 7 | 线性 | 0.1 | 0.0 | |
| R50+ViT-L/16 号机车 | JFT - 300M (韩语) | 7/14 | 线性 | 0.1 | 0.0 | |
| 维T-B/{16,32} | 图像Net- 21k | 90 | 线性 | 0.03 | 0.1 | |
| 维T-L/{16,32} | 图像Net- 21k | 30/90 | 线性 | 0.03 | 0.1 | |
| VIT -号 | 图像网 | 300 | 余弦线 | 0.3 | 0.1 |
表3:培训的超参数。 所有型号都经过批量规模4096的训练,学习热速达到10k步. 对于ImageNet,我们发现它有利于在全球规范1中进一步应用梯度剪切。 培训决议是224。
Table 3 summarizes our training setups for our different models. We found strong regularization to be key when training models from scratch on ImageNet. Dropout, when used, is applied after every dense layer except for the the qkv-projections and directly after adding positional- to patch embeddings. Hybrid models are trained with the exact setup as their ViT counterparts. Finally, all training is done on resolution 224.
表3概述了我们为不同模式举办的培训。 我们发现,在ImageNet从零开始培训模型时,强有力的规范化是关键。 在使用时,除qkv-投影外,在每个稠密层后应用Dropout,并在添加位置-后直接应用补丁嵌入. 混合型号作为VIT对应型号进行精确设置训练. 最后,所有培训都是就第224号决议进行的。
B.1.1 Fine-tuning
| Dataset | Steps | Base LR |
|---|---|---|
| ImageNet | 20 000 | {0.003, 0.01, 0.03, 0.06} |
| CIFAR100 | 10 000 | {0.001, 0.003, 0.01, 0.03} |
| CIFAR10 | 10 000 | {0.001, 0.003, 0.01, 0.03} |
| Oxford-IIIT Pets | 500 | {0.001, 0.003, 0.01, 0.03} |
| Oxford Flowers-102 | 500 | {0.001, 0.003, 0.01, 0.03} |
| VTAB (19 tasks) | 2 500 | 0.01 |
Table 4: Hyperparameters for fine-tuning. All models are fine-tuned with cosine learning rate decay, a batch size of 512, no weight decay, and grad clipping at global norm 1. If not mentioned otherwise, fine-tuning resolution is 384.
B.1.1 微调
- 数据集 * 步骤 * 基准 LR *
|,|也.
################################################################################################################################################################################################################################################################
===============================================================================================================================================================================================================================================================
CIFAR10=10 000============================================================================================================
牛津 - IIIT 宠物 | 500 | {0.001, 0.003, 0.01, 0.03} |
{\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}牛津百花
暨VTAB(19项任务)
表4:用于微调的超参数。 所有模型均以余弦学习速率衰变进行微调,批量尺寸为512,无重量衰变,分级剪切达到全球规范1. 如果没有提及,微调分辨率为384。
We fine-tune all ViT models using SGD with a momentum of 0.9. We run a small grid search over learning rates, see learning rate ranges in Table 4. To do so, we use small sub-splits from the training set (10% for Pets and Flowers, 2% for CIFAR, 1% ImageNet) as development set and train on the remaining data. For final results we train on the entire training set and evaluate on the respective test data. For fine-tuning ResNets and hybrid models we use the exact same setup, with the only exception of ImageNet where we add another value to the learning rate sweep. Additionally, for ResNets we also run the setup of Kolesnikov et al. 2020 and select the best results across this run and our sweep. Finally, if not mentioned otherwise, all fine-tuning experiments run at 384 resolution (running fine-tuning at different resolution than training is common practice (Kolesnikov et al. 2020)).
我们用0.9的动力对使用SGD的所有VIT模型进行微调. 我们运行一个小网格搜索 相对于学习率, 参见学习率范围 在表4。 为此,我们使用训练集的小分集(Pets and Flowers为10%,CIFAR为2%,ImageNet为1%)作为开发集,并在剩余数据上进行培训. 关于最后结果,我们对整个培训单元进行培训,并评估相应的测试数据。 对于微调ResNet和混合模型,我们使用完全相同的设置,唯一的例外是ImageNet,我们在学习速率扫描中增加了另一个值. 此外,对于ResNets,我们还运行了科列斯尼科夫等人2020年的设置,并选择了这次运行和我们扫荡的最好结果。 最后,如果不另提的话,以384分辨率进行的所有微调试验(以不同分辨率进行微调是常见的做法(Kolesnikov等人,2020年))。
When transferring ViT models to another dataset, we remove the whole head (two linear layers) and replace it by a single, zero-initialized linear layer outputting the number of classes required by the target dataset. We found this to be a little more robust than simply re-initializing the very last layer.
在将VIT模型转移到另一个数据集时,我们移除了整个头(两个线性层),代之以一个零初始化的线性层输出目标数据集所需的类数. 我们发现这比重新启动最后一层要强一些。
For VTAB we follow the protocol in Kolesnikov et al. 2020, and use the same hyperparameter setting for all tasks. We use a learning rate of and train for steps (Tab. 4). We chose this setting by running a small sweep over two learning rates and two schedules, and selecting the setting with the highest VTAB score on the 200-example validation sets. We follow the pre-processing used in Kolesnikov et al. 2020, except that we do not use task-specific input resolutions. Instead we find that Vision Transformer benefits most from a high resolution () for all tasks.
对于VTAB,我们遵循科列斯尼科夫等人2020年的协议,对所有任务使用相同的超参数设置. 我们使用的学习率,对步骤进行列车(Tab.4). 我们选择了这个设置,通过对两个学习率和两个时间表进行小扫射,并在200个实例验证集中选择VTAB得分最高的设置. 我们遵循科列斯尼科夫等人2020年采用的预处理方法,只是我们不使用特定任务输入分辨率. 相反,我们发现,《愿景变换器》从高分辨率()的所有任务中受益最大。
B.1.2 Self-supervision
We employ the masked patch prediction objective for preliminary self-supervision experiments. To do so we corrupt 50% of patch embeddings by either replacing their embeddings with a learnable [mask] embedding (80%), a random other patch embedding (10%) or just keeping them as is (10%). This setup is very similar to the one used for language by Devlin et al. 2019. Finally, we predict the 3-bit, mean color (i.e., 512 colors in total) of every corrupted patch using their respective patch representations.
B.1.2 自监督
我们使用“假补丁预测”目标进行初步自监督实验。 为此,我们将50%的补丁嵌入进行腐败,要么用可学习的[面具]嵌入(80%)、随机的其他补丁嵌入(10%),要么保持原样(10%)。 这种设置与Devlin等人2019年语言使用的设置非常相似. 最后,我们预测每个损坏的补丁使用各自补丁表示的3位,平均颜色(即共计512色).
We trained our self-supervised model for 1M steps (ca. 14 epochs) with batch size 4096 on JFT. We use Adam, with a base learning rate of , warmup of 10k steps and cosine learning rate decay. As prediction targets for pretraining we tried the following settings: 1) predicting only the mean, 3bit color (i.e., 1 prediction of 512 colors), 2) predicting a downsized version of the patch with 3bit colors in parallel (i.e., 16 predictions of 512 colors), 3) regression on the full patch using L2 (i.e., 256 regressions on the 3 RGB channels). Surprisingly, we found that all worked quite well, though L2 was slightly worse. We report final results only for option 1) because it has shown best few-shot performance. We also experimented with 15% corruption rate as used by Devlin et al. 2019 but results were also slightly worse on our few-shot metrics.
我们为1M步(约14个纪元)训练了自监督的型号,JFT上的批量尺寸为4096. 我们使用亚当, 基础学习率是, 热身10千步 和余弦学习率衰减。 作为预训的预测目标,我们尝试了以下设置:1)只预测平均,3bit颜色(即1预测512颜色),2)预测一个缩小版的补丁,3bit颜色平行(即16预测512颜色),3)利用L2(即3 RGB通道的256回归)在全补丁上回归. 令人惊奇的是,我们发现一切都很顺利,尽管L2稍差。 我们只报告备选方案1的最终结果,因为它显示了最佳的少许成绩。 我们还尝试了Devlin等人在2019年使用的15%的腐败率,但结果在我们的几率指标上也稍差一些。
Lastly, we would like to remark that our instantiation of masked patch prediction doesn’t require such an enormous amount of pretraining nor a large dataset such as JFT in order to lead to similar performance gains on ImageNet classification. That is, we observed diminishing returns on downstream performance after 100k pretraining steps, and see similar gains when pretraining on ImageNet.
最后,我们要指出的是,我们对蒙庵补丁的即兴预测不需要讶异的预训,也不需要像JFT这样的大数据集,以便取得ImageNet分类的类似性能收益。 也就是说,我们看到,在100公里的预训步骤之后,下游业绩的回报不断下降,在ImageNet预训时也看到类似的收益。
Appendix C Additional Results
附录 C 额外结果
We report detailed results corresponding to the figures presented in the paper. Table 5 corresponds to Figure 4 from the paper and shows transfer performance of different ViT models pre-trained on datasets of increasing size: ImageNet, ImageNet-21k, and JFT-300M. Table 6 corresponds to Figure 5 from the paper and shows the transfer performance of ViT, ResNet, and hybrid models of varying size, as well as the estimated computational cost of their pre-training.
我们报告与文件所列数字相对应的详细结果。 表5对应于纸面图4,并显示不同VIT模型在大小越来越大的数据集上预先训练的传输性能:ImageNet,ImageNet-21k,以及JFT-300M. 表6对应于文件中的图5,显示了VIT,ResNet和大小不同的混合模型的转移性能,以及其预训的估计计算成本.
| ViT-B/16 | ViT-B/32 | ViT-L/16 | ViT-L/32 | ViT-H/14 | ||
|---|---|---|---|---|---|---|
| ImageNet | CIFAR-10 | 98.13 | 97.77 | 97.86 | 97.94 | - |
| CIFAR-100 | 87.13 | 86.31 | 86.35 | 87.07 | - | |
| ImageNet | 77.91 | 73.38 | 76.53 | 71.16 | - | |
| ImageNet ReaL | 83.57 | 79.56 | 82.19 | 77.83 | - | |
| Oxford Flowers-102 | 89.49 | 85.43 | 89.66 | 86.36 | - | |
| Oxford-IIIT-Pets | 93.81 | 92.04 | 93.64 | 91.35 | - | |
| ImageNet-21k | CIFAR-10 | 98.95 | 98.79 | 99.16 | 99.13 | 99.27 |
| CIFAR-100 | 91.67 | 91.97 | 93.44 | 93.04 | 93.82 | |
| ImageNet | 83.97 | 81.28 | 85.15 | 80.99 | 85.13 | |
| ImageNet ReaL | 88.35 | 86.63 | 88.40 | 85.65 | 88.70 | |
| Oxford Flowers-102 | 99.38 | 99.11 | 99.61 | 99.19 | 99.51 | |
| Oxford-IIIT-Pets | 94.43 | 93.02 | 94.73 | 93.09 | 94.82 | |
| JFT-300M | CIFAR-10 | 99.00 | 98.61 | 99.38 | 99.19 | 99.50 |
| CIFAR-100 | 91.87 | 90.49 | 94.04 | 92.52 | 94.55 | |
| ImageNet | 84.15 | 80.73 | 87.12 | 84.37 | 88.04 | |
| ImageNet ReaL | 88.85 | 86.27 | 89.99 | 88.28 | 90.33 | |
| Oxford Flowers-102 | 99.56 | 99.27 | 99.56 | 99.45 | 99.68 | |
| Oxford-IIIT-Pets | 95.80 | 93.40 | 97.11 | 95.83 | 97.56 |
Table 5: Top1 accuracy (in %) of Vision Transformer on various datasets when pre-trained on ImageNet, ImageNet-21k or JFT300M. These values correspond to Figure 4 in the main text. Models are fine-tuned at 384 resolution. Note that the ImageNet results are computed without additional techniques (Polyak averaging and 512 resolution images) used to achieve results in Table 2.
| ViT-B/16号卫星 | pearli-B | 维T-L/16 | VIT-L/32号卫星 | 维特-H/14 | ||
|---|---|---|---|---|---|---|
| 图像网 | 科伊法尔-10 | 98.13 | 97.77 | 97.86 | 97.94 | - |
| CIFAR-100 (法语) | 87.13 | 86.31 | 86.35 | 87.07 | - | |
| 图像网 | 77.91 | 73.38 | 76.53 | 71.16 | - | |
| 图像网络ReaL | 83.57 | 79.56 | 82.19 | 77.83 | - | |
| 牛津花报-102 | 89.49 | 85.43 | 89.66 | 86.36 | - | |
| 牛津-IIIT-Pets | 93.81 | 92.04 | 93.64 | 91.35 | - | |
| 图像Net- 21k | 科伊法尔-10 | 98.95 | 98.79 | 99.16 | 99.13 | 99.27 |
| CIFAR-100 (法语) | 91.67 | 91.97 | 93.44 | 93.04 | 93.82 | |
| 图像网 | 83.97 | 81.28 | 85.15 | 80.99 | 85.13 | |
| 图像网络ReaL | 88.35 | 86.63 | 88.40 | 85.65 | 88.70 | |
| 牛津花报-102 | 99.38 | 99.11 | 99.61 | 99.19 | 99.51 | |
| 牛津-IIIT-Pets | 94.43 | 93.02 | 94.73 | 93.09 | 94.82 | |
| JFT - 300M (韩语) | 科伊法尔-10 | 99.00 | 98.61 | 99.38 | 99.19 | 99.50 |
| CIFAR-100 (法语) | 91.87 | 90.49 | 94.04 | 92.52 | 94.55 | |
| 图像网 | 84.15 | 80.73 | 87.12 | 84.37 | 88.04 | |
| 图像网络ReaL | 88.85 | 86.27 | 89.99 | 88.28 | 90.33 | |
| 牛津花报-102 | 99.56 | 99.27 | 99.56 | 99.45 | 99.68 | |
| 牛津-IIIT-Pets | 95.80 | 93.40 | 97.11 | 95.83 | 97.56 |
表5:在ImageNet,ImageNet-21k或JFT300M上预训时,各种数据集上的Vision Transformer的Top1精度(单位:%). 这些值与正文图4相对应. 模型微调384分辨率. 请注意,在计算ImageNet结果时没有采用其他技术(Polyak平均和512张分辨率图像),以在表2中取得成果。
| Epochs | ImageNet | ImageNet ReaL | CIFAR-10 | CIFAR-100 | Pets | Flowers | exaFLOPs | |
|---|---|---|---|---|---|---|---|---|
| name | ||||||||
| ViT-B/32 | 7 | 80.73 | 86.27 | 98.61 | 90.49 | 93.40 | 99.27 | 55 |
| ViT-B/16 | 7 | 84.15 | 88.85 | 99.00 | 91.87 | 95.80 | 99.56 | 224 |
| ViT-L/32 | 7 | 84.37 | 88.28 | 99.19 | 92.52 | 95.83 | 99.45 | 196 |
| ViT-L/16 | 7 | 86.30 | 89.43 | 99.38 | 93.46 | 96.81 | 99.66 | 783 |
| ViT-L/16 | 14 | 87.12 | 89.99 | 99.38 | 94.04 | 97.11 | 99.56 | 1567 |
| ViT-H/14 | 14 | 88.08 | 90.36 | 99.50 | 94.71 | 97.11 | 99.71 | 4262 |
| ResNet50x1 | 7 | 77.54 | 84.56 | 97.67 | 86.07 | 91.11 | 94.26 | 50 |
| ResNet50x2 | 7 | 82.12 | 87.94 | 98.29 | 89.20 | 93.43 | 97.02 | 199 |
| ResNet101x1 | 7 | 80.67 | 87.07 | 98.48 | 89.17 | 94.08 | 95.95 | 96 |
| ResNet152x1 | 7 | 81.88 | 87.96 | 98.82 | 90.22 | 94.17 | 96.94 | 141 |
| ResNet152x2 | 7 | 84.97 | 89.69 | 99.06 | 92.05 | 95.37 | 98.62 | 563 |
| ResNet152x2 | 14 | 85.56 | 89.89 | 99.24 | 91.92 | 95.75 | 98.75 | 1126 |
| ResNet200x3 | 14 | 87.22 | 90.15 | 99.34 | 93.53 | 96.32 | 99.04 | 3306 |
| R50x1+ViT-B/32 | 7 | 84.90 | 89.15 | 99.01 | 92.24 | 95.75 | 99.46 | 106 |
| R50x1+ViT-B/16 | 7 | 85.58 | 89.65 | 99.14 | 92.63 | 96.65 | 99.40 | 274 |
| R50x1+ViT-L/32 | 7 | 85.68 | 89.04 | 99.24 | 92.93 | 96.97 | 99.43 | 246 |
| R50x1+ViT-L/16 | 7 | 86.60 | 89.72 | 99.18 | 93.64 | 97.03 | 99.40 | 859 |
| R50x1+ViT-L/16 | 14 | 87.12 | 89.76 | 99.31 | 93.89 | 97.36 | 99.11 | 1668 |
Table 6: Detailed results of model scaling experiments. These correspond to Figure 5 in the main paper. We show transfer accuracy on several datasets, as well as the pre-training compute (in exaFLOPs).
| 埃波希斯语Name | 图像网 | 图像网络ReaL | 科伊法尔-10 | CIFAR-100 (法语) | 宠物 | 鲜花 | ExaFLOPs 软件 | |
|---|---|---|---|---|---|---|---|---|
| 名称 | ||||||||
| VIT-B/32号卫星 | 7 | 80.73 | 86.27 | 98.61 | 90.49 | 93.40 | 99.27 | 55 |
| ViT-B/16号卫星 | 7 | 84.15 | 88.85 | 99.00 | 91.87 | 95.80 | 99.56 | 224 |
| VIT-L/32号卫星 | 7 | 84.37 | 88.28 | 99.19 | 92.52 | 95.83 | 99.45 | 196 |
| 维T-L/16 | 7 | 86.30 | 89.43 | 99.38 | 93.46 | 96.81 | 99.66 | 783 |
| 维T-L/16 | 14 | 87.12 | 89.99 | 99.38 | 94.04 | 97.11 | 99.56 | 1567 |
| 维特-H/14 | 14 | 88.08 | 90.36 | 99.50 | 94.71 | 97.11 | 99.71 | 4262 |
| ResNet50x1 (英语). | 7 | 77.54 | 84.56 | 97.67 | 86.07 | 91.11 | 94.26 | 50 |
| ResNet50x2 (英语). | 7 | 82.12 | 87.94 | 98.29 | 89.20 | 93.43 | 97.02 | 199 |
| ResNet 101x1 (英语). | 7 | 80.67 | 87.07 | 98.48 | 89.17 | 94.08 | 95.95 | 96 |
| ResNet 152x1 (英语). | 7 | 81.88 | 87.96 | 98.82 | 90.22 | 94.17 | 96.94 | 141 |
| ResNet见上文 | 7 | 84.97 | 89.69 | 99.06 | 92.05 | 95.37 | 98.62 | 563 |
| ResNet见上文 | 14 | 85.56 | 89.89 | 99.24 | 91.92 | 95.75 | 98.75 | 1126 |
| ResNet 200x3 (英语). | 14 | 87.22 | 90.15 | 99.34 | 93.53 | 96.32 | 99.04 | 3306 |
| R50x1+维T-B/32 | 7 | 84.90 | 89.15 | 99.01 | 92.24 | 95.75 | 99.46 | 106 |
| R50x1+ViT-B/16 (英语). | 7 | 85.58 | 89.65 | 99.14 | 92.63 | 96.65 | 99.40 | 274 |
| R50x1+ViT-L/32 (英语). | 7 | 85.68 | 89.04 | 99.24 | 92.93 | 96.97 | 99.43 | 246 |
| R50x1+ 老维-L/16 | 7 | 86.60 | 89.72 | 99.18 | 93.64 | 97.03 | 99.40 | 859 |
| R50x1+ 老维-L/16 | 14 | 87.12 | 89.76 | 99.31 | 93.89 | 97.36 | 99.11 | 1668 |
表6:模型缩放实验的详细结果. 这些与主要文件中的图5相对应。 我们在几个数据集上 我们会显示传输精度 以及训练前计算( Board exaFLOPs).
Appendix D Additional Analyses
附录 D 额外分析
D.1 SGD vs. Adam for ResNets
D.1 ResNet 上 SGD 与 Adam
ResNets are typically trained with SGD and our use of Adam as optimizer is quite unconventional. Here we show the experiments that motivated this choice. Namely, we compare the fine-tuning performance of two ResNets – 50x1 and 152x2 – pre-trained on JFT with SGD and Adam. For SGD, we use the hyperparameters recommended by Kolesnikov et al. 2020. Results are presented in Table 7. Adam pre-training outperforms SGD pre-training on most datasets and on average. This justifies the choice of Adam as the optimizer used to pre-train ResNets on JFT. Note that the absolute numbers are lower than those reported by Kolesnikov et al. 2020, since we pre-train only for epochs, not .
ResNet是典型的SGD训练,我们使用Adam作为优化器相当不常规. 在这里,我们展示了促使这个选择的实验。 也就是说,我们比较了两个ResNet的微调性能 — — 50x1和152x2 — — 在JFT上预训的SGD和Adam. 对于SGD,我们使用科列斯尼科夫等人于2020年推荐的超参数. 结果见表7。 亚当的预训表现优于SGD在大多数数据集和平均的预训. 这为选择亚当作为JFT上预训ResNets的优化器提供了理由. 请注意,绝对数字低于科列斯尼科夫等人2020年报告的数字,因为我们只为时代进行预训,而不是.
| ResNet50 | ResNet152x2 | |||
|---|---|---|---|---|
| Dataset | Adam | SGD | Adam | SGD |
| ImageNet | ||||
| CIFAR10 | ||||
| CIFAR100 | ||||
| Oxford-IIIT Pets | ||||
| Oxford Flowers-102 | ||||
| Average |
Table 7: Fine-tuning ResNet models pre-trained with Adam and SGD.
| ResNet50 网络50 | ResNet 152x2 (英语). | |||
|---|---|---|---|---|
| 数据集 | 亚当 | SGD 软件 | 亚当 | SGD 软件 |
| 图像网 | ||||
| 中非共和国10 | ||||
| 中非共和国100 | ||||
| 牛津-IIIT 宠物 | ||||
| 牛津花报-102 | ||||
| nào。 |
表7:与Adam和SGD一起预训的微调ResNet型号.
D.2 Transformer shape
D.2 Transformer 形状
Figure 8: Scaling different model dimensions of the Vision Transformer.
图8:放大视野变换器的不同模型维度.
We ran ablations on scaling different dimensions of the Transformer architecture to find out which are best suited for scaling to very large models. Figure 8 shows 5-shot performance on ImageNet for different configurations. All configurations are based on a ViT model with layers, , and a patch size of , the intersection of all lines. We can see that scaling the depth results in the biggest improvements which are clearly visible up until 64 layers. However, diminishing returns are already visible after 16 layers. Interestingly, scaling the width of the network seems to result in the smallest changes. Decreasing the patch size and thus increasing the effective sequence length shows surprisingly robust improvements without introducing parameters. These findings suggest that compute might be a better predictor of performance than the number of parameters, and that scaling should emphasize depth over width if any. Overall, we find that scaling all dimensions proportionally results in robust improvements.
我们研究了“Transformer”结构的不同维度 以找出哪些最适合缩放到非常大的模型。 图8显示不同配置在ImageNet上的5拍性能. 所有配置都的平等权利 đã基于蓓丽花的ViT模型,拥有层,,和的补丁大小,是所有线路的交汇点. 我们可以看到,扩大深度的结果是最大的改进,这些改进可以清楚地看到到64层。 然而,在16层之后,回报率已经下降。 有趣的是,缩小网络的宽度似乎导致最小的变化. 减少补丁大小,从而增加有效序列长度,显示出令人惊讶的强劲改进,而没有引入参数。 这些调查结果表明,计算可能比参数数更能预测性能,如果有的话,衡量应强调深度与宽度。 总的来说,我们发现,按比例扩大所有层面,将带来有力的改进。
Figure 9: Comparison of class-token and global average pooling classifiers. Both work similarly well, but require different learning-rates.
图9:类集合与全球平均集合分类比较。 两者同样运作良好,但需要不同的学习率。
D.3 Head Type and class token
D.3 头类型与 class token
In order to stay as close as possible to the original Transformer model, we made use of an additional [class] token, which is taken as image representation. The output of this token is then transformed into a class prediction via a small multi-layer perceptron (MLP) with as non-linearity in the single hidden layer.
为了尽可能贴近原Transformer模型,我们使用了额外的[类]符号,该符号作为图像代表. 此令牌的输出随后通过一个小的多层受体(MLP)转化为一个类预测,其中作为单隐藏层的非线性.
This design is inherited from the Transformer model for text, and we use it throughout the main paper. An initial attempt at using only image-patch embeddings, globally average-pooling (GAP) them, followed by a linear classifier—just like ResNet’s final feature map—performed very poorly. However, we found that this is neither due to the extra token, nor to the GAP operation. Instead, the difference in performance is fully explained by the requirement for a different learning-rate, see Figure 9.
这个设计是从Transformer模型中继承下来的文本,我们在整个主纸中都使用它. 最初尝试只使用图像画板嵌入(GAP), 然而,我们发现,这既不是额外的象征,也不是GAP行动造成的。 相反,业绩差异完全是由于要求不同的学习率,见图9。
D.4 Positional Embedding
D.4 位置嵌入
We ran ablations on different ways of encoding spatial information using positional embedding. We tried the following cases:
我们 运行不同的方式 编码空间信息 使用位置嵌入。 我们审理了以下案件:
- • Providing no positional information: Considering the inputs as a bag of patches.
- • 1-dimensional positional embedding: Considering the inputs as a sequence of patches in the raster order (default across all other experiments in this paper).
- • 2-dimensional positional embedding: Considering the inputs as a grid of patches in two dimensions. In this case, two sets of embeddings are learned, each for one of the axes, -embedding, and -embedding, each with size . Then, based on the coordinate on the path in the input, we concatenate the and embedding to get the final positional embedding for that patch.
- • Relative positional embeddings: Considering the relative distance between patches to encode the spatial information as instead of their absolute position. To do so, we use 1-dimensional Relative Attention, in which we define the relative distance all possible pairs of patches. Thus, for every given pair (one as query, and the other as key/value in the attention mechanism), we have an offset , where each offset is associated with an embedding. Then, we simply run extra attention, where we use the original query (the content of query), but use relative positional embeddings as keys. We then use the logits from the relative attention as a bias term and add it to the logits of the main attention (content-based attention) before applying the softmax.
· 不提供位置信息:将输入视为包件*。
- 1维位置嵌入:将输入视为光栅顺序中的补丁序列(在本文其他所有实验中默认).
- 二维位置嵌入:将输入视为两个维的补丁网格。 在这种情况下,学习了两套嵌入式,每套为轴式之一,-嵌入式,和-嵌入式,每套为尺寸为. 然后,根据输入中路径上的坐标,我们将和嵌入来获得该补丁的最终位置嵌入.
- 相对位置嵌入:考虑补丁之间的相对距离,将空间信息编码为而不是绝对位置. 为此,我们使用一维相对注意,其中定义了所有可能的对补丁的相对距离. 因此,对于每个给定的对(一个是查询,另一个是注意机制中的键/值),我们有一个抵消,其中每个抵消与嵌入相关。 然后,我们只是运行额外的注意,在那里我们使用原始查询(查询的内容),但使用相对位置嵌入作为密钥. 然后,我们用相对关注的对数作为偏差词,在应用软马克斯之前加入主要关注的对数(基于内容的注意).
| Pos. Emb. | Default/Stem | Every Layer | Every Layer-Shared |
|---|---|---|---|
| No Pos. Emb. | 0.61382 | N/A | N/A |
| 1-D Pos. Emb. | 0.64206 | 0.63964 | 0.64292 |
| 2-D Pos. Emb. | 0.64001 | 0.64046 | 0.64022 |
| Rel. Pos. Emb. | 0.64032 | N/A | N/A |
Table 8: Results of the ablation study on positional embeddings with ViT-B/16 model evaluated on ImageNet 5-shot linear.
| 阿宝,阿宝 | 默认/stem | 每据以层 | 每层共享 |
|---|---|---|---|
| 没有阿宝,安布 | 0.61382 | 不详 | 不详 |
| 1D Pos,安贝。 | 0.64206 | 0.63964 | 0.64292 |
| 二维宝斯,安布 | 0.64001 | 0.64046 | 0.64022 |
| (原始内容存档于2018-03-27). Rel. Pos, Emb. | 0.64032 | 24 . . . . . . . . . . . . . . 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 | 24 . . . . . . . . . . . . . . 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 动 |
表8:在ImageNet 5-shot线性上评价的VIT-B/16模型定位嵌入法研究的结果.
Figure 10: Position embeddings of models trained with different hyperparameters.
图10:使用不同超参数培训的模型的位置嵌入。
In addition to different ways of encoding spatial information, we also tried different ways of incorporating this information in our model. For the 1-dimensional and 2-dimensional positional embeddings, we tried three different cases: (1) add positional embeddings to the inputs right after the stem of them model and before feeding the inputs to the Transformer encoder (default across all other experiments in this paper); (2) learn and add positional embeddings to the inputs at the beginning of each layer; (3) add a learned positional embeddings to the inputs at the beginning of each layer (shared between layers).
除了对空间信息进行编码的不同方式外,我们还尝试了将这种信息纳入我们的模型的不同方式. 对于1维和2维位置嵌入为其不同的情况,我们尝试了三种不同的情况:(1)在其模型的干线之后并在将输入输入输入输入输入输入输入之前添加位置嵌入(本论文中其他所有实验的默认);(2)在每个层的开头学习和在输入中添加位置嵌入;(3)在每个层的开头添加一个学习到的位置嵌入(层间共享).
Table 8 summarizes the results from this ablation study on a ViT-B/16 model. As we can see, while there is a large gap between the performances of the model with no positional embedding and models with positional embedding, there is little to no difference between different ways of encoding positional information. We speculate that since our Transformer encoder operates on patch-level inputs, as opposed to pixel-level, the differences in how to encode spatial information is less important. More precisely, in patch-level inputs, the spatial dimensions are much smaller than the original pixel-level inputs, e.g., as opposed to , and learning to represent the spatial relations in this resolution is equally easy for these different positional encoding strategies. Even so, the specific pattern of position embedding similarity learned by the network depends on the training hyperparameters (Figure 10).
表8总结了这一关于Vit-B/16模型的研究的结果。 如我们所见,虽然模型的性能之间没有位置嵌入和有位置嵌入的模型存在很大差距,但不同方式编码位置信息之间几乎没有或没有区别. 我们推测,由于我们的Transformer编码器运行在补丁级输入上,而不是像素级,因此如何编码空间信息的差异就不那么重要了. 更确切地说,在补丁级输入中,空间维度远小于原始像素级输入,如相对于,学习在本解析中代表空间关系对于这些不同的位置编码策略同样容易. 即便如此,网络所学到的嵌入相似性的具体位置模式取决于训练超参数(图10).
Figure 11: Size of attended area by head and network depth. Attention distance was computed for 128 example images by averaging the distance between the query pixel and all other pixels, weighted by the attention weight. Each dot shows the mean attention distance across images for one of 16 heads at one layer. Image width is 224 pixels.
图11:按头部和网络深度分列的受关注区域大小。 128个实例图像的注意距离是通过平均查询像素和所有其他像素之间的距离计算的,以注意重量加权。 每个点都显示一个层上16个头中的一个的图像之间的平均关注距离. 图像宽度为224像素.
D.5 Empirical Computational Costs
D.5 经验计算开销
We are also interested in real-world speed of the architectures on our hardware, which is not always well predicted by theoretical FLOPs due to details like lane widths and cache sizes. For this purpose, we perform timing of inference speed for the main models of interest, on a TPUv3 accelerator; the difference between inference and backprop speed is a constant model-independent factor.
我们也对硬件上的架构的真实世界速度感兴趣,由于车道宽度和缓存尺寸等细节,理论FLOP并不总是很好地预测. 为此,我们对主要的 真趣模型进行推断速度的定时,在一个TPUv3加速器上;推论速度与反推速度的区别是一个恒定的模型独立因素.
Figure 12 (left) shows how many images one core can handle per second, across various input sizes. Every single point refers to the peak performance measured across a wide range of batch-sizes. As can be seen, the theoretical bi-quadratic scaling of ViT with image size only barely starts happening for the largest models at the largest resolutions.
图12(左)显示一个核心能够处理多少个每秒,跨不同输入大小的图像. 每一个点都指在广泛的批量尺寸中测量的峰值性能. 可以看出,ViT具有图像大小的理论双夸度缩放,对于最大的分辨率最大的模型,只是勉强开始出现.
Another quantity of interest is the largest batch-size each model can fit onto a core, larger being better for scaling to large datasets. Figure 12 (right) shows this quantity for the same set of models. This shows that large ViT models have a clear advantage in terms of memory-efficiency over ResNet models.
另一种兴趣是每个模型可以装入一个核心的最大批量尺寸,更大的更适合缩放大型数据集。 图12(右)显示同一套模型的这一数量。 这表明大型VIT模型在内存效率方面比ResNet模型有明显的优势.
Figure 12: Left: Real wall-clock timings of various architectures across input sizes. ViT models have speed comparable to similar ResNets. Right: Largest per-core batch-size fitting on device with various architectures across input sizes. ViT models are clearly more memory-efficient.
图12:左: 不同输入大小的各种架构真实的墙钟时间. VIT模型具有类似ResNet的可比较速度. ** 权利**: 安装在设备上的最大每个核心批量尺寸,不同输入大小的各种结构。 VIT模型显然更具有内存效率.
D.6 Axial Attention
D.6 轴向注意力
Axial Attention (Huang et al. 2020; Ho et al. 2019) is a simple, yet effective technique to run self-attention on large inputs that are organized as multidimensional tensors. The general idea of axial attention is to perform multiple attention operations, each along a single axis of the input tensor, instead of applying 1-dimensional attention to the flattened version of the input. In axial attention, each attention mixes information along a particular axis, while keeping information along the other axes independent. Along this line, Wang et al. 2020b proposed the AxialResNet model in which all the convolutions with kernel size in a ResNet50 are replaced by axial self-attention, i.e. a row and column attention, augmented by relative positional encoding. We have implemented AxialResNet as a baseline model.[^3].
轴心注意(Huang等人,2020年;Ho等人,2019年)是一种简单而有效的技术,可以对作为多维抗震器组织的大量投入进行自觉操作. 一般的轴感注意思想是进行多点注意操作,每个操作都沿着输入式的收音机的单轴进行,而不是对输入式的扁平版应用一维的注意. 在轴心注意中,每个注意沿特定轴线混合信息,同时沿其他轴线保持信息独立. 沿着这条线,Wang et al. 2020b提出了AxialResNet模型,其中ResNet50中所有具有内核大小的的演化被轴自觉所取代,即一行和列的注意力,并辅以相对位置编码. 我们实施了AxialResNet作为基线模型。
Moreover, we have modified ViT to process inputs in the 2-dimensional shape, instead of a 1-dimensional sequence of patches, and incorporate Axial Transformer blocks, in which instead of a self-attention followed by an MLP, we have a a row-self-attention plus an MLP followed by a column-self-attention plus an MLP.
此外,我们修改了ViT,以2维形状处理输入,而不是1维的补丁序列,并加入了轴Transformer块,在其中,我们没有自我注意,而是有一个行自注意,加上一个MLP,然后是列自注意,加上一个MLP.
Figure 13: Performance of Axial-Attention based models, in terms of top-1 accuracy on ImageNet 5-shot linear, versus their speed in terms of number of FLOPs (left) and inference time (left).
图13:以Axial-Attention为基础的模型的性能,以ImageNet上5射线的顶1精度表示,以FLOP数量(左)和推断时间(左)表示,相对于其速度.
Figure 13, present the performance of Axial ResNet, Axial-ViT-B/32 and Axial-ViT-B/16 on ImageNet 5shot linear, when pretrained on JFT dataset, verses the pretraining compute, both in terms of number of FLOPs and inference time (example per seconds). As we can see, both Axial-ViT-B/32 and Axial-ViT-B/16 do better than their ViT-B counterpart in terms of performance, but it comes at the cost of more compute. This is because in Axial-ViT models, each Transformer block with global self-attention is replaced by two Axial Transformer blocks, one with row and one with column self-attention and although the sequence length that self-attention operates on is smaller in axial case, there is a extra MLP per Axial-ViT block. For the AxialResNet, although it looks reasonable in terms of accuracy/compute trade-off (Figure 13, left), the naive implementation is extremely slow on TPUs (Figure 13, right).
图13,在ImageNet 5shot线性上,在JFT数据集上预训时,以FLOP的数量和推断时间(每秒实例)来显示Axial ResNet,Axial-ViT-B/32和Axial-ViT-B/16的性能. 正如我们所见,Axial-ViT-B/32和Axial-ViT-B/16在性能上都比他们的ViT-B对应方做得更好,但代价是更计算. 这是因为在Axial-ViT模型中,每个具有全局自控功能的Transformer块被两个轴自控器块所取代,一个带排,一个带列自控功能的Transformer块,虽然自控运行的序列长度在轴箱中较小,但每个Transformer块都有一个额外的MLP. 对于AxialResNet来说,虽然在准确性/计算权衡上看起来是合理的(图13,左),但对于TPU来说,天真的执行极其缓慢(图13,右).
D.7 Attention Distance
D.7 注意力距离
To understand how ViT uses self-attention to integrate information across the image, we analyzed the average distance spanned by attention weights at different layers (Figure 11). This “attention distance” is analogous to receptive field size in CNNs. Average attention distance is highly variable across heads in lower layers, with some heads attending to much of the image, while others attend to small regions at or near the query location. As depth increases, attention distance increases for all heads. In the second half of the network, most heads attend widely across tokens.
为了了解ViT如何使用自我关注来整合整个图像中的信息,我们分析了不同层的注意力权重所跨越的平均距离(图11). 这种“注意距离”类似于有线电视新闻网所接受的实地规模。 平均关注距离在低层头部之间变化很大,有些头部关注大部分图像,而另一些则关注查询位置或附近小区域. 随着深度的增加,所有头部的注意力距离都会增加. 在网络的下半部分,大多数头目都广泛参加跨标志赛.
D.8 Attention Maps
D.8 注意力图
To compute maps of the attention from the output token to the input space (Figures 6 and 14), we used Attention Rollout (Abnar & Zuidema 2020). Briefly, we averaged attention weights of ViT-L/16 across all heads and then recursively multiplied the weight matrices of all layers. This accounts for the mixing of attention across tokens through all layers.
为了计算从输出符到输入空间(图6和图14)的注意力的地图,我们使用了"注意推出"(Abnar & Zuidema 2020). 简而言之,我们平均所有头部的注意力重量为VIT-L/16,然后递增各层的重量矩阵。 这说明在各种层次上将注意力混合到各种标志上。
Figure 14: Further example attention maps as in Figure 6 (random selection).
图14:图6(随机选择)中的更多关注地图。
D.9 ObjectNet Results
D.9 ObjectNet 结果
We also evaluate our flagship ViT-H/14 model on the ObjectNet benchmark following the evaluation setup in Kolesnikov et al. 2020, resulting in 82.1% top-5 accuracy and 61.7% top-1 accuracy.
我们还在Kolesnikov等人2020年的评估设置之后,对ObjectNet基准上的旗舰ViT-H/14模型进行了评估,结果最高精确度达到82.1%,最高精确度达到61.7%。
D.10 VTAB Breakdown
D.10 VTAB 细分
Table 9 shows the scores attained on each of the VTAB-1k tasks.
表9显示每个VTAB-1k任务取得的分数。
| Caltech101 | CIFAR-100 | DTD | Flowers102 | Pets | Sun397 | SVHN | Camelyon | EuroSAT | Resisc45 | Retinopathy | Clevr-Count | Clevr-Dist | DMLab | dSpr-Loc | dSpr-Ori | KITTI-Dist | sNORB-Azim | sNORB-Elev | Mean | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ViT-H/14 (JFT) | 95.3 | 85.5 | 75.2 | 99.7 | 97.2 | 65.0 | 88.9 | 83.3 | 96.7 | 91.4 | 76.6 | 91.7 | 63.8 | 53.1 | 79.4 | 63.3 | 84.5 | 33.2 | 51.2 | 77.6 |
| ViT-L/16 (JFT) | 95.4 | 81.9 | 74.3 | 99.7 | 96.7 | 63.5 | 87.4 | 83.6 | 96.5 | 89.7 | 77.1 | 86.4 | 63.1 | 49.7 | 74.5 | 60.5 | 82.2 | 36.2 | 51.1 | 76.3 |
| ViT-L/16 (I21k) | 90.8 | 84.1 | 74.1 | 99.3 | 92.7 | 61.0 | 80.9 | 82.5 | 95.6 | 85.2 | 75.3 | 70.3 | 56.1 | 41.9 | 74.7 | 64.9 | 79.9 | 30.5 | 41.7 | 72.7 |
Table 9: Breakdown of VTAB-1k performance across tasks.
| 加州理工学院101 | CIFAR-100 (法语) | 发展 | 鲜花102 | 宠物 | 太阳397 | 斯洛伐克 | 来宾 | 欧元卫星 | 缩写c45 | 复健症 | 客户 | 宽度 | 杜马拉布 | dSpr-Loc (英语). | dspr-Ori (英语). | KITTI 移动 | 斯洛伐克-阿齐姆 | 斯诺布-埃列夫 | 平均值 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VIT-H/14 (日本电信电信协会) | 95.3 | 85.5 | 75.2 | 99.7 | 97.2 | 65.0 | 88.9 | 83.3 | 96.7 | 91.4 | 76.6 | 91.7 | 63.8 | 53.1 | 79.4 | 63.3 | 84.5 | 33.2 | 51.2 | 77.6 |
| VIT-L/16 (日本电信电信公司) | 95.4 | 81.9 | 74.3 | 99.7 | 96.7 | 63.5 | 87.4 | 83.6 | 96.5 | 89.7 | 77.1 | 86.4 | 63.1 | 49.7 | 74.5 | 60.5 | 82.2 | 36.2 | 51.1 | 76.3 |
| ViT-L/16 (I21k) (英语). | 90.8 | 84.1 | 74.1 | 99.3 | 92.7 | 61.0 | 80.9 | 82.5 | 95.6 | 85.2 | 75.3 | 70.3 | 56.1 | 41.9 | 74.7 | 64.9 | 79.9 | 30.5 | 41.7 | 72.7 |
表9:跨任务VTAB-1k业绩细目。