ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表
摘要1 引言2 背景2.1 视觉–语言模型的类型学2.2 模态交互方式2.3 视觉嵌入方式区域特征网格特征图像块投影3 视觉–语言 Transformer3.1 模型概览3.2 预训练目标图像–文本匹配掩码语言建模3.3 整词掩码3.4 图像增强4 实验4.1 概览4.2 实现细节4.3 分类任务视觉问答视觉推理的自然语言4.4 检索任务4.5 消融实验4.6 VLP 模型的复杂度分析4.7 可视化5 结论与后续工作可扩展性视觉输入的掩码建模增强策略参考文献
返回论文列表
目录

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

收录于 ICML 2021 会议论文（Long Presentation）被引 2,496 次
打开论文来源

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

ViLT：不依赖卷积或区域监督的视觉–语言 Transformer

Wonjae Kim Affiliation:  Kakao Enterprise, Seongnam, Gyeonggi, Republic of Korea Correspondence to: wonjae.kim@navercorp.com    Bokyung Son Affiliation:  Kakao Enterprise, Seongnam, Gyeonggi, Republic of Korea    Ildoo Kim Affiliation:  Kakao Brain, Seongnam, Gyeonggi, Republic of Korea

Abstract

摘要

Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.

视觉–语言预训练（VLP）已经提升了多种视觉与语言联合下游任务的表现。现有 VLP 方法严重依赖图像特征提取，其中多数既用到区域监督（例如目标检测），也用到卷积架构（例如 ResNet）。文献里这一点常被放过，我们却认为它在两方面都成问题：（1）效率／速度上，光是提取输入特征所需的计算就远多于多模态交互步骤；（2）表达能力上，它被视觉嵌入器及其预先定义的视觉词表卡住了上限。本文提出一个尽量精简的 VLP 模型——视觉–语言 Transformer（ViLT）。它是单体式的：处理视觉输入的方式被大幅简化，几乎与处理文本输入一样，不再使用卷积。我们表明，ViLT 比以往 VLP 模型最多快几十倍，下游任务表现却具有竞争力，有时还更好。代码和预训练权重见 https://github.com/dandelin/vilt 。

Figure 1: Visual comparison of conventional VLP architectures and our proposed ViLT. We have entirely removed convolutional neural networks from the VLP pipeline without hurting performance on downstream tasks. ViLT is the first VLP model of which the modal-specific components require less computation than the transformer component for multimodal interactions.

图 1：传统 VLP 架构与本文提出的 ViLT 的直观对比。我们把卷积神经网络从 VLP 流水线中完全去掉，同时没有损害下游任务表现。ViLT 是第一个模态专用组件所需计算少于用于多模态交互的 Transformer 组件的 VLP 模型。

Figure 2: Four categories of vision-and-language models. The height of each rectangle denotes its relative computational size. VE, TE, and MI are short for visual embedder, textual embedder, and modality interaction, respectively.

图 2：视觉–语言模型的四种类型。每个矩形的高度表示其相对计算规模。VE、TE 和 MI 分别是视觉嵌入器、文本嵌入器和模态交互的简称。

1 Introduction

1 引言

The pre-train-and-fine-tune scheme has been expanded to a joint domain of vision and language, giving birth to the category of Vision-and-Language Pre-training (VLP) models (Lu et al. 2019; Chen et al. 2019; Su et al. 2019; Li et al. 2019; Tan & Bansal 2019; Li et al. 2020a; Lu et al. 2020; Cho et al. 2020; Qi et al. 2020; Zhou et al. 2020; Huang et al. 2020; Li et al. 2020b; Gan et al. 2020; Yu et al. 2020; Zhang et al. 2021). These models are pre-trained with image text matching and masked language modeling objectives[^1] on images and their aligned descriptions, and are fine-tuned on vision-and-language downstream tasks where the inputs involve two modalities.

预训练再微调的范式已经扩展到视觉与语言的联合领域，并催生出一类视觉–语言预训练（VLP）模型（Lu 等 2019；Chen 等 2019；Su 等 2019；Li 等 2019；Tan 与 Bansal 2019；Li 等 2020a；Lu 等 2020；Cho 等 2020；Qi 等 2020；Zhou 等 2020；Huang 等 2020；Li 等 2020b；Gan 等 2020；Yu 等 2020；Zhang 等 2021）。这些模型在图像及其对齐描述上，用图像–文本匹配和掩码语言建模目标做预训练，再在输入包含两种模态的视觉–语言下游任务上微调。

To be fed into VLP models, image pixels need to be initially embedded in a dense form alongside language tokens. Since the seminal work of Krizhevsky et al. 2012, deep convolutional networks have been regarded as essential for this visual embedding step. Most VLP models employ an object detector pre-trained on the Visual Genome dataset (Krishna et al. 2017) annotated with 1,600 object classes and 400 attribute classes as in Anderson et al. 2018. Pixel-BERT (Huang et al. 2020) is one exception of this trend, as it uses ResNet variants (He et al. 2016; Xie et al. 2017) pre-trained on ImageNet classification (Russakovsky et al. 2015) embedding pixels in place of object detection modules.

要送进 VLP 模型，图像像素需要先和语言标记一样被嵌入成稠密形式。自 Krizhevsky 等 2012 以来，深度卷积网络一直被视为这一视觉嵌入步骤的必备部件。多数 VLP 模型使用在 Visual Genome 数据集（Krishna 等 2017）上预训练的目标检测器，该数据集聚了 1,600 个物体类别和 400 个属性类别，做法见 Anderson 等 2018。Pixel-BERT（Huang 等 2020）是这一趋势的一个例外：它用在 ImageNet 分类（Russakovsky 等 2015）上预训练的 ResNet 变体（He 等 2016；Xie 等 2017）来嵌入像素，而不使用目标检测模块。

To this date, most VLP studies have focused on improving performance by increasing the power of visual embedders. The shortcomings of having a heavy visual embedder are often disregarded in academic experiments because region features are commonly cached in advance at training time to ease the burden of feature extraction. However, the limitations are still evident in real-world applications as the queries in the wild have to undergo a slow extraction process.

迄今为止，多数 VLP 研究都把精力放在增强视觉嵌入器上，以求提高性能。沉重的视觉嵌入器在学术实验里往往不显眼，因为区域特征通常会在训练时预先缓存，以减轻特征提取负担。但在真实应用中，限制依然明显：野外查询必须经过一段缓慢的提取过程。

To this end, we shift our attention to the lightweight and fast embedding of visual inputs. Recent work (Dosovitskiy et al. 2020; Touvron et al. 2020) demonstrated that using a simple linear projection of a patch is effective enough to embed pixels before feeding them into transformers. Whereas being the solid mainstream for text (Devlin et al. 2019), it is only recently that transformers (Vaswani et al. 2017) are used for images as well. We presume that the transformer module– used for modality interaction in VLP models– can also manage to process visual features in place of a convolutional visual embedder, just as it processes textual features.

为此，我们把注意力转到视觉输入的轻量、快速嵌入。近期工作（Dosovitskiy 等 2020；Touvron 等 2020）表明，对图像块做一次简单的线性投影，就足以在把像素送进 Transformer 之前完成嵌入。这种方式在文本上早已是主流（Devlin 等 2019），用于图像则是最近的事。我们推测，VLP 模型里用来做模态交互的 Transformer 模块，也可以代替卷积视觉嵌入器来处理视觉特征，就像它处理文本特征那样。

This paper proposes the Vision-and-Language Transformer (ViLT) that handles two modalities in a single unified manner. It mainly differs from previous VLP models in its shallow, convolution-free embedding of pixel-level inputs. Removing deep embedders solely dedicated to visual inputs significantly cuts down the model size and running time by design. Figure 1 shows that our parameter-efficient model is tens of times faster than VLP models with region features and at least four times faster than those with grid features while exhibiting similar or even better performance on vision-and-language downstream tasks.

本文提出视觉–语言 Transformer（ViLT），用单一统一的方式处理两种模态。它与以往 VLP 模型的主要差别，在于对像素级输入采用浅层、无卷积的嵌入。去掉专门服务视觉输入的深层嵌入器，从设计上就显著减小了模型规模并缩短了运行时间。图 1 显示，这一参数更省的模型比使用区域特征的 VLP 模型快几十倍，也比使用网格特征的模型至少快四倍，同时在视觉–语言下游任务上表现相近，有时还更好。

Our key contributions can be summarized as follows:

本文的主要贡献可以概括如下：

  • • ViLT is the simplest architecture by far for a vision-and-language model as it commissions the transformer module to extract and process visual features in place of a separate deep visual embedder. This design inherently leads to significant runtime and parameter efficiency.
  • ViLT 是目前结构最简单的视觉–语言模型：它把提取和处理视觉特征的工作交给 Transformer 模块，而不再单独配备深层视觉嵌入器。这一设计本身就会带来显著的运行时间和参数效率。
  • • For the first time, we achieve competent performance on vision-and-language tasks without using region features or deep convolutional visual embedders in general.
  • 我们首次在不使用区域特征、也不使用深层卷积视觉嵌入器的情况下，在视觉–语言任务上达到有竞争力的表现。
  • • Also, for the first time, we empirically show that whole word masking and image augmentations that were unprecedented in VLP training schemes further drive downstream performance.
  • 我们也首次用实验表明，以往 VLP 训练方案里未见采用的整词掩码和图像增强，还能进一步推高下游性能。

2 Background

2 背景

2.1 Taxonomy of Vision-and-Language Models

2.1 视觉–语言模型的类型学

We propose a taxonomy of vision-and-language models based on two points: (1) whether the two modalities have an even level of expressiveness in terms of dedicated parameters and/or computation; and (2) whether the two modalities interact in a deep network. A combination of these points leads to four archetypes in Figure 2.

我们按两点给视觉–语言模型分类：（1）两种模态在专用参数和／或计算上的表达能力是否相当；（2）两种模态是否在深层网络中交互。这两点的组合对应图 2 中的四种原型。

The visual semantic embedding (VSE) models such as VSE++ (Faghri et al. 2017) and SCAN (Lee et al. 2018) belong to Figure 2(a). They use separate embedders for image and text, with the former being much heavier. Then, they represent the similarity of the embedded features from the two modalities with simple dot products or shallow attention layers.

视觉语义嵌入（VSE）模型，例如 VSE++（Faghri 等 2017）和 SCAN（Lee 等 2018），属于图 2(a)。它们为图像和文本使用分开的嵌入器，且前者要重得多。然后用简单的点积或浅层注意力来表示两种模态嵌入特征之间的相似度。

CLIP (Radford et al. 2021) belongs to Figure 2(b) as it uses separate but equally expensive transformer embedders for each modality. Interaction between the pooled image vector and text vector is still shallow (dot product). Despite CLIP’s remarkable zero-shot performance on image-to-text retrieval, we could not observe the same level of performance on other vision-and-language downstream tasks. For instance, fine-tuning the MLP head on NLVR2 (Suhr et al. 2018) with the dot product of pooled visual and textual vectors from CLIP as the multimodal representation gives a low dev accuracy of 50.99 ±\pm± 0.38 (ran with three different seeds); as chance level accuracy is 0.5, we conclude that the representations are incapable of learning this task. It also matches the findings of Suhr et al. 2018 that all models with simply fused multimodal representation failed to learn NLVR2.

CLIP（Radford 等 2021）属于图 2(b)：它为每种模态使用分开但同样昂贵的 Transformer 嵌入器。汇聚后的图像向量与文本向量之间的交互仍然很浅（点积）。尽管 CLIP 在图像到文本检索上的零样本表现出色，我们在其他视觉–语言下游任务上并没有看到同等水平。例如，用 CLIP 汇聚后的视觉与文本向量的点积作为多模态表示，再微调 NLVR2（Suhr 等 2018）上的 MLP 头，开发集准确率只有 50.99±0.3850.99 \pm 0.3850.99±0.38（三个不同随机种子）；随机水平是 0.5，因此可以认为这些表示学不会这项任务。这也与 Suhr 等 2018 的发现一致：所有只用简单融合多模态表示的模型都没能学会 NLVR2。

This result backs up our speculation that simple fusion of outputs even from high-performing unimodal embedders may not be sufficient to learn complex vision-and-language tasks, bolstering the need for a more rigorous inter-modal interaction scheme.

这一结果支持我们的判断：即便单模态嵌入器本身很强，只做简单融合也可能不足以学会复杂的视觉–语言任务，因而需要更严格的跨模态交互机制。

Unlike models with shallow interaction, the more recent VLP models that fall under Figure 2(c) use a deep transformer to model the interaction of image and text features. Aside from the interaction module, however, convolutional networks are still involved in extracting and embedding image features, which accounts for most of the computation as depicted in Figure 1. Modulation-based vision-and-language models (Perez et al. 2018; Nguyen et al. 2020) also fall under Figure 2(c), with their visual CNN stems corresponding to visual embedder, RNNs producing the modulation parameters to textual embedder, and modulated CNNs to modality interaction.

与浅层交互的模型不同，更新近的、属于图 2(c) 的 VLP 模型用深层 Transformer 来建模图像和文本特征的交互。可在交互模块之外，提取和嵌入图像特征时仍然用到卷积神经网络，而且如图 1 所示，这部分占了大部分计算。基于调制的视觉–语言模型（Perez 等 2018；Nguyen 等 2020）也属于图 2(c)：它们的视觉 CNN 主干对应视觉嵌入器，产生调制参数的 RNN 对应文本嵌入器，被调制的 CNN 则对应模态交互。

Our proposed ViLT is the first model of type Figure 2(d) where the embedding layers of raw pixels are shallow and computationally light as of text tokens. This architecture thereby concentrates most of the computation on modeling modality interactions.

我们提出的 ViLT 是图 2(d) 类型的第一个模型：原始像素的嵌入层很浅，计算量也和文本标记相当。这样，大部分计算就可以集中在建模模态交互上。

2.2 Modality Interaction Schema

2.2 模态交互方式

At the very core of contemporary VLP models lie transformers. They get visual and textual embedding sequences as input, model inter-modal and optionally intra-modal interactions throughout layers, then output a contextualized feature sequence.

当代 VLP 模型的核心是 Transformer。它们接收视觉和文本嵌入序列，在各层中建模跨模态交互，并可选地建模模态内交互，然后输出上下文化的特征序列。

Bugliarello et al. 2020 classifies interaction schema into two categories: (1) single-stream approaches (e.g., VisualBERT (Li et al. 2019), UNITER (Chen et al. 2019)) where layers collectively operate on a concatenation of image and text inputs; and (2) dual-stream approaches (e.g., ViLBERT (Lu et al. 2019), LXMERT (Tan & Bansal 2019)) where the two modalities are not concatenated at the input level. We follow the single-stream approach for our interaction transformer module because the dual-stream approach introduces additional parameters.

Bugliarello 等 2020 把交互方式分成两类：（1）单流方法（例如 VisualBERT（Li 等 2019）、UNITER（Chen 等 2019）），各层共同作用于图像和文本输入的拼接；（2）双流方法（例如 ViLBERT（Lu 等 2019）、LXMERT（Tan 与 Bansal 2019）），两种模态在输入阶段并不拼接。我们的交互 Transformer 模块采用单流方法，因为双流方法会引入额外参数。

Figure 3: Model overview. Illustration inspired by Dosovitskiy et al. 2020.

图 3：模型概览。示意图受 Dosovitskiy 等 2020 启发。

2.3 Visual Embedding Schema

2.3 视觉嵌入方式

Whereas all performant VLP models share the same textual embedder– tokenizer from pre-trained BERT, word and position embeddings resembling those of BERT– they differ on visual embedders. Still, in most (if not all) cases, visual embedding is the bottleneck of existing VLP models. We focus on cutting corners on this step by introducing patch projection instead of using region or grid features for which heavy extraction modules are used.

表现好的 VLP 模型在文本嵌入器上是统一的——都用预训练 BERT 的分词器，以及类似 BERT 的词嵌入和位置嵌入——差别主要在视觉嵌入器。而且在绝大多数情况下，视觉嵌入仍是现有 VLP 模型的瓶颈。我们把这一步尽量简化，改用图像块投影，而不再使用需要沉重提取模块的区域特征或网格特征。

Region Feature.

VLP models dominantly utilize region features, also known as bottom-up features (Anderson et al. 2018). They are obtained from an off-the-shelf object detector like Faster R-CNN (Ren et al. 2016).

区域特征

VLP 模型主要使用区域特征，也称为自下而上特征（Anderson 等 2018）。它们来自现成的目标检测器，例如 Faster R-CNN（Ren 等 2016）。

The general pipeline of generating region features is as follows. First, a region proposal network (RPN) proposes regions of interest (RoI) based on the grid features pooled from the CNN backbone. Non-maximum suppression (NMS) then reduces the number of RoIs to a few thousand. After being pooled by operations such as RoI Align (He et al. 2017), the RoIs go through RoI heads and become region features. NMS is again applied to every class, finally reducing the number of features under a hundred.

生成区域特征的一般流程如下。首先，区域提议网络（RPN）根据从 CNN 主干汇聚得到的网格特征提出感兴趣区域（RoI）。非极大值抑制（NMS）再把 RoI 数量降到几千。经过 RoI Align（He 等 2017）一类汇聚操作后，这些 RoI 进入 RoI 头，变成区域特征。然后再对每个类别做一次 NMS，最终把特征数量压到一百以下。

The above process involves several factors that affect the performance and runtime: the backbone, the style of NMS, the RoI heads. Previous works were lenient with controlling these factors, making varying choices from each other as listed in Table 7.[^2]

上述过程涉及若干影响性能和运行时间的因素：主干网络、NMS 的方式、RoI 头。以往工作对这些因素控制得并不严格，彼此选择各不相同，见表 7。

  • • Backbone: ResNet-101 (Lu et al. 2019; Tan & Bansal 2019; Su et al. 2019) and ResNext-152 (Li et al. 2019; Li et al. 2020a; Zhang et al. 2021) are two commonly used backbones.
  • 主干：常用的两种是 ResNet-101（Lu 等 2019；Tan 与 Bansal 2019；Su 等 2019）和 ResNeXt-152（Li 等 2019；Li 等 2020a；Zhang 等 2021）。
  • • NMS: NMS is typically done in a per-class fashion. Applying NMS to each and every class becomes a major runtime bottleneck with a large number of classes, e.g. 1.6K in the VG dataset (Jiang et al. 2020). Class-agnostic NMS was recently introduced to tackle this issue (Zhang et al. 2021).
  • NMS：通常按类别分别进行。类别很多时，对每一个类别都做 NMS 会成为主要的运行时间瓶颈，例如 VG 数据集中的 1.6K 类（Jiang 等 2020）。最近有工作引入类别无关的 NMS 来应对这一问题（Zhang 等 2021）。
  • • RoI head: C4 heads were initially used (Anderson et al. 2018). FPN-MLP heads were introduced later (Jiang et al. 2018). As heads operate for each and every RoI, they pose a substantial runtime burden.
  • RoI 头：最初使用 C4 头（Anderson 等 2018），后来出现了 FPN-MLP 头（Jiang 等 2018）。头要对每一个 RoI 运行，会带来可观的运行时间负担。

However lightweight, object detectors are less likely to be faster than the backbone or a single-layer convolution. Freezing the visual backbone and caching the region features in advance only helps at training time and not during inference, not to mention that it could hold performance back.

即便尽量做轻，目标检测器也很难比主干或单层卷积更快。冻结视觉主干并预先缓存区域特征，只对训练有帮助，推理时仍然慢，更可能拖累性能。

Grid Feature.

Besides detector heads, the output feature grid of convolutional neural networks such as ResNets can also be used as visual features for vision-and-language pre-training. Direct use of grid features was first proposed by VQA-specific models (Jiang et al. 2020; Nguyen et al. 2020), mainly to avoid using severely slow region selection operations.

网格特征

除了检测头，ResNet 一类卷积神经网络输出的特征网格也可以当作视觉–语言预训练的视觉特征。直接使用网格特征最早由面向 VQA 的模型提出（Jiang 等 2020；Nguyen 等 2020），主要是为了避开极慢的区域选择操作。

X-LXMERT (Cho et al. 2020) revisited grid features by fixing the region proposals to grids instead of those from the region proposal networks. However, their caching of features excluded further tuning of the backbone.

X-LXMERT（Cho 等 2020）重新审视了网格特征：把区域提议固定成网格，而不再用区域提议网络给出的框。不过他们缓存特征时，没有再继续微调主干。

Pixel-BERT is the only VLP model that replaces the VG-pre-trained object detector with a ResNet variant backbone pre-trained with ImageNet classification. Unlike frozen detectors in region-feature-based VLP models, the backbone of Pixel-BERT is tuned during vision-and-language pre-training. The downstream performance of Pixel-BERT with ResNet-50 falls below region-feature-based VLP models, but it matches that of other competitors with the use of a much heavier ResNeXt-152.

Pixel-BERT 是唯一用在 ImageNet 分类上预训练的 ResNet 变体主干，来替换在 VG 上预训练的目标检测器的 VLP 模型。与基于区域特征的 VLP 模型中被冻结的检测器不同，Pixel-BERT 的主干会在视觉–语言预训练中一起调整。使用 ResNet-50 时，Pixel-BERT 的下游表现低于基于区域特征的 VLP 模型；换上重得多的 ResNeXt-152 后，才和其他对手打平。

We claim that grid features are not the go-to option, however, since deep CNNs are still expensive that they account for a large portion of the whole computation as in Figure 1.

不过我们认为网格特征也不是首选，因为深层 CNN 仍然昂贵，正如图 1 所示，它们仍占整体计算的很大一部分。

Patch Projection.

To minimize overhead, we adopt the simplest visual embedding scheme: linear projection that operates on image patches. The patch projection embedding was introduced by ViT (Dosovitskiy et al. 2020) for image classification tasks. Patch projection drastically simplifies the visual embedding step to the level of textual embedding, which also consists of simple projection (lookup) operations.

图像块投影

为把额外开销降到最低，我们采用最简单的视觉嵌入方案：作用在图像块上的线性投影。图像块投影嵌入由 ViT（Dosovitskiy 等 2020）引入，用于图像分类。它把视觉嵌入步骤大幅简化到与文本嵌入相当的程度，而文本嵌入同样只是简单的投影（查表）操作。

We use a 32 ×\times× 32 patch projection which only requires 2.4M parameters. This is in sharp contrast to complex ResNe(X)t backbones[^3] and detection components. Its running time is also ignorable as shown in Figure 1. We make a detailed runtime analysis in Section 4.6.

我们使用 32×3232 \times 3232×32 的图像块投影，只需 240 万参数。这与复杂的 ResNe(X)t 主干以及检测组件形成鲜明对比。其运行时间也几乎可以忽略，如图 1 所示。我们在第 4.6 节给出详细的运行时间分析。

3 Vision-and-Language Transformer

3 视觉–语言 Transformer

3.1 Model Overview

3.1 模型概览

ViLT has a succinct architecture as a VLP model with a minimal visual embedding pipeline and following the single-stream approach.

作为 VLP 模型，ViLT 的结构很简洁：视觉嵌入流水线尽量精简，并采用单流方法。

We deviate from the literature that we initialize the interaction transformer weights from pre-trained ViT instead of BERT. Such initialization exploits the power of the interaction layers to process visual features while lacking a separate deep visual embedder. [^4]

与文献中的常见做法不同，我们用预训练 ViT 而不是 BERT 来初始化交互 Transformer 的权重。这样初始化，是为了在没有单独深层视觉嵌入器的情况下，借助交互层本身的能力来处理视觉特征。

tˉ=[tclass;t1T;⋯ ;tLT]+Tposvˉ=[vclass;v1V;⋯ ;vNV]+Vposz0=[tˉ+ttype;vˉ+vtype]z^d=MSA(LN(zd−1))+zd−1,d=1…Dzd=MLP(LN(z^d))+z^d,d=1…Dp=tanh(z0DWpool)(1)\displaystyle\bar{t} \\ \displaystyle=[t_{\text{class}};t_{1}T;\cdots;t_{L}T]+T^{\text{pos}} \\ \displaystyle\bar{v} \\ \displaystyle=[v_{\text{class}};v_{1}V;\cdots;v_{N}V]+V^{\text{pos}} \\ \displaystyle z^{0} \\ \displaystyle=[\bar{t}+t^{\text{type}};\bar{v}+v^{\text{type}}] \\ \displaystyle\hat{z}^{d} \\ \displaystyle=\text{MSA}(\text{LN}(z^{d-1}))+z^{d-1}, \\ \displaystyle d=1\ldots D \\ \displaystyle z^{d} \\ \displaystyle=\text{MLP}(\text{LN}(\hat{z}^{d}))+\hat{z}^{d}, \\ \displaystyle d=1\ldots D \\ \displaystyle p \\ \displaystyle=\text{tanh}(z^{D}_{0}W_{\text{pool}}) \tag{1}tˉ=[tclass​;t1​T;⋯;tL​T]+Tposvˉ=[vclass​;v1​V;⋯;vN​V]+Vposz0=[tˉ+ttype;vˉ+vtype]z^d=MSA(LN(zd−1))+zd−1,d=1…Dzd=MLP(LN(z^d))+z^d,d=1…Dp=tanh(z0D​Wpool​)(1)

ViT consists of stacked blocks that include a multiheaded self-attention (MSA) layer and an MLP layer. The position of layer normalization (LN) in ViT is the only difference from BERT: LN comes after MSA and MLP in BERT (“post-norm”) and before in ViT (“pre-norm”). The input text t∈RL×∣V∣t\in\mathbb{R}^{L\times|V|}t∈RL×∣V∣ is embedded to tˉ∈RL×H\bar{t}\in\mathbb{R}^{L\times H}tˉ∈RL×H with a word embedding matrix T∈R∣V∣×HT\in\mathbb{R}^{|V|\times H}T∈R∣V∣×H and a position embedding matrix Tpos∈R(L+1)×HT^{\text{pos}}\in\mathbb{R}^{(L+1)\times H}Tpos∈R(L+1)×H.

ViT 由堆叠的块组成，每个块包含一个多头自注意力（MSA）层和一个 MLP 层。ViT 与 BERT 的唯一差别是层归一化（LN）的位置：BERT 把 LN 放在 MSA 和 MLP 之后（“后归一化”），ViT 则放在之前（“前归一化”）。输入文本 t∈RL×∣V∣t\in\mathbb{R}^{L\times|V|}t∈RL×∣V∣ 通过词嵌入矩阵 T∈R∣V∣×HT\in\mathbb{R}^{|V|\times H}T∈R∣V∣×H 和位置嵌入矩阵 Tpos∈R(L+1)×HT^{\text{pos}}\in\mathbb{R}^{(L+1)\times H}Tpos∈R(L+1)×H 嵌入为 tˉ∈RL×H\bar{t}\in\mathbb{R}^{L\times H}tˉ∈RL×H。

The input image I∈RC×H×WI\in\mathbb{R}^{C\times H\times W}I∈RC×H×W is sliced into patches and flattened to v∈RN×(P2⋅C)v\in\mathbb{R}^{N\times(P^{2}\cdot C)}v∈RN×(P2⋅C) where (P,P)(P,P)(P,P) is the patch resolution and N=HW/P2N=HW/P^{2}N=HW/P2. Followed by linear projection V∈R(P2⋅C)×HV\in\mathbb{R}^{(P^{2}\cdot C)\times H}V∈R(P2⋅C)×H and position embedding Vpos∈R(N+1)×HV^{\text{pos}}\in\mathbb{R}^{(N+1)\times H}Vpos∈R(N+1)×H, vvv is embedded into vˉ∈RN×H\bar{v}\in\mathbb{R}^{N\times H}vˉ∈RN×H.

输入图像 I∈RC×H×WI\in\mathbb{R}^{C\times H\times W}I∈RC×H×W 被切成图像块并展平为 v∈RN×(P2⋅C)v\in\mathbb{R}^{N\times(P^{2}\cdot C)}v∈RN×(P2⋅C)，其中 (P,P)(P,P)(P,P) 是块的分辨率，N=HW/P2N=HW/P^{2}N=HW/P2。再经过线性投影 V∈R(P2⋅C)×HV\in\mathbb{R}^{(P^{2}\cdot C)\times H}V∈R(P2⋅C)×H 和位置嵌入 Vpos∈R(N+1)×HV^{\text{pos}}\in\mathbb{R}^{(N+1)\times H}Vpos∈R(N+1)×H，vvv 被嵌入为 vˉ∈RN×H\bar{v}\in\mathbb{R}^{N\times H}vˉ∈RN×H。

The text and image embeddings are summed with their corresponding modal-type embedding vectors ttype,vtype∈RHt^{\text{type}},v^{\text{type}}\in\mathbb{R}^{H}ttype,vtype∈RH, then are concatenated into a combined sequence z0z^{0}z0. The contextualized vector zzz is iteratively updated through DDD-depth transformer layers up until the final contextualized sequence zDz^{D}zD. ppp is a pooled representation of the whole multimodal input, and is obtained by applying linear projection Wpool∈RH×HW_{\text{pool}}\in\mathbb{R}^{H\times H}Wpool​∈RH×H and hyperbolic tangent upon the first index of sequence zDz^{D}zD.

文本和图像嵌入分别加上对应的模态类型嵌入向量 ttype,vtype∈RHt^{\text{type}},v^{\text{type}}\in\mathbb{R}^{H}ttype,vtype∈RH，再拼接成组合序列 z0z^{0}z0。上下文化向量 zzz 经过 DDD 层 Transformer 迭代更新，直到最终的上下文化序列 zDz^{D}zD。ppp 是整个多模态输入的汇聚表示，由线性投影 Wpool∈RH×HW_{\text{pool}}\in\mathbb{R}^{H\times H}Wpool​∈RH×H 和双曲正切作用在序列 zDz^{D}zD 的第一个位置上得到。

For all experiments, we use weights from ViT-B/32 pre-trained on ImageNet, hence the name ViLT-B/32.[^5] Hidden size HHH is 768, layer depth DDD is 12, patch size PPP is 32, MLP size is 3,072, and the number of attention heads is 12.

全部实验都使用在 ImageNet 上预训练的 ViT-B/32 权重，因此模型名为 ViLT-B/32。隐藏维度 HHH 为 768，层数 DDD 为 12，图像块大小 PPP 为 32，MLP 宽度为 3,072，注意力头数为 12。

3.2 Pre-training Objectives

3.2 预训练目标

We train ViLT with two objectives commonly used to train VLP models: image text matching (ITM) and masked language modeling (MLM).

我们用训练 VLP 模型时常用的两个目标来训练 ViLT：图像–文本匹配（ITM）和掩码语言建模（MLM）。

Image Text Matching.

We randomly replace the aligned image with a different image with the probability of 0.5. A single linear layer ITM head projects the pooled output feature ppp to logits over binary class, and we compute negative log-likelihood loss as our ITM loss.

图像–文本匹配

我们以 0.5 的概率把对齐的图像随机替换成另一张图像。一层线性的 ITM 头把汇聚输出特征 ppp 投影成二分类的 logits，ITM 损失取负对数似然。

Plus, inspired by the word region alignment objective in Chen et al. 2019, we design word patch alignment (WPA) that computes the alignment score between two subsets of zDz^{D}zD: zD∣tz^{D}|_{t}zD∣t​ (textual subset) and zD∣vz^{D}|_{v}zD∣v​ (visual subset), using the inexact proximal point method for optimal transports (IPOT) (Xie et al. 2020). We set the hyperparameters of IPOT following Chen et al. 2019 (β=0.5,N=50\beta=0.5,N=50β=0.5,N=50), and add the approximate wasserstein distance multiplied by 0.1 to the ITM loss.

此外，受 Chen 等 2019 中词–区域对齐目标的启发，我们设计了词–图像块对齐（WPA）：用非精确近端点最优传输方法（IPOT）（Xie 等 2020），计算 zDz^{D}zD 的两个子集——文本子集 zD∣tz^{D}|_{t}zD∣t​ 和视觉子集 zD∣vz^{D}|_{v}zD∣v​——之间的对齐分数。IPOT 的超参数沿用 Chen 等 2019（β=0.5,N=50\beta=0.5,N=50β=0.5,N=50），并把近似 Wasserstein 距离乘以 0.1 后加到 ITM 损失上。

Masked Language Modeling.

This objective is to predict the ground truth labels of masked text tokens tmaskedt_{\text{masked}}tmasked​ from its contextualized vector zmaskedD∣tz^{D}_{\text{masked}}|_{t}zmaskedD​∣t​. Following the heuristics of Devlin et al. 2019, we randomly mask ttt with the probability of 0.15.

掩码语言建模

这一目标是根据被掩码文本标记 tmaskedt_{\text{masked}}tmasked​ 的上下文化向量 zmaskedD∣tz^{D}_{\text{masked}}|_{t}zmaskedD​∣t​，预测其真值标签。遵循 Devlin 等 2019 的做法，我们以 0.15 的概率随机掩码 ttt。

We use a two-layer MLP MLM head that inputs zmaskedD∣tz^{D}_{\text{masked}}|_{t}zmaskedD​∣t​ and outputs logits over vocabulary, just as the MLM objective of BERT. The MLM loss is then computed as the negative log-likelihood loss for the masked tokens.

我们使用两层 MLP 作为 MLM 头，输入 zmaskedD∣tz^{D}_{\text{masked}}|_{t}zmaskedD​∣t​，输出词表上的 logits，与 BERT 的 MLM 目标相同。MLM 损失是这些被掩码标记上的负对数似然。

3.3 Whole Word Masking

3.3 整词掩码

Whole word masking is a masking technique that masks all consecutive subword tokens that compose a whole word. It is shown to be effective on downstream tasks when applied to original and Chinese BERT (Cui et al. 2019).

整词掩码是一种掩码技术：组成一个整词的连续子词标记会全部被掩住。它在原版 BERT 和中文 BERT 上被证明对下游任务有效（Cui 等 2019）。

We hypothesize that whole word masking is particularly crucial for VLP in order to make full use of information from the other modality. For example, the word “giraffe” is tokenized into three wordpiece tokens ["gi", "##raf", "##fe"] with the pre-trained bert-base-uncased tokenizer. If not all tokens are masked, say, ["gi", "[MASK]", "##fe"], the model may solely rely on the nearby two language tokens ["gi", "##fe"] to predict the masked "##raf" rather than using the information from the image.

我们推测，整词掩码对 VLP 尤其关键，这样才能充分利用另一模态的信息。例如，单词 “giraffe” 经预训练的 bert-base-uncased 分词器会变成三个 wordpiece 标记 ["gi", "##raf", "##fe"]。如果没有把它们全部掩住，比如说变成 ["gi", "[MASK]", "##fe"]，模型可能只靠邻近的两个语言标记 ["gi", "##fe"] 去预测被掩住的 "##raf"，而不去使用图像中的信息。

We mask whole words with a mask probability of 0.15 during pre-training. We discuss its impact in Section 4.5.

预训练时我们以 0.15 的掩码概率做整词掩码。其影响在第 4.5 节讨论。

3.4 Image Augmentation

3.4 图像增强

Image augmentation reportedly improves the generalization power of vision models (Shorten & Khoshgoftaar 2019). DeiT (Touvron et al. 2020) that builds on ViT experimented with various augmentation techniques (Zhang et al. 2017; Yun et al. 2019; Berman et al. 2019; Hoffer et al. 2020; Cubuk et al. 2020), and found them beneficial for ViT training. However, the effects of image augmentation have not been explored within VLP models. Caching visual features restrains region-feature-based VLP models from using image augmentation. Notwithstanding its applicability, neither did Pixel-BERT study its effects.

据报道，图像增强能提高视觉模型的泛化能力（Shorten 与 Khoshgoftaar 2019）。建立在 ViT 之上的 DeiT（Touvron 等 2020）试验了多种增强技术（Zhang 等 2017；Yun 等 2019；Berman 等 2019；Hoffer 等 2020；Cubuk 等 2020），并发现它们对 ViT 训练有帮助。但图像增强的效果在 VLP 模型中尚未被探讨。缓存视觉特征会妨碍基于区域特征的 VLP 模型使用图像增强。Pixel-BERT 虽然用得上增强，也没有研究其效果。

To this end, we apply RandAugment (Cubuk et al. 2020) during fine-tuning. We use all the original policies except two: color inversion, because texts often contain color information as well, and cutout, as it may clear out small but important objects dispersed throughout the whole image. We use N=2,M=9N=2,M=9N=2,M=9 as the hyperparameters. We discuss its impact in Section 4.5 and Section 5.

为此，我们在微调时使用 RandAugment（Cubuk 等 2020）。原有策略里我们去掉两项：颜色反转，因为文本里也常常包含颜色信息；以及 cutout，因为它可能抹掉分散在整张图里的小而关键的物体。超参数取 N=2,M=9N=2,M=9N=2,M=9。其影响在第 4.5 节和第 5 节讨论。

4 Experiments

4 实验

4.1 Overview

4.1 概览

Dataset # Images # Captions Caption Length
MSCOCO 113K 567K 11.81 ±\pm± 2.81
VG 108K 5.41M 5.53 ±\pm± 1.76
GCC† 3.01M 3.01M 10.66 ±\pm± 4.93
SBU† 867K 867K 15.0 ±\pm± 7.74

Table 1: Pre-training dataset statistics. Caption length is the length of tokens from pre-trained bert-base-uncased tokenizer. † GCC and SBU provide only image urls, so we collect the images from urls which were still accessible.

数据集 图像数 描述数 描述长度
MSCOCO 113K 567K 11.81 ±\pm± 2.81
VG 108K 5.41M 5.53 ±\pm± 1.76
GCC† 3.01M 3.01M 10.66 ±\pm± 4.93
SBU† 867K 867K 15.0 ±\pm± 7.74

表 1：预训练数据集统计。描述长度是预训练 bert-base-uncased 分词器给出的标记长度。† GCC 和 SBU 只提供图像网址，我们收集了当时仍可访问的网址对应的图像。

We use four datasets for pre-training: Microsoft COCO (MSCOCO) (Lin et al. 2014), Visual Genome (VG) (Krishna et al. 2017), SBU Captions (SBU) (Ordonez et al. 2011), and Google Conceptual Captions (GCC) (Sharma et al. 2018). Table 1 reports the dataset statistics.

预训练使用四个数据集：Microsoft COCO（MSCOCO）（Lin 等 2014）、Visual Genome（VG）（Krishna 等 2017）、SBU Captions（SBU）（Ordonez 等 2011）和 Google Conceptual Captions（GCC）（Sharma 等 2018）。表 1 报告了数据集统计。

We evaluate ViLT on two widely explored types of vision-and-language downstream tasks: for classification, we use VQAv2 (Goyal et al. 2017) and NLVR2 (Suhr et al. 2018), and for retrieval, we use MSCOCO and Flickr30K (F30K) (Plummer et al. 2015) re-splited by Karpathy & Fei-Fei 2015. For the classification tasks, we fine-tune three times with different initialization seeds for the head and data ordering and report the mean scores. We report the standard deviation in Table 5 along with ablation studies. For the retrieval tasks, we only fine-tune once.

我们在两类广为研究的视觉–语言下游任务上评估 ViLT：分类任务使用 VQAv2（Goyal 等 2017）和 NLVR2（Suhr 等 2018），检索任务使用由 Karpathy 与 Fei-Fei 2015 重新划分的 MSCOCO 和 Flickr30K（F30K）（Plummer 等 2015）。分类任务上，我们用不同的头初始化种子和数据顺序微调三次，报告平均分数。标准差与消融实验一起列在表 5。检索任务只微调一次。

4.2 Implementation Details

4.2 实现细节

For all experiments, we use AdamW optimizer (Loshchilov & Hutter 2018) with base learning rate of 10−410^{-4}10−4 and weight decay of 10−210^{-2}10−2. The learning rate was warmed up for 10% of the total training steps and was decayed linearly to zero for the rest of the training. Note that downstream performance may be further improved if we customize the hyperparameters to each task.

全部实验使用 AdamW 优化器（Loshchilov 与 Hutter 2018），基础学习率为 10−410^{-4}10−4，权重衰减为 10−210^{-2}10−2。学习率在总训练步数的前 10% 做预热，其余时间线性衰减到零。需要说明，如果为每个任务单独调整超参数，下游表现还有进一步提高的空间。

We resize the shorter edge of input images to 384 and limit the longer edge to under 640 while preserving the aspect ratio. This resizing scheme is also used during object detection in other VLP models, but with a larger size of the shorter edge (800). Patch projection of ViLT-B/32 yields 12 ×\times× 20 = 240 patches for an image with a resolution of 384 ×\times× 640. As this is a rarely reached upper limit, we sample 200 patches at maximum during pre-training. We interpolate VposV^{\text{pos}}Vpos of ViT-B/32 to fit the size of each image and pad the patches for batch training. Note that the resulting image resolution is four times smaller than 800 ×\times× 1,333, which is the size that all other VLP models use for inputs to their visual embedders.

我们把输入图像的短边缩放到 384，长边限制在 640 以内，并保持宽高比。其他 VLP 模型在做目标检测时也使用这种缩放方式，只是短边更大（800）。ViLT-B/32 的图像块投影对分辨率为 384×640384 \times 640384×640 的图像会产生 12×20=24012 \times 20 = 24012×20=240 个块。这是很少达到的上限，因此预训练时我们最多采样 200 个块。我们把 ViT-B/32 的 VposV^{\text{pos}}Vpos 插值到每张图像的尺寸，并在批量训练时对块做填充。需要注意，由此得到的图像分辨率比 800×1,333800 \times 1{,}333800×1,333 小四倍，而后者是其他所有 VLP 模型送进视觉嵌入器的输入尺寸。

We use the bert-base-uncased tokenizer to tokenize text inputs. Instead of fine-tuning from pre-trained BERT, we learn the textual embedding-related parameters tclasst_{\text{class}}tclass​, TTT, and TposT^{\text{pos}}Tpos from scratch. Although beneficial prima facie, employing a pre-trained text-only BERT does not guarantee performance gain for vision and language downstream tasks. Counterevidence has already been reported by Tan & Bansal 2019, where initializing with pre-trained BERT parameters led to weaker performance than pre-training from scratch.

文本输入使用 bert-base-uncased 分词器。我们不从预训练 BERT 微调，而是从头学习与文本嵌入相关的参数 tclasst_{\text{class}}tclass​、TTT 和 TposT^{\text{pos}}Tpos。表面上这像是吃亏，但采用只在文本上预训练的 BERT 并不保证视觉–语言下游任务一定更好。Tan 与 Bansal 2019 已经报告过反例：用预训练 BERT 参数初始化，反而比从头预训练更弱。

We pre-train ViLT-B/32 for 100K or 200K steps on 64 NVIDIA V100 GPUs with a batch size of 4,096. For all downstream tasks, we train for ten epochs with a batch size of 256 for VQAv2/retrieval tasks and 128 for NLVR2.

我们在 64 块 NVIDIA V100 GPU 上，以批量大小 4,096 将 ViLT-B/32 预训练 100K 或 200K 步。所有下游任务训练 10 个 epoch，VQAv2／检索任务的批量大小为 256，NLVR2 为 128。

4.3 Classification Tasks

4.3 分类任务

Visual Embed Model Time (ms) VQAv2 NLVR2
test-dev dev test-P
Region w/o VLP SOTA ~900 70.63 54.80 53.50
ViLBERT ~920 70.55 - -
VisualBERT ~925 70.80 67.40 67.00
LXMERT ~900 72.42 74.90 74.50
UNITER-Base ~900 72.70 75.85 75.80
OSCAR-Base† ~900 73.16 78.07 78.36
VinVL-Base†‡ ~650 75.95 82.05 83.08
Grid Pixel-BERT-X152 ~160 74.45 76.50 77.20
Pixel-BERT-R50 ~60 71.35 71.70 72.40
Linear ViLT-B/32 ~15 70.33 74.41 74.57
ViLT-B/32ⓐ ~15 70.85 74.91 75.57
ViLT-B/32ⓐ+⃝ ~15 71.26 75.70 76.13

Table 2: Comparison of ViLT-B/32 with other models on downstream classification tasks. We use MCAN (Yu et al. 2019) and MaxEnt (Suhr et al. 2018) for VQAv2 and NLVR2 w/o VLP SOTA results. † additionally used GQA, VQAv2, VG-QA for pre-training. ‡ made additional use of the Open Images (Kuznetsova et al. 2020) dataset. ⓐ indicates RandAugment is applied during fine-tuning. +⃝ indicates model trained for a longer 200K pre-training steps.

视觉嵌入 模型 时间 (ms) VQAv2 NLVR2
test-dev dev test-P
区域 无 VLP 的当时最好结果 ~900 70.63 54.80 53.50
ViLBERT ~920 70.55 - -
VisualBERT ~925 70.80 67.40 67.00
LXMERT ~900 72.42 74.90 74.50
UNITER-Base ~900 72.70 75.85 75.80
OSCAR-Base† ~900 73.16 78.07 78.36
VinVL-Base†‡ ~650 75.95 82.05 83.08
网格 Pixel-BERT-X152 ~160 74.45 76.50 77.20
Pixel-BERT-R50 ~60 71.35 71.70 72.40
线性 ViLT-B/32 ~15 70.33 74.41 74.57
ViLT-B/32ⓐ ~15 70.85 74.91 75.57
ViLT-B/32ⓐ+⃝ ~15 71.26 75.70 76.13

表 2：ViLT-B/32 与其他模型在下游分类任务上的比较。无 VLP 的当时最好结果中，VQAv2 使用 MCAN（Yu 等 2019），NLVR2 使用 MaxEnt（Suhr 等 2018）。† 额外使用了 GQA、VQAv2、VG-QA 做预训练。‡ 额外使用了 Open Images（Kuznetsova 等 2020）数据集。ⓐ 表示微调时使用了 RandAugment。+⃝ 表示模型预训练了更长的 200K 步。

Visual Embed Model Time (ms) Zero-Shot Text Retrieval Zero-Shot Image Retrieval
Flickr30k (1K) MSCOCO (5K) Flickr30k (1K) MSCOCO (5K)
R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10
Region ViLBERT ~900 - - - - - - 31.9 61.1 72.8 - - -
Unicoder-VL ~925 64.3 85.8 92.3 - - - 48.4 76.0 85.2 - - -
UNITER-Base ~900 80.7 95.7 98.0 - - - 66.2 88.4 92.9 - - -
ImageBERT† ~925 70.7 90.2 94.0 44.0 71.2 80.4 54.3 79.6 87.5 32.3 59.0 70.2
Linear ViLT-B/32 ~15 69.7 91.0 96.0 53.4 80.7 88.8 51.3 79.9 87.9 37.3 67.4 79.0
ViLT-B/32+⃝ ~15 73.2 93.6 96.5 56.5 82.6 89.6 55.0 82.5 89.8 40.4 70.0 81.1

Table 3: Comparison of ViLT-B/32 with other VLP models on downstream zero-shot retrieval tasks. We exclude the models of which zero-shot retrieval performances were not reported in their original papers. † is pre-trained with a 10M proprietary vision-and-language dataset in addition to the 4M dataset of GCC+SBU. +⃝ indicates model trained for a longer 200K pre-training steps.

视觉嵌入 模型 时间 (ms) 零样本文本检索 零样本图像检索
Flickr30k (1K) MSCOCO (5K) Flickr30k (1K) MSCOCO (5K)
R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10
区域 ViLBERT ~900 - - - - - - 31.9 61.1 72.8 - - -
Unicoder-VL ~925 64.3 85.8 92.3 - - - 48.4 76.0 85.2 - - -
UNITER-Base ~900 80.7 95.7 98.0 - - - 66.2 88.4 92.9 - - -
ImageBERT† ~925 70.7 90.2 94.0 44.0 71.2 80.4 54.3 79.6 87.5 32.3 59.0 70.2
线性 ViLT-B/32 ~15 69.7 91.0 96.0 53.4 80.7 88.8 51.3 79.9 87.9 37.3 67.4 79.0
ViLT-B/32+⃝ ~15 73.2 93.6 96.5 56.5 82.6 89.6 55.0 82.5 89.8 40.4 70.0 81.1

表 3：ViLT-B/32 与其他 VLP 模型在下游零样本检索任务上的比较。原论文未报告零样本检索结果的模型未列入。† 除 GCC+SBU 的 4M 数据集外，还在一个 10M 的专有视觉–语言数据集上预训练。+⃝ 表示模型预训练了更长的 200K 步。

Visual Embed Model Time (ms) Text Retrieval Image Retrieval
Flickr30k (1K) MSCOCO (5K) Flickr30k (1K) MSCOCO (5K)
R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10
Region w/o VLP SOTA ~900 67.4 90.3 95.8 50.4 82.2 90.0 48.6 77.7 85.2 38.6 69.3 80.4
ViLBERT-Base ~920 - - - - - - 58.2 84.9 91.5 - - -
Unicoder-VL ~925 86.2 96.3 99.0 62.3 87.1 92.8 71.5 91.2 95.2 48.4 76.7 85.9
UNITER-Base ~900 85.9 97.1 98.8 64.4 87.4 93.1 72.5 92.4 96.1 50.3 78.5 87.2
OSCAR-Base† ~900 - - - 70.0 91.1 95.5 - - - 54.0 80.8 88.5
VinVL-Base†‡ ~650 - - - 74.6 92.6 96.3 - - - 58.1 83.2 90.1
Grid Pixel-BERT-X152 ~160 87.0 98.9 99.5 63.6 87.5 93.6 71.5 92.1 95.8 50.1 77.6 86.2
Pixel-BERT-R50 ~60 75.7 94.7 97.1 59.8 85.5 91.6 53.4 80.4 88.5 41.1 69.7 80.5
Linear ViLT-B/32 ~15 81.4 95.6 97.6 61.8 86.2 92.6 61.9 86.8 92.8 41.3 72.0 82.5
ViLT-B/32ⓐ ~15 83.7 97.2 98.1 62.9 87.1 92.7 62.2 87.6 93.2 42.6 72.8 83.4
ViLT-B/32ⓐ+⃝ ~15 83.5 96.7 98.6 61.5 86.3 92.7 64.4 88.7 93.8 42.7 72.9 83.1

Table 4: Comparison of ViLT-B/32 with other models on downstream retrieval tasks. We use SCAN for w/o VLP SOTA results. † additionally used GQA, VQAv2, VG-QA for pre-training. ‡ additionally used the Open Images dataset. ⓐ indicates RandAugment is applied during fine-tuning. +⃝ indicates model trained for a longer 200K pre-training steps.

视觉嵌入 模型 时间 (ms) 文本检索 图像检索
Flickr30k (1K) MSCOCO (5K) Flickr30k (1K) MSCOCO (5K)
R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10
区域 无 VLP 的当时最好结果 ~900 67.4 90.3 95.8 50.4 82.2 90.0 48.6 77.7 85.2 38.6 69.3 80.4
ViLBERT-Base ~920 - - - - - - 58.2 84.9 91.5 - - -
Unicoder-VL ~925 86.2 96.3 99.0 62.3 87.1 92.8 71.5 91.2 95.2 48.4 76.7 85.9
UNITER-Base ~900 85.9 97.1 98.8 64.4 87.4 93.1 72.5 92.4 96.1 50.3 78.5 87.2
OSCAR-Base† ~900 - - - 70.0 91.1 95.5 - - - 54.0 80.8 88.5
VinVL-Base†‡ ~650 - - - 74.6 92.6 96.3 - - - 58.1 83.2 90.1
网格 Pixel-BERT-X152 ~160 87.0 98.9 99.5 63.6 87.5 93.6 71.5 92.1 95.8 50.1 77.6 86.2
Pixel-BERT-R50 ~60 75.7 94.7 97.1 59.8 85.5 91.6 53.4 80.4 88.5 41.1 69.7 80.5
线性 ViLT-B/32 ~15 81.4 95.6 97.6 61.8 86.2 92.6 61.9 86.8 92.8 41.3 72.0 82.5
ViLT-B/32ⓐ ~15 83.7 97.2 98.1 62.9 87.1 92.7 62.2 87.6 93.2 42.6 72.8 83.4
ViLT-B/32ⓐ+⃝ ~15 83.5 96.7 98.6 61.5 86.3 92.7 64.4 88.7 93.8 42.7 72.9 83.1

表 4：ViLT-B/32 与其他模型在下游检索任务上的比较。无 VLP 的当时最好结果使用 SCAN。† 额外使用了 GQA、VQAv2、VG-QA 做预训练。‡ 额外使用了 Open Images 数据集。ⓐ 表示微调时使用了 RandAugment。+⃝ 表示模型预训练了更长的 200K 步。

We evaluate ViLT-B/32 on two commonly used datasets: VQAv2 and NLVR2. We use a two-layer MLP of hidden size 1,536 as the fine-tuned downstream head.

我们在两个常用数据集上评估 ViLT-B/32：VQAv2 和 NLVR2。微调用的下游头是隐藏大小为 1,536 的两层 MLP。

Visual Question Answering.

The VQAv2 task asks for answers given pairs of an image and a question in natural language. The annotated answers are originally in free-form natural language, but it is a common practice to convert the task to a classification task with 3,129 answer classes. Following this practice, we fine-tune ViLT-B/32 on the VQAv2 train and validation sets while reserving 1,000 validation images and their related questions for internal validation.

视觉问答

VQAv2 任务要求根据一对图像和自然语言问题给出答案。标注答案原本是自由形式的自然语言，通行做法是把它转成有 3,129 个答案类别的分类任务。我们沿用这一做法，在 VQAv2 的训练集和验证集上微调 ViLT-B/32，同时留出 1,000 张验证图像及其相关问题做内部验证。

We report the test-dev score results[^6] from the submission to the evaluation server. ViLT falls short of VQA score compared to other VLP models with a heavy visual embedder. We suspect a detached object representation generated by the object detector eases the training of VQA since questions in VQA typically ask about objects.

我们报告提交到评估服务器后得到的 test-dev 分数。与带有沉重视觉嵌入器的其他 VLP 模型相比，ViLT 的 VQA 分数偏低。我们猜测，目标检测器给出的、已经分离好的物体表示会降低 VQA 的训练难度，因为 VQA 中的问题通常问的就是物体。

Natural Language for Visual Reasoning.

The NLVR2 task is a binary classification task given triplets of two images and a question in natural language. As there are two input images unlike the pre-training setup, multiple strategies exist[^7]. Following OSCAR (Li et al. 2020b) and VinVL (Zhang et al. 2021), we use the pair method. Here, the triplet input is reformulated into two pairs (question, image1) and (question, image2), and each pair goes through the ViLT. The head takes the concatenation of two pooled representations (ppp) as input and outputs the binary prediction.

视觉推理的自然语言

NLVR2 任务是给定两张图像和一个自然语言问题组成的三元组，做二分类。与预训练设定不同，这里有两张输入图像，因此存在多种策略。我们跟随 OSCAR（Li 等 2020b）和 VinVL（Zhang 等 2021），采用 pair 方法：把三元组改写成两个配对（问题，图像 1）和（问题，图像 2），每一对分别通过 ViLT。头接收两个汇聚表示（ppp）的拼接，并输出二分类预测。

Table 2 shows the results. ViLT-B/32 maintains competitive performance on both datasets considering its remarkable inference speed.

表 2 给出了结果。考虑到推理速度上的明显优势，ViLT-B/32 在两个数据集上都保持了有竞争力的表现。

4.4 Retrieval Tasks

4.4 检索任务

We fine-tune ViLT-B/32 on the Karpathy & Fei-Fei 2015 split of MSCOCO and F30K. For image-to-text and text-to-image retrieval, we measure both zero-shot and fine-tuned performance[^8]. We initialize the similarity score head from the pre-trained ITM head, particularly the part that computes the true-pair logits. We sample 15 random texts as negative samples and tune the model with cross-entropy loss that maximizes the scores on positive pairs.

我们在 Karpathy 与 Fei-Fei 2015 划分的 MSCOCO 和 F30K 上微调 ViLT-B/32。对图像到文本和文本到图像检索，我们同时测量零样本和微调后的表现。相似度分数头从预训练的 ITM 头初始化，具体是计算正配对 logits 的那一部分。我们采样 15 个随机文本作为负样本，用交叉熵损失微调，使正配对上的分数尽量高。

We report the zero shot retrieval results in Table 3 and fine-tuned results in Table 4. At zero-shot retrieval, ViLT-B/32 performs better in general than ImageBERT despite ImageBERT’s pre-training on a larger (14M) dataset. At fine-tuned retrieval, recalls for ViLT-B/32 are higher by a large margin than the second fastest model (Pixel-BERT-R50).

零样本检索结果见表 3，微调结果见表 4。零样本检索上，ViLT-B/32 总体上优于 ImageBERT，尽管后者还在更大的（14M）数据集上做了预训练。微调检索上，ViLT-B/32 的召回率也以较大幅度高于次快的模型（Pixel-BERT-R50）。

4.5 Ablation Study

4.5 消融实验

Training Steps Ablation VQAv2 NLVR2 Flickr30k R@1 (1K) MSCOCO R@1 (5K)
ⓦ ⓜ ⓐ test-dev dev test-P TR (ZS) IR (ZS) TR (ZS) IR (ZS)
25K X X X 68.96 ±\pm± 0.07 70.83 ±\pm± 0.19 70.83 ±\pm± 0.23 75.39 (45.12) 52.52 (31.80) 53.72 (31.55) 34.88 (21.58)
50K X X X 69.80 ±\pm± 0.01 71.93 ±\pm± 0.27 72.92 ±\pm± 0.82 78.13 (55.57) 57.36 (40.94) 57.00 (39.56) 37.47 (27.51)
100K X X X 70.16 ±\pm± 0.01 73.54 ±\pm± 0.02 74.15 ±\pm± 0.27 79.39 (66.99) 60.50 (47.62) 60.15 (51.25) 40.45 (34.59)
100K O X X 70.33 ±\pm± 0.01 74.41 ±\pm± 0.21 74.57 ±\pm± 0.09 81.35 (69.73) 61.86 (51.28) 61.79 (53.40) 41.25 (37.26)
100K O O X 70.21 ±\pm± 0.05 72.76 ±\pm± 0.50 73.54 ±\pm± 0.47 78.91 (63.67) 58.76 (46.96) 59.53 (47.75) 40.08 (32.28)
100K O X O 70.85 ±\pm± 0.13 74.91 ±\pm± 0.29 75.57 ±\pm± 0.61 83.69 (69.73) 62.22 (51.28) 62.88 (53.40) 42.62 (37.26)
200K O X O 71.26 ±\pm± 0.06 75.70 ±\pm± 0.32 76.13 ±\pm± 0.39 83.50 (73.24) 64.36 (54.96) 61.49 (56.51) 42.70 (40.42)

Table 5: Ablation study of ViLT-B/32. ⓦ denotes whether whole word masking is used for pre-training. ⓜ denotes whether MPP objective is used for pre-training. ⓐ denotes whether RandAugment is used during fine-tuning.

训练步数 消融 VQAv2 NLVR2 Flickr30k R@1 (1K) MSCOCO R@1 (5K)
ⓦ ⓜ ⓐ test-dev dev test-P TR (ZS) IR (ZS) TR (ZS) IR (ZS)
25K X X X 68.96 ±\pm± 0.07 70.83 ±\pm± 0.19 70.83 ±\pm± 0.23 75.39 (45.12) 52.52 (31.80) 53.72 (31.55) 34.88 (21.58)
50K X X X 69.80 ±\pm± 0.01 71.93 ±\pm± 0.27 72.92 ±\pm± 0.82 78.13 (55.57) 57.36 (40.94) 57.00 (39.56) 37.47 (27.51)
100K X X X 70.16 ±\pm± 0.01 73.54 ±\pm± 0.02 74.15 ±\pm± 0.27 79.39 (66.99) 60.50 (47.62) 60.15 (51.25) 40.45 (34.59)
100K O X X 70.33 ±\pm± 0.01 74.41 ±\pm± 0.21 74.57 ±\pm± 0.09 81.35 (69.73) 61.86 (51.28) 61.79 (53.40) 41.25 (37.26)
100K O O X 70.21 ±\pm± 0.05 72.76 ±\pm± 0.50 73.54 ±\pm± 0.47 78.91 (63.67) 58.76 (46.96) 59.53 (47.75) 40.08 (32.28)
100K O X O 70.85 ±\pm± 0.13 74.91 ±\pm± 0.29 75.57 ±\pm± 0.61 83.69 (69.73) 62.22 (51.28) 62.88 (53.40) 42.62 (37.26)
200K O X O 71.26 ±\pm± 0.06 75.70 ±\pm± 0.32 76.13 ±\pm± 0.39 83.50 (73.24) 64.36 (54.96) 61.49 (56.51) 42.70 (40.42)

表 5：ViLT-B/32 的消融实验。ⓦ 表示预训练是否使用整词掩码。ⓜ 表示预训练是否使用 MPP 目标。ⓐ 表示微调是否使用 RandAugment。

In Table 5, we perform various ablations. More training steps, whole word masking, and image augmentation come to be beneficial, whereas an additional training objective does not help.

表 5 中我们做了多项消融。更长的训练步数、整词掩码和图像增强都有帮助，而额外的训练目标没有带来提升。

It has been reported that the number of training iterations affects the performance of self-supervised models (Devlin et al. 2019; Chen et al. 2020a; Chen et al. 2020b). As VLP is also a form of self-supervised training, we examine the effects of training durations. As expected, the performance constantly increases as we train the model for longer training steps (rows 13). Masking whole words for the MLM objective (rows 34) and fine-tuning with augmentation (row 6) also drive performance. Further increase in training iterations to 200K improved performance on VQAv2, NLVR2, and zero-shot retrieval. We stop increasing the number of iterations over 200K as the fine-tuned text retrieval performance decreases afterward.

已有报道指出，训练迭代次数会影响自监督模型的表现（Devlin 等 2019；Chen 等 2020a；Chen 等 2020b）。VLP 也是一种自监督训练，因此我们考察了训练时长的影响。正如所料，训练步数增加时性能持续上升（第 1 至 3 行）。为 MLM 目标做整词掩码（第 3 至 4 行）以及带增强的微调（第 6 行）也能推高性能。把迭代次数进一步增加到 200K，会改善 VQAv2、NLVR2 和零样本检索。我们没有再超过 200K，因为之后微调后的文本检索性能会下降。

An additional masked region modeling (MRM) objective has been the key for performance boost in VLP models such as Chen et al. 2019. We experiment with masked patch prediction (MPP) (Dosovitskiy et al. 2020) which mimics the effect of MRM in a form compatible with patch projections. The patch vvv is masked with the probability of 0.15, and the model predicts the mean RGB value of the masked patch from its contextualized vector zmaskedD∣vz^{D}_{\text{masked}}|_{v}zmaskedD​∣v​. However, MPP turns out not to be contributing to downstream performance (rows 4~5). This result is in sharp contrast to the MRM objective on supervision signals from object detection.

额外的掩码区域建模（MRM）目标，曾是 Chen 等 2019 一类 VLP 模型提升性能的关键。我们试验了掩码图像块预测（MPP）（Dosovitskiy 等 2020），它以与图像块投影相容的形式模仿 MRM 的作用：以 0.15 的概率掩码图像块 vvv，模型根据其上下文化向量 zmaskedD∣vz^{D}_{\text{masked}}|_{v}zmaskedD​∣v​ 预测被掩码块的平均 RGB 值。不过 MPP 对下游性能没有贡献（第 4 至 5 行）。这与作用在目标检测监督信号上的 MRM 目标形成鲜明对比。

4.6 Complexity Analysis of VLP Models

4.6 VLP 模型的复杂度分析

Visual Embed Model #Params (M) #FLOPs (G) Time (ms)
Region ViLBERT36+36 274.3 958.1 ~900
VisualBERT36+128 170.3 425.0 ~925
LXMERT36+20 239.8 952.0 ~900
UNITER-Base36+60 154.7 949.9 ~900
OSCAR-Base50+35 154.7 956.4 ~900
VinVL-Base50+35 157.3 1023.3 ~650
Unicoder-VL100+? 170.3 419.7 ~925
ImageBERT100+44 170.3 420.6 ~925
Grid Pixel-BERT-X152146+? 144.3 185.8 ~160
Pixel-BERT-R50260+? 94.9 136.8 ~60
Linear ViLT-B/32200+40 87.4 55.9 ~15

Table 6: Comparison of VLP models in terms of parameter size, FLOPs, and inference latency. Since FLOPs are proportional to input size, we denote the number of input tokens (image+text) as superscripts ("?" when text length is unreported; we arbitrarily use length 40). Although not captured in FLOPs count nor parameter size (because it is not a tensor operation), note that per-class NMS for 1,600 classes amounts to more than 500 ms in latency. NMS latency varies a lot according to the number of detected classes.

视觉嵌入 模型 参数量 (M) FLOPs (G) 时间 (ms)
区域 ViLBERT36+36 274.3 958.1 ~900
VisualBERT36+128 170.3 425.0 ~925
LXMERT36+20 239.8 952.0 ~900
UNITER-Base36+60 154.7 949.9 ~900
OSCAR-Base50+35 154.7 956.4 ~900
VinVL-Base50+35 157.3 1023.3 ~650
Unicoder-VL100+? 170.3 419.7 ~925
ImageBERT100+44 170.3 420.6 ~925
网格 Pixel-BERT-X152146+? 144.3 185.8 ~160
Pixel-BERT-R50260+? 94.9 136.8 ~60
线性 ViLT-B/32200+40 87.4 55.9 ~15

表 6：VLP 模型在参数量、FLOPs 和推理延迟上的比较。由于 FLOPs 与输入大小成正比，我们用上标标出输入标记数（图像+文本）（原文未报告文本长度时记为“?”，此时我们权且使用长度 40）。需要注意的是，按类别对 1,600 个类做 NMS 会贡献超过 500 毫秒的延迟，这一点既不会反映在 FLOPs 计数里，也不会反映在参数量里，因为它不是张量运算。NMS 延迟还会随检出类别数大幅变化。

Visual Embed Model CNN Backbone RoI Head NMS Trans. Layers
Region ViLBERT R101 C4 PC ~15
VisualBERT X152 FPN PC 12
LXMERT R101 C4 PC ~12
UNITER-Base R101 C4 PC 12
OSCAR-Base R101 C4 PC 12
VinVL-Base X152 C4 CA 12
Unicoder-VL X152 FPN PC 12
ImageBERT X152 FPN PC 12
Grid Pixel-BERT-X152 X152 - - 12
Pixel-BERT-R50 R50 - - 12
Linear ViLT-B/32 - - - 12

Table 7: VLP model components. "PC" is for per-class manner NMS and "CA" is for class-agnostic. Following Tan & Bansal 2019, one single-modality layer is counted as 0.5 multi-modality layer.

视觉嵌入 模型 CNN 主干 RoI 头 NMS Transformer 层
区域 ViLBERT R101 C4 按类 ~15
VisualBERT X152 FPN 按类 12
LXMERT R101 C4 按类 ~12
UNITER-Base R101 C4 按类 12
OSCAR-Base R101 C4 按类 12
VinVL-Base X152 C4 类别无关 12
Unicoder-VL X152 FPN 按类 12
ImageBERT X152 FPN 按类 12
网格 Pixel-BERT-X152 X152 - - 12
Pixel-BERT-R50 R50 - - 12
线性 ViLT-B/32 - - - 12

表 7：VLP 模型的组成。“按类”表示按类别做 NMS，“类别无关”表示 class-agnostic。按照 Tan 与 Bansal 2019，一层单模态层计为 0.5 层多模态层。

Figure 4: Visualizations of transportation plan of word patch alignment. Best viewed zoomed in.

图 4：词–图像块对齐传输方案的可视化。放大后更清晰。

We analyze the complexity of VLP models in various terms. In Table 6, we report the number of parameters, the number of floating-point operations (FLOPs), and the inference latency of the visual embedder and transformer. We exclude the textual embedder because it is shared by all VLP models[^9]. The latency is averaged over 10K times on a Xeon E5-2650 CPU and an NVIDIA P40 GPU.

我们从多个角度分析 VLP 模型的复杂度。表 6 报告了视觉嵌入器与 Transformer 的参数量、浮点运算次数（FLOPs）以及推理延迟。文本嵌入器被所有 VLP 模型共享，因此未计入。延迟在 Xeon E5-2650 CPU 和 NVIDIA P40 GPU 上对 10K 次取平均。

The input size in terms of image resolution and the length of concatenated multimodal input sequence affects the number of FLOPs. We co-note the sequence lengths. The image resolution is 800 ×\times× 1,333 for region-based VLP models and Pixel-BERT-R50, 600 ×\times× 1,000 for Pixel-BERT-X152, and 384 ×\times× 640 for ViLT-B/32.

图像分辨率以及拼接后的多模态输入序列长度会影响 FLOPs。我们一并标出了序列长度。基于区域的 VLP 模型和 Pixel-BERT-R50 的图像分辨率为 800×1,333800 \times 1{,}333800×1,333，Pixel-BERT-X152 为 600×1,000600 \times 1{,}000600×1,000，ViLT-B/32 为 384×640384 \times 640384×640。

In Pixel-BERT and ViLT, visual tokens are sampled during pre-training and used in full during fine-tuning. We report the maximum number of visual tokens.

在 Pixel-BERT 和 ViLT 中，视觉标记在预训练时采样，微调时使用全部标记。我们报告视觉标记数的上限。

We observe that the runtime of BERT-base-like transformers varies only by <1<1<1 ms for input sequences of length under 300. Since patch projection of ViLT-B/32 generates at most 240 image tokens, our model can still be efficient even though it receives a combination of image and text tokens.

我们观察到，对长度在 300 以下的输入序列，类 BERT-base 的 Transformer 运行时间变化不到 1 毫秒。由于 ViLT-B/32 的图像块投影最多产生 240 个图像标记，即使它接收的是图像和文本标记的组合，模型仍然可以保持高效。

4.7 Visualization

4.7 可视化

Figure 4 is an example of a cross-modal alignment. The transportation plan of WPA expresses a heatmap for a text token highlighted in pink color. Each square tile represents a patch, and its opacity indicates how much mass is transported from the highlighted word token.

图 4 是一个跨模态对齐的例子。WPA 的传输方案对粉色高亮的文本标记表现为热图。每个方格代表一个图像块，不透明度表示从该高亮词标记传输过去的质量。

More IPOT iterations– more than over 50 as in the training phase– help the visualization heatmap converge; empirically, 1,000 iterations are sufficient to get a clearly identifiable heatmap. We z-normalize the plan for each token and clamp the values to [1.0, 3.0].

比训练阶段更多的 IPOT 迭代——超过 50 次——有助于可视化热图收敛；经验上看，1,000 次迭代就足以得到轮廓清晰的热图。我们对每个标记的方案做 z 标准化，并把数值截断到 [1.0, 3.0]。

5 Conclusion and Future Work

5 结论与后续工作

In this paper, we present a minimal VLP architecture, Vision-and-Langauge Transformer (ViLT). ViLT is competent to competitors which are heavily equipped with convolutional visual embedding networks (e.g., Faster R-CNN and ResNets). We ask for future work on VLP to focus more on the modality interactions inside the transformer module rather than engaging in an arms race that merely powers up unimodal embedders.

本文提出一个尽量精简的 VLP 架构：视觉–语言 Transformer（ViLT）。与大量配备卷积视觉嵌入网络（例如 Faster R-CNN 和 ResNet）的对手相比，ViLT 仍然具有竞争力。我们希望后续的 VLP 工作把更多注意力放在 Transformer 模块内部的模态交互上，而不是只比赛谁把单模态嵌入器做得更强。

Although remarkable as it is, ViLT-B/32 is more of a proof of concept that efficient VLP models free of convolution and region supervision can still be competent. We wrap up by pointing out a few factors that may add to the ViLT family.

尽管已经相当出色，ViLT-B/32 更像是一个概念验证：不依赖卷积和区域监督的高效 VLP 模型仍然可以具有竞争力。最后我们指出几个可能让 ViLT 家族继续增强的因素。

Scalability.

As shown in papers on large-scale transformers (Devlin et al. 2019; Dosovitskiy et al. 2020), the performance of pre-trained transformers scale well given an appropriate amount of data. This observation paves the way for even better performing ViLT variants (e.g., ViLT-L (large) and ViLT-H (huge)). We leave training larger models for future work because aligned vision-and-language datasets are yet scarce.

可扩展性

关于大规模 Transformer 的论文（Devlin 等 2019；Dosovitskiy 等 2020）表明，只要数据量合适，预训练 Transformer 的性能可以很好地随规模提升。这一观察为表现更好的 ViLT 变体（例如 ViLT-L（large）和 ViLT-H（huge））铺平了道路。由于对齐的视觉–语言数据仍然偏少，我们把训练更大模型留作后续工作。

Masked Modeling for Visual Inputs.

Considering the success of MRM, we speculate that the masked modeling objective for the visual modality helps by preserving the information up until the last layer of the transformer. However, as observed in Table 5, a naive variant of MRM on image patches (MPP) fails.

视觉输入的掩码建模

考虑到 MRM 的成功，我们推测视觉模态上的掩码建模目标之所以有帮助，是因为它能把信息保持到 Transformer 的最后一层。但如表 5 所示，把 MRM 简单地改成作用在图像块上的变体（MPP）并没有成功。

Cho et al. 2020 proposed to train their grid RoIs on masked object classification (MOC) tasks. However, the visual vocabulary cluster in this work was fixed during the vision and language pre-training together with the visual backbone. For trainable visual embedders, one-time clustering is not a viable option. We believe that alternating clustering (Caron et al. 2018; Caron et al. 2019) or simultaneous clustering (Asano et al. 2019; Caron et al. 2020) methods studied in visual unsupervised learning research could be applied.

Cho 等 2020 提出在网格 RoI 上训练掩码物体分类（MOC）任务。不过这项工作中的视觉词表聚类在视觉–语言预训练期间是固定的，视觉主干也同样被冻结。对可训练的视觉嵌入器来说，一次性聚类并不可行。我们认为，视觉无监督学习中研究过的交替聚类（Caron 等 2018；Caron 等 2019）或同时聚类（Asano 等 2019；Caron 等 2020）方法可以派上用场。

We encourage future work that does not use region supervision to devise a more sophisticated masking objective for the visual modality.

我们鼓励不使用区域监督的后续工作，为视觉模态设计更精细的掩码目标。

Augmentation Strategies.

Previous work on contrastive visual representation learning (Chen et al. 2020a; Chen et al. 2020b) showed that gaussian blur, not employed by RandAugment, brings noticeable gains to downstream performance compared with a simpler augmentation strategy (He et al. 2020). Exploration of appropriate augmentation strategies for textual and visual inputs would be a valuable addition.

增强策略

对比视觉表示学习的以往工作（Chen 等 2020a；Chen 等 2020b）表明，RandAugment 没有采用的高斯模糊，相比更简单的增强策略（He 等 2020），会给下游性能带来明显增益。探索适合文本和视觉输入的增强策略，会是一项有价值的补充。

References

参考文献

Anderson et al. (2018) Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 6077–6086, 2018.

Asano et al. (2019) Asano, Y., Rupprecht, C., and Vedaldi, A. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations , 2019.

Berman et al. (2019) Berman, M., Jégou, H., Vedaldi, A., Kokkinos, I., and Douze, M. Multigrain: a unified image embedding for classes and instances. arXiv preprint arXiv:1902.05509 , 2019.

Bugliarello et al. (2020) Bugliarello, E., Cotterell, R., Okazaki, N., and Elliott, D. Multimodal pretraining unmasked: Unifying the vision and language berts. arXiv preprint arXiv:2011.15124 , 2020.

Caron et al. (2018) Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV) , pp. 132–149, 2018.

Caron et al. (2019) Caron, M., Bojanowski, P., Mairal, J., and Joulin, A. Unsupervised pre-training of image features on non-curated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 2959–2968, 2019.

Caron et al. (2020) Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882 , 2020.

Chen et al. (2020a) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pp. 1597–1607. PMLR, 2020a.

Chen et al. (2020b) Chen, X., Fan, H., Girshick, R., and He, K. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020b.

Chen et al. (2019) Chen, Y.-C., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 , 2019.

Cho et al. (2020) Cho, J., Lu, J., Schwenk, D., Hajishirzi, H., and Kembhavi, A. X-lxmert: Paint, caption and answer questions with multi-modal transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 8785–8805, 2020.

Cubuk et al. (2020) Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pp. 702–703, 2020.

Cui et al. (2019) Cui, Y., Che, W., Liu, T., Qin, B., Yang, Z., Wang, S., and Hu, G. Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101 , 2019.

Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 4171–4186, 2019.

Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Faghri et al. (2017) Faghri, F., Fleet, D. J., Kiros, J. R., and Fidler, S. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612 , 2017.

Gan et al. (2020) Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., and Liu, J. Large-scale adversarial training for vision-and-language representation learning. arXiv preprint arXiv:2006.06195 , 2020.

Goyal et al. (2017) Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 6904–6913, 2017.

He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770–778, 2016.

He et al. (2017) He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pp. 2961–2969, 2017.

He et al. (2020) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9729–9738, 2020.

Hoffer et al. (2020) Hoffer, E., Ben-Nun, T., Hubara, I., Giladi, N., Hoefler, T., and Soudry, D. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8129–8138, 2020.

Huang et al. (2020) Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849 , 2020.

Jiang et al. (2020) Jiang, H., Misra, I., Rohrbach, M., Learned-Miller, E., and Chen, X. In defense of grid features for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10267–10276, 2020.

Jiang et al. (2018) Jiang, Y., Natarajan, V., Chen, X., Rohrbach, M., Batra, D., and Parikh, D. Pythia v0. 1: the winning entry to the vqa challenge 2018. arXiv preprint arXiv:1807.09956 , 2018.

Karpathy & Fei-Fei (2015) Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 3128–3137, 2015.

Krishna et al. (2017) Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision , 123(1):32–73, 2017.

Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In NIPS , 2012.

Kuznetsova et al. (2020) Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. The open images dataset v4. International Journal of Computer Vision , pp. 1–26, 2020.

Lee et al. (2018) Lee, K.-H., Chen, X., Hua, G., Hu, H., and He, X. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV) , pp. 201–216, 2018.

Li et al. (2020a) Li, G., Duan, N., Fang, Y., Gong, M., Jiang, D., and Zhou, M. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI , pp. 11336–11344, 2020a.

Li et al. (2019) Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019.

Li et al. (2020b) Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision , pp. 121–137. Springer, 2020b.

Lin et al. (2014) Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European conference on computer vision , pp. 740–755. Springer, 2014.

Loshchilov & Hutter (2018) Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

Lu et al. (2019) Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems , pp. 13–23, 2019.

Lu et al. (2020) Lu, J., Goswami, V., Rohrbach, M., Parikh, D., and Lee, S. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10437–10446, 2020.

Nguyen et al. (2020) Nguyen, D.-K., Goswami, V., and Chen, X. Revisiting modulated convolutions for visual counting and beyond. arXiv preprint arXiv:2004.11883 , 2020.

Ordonez et al. (2011) Ordonez, V., Kulkarni, G., and Berg, T. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems , 24:1143–1151, 2011.

Perez et al. (2018) Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Plummer et al. (2015) Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision , pp. 2641–2649, 2015.

Qi et al. (2020) Qi, D., Su, L., Song, J., Cui, E., Bharti, T., and Sacheti, A. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprint arXiv:2001.07966 , 2020.

Radford et al. (2021) Radford, A., Sutskever, I., Kim, J., Krueger, G., and Agarwal, S. Learning transferable visual models from natural language supervision, 2021.

Ren et al. (2016) Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence , 39(6):1137–1149, 2016.

Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115(3):211–252, 2015.

Sharma et al. (2018) Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 2556–2565, 2018.

Shorten & Khoshgoftaar (2019) Shorten, C. and Khoshgoftaar, T. M. A survey on image data augmentation for deep learning. Journal of Big Data , 6(1):1–48, 2019.

Su et al. (2019) Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., and Dai, J. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 , 2019.

Suhr et al. (2018) Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., and Artzi, Y. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491 , 2018.

Tan & Bansal (2019) Tan, H. and Bansal, M. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 , 2019.

Touvron et al. (2020) Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 , 2020.

Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems , 30:5998–6008, 2017.

Xie et al. (2017) Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1492–1500, 2017.

Xie et al. (2020) Xie, Y., Wang, X., Wang, R., and Zha, H. A fast proximal point method for computing exact wasserstein distance. In Uncertainty in Artificial Intelligence , pp. 433–453. PMLR, 2020.

Yu et al. (2020) Yu, F., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H., and Wang, H. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934 , 2020.

Yu et al. (2019) Yu, Z., Yu, J., Cui, Y., Tao, D., and Tian, Q. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 6281–6290, 2019.

Yun et al. (2019) Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 6023–6032, 2019.

Zhang et al. (2017) Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Zhang et al. (2021) Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., and Gao, J. Vinvl: Making visual representations matter in vision-language models. arXiv preprint arXiv:2101.00529 , 2021.

Zhou et al. (2020) Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J. J., and Gao, J. Unified vision-language pre-training for image captioning and vqa. In AAAI , pp. 13041–13049, 2020.