ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

ViLBERT：面向视觉-语言任务的任务无关视觉语言表示预训练

Jiasen Lu Affiliation: Georgia Institute of Technology Dhruv Batra Affiliation: Georgia Institute of Technology Affiliation: Facebook AI Research Devi Parikh Affiliation: Georgia Institute of Technology Affiliation: Facebook AI Research Stefan Lee Affiliation: Georgia Institute of Technology Affiliation: Oregon State University

Abstract

摘要

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, processing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks - visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval - by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models - achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.

我们提出 ViLBERT（Vision-and-Language BERT 的简称）：一种学习图像内容与自然语言的任务无关联合表示的模型。我们把流行的 BERT 架构扩展为多模态双流模型，视觉与文本输入分别在独立流中处理，并通过协同注意力 Transformer 层交互。模型在大规模自动采集的 Conceptual Captions 数据集上，经两项代理任务预训练，再仅对基础架构做少量增补，迁移到多个成熟的视觉-语言任务——视觉问答、视觉常识推理、指称表达式与基于标题的图像检索。相对既有任务专用模型，我们在各项任务上观察到显著提升，并在全部四个任务上达到当时最优。这项工作标志着：视觉 grounding 不再只作为任务训练的一部分来学，而应被当作可预训练、可迁移的能力。

1 Introduction

1 引言

“\dots spend the summer linking a camera to a computer and getting the computer to describe what it saw.”

「\dots\dots花一个夏天把相机接到计算机上，让计算机描述它所看到的。」

Marvin Minsky on the goal of a 1966 undergraduate summer research project [1]

Marvin Minsky 谈 1966 年本科生暑期研究项目的目标 [1]

Since this now famously ambitious summer project, steady progress has been made towards systems that can demonstrate their visual understanding by generating or responding to natural language in the context of images, videos, or even full 3D environments [2, 3, 4, 5, 6, 7, 8]. These approaches and corresponding tasks have come to be referred to under the common banner of ‘vision-and-language’. However, despite the common need to align natural language and visual stimuli - i.e . to perform visual grounding - approaches for vision-and-language tasks lack a unified foundation to gain this capability. Instead, the dominant strategy is to start with separate language and vision models pretrained for other large-scale tasks and then learn grounding as part of task training - often resulting in myopic groundings that generalize poorly when paired visiolinguistic data is limited or biased [9, 10].

自这一如今著名的雄心勃勃的暑期项目以来，能在图像、视频乃至完整三维环境中用自然语言生成或回应以展示视觉理解的系统，已稳步推进 [2, 3, 4, 5, 6, 7, 8]。这些方法与对应任务被统称为「视觉与语言」。然而，尽管都需要把自然语言与视觉刺激对齐——即做 视觉 grounding ——视觉-语言任务的方法仍缺少获得该能力的统一基础。主导策略是：先分别用为其他大规模任务预训练的语言与视觉模型起步，再在任务训练中学习 grounding——当成对视觉语言数据有限或有偏时，往往得到短视、泛化差的 grounding [9, 10]。

This pretrain-then-transfer learning approach to vision-and-language tasks follows naturally from its widespread use in both computer vision and natural language processing where it has become the de facto standard due to the ease-of-use and strong representational power of large, publicly-available models [11, 12, 13, 14] trained on large-scale data sources [15, 16, 17, 18, 19]. In these domains, pretrained models can provide useful information for target tasks, e.g . dog breed-sensitive image features or a well-calibrated semantic distance between words. While visual and linguistic understandings like these are of course essential to vision-and-language tasks, equally important is how they relate to one another - e.g . a perfect visual representation of dog breeds is of little use if a downstream vision-and-language model fails to associate it with appropriate phrases like “beagle” or “shepherd”. We are therefore interested in developing a common model for visual grounding that can learn these connections and leverage them on a wide array of vision-and-language tasks - i.e ., we seek to pretrain for visual grounding.

这种 预训练再迁移 的视觉-语言学习路径，自然延续自计算机视觉与自然语言处理中的广泛做法：因公开、大规模数据源 [15, 16, 17, 18, 19] 上训练的大型模型 [11, 12, 13, 14] 易用且表示力强，已成为事实标准。在这些领域，预训练模型能为目标任务提供有用信息，例如对犬种敏感的图像特征，或词与词之间校准良好的语义距离。这类视觉与语言理解对视觉-语言任务固然必要，但同样重要的是二者如何相互关联——例如，若下游视觉-语言模型无法把「beagle」或「shepherd」等短语与之对应，再完美的犬种视觉表示也无济于事。因此我们希望为视觉 grounding 建立通用模型，学到这些联系并迁移到广泛的视觉-语言任务上——亦即，我们寻求为视觉 grounding 做预训练。

\sim

为学习这些联合视觉-语言表示，我们借鉴自监督学习的近期成功：通过训练模型完成所谓「代理」任务，从大规模无标注数据中捕获丰富的语义与结构信息。代理任务利用数据内部结构自动生成监督信号（例如图像上色 [20] 或重构文本中被掩码的词 [12]）。视觉社区内的工作前景日增 [21, 22, 23]，但迄今影响最大的仍是 ELMo [13]、BERT [12]、GPT [14] 等语言模型，它们在多项 NLP 任务上刷新了水位线。要以类似方式学习视觉 grounding，需找到视觉与语言对齐可用的合适数据源。本文考虑新近发布的 Conceptual Captions [24]：约 330 万张图像，配有从网页带 alt-text 图像自动采集的弱关联描述性标题。

We present a joint model for learning task-agnostic visual grounding from paired visiolinguistic data which we call Vision & Language BERT (ViLBERT for short). Our approach extends the recently developed BERT [12] language model to jointly reason about text and images. Our key technical innovation is introducing separate streams for vision and language processing that communicate through co-attentional transformer layers. This structure can accommodate the differing processing needs of each modality and provides interaction between modalities at varying representation depths. We demonstrate that this structure outperforms a single-stream unified model in our experiments.

我们提出从成对视觉语言数据中学习任务无关视觉 grounding 的联合模型，称为 Vision & Language BERT（简称 ViLBERT）。方法把新近的 BERT [12] 语言模型扩展到联合推理文本与图像。关键技术创新是引入视觉与语言的分离处理流，并通过协同注意力 Transformer 层通信。该结构可容纳各模态不同的处理需求，并在不同表示深度提供模态间交互。实验表明，相对单流统一模型，这一结构表现更好。

In analogy to the training tasks in [12], we train our model on Conceptual Captions on two proxy tasks: predicting the semantics of masked words and image regions given the unmasked inputs, and predicting whether an image and text segment correspond. We apply our pretrained model as a base for four established vision-and-language tasks - visual question answering [3], visual commonsense reasoning [25], referring expressions [2], and caption-based image retrieval [26] - setting state-of-the-art on all four tasks. We find improvements of 2 to 10 percentage points across these tasks when compared to state-of-the-art task-specific baselines using separately pretrained vision and language models. Furthermore, our structure is simple to modify for each of these tasks - serving as a common foundation for visual grounding across multiple vision-and-language tasks.

类比 [12] 中的训练任务，我们在 Conceptual Captions 上用两项代理任务训练模型：在给定未掩码输入时预测被掩码词与图像区域的语义，以及预测图像与文本片段是否对应。把预训练模型作为基础，应用到四个成熟视觉-语言任务——视觉问答 [3]、视觉常识推理 [25]、指称表达式 [2] 与基于标题的图像检索 [26]——并在全部四个任务上达到当时最优。相对分别预训练视觉与语言模型的当时最优任务专用基线，提升约 2 到 10 个百分点。此外，对各任务修改结构很简单——作为跨多项视觉-语言任务的视觉 grounding 共同基础。

2 Approach

2 方法

In this section, we first briefly summarize the BERT language model (Sec. 2.1) and then describe how we extend it to jointly represent vision and language data (Sec. 2.2).

本节先简要概述 BERT 语言模型（第 2.1 节），再说明如何将其扩展为联合表示视觉与语言数据（第 2.2 节）。

2.1 Preliminaries: Bidirectional Encoder Representations from Transformers (BERT)

2.1 预备：来自 Transformer 的双向编码器表示（BERT）

Figure 1: Our ViLBERT model consists of two parallel streams for visual (green) and linguistic (purple) processing that interact through novel co-attentional transformer layers. This structure allows for variable depths for each modality and enables sparse interaction through co-attention. Dashed boxes with multiplier subscripts denote repeated blocks of layers.

图 1：我们的 ViLBERT 模型由视觉（绿）与语言（紫）两条并行流构成，经新颖的协同注意力 Transformer 层交互。该结构允许各模态深度可变，并通过协同注意力实现稀疏交互。带倍增下标的虚线框表示重复的层块。

The BERT model introduced by [12] is an attention-based bidirectional language model. When pretrained on a large language corpus, BERT has proven to be very effective for transfer learning to multiple natural language processing tasks.

[12] 提出的 BERT 是基于注意力的双向语言模型。在大规模语料上预训练后，BERT 已被证明能有效迁移到多项自然语言处理任务。

w_{0},\dots,w_{T}

w_{0},\dots,w_{T}

Text Representation. BERT operates over sequences of discrete tokens comprised of vocabulary words and a small set of special tokens: SEP, CLS, and MASK. For a given token, the input representation is a sum of a token-specific learned embedding [28] and encodings for position (i.e . token’s index in the sequence) and segment (i.e . index of the token’s sentence if multiple exist).

文本表示 。 BERT 处理由词表词与一小套特殊 token（SEP、CLS、MASK）组成的离散 token 序列。对给定 token，输入表示是 token 专用学得嵌入 [28] 与位置编码（即 token 在序列中的下标）及片段编码（若存在多句，则为 token 所属句子的下标）之和。

Training Tasks and Objectives. The BERT model is trained end-to-end on a large language-corpus under two tasks: masked language modelling and next sentence prediction .

训练任务与目标 。 BERT 在大规模语料上端到端训练，含两项任务： 掩码语言建模 与 下一句预测 。

X_{M}

X_{M}

A

A

(a) Standard encoder transformer block

(a) 标准编码器 Transformer 块

2.2 ViLBERT: Extending BERT to Jointly Represent Images and Text

2.2 ViLBERT：把 BERT 扩展到联合表示图像与文本

Inspired by BERT’s success at language modeling, we would like to develop analogous models and training tasks to learn joint representations of language and visual content from paired data. Specifically, we consider jointly representing static images and corresponding descriptive text.

受 BERT 在语言建模上成功的启发，我们希望发展类似模型与训练任务，从成对数据中学习语言与视觉内容的联合表示。具体地，我们考虑联合表示静态图像与对应的描述性文本。

One straightforward approach is to make minimal changes to BERT -- simply discretizing the space of visual inputs via clustering, treat these visual ‘tokens’ exactly like text inputs, and start from a pretrained BERT model[^1]. This architecture suffers from a number of drawbacks. First, initial clustering may result in discretization error and lose important visual details. Second, it treats inputs from both modalities identically, ignoring that they may need different levels of processing due to either their inherent complexity or the initial level of abstraction of their input representations. For instance, image regions may have weaker relations than words in a sentence and visual features are themselves often already the output of a very deep network. Finally, forcing the pretrained weights to accommodate the large set of additional visual ‘tokens’ may damage the learned BERT language model. Instead, we develop a two-stream architecture modelling each modality separately and then fusing them through a small set of attention-based interactions. This approach allows for variable network depth for each modality and enables cross-modal connections at different depths.

一种直接做法是对 BERT 做最少改动——通过聚类离散化视觉输入空间，把这些视觉「token」与文本输入一视同仁，并从预训练 BERT 起步[^1]。该架构有若干缺陷。首先，初始聚类可能导致离散化误差并丢失重要视觉细节。其次，它同等对待两模态输入，忽略了因固有复杂度或输入表示初始抽象程度不同，它们可能需要不同层次的处理。例如，图像区域之间的关系可能弱于句子中词与词的关系，且视觉特征本身往往已是很深网络的输出。最后，迫使预训练权重容纳大量额外视觉「token」可能损害已学到的 BERT 语言模型。相反，我们发展双流架构：各模态单独建模，再经少量基于注意力的交互融合。该方法允许各模态网络深度可变，并在不同深度启用跨模态连接。

I

v_{1},\dots,v_{\mathcal{T}}

H^{(i)}_{V}

H^{(i)}_{V}

Image Representations. We generate image region features by extracting bounding boxes and their visual features from a pre-trained object detection network (see Sec. 3.1). Unlike words in text, image regions lack a natural ordering. we encode spatial location instead, constructing a 5-d vector from region position (normalized top-left and bottom-right coordinates) and the fraction of image area covered. This is then projected to match the dimension of the visual feature and they are summed.

图像表示 。 我们用预训练目标检测网络提取边界框及其视觉特征，以生成图像区域特征（见第 3.1 节）。与文本中的词不同，图像区域没有自然顺序。我们改为编码空间位置：由区域位置（归一化左上与右下坐标）及所占图像面积比例构造 5 维向量，再投影到与视觉特征同维并相加。

We mark the beginning of an image region sequence with a special IMG token representing the entire image (i.e. mean-pooled visual features with a spatial encoding corresponding to the entire image).

我们用特殊 IMG token 标记图像区域序列的开头，表示整幅图像（即均值池化的视觉特征，配以对应整幅图像的空间编码）。

(a) Masked multi-modal learning

(a) 掩码多模态学习

Training Tasks and Objectives. In analogy to those described in the previous section, we consider two pretraining tasks: masked multi-modal modelling and multi-modal alignment prediction .

训练任务与目标 。 类比上一节所述，我们考虑两项预训练任务： 掩码多模态建模 与 多模态对齐预测 。

The masked multi-modal modelling task (shown in Fig. 3(a)) follows from the masked language modelling task in standard BERT - masking approximately 15% of both words and image region inputs and tasking the model with reconstructing them given the remaining inputs. Masked image regions have their image features zeroed out 90% of the time and are unaltered 10%. Masked text inputs are handled as in BERT. Rather than directly regressing the masked feature values, the model instead predicts a distribution over semantic classes for the corresponding image region. To supervise this, we take the output distribution for the region from the same pretrained detection model used in feature extraction. We train the model to minimize the KL divergence between these two distributions. This choice reflects the notion that language often only identifies high-level semantics of visual content and is unlikely to be able to reconstruct exact image features. Further, applying a regression loss could make it difficult to balance losses incurred by masked image and text inputs.

掩码多模态建模任务（图 3(a)）沿袭标准 BERT 的掩码语言建模——约掩码 15% 的词与图像区域输入，并要求模型在给定其余输入时重构它们。被掩码图像区域的图像特征有 90% 时间置零，10% 保持不变；被掩码文本输入按 BERT 方式处理。模型不直接回归被掩码特征值，而是预测对应图像区域语义类别上的分布。监督信号取自特征提取所用同一预训练检测模型对该区域的输出分布；训练目标是最小化两分布之间的 KL 散度。这一选择反映：语言往往只识别视觉内容的高层语义，不太可能重构精确图像特征；此外，回归损失也可能难以平衡掩码图像与文本输入带来的损失。

v_{1},\dots,v_{\mathcal{T}},

v_{1},\dots,v_{\mathcal{T}},

3 Experimental Settings

3 实验设定

In this section, we describe how we train our model and provide overviews of the vision-and-language tasks to which we transfer the trained model.

本节说明如何训练模型，并概述我们迁移预训练模型所至的视觉-语言任务。

3.1 Training ViLBERT

3.1 训练 ViLBERT

To train our full ViLBERT model, we apply the training tasks presented in Sec. 2.2 to the Conceptual Captions dataset [24]. Conceptual Captions is a collection of 3.3 million image-caption pairs automatically scraped from alt-text enabled web images. The automatic collection and sanitation process leaves some noise and the ‘captions’ are sometimes not human-like or short on details (e.g . “actors attend the premiere at festival”). However, it presents a huge diversity of visual content and serves as an excellent dataset for our purposes. Since some links had become broken by the time we downloaded the data, our model is trained with around 3.1 million image-caption pairs.

为训练完整 ViLBERT 模型，我们把第 2.2 节给出的训练任务施加于 Conceptual Captions 数据集 [24]。Conceptual Captions 是从带 alt-text 的网页图像自动抓取的 330 万图像-标题对集合。自动采集与清洗过程留下噪声，「标题」有时不像人写的话或细节不足（例如「actors attend the premiere at festival」）。但它视觉内容多样性极大，非常适合我们的目的。下载时部分链接已失效，因此模型实际用约 310 万图像-标题对训练。

{}_{\texttt{BASE}}

{}_{\texttt{BASE}}

i

i

We train on 8 TitanX GPUs with a total batch size of 512 for 10 epochs. We use the Adam optimizer with initial learning rates of 1e-4. We use a linear decay learning rate schedule with warm up to train the model. Both training task losses are weighed equally.

我们在 8 块 TitanX GPU 上训练，总批大小 512，共 10 个 epoch。使用 Adam 优化器，初始学习率 1e-4；采用带预热的线性衰减学习率日程。两项训练任务损失等权。

Figure 4: Examples for each vision-and-language task we transfer ViLBERT to in our experiments.

图 4：实验中把 ViLBERT 迁移到的各视觉-语言任务示例。

3.2 Vision-and-Language Transfer Tasks

3.2 视觉-语言迁移任务

We transfer our pretrained ViLBERT model to a set of four established vision-and-language tasks (see examples in Fig.4) and one diagnostic task. We follow a fine-tuning strategy where we modify the pretrained base model to perform the new task and then train the entire model end-to-end. In all cases, the modification is trivial - typically amounting to learning a classification layer. This is in stark contrast to the significant efforts made within the community to develop specialized models for each of these tasks. We describe the problem, dataset, model modifications, and training objective for each task below.

我们把预训练 ViLBERT 迁移到四个成熟视觉-语言任务（示例见图 4）以及一项诊断任务。采用微调策略：修改预训练基础模型以完成新任务，再端到端训练整个模型。所有情形下修改都很琐碎——通常只需学习一个分类层。这与社区为各任务开发专用模型的大量努力形成鲜明对比。下文分别描述各任务的问题、数据集、模型修改与训练目标。

h_{\mbox{{IMG}}}

h_{\mbox{{IMG}}}

\rightarrow

\rightarrow

h_{v_{i}}

i

Caption-Based Image Retrieval. Caption-based image retrieval is the task of identifying an image from a pool given a caption describing its content. We train and evaluate on the Flickr30k dataset [26] consisting of 31,000 images from Flickr with five captions each. Following the splits in [35], we use 1,000 images for validation and test each and train on the rest. These captions are well-grounded in and descriptive of the visual content and are qualitatively different than the automatically collected Conceptual Captions. We train in a 4-way multiple-choice setting by randomly sampling three distractors for each image-caption pair - substituting a random caption, a random image, or a hard negative from among the 100 nearest neighbors of the target image. We compute the alignment score (as in alignment prediction pretraining) for each and apply a softmax. We train this model under a cross-entropy loss to select the true image-caption pair for 20 epochs with a batch size of 64 and an initial learning rate of 2e-5. At inference, we score each caption-image pair in the test set and then sort. For efficiency, we cache the linguistic stream representation before the first Co-TRM layer - effectively freezing the linguistic representation before fusion.

基于标题的图像检索 。 该任务是在给定描述内容的标题时，从池中识别图像。我们在 Flickr30k 数据集 [26] 上训练与评估：来自 Flickr 的 31000 张图像，每张五条标题。按 [35] 的划分，验证与测试各用 1000 张，其余训练。这些标题与视觉内容 grounding 良好且描述充分，与自动采集的 Conceptual Captions 在质感上不同。我们以 4 选 1 设定训练：对每个图像-标题对随机采样三个干扰项——替换随机标题、随机图像，或从目标图像的 100 近邻中取难负例。对每个计算对齐分数（同对齐预测预训练），再施加 softmax；以交叉熵损失训练模型选出真实图像-标题对，共 20 个 epoch，批大小 64，初始学习率 2e-5。推理时对测试集每个标题-图像对打分再排序。为提效，我们在第一个 Co-TRM 层之前缓存语言流表示——等效于在融合前冻结语言表示。

‘Zero-shot’ Caption-Based Image Retrieval. The previous tasks are all transfer tasks that include dataset specific fine-tuning. In this ‘zero-shot’ task, we directly apply the pretrained the multi-modal alignment prediction mechanism to caption-based image retrieval in Flickr30k [26] without fine-tuning (thus the description as ‘zero-shot’). The goal of this task is to demonstrate that the pretraining has developed the ability to ground text and that this can generalize to visual and linguistic variation without any task specific fine-tuning. We directly use the ViLBERT model trained on Conceptual Captions dataset described in Sec. 3.1. We use the alignment prediction objective as a scoring function and test on the same split as the caption-based image retrieval task described above.

「零样本」基于标题的图像检索 。 前述任务都是含数据集专用微调的迁移任务。在本「零样本」任务中，我们 不做微调 ，直接把预训练的多模态对齐预测机制用于 Flickr30k [26] 上的基于标题图像检索（故称「零样本」）。目标是证明预训练已发展出接地文本的能力，并能在无任务专用微调的情况下泛化到视觉与语言变化。我们直接使用第 3.1 节所述在 Conceptual Captions 上训练的 ViLBERT；用对齐预测目标作为打分函数，并在与上述检索任务相同的划分上测试。

4 Results and Analysis

4 结果与分析

Baselines. We compare our pretrained ViLBERT model against two ablative baselines:

基线 。 我们把预训练 ViLBERT 与两个消融基线对比：

{}_{\texttt{BASE}}

{}_{\texttt{BASE}}

For both baselines and our model, we finetune the transfer tasks as described in the previous section.

对两个基线与我们的模型，均按上一节所述微调迁移任务。

Task-Specific Baselines. To put our results in context, we present published results of problem-specific methods that are to our knowledge state-of-the-art in each task: DFAF [36] for VQA, R2C [25] for VCR, MAttNet [33] for RefCOCO+, and SCAN [35] for caption-based image retrieval.

任务专用基线 。 为把结果放入语境，我们给出据我们所知各任务当时最优的问题专用方法已发表结果：VQA 用 DFAF [36]，VCR 用 R2C [25]，RefCOCO+ 用 MAttNet [33]，基于标题的图像检索用 SCAN [35]。

\rightarrow

		VQA [3]	VCR [25]	RefCOCO+ [32]	Image Retrieval [26]	ZS Image Retrieval
	Method	test-dev (test-std)	Q $\rightarrow$ A	QA $\rightarrow$ R	Q $\rightarrow$ AR	val	testA	testB	R1	R5	R10	R1	R5	R10
SOTA	DFAF [36]	70.22 (70.34)	-	-	-	-	-	-	-	-	-	-	-	-
	R2C [25]	-	63.8 (65.1)	67.2 (67.3)	43.1 (44.0)	-	-	-	-	-	-	-	-	-
	MAttNet [33]	-	-	-	-	65.33	71.62	56.02	-	-	-	-	-	-
	SCAN [35]	-	-	-	-	-	-	-	48.60	77.70	85.20	-	-	-
Ours	Single-Stream†	65.90	68.15	68.89	47.27	65.64	72.02	56.04	-	-	-	-	-	-
	Single-Stream	68.85	71.09	73.93	52.73	69.21	75.32	61.02	-	-	-	-	-	-
	ViLBERT†	68.93	69.26	71.01	49.48	68.61	75.97	58.44	45.50	76.78	85.02	0.00	0.00	0.00
	ViLBERT	70.55 (70.92)	72.42 (73.3)	74.47 (74.6)	54.04 (54.8)	72.34	78.52	62.61	58.20	84.90	91.52	31.86	61.12	72.80

\rightarrow

		VQA [3]	VCR [25]	RefCOCO+ [32]	图像检索 [26]	零样本图像检索
	方法	test-dev (test-std)	Q $\rightarrow$ A	QA $\rightarrow$ R	Q $\rightarrow$ AR	val	testA	testB	R1	R5	R10	R1	R5	R10
SOTA	DFAF [36]	70.22 (70.34)	-	-	-	-	-	-	-	-	-	-	-	-
	R2C [25]	-	63.8 (65.1)	67.2 (67.3)	43.1 (44.0)	-	-	-	-	-	-	-	-	-
	MAttNet [33]	-	-	-	-	65.33	71.62	56.02	-	-	-	-	-	-
	SCAN [35]	-	-	-	-	-	-	-	48.60	77.70	85.20	-	-	-
Ours	Single-Stream†	65.90	68.15	68.89	47.27	65.64	72.02	56.04	-	-	-	-	-	-
	Single-Stream	68.85	71.09	73.93	52.73	69.21	75.32	61.02	-	-	-	-	-	-
	ViLBERT†	68.93	69.26	71.01	49.48	68.61	75.97	58.44	45.50	76.78	85.02	0.00	0.00	0.00
	ViLBERT	70.55 (70.92)	72.42 (73.3)	74.47 (74.6)	54.04 (54.8)	72.34	78.52	62.61	58.20	84.90	91.52	31.86	61.12	72.80

Results. Tab. 1 shows results across all transfer tasks and we highlight key findings below:

结果 。 表 1 给出全部迁移任务结果，关键发现如下：

- Our architecture improves performance over a single-stream model. We observe improvements across tasks for ViLBERT over the single-stream baseline for both pretrained (Single-Stream vs. ViLBERT) and non-pretrained (Single-Stream† vs. ViLBERT†). Most significant gains are observed for VQA and RefCOCO+. - Our pretraining tasks result in improved visiolinguistic representations. Our models further improve by between 2% and 13% across tasks when using a ViLBERT model that has been pretrained under our proxy tasks (ViLBERT vs ViLBERT† ). We also observe improvements on Single-Stream which verifies our proxy tasks can generalize to different model architectures. - Finetuning from ViLBERT is a powerful strategy for vision-and-language tasks. With a single base architecture, our transfer task performance exceeds state-of-the-art task-specific models for all four established tasks. We set state-of-the-art for VCR, RefCOCO+ and image retrieval by significant margins (7-10 percentage points improvement). Further, extending to these tasks was simple - requiring the addition of a single classifier for each task.

- 我们的架构优于单流模型 。 无论预训练（Single-Stream vs. ViLBERT）还是未预训练（Single-Stream† vs. ViLBERT†），ViLBERT 在各项任务上均相对单流基线有提升。VQA 与 RefCOCO+ 上收益最显著。 - 我们的预训练任务带来更好的视觉语言表示 。 使用经代理任务预训练的 ViLBERT 时，相对 ViLBERT† 各项任务再提升约 2% 到 13%。单流模型上也观察到提升，说明代理任务可泛化到不同模型架构。 - 从 ViLBERT 微调是视觉-语言任务的有力策略 。 以单一基础架构，我们的迁移任务表现超过全部四个成熟任务的任务专用当时最优模型。在 VCR、RefCOCO+ 与图像检索上以显著幅度（提升 7-10 个百分点）刷新当时最优。此外，扩展到这些任务很简单——每个任务只需加一个分类器。

Overall, these results demonstrate that our ViLBERT model is able to learn important visual-linguistic relationships that can be exploited by downstream tasks.

总体而言，这些结果表明 ViLBERT 能够学到下游任务可利用的重要视觉-语言关系。

\rightarrow

	VQA [3]	VCR [25]	RefCOCO+ [32]	Image Retrieval [26]	ZS Image Retrieval [26]
Method	test-dev	Q $\rightarrow$ A	QA $\rightarrow$ R	Q $\rightarrow$ AR	val	testA	testB	R1	R5	R10	R1	R5	R10
ViLBERT (2-layer)	69.92	72.44	74.80	54.40	71.74	78.61	62.28	55.68	84.26	90.56	26.14	56.04	68.80
ViLBERT (4-layer)	70.22	72.45	74.00	53.82	72.07	78.53	63.14	55.38	84.10	90.62	26.28	54.34	66.08
ViLBERT (6-layer)	70.55	72.42	74.47	54.04	72.34	78.52	62.61	58.20	84.90	91.52	31.86	61.12	72.80
ViLBERT (8-layer)	70.47	72.33	74.15	53.79	71.66	78.29	62.43	58.78	85.60	91.42	32.80	63.38	74.62

\rightarrow

	VQA [3]	VCR [25]	RefCOCO+ [32]	图像检索 [26]	零样本图像检索 [26]
方法	test-dev	Q $\rightarrow$ A	QA $\rightarrow$ R	Q $\rightarrow$ AR	val	testA	testB	R1	R5	R10	R1	R5	R10
ViLBERT (2-layer)	69.92	72.44	74.80	54.40	71.74	78.61	62.28	55.68	84.26	90.56	26.14	56.04	68.80
ViLBERT (4-layer)	70.22	72.45	74.00	53.82	72.07	78.53	63.14	55.38	84.10	90.62	26.28	54.34	66.08
ViLBERT (6-layer)	70.55	72.42	74.47	54.04	72.34	78.52	62.61	58.20	84.90	91.52	31.86	61.12	72.80
ViLBERT (8-layer)	70.47	72.33	74.15	53.79	71.66	78.29	62.43	58.78	85.60	91.42	32.80	63.38	74.62

\rightarrow

\rightarrow

Benefits of Large Training Sets. We also studied the impact of the size of the pretraining dataset. For this experiment, we take random subsets of 25% and 50% from the conceptual caption dataset, and pretrain and finetune ViLBERT using the same setup as above. We can see that the accuracy grows monotonically as the amount of data increases, which suggests that ViLBERT may benefit from even more pretraining data.

大规模训练集的收益 。 我们还研究了预训练数据集规模的影响。实验中从 Conceptual Captions 随机取 25% 与 50% 子集，按上述相同设定预训练并微调 ViLBERT。可见准确率随数据量单调上升，提示 ViLBERT 或可从更多预训练数据中继续受益。

\rightarrow

	VQA [3]	VCR [25]	RefCOCO+ [32]	Image Retrieval [26]	ZS Image Retrieval [26]
Method	test-dev	Q $\rightarrow$ A	QA $\rightarrow$ R	Q $\rightarrow$ AR	val	testA	testB	R1	R5	R10	R1	R5	R10
ViLBERT (0 %)	68.93	69.26	71.01	49.48	68.61	75.97	58.44	45.50	76.78	85.02	0.00	0.00	0.00
ViLBERT (25 %)	69.82	71.61	73.00	52.66	69.90	76.83	60.99	53.08	80.80	88.52	20.40	48.54	62.06
ViLBERT (50 %)	70.30	71.88	73.60	53.03	71.16	77.35	61.57	54.84	83.62	90.10	26.76	56.26	68.80
ViLBERT (100 %)	70.55	72.42	74.47	54.04	72.34	78.52	62.61	58.20	84.90	91.52	31.86	61.12	72.80

\rightarrow

	VQA [3]	VCR [25]	RefCOCO+ [32]	图像检索 [26]	零样本图像检索 [26]
方法	test-dev	Q $\rightarrow$ A	QA $\rightarrow$ R	Q $\rightarrow$ AR	val	testA	testB	R1	R5	R10	R1	R5	R10
ViLBERT (0 %)	68.93	69.26	71.01	49.48	68.61	75.97	58.44	45.50	76.78	85.02	0.00	0.00	0.00
ViLBERT (25 %)	69.82	71.61	73.00	52.66	69.90	76.83	60.99	53.08	80.80	88.52	20.40	48.54	62.06
ViLBERT (50 %)	70.30	71.88	73.60	53.03	71.16	77.35	61.57	54.84	83.62	90.10	26.76	56.26	68.80
ViLBERT (100 %)	70.55	72.42	74.47	54.04	72.34	78.52	62.61	58.20	84.90	91.52	31.86	61.12	72.80

Figure 5: Qualitative examples of sampled image descriptions from a ViLBERT model after our pretraining tasks, but before task-specific fine-tuning.

图 5：预训练任务之后、任务专用微调之前，从 ViLBERT 采样得到的图像描述定性示例。

What does ViLBERT learn during pretraining? To get a sense for what ViLBERT learns during Conceptual Caption pretraining, we look at zero-shot caption-based image retreival and some qualitative examples. While zero-shot performance (Tab. 1, right) is significantly lower than the fine-tuned model (31.86 vs 58.20 R1) it performs reasonably without having seen a Flickr30k image or caption (31.86 vs 48.60 R1 for prior SOTA) - indicating that ViLBERT has learned a semantically meaningful alignment between vision and language during pretraining. We also qualitatively inspect the pretrained ViLBERT model by inputting images and sampling out image-conditioned text. This is essentially image captioning. Note that without fine-tuning the model on clean, human-annotated captioning data, the outputs are not likely to be high quality. However, they still serve as a mechanism to inspect what the pretrained model has learned. Fig. 5 shows some of these sampled ‘captions’. Generating text from BERT-style models is an open research area and we follow the sampling procedure from [37] - initializing the text stream with all MASK tokens and then sequentially resampling predicted output tokens in a Markov Chain Monte Carlo fashion. We find many images produce captions that describe image contents; however, due to the collection procedure of Conceptual Captions from web-image alt-text, many of these captions are editorialized (see fireworks example on the top left) and include references to non-visual concepts.

预训练期间 ViLBERT 学到了什么 ？ 为感知 Conceptual Captions 预训练所学内容，我们考察零样本基于标题的图像检索及若干定性示例。零样本表现（表 1 右）虽显著低于微调模型（R1：31.86 vs 58.20），但在从未见过 Flickr30k 图像或标题的情况下仍合理（相对先前 SOTA 的 R1 48.60，我们为 31.86）——表明预训练已学到语义有意义的视觉-语言对齐。我们还通过输入图像并采样图像条件文本，定性检查预训练 ViLBERT；这本质上是图像描述。注意：未在干净的人工标注描述数据上微调时，输出质量不太可能很高，但仍可作为检视预训练所学的机制。图 5 给出部分采样「标题」。从 BERT 风格模型生成文本仍是开放研究问题；我们遵循 [37] 的采样程序——把文本流全部初始化为 MASK token，再以马尔可夫链蒙特卡洛方式依次重采样预测输出 token。许多图像产生描述图像内容的标题；但因 Conceptual Captions 采自网页图像 alt-text，不少标题带有编辑化色彩（见左上烟花示例），并包含非视觉概念的指称。

5 Related Work

5 相关工作

Self-Supervised Learning. There has been substantial recent interest in both vision [38, 39, 40, 41, 42, 43] and language around self-supervised representation learning. In this paradigm, deep models are trained for tasks where regularities in existing data can be turned into supervision automatically. While there has been progress on the vision side, self-supervised image representations still lag behind those from models trained under image classification tasks. Self-supervised language models on the other hand have resulted in significant improvements over prior work [12, 14, 13, 44]. In this work, we develop a model and proxy tasks for learning joint visual-linguistic representations - extending the popular BERT [12] model.

自监督学习 。 视觉 [38, 39, 40, 41, 42, 43] 与语言两侧近期都对自监督表示学习兴趣浓厚。该范式下，深度模型被训练完成可把现有数据规律自动变成监督的任务。视觉侧已有进展，但自监督图像表示仍落后于在图像分类任务上训练的模型；自监督语言模型则相对先前工作带来显著提升 [12, 14, 13, 44]。本文发展用于学习联合视觉-语言表示的模型与代理任务——扩展流行的 BERT [12] 模型。

Vision-and-Language. While we address many vision-and-language tasks in Sec. 3.2, we do miss some families of tasks including visually grounded dialog [4, 45], embodied tasks like question answering [7] and instruction following [8], and text generation tasks like image and video captioning [5]. These tasks may also benefit from a self-supervised approach similar to what we have presented. There are open questions on how to incorporate long sequences of images and text found in dialog, embodied tasks, and video processing. Further, it is unclear how to effectively decode output text from our bidirectional model as existing greedy decoders like beam-search do not apply.

视觉与语言 。 尽管第 3.2 节覆盖多项视觉-语言任务，我们仍错过若干任务族，包括视觉 grounding 对话 [4, 45]、具身任务如问答 [7] 与指令跟随 [8]，以及图像与视频描述等文本生成任务 [5]。这些任务也可能受益于类似本文的自监督路径。如何把对话、具身任务与视频处理中的长图像-文本序列纳入，仍有开放问题。此外，尚不清楚如何有效地从我们的双向模型解码输出文本，因为既有贪心解码器如束搜索并不适用。

Self-Supervised Learning for Vision-And-Language. Most related to our approach is concurrent work on learning joint representations between video and language [29]. In this work, self-supervised tasks paralleling our own are derived from cooking videos paired with text-to-speech transcribed audio. They present a unified BERT architecture for both the visual and linguistic inputs similar to the Single-Stream baseline we consider here. They apply the learned model to two tasks on cooking videos: zero-shot activity recognition and blank-filling on audio transcripts. In contrast, we learn representations of images and descriptive text on a wide range of images from the web and focus extensively on transfer learning from this model for well-established vision-and-language tasks.

视觉-语言的自监督学习 。 与我们方法最相关的是同期关于学习视频与语言联合表示的工作 [29]。该工作从配有语音转写音频的烹饪视频导出与我们平行的自监督任务。他们提出统一 BERT 架构同时处理视觉与语言输入，类似我们此处考虑的单流基线；并把学到的模型用于烹饪视频上的两项任务：零样本活动识别与音频转写填空。相对之下，我们在来自网页的广泛图像上学习图像与描述性文本的表示，并聚焦于把该模型迁移到成熟视觉-语言任务。

6 Conclusion

6 结论

We develop a joint model for image content and text and pretrain it on a large, automatically-collected dataset to learn visual grounding. Our ViLBERT model introduces a novel two-stream architecture with co-attentional transformer blocks that outperforms sensible ablations and exceeds state-of-the-art when transferred to multiple established vision-and-language tasks. Furthermore, transferring our model to these tasks is simple and easy to implement - requiring only the addition of a classifier for each task we examined here. We consider extensions of our model to other vision-and-language tasks (including those requiring generation) as well as multi-task learning as exciting future work.

我们为图像内容与文本建立联合模型，并在大规模自动采集数据集上预训练以学习视觉 grounding。ViLBERT 引入带协同注意力 Transformer 块的新颖双流架构，优于合理消融，并在迁移到多个成熟视觉-语言任务时超过当时最优。此外，把模型迁移到这些任务简单易实现——此处考察的每个任务只需加一个分类器。把模型扩展到其他视觉-语言任务（含需要生成的任务）以及多任务学习，是令人兴奋的未来工作。

Acknowledgement. This work was supported in part by NSF, AFRL, DARPA, ONR YIPs, ARO PECASE. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.

致谢。本研究部分得到 NSF、AFRL、DARPA、ONR YIPs、ARO PECASE 支持。文中观点与结论仅代表作者，不应被解释为必然代表美国政府或任何资助方的官方政策或背书（明示或暗示） 。

References

参考文献

[1] Margaret A. Boden. Mind as Machine: A History of Cognitive Science . Oxford University Press, 2008. [2] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. Referit game: Referring to objects in photographs of natural scenes. In EMNLP , 2014a. [3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In ICCV , 2015. [4] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR , 2017. [5] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR , abs/1504.00325, 2015. [6] Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurelie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. "foil it! find one mismatch between image and language caption". In ACL , 2017. [7] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Question Answering. In CVPR , 2018. [8] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR) , 2018a. [9] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don’t just assume; look and answer: Overcoming priors for visual question answering. In CVPR , 2018a. [10] Harsh Agrawal, Karan Desai, Xinlei Chen, Rishabh Jain, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. arXiv preprint arXiv:1812.08658 , 2018b.

[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016. [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. [13] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NACCL , 2018. [14] Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI, 2018. [15] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV , 2015. [16] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In arXiv , 2016. URL https://arxiv.org/abs/1602.07332 . [17] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV , 2015. [18] English wikipedia, 2019. URL https://en.wikipedia.org/ . [19] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. In arXiv , 2014. [20] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR , pages 6874-6883, 2017.

[21] Dinesh Jayaraman, Ruohan Gao, and Kristen Grauman. Shapecodes: self-supervised feature learning by lifting views to viewgrids. In ECCV , pages 120-136, 2018. [22] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV , pages 609-617, 2017. [23] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In CVPR , pages 2701-2710, 2017. [24] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL , 2018. [25] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR , 2019. [26] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL , 2014. [27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017. [28] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 , 2016. [29] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. arXiv preprint arXiv:1904.01766 , 2019. [30] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR , pages 6077-6086, 2018b.

[31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NuerIPS , pages 91-99, 2015. [32] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP , 2014b. [33] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In CVPR , 2018. [34] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , pages 2961-2969, 2017. [35] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV , pages 201-216, 2018. [36] Gao Peng, Hongsheng Li, Haoxuan You, Zhengkai Jiang, Pan Lu, Steven Hoi, and Xiaogang Wang. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. arXiv preprint arXiv:1812.05252 , 2018. [37] Alex Wang and Kyunghyun Cho. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv:1902.04094 , 2019. [38] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV , pages 1422-1430, 2015. [39] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV , pages 649-666. Springer, 2016. [40] Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE PAMI , 38(9):1734-1747, 2015.

[41] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR , pages 2536-2544, 2016. [42] Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In CVPR , pages 1413-1421, 2015. [43] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV , pages 527-544. Springer, 2016. [44] Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291 , 2019. [45] Harm de Vries, Florian Strub, Jeremie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C. Courville. Modulating early visual processing by language. In NuerIPS , 2017.

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

ViLBERT：面向视觉–语言任务的任务无关视觉语言表示预训练

Abstract

摘要

1 Introduction

1 引言

2 Approach

2 方法

2.1 Preliminaries: Bidirectional Encoder Representations from Transformers (BERT)

2.1 预备：来自 Transformer 的双向编码器表示（BERT）

2.2 ViLBERT: Extending BERT to Jointly Represent Images and Text

2.2 ViLBERT：把 BERT 扩展到联合表示图像与文本

3 Experimental Settings

3 实验设定

3.1 Training ViLBERT

3.1 训练 ViLBERT

3.2 Vision-and-Language Transfer Tasks

3.2 视觉–语言迁移任务

4 Results and Analysis

4 结果与分析

5 Related Work

5 相关工作

6 Conclusion

6 结论

References

参考文献