LXMERT: Learning Cross-Modality Encoder Representations from Transformers

LXMERT：从 Transformer 学习跨模态编码器表示

Hao Tan         Mohit Bansal Affiliation:  UNC Chapel Hill Affiliation:  {haotan, mbansal}@cs.unc.edu

Abstract

摘要

Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, $\text{NLVR}^{2}$, and improve the previous best result by $22\%$ absolute ($54\%$ to $76\%$). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pre-training strategies significantly contribute to our strong results; and also present several attention visualizations for the different encoders.[^1]

视觉—语言推理需要理解视觉概念、语言语义，以及两者之间的对齐与关系。为此我们提出 LXMERT（Learning Cross-Modality Encoder Representations from Transformers）框架，专门学习这些跨模态联系。LXMERT 构建大规模 Transformer，包含三个编码器：物体关系编码器、语言编码器与跨模态编码器。为使模型具备连接视觉与语言语义的能力，我们在大量图文对上用五项有代表性的预训练任务训练：掩码语言建模、掩码物体预测（特征回归与标签分类）、跨模态匹配，以及图像问答。这些任务同时促进模态内与跨模态关系的学习。从预训练参数微调后，模型在两个视觉问答数据集（VQA 与 GQA）上达到当时最优。我们还将预训练的跨模态模型迁移到有挑战的视觉推理任务 $\text{NLVR}^{2}$，相对此前最佳结果绝对提升 $22\%$（$54\%$→$76\%$）。最后用详细消融说明新模型组件与预训练策略都显著贡献了强结果，并展示各编码器的若干注意力可视化。[^1]

1 Introduction

1 引言

Vision-and-language reasoning requires the understanding of visual contents, language semantics, and cross-modal alignments and relationships. There has been substantial past works in separately developing backbone models with better representations for the single modalities of vision and of language. For visual-content understanding, people have developed several backbone models Simonyan and Zisserman 2014; Szegedy et al. 2015; He et al. 2016 and shown their effectiveness on large vision datasets Deng et al. 2009; Lin et al. 2014; Krishna et al. 2017. Pioneering works Girshick et al. 2014; Xu et al. 2015 also show the generalizability of these pre-trained (especially on ImageNet) backbone models by fine-tuning them on different tasks. In terms of language understanding, last year, we witnessed strong progress towards building a universal backbone model with large-scale contextualized language model pre-training Peters et al. 2018; Radford et al. 2018; Devlin et al. 2019, which has improved performances on various tasks Rajpurkar et al. 2016; Wang et al. 2018 to significant levels. Despite these influential single-modality works, large-scale pretraining and fine-tuning studies for the modality-pair of vision and language are still under-developed.

视觉—语言推理需要理解视觉内容、语言语义以及跨模态对齐与关系。过去在单模态骨干上已有大量工作：视觉侧有多种骨干网络（Simonyan and Zisserman 2014；Szegedy et al. 2015；He et al. 2016），并在大规模视觉数据集上验证（Deng et al. 2009；Lin et al. 2014；Krishna et al. 2017）；开创性工作（Girshick et al. 2014；Xu et al. 2015）也表明在 ImageNet 等上预训练的骨干可微调到不同任务。语言侧，前一年大规模上下文语言模型预训练（Peters et al. 2018；Radford et al. 2018；Devlin et al. 2019）把多项任务（Rajpurkar et al. 2016；Wang et al. 2018）推到新高度。尽管单模态进展显著，视觉与语言这对模态的大规模预训练与微调仍相对不足。

Therefore, we present one of the first works in building a pre-trained vision-and-language cross-modality framework and show its strong performance on several datasets. We name this framework “LXMERT: Learning Cross-Modality Encoder Representations from Transformers” (pronounced: ‘leksmert’). This framework is modeled after recent BERT-style innovations while further adapted to useful cross-modality scenarios. Our new cross-modality model focuses on learning vision-and-language interactions, especially for representations of a single image and its descriptive sentence. It consists of three Transformer Vaswani et al. 2017 encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. In order to better learn the cross-modal alignments between vision and language, we next pre-train our model with five diverse representative tasks: (1) masked cross-modality language modeling, (2) masked object prediction via RoI-feature regression, (3) masked object prediction via detected-label classification, (4) cross-modality matching, and (5) image question answering. Different from single-modality pre-training (e.g., masked LM in BERT), this multi-modality pre-training allows our model to infer masked features either from the visible elements in the same modality, or from aligned components in the other modality. In this way, it helps build both intra-modality and cross-modality relationships.

因此，我们给出早期跨模态预训练框架之一，并在多个数据集上展示强表现。框架命名为 LXMERT：Learning Cross-Modality Encoder Representations from Transformers（读作 “leksmert”）。设计上借鉴 BERT 式思路，并面向跨模态场景改造。新模型聚焦视觉—语言交互，尤其是单张图像与其描述句的表示，由三个 Transformer（Vaswani et al. 2017）编码器组成：物体关系编码器、语言编码器与跨模态编码器。为更好学习跨模态对齐，接着用五项任务预训练：(1) 掩码跨模态语言建模；(2) 经 RoI 特征回归的掩码物体预测；(3) 经检测标签分类的掩码物体预测；(4) 跨模态匹配；(5) 图像问答。与单模态预训练（如 BERT 的掩码 LM）不同，多模态预训练允许模型从同模态可见元素或另一模态的对齐成分推断被掩成分，从而同时建立模态内与跨模态关系。

Empirically, we first evaluate LXMERT on two popular visual question-answering datasets, VQA Antol et al. 2015 and GQA Hudson and Manning 2019. Our model outperforms previous works in all question categories (e.g., Binary, Number, Open) and achieves state-of-the-art results in terms of overall accuracy. Further, to show the generalizability of our pre-trained model, we fine-tune LXMERT on a challenging visual reasoning task, Natural Language for Visual Reasoning for Real ($\text{NLVR}^{2}$) Suhr et al. 2019, where we do not use the natural images in their dataset for our pre-training, but fine-tune and evaluate on these challenging, real-world images. In this setup, we achieve a large improvement of $22\%$ absolute in accuracy ($54\%$ to $76\%$, i.e., 48% relative error reduction) and $30\%$ absolute in consistency ($12\%$ to $42\%$, i.e., 34% relative error reduction). Lastly, we conduct several analysis and ablation studies to prove the effectiveness of our model components and diverse pre-training tasks by removing them or comparing them with their alternative options. Especially, we use several ways to take the existing BERT model and its variants, and show their ineffectiveness in vision-and-language tasks, which overall proves the need of our new cross-modality pre-training framework. We also present several attention visualizations for the different language, object-relationship, and cross-modality encoders.

实验上，先在流行视觉问答数据集 VQA（Antol et al. 2015）与 GQA（Hudson and Manning 2019）上评测：各类问题（Binary、Number、Open）均超过此前工作，整体准确率达当时最优。为检验泛化，再在有挑战的 Natural Language for Visual Reasoning for Real（$\text{NLVR}^{2}$，Suhr et al. 2019）上微调——预训练并未使用该集的自然图像，却仍在这些真实世界图像上取得大幅提升：准确率绝对提高 $22\%$（$54\%$→$76\%$，相对误差约降 48%），一致性绝对提高 $30\%$（$12\%$→$42\%$，相对误差约降 34%）。随后通过移除组件或与替代方案对比，分析模型结构与多样预训练任务的作用；尤其用多种方式接入现成 BERT 及其变体，说明其在视觉—语言任务上的不足，从而印证新跨模态预训练框架的必要。文中还给出语言、物体关系与跨模态编码器的若干注意力可视化。

Figure 1: The LXMERT model for learning vision-and-language cross-modality representations. ‘Self’ and ‘Cross’ are abbreviations for self-attention sub-layers and cross-attention sub-layers, respectively. ‘FF’ denotes a feed-forward sub-layer.

图 1：用于学习视觉—语言跨模态表示的 LXMERT 模型。Self / Cross 分别表示自注意力与交叉注意力子层；FF 表示前馈子层。

2 Model Architecture

2 模型结构

We build our cross-modality model with self-attention and cross-attention layers following the recent progress in designing natural language processing models (e.g., transformers Vaswani et al. 2017). As shown in Fig. 1, our model takes two inputs: an image and its related sentence (e.g., a caption or a question). Each image is represented as a sequence of objects, and each sentence is represented as a sequence of words. Via careful design and combination of these self-attention and cross-attention layers, our model is able to generate language representations, image representations, and cross-modality representations from the inputs. Next, we describe the components of this model in detail.

我们按近年来 NLP 模型（如 Transformer，Vaswani et al. 2017）的设计，用自注意力与交叉注意力搭建跨模态模型。如图 1，模型接受图像及其相关句子（说明或问题）两路输入：图像表示为物体序列，句子表示为词序列。通过对这两类注意力层的细致组合，模型可从输入生成语言表示、图像表示与跨模态表示。下面分述各组件。

2.1 Input Embeddings

2.1 输入嵌入

The input embedding layers in LXMERT convert the inputs (i.e., an image and a sentence) into two sequences of features: word-level sentence embeddings and object-level image embeddings. These embedding features will be further processed by the latter encoding layers.

LXMERT 的输入嵌入层把输入（图像与句子）转为两段特征序列：词级句子嵌入与物体级图像嵌入，再交给后续编码层处理。

Word-Level Sentence Embeddings

A sentence is first split into words $\left\{w_{1},\ldots,w_{n}\right\}$ with length of $n$ by the same WordPiece tokenizer Wu et al. 2016 in Devlin et al. 2019. Next, as shown in Fig. 1, the word $w_{i}$ and its index $i$ ($w_{i}$’s absolute position in the sentence) are projected to vectors by embedding sub-layers, and then added to the index-aware word embeddings:

词级句子嵌入

句子先由与 Devlin et al. 2019 相同的 WordPiece 分词器（Wu et al. 2016）切成词 $\left\{w_{1},\ldots,w_{n}\right\}$，长度为 $n$。如图 1，词 $w_{i}$ 与其下标 $i$（在句中的绝对位置）经嵌入子层投影后相加，得到位置感知的词嵌入：

$$\displaystyle\hat{w}_{i} \\ \displaystyle=\mathrm{WordEmbed}\left(w_{i}\right) \\ \displaystyle\hat{u}_{i} \\ \displaystyle=\mathrm{IdxEmbed}\left(i\right) \\ \displaystyle h_{i} \\ \displaystyle=\mathrm{LayerNorm}\left(\hat{w}_{i}+\hat{u}_{i}\right)$$

Object-Level Image Embeddings

Instead of using the feature map output by a convolutional neural network, we follow Anderson et al. 2018 in taking the features of detected objects as the embeddings of images. Specifically, the object detector detects $m$ objects $\left\{o_{1},\ldots,o_{m}\right\}$ from the image (denoted by bounding boxes on the image in Fig. 1). Each object $o_{j}$ is represented by its position feature (i.e., bounding box coordinates) $p_{j}$ and its $2048$-dimensional region-of-interest (RoI) feature $f_{j}$. Instead of directly using the RoI feature $f_{j}$ without considering its position $p_{j}$ in Anderson et al. 2018, we learn a position-aware embedding $v_{j}$ by adding outputs of 2 fully-connected layers:

物体级图像嵌入

我们不直接用 CNN 的特征图，而遵循 Anderson et al. 2018，以检测到的物体特征作为图像嵌入。检测器从图中检出 $m$ 个物体 $\left\{o_{1},\ldots,o_{m}\right\}$（图 1 中以边界框标出）。每个物体 $o_{j}$ 由其位置特征（边界框坐标）$p_{j}$ 与 $2048$ 维 RoI 特征 $f_{j}$ 表示。与 Anderson et al. 2018 直接使用 $f_{j}$、不考虑 $p_{j}$ 不同，我们用两个全连接层的输出相加，学习位置感知嵌入 $v_{j}$：

$$\displaystyle\hat{f}_{j} \\ \displaystyle=\mathrm{LayerNorm}\left(W_{\textsc{f}}f_{j}+b_{\textsc{f}}\right) \\ \displaystyle\hat{p}_{j} \\ \displaystyle=\mathrm{LayerNorm}\left(W_{\textsc{p}}p_{j}+b_{\textsc{p}}\right) \\ \displaystyle v_{j} \\ \displaystyle=\left(\hat{f}_{j}+\hat{p}_{j}\right)/2 \tag{1}$$

In addition to providing spatial information in visual reasoning, the inclusion of positional information is necessary for our masked object prediction pre-training task (described in Sec. 3.1.2). Since the image embedding layer and the following attention layers are agnostic to the absolute indices of their inputs, the order of the object is not specified. Lastly, in Equation 1, the layer normalization is applied to the projected features before summation so as to balance the energy of the two different types of features.

位置信息除了服务视觉推理外，对第 3.1.2 节的掩码物体预测预训练也必不可少。图像嵌入层与后续注意力层对输入绝对下标不敏感，因此物体顺序无需指定。式 (1) 中，两类特征在求和前分别做层归一化，以平衡能量差异。

2.2 Encoders

2.2 编码器

We build our encoders, i.e., the language encoder, the object-relationship encoder, and the cross-modality encoder, mostly on the basis of two kinds of attention layers: self-attention layers and cross-attention layers. We first review the definition and notations of attention layers and then discuss how they form our encoders.

语言编码器、物体关系编码器与跨模态编码器，主要建立在自注意力与交叉注意力两类层之上。下面先回顾注意力记号，再说明如何组成各编码器。

Background: Attention Layers

Attention layers Bahdanau et al. 2014; Xu et al. 2015 aim to retrieve information from a set of context vectors $\{y_{j}\}$ related to a query vector $x$. An attention layer first calculates the matching score $a_{j}$ between the query vector $x$ and each context vector $y_{j}$. Scores are then normalized by softmax:

背景：注意力层

注意力层（Bahdanau et al. 2014；Xu et al. 2015）旨在从一组上下文向量 $\{y_{j}\}$ 中，相对查询向量 $x$ 检索信息。先算 $x$ 与各 $y_{j}$ 的匹配分数 $a_{j}$，再经 softmax 归一化：

$$\displaystyle\vskip-5.0pta_{j} \\ \displaystyle=\mathrm{score}(x,y_{j}) \\ \displaystyle\alpha_{j} \\ \displaystyle=\exp(a_{j})/\sum\nolimits_{k}\exp(a_{k})$$

The output of an attention layer is the weighted sum of the context vectors w.r.t. the softmax-normalized score: $\mathrm{Att}_{\textsc{x}\rightarrow\textsc{y}}\left(x,\{y_{j}\}\right)=\sum\nolimits_{j}\alpha_{j}y_{j}$. An attention layer is called self-attention when the query vector $x$ is in the set of context vectors $\{y_{j}\}$. Specifically, we use the multi-head attention following Transformer Vaswani et al. 2017.

注意力输出是上下文向量按 softmax 权重的加权和：$\mathrm{Att}_{\textsc{x}\rightarrow\textsc{y}}\left(x,\{y_{j}\}\right)=\sum\nolimits_{j}\alpha_{j}y_{j}$。当查询 $x$ 属于上下文集合 $\{y_{j}\}$ 时称为自注意力。具体采用 Transformer（Vaswani et al. 2017）中的多头注意力。

Single-Modality Encoders

After the embedding layers, we first apply two transformer encoders Vaswani et al. 2017, i.e., a language encoder and an object-relationship encoder, and each of them only focuses on a single modality (i.e., language or vision). Different from BERT Devlin et al. 2019, which applies the transformer encoder only to language inputs, we apply it to vision inputs as well (and to cross-modality inputs as described later below). Each layer (left dashed blocks in Fig. 1) in a single-modality encoder contains a self-attention (‘Self’) sub-layer and a feed-forward (‘FF’) sub-layer, where the feed-forward sub-layer is further composed of two fully-connected sub-layers. We take $\textbf{N}_{\textbf{L}}$ and $\textbf{N}_{\textbf{R}}$ layers in the language encoder and the object-relationship encoder, respectively. We add a residual connection and layer normalization (annotated by the ‘+’ sign in Fig. 1) after each sub-layer as in Vaswani et al. 2017.

单模态编码器

嵌入之后，先各用一个 Transformer 编码器（Vaswani et al. 2017）——语言编码器与物体关系编码器——各自只处理单一模态。与 BERT（Devlin et al. 2019）仅把 Transformer 用于语言不同，我们也把它用于视觉（以及后文的跨模态输入）。单模态编码器每层（图 1 左侧虚线框）含自注意力（Self）与前馈（FF）子层，前馈由两个全连接构成。语言与物体关系编码器分别堆叠 $\textbf{N}_{\textbf{L}}$、 $\textbf{N}_{\textbf{R}}$ 层。每个子层后加残差连接与层归一化（图 1 中 “+”），同 Vaswani et al. 2017。

Figure 2: Pre-training in LXMERT. The object RoI features and word tokens are masked. Our five pre-training tasks learn the feature representations based on these masked inputs. Special tokens are in brackets and classification labels are in braces.

图 2：LXMERT 的预训练。物体 RoI 特征与词 token 被掩码；五项预训练任务基于这些掩码输入学习表示。特殊 token 写在方括号内，分类标签写在花括号内。

Cross-Modality Encoder

Each cross-modality layer (the right dashed block in Fig. 1) in the cross-modality encoder consists of two self-attention sub-layers, one bi-directional cross-attention sub-layer, and two feed-forward sub-layers. We stack (i.e., using the output of $k$-th layer as the input of $(k\mbox{+}1)$-th layer) $\textbf{N}_{\textbf{X}}$ these cross-modality layers in our encoder implementation. Inside the $k$-th layer, the bi-directional cross-attention sub-layer (‘Cross’) is first applied, which contains two uni-directional cross-attention sub-layers: one from language to vision and one from vision to language. The query and context vectors are the outputs of the $(k\mbox{-}1)$-th layer (i.e., language features $\{h_{i}^{k-1}\}$ and vision features $\{v_{j}^{k-1}\}$):

跨模态编码器

跨模态编码器中每一层（图 1 右侧虚线框）含两个自注意力子层、一个双向交叉注意力子层，以及两个前馈子层。我们堆叠 $\textbf{N}_{\textbf{X}}$ 层（第 $k$ 层输出作为第 $(k{+}1)$ 层输入）。在第 $k$ 层内先做双向交叉注意力（Cross）：含语言→视觉与视觉→语言两个单向交叉注意力；查询与上下文取自第 $(k{-}1)$ 层输出（语言特征 $\{h_{i}^{k-1}\}$、视觉特征 $\{v_{j}^{k-1}\}$）：

$$\displaystyle\hat{h}^{k}_{i} \\ \displaystyle=\mathrm{CrossAtt}_{\textsc{l}\rightarrow\textsc{r}}\left({h}^{k-1}_{i}{,}\{{v}^{k-1}_{1}{,}\ldots,{v}^{k-1}_{m}\}\right) \\ \displaystyle\hat{v}^{k}_{j} \\ \displaystyle=\mathrm{CrossAtt}_{\textsc{r}\rightarrow\textsc{l}}\left({v}^{k-1}_{j}{,}\{{h}^{k-1}_{1}{,}\ldots,{h}^{k-1}_{n}\}\right)$$

The cross-attention sub-layer is used to exchange the information and align the entities between the two modalities in order to learn joint cross-modality representations. For further building internal connections, the self-attention sub-layers (‘Self’) are then applied to the output of the cross-attention sub-layer:

交叉注意力用于在两模态间交换信息并对齐实体，以学习联合跨模态表示。随后对交叉注意力输出再施加自注意力（Self），以加强内部联系：

$$\displaystyle\tilde{h}^{k}_{i} \\ \displaystyle=\mathrm{SelfAtt}_{\textsc{l}\rightarrow\textsc{l}}\left(\hat{h}^{k}_{i},\{\hat{h}^{k}_{1},\ldots,\hat{h}^{k}_{n}\}\right) \\ \displaystyle\tilde{v}^{k}_{j} \\ \displaystyle=\mathrm{SelfAtt}_{\textsc{r}\rightarrow\textsc{r}}\left(\hat{v}^{k}_{j},\{\hat{v}^{k}_{1},\ldots,\hat{v}^{k}_{m}\}\right)$$

Lastly, the $k$-th layer output $\{h^{k}_{i}\}$ and $\{v^{k}_{j}\}$ are produced by feed-forward sub-layers (‘FF’) on top of $\{\hat{h}^{k}_{i}\}$ and $\{\hat{v}^{k}_{j}\}$. We also add a residual connection and layer normalization after each sub-layer, similar to the single-modality encoders.

最后，第 $k$ 层输出 $\{h^{k}_{i}\}$、 $\{v^{k}_{j}\}$ 由前馈子层（FF）作用在 $\{\hat{h}^{k}_{i}\}$、 $\{\hat{v}^{k}_{j}\}$ 上得到。与单模态编码器一样，每个子层后同样加残差与层归一化。

2.3 Output Representations

2.3 输出表示

As shown in the right-most part of Fig. 1, our LXMERT cross-modality model has three outputs for language, vision, and cross-modality, respectively. The language and vision outputs are the feature sequences generated by the cross-modality encoder. For the cross-modality output, following the practice in Devlin et al. 2019, we append a special token [CLS] (denoted as the top yellow block in the bottom branch of Fig. 1) before the sentence words, and the corresponding feature vector of this special token in language feature sequences is used as the cross-modality output.

如图 1 最右侧所示，LXMERT 跨模态模型分别给出语言、视觉与跨模态三路输出。语言与视觉输出是跨模态编码器生成的特征序列。跨模态输出则沿用 Devlin et al. 2019：在句首附加特殊 token [CLS]（图 1 下支最上方黄色块），取语言特征序列中对应该 token 的向量作为跨模态输出。

Image Split	Images	Sentences (or Questions)
COCO-Cap	VG-Cap	VQA	GQA	VG-QA	All
MS COCO - VG	72K	361K	-	387K	-	-	0.75M
MS COCO $\cap$ VG	51K	256K	2.54M	271K	515K	724K	4.30M
VG - MS COCO	57K	-	2.85M	-	556K	718K	4.13M
All	180K	617K	5.39M	658K	1.07M	1.44M	9.18M

Table 1: Amount of data for pre-training. Each image has multiple sentences/questions. ‘Cap’ is caption. ‘VG’ is Visual Genome. Since MS COCO and VG share $51$K images, we list it separately to ensure disjoint image splits.

图像划分	图像数	句子（或问题）数
COCO-Cap	VG-Cap	VQA	GQA	VG-QA	合计
MS COCO − VG	72K	361K	-	387K	-	-	0.75M
MS COCO $\cap$ VG	51K	256K	2.54M	271K	515K	724K	4.30M
VG − MS COCO	57K	-	2.85M	-	556K	718K	4.13M
合计	180K	617K	5.39M	658K	1.07M	1.44M	9.18M

表 1：预训练数据量。每张图对应多句/多问。Cap 表示说明；VG 为 Visual Genome。MS COCO 与 VG 共享 $51$K 张图，单独列出以保证图像划分互斥。

3 Pre-Training Strategies

3 预训练策略

In order to learn a better initialization which understands connections between vision and language, we pre-train our model with different modality pre-training tasks on a large aggregated dataset.

为学到更好理解视觉—语言联系的初始化，我们在大规模聚合数据上用多种模态预训练任务训练模型。

3.1 Pre-Training Tasks

3.1 预训练任务

3.1.1 Language Task: Masked Cross-Modality LM

On the language side, we take the masked cross-modality language model (LM) task. As shown in the bottom branch of Fig. 2, the task setup is almost same to BERT Devlin et al. 2019: words are randomly masked with a probability of $0.15$ and the model is asked to predict these masked words. In addition to BERT where masked words are predicted from the non-masked words in the language modality, LXMERT, with its cross-modality model architecture, could predict masked words from the vision modality as well, so as to resolve ambiguity. For example, as shown in Fig. 2, it is hard to determine the masked word ‘carrot’ from its language context but the word choice is clear if the visual information is considered. Hence, it helps building connections from the vision modality to the language modality, and we refer to this task as masked cross-modality LM to emphasize this difference. We also show that loading BERT parameters into LXMERT will do harm to the pre-training procedure in Sec. 5.1 since BERT can perform relatively well in the language modality without learning these cross-modality connections.

3.1.1 语言任务：掩码跨模态 LM

语言侧采用掩码跨模态语言建模（LM）。如图 2 下支，设定几乎与 BERT（Devlin et al. 2019）相同：词以 $0.15$ 概率随机掩码，模型需预测被掩词。BERT 只能从语言模态未掩词推断；LXMERT 借助跨模态结构也可从视觉侧推断，从而消歧。例如图 2 中，仅凭语言上下文很难确定被掩词 “carrot”，结合视觉则一目了然。这有助于建立从视觉到语言的联系，故称为掩码跨模态 LM 以强调差异。第 5.1 节还表明：把 BERT 参数载入 LXMERT 会损害预训练，因为 BERT 可主要靠语言模态表现尚可，而不必学习这些跨模态联系。

3.1.2 Vision Task: Masked Object Prediction

As shown in the top branch of Fig. 2, we pre-train the vision side by randomly masking objects (i.e., masking RoI features with zeros) with a probability of $0.15$ and asking the model to predict proprieties of these masked objects. Similar to the language task (i.e., masked cross-modality LM), the model can infer the masked objects either from visible objects or from the language modality. Inferring the objects from the vision side helps learn the object relationships, and inferring from the language side helps learn the cross-modality alignments. Therefore, we perform two sub-tasks: RoI-Feature Regression regresses the object RoI feature $f_{j}$ with L2 loss, and Detected-Label Classification learns the labels of masked objects with cross-entropy loss. In the ‘Detected-Label Classification’ sub-task, although most of our pre-training images have object-level annotations, the ground truth labels of the annotated objects are inconsistent in different datasets (e.g., different number of label classes). For these reasons, we take detected labels output by Faster R-CNN Ren et al. 2015. Although detected labels are noisy, experimental results show that these labels contribute to pre-training in Sec. 5.3.

3.1.2 视觉任务：掩码物体预测

如图 2 上支，视觉侧以 $0.15$ 概率随机掩码物体（将 RoI 特征置零），并要求模型预测这些被掩物体的属性。与语言任务类似，模型可从可见物体或从语言模态推断。从视觉侧推断有助于学物体关系，从语言侧推断有助于学跨模态对齐。因此设两个子任务：RoI 特征回归用 L2 损失回归物体 RoI 特征 $f_{j}$；检测标签分类用交叉熵学习被掩物体标签。后一任务中，虽多数预训练图像有物体级标注，但各数据集标签体系不一致（类别数不同），故采用 Faster R-CNN（Ren et al. 2015）的检测标签。标签虽有噪声，第 5.3 节实验表明它们仍有助于预训练。

3.1.3 Cross-Modality Tasks

As shown in the middle-rightmost part of Fig. 2, to learn a strong cross-modality representation, we pre-train the LXMERT model with 2 tasks that explicitly need both language and vision modalities.

3.1.3 跨模态任务

如图 2 中右部，为学到强跨模态表示，再用两项显式需要语言与视觉的任务预训练 LXMERT。

Cross-Modality Matching

For each sentence, with a probability of $0.5$, we replace it with a mis-matched[^2] sentence. Then, we train a classifier to predict whether an image and a sentence match each other. This task is similar to ‘Next Sentence Prediction’ in BERT Devlin et al. 2019.

跨模态匹配

对每个句子，以 $0.5$ 概率替换为不匹配[^2]句子，再训练分类器判断图文是否匹配。类似 BERT（Devlin et al. 2019）的 “下一句预测”。

Image Question Answering (QA)

In order to enlarge the pre-training dataset (see details in Sec. 3.2), around $1/3$ sentences in the pre-training data are questions about the images. We ask the model to predict the answer to these image-related questions when the image and the question are matched (i.e., not randomly replaced in the cross-modality matching task). We show that pre-training with this image QA leads to a better cross-modality representation in Sec. 5.2.

图像问答（QA）

为扩大预训练数据（见第 3.2 节），约 $1/3$ 预训练句子是关于图像的问题。当图文匹配（即未在跨模态匹配中被随机替换）时，要求模型预测这些问题的答案。第 5.2 节表明，带图像 QA 的预训练能得到更好的跨模态表示。

3.2 Pre-Training Data

3.2 预训练数据

As shown in Table. 1, we aggregate pre-training data from five vision-and-language datasets whose images come from MS COCO Lin et al. 2014 or Visual Genome Krishna et al. 2017. Besides the two original captioning datasets, we also aggregate three large image question answering (image QA) datasets: VQA v2.0 Antol et al. 2015, GQA balanced version Hudson and Manning 2019, and VG-QA Zhu et al. 2016. We only collect train and dev splits in each dataset to avoid seeing any test data in pre-training. We conduct minimal pre-processing on the five datasets to create aligned image-and-sentence pairs. For each image question answering dataset, we take questions as sentences from the image-and-sentence data pairs and take answers as labels in the image QA pre-training task (described in Sec. 3.1.3). This provides us with a large aligned vision-and-language dataset of $9.18$M image-and-sentence pairs on $180$K distinct images. In terms of tokens, the pre-training data contain around $100$M words and $6.5$M image objects.

如表 1，我们聚合五个视觉—语言数据集，图像来自 MS COCO（Lin et al. 2014）或 Visual Genome（Krishna et al. 2017）。除两个原始说明数据集外，还并入三个大型图像问答集：VQA v2.0（Antol et al. 2015）、GQA 平衡版（Hudson and Manning 2019）与 VG-QA（Zhu et al. 2016）。仅收集各集的 train 与 dev 划分，避免预训练见到任何测试数据。对五份数据做最小预处理以得到对齐图文对。对图像问答集，把问题当作图文对中的句子，答案作为图像 QA 预训练任务的标签（见第 3.1.3 节）。最终得到约 $9.18$M 图文对、覆盖 $180$K 张不同图像；按 token 计约 $100$M 词与 $6.5$M 图像物体。

Method	VQA	GQA	$\text{NLVR}^{2}$
Binary	Number	Other	Accu	Binary	Open	Accu	Cons	Accu
Human	-	-	-	-	91.2	87.4	89.3	-	96.3
Image Only	-	-	-	-	36.1	1.74	17.8	7.40	51.9
Language Only	66.8	31.8	27.6	44.3	61.9	22.7	41.1	4.20	51.1
State-of-the-Art	85.8	53.7	60.7	70.4	76.0	40.4	57.1	12.0	53.5
LXMERT	88.2	54.2	63.1	72.5	77.8	45.0	60.3	42.1	76.2

Table 2: Test-set results. VQA/GQA results are reported on the ‘test-standard’ splits and $\text{NLVR}^{2}$ results are reported on the unreleased test set (‘Test-U’). The highest method results are in bold. Our LXMERT framework outperforms previous (comparable) state-of-the-art methods on all three datasets w.r.t. all metrics.

方法	VQA	GQA	$\text{NLVR}^{2}$
Binary	Number	Other	Accu	Binary	Open	Accu	Cons	Accu
Human	-	-	-	-	91.2	87.4	89.3	-	96.3
Image Only	-	-	-	-	36.1	1.74	17.8	7.40	51.9
Language Only	66.8	31.8	27.6	44.3	61.9	22.7	41.1	4.20	51.1
State-of-the-Art	85.8	53.7	60.7	70.4	76.0	40.4	57.1	12.0	53.5
LXMERT	88.2	54.2	63.1	72.5	77.8	45.0	60.3	42.1	76.2

表 2：测试集结果。VQA/GQA 报 test-standard；$\text{NLVR}^{2}$ 报未公开测试集（Test-U）。最优加粗。LXMERT 在三个数据集全部指标上超过此前（可比）最优方法。

3.3 Pre-Training Procedure

3.3 预训练流程

We pre-train our LXMERT model on the large aggregated dataset (discussed in Sec. 3.2) via the pre-training tasks (Sec. 3.1). The details about the data splits are in the Appendix. The input sentences are split by the WordPiece tokenizer Wu et al. 2016 provided in BERT Devlin et al. 2019. The objects are detected by Faster R-CNN Ren et al. 2015 which is pre-trained on Visual Genome (provided by Anderson et al. 2018). We do not fine-tune the Faster R-CNN detector and freeze it as a feature extractor. Different from detecting variable numbers of objects in Anderson et al. 2018, we consistently keep $36$ objects for each image to maximize the pre-training compute utilization by avoiding padding. For the model architecture, we set the numbers of layers $\textbf{N}_{\textbf{L}}$, $\textbf{N}_{\textbf{X}}$, and $\textbf{N}_{\textbf{R}}$ to $9$, $5$, and $5$ respectively.[^3] More layers are used in the language encoder to balance the visual features extracted from $101$-layer Faster R-CNN. The hidden size $768$ is the same as $\text{BERT}_{\text{BASE}}$. We pre-train all parameters in encoders and embedding layers from scratch (i.e., model parameters are randomly initialized or set to zero). We also show results of loading pre-trained BERT parameters in Sec. 5.1. LXMERT is pre-trained with multiple pre-training tasks and hence multiple losses are involved. We add these losses with equal weights. For the image QA pre-training tasks, we create a joint answer table with $9500$ answer candidates which roughly cover $90\%$ questions in all three image QA datasets.

我们在大规模聚合数据（第 3.2 节）上，用前述预训练任务（第 3.1 节）训练 LXMERT；划分细节见附录。输入句子用 BERT（Devlin et al. 2019）提供的 WordPiece 分词器（Wu et al. 2016）切分。物体由在 Visual Genome 上预训练的 Faster R-CNN（Ren et al. 2015；Anderson et al. 2018 提供）检测。检测器不微调，冻结为特征提取器。与 Anderson et al. 2018 检测可变数量物体不同，我们每张图固定保留 $36$ 个物体，避免 padding、提高算力利用率。结构上设 $\textbf{N}_{\textbf{L}}=9$、$\textbf{N}_{\textbf{X}}=5$、$\textbf{N}_{\textbf{R}}=5$。[^3] 语言编码器层数更多，以平衡来自 $101$ 层 Faster R-CNN 的视觉特征。隐层大小 $768$，与 $\text{BERT}_{\text{BASE}}$ 相同。编码器与嵌入层全部从零预训练（随机初始化或置零）。第 5.1 节另报加载预训练 BERT 参数的结果。多任务损失等权相加。图像 QA 预训练建有约 $9500$ 个候选答案的联合表，大约覆盖三份问答数据中 $90\%$ 的问题。

We take Adam Kingma and Ba 2014 as the optimizer with a linear-decayed learning-rate schedule Devlin et al. 2019 and a peak learning rate at $1e-4$. We train the model for $20$ epochs (i.e., roughly $670$K[^4] optimization steps) with a batch size of $256$. We only pre-train with image QA task (see Sec. 3.1.3) for the last $10$ epochs, because this task converges faster and empirically needs a smaller learning rate. The whole pre-training process takes $10$ days on $4$ Titan Xp.

优化器用 Adam（Kingma and Ba 2014），学习率按 Devlin et al. 2019 线性衰减，峰值 $1e{-4}$。训练 $20$ 个 epoch（约 $670$K[^4] 步），批大小 $256$。图像 QA 任务（第 3.1.3 节）只在后 $10$ 个 epoch 启用，因其收敛更快且经验上需要更小学习率。全程在 $4$ 块 Titan Xp 上约 $10$ 天。

Fine-tuning

Fine-tuning is fast and robust. We only perform necessary modification to our model with respect to different tasks (details in Sec. 4.2). We use a learning rate of $1e-5$ or $5e-5$, a batch size of $32$, and fine-tune the model from our pre-trained parameters for $4$ epochs.

微调

微调快速且稳健。仅按任务做必要改动（细节见第 4.2 节）。学习率取 $1e{-5}$ 或 $5e{-5}$，批大小 $32$，从预训练参数微调 $4$ 个 epoch。

4 Experimental Setup and Results

4 实验设置与结果

In this section, we first introduce the datasets that are used to evaluate our LXMERT framework and empirically compare our single-model results with previous best results.

本节先介绍用于评测 LXMERT 的数据集，再把单模型结果与此前最佳结果对照。

4.1 Evaluated Datasets

4.1 评测数据集

We use three datasets for evaluating our LXMERT framework: VQA v2.0 dataset Goyal et al. 2017, GQA Hudson and Manning 2019, and $\text{NLVR}^{2}$. See details in Appendix.

评测使用三个数据集：VQA v2.0（Goyal et al. 2017）、GQA（Hudson and Manning 2019）与 $\text{NLVR}^{2}$。细节见附录。

4.2 Implementation Details

4.2 实现细节

On VQA and GQA, we fine-tune our model from the pre-trained snapshot without data augmentation (analysis in Sec. 5.2). When training GQA, we only take raw questions and raw images as inputs and do not use other supervisions (e.g., functional programs and scene graphs). Since each datum in $\text{NLVR}^{2}$ has two natural images $\mathit{img}_{0},\mathit{img}_{1}$ and one language statement $s$, we use LXMERT to encode the two image-statement pairs $(\mathit{img}_{0},s)$ and $(\mathit{img}_{1},s)$, then train a classifier based on the concatenation of the two cross-modality outputs. More details in Appendix.

在 VQA 与 GQA 上，从预训练快照微调且不做数据增强（分析见第 5.2 节）。训练 GQA 时只以原始问题与原始图像为输入，不用功能程序、场景图等额外监督。$\text{NLVR}^{2}$ 每条样本含两张自然图像 $\mathit{img}_{0},\mathit{img}_{1}$ 与一条语句 $s$，我们用 LXMERT 分别编码 $(\mathit{img}_{0},s)$ 与 $(\mathit{img}_{1},s)$，再拼接两路跨模态输出训练分类器。更多细节见附录。

4.3 Empirical Comparison Results

4.3 经验对比结果

We compare our single-model results with previous best published results on VQA/GQA test-standard sets and $\text{NLVR}^{2}$ public test set. Besides previous state-of-the-art (SotA) methods, we also show the human performance and image-only/language-only results when available.

在 VQA/GQA 的 test-standard 与 $\text{NLVR}^{2}$ 公开测试集上，把单模型结果与此前最佳发表结果对比；除 SotA 外，在可得时也给出人类表现与仅图像/仅语言结果。

VQA

The SotA result is BAN+Counter in Kim et al. 2018, which achieves the best accuracy among other recent works: MFH Yu et al. 2018, Pythia Jiang et al. 2018, DFAF Gao et al. 2019a, and Cycle-Consistency Shah et al. 2019.[^5] LXMERT improves the SotA overall accuracy (‘Accu’ in Table 2) by $2.1\%$ and has $2.4\%$ improvement on the ‘Binary’/‘Other’ question sub-categories. Although LXMERT does not explicitly take a counting module as in BAN+Counter, our result on the counting-related questions (‘Number’) is still equal or better.[^6]

VQA

SotA 为 Kim et al. 2018 的 BAN+Counter，优于 MFH（Yu et al. 2018）、Pythia（Jiang et al. 2018）、DFAF（Gao et al. 2019a）、Cycle-Consistency（Shah et al. 2019）等。[^5] LXMERT 将整体准确率（表 2 的 Accu）再提高 $2.1\%$，Binary/Other 子类提高 $2.4\%$。虽未像 BAN+Counter 显式加入计数模块，在计数相关问题（Number）上仍持平或更好。[^6]

GQA

The GQA Hudson and Manning 2019 SotA result is taken from BAN Kim et al. 2018 on the public leaderbaord. Our $3.2\%$ accuracy gain over the SotA GQA method is higher than VQA, possibly because GQA requires more visual reasoning. Thus our framework, with novel encoders and cross-modality pre-training, is suitable and achieves a $4.6\%$ improvement on open-domain questions (‘Open’ in Table 2).[^7]

GQA

GQA（Hudson and Manning 2019）公开榜 SotA 取自 BAN（Kim et al. 2018）。相对该 SotA，$3.2\%$ 的准确率增益高于 VQA，可能因 GQA 更依赖视觉推理；带新编码器与跨模态预训练的本框架因此更合适，在开放域问题（表 2 的 Open）上提高 $4.6\%$。[^7]

$\text{NLVR}^{2}$

$\text{NLVR}^{2}$ Suhr et al. 2019 is a challenging visual reasoning dataset where some existing approaches Hu et al. 2017; Perez et al. 2018 fail, and the SotA method is ‘MaxEnt’ in Suhr et al. 2019. The failure of existing methods (and our model w/o pre-training in Sec. 5.1) indicates that the connection between vision and language may not be end-to-end learned in a complex vision-and-language task without large-scale pre-training. However, with our novel pre-training strategies in building the cross-modality connections, we significantly improve the accuracy (‘Accu’ of 76.2% on unreleased test set ‘Test-U’, in Table 2) by $22\%$. Another evaluation metric consistency measures the proportion of unique sentences for which all related image pairs[^8] are correctly predicted. Our LXMERT model improves consistency (‘Cons’) to 42.1% (i.e., by $3.5$ times).[^9]

$\text{NLVR}^{2}$

$\text{NLVR}^{2}$（Suhr et al. 2019）是有挑战的视觉推理集，部分已有方法（Hu et al. 2017；Perez et al. 2018）失败，SotA 为 Suhr et al. 2019 的 MaxEnt。已有方法（以及第 5.1 节无预训练的本模型）的失败表明：在复杂视觉—语言任务上，若不做大规模预训练，视觉与语言的联系未必能端到端学好。借助建立跨模态联系的预训练策略，我们把准确率（表 2 未公开 Test-U 上 Accu $76.2\%$）绝对提高 $22\%$。另一指标一致性衡量：对唯一语句，其全部相关图像对[^8]是否都被正确预测的比例；LXMERT 将一致性（Cons）提到 $42.1\%$（约 $3.5$ 倍）。[^9]

Method	VQA	GQA	$\text{NLVR}^{2}$
LSTM + BUTD	63.1	50.0	52.6
BERT + BUTD	62.8	52.1	51.9
BERT + 1 CrossAtt	64.6	55.5	52.4
BERT + 2 CrossAtt	65.8	56.1	50.9
BERT + 3 CrossAtt	66.4	56.6	50.9
BERT + 4 CrossAtt	66.4	56.0	50.9
BERT + 5 CrossAtt	66.5	56.3	50.9
Train + BERT	65.5	56.2	50.9
Train + scratch	65.1	50.0	50.9
Pre-train + BERT	68.8	58.3	70.1
Pre-train + scratch	69.9	60.0	74.9

Table 3: Dev-set accuracy of using BERT.

方法	VQA	GQA	$\text{NLVR}^{2}$
LSTM + BUTD	63.1	50.0	52.6
BERT + BUTD	62.8	52.1	51.9
BERT + 1 CrossAtt	64.6	55.5	52.4
BERT + 2 CrossAtt	65.8	56.1	50.9
BERT + 3 CrossAtt	66.4	56.6	50.9
BERT + 4 CrossAtt	66.4	56.0	50.9
BERT + 5 CrossAtt	66.5	56.3	50.9
Train + BERT	65.5	56.2	50.9
Train + scratch	65.1	50.0	50.9
Pre-train + BERT	68.8	58.3	70.1
Pre-train + scratch	69.9	60.0	74.9

表 3：使用 BERT 时的开发集准确率。

5 Analysis

5 分析

In this section, we analyze our LXMERT framework by comparing it with some alternative choices or by excluding certain model components/pre-training strategies.

本节通过与替代方案对比，或剔除某些模型组件/预训练策略，分析 LXMERT。

5.1 BERT versus LXMERT

5.1 BERT 对比 LXMERT

BERT Devlin et al. 2019 is a pre-trained language encoder which improves several language tasks. As shown in Table 3, we discuss several ways to incorporate a $\text{BERT}_{\text{BASE}}$ pre-trained model for vision-language tasks and empirically compare it with our LXMERT approach. Although our full model achieves accuracy of $74.9\%$ on $\text{NLVR}^{2}$, all results without LXMERT pre-training is around $22\%$ absolute lower.

BERT（Devlin et al. 2019）是提升多项语言任务的预训练语言编码器。如表 3，我们讨论多种把 $\text{BERT}_{\text{BASE}}$ 用于视觉—语言任务的方式，并与 LXMERT 对照。完整模型在 $\text{NLVR}^{2}$ 上准确率 $74.9\%$，而所有不做 LXMERT 预训练的结果大约低 $22$ 个百分点。

BERT+BUTD

Bottom-Up and Top-Down (BUTD) attention Anderson et al. 2018 method encodes questions with GRU Chung et al. 2015, then attends to object RoI features $\{f_{j}\}$ to predict the answer. We apply BERT to BUTD by replacing its GRU language encoder with BERT. As shown in the first block of Table. 3, results of BERT encoder is comparable to LSTM encoder.

BERT+BUTD

自下而上—自上而下（BUTD）注意力（Anderson et al. 2018）用 GRU（Chung et al. 2015）编码问题，再对物体 RoI 特征 $\{f_{j}\}$ 做注意力以预测答案。我们用 BERT 替换其 GRU 语言编码器。如表 3 第一块，BERT 编码器结果与 LSTM 相当。

BERT+CrossAtt

Since BUTD only takes the raw RoI features $\{f_{j}\}$ without considering the object positions $\{p_{j}\}$ and object relationships, we enhance BERT+BUTD with our novel position-aware object embedding (in Sec. 2.1) and cross-modality layers (in Sec. 2.2). As shown in the second block of Table 3, the result of $1$ cross-modality layer is better than BUTD, while stacking more cross-modality layers further improves it. However, without our cross-modality pre-training (BERT is language-only pre-trained), results become stationary after adding $3$ cross-attention layers and have a $3.4\%$ gap to our full LXMERT framework (the last bold row in Table 3).

BERT+CrossAtt

BUTD 只用原始 RoI 特征 $\{f_{j}\}$，未考虑物体位置 $\{p_{j}\}$ 与物体关系。我们为 BERT+BUTD 加上位置感知物体嵌入（第 2.1 节）与跨模态层（第 2.2 节）。如表 3 第二块，$1$ 层跨模态已优于 BUTD，继续堆叠还能再涨；但若无跨模态预训练（BERT 仅语言预训练），加到 $3$ 层交叉注意力后趋于平台，相对完整 LXMERT（表 3 末行加粗）仍差 $3.4\%$。

BERT+LXMERT

We also try loading BERT parameters[^10] into LXMERT, and use it in model training (i.e., without LXMERT pre-training) or in pre-training. We show results in the last block of Table. 3. Compared to the ‘from scratch’ (i.e., model parameters are randomly initialized) approach, BERT improves the fine-tuning results but it shows weaker results than our full model. Empirically, pre-training LXMERT initialized with BERT parameters has lower (i.e., better) pre-training loss for the first $3$ pre-training epochs but was then caught up by our ‘from scratch’ approach. A possible reason is that BERT is already pre-trained with single-modality masked language model, and thus could do well based only on the language modality without considering the connection to the vision modality (as discussed in Sec. 3.1.1).

BERT+LXMERT

我们也尝试把 BERT 参数[^10]载入 LXMERT，用于直接训练（不做 LXMERT 预训练）或用于预训练，结果见表 3 最后一块。相对 “from scratch”（随机初始化），BERT 能改善微调，但仍弱于完整模型。经验上，用 BERT 初始化的 LXMERT 预训练在前 $3$ 个 epoch 损失更低（更好），之后被 from-scratch 追上。一种可能是：BERT 已在单模态掩码 LM 上预训练，仅靠语言模态即可表现不错，而不必建立与视觉的联系（见第 3.1.1 节）。

Method	VQA	GQA	$\text{NLVR}^{2}$
1. P20 + DA	68.0	58.1	-
2. P20 + FT	68.9	58.2	72.4
3. P10+QA10 + DA	69.1	59.2	-
4. P10+QA10 + FT	69.9	60.0	74.9

Table 4: Dev-set accuracy showing the importance of the image-QA pre-training task. P10 means pre-training without the image-QA loss for $10$ epochs while QA10 means pre-training with the image-QA loss. DA and FT mean fine-tuning with and without Data Augmentation, resp.

方法	VQA	GQA	$\text{NLVR}^{2}$
1. P20 + DA	68.0	58.1	-
2. P20 + FT	68.9	58.2	72.4
3. P10+QA10 + DA	69.1	59.2	-
4. P10+QA10 + FT	69.9	60.0	74.9

表 4：开发集准确率，说明图像 QA 预训练任务的重要性。P10 表示不含图像 QA 损失预训练 $10$ 个 epoch；QA10 表示含该损失。DA / FT 分别表示带/不带数据增强的微调。

5.2 Effect of the Image QA Pre-training Task

5.2 图像 QA 预训练任务的作用

We show the importance of image QA pre-training task (introduced in Sec. 3.1.3) by excluding it or comparing it with its alternative: data augmentation.

通过剔除图像 QA 预训练（第 3.1.3 节），或与替代方案——数据增强——对比，说明其重要性。

Pre-training w/ or w/o Image QA

To fairly compare with our original pre-training procedure (10 epochs w/o QA + 10 epochs w/ QA, details in Sec. 3.3) , we pre-train LXMERT model without image QA task for $20$ epochs. As shown in Table 4 rows 2 and 4, pre-training with QA loss improves the result on all three datasets. The $2.1\%$ improvement on $\text{NLVR}^{2}$ shows the stronger representations learned with image-QA pre-training, since all data (images and statements) in $\text{NLVR}^{2}$ are not used in pre-training.

有/无图像 QA 的预训练

为公平对照原始流程（$10$ epoch 无 QA + $10$ epoch 有 QA，见第 3.3 节），我们在无图像 QA 任务下预训练 $20$ 个 epoch。如表 4 第 2、4 行，带 QA 损失在三个数据集上都更好。$\text{NLVR}^{2}$ 上 $2.1\%$ 的提升说明学到了更强表示，因为该集的图像与语句均未参与预训练。

Method	VQA	GQA	$\text{NLVR}^{2}$
1. No Vision Tasks	66.3	57.1	50.9
2. Feat	69.2	59.5	72.9
3. Label	69.5	59.3	73.5
4. Feat + Label	69.9	60.0	74.9

Table 5: Dev-set accuracy of different vision pre-training tasks. ‘Feat’ is RoI-feature regression; ‘Label’ is detected-label classification.

方法	VQA	GQA	$\text{NLVR}^{2}$
1. No Vision Tasks	66.3	57.1	50.9
2. Feat	69.2	59.5	72.9
3. Label	69.5	59.3	73.5
4. Feat + Label	69.9	60.0	74.9

表 5：不同视觉预训练任务的开发集准确率。Feat 为 RoI 特征回归；Label 为检测标签分类。

Pre-training versus Data Augmentation

Data augmentation (DA) is a technique which is used in several VQA implementations Anderson et al. 2018; Kim et al. 2018; Jiang et al. 2018. It increases the amount of training data by adding questions from other image QA datasets. Our LXMERT framework instead uses multiple QA datasets in pre-training and is fine-tuned only on one specific dataset. Since the overall amounts of data used in pre-training and DA are similar, we thus can fairly compare these two strategies, and results show that our QA pre-training approach outperforms DA. We first exclude the QA task in our pre-training and show the results of DA fine-tuning. As shown in Table. 4 row 1, DA fine-tuning decreases the results compared to non-DA fine-tuning in row 2. Next, we use DA after QA-pre-training (row 3) and DA also drops the results.

预训练对比数据增强

数据增强（DA）常见于多种 VQA 实现（Anderson et al. 2018；Kim et al. 2018；Jiang et al. 2018）：把其他图像问答集的问题并入训练以扩数据。LXMERT 则在预训练阶段使用多份 QA 数据，微调只在单一目标集上。两者总体数据量相近，可公平比较；结果显示 QA 预训练优于 DA。先去掉预训练中的 QA，再做 DA 微调：表 4 第 1 行相对第 2 行非 DA 微调反而更差。在 QA 预训练后再做 DA（第 3 行）同样掉点。

5.3 Effect of Vision Pre-training tasks

5.3 视觉预训练任务的作用

We analyze the effect of different vision pre-training tasks in Table 5. Without any vision tasks in pre-training (i.e., only using the language and cross-modality pre-training tasks), the results (row $1$ of Table 5) are similar to BERT+3 CrossAtt in Table 3. The two visual pre-training tasks (i.e., RoI-feature regression and detected-label classification) could get reasonable results (row $2$ and row $3$) on their own, and jointly pre-training with these two tasks achieves the highest results (row $4$).

表 5 分析不同视觉预训练任务。预训练完全不做视觉任务（只用语言与跨模态任务）时，结果（表 5 第 1 行）类似表 3 的 BERT+3 CrossAtt。两项视觉任务（RoI 特征回归与检测标签分类）单独也已不错（第 2、3 行），联合训练最高（第 4 行）。

5.4 Visualizing LXMERT Behavior

5.4 可视化 LXMERT 行为

In the appendix, we show the behavior of LXMERT by visualizing its attention graphs in the language encoder, object-relationship encoder, and cross-modality encoder, respectively.

附录中通过语言编码器、物体关系编码器与跨模态编码器的注意力图，展示 LXMERT 的行为。

6 Related Work

6 相关工作

Model Architecture

Our model is closely related to three ideas: bi-directional attention, Transformer, and BUTD. Lu et al. 2016 applies bi-directional attention to the vision-and-language tasks while its concurrent work BiDAF Seo et al. 2017 adds modeling layers in solving reading comprehension. Transformer Vaswani et al. 2017 is first used in machine translation, we utilize it as our single-modality encoders and design our cross-modality encoder based on it. BUTD Anderson et al. 2018 embeds images with the object RoI features, we extend it with object positional embeddings and object relationship encoders.

模型结构

本模型与三个思路密切相关：双向注意力、Transformer 与 BUTD。Lu et al. 2016 把双向注意力用于视觉—语言，同期 BiDAF（Seo et al. 2017）在阅读理解中加入建模层。Transformer（Vaswani et al. 2017）最初用于机器翻译；我们用作单模态编码器，并据此设计跨模态编码器。BUTD（Anderson et al. 2018）用物体 RoI 特征嵌入图像；我们扩展以物体位置嵌入与物体关系编码器。

Pre-training

After ELMo Peters et al. 2018, GPT Radford et al. 2018, and BERT Devlin et al. 2019 show improvements in language understanding tasks with large-scale pre-trained language model, progress has been made towards the cross-modality pre-training. XLM Lample and Conneau 2019 learns the joint cross-lingual representations by leveraging the monolingual data and parallel data. VideoBert Sun et al. 2019 takes masked LM on the concatenation of language words and visual tokens, where the visual tokens are converted from video frames by vector quantization. However, these methods are still based on a single transformer encoder and BERT-stype token-based pre-training, thus we develop a new model architecture and novel pre-training tasks to satisfy the need of cross-modality tasks.

预训练

ELMo（Peters et al. 2018）、GPT（Radford et al. 2018）与 BERT（Devlin et al. 2019）以大规模预训练语言模型提升语言理解之后，跨模态预训练也开始推进。XLM（Lample and Conneau 2019）利用单语与平行数据学习跨语言联合表示。VideoBert（Sun et al. 2019）对语言词与视觉 token 的拼接做掩码 LM，视觉 token 由视频帧量化得到。这些方法仍基于单一 Transformer 与 BERT 式 token 预训练，因而我们另建模型结构与预训练任务，以更好服务跨模态任务。

Recent works since our EMNLP submission

This version of our paper (and all current results) was submitted to EMNLP[^11] and was used to participate in the VQA and GQA challenges in May 2019. Since our EMNLP submission, a few other useful preprints have recently been released (in August) on similar cross-modality pre-training directions: ViLBERT Lu et al. 2019 and VisualBERT Li et al. 2019. Our LXMERT methods differs from them in multiple ways: we use a more detailed, multi-component design for the cross-modality model (i.e., with an object-relationship encoder and cross-modality layers) and we employ additional, useful pre-training tasks (i.e., RoI-feature regression and image question answering). These differences result in the current best performance (on overlapping reported tasks): a margin of 1.5% accuracy on VQA 2.0 and a margin of 9% accuracy on $\text{NLVR}^{2}$ (and 15% in consistency). LXMERT is also the only method which ranks in the top-3 on both the VQA and GQA challenges among more than 90 teams. We provide a detailed analysis to show how these additional pre-training tasks contribute to the fine-tuning performance in Sec. 5.2 and Sec. 5.3.

EMNLP 投稿之后的近期工作

本稿（及全部当前结果）投往 EMNLP[^11]，并用于 2019 年 5 月的 VQA 与 GQA 挑战。投稿后，8 月前后出现若干相近方向的预印本：ViLBERT（Lu et al. 2019）与 VisualBERT（Li et al. 2019）。LXMERT 与之有多处不同：跨模态模型组件更细（含物体关系编码器与跨模态层），并采用额外有效的预训练任务（RoI 特征回归与图像问答）。这些差异带来当前最佳表现（在重叠任务上）：VQA 2.0 准确率高约 1.5 个百分点，$\text{NLVR}^{2}$ 准确率高约 9 个百分点（一致性高约 15 个百分点）。在逾 90 支队伍中，LXMERT 也是唯一同时进入 VQA 与 GQA 挑战前三的方法。第 5.2、5.3 节详细分析了这些额外预训练任务对微调的贡献。

7 Conclusion

7 结论

We presented a cross-modality framework, LXMERT, for learning the connections between vision and language. We build the model based on Transfermer encoders and our novel cross-modality encoder. This model is then pre-trained with diverse pre-training tasks on a large-scale dataset of image-and-sentence pairs. Empirically, we show state-of-the-art results on two image QA datasets (i.e., VQA and GQA) and show the model generalizability with a $22\%$ improvement on the challenging visual reasoning dataset of $\text{NLVR}^{2}$. We also show the effectiveness of several model components and training methods via detailed analysis and ablation studies.

我们提出跨模态框架 LXMERT，用于学习视觉与语言之间的联系。模型基于 Transformer 编码器与新的跨模态编码器，并在大规模图文对上用多样预训练任务训练。经验上，在两个图像问答数据集（VQA、GQA）上达到当时最优，并在有挑战的 $\text{NLVR}^{2}$ 上取得 $22\%$ 提升，体现泛化。详细分析与消融也表明若干模型组件与训练方法有效。

Acknowledgments

致谢

We thank the reviewers for their helpful comments. This work was supported by ARO-YIP Award #W911NF-18-1-0336, and awards from Google, Facebook, Salesforce, and Adobe. The views, opinions, and/or findings contained in this article are those of the authors and should not be interpreted as representing the official views or policies, either expressed or implied, of the funding agency. We also thank Alane Suhr for evaluation on NLVR2.

感谢审稿人的有益意见。本工作获 ARO-YIP Award #W911NF-18-1-0336，以及 Google、Facebook、Salesforce、Adobe 的资助。文中观点不代表资助方官方立场。亦感谢 Alane Suhr 协助 NLVR2 评测。

References

参考文献

Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6077–6086.

Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425–2433.

Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 .

Chung et al. (2015) Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2015. Gated feedback recurrent neural networks. In International Conference on Machine Learning , pages 2067–2075.

Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248–255. IEEE.

Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT .

Gao et al. (2019a) Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven C. H. Hoi, Xiaogang Wang, and Hongsheng Li. 2019a. Dynamic fusion with intra- and inter-modality attention flow for visual question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

Gao et al. (2019b) Peng Gao, Haoxuan You, Zhanpeng Zhang, Xiaogang Wang, and Hongsheng Li. 2019b. Multi-modality latent interaction network for visual question answering. arXiv preprint arXiv:1908.04289 .

Girshick et al. (2014) Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 580–587.

Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6904–6913.

He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770–778.

Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. https://openreview.net/forum?id=Bk0MRI5lg .

Hoover et al. (2019) Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann. 2019. exbert: A visual analysis tool to explore learned representations in transformers models. arXiv preprint arXiv:1910.05276 .

Hu et al. (2017) Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision , pages 804–813.

Hudson and Manning (2019) Drew A Hudson and Christopher D Manning. 2019. Gqa: a new dataset for compositional question answering over real-world images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition .

Jiang et al. (2018) Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2018. Pythia v0. 1: the winning entry to the vqa challenge 2018. arXiv preprint arXiv:1807.09956 .

Kim et al. (2018) Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018. Bilinear attention networks. In Advances in Neural Information Processing Systems , pages 1564–1574.

Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In International Conference on Learning Representations .

Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision , 123(1):32–73.

Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291 .

Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 .

Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740–755. Springer.

Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265 .

Lu et al. (2016) Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Information Processing Systems , pages 289–297.

Perez et al. (2018) Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence .

Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT , pages 2227–2237.

Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf .

Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 2383–2392.

Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems , pages 91–99.

Seo et al. (2017) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In International Conference on Learning Representations .

Shah et al. (2019) Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. 2019. Cycle-consistency for robust visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition .

Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 .

Suhr et al. (2019) Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics .

Sun et al. (2019) Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. arXiv preprint arXiv:1904.01766 .

Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1–9.

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems , pages 5998–6008.

Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018 , page 353.

Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 .

Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning , pages 2048–2057.

Yu et al. (2019a) Zhou Yu, Yuhao Cui, Jun Yu, Dacheng Tao, and Qi Tian. 2019a. Multimodal unified attention networks for vision-and-language interactions. arXiv preprint arXiv:1908.04107 .

Yu et al. (2019b) Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. 2019b. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6281–6290.

Yu et al. (2018) Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. 2018. Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering. IEEE Transactions on Neural Networks and Learning Systems , 29(12):5947–5959.

Zhu et al. (2016) Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4995–5004.

Appendix

附录

Appendix A Evaluated Datasets Description

附录 A 评测数据集说明

We use three datasets for evaluating our LXMERT framework.

评测 LXMERT 使用以下三个数据集。

VQA

The goal of visual question answering (VQA) Antol et al. 2015 is to answer a natural language question related to an image. We take VQA v2.0 dataset Goyal et al. 2017 which reduces the answer bias compared to VQA v1.0. The dataset contains an average of $5.4$ questions per image and the total amount of questions is $1.1$M.

VQA

视觉问答（VQA，Antol et al. 2015）要求回答与图像相关的自然语言问题。我们采用相对 v1.0 降低答案偏置的 VQA v2.0（Goyal et al. 2017）：平均每图 $5.4$ 个问题，共约 $1.1$M 问。

GQA

The task of GQA Hudson and Manning 2019 is same as VQA (i.e., answer single-image related questions), but GQA requires more reasoning skills (e.g., spatial understanding and multi-step inference). $22$M questions in the dataset are generated from ground truth image scene graph to explicitly control the question quality.

GQA

GQA（Hudson and Manning 2019）任务与 VQA 相同（单图相关问答），但更强调推理（如空间理解与多步推断）。约 $22$M 问题由真值场景图生成，以显式控制问题质量。

$\text{NLVR}^{2}$

Since the previous two datasets are used in pre-training for increasing the amount of pre-training data to a certain scale, we evaluate our LXMERT framework on another challenging visual reasoning dataset $\text{NLVR}^{2}$ where all the sentences and images are not covered in pre-training. Each datum in $\text{NLVR}^{2}$ contains two related natural images and one natural language statement. The task is to predict whether the statement correctly describes these two images or not. $\text{NLVR}^{2}$ has $86$K, $7$K, $7$K data in training, development, and test sets, respectively.

$\text{NLVR}^{2}$

前两个数据集已用于预训练以扩规模，因此另在 $\text{NLVR}^{2}$ 上评测：其语句与图像均未出现在预训练中。每条样本含两张相关自然图像与一条自然语言陈述，任务是判断陈述是否正确描述这两张图。训练/开发/测试分别约 $86$K / $7$K / $7$K 条。

Appendix B Details of $\text{NLVR}^{2}$ Fine-tuning

附录 B $\text{NLVR}^{2}$ 微调细节

Each datum in $\text{NLVR}^{2}$ consists of a two-image pair ($img_{0}$, $img_{1}$), one statement $s$, and a ground truth label $y^{*}$ indicating whether the statement correctly describe the two images. The task is to predict the label $y$ given the images and the statement.

$\text{NLVR}^{2}$ 每条数据含图像对 $(img_{0}, img_{1})$、陈述 $s$，以及真值标签 $y^{*}$（陈述是否正确描述两图）。任务是在给定图像与陈述时预测 $y$。

To use our LXMERT model on $\text{NLVR}^{2}$, we concatenate the cross-modality representations of the two images and then build the classifier with GeLU activationHendrycks and Gimpel 2016. Suppose that $\mathrm{LXMERT}(\mathit{img},\mathit{sent})$ is the single-vector cross-modality representation, the predicted probability is:

使用 LXMERT 时，拼接两图的跨模态表示，再用 GeLU 激活（Hendrycks and Gimpel 2016）建分类器。设 $\mathrm{LXMERT}(\mathit{img},\mathit{sent})$ 为单向量跨模态表示，则预测概率为：

$$\displaystyle x_{0} \\ \displaystyle=\mathrm{LXMERT}(\mathit{img}_{0},s) \\ \displaystyle x_{1} \\ \displaystyle=\mathrm{LXMERT}(\mathit{img}_{1},s) \\ \displaystyle z^{0} \\ \displaystyle=W_{0}[x_{0};x_{1}]+b_{0} \\ \displaystyle z^{1} \\ \displaystyle=\mathrm{LayerNorm}\left(\mathrm{GeLU}(z^{0})\right) \\ \displaystyle\mathit{prob} \\ \displaystyle=\sigma(W_{1}z^{1}+b_{1})$$

where $\sigma$ is sigmoid function. The model is optimized by maximizing the log-likelihood, which is equivalent to minimize the binary cross entropy loss:

其中 $\sigma$ 为 sigmoid。最大化对数似然等价于最小化二元交叉熵：

$$\displaystyle\mathcal{L}=\mbox{-}y^{*}\log\mathit{prob}-(1-y^{*})\log(1-\mathit{prob})$$

Appendix C Training, Validation, and Testing Splits

附录 C 训练、验证与测试划分

We carefully split each dataset to ensure that all testing images are not involved in any pre-training or fine-tuning steps. Our data splits for each dataset and reproducible code are available at https://github.com/airsplay/lxmert.

我们仔细划分各数据集，确保任何测试图像都不进入预训练或微调。各集划分与可复现代码见 https://github.com/airsplay/lxmert。

LXMERT Pre-Traininig

Since MS COCO has a relative large validation set, we sample a set of $5$k images from the MS COCO validation set as the mini-validation (minival) set. The rest of the images in training and validation sets (i.e., COCO training images, COCO validation images besides minival, and all the other images in Visual Genome) are used in pre-training. Although the captions and questions of the MS COCO test sets are available, we exclude all of them to make sure that testing images are not seen in pre-training.

LXMERT 预训练

MS COCO 验证集相对较大，我们从中采样 $5$k 张图作为迷你验证（minival）；其余训练与验证图像（COCO 训练、除 minival 外的 COCO 验证，以及 Visual Genome 全部图像）用于预训练。虽可取得 MS COCO 测试集的说明与问题，但全部排除，以保证测试图像未在预训练中出现。

Fine-tuning

For training and validating VQA v2.0, we take the same split convention as in our LXMERT pre-training. The data related to images in LXMERT mini-validation set is used to validate model performance and the rest of the data in train+val are used in fine-tuning. We test our model on the VQA v2.0 ‘test-dev’ and ‘test-standard’ splits. For GQA fine-tuning, we follow the suggestions in official GQA guidelines[^12] to take testdev as our validation set and fine-tune our model on the joint train + validation sets. We test our GQA model on GQA ‘test-standard’ split. The images in $\text{NLVR}^{2}$ are not from either MS COCO or Visual Genome, we thus keep using the original split: fine-tune on train split, validate the model choice on val split, and test on the public (‘Test-P’) and unreleased (‘Test-U’) test splits.

微调

VQA v2.0 的训练与验证沿用与 LXMERT 预训练相同的划分约定：与 minival 图像相关的数据用于验证，train+val 其余用于微调；在 test-dev 与 test-standard 上测试。GQA 按官方指南[^12]以 testdev 为验证集，在 train+validation 联合集上微调，在 test-standard 上测试。$\text{NLVR}^{2}$ 图像既非 MS COCO 也非 Visual Genome，故保留原始划分：train 微调、val 选模，在公开（Test-P）与未公开（Test-U）测试集上评测。

Appendix D Training Details of ‘BERT versus LXMERT’

附录 D “BERT 对比 LXMERT” 的训练细节

When training with BERT only, we train each experiments for $20$ epochs with a batch size $64$/$128$ since it was not pre-trained on these cross-modality datasets. The learning rate is set to $1e-4$ instead of $5e-5$.

仅用 BERT 训练时，因未在这些跨模态数据上预训练，各实验训练 $20$ 个 epoch，批大小 $64$/$128$，学习率设为 $1e{-4}$ 而非 $5e{-5}$。

Figure 3: Attention graphs reveal similar behavior in the LXMERT language encoder (a, c) and in the original BERT encoder (b, d). Fig. a & b show the attention pointing to next words while Fig. c & d show the attention pointing to previous words.

图 3：注意力图显示 LXMERT 语言编码器（a、c）与原始 BERT 编码器（b、d）行为相近。a、b 为指向后继词的注意力；c、d 为指向前驱词的注意力。

Figure 4: The attention graph (a) and its recovered scene graph (b) in the first layer of LXMERT’s object-relationship encoder.

图 4：LXMERT 物体关系编码器第一层的注意力图（a）及其恢复的场景图（b）。

Appendix E Visualizing LXMERT Behavior

附录 E 可视化 LXMERT 行为

In this section, we show the behavior of LXMERT by visualizing its attention graphs in the language encoder, object-relationship encoder, and cross-modality encoder, respectively.

本节通过语言编码器、物体关系编码器与跨模态编码器的注意力图，展示 LXMERT 行为。

E.1 Language Encoder

E.1 语言编码器

In Fig. 3, we reveal that the LXMERT language encoder has similar behaviour as the original BERT encoder, by using the same sentence “Is it warm enough for him to be wearing shorts?” as the input to both models. LXMERT’s attention graphs (in Fig. 3(a, c)) are extracted from the pre-trained LXMERT without fine-tuning on a specific task. BERT’s attention graphs (in Fig. 3(b, d)) come from Hoover et al. 2019.[^13] We find that both the second LXMERT layer (Fig. 3(a)) and third BERT layer (Fig. 3(b)) point to the next words while both the fourth LXMERT layer (Fig. 3(c)) and fourth BERT layer (Fig. 3(d)) point to the previous words, thus showing the similar behaviour of the two encoders.

图 3 中，我们用同一句子 “Is it warm enough for him to be wearing shorts?” 作为两模型输入，说明 LXMERT 语言编码器与原始 BERT 行为相近。LXMERT 注意力图（图 3(a,c)）来自未针对具体任务微调的预训练模型；BERT 注意力图（图 3(b,d)）来自 Hoover et al. 2019。[^13] 可见 LXMERT 第 2 层（图 3(a)）与 BERT 第 3 层（图 3(b)）都指向后继词，LXMERT 第 4 层（图 3(c)）与 BERT 第 4 层（图 3(d)）都指向前驱词，行为相似。

Figure 5: Attention graphs in LXMERT’s cross-modality encoder showing that the attention focus on pronouns (marked in pink), nouns (marked in blue), and articles (marked in red).

图 5：LXMERT 跨模态编码器的注意力图，显示注意力集中在代词（粉）、名词（蓝）与冠词（红）。

E.2 Object-Relationship Encoder

E.2 物体关系编码器

In Fig. 4, we visualize the attention graph of the first layer in LXMERT’s object-relationship encoder. We only highlight the objects with the highest attention scores while the other objects are mostly not attended to. We manually build the connections between objects (marked as yellow lines in Fig. 4(b)) according to the attention graph. These connections faithfully draw a scene graph of the figure, which indicates that the object-relationship encoder might be learning a reasonably good network of the relationships between objects.

图 4 可视化 LXMERT 物体关系编码器第一层的注意力。仅高亮注意力分数最高的物体，其余大多不被关注。我们按注意力图手工连边（图 4(b) 黄线）；这些连接如实勾勒出场景图，表明物体关系编码器可能在学习较合理的物体关系网络。

E.3 Cross-Modality Encoder

E.3 跨模态编码器

In Fig. 5, we visualize the attention in LXMERT’s cross-modality encoder to reveal the connections between objects and words. We find that the attention focuses on nouns and pronouns as shown in the top figure of Fig. 5 because they are the most informative words in current vision-and-language tasks. However, for non-plural nouns (as shown in the bottom example in Fig. 5), the attention will focus on the articles. Although we do not specifically design for this behavior, we think that articles are possibly serving as special tokens (e.g., [CLS], [SEP] in BERT), thus providing unified target entries for the attention layers. Next, we are also looking at how to utilize pre-training tasks which directly capture pairwise noun-noun and noun-verb relationships between the images and text sentences.

图 5 可视化跨模态编码器中物体与词之间的注意力。上例显示注意力多落在名词与代词上，因它们在当前视觉—语言任务中信息量最大；对非复数名词（图 5 下例），注意力会落到冠词上。虽非特意设计，我们认为冠词可能充当类似 BERT 中 [CLS]/[SEP] 的特殊 token，为注意力层提供统一目标入口。接下来我们也在探索能直接捕捉图文之间名词—名词、名词—动词成对关系的预训练任务。