Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Oscar：面向视觉-语言任务的物体语义对齐预训练

Xiujun Li ♡ ​ ♠ \heartsuit\spadesuit    Xi Yin ♡ \heartsuit    Chunyuan Li ♡ \heartsuit    Pengchuan Zhang ♡ \heartsuit    Xiaowei Hu ♡ \heartsuit    Lei Zhang ♡ \heartsuit    Lijuan Wang ♡ \heartsuit    Houdong Hu ♡ \heartsuit    Li Dong ♡ \heartsuit    Furu Wei ♡ \heartsuit    Yejin Choi ♠ \spadesuit    Jianfeng Gao ♡ \heartsuit Affiliation:  Microsoft Corporation      University of Washington

Abstract

摘要

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar[^1], which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of $6.5$ million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks.[^2]

在图文对上学跨模态表征的大规模预训练，正越来越多地用到视觉-语言任务上。已有方法通常只是把图像区域特征和文本特征拼在一起送进待预训练的模型，再靠自注意力硬做图文语义对齐。本文提出一种新的学习方法 Oscar，用图像里检测出的物体标签作为锚点（anchor points），显著降低对齐学习的难度。做法的依据是：图像中的显著物体已能被较准确地检测出来，而且往往也会在配对文本里被提到。我们在公开语料的 650 万组图文对上预训练 Oscar，再在下游任务上微调，从而在六项成熟的视觉-语言理解与生成任务上取得了当时最好的成绩。

1 Introduction

1 引言

Keywords: Object Semantics, Vision-and-Language, Pre-training

Learning cross-modal representations is fundamental to a wide range of vision-language (V+L) tasks, such as visual question answering, image-text retrieval, image captioning. Recent studies [22, 38, 5, 35, 20, 19, 46] on vision-language pre-training (VLP) have shown that it can effectively learn generic representations from massive image-text pairs, and that fine-tuning VLP models on task-specific data achieves state-of-the-art (SoTA) results on well-established V+L tasks.

学习跨模态表征，是视觉问答、图文检索、图像描述等许多视觉-语言（V+L）任务的基础。近期关于视觉-语言预训练（VLP）的研究[22, 38, 5, 35, 20, 19, 46]表明，可以从大规模图文对中有效学到通用表征；再把 VLP 模型在任务数据上微调，就能在成熟的 V+L 任务上取得当时最好的结果。

These VLP models are based on multi-layer Transformers [39]. To pre-train such models, existing methods simply concatenate image region features and text features as input and resort to the self-attention mechanism to learn semantic alignments between image regions and text in a brute force manner. However, the lack of explicit alignment information between the image regions and text poses alignment modeling a weakly-supervised learning task. In addition, visual regions are often over-sampled [2], noisy and ambiguous, which makes the task even more challenging.

这些 VLP 模型都以多层 Transformer[39] 为骨架。预训练时，已有方法通常只是把图像区域特征和文本特征拼接成输入，再依靠自注意力，用近乎硬做的方式去学区域与文本之间的语义对齐。由于区域和文本之间缺少显式对齐信息，对齐建模实际上是一项弱监督任务。再加上视觉区域常常被过采样[2]，既嘈杂又含糊，任务就更难了。

In this study, we show that the learning of cross-modal representations can be significantly improved by introducing object tags detected in images as anchor points to ease the learning of semantic alignments between images and texts. We propose a new VLP method Oscar, where we define the training samples as triples, each consisting of a word sequence, a set of object tags, and a set of image region features. Our method is motivated by the observation that the salient objects in an image can be accurately detected by modern object detectors [28], and that these objects are often mentioned in the paired text. For example, on the MS COCO dataset [21], the percentages that an image and its paired text share at least $1$, $2$, $3$ objects are $49.7\%$, $22.2\%$, $12.9\%$, respectively. Our Oscar model is pre-trained on a large-scale V+L dataset composed of $6.5$ million pairs, and is fine-tuned and evaluated on seven V+L understanding and generation tasks. The overall setting is illustrated in Fig 1.

本研究表明，如果把图像中检测出的物体标签作为锚点引入，跨模态表征的学习可以明显改善，图文之间的语义对齐也更容易学。我们提出一种新的 VLP 方法 Oscar，把训练样本定义成三元组：词序列、一组物体标签，以及一组图像区域特征。这一做法来自一个观察：现代物体检测器[28]已经能够较准确地检出图像中的显著物体，而这些物体往往也会在配对文本里出现。例如在 MS COCO 数据集[21]上，一张图像与其配对文本至少共享 1、2、3 个物体的比例分别是 49.7%、22.2%、12.9%。Oscar 在由 650 万对样本组成的大规模 V+L 数据上预训练，再在七项 V+L 理解与生成任务上微调并评测。整体设置如图 1 所示。

Figure 1: Oscar pipeline. The model takes a triple as input, is pre-trained with two losses (a masked token loss over words & tags, and a contrastive loss between tags and others), and fine-tuned for 5 understanding and 2 generation tasks (detailed in Sec. 4).

图 1：Oscar 流程。模型以三元组为输入，用两项损失预训练（对词和标签的掩码词元损失，以及标签与其他内容之间的对比损失），再微调到 5 项理解任务和 2 项生成任务（详见第 4 节）。

Although the use of anchor points for alignment modeling has been explored in natural language processing e.g., [3], to the best of our knowledge, this work is the first that explores the idea for VLP. There have been previous works that use object or image tags in V+L tasks for the sake of enhancing the feature representation of image regions, rather than for learning image-text alignments. For example, Zhou et al. [46] uses the object prediction probability as a soft label and concatenate it with its corresponding region features. Wu et al. [42] and You et al. [43] introduce image-level labels or attributes to improve image-level visual representations.

用锚点来建模对齐，在自然语言处理里已经有过探索，例如[3]；据我们所知，把这一想法用到 VLP 上，本文是第一次。此前也有工作在 V+L 任务里使用物体或图像标签，但目的是增强图像区域的特征表示，而不是学习图文对齐。例如，Zhou 等[46]把物体预测概率当作软标签，再与对应的区域特征拼接；Wu 等[42]和 You 等[43]则引入图像级标签或属性，以改善图像级视觉表征。

The main contributions of this work can be summarized as follows: $(\textup{\it i})$ We introduce Oscar, a powerful VLP method to learn generic image-text representations for V+L understanding and generation tasks. $(\textup{\it ii})$ We have developed an Oscar model that achieves new SoTA on multiple V+L benchmarks, outperforming existing approaches by a significant margin; $(\textup{\it iii})$ We present extensive experiments and analysis to provide insights on the effectiveness of using object tags as anchor points for cross-modal representation learning and downstream tasks.

本文的主要贡献可以概括为：$(\textup{\it i})$ 提出 Oscar，一种用于学习通用图文表征、面向 V+L 理解与生成任务的有力 VLP 方法；$(\textup{\it ii})$ 训练出的 Oscar 模型在多项 V+L 基准上取得新的最好成绩，明显超过已有方法；$(\textup{\it iii})$ 通过大量实验与分析，说明把物体标签当作锚点，对跨模态表征学习和下游任务为什么有效。

2 Background

2 背景

The training data for many V+L tasks consists of image-text pairs, as shown in Fig. 2(a). We denote a dataset of size $N$ by $\mathcal{D}=\{({\bf I}_{i},\boldsymbol{w}_{i})\}_{i=1}^{N}$, with image ${\bf I}$ and text sequence $\boldsymbol{w}$. The goal of pre-training is to learn cross-modal representations of image-text pairs in a self-supervised manner, which can be adapted to serve various down-stream tasks via fine-tuning.

许多 V+L 任务的训练数据都是图文对，如图 2(a) 所示。我们把规模为 $N$ 的数据集记为 $\mathcal{D}=\{({\bf I}_{i},\boldsymbol{w}_{i})\}_{i=1}^{N}$，其中 ${\bf I}$ 是图像，$\boldsymbol{w}$ 是文本序列。预训练的目标，是以自监督方式学习图文对的跨模态表征，再通过微调适配到各种下游任务。

VLP typically employs multi-layer self-attention Transformers [39] to learn cross-modal contextualized representations, based on the singular embedding of each modality. Hence, the success of VLP fundamentally relies on the quality of the input singular embeddings. Existing VLP methods take visual region features $\boldsymbol{v}=\{v_{1},\cdots,v_{K}\}$ of an image and word embeddings $\boldsymbol{w}=\{w_{1},\cdots,w_{T}\}$ of its paired text as input, and relies on the self-attention mechanism to learn image-text alignments and produce cross-modal contextual representations.

VLP 通常用多层自注意力 Transformer[39]，在各模态各自的单独嵌入之上，学习跨模态的上下文化表征。因此，VLP 是否成功，根本上取决于输入的单独嵌入质量。已有 VLP 方法把一幅图像的视觉区域特征 $\boldsymbol{v}=\{v_{1},\cdots,v_{K}\}$ 与配对文本的词嵌入 $\boldsymbol{w}=\{w_{1},\cdots,w_{T}\}$ 作为输入，再依靠自注意力学习图文对齐，并产出跨模态上下文表征。

Though intuitive and effective, existing VLP methods suffer from two issues: $(\textup{\it i})$ Ambiguity. The visual region features are usually extracted from over-sampled regions [2] via Faster R-CNN object detectors [28], which inevitably results in overlaps among image regions at different positions. This renders ambiguities for the extracted visual embeddings. For example, in Fig. 2(a) the region features for $\mathtt{dog}$ and $\mathtt{couch}$ are not easily distinguishable, as their regions heavily overlap. $(\textup{\it ii})$ Lack of grounding. VLP is naturally a weakly-supervised learning problem because there is no explicitly labeled alignments between regions or objects in an image and words or phrases in text. However, we can see that salient objects such as $\mathtt{dog}$ and $\mathtt{couch}$ are presented in both image and its paired text as in Fig. 2(a), and can be used as anchor points for learning semantic alignments between image regions and textual units as in Fig. 2(b). In this paper we propose a new VLP method that utilizes these anchor points to address the aforementioned issues.

虽然直观也有效，已有 VLP 方法仍有两个问题：$(\textup{\it i})$ 含糊性。视觉区域特征通常由 Faster R-CNN 物体检测器[28]从过采样区域[2]中抽出，不同位置的区域难免重叠，抽到的视觉嵌入因此含糊。例如在图 2(a) 中，$\mathtt{dog}$ 和 $\mathtt{couch}$ 的区域特征并不容易区分，因为它们的区域大量重叠。$(\textup{\it ii})$ 缺少锚定（grounding）。VLP 天然是弱监督学习问题，因为图像中的区域或物体，与文本中的词或短语之间，并没有显式标注的对齐。不过可以看到，像 $\mathtt{dog}$ 和 $\mathtt{couch}$ 这样的显著物体，在图 2(a) 的图像和配对文本里都会出现，因而可以当作锚点，用来学习图像区域与文本单元之间的语义对齐，如图 2(b)。本文提出一种新的 VLP 方法，正是利用这些锚点来应对上述问题。

(a) Image-text pair	(b) Objects as anchor points	(c) Semantics spaces

Figure 2: Illustration on the process that Oscar represents an image-text pair into semantic space via dictionary look up. (a) An example of input image-text pair (b) The object tags are used as anchor points to align image regions with word embeddings of pre-trained language models. (c) The word semantic space is more representative than image region features. In this example, $\mathtt{dog}$ and $\mathtt{couch}$ are similar in the visual feature space due to the overlap regions, but distinctive in the word embedding space.

(a) 图文对	(b) 物体作为锚点	(c) 语义空间

图 2：Oscar 通过词典查找，把图文对表示到语义空间的过程示意。(a) 输入图文对的例子；(b) 用物体标签作锚点，把图像区域与预训练语言模型的词嵌入对齐；(c) 词语义空间比图像区域特征更有区分度。在这个例子里，$\mathtt{dog}$ 和 $\mathtt{couch}$ 因为区域重叠，在视觉特征空间里很相近，但在词嵌入空间里是分开的。

3 Oscar Pre-training

3 Oscar 预训练

Humans perceive the world through many channels. Even though any individual channel might be incomplete or noisy, important factors are still perceivable since they tend to be shared among multiple channels (e.g.,$\mathtt{dog}$ can be described visually and verbally, as in Fig. 2). With this motivation, we propose a new VLP method Oscar to learn representations that capture channel-invariant (or modality-invariant) factors at the semantic level. Oscar differs from existing VLP in the way that the input image-text pairs are represented and the pre-training objective, as outlined in Fig. 3.

人通过许多通道感知世界。即便单个通道不完整或有噪声，重要因素往往仍能被感知到，因为它们会在多个通道之间共享（例如，$\mathtt{dog}$ 既可以在视觉上描述，也可以用语言描述，如图 2）。出于这一动机，我们提出新的 VLP 方法 Oscar，学习在语义层面捕获通道不变（或模态不变）因素的表征。Oscar 与已有 VLP 的差别，在于图文对的输入表示方式，以及预训练目标，如图 3 所示。

Figure 3: Illustration of Oscar. We represent the image-text pair as a triple [word tokens, object tags, region features], where the object tags (e.g.,“dog” or “couch”) are proposed to align the cross-domain semantics; when removed, Oscar reduces to previous VLP methods. The input triple can be understood from two perspectives: a modality view and a dictionary view.

图 3：Oscar 示意。我们把图文对表示成三元组 [词元、物体标签、区域特征]，其中物体标签（例如 “dog” 或 “couch”）用来对齐跨域语义；若去掉标签，Oscar 就退化为已有的 VLP 方法。输入三元组可以从两个视角来理解：模态视角和词典视角。

Input

Oscar represents each input image-text pair as a Word-Tag-Image triple $(\boldsymbol{w},{\boldsymbol{q}},\boldsymbol{v})$, where $\boldsymbol{w}$ is the sequence of word embeddings of the text, ${\boldsymbol{q}}$ is the word embedding sequence of the object tags (in text) detected from the image, and $\boldsymbol{v}$ is the set of region vectors of the image.

输入

Oscar 把每个输入图文对表示成词-标签-图像三元组 $(\boldsymbol{w},{\boldsymbol{q}},\boldsymbol{v})$，其中 $\boldsymbol{w}$ 是文本的词嵌入序列，${\boldsymbol{q}}$ 是从图像中检出的物体标签（以文本形式）的词嵌入序列，$\boldsymbol{v}$ 是该图像的区域向量集合。

Existing VLP methods represent each input pair as $(\boldsymbol{w},\boldsymbol{v})$. Oscar introduces ${\boldsymbol{q}}$ as anchor points to ease the learning of image-text alignment. This is motivated by the observation that in training data, important objects in an image are often also presented in the image-paired text, using either the same words as object tags or different but semantically similar or related words. Since the alignments between ${\boldsymbol{q}}$ and $\boldsymbol{w}$, both in text, are relatively easy to identified by using pre-trained BERT models [6], which are used as initialization for VLP in Oscar, the image regions from which the object tags are detected are likely to have higher attention weights than other regions, when queried by the semantically related words in the text. This alignment learning process is conceptually illustrated in Fig. 2(b). The process can also be interpreted as learning to ground the image objects, which might be ambiguously represented in the vision space such as $\mathtt{dog}$ and $\mathtt{couch}$ in Fig. 2(a), in distinctive entities represented in the language space, as illustrated in Fig. 2(c).

已有 VLP 方法把每个输入对表示成 $(\boldsymbol{w},\boldsymbol{v})$。Oscar 引入 ${\boldsymbol{q}}$ 作为锚点，以降低图文对齐的学习难度。这来自训练数据里的一个观察：图像中的重要物体，常常也会出现在配对文本里，用的要么是与物体标签相同的词，要么是语义相近或相关的不同词。由于 ${\boldsymbol{q}}$ 与 $\boldsymbol{w}$ 都在文本一侧，它们之间的对齐相对容易由预训练 BERT 模型[6]识别出来——Oscar 正是用这些 BERT 来初始化 VLP。于是，当被文本里语义相关的词查询时，检出这些物体标签的那些图像区域，就更可能比其他区域获得更高的注意力权重。这一对齐学习过程在概念上如图 2(b) 所示。也可以把它理解成：学习把视觉空间里可能表示含糊的图像物体（如图 2(a) 中的 $\mathtt{dog}$ 和 $\mathtt{couch}$），锚定到语言空间里可区分的实体上，如图 2(c)。

Specifically, $\boldsymbol{v}$ and ${\boldsymbol{q}}$ are generated as follows. Given an image with $K$ regions of objects (normally over-sampled and noisy), Faster R-CNN [28] is used to extract the visual semantics of each region as $(v^{\prime},z)$, where region feature $v^{\prime}\in\mathbb{R}^{P}$ is a $P$-dimensional vector (i.e.,$P=2048$), and region position $z$ a $R$-dimensional vector (i.e.,$R=4$ or $6$)[^3]. We concatenate $v^{\prime}$ and $z$ to form a position-sensitive region feature vector, which is further transformed into $v$ using a linear projection to ensure that it has the same vector dimension as that of word embeddings. Meanwhile, the same Faster R-CNN is used to detect a set of high precision object tags. ${\boldsymbol{q}}$ is the sequence of word embeddings of the object tags.

具体而言，$\boldsymbol{v}$ 和 ${\boldsymbol{q}}$ 这样得到。给定一幅含有 $K$ 个物体区域的图像（通常过采样且嘈杂），用 Faster R-CNN[28] 抽取每个区域的视觉语义 $(v^{\prime},z)$，其中区域特征 $v^{\prime}\in\mathbb{R}^{P}$ 是 $P$ 维向量（即 $P=2048$），区域位置 $z$ 是 $R$ 维向量（即 $R=4$ 或 $6$）。我们把 $v^{\prime}$ 与 $z$ 拼接成对位置敏感的区域特征向量，再经线性投影变换为 $v$，使其与词嵌入具有相同的向量维数。与此同时，同一套 Faster R-CNN 还用于检出一组高精度物体标签。${\boldsymbol{q}}$ 就是这些物体标签的词嵌入序列。

Pre-Training Objective

The Oscar input can be viewed from two different perspectives as

预训练目标

Oscar 的输入可以从两个不同视角来看，即

\displaystyle\boldsymbol{x}\triangleq[\underbrace{~\boldsymbol{w}_{~}}_{\text{{\color[rgb]{1,0.5,0.5}language}}},\underbrace{~{\boldsymbol{q}},\boldsymbol{v}~}_{\text{{\color[rgb]{0.2344,0.707,0.293}image}}}]=[\underbrace{~\boldsymbol{w},{\boldsymbol{q}}}_{\text{{\color[rgb]{1,0.5,0.5}language}}},~\underbrace{~\boldsymbol{v}_{~{}_{~}}}_{\text{{\color[rgb]{0.2344,0.707,0.293}image}}}]\triangleq\boldsymbol{x}^{\prime} \tag{1}

where $\boldsymbol{x}$ is the modality view to distinguish the representations between a text and an image; while $\boldsymbol{x}^{\prime}$ is the dictionary view[^4] to distinguish the two different semantic spaces, in which the input is represented. The two-view perspective allows us to design a novel pre-training objective.

其中 $\boldsymbol{x}$ 是模态视角，用来区分文本与图像的表示；而 $\boldsymbol{x}^{\prime}$ 是词典视角，用来区分输入所在的两个不同语义空间。这两个视角让我们可以设计新的预训练目标。

A Dictionary View: Masked Token Loss.

The use of different dictionaries determines the semantic spaces utilized to represent different sub-sequences. Specifically, the object tags and word tokens share the same linguistic semantic space, while the image region features lie in the visual semantic space. We define the discrete token sequence as ${\boldsymbol{h}}\triangleq[\boldsymbol{w},{\boldsymbol{q}}]$, and apply the Masked Token Loss (MTL) for pre-training. At each iteration, we randomly mask each input token in ${\boldsymbol{h}}$ with probability $15\%$, and replace the masked one $h_{i}$ with a special token $\mathtt{[MASK]}$. The goal of training is to predict these masked tokens based on their surrounding tokens ${\boldsymbol{h}}_{\backslash i}$ and all image features $\boldsymbol{v}$ by minimizing the negative log-likelihood:

词典视角：掩码词元损失。

使用不同词典，决定了用怎样的语义空间来表示不同的子序列。具体来说，物体标签和词元共享同一个语言语义空间，而图像区域特征位于视觉语义空间。我们把离散词元序列定义为 ${\boldsymbol{h}}\triangleq[\boldsymbol{w},{\boldsymbol{q}}]$，并用掩码词元损失（Masked Token Loss，MTL）做预训练。每次迭代时，我们以 15% 的概率随机掩码 ${\boldsymbol{h}}$ 中的每个输入词元，把被掩码的 $h_{i}$ 替换成特殊词元 $\mathtt{[MASK]}$。训练目标是根据其周围词元 ${\boldsymbol{h}}_{\backslash i}$ 以及全部图像特征 $\boldsymbol{v}$ 来预测这些被掩码的词元，即最小化负对数似然：

$$\displaystyle\hskip 0.0pt\mathcal{L}_{\text{MTL}}=-\mathbb{E}_{(\boldsymbol{v},{\boldsymbol{h}})\sim\mathcal{D}}\log p(h_{i}|{\boldsymbol{h}}_{\backslash i},\boldsymbol{v}) \tag{2}$$

This is similar to masked language model used by BERT. The masked word or tag needs to be recovered from its surroundings, with additional image information attended to help ground the learned word embeddings in the vision context.

这与 BERT 使用的掩码语言模型类似。被掩码的词或标签需要从上下文中恢复，并额外关注图像信息，以便把学到的词嵌入锚定在视觉语境里。

A Modality View: Contrastive Loss.

For each input triple, we group ${\boldsymbol{h}}^{\prime}\triangleq[{\boldsymbol{q}},\boldsymbol{v}]$ to represent the image modality, and consider $\boldsymbol{w}$ as the language modality. We then sample a set of “polluted” image representations by replacing ${\boldsymbol{q}}$ with probability 50% with a different tag sequence randomly sampled from the dataset $\mathcal{D}$. Since the encoder output on the special token $\mathtt{[CLS]}$ is the fused vision-language representation of $({\boldsymbol{h}}^{\prime},\boldsymbol{w})$, we apply a fully-connected (FC) layer on the top of it as a binary classifier $f(.)$ to predict whether the pair contains the original image representation ($y=1$) or any polluted ones ($y=0$). The contrastive loss is defined as

模态视角：对比损失。

对每个输入三元组，我们把 ${\boldsymbol{h}}^{\prime}\triangleq[{\boldsymbol{q}},\boldsymbol{v}]$ 归为图像模态，把 $\boldsymbol{w}$ 视为语言模态。然后以 50% 的概率，用从数据集 $\mathcal{D}$ 中随机抽样的另一段标签序列替换 ${\boldsymbol{q}}$，从而构造一组“被污染”的图像表示。由于编码器在特殊词元 $\mathtt{[CLS]}$ 上的输出是 $({\boldsymbol{h}}^{\prime},\boldsymbol{w})$ 的融合视觉-语言表征，我们在其上加一个全连接（FC）层作为二分类器 $f(.)$，预测该对包含的是原始图像表示（$y=1$）还是被污染的表示（$y=0$）。对比损失定义为

$$\displaystyle\mathcal{L}_{\text{C}}=-\mathbb{E}_{({\boldsymbol{h}}^{\prime},\boldsymbol{w})\sim\mathcal{D}}\log p(y|f({\boldsymbol{h}}^{\prime},\boldsymbol{w})). \tag{3}$$

During the cross-modal pre-training, we utilize object tags as the proxy of images to adjust the word embedding space of BERT, where a text is similar to its paired image (or more specifically, the object tags detected from the image), and dissimilar to the polluted ones.

在跨模态预训练中，我们把物体标签当作图像的代理，去调整 BERT 的词嵌入空间：一段文本应与其配对图像（更具体地说，是从该图像检出的物体标签）相近，而与被污染的样本相远。

The full pre-training objective of Oscar is:

Oscar 完整的预训练目标是：

$$\displaystyle\mathcal{L}_{\text{Pre-training}}=\mathcal{L}_{\text{MTL}}+\mathcal{L}_{\text{C}}. \tag{4}$$

Discussion.

Although other loss function designs can be considered as pre-training objectives, we perform experiments with these two losses for two reasons: $(\textup{\it i})$ Each loss provides a representative learning signal from its own perspective. We deliberately keep a clear and simple form for the joint loss to study the effectiveness of the proposed dictionary and modality views, respectively. $(\textup{\it ii})$ Though the overall loss is much simpler than those of existing VLP methods, it yields superior performance in our experiments.

讨论。

预训练目标也可以考虑其他损失函数设计，我们用这两项损失做实验，出于两点：$(\textup{\it i})$ 每项损失都从各自视角给出有代表性的学习信号。我们刻意保持联合损失的形式清楚而简单，以便分别考察所提出的词典视角与模态视角是否有效。$(\textup{\it ii})$ 尽管总体损失比已有 VLP 方法简单得多，实验中却得到了更好的表现。

Pre-training Corpus

We have built the pre-training corpus based on the existing V+L datasets, including COCO [21], Conceptual Captions (CC) [31], SBU captions [26], flicker30k [44], GQA [13] etc.. In total, the unique image set is 4.1 million, and the corpus consists of 6.5 million text-tag-image triples. The detail is in Appendix.

预训练语料

预训练语料基于已有的 V+L 数据集构建，包括 COCO[21]、Conceptual Captions（CC）[31]、SBU captions[26]、Flickr30k[44]、GQA[13] 等。不重复的图像集合共 410 万张，语料包含 650 万组文本-标签-图像三元组。细节见附录。

Implementation Details

We pre-train two model variants, denoted as Oscar${}_{\text{B}}$ and Oscar${}_{\text{L}}$, initialized with parameters $\boldsymbol{\theta}_{\text{BERT}}$ of BERT base ($H=768$) and large ($H=1024$), respectively, where $H$ is the hidden size. To ensure that the image region features have the same input embedding size as BERT, we transform the position-sensitive region features using a linear projection via matrix ${{\bf W}}$. The trainable parameters are $\boldsymbol{\theta}=\{\boldsymbol{\theta}_{\text{BERT}},{{\bf W}}\}$. The AdamW Optimizer is used. Oscar${}_{\text{B}}$ is trained for at least $1.0$M steps, with learning rate $5e^{-5}$ and batch size $768$. Oscar${}_{\text{L}}$ is trained for at least $900$k steps, with learning rate $1e^{-5}$ and batch size $512$. The sequence length of discrete tokens ${\boldsymbol{h}}$ and region features $\boldsymbol{v}$ are $35$ and $50$, respectively.

实现细节

我们预训练两个模型变体，记为 Oscar${}_{\text{B}}$ 和 Oscar${}_{\text{L}}$，分别用 BERT base（$H=768$）和 large（$H=1024$）的参数 $\boldsymbol{\theta}_{\text{BERT}}$ 初始化，其中 $H$ 是隐层大小。为使图像区域特征与 BERT 具有相同的输入嵌入维数，我们用矩阵 ${{\bf W}}$ 对位置敏感的区域特征做线性投影。可训练参数为 $\boldsymbol{\theta}=\{\boldsymbol{\theta}_{\text{BERT}},{{\bf W}}\}$。优化器使用 AdamW。Oscar${}_{\text{B}}$ 至少训练 $1.0$M 步，学习率为 $5e^{-5}$，批大小为 $768$。Oscar${}_{\text{L}}$ 至少训练 $900$k 步，学习率为 $1e^{-5}$，批大小为 $512$。离散词元 ${\boldsymbol{h}}$ 和区域特征 $\boldsymbol{v}$ 的序列长度分别是 $35$ 和 $50$。

4 Adapting to V+L Tasks

4 适配到 V+L 任务

We adapt the pre-trained models to seven downstream V+L tasks, including five understanding tasks and two generation tasks. Each task poses different challenges for adaptation. We introduce the tasks and our fine-tuning strategy in this section, and leave the detailed description of datasets and evaluation metrics to Appendix.

我们把预训练模型适配到七项下游 V+L 任务，其中包括五项理解任务和两项生成任务。每项任务对适配都提出了不同的挑战。本节介绍这些任务以及我们的微调策略，数据集与评测指标的详细说明放在附录。

Image-Text Retrieval

heavily relies on the joint representations. There are two sub-tasks: image retrieval and text retrieval, depending on which modality is used as the retrieved target. During training, we formulate it as a binary classification problem. Given an aligned image-text pair, we randomly select a different image or a different caption to form an unaligned pair. The final representation of $\mathtt{[CLS]}$ is used as the input to the classifier to predict whether the given pair is aligned or not. We did not use ranking losses [14, 18], as we found that the binary classification loss works better, similarly as reported in [27]. In the testing stage, the probability score is used to rank the given image-text pairs of a query. Following [19], we report the top-$K$ retrieval results on both the $1$K and $5$K COCO test sets.

图文检索

高度依赖联合表征。按检索目标的模态，可分为两个子任务：图像检索和文本检索。训练时，我们把它做成二分类问题。给定一对对齐的图文，再随机另选一张图像或一句描述，构成未对齐的一对。$\mathtt{[CLS]}$ 的最终表征送入分类器，预测给定对是否对齐。我们没有使用排序损失[14, 18]，因为发现二分类损失效果更好，这与[27]中的报告类似。测试阶段，用概率分数对查询所对应的图文对排序。沿用[19]，我们在 COCO 的 $1$K 和 $5$K 测试集上都报告前 $K$ 的检索结果。

Image Captioning

requires the model to generate a natural language description of the content of an image. To enable sentence generation, we fine-tune Oscar using the seq2seq objective. The input samples are processed to triples consisting of image region features, captions, and object tags, in the same way as that during the pre-training. We randomly mask out $15\%$ of the caption tokens and use the corresponding output representations to perform classification to predict the token ids. Similar to VLP [46], the self-attention mask is constrained such that a caption token can only attend to the tokens before its position to simulate a uni-directional generation process. Note that all caption tokens will have full attentions to image regions and object tags but not the other way around.

图像描述

要求模型生成一句描述图像内容的自然语言。为了能够生成句子，我们用 seq2seq 目标微调 Oscar。输入样本被处理成由图像区域特征、描述文本和物体标签组成的三元组，方式与预训练时相同。我们随机掩掉 15% 的描述词元，并用对应的输出表征做分类，以预测词元编号。与 VLP[46] 类似，自注意力掩码被约束成：一个描述词元只能注意到它位置之前的词元，以模拟单向生成过程。需要注意的是，所有描述词元都可以完整地注意到图像区域和物体标签，反过来则不行。

During inference, we first encode the image regions, object tags, and a special token $\mathtt{[CLS]}$ as input. Then the model starts the generation by feeding in a $\mathtt{[MASK]}$ token and sampling a token from the vocabulary based on the likelihood output. Next, the $\mathtt{[MASK]}$ token in the previous input sequence is replaced with the sampled token and a new $\mathtt{[MASK]}$ is appended for the next word prediction. The generation process terminates when the model outputs the $\mathtt{[STOP]}$ token. We use beam search (*i.e.,*beam size = 5) [2] in our experiments and report our results on the COCO image captioning dataset.

推理时，我们先把图像区域、物体标签和一个特殊词元 $\mathtt{[CLS]}$ 作为输入进行编码。然后模型开始生成：喂入一个 $\mathtt{[MASK]}$ 词元，并根据似然输出从词表中采样一个词元。接着，把上一输入序列里的 $\mathtt{[MASK]}$ 换成刚采样到的词元，再在末尾追加一个新的 $\mathtt{[MASK]}$，用于预测下一个词。当模型输出 $\mathtt{[STOP]}$ 词元时，生成过程结束。实验中我们使用束搜索（束宽为 5）[2]，并在 COCO 图像描述数据集上报告结果。

Novel Object Captioning (NoCaps)

[1] extends the image captioning task, and provides a benchmark with images from the Open Images dataset [17] to test models’ capability of describing novel objects which are not seen in the training corpus. Following the restriction guideline of NoCaps, we use the predicted Visual Genome and Open Images labels to form tag sequences, and train Oscar on COCO without the initialization of pre-training.

新颖物体描述（NoCaps）

[1] 扩展了图像描述任务，并提供一份来自 Open Images 数据集[17]的基准，用来检验模型描述训练语料中未见过的新颖物体的能力。按照 NoCaps 的限制指南，我们用预测得到的 Visual Genome 和 Open Images 标签组成标签序列，并在 COCO 上训练 Oscar，且不以预训练权重初始化。

VQA

[9] requires the model to answer natural language questions based on an image. Given an image and a question, the task is to select the correct answer from a multi-choice list. Here we conduct experiments on the widely-used VQA v2.0 dataset [9], which is built based on the MSCOCO [21] image corpus. The dataset is split into training (83k images and 444k questions), validation (41k images and 214k questions), and test (81k images and 448k questions) sets. Following [2], for each question, the model picks the corresponding answer from a shared set consisting of 3,129 answers.

VQA

[9] 要求模型根据图像回答自然语言问题。给定一幅图像和一个问题，任务是从多项选项中选出正确答案。我们在广泛使用的、基于 MSCOCO[21] 图像语料构建的 VQA v2.0 数据集[9]上做实验。数据划分为训练集（8.3 万张图像、44.4 万个问题）、验证集（4.1 万张图像、21.4 万个问题）和测试集（8.1 万张图像、44.8 万个问题）。沿用[2]，对每个问题，模型从由 3,129 个答案组成的共享集合中选出对应答案。

When fine-tuning on the VQA task, we construct one input sequence, which contains the concatenation of a given question, object tags and region features, and then the $\mathtt{[CLS]}$ output from Oscar is fed to a task-specific linear classifier for answer prediction. We treat VQA as a multi-label classification problem [2] – assigning a soft target score to each answer based on its relevancy to the human answer responses, and then we fine-tune the model by minimizing the cross-entropy loss computed using the predicted scores and the soft target scores. At inference, we simply use a Softmax function for prediction.

在 VQA 任务上微调时，我们构造一个输入序列，把给定问题、物体标签和区域特征拼接在一起，再把 Oscar 的 $\mathtt{[CLS]}$ 输出送入任务专用的线性分类器做答案预测。我们把 VQA 当作多标签分类问题[2]——根据每个答案与人工作答的相关程度，给它赋一个软目标分数，再最小化由预测分数与软目标分数计算的交叉熵损失，来微调模型。推理时，我们直接用 Softmax 做预测。

GQA

[13] is similar to VQA, except that GQA tests the reasoning capability of the model to answer a question. We conduct experiments on the public GQA dataset [13]. For each question, the model chooses an answer from a shared set of $1,852$ candidate answers. We develop two fine-tuned models using OscarB. One is similar to that of VQA. The other, denoted as Oscar${}_{B}^{*}$ in Table 2(d), is first fine-tuned on unbalanced “all-split” for $5$ epochs, and then fine-tuned on the “balanced-split” for $2$ epochs, as suggested in [4].

GQA

[13] 与 VQA 类似，不同的是 GQA 检验的是模型为了回答问题而进行推理的能力。我们在公开的 GQA 数据集[13]上做实验。对每个问题，模型从包含 $1,852$ 个候选答案的共享集合中选择一个答案。我们用 OscarB 训练了两个微调模型。一个与 VQA 的做法类似。另一个在表 2(d) 中记为 Oscar${}_{B}^{*}$：先在不均衡的 “all-split” 上微调 $5$ 个 epoch，再在 “balanced-split” 上微调 $2$ 个 epoch，这一安排参考了[4]的建议。

Natural Language Visual Reasoning for Real (NLVR2)

[36] takes a pair of images and a natural language, and the goal is to determine whether the natural language statement is true about the image pair. When fine-tuning on the NLVR2 task, we first construct two input sequences, each containing the concatenation of the given sentence (the natural language description) and one image, and then two $\mathtt{[CLS]}$ outputs from Oscar are concatenated as the joint input for a binary classifier, implemented by an MLP[^5].

面向真实场景的自然语言视觉推理（NLVR2）

[36] 的输入是一对图像和一段自然语言，目标是判断这段话对这一对图像是否为真。在 NLVR2 上微调时，我们先构造两个输入序列，每个序列都把给定句子（自然语言描述）与其中一幅图像拼接起来，再把 Oscar 给出的两个 $\mathtt{[CLS]}$ 输出拼接，作为由 MLP 实现的二分类器的联合输入。

5 Experimental Results & Analysis

5 实验结果与分析

5.1 Performance Comparison with SoTA

5.1 与当时最好结果的比较

To account for parameter efficiency, we compare Oscar against three types of SoTA’s: $(\textup{\it i})$ SoTAS indicates the best performance achieved by small models prior to the Transformer-based VLP models. $(\textup{\it ii})$ SoTAB indicates the best performance achieved by VLP models of similar size to BERT base. $(\textup{\it iii})$ SoTAL indicates the best performance yielded by models that have a similar size to BERT large. To the best of our knowledge, UNITER [5] is the only model of BERT large size.

为了同时考虑参数效率，我们把 Oscar 与三类当时最好结果比较：$(\textup{\it i})$ SoTAS 表示在基于 Transformer 的 VLP 模型出现之前，小模型取得的最好成绩；$(\textup{\it ii})$ SoTAB 表示与 BERT base 规模相近的 VLP 模型取得的最好成绩；$(\textup{\it iii})$ SoTAL 表示与 BERT large 规模相近的模型取得的最好成绩。据我们所知，UNITER[5] 是当时唯一达到 BERT large 规模的模型。

Table 1 summarizes the overall results on all tasks[^6]. For all the tables in this paper, Blue indicates the best result for a task, and gray background indicates results produced by Oscar. As shown in the table, our base model outperforms previous large models on most tasks, often by a significantly large margin. It demonstrates that the proposed Oscar is highly parameter-efficient, partially because the use of object tags as anchor points significantly eases the learning of semantic alignments between images and texts. Note that Oscar is pre-trained on $6.5$ million pairs, which is less than $9.6$ million pairs used for UNITER pre-training and $9.18$ million pairs for LXMERT.

表 1 汇总了全部任务上的总体结果。对本文中的所有表格，蓝色表示一项任务上的最好结果，灰色背景表示由 Oscar 得到的结果。如表所示，我们的基模型在多数任务上超过了先前的大模型，而且往往领先不少。这说明所提出的 Oscar 参数效率很高，部分原因是把物体标签当作锚点，显著降低了图文语义对齐的学习难度。需要说明的是，Oscar 在 650 万对样本上预训练，少于 UNITER 预训练所用的 960 万对，以及 LXMERT 的 918 万对。

Task	Image Retrieval	Text Retrieval	Image Captioning	NoCaps	VQA	NLVR2
R@1	R@5	R@10	R@1	R@5	R@10	B@4	M	C	S	C	S	test-std	test-P
SoTAS	$39.2$	$68.0$	$81.3$	$56.6$	$84.5$	$92.0$	$38.9$	$29.2$	$129.8$	$22.4$	$61.5$	$9.2$	$70.90$	$53.50$
SoTAB	$48.4$	$76.7$	$85.9$	$63.3$	$87.0$	$93.1$	$39.5$	$29.3$	$129.3$	$23.2$	$73.1$	$11.2$	$72.54$	$78.87$
SoTAL	$51.7$	$78.4$	$86.9$	$66.6$	$89.4$	$94.3$	$-$	$-$	$-$	$-$	$-$	$-$	$73.40$	$79.50$
Oscar${}_{\text{B}}$	$\bf 54.0$	$\bf 80.8$	$\bf 88.5$	$\bf 70.0$	$\bf 91.1$	$\bf 95.5$	$\bf 40.5$	$\bf 29.7$	$\bf 137.6$	$22.8$	$\bf 78.8$	$\bf 11.7$	$\bf 73.44$	$78.36$
Oscar${}_{\text{L}}$	$\bf 57.5$	$\bf 82.8$	$\bf 89.8$	$\bf 73.5$	$\bf 92.2$	$\bf 96.0$	$\bf 41.7$	$\bf 30.6$	$\bf 140.0$	$\bf 24.5$	$\bf 80.9$	$\bf 11.3$	$\bf 73.82$	$\bf 80.37$
$\Delta$	$\bf 5.8\uparrow$	$\bf 4.4\uparrow$	$\bf 2.9\uparrow$	$\bf 6.9\uparrow$	$\bf 2.8\uparrow$	$\bf 1.7\uparrow$	$\bf 2.2\uparrow$	$\bf 1.3\uparrow$	$\bf 10.7\uparrow$	$\bf 1.3\uparrow$	$\bf 7.8\uparrow$	$\bf 0.5\uparrow$	$\bf 0.42\uparrow$	$\bf 0.87\uparrow$

Table 1: Overall results on six tasks. $\Delta$ indicates the improvement over SoTA. SoTA with subscript S, B, L indicates performance achieved by small models, VLP of similar size to BERT base and large model, respectively. Most results are from [5], except that image captioning results are from[11, 46], NoCaps results are from [1], VQA results are from [38].

任务	图像检索	文本检索	图像描述	NoCaps	VQA	NLVR2
R@1	R@5	R@10	R@1	R@5	R@10	B@4	M	C	S	C	S	test-std	test-P
SoTAS	$39.2$	$68.0$	$81.3$	$56.6$	$84.5$	$92.0$	$38.9$	$29.2$	$129.8$	$22.4$	$61.5$	$9.2$	$70.90$	$53.50$
SoTAB	$48.4$	$76.7$	$85.9$	$63.3$	$87.0$	$93.1$	$39.5$	$29.3$	$129.3$	$23.2$	$73.1$	$11.2$	$72.54$	$78.87$
SoTAL	$51.7$	$78.4$	$86.9$	$66.6$	$89.4$	$94.3$	$-$	$-$	$-$	$-$	$-$	$-$	$73.40$	$79.50$
Oscar${}_{\text{B}}$	$\bf 54.0$	$\bf 80.8$	$\bf 88.5$	$\bf 70.0$	$\bf 91.1$	$\bf 95.5$	$\bf 40.5$	$\bf 29.7$	$\bf 137.6$	$22.8$	$\bf 78.8$	$\bf 11.7$	$\bf 73.44$	$78.36$
Oscar${}_{\text{L}}$	$\bf 57.5$	$\bf 82.8$	$\bf 89.8$	$\bf 73.5$	$\bf 92.2$	$\bf 96.0$	$\bf 41.7$	$\bf 30.6$	$\bf 140.0$	$\bf 24.5$	$\bf 80.9$	$\bf 11.3$	$\bf 73.82$	$\bf 80.37$
$\Delta$	$\bf 5.8\uparrow$	$\bf 4.4\uparrow$	$\bf 2.9\uparrow$	$\bf 6.9\uparrow$	$\bf 2.8\uparrow$	$\bf 1.7\uparrow$	$\bf 2.2\uparrow$	$\bf 1.3\uparrow$	$\bf 10.7\uparrow$	$\bf 1.3\uparrow$	$\bf 7.8\uparrow$	$\bf 0.5\uparrow$	$\bf 0.42\uparrow$	$\bf 0.87\uparrow$

表 1：六项任务上的总体结果。$\Delta$ 表示相对当时最好结果的提升。下标为 S、B、L 的 SoTA 分别表示小模型、与 BERT base 规模相近的 VLP，以及大模型取得的成绩。多数结果来自[5]，图像描述结果来自[11, 46]，NoCaps 结果来自[1]，VQA 结果来自[38]。

(a) Image-text retrieval
(b) VQA
(c) NLVR2
(d) GQA	(e) Image captioning on COCO
(f) Evaluation on NoCaps Val. Models are trained on COCO only without pre-training.

Table 2: Detailed results on V+L tasks.

(a) 图文检索
(b) VQA
(c) NLVR2
(d) GQA	(e) COCO 上的图像描述
(f) 在 NoCaps 验证集上的评测。模型仅在 COCO 上训练，不做预训练。

表 2：视觉-语言任务上的详细结果。

We report the detailed comparison on each task in Table 2. $(\textup{\it i})$ VLP methods dominate empirical performance across many V+L tasks, compared with small models. Oscar outperforms all existing VLP methods on all seven tasks, and achieves new SoTA on six of them. On GQA, neural state machine (NSM) [12] relies on a strong structural prior, which can also be incorporated into Oscar for improvement in the future. $(\textup{\it ii})$ 12-in-1 is a recently proposed multi-task learning model [23] for V+L, implemented on BERT base. We see that Oscar${}_{\text{B}}$ outperforms 12-in-1 on almost all the tasks, except on Test-P of NLVR2. Given that our method is based on single task fine-tuning, the result demonstrates the effectiveness of our proposed pre-training scheme. $(\textup{\it iii})$ overall, Oscar is the best performer on both understanding and generation tasks. On the captioning task, we further fine-tune Oscar with self-critical sequence training (SCST) [30] to improve sequence-level learning. The only comparable VLP method for captioning is [46]. The results in Table 2 (e) show that Oscar yields a much better performance, *e.g.,*improving BLEU@4 and CIDEr by more than 2 and 10 points, respectively. $(\textup{\it iv})$ The NoCaps guideline requires to only use the COCO captioning training set. Hence, we initialize with BERT, and train Oscar on the COCO training set. Constrained beam search (CBS) is used. The results in Table 2 (f) show that the variants of Oscar consistently outperform the previous SoTA method UpDown [1]. The gap is much larger on the near-domain or out-of-domain cases, demonstrating the strong generalization ability of Oscar.

我们在表 2 中报告各项任务上的详细比较。$(\textup{\it i})$ 与小模型相比，VLP 方法在许多 V+L 任务上占据了经验表现的主导地位。Oscar 在全部七项任务上超过已有 VLP 方法，并在其中六项上取得新的最好成绩。在 GQA 上，神经状态机（NSM）[12] 依赖较强的结构先验，这一先验也可以在未来纳入 Oscar 以进一步提升。$(\textup{\it ii})$ 12-in-1 是近期提出的 V+L 多任务学习模型[23]，实现在 BERT base 上。可以看到，Oscar${}_{\text{B}}$ 在几乎全部任务上都超过 12-in-1，只在 NLVR2 的 Test-P 上略逊。考虑到我们的方法是单任务微调，这一结果说明所提出的预训练方案是有效的。$(\textup{\it iii})$ 总体而言，Oscar 在理解和生成任务上都是最好的。在描述任务上，我们还用自临界序列训练（SCST）[30] 继续微调 Oscar，以加强序列级学习。当时唯一可比较的、面向描述的 VLP 方法是[46]。表 2(e) 的结果表明 Oscar 明显更好，例如 BLEU@4 和 CIDEr 分别提高了 2 个点和 10 个点以上。$(\textup{\it iv})$ NoCaps 的指南要求只使用 COCO 描述训练集。因此我们从 BERT 初始化，并在 COCO 训练集上训练 Oscar，解码时使用约束束搜索（CBS）。表 2(f) 的结果表明，Oscar 的各个变体都稳定超过先前最好的 UpDown[1]。在近域或域外情形上差距更大，说明 Oscar 的泛化能力很强。

5.2 Qualitative Studies

5.2 定性研究

(a) Oscar	(b) Baseline (No tags)

Figure 4: 2D visualization using $t$-SNE. The points from the same object class share the same color. Please refer Appendix for full visualization.

(a) Oscar	(b) 基线（无标签）

图 4：用 $t$-SNE 做的二维可视化。同一物体类别的点共用一种颜色。完整可视化见附录。

We visualize the learned semantic feature space of image-text pairs of the COCO test set on a 2D map using $t$-SNE [24]. For each image region and word token, we pass it through the model, and use its last-layer output as features. Pre-trained models with and without object tags are compared. The results in Fig 4 reveal some interesting findings. $(\textup{\it i})$ Intra-class. With the aid of object tags, the distance of the same object between two modalities is substantially reduced. For example, the visual and textual representations for $\mathtt{person}$ (or $\mathtt{zebra}$) in Oscar is much closer than that in the baseline method. $(\textup{\it ii})$ Inter-class. Object classes of related semantics are getting closer (but still distinguishable) after adding tags, while there are some mixtures in the baseline, such as animal ($\mathtt{person}$, $\mathtt{zebra}$, $\mathtt{sheep}$, $\mathtt{bird}$), furniture ($\mathtt{chair}$, $\mathtt{couch}$, $\mathtt{bench}$), and transportation ($\mathtt{bus}$, $\mathtt{train}$, $\mathtt{truck}$, $\mathtt{motorcycle}$, $\mathtt{car}$). This verifies the importance of object tags in alignment learning: it plays the role of anchor points in linking and regularizing the cross-modal feature learning.

我们用 $t$-SNE[24] 把 COCO 测试集中图文对学到的语义特征空间画到二维图上。对每个图像区域和词元，我们让它通过模型，并取其最后一层输出作为特征。比较了带物体标签和不带物体标签的预训练模型。图 4 的结果揭示了一些有意思的发现。$(\textup{\it i})$ 类内。借助物体标签，同一物体在两个模态之间的距离明显缩小。例如，Oscar 中 $\mathtt{person}$（或 $\mathtt{zebra}$）的视觉表征与文本表征，要比基线方法近得多。$(\textup{\it ii})$ 类间。加上标签之后，语义相关的物体类别会靠得更近（但仍可区分），而基线里则有一些混在一起的情况，例如动物（$\mathtt{person}$、$\mathtt{zebra}$、$\mathtt{sheep}$、$\mathtt{bird}$）、家具（$\mathtt{chair}$、$\mathtt{couch}$、$\mathtt{bench}$）以及交通工具（$\mathtt{bus}$、$\mathtt{train}$、$\mathtt{truck}$、$\mathtt{motorcycle}$、$\mathtt{car}$）。这印证了物体标签在对齐学习中的重要性：它扮演锚点的角色，把跨模态特征学习连接起来并加以约束。

Figure 5: Examples of image captioning. Objects are colored, based on their appearance against the groud-truth (GT): all, Oscar & tags, tags only.

图 5：图像描述的例子。物体按它们相对标准答案（GT）的出现情况着色：全部模型都提到、Oscar 与标签、仅标签。

We compare generated captions of different models in Fig. 5. The baseline method is VLP without object tags. We see that Oscar generates more detailed descriptions of images than the baseline, due to the use of the accurate and diverse object tags detected by Faster R-CNN. They are the anchor points in the word embedding space, guiding the text generation process.

我们在图 5 中比较不同模型生成的描述。基线方法是不使用物体标签的 VLP。可以看到，Oscar 对图像的描述比基线更细致，这是因为 Faster R-CNN 检出的物体标签又准又多样。它们是词嵌入空间中的锚点，引导着文本生成过程。

5.3 Ablation Analysis

5.3 消融分析

We perform ablation experiments over a number of design choices of Oscar in both pre-training and fine-tuning to better understand their relative importance to four representative downstream tasks. All the ablation experiments are conducted on the base model.

我们对 Oscar 在预训练和微调中的若干设计选择做了消融，以便更好地理解它们对四项有代表性的下游任务的相对重要性。全部消融实验都在基模型上进行。

Figure 6: The learning curves of fine-tuning downstream tasks with different object tags. Each curve is with 3 runs.

(a) VQA
(b) Image Retrieval R@1
(c) Image Captioning

图 6：用不同物体标签微调下游任务的学习曲线。每条曲线都来自 3 次运行。

(a) VQA
(b) 图像检索 R@1
(c) 图像描述

The Effect of Object Tags

To study the effect of object tags, we experiment three different settings: $(\textup{\it i})$ Baseline (No Tags): this reduces the models to their previous VLP counterparts, where no tag information is exploited. $(\textup{\it ii})$ Predicted Tags: we use an off-the-shelf object detector (trained on COCO dataset) to predict object tags. $(\textup{\it iii})$ Ground-truth Tags: The ground-truth tags from COCO dataset are utilized to serve as a performance “upper bound” for our method. The experiments are conducted with the same BERT base model on three representative tasks, including VQA, image retrieval, and image captioning. As shown in Fig. 6, the learning curves for fine-tuning with object tags converges significantly faster and better than the VLP method without tags on all tasks. On the VQA and retrieval tasks, training using tags only takes half of the training time to achieve the final performance of the baseline, showing that Oscar is a more practical and efficient scheme for VLP. With more accurate object detectors developed in the future, Oscar can achieve even better performance, closing the gap demonstrated by using the ground-truth tags.

物体标签的作用

为了研究物体标签的作用，我们试验了三种设置：$(\textup{\it i})$ 基线（无标签）：模型退化为先前的 VLP 对应版本，不利用任何标签信息。$(\textup{\it ii})$ 预测标签：用现成的物体检测器（在 COCO 数据集上训练）预测物体标签。$(\textup{\it iii})$ 标准标签：使用 COCO 数据集中的标准标签，作为本方法表现的“上界”。实验在同一套 BERT base 模型上，针对 VQA、图像检索和图像描述这三项有代表性的任务进行。如图 6 所示，带物体标签的微调学习曲线，在所有任务上都比不使用标签的 VLP 方法收敛得明显更快、也更好。在 VQA 和检索任务上，使用标签的训练只需大约一半时间，就能达到基线的最终表现，说明 Oscar 是一种更实用、更高效的 VLP 方案。随着未来出现更准确的物体检测器，Oscar 还可以取得更好的成绩，缩小与使用标准标签时所展示的差距。

Table 3: Retrieval results on the COCO 1K test set, with different types of attention interactions.

表 3：在 COCO 1K 测试集上、不同注意力交互方式下的检索结果。

Attention Interaction

To further understand the interaction among the text, object tags and object regions, we conduct fine-tuning experiments by varying the attention masks for image-text retrieval. The default setting uses full attentions across all modalities. We then enable certain part of the attention masks. All models are initialized from BERT base without pre-training. Table 5.3 reports the performance on the COCO $1$K test set. By comparing the results of using full attention and partial attention $\boldsymbol{w}$-$\boldsymbol{v}$, we see that it is beneficial to add object tags. Moreover, region features are more informative than object tags ($\boldsymbol{w}$-$\boldsymbol{v}$, vs. $\boldsymbol{v}$-${\boldsymbol{q}}$) in representing an image. This suggests that tags yield minor improvement when used as features; a more promising way is to use them as anchor points, as done in Oscar.

注意力交互

为了进一步理解文本、物体标签和物体区域之间的交互，我们在图文检索的微调中改变注意力掩码。默认设置是在所有模态之间使用完整注意力。然后我们再只打开注意力掩码的某些部分。所有模型都从未经预训练的 BERT base 初始化。表 3 报告了 COCO $1$K 测试集上的表现。比较使用完整注意力与只使用部分注意力 $\boldsymbol{w}$-$\boldsymbol{v}$ 的结果，可以看到加入物体标签是有益的。此外，在表示一幅图像时，区域特征比物体标签更有信息量（$\boldsymbol{w}$-$\boldsymbol{v}$ 对比 $\boldsymbol{v}$-${\boldsymbol{q}}$）。这说明，标签若只当作特征来用，提升有限；更有希望的用法是像 Oscar 那样把它们当作锚点。

Object Tags in Pre-training

To study the impact of different object tag sets in pre-trained models, we pre-train two variants: Oscar${}^{\text{VG}}$ and Oscar${}^{\text{OI}}$ utilizes object tags produced by the object detector trained on the visual genome (VG) dataset [16] and the open images (OI) dataset [17], respectively. In this ablation, all the models are pre-trained for 589k steps. The results are shown in Table 4, where Baseline (No Tags) is also listed for comparison. It is clear that the Oscar scheme of using object tags as anchor points improves the baseline, regardless of which set of object tags is used. VG tags performs slightly better than OI. We hypothesize that the object detector trained on VG has a more diverse set of objects, although the object detector trained on OI has a higher precision.

预训练中的物体标签

为了研究预训练模型中不同物体标签集合的影响，我们预训练了两个变体：Oscar${}^{\text{VG}}$ 和 Oscar${}^{\text{OI}}$ 分别使用在 Visual Genome（VG）数据集[16]和 Open Images（OI）数据集[17]上训练的物体检测器所产生的物体标签。在这项消融中，所有模型都预训练 589k 步。结果如表 4 所示，同时列出了基线（无标签）以便比较。可以清楚地看到，无论使用哪一套物体标签，把物体标签当作锚点的 Oscar 方案都优于基线。VG 标签比 OI 略好。我们推测，在 VG 上训练的物体检测器具有更多样的物体集合，尽管在 OI 上训练的检测器具有更高的精度。

Pre-train	VQA	Text Retrieval	Image Retrieval	Image Captioning
dev	R@1	R@5	R@10	R@1	R@5	R@10	B@4	M	C	S
Baseline (No Tags)	$70.93$	$84.4$	$98.1$	$99.5$	$73.1$	$94.5$	$97.9$	$34.5$	$29.1$	$115.6$	$21.9$
Oscar${}^{\text{VG}}$	$71.70$	$88.4$	$99.1$	$99.8$	$75.7$	$95.2$	$98.3$	$36.4$	$30.3$	$123.4$	$23.0$
Oscar${}^{\text{OI}}$	$71.15$	$85.9$	$97.9$	$99.5$	$72.9$	$94.3$	$97.6$	$35.3$	$29.6$	$119.5$	$22.6$

Table 4: Results with various pre-training schemes.

预训练	VQA	文本检索	图像检索	图像描述
dev	R@1	R@5	R@10	R@1	R@5	R@10	B@4	M	C	S
基线（无标签）	$70.93$	$84.4$	$98.1$	$99.5$	$73.1$	$94.5$	$97.9$	$34.5$	$29.1$	$115.6$	$21.9$
Oscar${}^{\text{VG}}$	$71.70$	$88.4$	$99.1$	$99.8$	$75.7$	$95.2$	$98.3$	$36.4$	$30.3$	$123.4$	$23.0$
Oscar${}^{\text{OI}}$	$71.15$	$85.9$	$97.9$	$99.5$	$72.9$	$94.3$	$97.6$	$35.3$	$29.6$	$119.5$	$22.6$

表 4：不同预训练方案下的结果。

6 Related Work

6 相关工作

Vision-Language Pre-training

There is a growing interest in pre-training generic models to solve a variety of V+L problems, such as visual question-answering (VQA), image-text retrieval and image captioning etc.  The existing methods [37, 38, 22, 5, 46, 35, 19, 10] employ BERT-like objectives [6] to learn cross-modal representations from a concatenated-sequence of visual region features and language token embeddings. They heavily rely on the self-attention mechanism of Transformers to learn joint representations that are appropriately contextualized in both modalities. For example, early efforts such as [22, 38] propose a two-stream and three-stream Transformer-based framework with co-attention to fuse the two modalities, respectively. Chen et al.[5] conduct comprehensive studies on the effects of different pre-training objectives for the learned generic representations. Zhou et al.[46] propose the first unified model to deal with both understanding and generation tasks, using only VQA and image captioning as the downstream tasks. In this paper, the Oscar models have been applied to a wider range of downstream tasks, including both understanding and generation tasks, and have achieved new SoTA in most of them. Compared to existing VLP methods, the most salient difference of the proposed Oscar is the use of object tags for aligning elements in two modalities. It alleviates the challenge of VLP models having to figure out the cross-modal semantic alignment from scratch, and thus improves the learning efficiency. In fact, our base model already outperforms the existing large VLP models on most V+L tasks.

视觉-语言预训练

预训练通用模型以解决视觉问答（VQA）、图文检索和图像描述等各类 V+L 问题，正受到越来越多关注。已有方法[37, 38, 22, 5, 46, 35, 19, 10]采用类似 BERT 的目标[6]，从视觉区域特征与语言词元嵌入拼接而成的序列中学习跨模态表征。它们很大程度上依靠 Transformer 的自注意力，来学习在两个模态中都得到适当上下文化的联合表征。例如，早期工作如[22, 38]分别提出基于双流和三流 Transformer、并用协同注意力融合两个模态的框架。Chen 等[5]系统研究了不同预训练目标对所学通用表征的影响。Zhou 等[46]提出第一个同时处理理解与生成任务的统一模型，当时只用 VQA 和图像描述作为下游任务。本文将 Oscar 模型应用到更广的下游任务上，既包括理解也包括生成，并在其中多数任务上取得了新的最好成绩。与已有 VLP 方法相比，Oscar 最显著的差别是用物体标签来对齐两个模态中的元素。这减轻了 VLP 模型必须从零开始找出跨模态语义对齐的负担，从而提高了学习效率。事实上，我们的基模型在多数 V+L 任务上已经超过已有的大型 VLP 模型。

Object Tags

Anderson et al. [2] introduce the bottom-up mechanism to represent an image as a set of visual regions via Faster R-CNN [28], each with an associated feature vector. It enables attention to be computed at the object level, and has quickly become the de facto standard for fine-grained image understanding tasks. In this paper, we propose to use object tags to align the object-region features in [2] in the pre-trained linguistic semantic space. The idea of utilizing object tags has been explored for image understanding [42, 43, 46]. Based on grid-wise region features of CNNs, Wu et al. [42] employ the predicted object tags only as the input to LSTM for image captioning, while You et al. [43] consider both tags and region features. Based on salient regions proposed by object detectors, Zhou et al. [46] concatenate the object prediction probability vector with region features as the visual input for VLP. Unfortunately, the tags in these works are not simultaneously associated with both object regions and word embeddings of text, resulting in a lack of grounding. Our construction of object tags with their corresponding region features & word embeddings yields more complete and informative representations for objects, particularly when the linguistic entity embeddings are pre-trained, as described next.

物体标签

Anderson 等[2]引入自底向上机制，通过 Faster R-CNN[28] 把一幅图像表示成一组视觉区域，每个区域带有相应的特征向量。这使得注意力可以在物体级别上计算，并迅速成为细粒度图像理解任务的事实标准。本文提出用物体标签，把[2]中的物体区域特征对齐到预训练的语言语义空间中。利用物体标签的想法，在图像理解中已有探索[42, 43, 46]。基于 CNN 的网格区域特征，Wu 等[42]只把预测得到的物体标签作为 LSTM 的输入来做图像描述，而 You 等[43]同时考虑标签和区域特征。基于物体检测器提出的显著区域，Zhou 等[46]把物体预测概率向量与区域特征拼接，作为 VLP 的视觉输入。可惜的是，这些工作中的标签并没有同时与物体区域和文本的词嵌入关联起来，因而缺少锚定。我们把物体标签与对应的区域特征和词嵌入放在一起构造，从而得到对物体更完整、也更有信息量的表示，尤其是在语言实体嵌入已经预训练好的时候，如下文所述。

Multimodal Embeddings

It has been shown that V+L tasks can benefit from a shared embedding space to align the inter-modal correspondences between images and text. Early attempts from Socher et al. [33] project words and image regions into a common space using kernelized canonical correlation analysis, and achieve good results for annotation and segmentation. Similar ideas are employed for image captioning [14] and text-based image retrieval [29]. In particular, the seminal work DeViSE [8] proposes to identify visual objects using semantic information gleaned from un-annotated text. This semantic information is exploited to make predictions of image labels that are not observed during training, and improves zero-shot predictions dramatically across thousands of novel labels that have never been seen by the vision model. The idea has been extended in [34, 15, 25], showing that leveraging pre-trained linguistic knowledge is highly effective for aligning semantics and improving sample efficiency in cross-modal transfer learning. Inspired by this line of research, we revisit the idea and propose to leverage the rich semantics from the learned word embeddings in the era of neural language model pre-training. Indeed, our results on novel object captioning demonstrate that Oscar helps improve the generalizability of the pre-trained models.

多模态嵌入

已经有研究表明，V+L 任务可以从一个共享嵌入空间中获益，用来对齐图像与文本之间的跨模态对应关系。早期尝试来自 Socher 等[33]，他们用核化典型相关分析把词和图像区域投影到同一空间，并在标注与分割上取得了不错的结果。类似想法也被用于图像描述[14]和基于文本的图像检索[29]。其中，开创性工作 DeViSE[8] 提出利用从未标注文本中获得的语义信息来识别视觉物体。这些语义信息被用来预测训练时未见过的图像标签，并在视觉模型从未见过的数千个新颖标签上显著改善了零样本预测。这一想法在[34, 15, 25]中得到扩展，表明借助预训练的语言知识，对对齐语义、提高跨模态迁移学习的样本效率都非常有效。受这一脉络启发，我们重新审视这一想法，并在神经语言模型预训练的时代，利用已学到的词嵌入中的丰富语义。事实上，我们在新颖物体描述上的结果也表明，Oscar 有助于提高预训练模型的泛化能力。

7 Conclusion

7 结论

In this paper, we have presented a new pre-training method Oscar, which uses object tags as anchor points to align the image and language modalities in a shared semantic space. We validate the schema by pre-training Oscar models on a public corpus with 6.5 million text-image pairs. The pre-trained models archive new state-of-the-arts on six established V+L understanding and generation tasks.

本文提出了一种新的预训练方法 Oscar，用物体标签作为锚点，在共享的语义空间中对齐图像模态与语言模态。我们在包含 650 万组图文对的公开语料上预训练 Oscar 模型，验证了这一方案。预训练模型在六项成熟的 V+L 理解与生成任务上取得了新的最好成绩。

References

参考文献

[1] Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., Anderson, P.: nocaps: novel object captioning at scale. In: ICCV (2019)

[2] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: CVPR (2018)

[3] Brown, P.F., Lai, J.C., Mercer, R.L.: Aligning sentences in parallel corpora. In: Proceedings of the 29th annual meeting on Association for Computational Linguistics (1991)

[4] Chen, W., Gan, Z., Li, L., Cheng, Y., Wang, W., Liu, J.: Meta module network for compositional visual reasoning. arXiv preprint arXiv:1910.03230 (2019)

[5] Chen, Y.C., Li, L., Yu, L., Kholy, A.E., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 (2019)

[6] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL (2019)

[7] Faghri, F., Fleet, D.J., Kiros, J.R., Fidler, S.: Vse++: Improved visual-semantic embeddings. arXiv preprint arXiv:1707.05612 2 (7),  8 (2017)

[8] Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., Mikolov, T.: DeViSE: A deep visual-semantic embedding model. In: NeurIPS (2013)

[9] Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In: CVPR (2017)

[10] Hao, W., Li, C., Li, X., Carin, L., Gao, J.: Towards learning a generic agent for vision-and-language navigation via pre-training. CVPR (2020)

[11] Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning. In: ICCV (2019)

[12] Hudson, D., Manning, C.D.: Learning by abstraction: The neural state machine. In: NeurIPS (2019)

[13] Hudson, D.A., Manning, C.D.: GQA: A new dataset for real-world visual reasoning and compositional question answering. arXiv preprint arXiv:1902.09506 (2019)

[14] Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR (2015)

[15] Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 (2014)

[16] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123 (1), 32–73 (2017)

[17] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Duerig, T., et al.: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00982 (2018)

[18] Lee, K.H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: ECCV (2018)

[19] Li, G., Duan, N., Fang, Y., Jiang, D., Zhou, M.: Unicoder-VL: A universal encoder for vision and language by cross-modal pre-training. arXiv preprint arXiv:1908.06066 (2019)

[20] Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019)

[21] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV (2014)

[22] Lu, J., Batra, D., Parikh, D., Lee, S.: VilBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NeurIPS (2019)

[23] Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-Task vision and language representation learning. arXiv preprint arXiv:1912.02315 (2019)

[24] Maaten, L.v.d., Hinton, G.: Visualizing data using t-SNE. Journal of machine learning research (2008)

[25] Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G.S., Dean, J.: Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv:1312.5650 (2013)

[26] Ordonez, V., Kulkarni, G., Berg, T.L.: Im2text: Describing images using 1 million captioned photographs. In: NeurIPS (2011)

[27] Qi, D., Su, L., Song, J., Cui, E., Bharti, T., Sacheti, A.: Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprint arXiv:2001.07966 (2020)

[28] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015)

[29] Ren, Z., Jin, H., Lin, Z., Fang, C., Yuille, A.: Joint image-text representation by gaussian visual-semantic embedding. In: Multimedia (2016)

[30] Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: CVPR (2017)

[31] Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Annual Meeting of the Association for Computational Linguistics (2018)

[32] Shi, B., Ji, L., Lu, P., Niu, Z., Duan, N.: Knowledge aware semantic concept expansion for image-text matching. In: IJCAI (2019)

[33] Socher, R., Fei-Fei, L.: Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In: CVPR (2010)

[34] Socher, R., Ganjoo, M., Manning, C.D., Ng, A.: Zero-shot learning through cross-modal transfer. In: NeurIPS (2013)

[35] Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: VL-BERT: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 (2019)

[36] Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., Artzi, Y.: A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491 (2018)

[37] Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: VideoBERT: A joint model for video and language representation learning. ICCV (2019)

[38] Tan, H., Bansal, M.: LXMERT: Learning cross-modality encoder representations from transformers. EMNLP (2019)

[39] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017)

[40] Wang, Y., Yang, H., Qian, X., Ma, L., Lu, J., Li, B., Fan, X.: Position focused attention network for image-text matching. arXiv preprint arXiv:1907.09748 (2019)

[41] Wang, Z., Liu, X., Li, H., Sheng, L., Yan, J., Wang, X., Shao, J.: CAMP: Cross-Modal adaptive message passing for text-image retrieval. In: ICCV (2019)

[42] Wu, Q., Shen, C., Liu, L., Dick, A., Van Den Hengel, A.: What value do explicit high level concepts have in vision to language problems? In: CVPR (2016)

[43] You, Q., Jin, H., Wang, Z., Fang, C., Luo, J.: Image captioning with semantic attention. In: CVPR (2016)

[44] Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 , 67–78 (2014)

[45] Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Shen, Y.D.: Dual-path convolutional image-text embedding with instance loss. arXiv preprint arXiv:1711.05535 (2017)

[46] Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J.J., Gao, J.: Unified vision-language pre-training for image captioning and VQA. AAAI (2020)

Appendix 0.A Fine-tuning Settings

附录 0.A 微调设置

Image-Text Retrieval

We adopt the widely used Karpathy split [14] on the COCO caption dataset [21] to conduct our experiments. Specifically, the dataset consists of $113,287$ images for training, $5,000$ images for validation, and $5,000$ images for testing. Each image is associated with $5$ human-generated captions. For the Oscar${}_{\text{B}}$ model, we fine-tune with a batch size of $256$ for $40$ epochs. The initial learning rate is set to $2e^{-5}$ and linearly decreases. For the Oscar${}_{\text{L}}$ model, we fine-tune with a batch size of $128$ for $40$ epochs. The initial learning rate is set to $1e^{-5}$ and linearly decreases. We use the validation set for parameter tuning. We compare with several existing methods, including DVSA [14], VSE++ [7], DPC [45], CAMP [41], SCAN [18], SCG [32], PFAN [40], Unicoder-VL [19], 12-in-1 [23], UNITER [5].

图文检索

我们采用 COCO 描述数据集[21]上广泛使用的 Karpathy 划分[14]进行实验。具体来说，数据集包含 $113,287$ 张训练图像、$5,000$ 张验证图像和 $5,000$ 张测试图像。每张图像配有 $5$ 句人工撰写的描述。对 Oscar${}_{\text{B}}$ 模型，我们以批大小 $256$ 微调 $40$ 个 epoch。初始学习率设为 $2e^{-5}$，并线性下降。对 Oscar${}_{\text{L}}$ 模型，我们以批大小 $128$ 微调 $40$ 个 epoch。初始学习率设为 $1e^{-5}$，并线性下降。用验证集做参数选择。比较的已有方法包括 DVSA[14]、VSE++[7]、DPC[45]、CAMP[41]、SCAN[18]、SCG[32]、PFAN[40]、Unicoder-VL[19]、12-in-1[23]、UNITER[5]。

Image Captioning

Though the training objective (*i.e.,*seq2seq) for image captioning is different from that used in pre-training (*i.e.,*bidirectional attention-based mask token loss), we directly fine-tune Oscar for image captioning on COCO without additional pre-training on Conceptual Captions [31]. This is to validate the generalization ability of the Oscar models for generation tasks. We use the same Karpathy split [14]. During training, we randomly select $15\%$ of caption tokens with a maximum of $3$ tokens per caption to be masked out. For the Oscar${}_{\text{B}}$ model, we fine-tune with cross-entropy loss for $40$ epochs with a batch size of $256$ and an initial learning rate of $3e^{-5}$ and then with CIDEr optimization [30] for $5$ epochs with a batch size of $64$ and initial learning rate of $1e^{-6}$. For the Oscar${}_{\text{L}}$ model, we fine-tune for $30$ epochs with a batch size of $128$ and an initial learning rate of $1e^{-5}$ and then with CIDEr optimization for another $3$ epochs with a batch size of $48$ and learning rate of {$1e^{-6}$, $5e^{-7}$}. We compare with several existing methods, including BUTD [2], VLP [46], AoANet [11].

图像描述

尽管图像描述的训练目标（即 seq2seq）与预训练所用目标（即基于双向注意力的掩码词元损失）不同，我们仍直接在 COCO 上微调 Oscar 做图像描述，而不再先在 Conceptual Captions[31] 上额外预训练。这样做是为了检验 Oscar 模型在生成任务上的泛化能力。我们使用同样的 Karpathy 划分[14]。训练时，随机选择每句描述中 15% 的词元掩掉，每句最多掩 $3$ 个词元。对 Oscar${}_{\text{B}}$ 模型，我们先以交叉熵损失微调 $40$ 个 epoch，批大小为 $256$，初始学习率为 $3e^{-5}$；再以 CIDEr 优化[30]训练 $5$ 个 epoch，批大小为 $64$，初始学习率为 $1e^{-6}$。对 Oscar${}_{\text{L}}$ 模型，我们先微调 $30$ 个 epoch，批大小为 $128$，初始学习率为 $1e^{-5}$；再用 CIDEr 优化训练 $3$ 个 epoch，批大小为 $48$，学习率为 $1e^{-6}$ 或 $5e^{-7}$。比较的已有方法包括 BUTD[2]、VLP[46]、AoANet[11]。

NoCaps

Since NoCaps images are collected from Open Images. We train an object detector using the Open Images training set and applied it to generate the tags. We conduct experiments from BERT model directly without pre-training as required by the task guidelines. For the Oscar${}_{\text{B}}$ model, we train $40$ epoch with a batch size of $256$ and learning rate $3e^{-5}$; further we perform CIDEr optimization with learning rate $1e^{-6}$ and batch size $64$ for $5$ epochs. During inference, we use constrained beam search for decoding. We compare Oscar with UpDown [1] on this task.

NoCaps

由于 NoCaps 的图像采集自 Open Images，我们用 Open Images 训练集训练了一个物体检测器，并用它生成标签。按照任务指南的要求，我们直接从 BERT 模型出发做训练，不做预训练。对 Oscar${}_{\text{B}}$ 模型，我们训练 $40$ 个 epoch，批大小为 $256$，学习率为 $3e^{-5}$；再以学习率 $1e^{-6}$、批大小 $64$ 做 $5$ 个 epoch 的 CIDEr 优化。推理时使用约束束搜索解码。我们在该任务上与 UpDown[1] 进行比较。

VQA

For VQA training, we random sample a set of 2k images from the MS COCO validation set as our validation set, the rest of images in the training and validation are used in the VQA finetuning. For the Oscar${}_{\text{B}}$ model, we fine-tune for $25$ epochs with a learning rate of $5e^{-5}$ and a batch size of $128$. For the Oscar${}_{\text{L}}$ model, we fine-tune for $25$ epochs with with a learning rate of $3e^{-5}$ and a batch size of $96$.

VQA

VQA 训练时，我们从 MS COCO 验证集中随机抽样 2 千张图像作为自己的验证集，训练集约与验证集中的其余图像都用于 VQA 微调。对 Oscar${}_{\text{B}}$ 模型，我们微调 $25$ 个 epoch，学习率为 $5e^{-5}$，批大小为 $128$。对 Oscar${}_{\text{L}}$ 模型，我们微调 $25$ 个 epoch，学习率为 $3e^{-5}$，批大小为 $96$。

GQA

The fine-tuning procedure of GQA is similar to that of VQA. For the Oscar${}_{\text{B}}$ model, we fine-tune for $5$ epochs with a learning rate of $5e^{-5}$ and a batch size of $128$. We compare with four existing methods, including LXMERT [38], MMN [4], 12-in-1 [23], NSM [12].

GQA

GQA 的微调流程与 VQA 类似。对 Oscar${}_{\text{B}}$ 模型，我们微调 $5$ 个 epoch，学习率为 $5e^{-5}$，批大小为 $128$。比较的已有方法有四种，包括 LXMERT[38]、MMN[4]、12-in-1[23]、NSM[12]。

NLVR2

For the Oscar${}_{\text{B}}$ model, we fine-tune for $20$ epochs with learning rate {$2e^{-5}$, $3e^{-5}$, $5e^{-5}$} and a batch size of $72$. For the Oscar${}_{\text{L}}$ model, we fine-tune for $20$ epochs with learning rate of {$2e^{-5}$, $3e^{-5}$} and a batch size of $48$.

NLVR2

对 Oscar${}_{\text{B}}$ 模型，我们微调 $20$ 个 epoch，学习率为 $2e^{-5}$、$3e^{-5}$ 或 $5e^{-5}$，批大小为 $72$。对 Oscar${}_{\text{L}}$ 模型，我们微调 $20$ 个 epoch，学习率为 $2e^{-5}$ 或 $3e^{-5}$，批大小为 $48$。

Appendix 0.B Pre-training Corpus

附录 0.B 预训练语料

Table 5 shows the statistics of image and text of the corpus.

表 5 给出了语料中图像与文本的统计。

Source	COCO	CC	SBU	Flicker30k	VQA	GQA	VG-QA	Total
(train)	(all)	(all)	(train)	(train)	(bal-train)	(train)
Image/Text	112k/560k	3.0M/3.0M	840k/840k	29k/145k	83k/444k	79k/1026k	48k/484k	4.1M/6.5M

Table 5: Statistics of the pre-training corpus.

来源	COCO	CC	SBU	Flickr30k	VQA	GQA	VG-QA	合计
(train)	(all)	(all)	(train)	(train)	(bal-train)	(train)
图像/文本	112k/560k	3.0M/3.0M	840k/840k	29k/145k	83k/444k	79k/1026k	48k/484k	4.1M/6.5M

表 5：预训练语料的统计。

Appendix 0.C More Results

附录 0.C 更多结果

The enlarged $t$-SNE visualization results of Oscar and baseline (no tags) are shown in Fig. 7 and Fig. 8, respectively.

Oscar 与基线（无标签）放大后的 $t$-SNE 可视化结果分别见图 7 和图 8。

Figure 7: Feature visualization of Oscar. We observe small distances between text and image features of the same object; some of them are perfectly aligned, as demonstrated by the overlapping regions.

图 7：Oscar 的特征可视化。我们观察到同一物体的文本特征与图像特征距离很小；其中一些完全对齐，重叠区域即是例证。

Figure 8: Feature visualization of baseline (no tags). For several object classes, their text and image features are largely separated (*e.g.,*person, umbrella, zebra). The distance of image features between some objects is too small (*e.g.,*bench, chair, couch).

图 8：基线（无标签）的特征可视化。对若干物体类别，其文本特征与图像特征分得很开（例如 person、umbrella、zebra）。有些物体之间的图像特征距离又过小（例如 bench、chair、couch）。

Acknowledgement

致谢

We thank Yonatan Bisk, Hannaneh Hajishirzi, Xiaodong Liu, Sachin Mehta, Hamid Palangi and Arun Sacheti, Rowan Zellers for valuable discussions and comments, and the Microsoft Research Technical Support team for managing the GPU clusters.

我们感谢 Yonatan Bisk、Hannaneh Hajishirzi、Xiaodong Liu、Sachin Mehta、Hamid Palangi、Arun Sacheti 和 Rowan Zellers 的宝贵讨论与意见，并感谢微软研究院技术支持团队对 GPU 集群的管理。