Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
用嘈杂文本监督放大视觉与视觉–语言表征学习
Chao Jia Affiliation: Google Research Correspondence to: chaojia@google.com Yinfei Yang Affiliation: Google Research Correspondence to: yinfeiy@google.com Ye Xia Affiliation: Google Research Yi-Ting Chen Affiliation: Google Research Zarana Parekh Affiliation: Google Research Hieu Pham Affiliation: Google Research Quoc V. Le Affiliation: Google Research Yunhsuan Sung Affiliation: Google Research Zhen Li Affiliation: Google Research Tom Duerig Affiliation: Google Research
Abstract
摘要
Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.
预训练表征对许多 NLP 与感知任务日益关键。NLP 表征学习已转向无需人工标注的原始文本训练，但视觉与视觉–语言表征仍严重依赖昂贵或需专家知识的精选训练集。视觉应用多在带显式类别标签的数据集（如 ImageNet、OpenImages）上学习表征；视觉–语言常用 Conceptual Captions、MSCOCO 或 CLIP 等，都涉及 nontrivial 的数据采集与清洗。昂贵的策展限制了数据规模，从而阻碍模型放大。本文利用逾十亿图像–alt-text 对的嘈杂数据集——相对 Conceptual Captions 流程，不做昂贵过滤或后处理。简单双塔编码器以对比损失对齐图像与文本表征。我们表明语料规模可弥补噪声，即便学习方案如此简单也能得到当时最优表征。视觉表征迁移到 ImageNet、VTAB 等分类任务表现强；对齐后的视觉与语言表征支持零样本图像分类，并在 Flickr30K 与 MSCOCO 图文检索上刷新结果，甚至超过更复杂的交叉注意力模型。表征还支持带复杂文本以及「文本+图像」查询的跨模态搜索。
Keywords:
Figure 1: A summary of our method, ALIGN. Visual and language representations are jointly learned from noisy image alt-text data. The representations can be used for vision-only or vision-language task transfer. Without any fine-tuning, ALIGN powers zero-shot visual classification and cross-modal search including image-to-text search, text-to-image search and even search with joint image+text queries.
关键词：
图 1：ALIGN 方法概要。视觉与语言表征从嘈杂图像 alt-text 数据联合学习。表征可用于纯视觉或视觉–语言任务迁移。无需微调，ALIGN 即可支持零样本视觉分类与跨模态搜索，包括图搜文、文搜图，甚至联合图像+文本查询。
1 Introduction
1 引言
In the existing literature, visual and vision-language representation learning are mostly studied separately with different training data sources. In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al. 2009), OpenImages (Kuznetsova et al. 2020), and JFT-300M (Sun et al. 2017; Kolesnikov et al. 2020) has proven to be critical for improving performance on downstream tasks via transfer learning. Curation of such pre-training datasets requires heavy work on data gathering, sampling, and human annotation, and hence is difficult to scale.
既有文献中，视觉与视觉–语言表征学习多在不同训练数据源上分别研究。视觉领域，在 ImageNet（Deng et al. 2009）、OpenImages（Kuznetsova et al. 2020）、JFT-300M（Sun et al. 2017；Kolesnikov et al. 2020）等大规模有监督数据上预训练，对迁移学习下游任务至关重要。策展这类预训练集需要大量数据采集、采样与人工标注，因而难以放大。
Pre-training has also become the de-facto approach in vision-language modeling (Lu et al. 2019; Chen et al. 2020c; Li et al. 2020). However, vision-language pre-training datasets such as Conceptual Captions (Sharma et al. 2018), Visual Genome Dense Captions (Krishna et al. 2016), and ImageBERT (Qi et al. 2020) require even heavier work on human annotation, semantic parsing, cleaning and balancing. As a result, the scales of these datasets are only in the realm of 10M examples. This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. 2019; Radford et al. 2019; Yang et al. 2019; Liu et al. 2019b; Raffel et al. 2020).
预训练也已成为视觉–语言建模的事实标准做法（Lu et al. 2019；Chen et al. 2020c；Li et al. 2020）。但 Conceptual Captions（Sharma et al. 2018）、Visual Genome Dense Captions（Krishna et al. 2016）、ImageBERT（Qi et al. 2020）等视觉–语言预训练集，对人工标注、语义解析、清洗与平衡的要求更高。结果是这些数据集规模仅约 10M 样本，至少比视觉领域对应数据小一个数量级，更远小于 NLP 预训练所用的互联网大规模文本语料（如 Devlin et al. 2019；Radford et al. 2019；Yang et al. 2019；Liu et al. 2019b；Raffel et al. 2020）。
In this work, we leverage a dataset of over one billion noisy image alt-text pairs to scale visual and vision-language representation learning. We follow the procedures described in the Conceptual Captions dataset (Sharma et al. 2018) to have a large noisy dataset. But instead of applying the complex filtering and post-processing steps as proposed by (Sharma et al. 2018) to clean the dataset, we only apply simple frequency-based filtering. The resulting dataset is noisy, but is two orders of magnitude larger than the Conceptual Captions dataset. We show that visual and vision-language representations pre-trained on our exascale dataset achieve very strong performance on a wide range of tasks.
本文利用逾十亿嘈杂图像–alt-text 对，放大视觉与视觉–语言表征学习。我们沿用 Conceptual Captions（Sharma et al. 2018）所述流程构建大规模嘈杂数据；但不做该文提出的复杂过滤与后处理，只做简单的基于频率的过滤。所得数据集更嘈杂，却比 Conceptual Captions 大约两个数量级。我们表明：在这一超大规模数据上预训练的视觉与视觉–语言表征，在广泛任务上表现很强。
To train our model, we use an objective that aligns the visual and language representations in a shared latent embedding space using a simple dual-encoder architecture. Similar objectives has been applied to learning visual-semantic embeddings (VSE) (Frome et al. 2013; Faghri et al. 2018). We name our model ALIGN: A Large-scale ImaGe and Noisy-text embedding. Image and text encoders are learned via a contrastive loss (formulated as normalized softmax) that pushes the embeddings of matched image-text pair together while pushing those of non-matched image-text pair apart. This is one of the most effective loss functions for both self-supervised (Chen et al. 2020b) and supervised (Zhai & Wu 2019; Musgrave et al. 2020) representation learning. Considering paired texts as fine-grained labels of images, our image-to-text contrastive loss is analogous to the conventional label-based classification objective; and the key difference is that the text encoder generates the “label” weights. The top-left of Figure 1 summarizes the method we use in ALIGN.
训练时，我们用简单双塔结构，在共享潜在嵌入空间对齐视觉与语言表征。类似目标曾用于学习视觉–语义嵌入（VSE）（Frome et al. 2013；Faghri et al. 2018）。模型命名为 ALIGN：A Large-scale ImaGe and Noisy-text embedding。图像与文本编码器经对比损失（形式为归一化 softmax）学习：匹配图文对的嵌入被拉近，不匹配对被推远。这是自监督（Chen et al. 2020b）与有监督（Zhai & Wu 2019；Musgrave et al. 2020）表征学习中最有效的损失之一。若把配对文本视为图像的细粒度标签，则图像到文本的对比损失类似于传统基于标签的分类目标；关键差异在于「标签」权重由文本编码器生成。图 1 左上概括了 ALIGN 所用方法。
The aligned image and text representations are naturally suited for cross-modality matching/retrieval tasks and achieve state-of-the-art (SOTA) results in corresponding benchmarks. For instance, ALIGN outperforms the previous SOTA method by over 7% in most zero-shot and fine-tuned R@1 metrics in Flickr30K and MSCOCO. Moreover, such cross-modality matching naturally enables zero-shot image classification when feeding the classnames into the text encoder, achieving 76.4% top-1 accuracy in ImageNet without using any of its training samples. The image representation itself also achieves superior performance in various downstream visual tasks. For example, ALIGN achieves 88.64% top-1 accuracy in ImageNet. Figure 1-bottom shows the cross-modal retrieval examples that come from a real retrieval system built by ALIGN.
对齐后的图像与文本表征天然适于跨模态匹配/检索，并在相应基准上达到当时最优（SOTA）。例如，在 Flickr30K 与 MSCOCO 多数零样本与微调 R@1 指标上，ALIGN 相对先前 SOTA 提升超过 7%。此外，跨模态匹配在把类名送入文本编码器时自然支持零样本图像分类，在 ImageNet 上不使用任何训练样本达到 76.4% top-1。图像表征本身在多种下游视觉任务上也表现优异，例如 ImageNet 上达到 88.64% top-1。图 1 下部展示了由 ALIGN 构建的真实检索系统给出的跨模态检索示例。
2 Related Work
2 相关工作
High-quality visual representations for classification or retrieval are usually pre-trained on large-scale labeled datasets (Mahajan et al. 2018; Kolesnikov et al. 2020; Dosovitskiy et al. 2021; Juan et al. 2020). Recently, self-supervised (Chen et al. 2020b; Tian et al. 2020; He et al. 2020; Misra & Maaten 2020; Li et al. 2021; Grill et al. 2020; Caron et al. 2020) and semi-supervised learning (Yalniz et al. 2019; Xie et al. 2020; Pham et al. 2020) have been studied as alternative paradigms. However, models trained by these methods so far show limited transferability to downstream tasks (Zoph et al. 2020).
用于分类或检索的高质量视觉表征，通常在大规模标注数据上预训练（Mahajan et al. 2018；Kolesnikov et al. 2020；Dosovitskiy et al. 2021；Juan et al. 2020）。近期自监督（Chen et al. 2020b；Tian et al. 2020；He et al. 2020；Misra & Maaten 2020；Li et al. 2021；Grill et al. 2020；Caron et al. 2020）与半监督学习（Yalniz et al. 2019；Xie et al. 2020；Pham et al. 2020）被作为替代范式研究，但这些方法训出的模型对下游任务的可迁移性迄今仍有限（Zoph et al. 2020）。
Leveraging images and natural language captions is another direction of learning visual representations. Joulin et al. 2015; Li et al. 2017; Desai & Johnson 2020; Sariyildiz et al. 2020; Zhang et al. 2020 show that a good visual representation can be learned by predicting the captions from images, which inspires our work. These works are however limited to small datasets such as Flickr (Joulin et al. 2015; Li et al. 2017) and COCO Captions (Desai & Johnson 2020; Sariyildiz et al. 2020), and the resulting models don’t produce a vision-language representation that is needed for tasks like cross-modal retrieval.
利用图像与自然语言标题是学习视觉表征的另一方向。Joulin et al. 2015；Li et al. 2017；Desai & Johnson 2020；Sariyildiz et al. 2020；Zhang et al. 2020 表明，通过从图像预测标题可学到良好视觉表征，启发了本文。但这些工作限于 Flickr（Joulin et al. 2015；Li et al. 2017）与 COCO Captions（Desai & Johnson 2020；Sariyildiz et al. 2020）等小数据集，所得模型也不产出跨模态检索所需的视觉–语言表征。
In the vision-language representation learning domain, visual-semantic embeddings (VSE) (Frome et al. 2013; Faghri et al. 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al. 2014; Karpathy et al. 2014; Kiros et al.; Nam et al. 2017; Li et al. 2019; Messina et al. 2020; Chen et al. 2020a) have been proposed. Recently more advanced models emerge with cross-modal attention layers (Liu et al. 2019a; Lu et al. 2019; Chen et al. 2020c; Huang et al. 2020b) and show superior performance in image-text matching tasks. However, they are orders of magnitudes slower and hence impractical for image-text retrieval systems in the real world. In contrast, our model inherits the simplest VSE form, but still outperforms all previous cross-attention models in image-text matching benchmarks.
在视觉–语言表征学习领域，已有视觉–语义嵌入（VSE）（Frome et al. 2013；Faghri et al. 2018）及其改进（如利用目标检测器、稠密特征图或多注意力层）（Socher et al. 2014；Karpathy et al. 2014；Kiros et al.；Nam et al. 2017；Li et al. 2019；Messina et al. 2020；Chen et al. 2020a）。近期带跨模态注意力层的更先进模型出现（Liu et al. 2019a；Lu et al. 2019；Chen et al. 2020c；Huang et al. 2020b），在图文匹配上表现更强，但速度慢几个数量级，难以用于真实世界图文检索系统。相反，我们的模型继承最简单的 VSE 形式，却仍在图文匹配基准上超过所有先前交叉注意力模型。
Closely related to our work is CLIP (Radford et al. 2021), which proposes visual representation learning via natural language supervision in a similar contrastive learning setting. Besides using different vision and language encoder architectures, the key difference is on training data: ALIGN follows the natural distribution of image-text pairs from the raw alt-text data, while CLIP collects the dataset by first constructing an allowlist of high-frequency visual concepts from English Wikipedia. We demonstrate that strong visual and vision-language representations can be learned with a dataset that doesn’t require expert knowledge to curate.
与本文密切相关的是 CLIP（Radford et al. 2021）：在类似对比学习设定下，借助自然语言监督做视觉表征学习。除视觉与语言编码器结构不同外，关键差异在训练数据：ALIGN 遵循原始 alt-text 数据中图文对的自然分布，而 CLIP 先从英文维基百科构造高频视觉概念白名单再收集数据。我们表明：无需专家知识策展的数据集，也能学到强视觉与视觉–语言表征。
3 A Large-Scale Noisy Image-Text Dataset
3 大规模嘈杂图文数据集
The focus of our work is to scale up visual and vision-language representation learning. For this purpose, we resort to a much larger dataset than existing ones. Specifically, we follow the methodology of constructing Conceptual Captions dataset (Sharma et al. 2018) to get a version of raw English alt-text data (image and alt-text pairs). The Conceptual Captions dataset was cleaned by heavy filtering and post-processing. Here, for the purpose of scaling, we trade quality for scale by relaxing most of the cleaning steps in the original work. Instead, we only apply minimal frequency-based filtering as detailed below. The result is a much larger (1.8B image-text pairs) but noisier dataset. Figure 2 shows some sample image-text pairs from the dataset.
本文重点是放大视觉与视觉–语言表征学习，因此求助于远大于既有集合的数据。具体而言，我们沿用 Conceptual Captions（Sharma et al. 2018）的构建方法，得到一份原始英文 alt-text 数据（图像与 alt-text 对）。Conceptual Captions 经重度过滤与后处理清洗；此处为放大规模，我们放宽原工作中多数清洗步骤，以质量换规模，仅做如下基于频率的最小过滤。结果是大得多（18 亿图文对）但更嘈杂的数据集。图 2 展示了数据集中的若干样本图文对。
Figure 2: Example image-text pairs randomly sampled from the training dataset of ALIGN. One clearly noisy text annotation is marked in italics.
图 2：从 ALIGN 训练数据中随机采样的示例图文对。一处明显嘈杂的文本标注以斜体标出。
Image-based filtering.
Following Sharma et al. 2018, we remove pornographic images and keep only images whose shorter dimension is larger than 200 pixels and aspect ratio is smaller than 3. Images with more than 1000 associated alt-texts are discarded. To ensure that we don’t train on test images, we also remove duplicates or near-duplicates of test images in all downstream evaluation datasets (e.g., ILSVRC-2012, Flickr30K, and MSCOCO). See Appendix A for more details.
基于图像的过滤。
沿用 Sharma et al. 2018，我们移除色情图像，只保留较短边大于 200 像素且宽高比小于 3 的图像。关联 alt-text 超过 1000 条的图像丢弃。为确保不在测试图像上训练，还移除所有下游评估集（如 ILSVRC-2012、Flickr30K、MSCOCO）测试图像的重复或近重复。详见附录 A。
Text-based filtering.
We exclude alt-texts that are shared by more than 10 images. These alt-texts are often irrelevant to the content of the images (e.g., “1920x1080”, “alt_img”, and “cristina”). We also discard alt-texts that contain any rare token (outside of 100 million most frequent unigrams and bigrams from the raw dataset), and those that are either too short (3 unigrams) or too long (20 unigrams). This removes noisy texts like “image_tid 25&id mggqpuweqdpd&cache 0&lan_code 0”, or texts that are too generic to be useful.
基于文本的过滤。
我们排除被超过 10 张图像共享的 alt-text；这类文本常与图像内容无关（如 “1920x1080”、“alt_img”、“cristina”）。还丢弃含任何稀有 token（不在原始数据最频繁的 1 亿个 unigram 与 bigram 内）的 alt-text，以及过短（3 个 unigram）或过长（20 个 unigram）者。由此去掉类似 “image_tid 25&id mggqpuweqdpd&cache 0&lan_code 0” 的嘈杂文本，或过于泛化而无用的文本。
4 Pre-training and Task Transfer
4 预训练与任务迁移
4.1 Pre-training on Noisy Image-Text Pairs
4.1 在嘈杂图文对上预训练
We pre-train ALIGN using a dual-encoder architecture. The model consists of a pair of image and text encoders with a cosine-similarity combination function at the top. We use EfficientNet with global pooling (without training the 1x1 conv layer in the classification head) as the image encoder and BERT with [CLS] token embedding as the text embedding encoder (we generate 100k wordpiece vocabulary from our training dataset). A fully-connected layer with linear activation is added on top of BERT encoder to match the dimension from the image tower. Both image and text encoders are trained from scratch.
我们用双塔结构预训练 ALIGN。模型由一对图像与文本编码器组成，顶部以余弦相似度组合。图像编码器为带全局池化的 EfficientNet（不训练分类头中的 1×1 卷积）；文本嵌入编码器为取 [CLS] token 嵌入的 BERT（词表为我们训练数据上生成的 10 万 wordpiece）。BERT 编码器之上加一层线性全连接，以匹配图像塔维度。图像与文本编码器均从零训练。
The image and text encoders are optimized via normalized softmax loss (Zhai & Wu 2019). In training, we treat matched image-text pairs as positive and all other random image-text pairs that can be formed in a training batch as negative.
图像与文本编码器经归一化 softmax 损失（Zhai & Wu 2019）优化。训练时，匹配图文对视为正例，同一训练 batch 内可组成的其余随机图文对视为负例。
We minimize the sum of two losses: one for image-to-text classification
我们最小化两项损失之和：一项用于图像到文本分类
and the other for text-to-image classification
另一项用于文本到图像分类
Here, and are the normalized embedding of image in the -th pair and that of text in the -th pair, respectively. is the batch size, and is the temperature to scale the logits. For in-batch negatives to be more effective, we concatenate embeddings from all computing cores to form a much larger batch. The temperature variable is crucial as both image and text embeddings are L2-normalized. Instead of manually sweeping for the optimal temperature value, we find that it can be effectively learned together with all the other parameters.
其中 与 分别为第 对中图像与第 对中文本的归一化嵌入； 为 batch 大小， 为缩放 logits 的温度。为使 batch 内负例更有效，我们拼接所有计算核心上的嵌入以形成更大 batch。由于图像与文本嵌入均经 L2 归一化，温度变量至关重要。我们不手工扫描最优温度，而发现它可与其他参数一并有效学习。
4.2 Transferring to Image-Text Matching & Retrieval
4.2 迁移到图文匹配与检索
We evaluate ALIGN models on image-to-text and text-to-image retrieval tasks, with and without finetuning. Two benchmark datasets are considered: Flickr30K (Plummer et al. 2015) and MSCOCO (Chen et al. 2015). We also evaluate ALIGN on Crisscrossed Captions (CxC) (Parekh et al. 2021), which is an extension of MSCOCO with additional human semantic similarity judgments for caption-caption, image-image, and image-caption pairs. With extended annotations, CxC enables four intra- and inter-modal retrieval tasks including image-to-text, text-to-image, text-to-text, and image-to-image retrieval, and three semantic similarity tasks including semantic textual similarity (STS), semantic image similarity (SIS), and semantic image-text similarity (SITS). As the training set is identical to the original MSCOCO, we can directly evaluate the MSCOCO fine-tuned ALIGN model on CxC annotations.
我们在有/无微调两种设定下，于图搜文与文搜图任务上评估 ALIGN，基准为 Flickr30K（Plummer et al. 2015）与 MSCOCO（Chen et al. 2015）。还在 Crisscrossed Captions（CxC）（Parekh et al. 2021）上评估——它是 MSCOCO 的扩展，额外提供标题–标题、图像–图像、图像–标题的人工语义相似度判断。借助扩展标注，CxC 支持图搜文、文搜图、文搜文、图搜图四类模态内/间检索，以及语义文本相似度（STS）、语义图像相似度（SIS）、语义图文相似度（SITS）三项相似度任务。因训练集与原 MSCOCO 相同，可直接在 CxC 标注上评估经 MSCOCO 微调的 ALIGN。
4.3 Transferring to Visual Classification
4.3 迁移到视觉分类
We first apply zero-shot transfer of ALIGN to visual classification tasks on ImageNet ILSVRC-2012 benchmark (Deng et al. 2009) and its variants including ImageNet-R(endition) (Hendrycks et al. 2020) (non-natural images such as art, cartoons, sketches), ImageNet-A(dversarial) (Hendrycks et al. 2021) (more challenging images for ML models), and ImageNet-V2 (Recht et al. 2019). All of these variants follow the same set (or a subset) of ImageNet classes, while the images in ImageNet-R and ImageNet-A are sampled from drastically different distributions from ImageNet.
我们先把 ALIGN 零样本迁移到 ImageNet ILSVRC-2012（Deng et al. 2009）及其变体上的视觉分类：ImageNet-R（rendition）（Hendrycks et al. 2020）（艺术、卡通、素描等非自然图像）、ImageNet-A（adversarial）（Hendrycks et al. 2021）（对 ML 模型更具挑战的图像）、ImageNet-V2（Recht et al. 2019）。这些变体遵循相同（或子集）的 ImageNet 类别，而 ImageNet-R 与 ImageNet-A 中的图像采样自与 ImageNet 截然不同的分布。
We also transfer the image encoder to downstream visual classification tasks. For this purpose, we use the ImageNet as well as a handful of smaller fine-grained classification datasets such as Oxford Flowers-102 (Nilsback & Zisserman 2008), Oxford-IIIT Pets (Parkhi et al. 2012), Stanford Cars (Krause et al. 2013), and Food101 (Bossard et al. 2014). For ImageNet, results from two settings are reported: training the top classification layer only (with frozen ALIGN image encoder) and fully fine-tuned. Only the latter setting is reported for fine-grained classification benchmarks. Following Kolesnikov et al. 2020, we also evaluate the robustness of our model on Visual Task Adaptation Benchmark (VTAB) (Zhai et al. 2019) which consists of 19 diverse (covering subgroups of natural, specialized and structured image classification tasks) visual classification tasks with 1000 training samples each.
我们还将图像编码器迁移到下游视觉分类。为此使用 ImageNet 以及若干更小的细粒度分类集，如 Oxford Flowers-102（Nilsback & Zisserman 2008）、Oxford-IIIT Pets（Parkhi et al. 2012）、Stanford Cars（Krause et al. 2013）、Food101（Bossard et al. 2014）。ImageNet 报告两种设定：只训顶部分类层（冻结 ALIGN 图像编码器）与全量微调；细粒度基准只报告后者。沿用 Kolesnikov et al. 2020，我们还在 Visual Task Adaptation Benchmark（VTAB）（Zhai et al. 2019）上评估稳健性——该基准含 19 项多样（覆盖自然、专用与结构化图像分类子组）视觉分类任务，各含 1000 个训练样本。
5 Experiments and Results
5 实验与结果
We train our ALIGN models from scratch, using the open-sourced implementation of EfficientNet as the image encoder and BERT as the text encoder. Unless in the ablation study, we use the results of ALIGN where the image encoder is EfficientNet-L2 and the text encoder is BERT-Large. The image encoder is trained at resolution of 289 289 pixels no matter what EfficientNet variant is used. We first resize input images to 346 346 resolution and then perform random crop (with additional random horizontal flip) in training and central crop in evaluation. For BERT we use wordpiece sequence of maximum 64 tokens since the input texts are no longer than 20 unigrams. The softmax temperature variable is initialized as 1.0 (this temperature variable is shared between image-to-text loss and text-to-image loss) and we use 0.1 as label smoothing parameter in the softmax losses. We use LAMB optimizer (You et al. 2020)[^1] with weight decay ratio 1e-5. The learning rate is warmed up linearly to 1e-3 from zero in 10k steps, and then linearly decay to zero in 1.2M steps (12 epochs). We train the model on 1024 Cloud TPUv3 cores with 16 positive pairs on each core. Therefore the total effective batch size is 16384.
我们从零训练 ALIGN：图像编码器使用开源 EfficientNet 实现，文本编码器为 BERT。除非消融实验另有说明，报告的 ALIGN 结果使用 EfficientNet-L2 图像编码器与 BERT-Large 文本编码器。无论 EfficientNet 变体如何，图像编码器均在 289289 分辨率训练：先将输入缩放到 346346，训练时随机裁剪（并随机水平翻转），评估时中心裁剪。BERT 使用最长 64 个 token 的 wordpiece 序列，因输入文本不超过 20 个 unigram。softmax 温度变量初始化为 1.0（图像到文本与文本到图像损失共享该温度），softmax 损失中标签平滑为 0.1。优化器用 LAMB（You et al. 2020）[^1]，权重衰减 1e-5。学习率在 10k 步内从 0 线性升至 1e-3，再在 1.2M 步（约 12 个 epoch）内线性衰减至 0。在 1024 个 Cloud TPUv3 核心上训练，每核 16 个正对，因而有效全局 batch 为 16384。
| Flickr30K (1K test set) | MSCOCO (5K test set) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| image text | text image | image text | text image | ||||||||||
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | ||
| Zero-shot | ImageBERT | 70.7 | 90.2 | 94.0 | 54.3 | 79.6 | 87.5 | 44.0 | 71.2 | 80.4 | 32.3 | 59.0 | 70.2 |
| UNITER | 83.6 | 95.7 | 97.7 | 68.7 | 89.2 | 93.9 | - | - | - | - | - | - | |
| CLIP | 88.0 | 98.7 | 99.4 | 68.7 | 90.6 | 95.2 | 58.4 | 81.5 | 88.1 | 37.8 | 62.4 | 72.2 | |
| ALIGN | 88.6 | 98.7 | 99.7 | 75.7 | 93.8 | 96.8 | 58.6 | 83.0 | 89.7 | 45.6 | 69.8 | 78.6 | |
| Fine-tuned | GPO | 88.7 | 98.9 | 99.8 | 76.1 | 94.5 | 97.1 | 68.1 | 90.2 | - | 52.7 | 80.2 | - |
| UNITER | 87.3 | 98.0 | 99.2 | 75.6 | 94.1 | 96.8 | 65.7 | 88.6 | 93.8 | 52.9 | 79.9 | 88.0 | |
| ERNIE-ViL | 88.1 | 98.0 | 99.2 | 76.7 | 93.6 | 96.4 | - | - | - | - | - | - | |
| VILLA | 87.9 | 97.5 | 98.8 | 76.3 | 94.2 | 96.8 | - | - | - | - | - | - | |
| Oscar | - | - | - | - | - | - | 73.5 | 92.2 | 96.0 | 57.5 | 82.8 | 89.8 | |
| ALIGN | 95.3 | 99.8 | 100.0 | 84.9 | 97.4 | 98.6 | 77.0 | 93.5 | 96.9 | 59.9 | 83.3 | 89.8 |
Table 1: Image-text retrieval results on Flickr30K and MSCOCO datasets (zero-shot and fine-tuned). ALIGN is compared with ImageBERT (Qi et al. 2020), UNITER (Chen et al. 2020c), CLIP (Radford et al. 2021), GPO (Chen et al. 2020a), ERNIE-ViL (Yu et al. 2020), VILLA (Gan et al. 2020), and Oscar (Li et al. 2020).
| Flickr30K (1K test set) | MSCOCO (5K test set) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| image text | text image | image text | text image | ||||||||||
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | ||
| Zero-shot | ImageBERT | 70.7 | 90.2 | 94.0 | 54.3 | 79.6 | 87.5 | 44.0 | 71.2 | 80.4 | 32.3 | 59.0 | 70.2 |
| UNITER | 83.6 | 95.7 | 97.7 | 68.7 | 89.2 | 93.9 | - | - | - | - | - | - | |
| CLIP | 88.0 | 98.7 | 99.4 | 68.7 | 90.6 | 95.2 | 58.4 | 81.5 | 88.1 | 37.8 | 62.4 | 72.2 | |
| ALIGN | 88.6 | 98.7 | 99.7 | 75.7 | 93.8 | 96.8 | 58.6 | 83.0 | 89.7 | 45.6 | 69.8 | 78.6 | |
| Fine-tuned | GPO | 88.7 | 98.9 | 99.8 | 76.1 | 94.5 | 97.1 | 68.1 | 90.2 | - | 52.7 | 80.2 | - |
| UNITER | 87.3 | 98.0 | 99.2 | 75.6 | 94.1 | 96.8 | 65.7 | 88.6 | 93.8 | 52.9 | 79.9 | 88.0 | |
| ERNIE-ViL | 88.1 | 98.0 | 99.2 | 76.7 | 93.6 | 96.4 | - | - | - | - | - | - | |
| VILLA | 87.9 | 97.5 | 98.8 | 76.3 | 94.2 | 96.8 | - | - | - | - | - | - | |
| Oscar | - | - | - | - | - | - | 73.5 | 92.2 | 96.0 | 57.5 | 82.8 | 89.8 | |
| ALIGN | 95.3 | 99.8 | 100.0 | 84.9 | 97.4 | 98.6 | 77.0 | 93.5 | 96.9 | 59.9 | 83.3 | 89.8 |
表 1：Flickr30K 与 MSCOCO 上的图文检索结果（零样本与微调）。ALIGN 与 ImageBERT（Qi et al. 2020）、UNITER（Chen et al. 2020c）、CLIP（Radford et al. 2021）、GPO（Chen et al. 2020a）、ERNIE-ViL（Yu et al. 2020）、VILLA（Gan et al. 2020）、Oscar（Li et al. 2020）比较。
| image text | text image | text text | image image | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | |
| VSE++ | 43.1 | 74.3 | 84.2 | 32.5 | 62.7 | 75.4 | 38.7 | 62.3 | 72.2 | 36.4 | 70.4 | 81.3 |
| VSRN | 52.4 | 81.9 | 90.0 | 40.1 | 71.1 | 81.5 | 41.0 | 64.8 | 74.5 | 44.2 | 76.7 | 86.2 |
| DEI2T | 53.9 | 82.7 | 91.2 | 39.8 | 70.2 | 80.9 | 26.0 | 47.1 | 57.5 | 38.3 | 74.1 | 85.0 |
| DET2T+I2T | 55.9 | 84.2 | 91.8 | 41.7 | 72.3 | 83.0 | 42.4 | 64.9 | 74.0 | 38.5 | 73.6 | 84.9 |
| ALIGN | 78.1 | 94.3 | 97.4 | 61.8 | 84.9 | 91.1 | 45.4 | 66.8 | 75.2 | 49.4 | 81.4 | 89.1 |
Table 2: Multimodal retrieval performance on Crisscrossed Captions (CxC) dataset. ALIGN is compared with VSE++ (Faghri et al. 2018), VSRN (Li et al. 2019), DEI2T (Parekh et al. 2021), and DET2T+I2T (Parekh et al. 2021).
| image text | text image | text text | image image | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | |
| VSE++ | 43.1 | 74.3 | 84.2 | 32.5 | 62.7 | 75.4 | 38.7 | 62.3 | 72.2 | 36.4 | 70.4 | 81.3 |
| VSRN | 52.4 | 81.9 | 90.0 | 40.1 | 71.1 | 81.5 | 41.0 | 64.8 | 74.5 | 44.2 | 76.7 | 86.2 |
| DEI2T | 53.9 | 82.7 | 91.2 | 39.8 | 70.2 | 80.9 | 26.0 | 47.1 | 57.5 | 38.3 | 74.1 | 85.0 |
| DET2T+I2T | 55.9 | 84.2 | 91.8 | 41.7 | 72.3 | 83.0 | 42.4 | 64.9 | 74.0 | 38.5 | 73.6 | 84.9 |
| ALIGN | 78.1 | 94.3 | 97.4 | 61.8 | 84.9 | 91.1 | 45.4 | 66.8 | 75.2 | 49.4 | 81.4 | 89.1 |
表 2：Crisscrossed Captions（CxC）上的多模态检索表现。ALIGN 与 VSE++（Faghri et al. 2018）、VSRN（Li et al. 2019）、DEI2T（Parekh et al. 2021）、DET2T+I2T（Parekh et al. 2021）比较。
| Model | STS | SIS | SITS | Mean Avg |
|---|---|---|---|---|
| avg std | avg std | avg std | ||
| VSE++ | 74.40.4 | 73.30.9 | 55.21.5 | 67.6 |
| VSRN | 73.00.4 | 70.11.0 | 60.41.3 | 67.8 |
| DEI2T | 50.90.6 | 81.30.7 | 61.61.4 | 64.6 |
| DET2T+I2T | 74.20.4 | 74.50.9 | 61.91.3 | 70.2 |
| ALIGN | 72.90.4 | 77.20.8 | 67.61.2 | 72.6 |
Table 3: Spearman’s R Bootstrap Correlation () on Crisscrossed Captions (CxC) dataset. ALIGN is compared with VSE++ (Faghri et al. 2018), VSRN (Li et al. 2019), DEI2T (Parekh et al. 2021), and DET2T+I2T (Parekh et al. 2021).
| Model | STS | SIS | SITS | Mean Avg |
|---|---|---|---|---|
| avg std | avg std | avg std | ||
| VSE++ | 74.40.4 | 73.30.9 | 55.21.5 | 67.6 |
| VSRN | 73.00.4 | 70.11.0 | 60.41.3 | 67.8 |
| DEI2T | 50.90.6 | 81.30.7 | 61.61.4 | 64.6 |
| DET2T+I2T | 74.20.4 | 74.50.9 | 61.91.3 | 70.2 |
| ALIGN | 72.90.4 | 77.20.8 | 67.61.2 | 72.6 |
表 3：Crisscrossed Captions（CxC）上的 Spearman R Bootstrap 相关（）。ALIGN 与 VSE++、VSRN、DEI2T、DET2T+I2T 比较。
5.1 Image-Text Matching & Retrieval
5.1 图文匹配与检索
We evaluate ALIGN on Flickr30K and MSCOCO cross-modal retrieval benchmarks, in both zero-shot and fully fine-tuned settings. We follow (Karpathy & Fei-Fei 2015) and most existing works to obtain the train/test splits. Specifically, for Flickr30K, we evaluate on the standard 1K test set, and finetune on the 30k training set. For MSCOCO, we evaluate on the 5K test set, and finetune on 82K training plus 30K additional validation images that are not in the 5K validation or 5K test sets.
我们在 Flickr30K 与 MSCOCO 跨模态检索基准上，于零样本与全量微调两种设定下评估 ALIGN。划分遵循 Karpathy & Fei-Fei 2015 与多数既有工作：Flickr30K 在标准 1K 测试集评估、在 30k 训练集微调；MSCOCO 在 5K 测试集评估，微调使用 82K 训练图外加不在 5K 验证/测试中的 30K 额外验证图。
During fine-tuning, the same loss function is used. But there can be false negatives when the batch size is comparable to the total number of training samples. So we reduce the global batch size from 16384 to 2048. We also reduce the initial learning rate to 1e-5 and train for 3K and 6K steps (with linear decay) respectively on Flickr30K and MSCOCO. All the other hyper-parameters are kept the same as pre-training.
微调时使用相同损失函数。但当 batch 大小与训练样本总数可比时可能出现假负例，故将全局 batch 从 16384 降至 2048。初始学习率降至 1e-5，在 Flickr30K 与 MSCOCO 上分别训练 3K 与 6K 步（线性衰减）。其余超参数与预训练保持一致。
Table 1 shows that, compared to previous works, ALIGN achieves SOTA results in all metrics of Flickr30K and MSCOCO benchmarks. In the zero-shot setting, ALIGN gets more than 7% improvement in image retrieval task compared to the previous SOTA, CLIP (Radford et al. 2021). With fine-tuning, ALIGN outperforms all existing methods by a large margin, including those that employ more complex cross-modal attention layers such as ImageBERT (Qi et al. 2020), UNITER (Chen et al. 2020c), ERNIE-ViL (Yu et al. 2020), VILLA (Gan et al. 2020) and Oscar (Li et al. 2020).
表 1 显示：相对先前工作，ALIGN 在 Flickr30K 与 MSCOCO 全部指标上达到 SOTA。零样本设定下，相对先前 SOTA CLIP（Radford et al. 2021），图像检索任务提升超过 7%。微调后，ALIGN 以较大优势超过所有现有方法，包括使用更复杂跨模态注意力层的 ImageBERT、UNITER、ERNIE-ViL、VILLA 与 Oscar。
Table 2 reports the performance of ALIGN on Crisscrossed Captions (CxC) retrieval tasks. Again, ALIGN achieves SOTA results in all metrics, especially by a large margin on image-to-text (+22.2% R@1) and text-to-image (20.1% R@1) tasks. Table 3 shows that ALIGN also outperforms the previous SOTA on SITS task with an improvement of 5.7%. One interesting observation is that, despite being much better on inter-modal tasks, ALIGN is not as impressive on intra-modal tasks. For instance, the improvements on text-to-text and image-to-image retrieval tasks (in particular the former) are less significant compared to those on image-to-text and text-to-image tasks. The performance on STS and SIS tasks is also slightly worse than VSE++ and DEI2T. We suspect it is because the training objective of ALIGN focuses on cross-modal (image-text) matching instead of intra-modal matching. Parekh et al. 2021 suggest multitask learning could produce more balanced representations. We leave it to the future work.
表 2 报告 ALIGN 在 Crisscrossed Captions（CxC）检索任务上的表现：再次在全部指标上达到 SOTA，尤其在图搜文（+22.2% R@1）与文搜图（+20.1% R@1）上优势很大。表 3 显示 ALIGN 在 SITS 任务上也超过先前 SOTA，提升 5.7%。有趣的是，尽管跨模态任务强很多，模态内任务上 ALIGN 不那么突出：例如文搜文与图搜图（尤其前者）的提升相对图搜文/文搜图更小；STS 与 SIS 也略弱于 VSE++ 与 DEI2T。我们怀疑是因为 ALIGN 训练目标聚焦跨模态（图文）匹配而非模态内匹配。Parekh et al. 2021 建议多任务学习可产生更均衡的表征，留待未来工作。
5.2 Zero-shot Visual Classification
5.2 零样本视觉分类
If we directly feed the texts of classnames into the text encoder, ALIGN is able to classify images into candidate classes via image-text retrieval. Table 4 compares ALIGN with CLIP on Imagenet and its variants. Similar to CLIP, ALIGN shows great robustness on classification tasks with different image distributions. In order to make a fair comparison, we use the same prompt ensembling method as CLIP. Each classname is expanded with a set of prompt templates defined by CLIP such as “A photo of a {classname}”. The class embedding is computed by averaging the embeddings of all templates followed by an L2-normalization. We find that such ensembling gives 2.9% improvement on ImageNet top-1 accuracy.
若直接把类名文本送入文本编码器，ALIGN 即可通过图文检索把图像分到候选类。表 4 在 ImageNet 及其变体上比较 ALIGN 与 CLIP。与 CLIP 类似，ALIGN 在不同图像分布的分类任务上表现出很强稳健性。为公平比较，我们采用与 CLIP 相同的 prompt 集成：每个类名用 CLIP 定义的一组模板展开，如 “A photo of a {classname}”；类嵌入由全部模板嵌入平均后再 L2 归一化得到。发现这种集成给 ImageNet top-1 带来 2.9% 提升。
| Model | ImageNet | ImageNet-R | ImageNet-A | ImageNet-V2 |
|---|---|---|---|---|
| CLIP | 76.2 | 88.9 | 77.2 | 70.1 |
| ALIGN | 76.4 | 92.2 | 75.8 | 70.1 |
Table 4: Top-1 Accuracy of zero-shot transfer of ALIGN to image classification on ImageNet and its variants.
| Model | ImageNet | ImageNet-R | ImageNet-A | ImageNet-V2 |
|---|---|---|---|---|
| CLIP | 76.2 | 88.9 | 77.2 | 70.1 |
| ALIGN | 76.4 | 92.2 | 75.8 | 70.1 |
表 4：ALIGN 零样本迁移到 ImageNet 及其变体图像分类的 top-1 准确率。
| Model (backbone) | Acc@1 w/ frozen features | Acc@1 | Acc@5 |
|---|---|---|---|
| WSL (ResNeXt-101 32x48d) | 83.6 | 85.4 | 97.6 |
| CLIP (ViT-L/14) | 85.4 | - | - |
| BiT (ResNet152 x 4) | - | 87.54 | 98.46 |
| NoisyStudent (EfficientNet-L2) | - | 88.4 | 98.7 |
| ViT (ViT-H/14) | - | 88.55 | - |
| Meta-Pseudo-Labels (EfficientNet-L2) | - | 90.2 | 98.8 |
| ALIGN (EfficientNet-L2) | 85.5 | 88.64 | 98.67 |
Table 5: ImageNet classification results. ALIGN is compared with WSL (Mahajan et al. 2018), CLIP (Radford et al. 2021), BiT (Kolesnikov et al. 2020), ViT (Dosovitskiy et al. 2021), NoisyStudent (Xie et al. 2020), and Meta-Pseudo-Labels (Pham et al. 2020).
| Model (backbone) | Acc@1 w/ frozen features | Acc@1 | Acc@5 |
|---|---|---|---|
| WSL (ResNeXt-101 32x48d) | 83.6 | 85.4 | 97.6 |
| CLIP (ViT-L/14) | 85.4 | - | - |
| BiT (ResNet152 x 4) | - | 87.54 | 98.46 |
| NoisyStudent (EfficientNet-L2) | - | 88.4 | 98.7 |
| ViT (ViT-H/14) | - | 88.55 | - |
| Meta-Pseudo-Labels (EfficientNet-L2) | - | 90.2 | 98.8 |
| ALIGN (EfficientNet-L2) | 85.5 | 88.64 | 98.67 |
表 5：ImageNet 分类结果。ALIGN 与 WSL（Mahajan et al. 2018）、CLIP（Radford et al. 2021）、BiT（Kolesnikov et al. 2020）、ViT（Dosovitskiy et al. 2021）、NoisyStudent（Xie et al. 2020）、Meta-Pseudo-Labels（Pham et al. 2020）比较。
5.3 Visual Classification w/ Image Encoder Only
5.3 仅用图像编码器的视觉分类
On the ImageNet benchmark, we first freeze the learned visual features and only train the classification head. Afterwards we fine-tune all layers. We use basic data augmentations including random cropping (same as in Szegedy et al. 2015) and horizontal flip. In evaluation we apply a single central crop with ratio of 0.875. Following Touvron et al. 2019, we use 0.8 scale ratio between training and evaluation to mitigate the resolution discrepancy introduced by random crop. Specifically, train/eval resolution is 289/360 with frozen visual features, and is 475/600 when fine-tuning all variables.
在 ImageNet 基准上，我们先冻结学到的视觉特征、只训练分类头，再微调全部层。数据增强包括随机裁剪（同 Szegedy et al. 2015）与水平翻转；评估时做比例 0.875 的单次中心裁剪。沿用 Touvron et al. 2019，训练与评估之间使用 0.8 的尺度比，以缓解随机裁剪引入的分辨率差异。具体而言：冻结特征时 train/eval 分辨率为 289/360；全量微调时为 475/600。
In both stages of training, we use a global batch size of 1024, SGD optimizer with momentum 0.9, and learning rate decayed every 30 epochs with ratio 0.2 (100 epochs in total). Weight decay is set to zero. With frozen visual features, we use the initial learning rate of 0.1. When fine-tuning all layers with use the initial learning rate of 0.01, and use 10x smaller learning rate on the backbone network compared to the classification head.
两阶段训练均使用全局 batch 1024、动量 0.9 的 SGD，学习率每 30 个 epoch 以 0.2 衰减（共 100 个 epoch）。权重衰减设为 0。冻结视觉特征时初始学习率 0.1；全量微调时初始学习率 0.01，且骨干网络学习率比分类头小 10 倍。
Table 5 compares ALIGN with previous methods on the ImageNet benchmark. With frozen features, ALIGN slightly outperforms CLIP and achieves SOTA result of 85.5% top-1 accuracy. After fine-tuning ALIGN achieves higher accuracy than BiT and ViT models, and is only worse than Meta Pseudo Labels which requires deeper interaction between ImageNet training and large-scale unlabeled data. Compared to NoisyStudent and Meta-Pseudeo-Labels which also use EfficientNet-L2, ALIGN saves 44% FLOPS by using smaller test resolution (600 instead of 800).
表 5 在 ImageNet 上比较 ALIGN 与先前方法。冻结特征时，ALIGN 略超 CLIP，达到 85.5% top-1 的 SOTA；全量微调后高于 BiT 与 ViT，仅弱于需要 ImageNet 训练与大规模未标注数据更深交互的 Meta Pseudo Labels。相对同样使用 EfficientNet-L2 的 NoisyStudent 与 Meta-Pseudo-Labels，ALIGN 因使用更小测试分辨率（600 而非 800）节省 44% FLOPS。
In VTAB eval, we follow a hyper-parameter sweep as shown in the Appendix I in (Zhai et al. 2019) with 50 trials for each task. Each task is trained on 800 images and the hyperparameters are selected using the validation set of 200 images. After the sweep, the selected hyperparameters are used to train on the combined training and validation splits of 1000 images for each task. Table 6 reports the mean accuracy (including the breakdown results on each subgroup) with standard deviation from three fine-tuning runs and shows that ALIGN outperforms BiT-L (Kolesnikov et al. 2020) with similar hyper-parameter selection method applied.
VTAB 评估遵循 Zhai et al. 2019 附录 I 的超参数扫描，每任务 50 次试验。每任务在 800 张图上训练，用 200 张验证集选超参；选定后在合并的 1000 张训练+验证上训练。表 6 报告三次微调的平均准确率（含各子组分解）及标准差，显示在类似超参选择下 ALIGN 超过 BiT-L（Kolesnikov et al. 2020）。
| Model | All tasks | Natural | Specialized | Structured |
|---|---|---|---|---|
| Bit-L | 78.72 | - | - | - |
| ALIGN | 79.990.15 | 83.38 | 87.56 | 73.25 |
Table 6: VTAB (19 tasks) comparison between ALIGN and BiT-L.
| Model | All tasks | Natural | Specialized | Structured |
|---|---|---|---|---|
| Bit-L | 78.72 | - | - | - |
| ALIGN | 79.990.15 | 83.38 | 87.56 | 73.25 |
表 6：ALIGN 与 BiT-L 在 VTAB（19 项任务）上的比较。
To evaluate on smaller fine-grained classification benchmarks, we adopt a simple fine-tuning strategy for all tasks. We use the same data augmentation and optimizer as in ImageNet fine-tuning. Similarly, we first train the classification head and then fine-tune all layers, except with batch norm statistics frozen. The train/eval resolution is fixed at 289/360. We use batch size 256 and weight decay 1e-5. The initial learning rate is set to 1e-2 and 1e-3 respectively, with cosine learning rate decay in 20k steps. Table 7 compares ALIGN with BiT-L (Kolesnikov et al. 2020) and SAM (Foret et al. 2021) which both apply same fine-tuning hyper-parameters for all tasks.[^2] For small tasks like these, details in fine-tuning matter. So we list the baseline results in (Foret et al. 2021) without using SAM optimization for a fairer comparison. Our result (average of three runs) is comparable to the SOTA results without tweaking on optimization algorithms.
在更小的细粒度分类基准上，我们对所有任务采用简单微调策略：数据增强与优化器同 ImageNet 微调；同样先训分类头再全量微调，但冻结 batch norm 统计量。train/eval 分辨率固定为 289/360；batch 256，权重衰减 1e-5；两阶段初始学习率分别为 1e-2 与 1e-3，余弦衰减 20k 步。表 7 比较 ALIGN 与 BiT-L（Kolesnikov et al. 2020）以及同样对所有任务使用相同微调超参的 SAM（Foret et al. 2021）。[^2] 对此类小任务，微调细节很关键，故列出 Foret et al. 2021 中未用 SAM 优化的基线以便更公平比较。我们的结果（三次运行平均）与不调优化算法时的 SOTA 相当。
| Model | Oxford | Oxford | Stanford | Food101 |
|---|---|---|---|---|
| Flowers | Pets | Cars | ||
| BiT-L | 99.63 | 96.62 | - | - |
| SAM-baseline | 99.60 | 96.92 | 95.07 | 96.03 |
| SAM-final | 99.65 | 97.10 | 95.96 | 96.18 |
| ALIGN | 99.65 | 96.19 | 96.13 | 95.88 |
Table 7: Transfer learning results on Fine-grained Classification Tasks. BiT-L (Kolesnikov et al. 2020) was trained with ResNet152 x 4 whereas SAM-baseline, SAM-final (Foret et al. 2021) and ALIGN were trained with EfficientNet-L2.
| Model | Oxford | Oxford | Stanford | Food101 |
|---|---|---|---|---|
| Flowers | Pets | Cars | ||
| BiT-L | 99.63 | 96.62 | - | - |
| SAM-baseline | 99.60 | 96.92 | 95.07 | 96.03 |
| SAM-final | 99.65 | 97.10 | 95.96 | 96.18 |
| ALIGN | 99.65 | 96.19 | 96.13 | 95.88 |
表 7：细粒度分类任务上的迁移学习结果。BiT-L 使用 ResNet152×4；SAM-baseline、SAM-final 与 ALIGN 使用 EfficientNet-L2。
6 Ablation Study
6 消融实验
In the ablation study, we compare model performance mostly on MSCOCO zero-shot retrieval and ImageNet K-Nearest-neighbor (KNN) tasks.[^3] We find these two metrics are representative and correlate well with other metrics reported in the section above. If not mentioned, hyper-parameters other than the ablated factor are kept the same as in the baseline model.
消融中，我们主要在 MSCOCO 零样本检索与 ImageNet K 近邻（KNN）任务上比较模型表现。[^3] 发现这两项指标有代表性，且与上一节其他指标相关性好。除非另有说明，被消融因素以外的超参数与基线模型保持一致。
6.1 Model Architectures
6.1 模型结构
We first study the performance of ALIGN models using different image and text backbones. We train EfficientNet from B1 to L2 for the image encoder and BERT-Mini to BERT-Large for the text encoder. We add an additional fully-connected layer with linear activation on top of B1, B3, B5 and L2 globally-pooled features to match the output dimension of B7 (640). A similar linear layer is added to all text encoders. We reduce the training steps to 1M in ablation to save some runtime.
我们先研究不同图像与文本骨干下 ALIGN 的表现：图像编码器从 EfficientNet-B1 到 L2，文本编码器从 BERT-Mini 到 BERT-Large。在 B1、B3、B5、L2 的全局池化特征上额外加线性全连接，以匹配 B7 的输出维度（640）；所有文本编码器也加类似线性层。消融中将训练步数减至 1M 以节省时间。
Figures 3 shows MSCOCO zero-shot retrieval and ImageNet KNN results with different combinations of image and text backbones. Model quality improves nicely with larger backbones except that the ImageNet KNN metric starts to saturate from BERT-Base to BERT-Large with EfficientNet-B7 and EfficientNet-L2. As expected, scaling up image encoder capacity is more important for vision tasks (e.g., even with BERT-Mini text tower, L2 performs better than B7 with BERT-Large). In image-text retrieval tasks the image and text encoder capacities are equally important. Based on the nice scaling property shown in Figure 3, we only fine-tune the model with EfficientNet-L2 + BERT-Large as reported in Section 5.
图 3 展示不同图像与文本骨干组合下的 MSCOCO 零样本检索与 ImageNet KNN 结果。除 EfficientNet-B7/L2 搭配时 ImageNet KNN 从 BERT-Base 到 BERT-Large 开始饱和外，模型质量随骨干变大而良好提升。如预期，放大图像编码器对视觉任务更重要（例如即便文本塔为 BERT-Mini，L2 也优于配 BERT-Large 的 B7）。图文检索任务上图像与文本编码器容量同等重要。基于图 3 所示良好缩放性质，第 5 节只微调 EfficientNet-L2 + BERT-Large 模型。
Figure 3: Zero-shot image-text retrieval and ImageNet KNN accuracy@1 with different image and text encoder sizes.
图 3：不同图像与文本编码器规模下的零样本图文检索与 ImageNet KNN accuracy@1。
We then study key architecture hyperparameters including embedding dimensions, number of random negatives in the batch, and the softmax temperature. Table 8 compares a number of model variants to a baseline model (first row) trained with the following settings: EfficientNet-B5 image encoder, BERT-Base text encoder, embedding dimension 640, all negatives in the batch, and a learnable softmax temperature.
接着研究关键结构超参：嵌入维度、batch 内随机负例数、softmax 温度。表 8 将若干模型变体与基线（首行）比较；基线设定为：EfficientNet-B5 图像编码器、BERT-Base 文本编码器、嵌入维度 640、使用 batch 内全部负例、可学习 softmax 温度。
Rows 2-4 of Table 8 show that model performance improves with higher embedding dimensions. Hence, we let the dimension scale with larger EfficientNet backbone (L2 uses 1376). Rows 5 and 6 show that using fewer in-batch negatives (50% and 25%) in the softmax loss will degrade the performance. Rows 7-9 study the effect of the temperature parameter in the softmax loss. Compared to the baseline model that learns the temperature parameter (converged to about 1/64), some hand-selected, fixed temperatures could be slightly better. However, we choose to use the learnable temperature as it performs competitively and makes learning easier. We also notice that the temperature usually quickly decrease to only around 1.2x of the converged values in the first 100k steps, and then slowly converges until the end of training.
表 8 第 2–4 行显示：更高嵌入维度带来更好表现，因此我们让维度随更大 EfficientNet 骨干放大（L2 使用 1376）。第 5–6 行显示 softmax 损失中使用更少 batch 内负例（50% 与 25%）会损害表现。第 7–9 行研究 softmax 温度：相对学习温度的基线（收敛约到 1/64），某些手工选定的固定温度可略好；但我们仍选择可学习温度，因其有竞争力且更易学习。还观察到温度通常在前 100k 步快速降到约收敛值的 1.2 倍，再缓慢收敛至训练结束。
| Model | MSCOCO | ImangeNet KNN | |
|---|---|---|---|
| I2T R@1 | T2I R@1 | R@1 | |
| B5 + BERT-base | 51.7 | 37.5 | 64.6 |
| w/ embedding dim=320 | 50.3 | 34.1 | 64.0 |
| w/ embedding dim=160 | 47.0 | 34.4 | 63.7 |
| w/ embedding dim=80 | 42.0 | 29.3 | 61.9 |
| w/ 50% in-batch negs | 50.2 | 37.0 | 63.8 |
| w/ 25% in-batch negs | 48.7 | 35.8 | 63.3 |
| w/ softmax temp=1/128 | 52.2 | 36.5 | 64.8 |
| w/ softmax temp=1/64 | 52.2 | 37.3 | 64.8 |
| w/ softmax temp=1/32 | 39.6 | 26.9 | 61.2 |
Table 8: Ablation study of key architecture parameters. Baseline model (first row) is trained with embedding dimension 640, using all negatives in the batch, and a learnable softmax temperature.
| Model | MSCOCO | ImangeNet KNN | |
|---|---|---|---|
| I2T R@1 | T2I R@1 | R@1 | |
| B5 + BERT-base | 51.7 | 37.5 | 64.6 |
| w/ embedding dim=320 | 50.3 | 34.1 | 64.0 |
| w/ embedding dim=160 | 47.0 | 34.4 | 63.7 |
| w/ embedding dim=80 | 42.0 | 29.3 | 61.9 |
| w/ 50% in-batch negs | 50.2 | 37.0 | 63.8 |
| w/ 25% in-batch negs | 48.7 | 35.8 | 63.3 |
| w/ softmax temp=1/128 | 52.2 | 36.5 | 64.8 |
| w/ softmax temp=1/64 | 52.2 | 37.3 | 64.8 |
| w/ softmax temp=1/32 | 39.6 | 26.9 | 61.2 |
表 8：关键结构参数消融。基线（首行）嵌入维度 640、使用 batch 内全部负例、可学习 softmax 温度。
6.2 Pre-training Datasets
6.2 预训练数据集
It’s also important to understand how the model performs when trained on different datasets with varying size. For this purpose, we train two models: EfficientNet-B7 + BERT-base and EfficientNet-B3 + BERT-mini on three different datasets: full ALIGN training data, 10% randomly sampled ALIGN training data, and Conceptual Captions (CC-3M, around 3M images). CC-3M is much smaller so we train the model with 1/10 of the default number of steps. All models are trained from scratch. As shown in Table 9, a large scale training set is essential to allow scaling up of our models and to achieve better performance. For instance, models trained on ALIGN data clearly outperform those trained on CC-3M data. On CC-3M, B7+BERT-base starts to overfit and performs even worse than B3+BERT-mini. Conversely, a larger model is required to fully utilize the larger dataset – the smaller B3+BERT-mini almost saturate at 10% of ALIGN data, while with the larger B7+BERT-base, there is a clear improvement with full ALIGN data.
理解模型在不同规模数据集上的表现也很重要。为此我们训练两个模型：EfficientNet-B7 + BERT-base 与 EfficientNet-B3 + BERT-mini，分别在三套数据上：完整 ALIGN 训练数据、随机采样的 10% ALIGN 数据、以及 Conceptual Captions（CC-3M，约 300 万图）。CC-3M 小得多，故训练步数为默认的 1/10。所有模型从零训练。如表 9 所示，大规模训练集对放大模型并取得更好表现必不可少：例如在 ALIGN 数据上训练的模型明显超过在 CC-3M 上训练者。在 CC-3M 上，B7+BERT-base 开始过拟合，甚至弱于 B3+BERT-mini。反过来，更大模型才能充分利用更大数据——较小的 B3+BERT-mini 在 10% ALIGN 数据上几乎饱和，而更大的 B7+BERT-base 在完整 ALIGN 数据上仍有明显提升。
| Model + Data | MSCOCO | ImangeNet KNN | |
|---|---|---|---|
| I2T R@1 | T2I R@1 | R@1 | |
| B7 + BERT-base | |||
| + ALIGN full data | 55.4 | 41.7 | 69.3 |
| + ALIGN 10% data | 52.0 | 39.2 | 68.8 |
| + CC-3M data | 18.9 | 15.5 | 48.7 |
| B3 + BERT-mini | |||
| + ALIGN full data | 37.4 | 24.5 | 56.5 |
| + ALIGN 10% data | 36.7 | 24.4 | 55.8 |
| + CC-3M data | 22.1 | 17.3 | 48.9 |
Table 9: Ablation study of different training datasets.
| Model + Data | MSCOCO | ImangeNet KNN | |
|---|---|---|---|
| I2T R@1 | T2I R@1 | R@1 | |
| B7 + BERT-base | |||
| + ALIGN full data | 55.4 | 41.7 | 69.3 |
| + ALIGN 10% data | 52.0 | 39.2 | 68.8 |
| + CC-3M data | 18.9 | 15.5 | 48.7 |
| B3 + BERT-mini | |||
| + ALIGN full data | 37.4 | 24.5 | 56.5 |
| + ALIGN 10% data | 36.7 | 24.4 | 55.8 |
| + CC-3M data | 22.1 | 17.3 | 48.9 |
表 9：不同训练数据集的消融。
To understand better how data size scaling wins over the increased noise, we further randomly sample 3M, 6M, and 12M ALIGN training data and compare them with the cleaned CC-3M data on B7+BERT-base model. Table 10 shows that while the ALIGN data performs much worse than CC data with the same size (3M), the model quality trained on 6M and 12M ALIGN data rapidly catches up. Despite being noisy, ALIGN data outperforms Conceptual Captions with only 4x size.
为更好理解数据规模如何胜过噪声增加，我们进一步随机采样 3M、6M、12M ALIGN 训练数据，在 B7+BERT-base 上与清洗后的 CC-3M 比较。表 10 显示：同等规模（3M）下 ALIGN 数据远弱于 CC；但在 6M 与 12M ALIGN 上模型质量迅速追上。尽管嘈杂，ALIGN 数据仅需约 4 倍规模即可超过 Conceptual Captions。
| Model + Data | MSCOCO | ImangeNet KNN | |
|---|---|---|---|
| I2T R@1 | T2I R@1 | R@1 | |
| B7 + BERT-base | |||
| + ALIGN 12M data | 23.8 | 17.5 | 51.4 |
| + ALIGN 6M data | 15.8 | 11.9 | 47.9 |
| + ALIGN 3M data | 8.1 | 6.3 | 41.3 |
| + CC-3M data | 18.9 | 15.5 | 48.7 |
Table 10: Tradeoff between training data size and quality.
| Model + Data | MSCOCO | ImangeNet KNN | |
|---|---|---|---|
| I2T R@1 | T2I R@1 | R@1 | |
| B7 + BERT-base | |||
| + ALIGN 12M data | 23.8 | 17.5 | 51.4 |
| + ALIGN 6M data | 15.8 | 11.9 | 47.9 |
| + ALIGN 3M data | 8.1 | 6.3 | 41.3 |
| + CC-3M data | 18.9 | 15.5 | 48.7 |
表 10：训练数据规模与质量之间的权衡。
7 Analysis of Learned Embeddings
7 所学嵌入的分析
We build a simple image retrieval system to study the behaviors of embeddings trained by ALIGN. For demonstration purposes, we use an index consisting of 160M CC-BY licensed images that are separate from our training set. Figure 4 shows the top 1 text-to-image retrieval results for a handful of text queries not existing in the training data. ALIGN can retrieve precise images given detailed descriptions of a scene, or fine-grained or instance-level concepts like landmarks and artworks. These examples demonstrate that our ALIGN model can align images and texts with similar semantics, and that ALIGN can generalize to novel complex concepts.
我们构建简单的图像检索系统，研究 ALIGN 训练出的嵌入行为。演示用索引含 1.6 亿张与训练集分离的 CC-BY 许可图像。图 4 给出若干训练数据中不存在的文本查询的 top-1 文搜图结果。ALIGN 能根据场景的细致描述，或地标、艺术品等细粒度/实例级概念，检索到精确图像。这些例子表明 ALIGN 能对齐语义相近的图像与文本，并能泛化到新颖的复杂概念。
Figure 4: Image retrieval with fine-grained text queries using ALIGN’s embeddings.
图 4：用 ALIGN 嵌入做细粒度文本查询的图像检索。
Figure 5: Image retrieval with imagetext queries. We add (or subtract) text query embedding to (or from) the image query embedding, and then use the resulting embedding to retrieve relevant images using cosine similarity.
图 5：用「图像±文本」查询做图像检索。我们将文本查询嵌入加到（或从）图像查询嵌入中减去，再用所得嵌入按余弦相似度检索相关图像。
Previously word2vec (Mikolov et al. 2013a; Mikolov et al. 2013b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs. We show that linear relationships between image and text embeddings also emerge in ALIGN. We perform image retrieval using a combined image+text query. Specifically, given a query image and a text string, we add their ALIGN embeddings together and use it to retrieve relevant images.[^4] Figure 5 shows results for a variety of image+text queries. These examples not only demonstrate great compositionality of ALIGN embeddings across vision and language domains, but also show the feasibility of a new paradigm of “search with multi-modal query” that would otherwise be hard using only text query or image query. For instance, one could now look for the “Australia” or “Madagascar” equivalence of pandas, or turn a pair of black shoes into identically-looking shoes with the color of “beige”. Finally, as shown in the last three rows of Figure 5, removing objects/attributes from a scene is possible by performing subtraction in the embedding space.
先前 word2vec（Mikolov et al. 2013a；Mikolov et al. 2013b）表明：训练词向量去预测句段中相邻词时，词向量之间会出现线性关系。我们表明 ALIGN 中图像与文本嵌入之间同样出现线性关系。我们用组合的图像+文本查询做图像检索：给定查询图像与文本串，将其 ALIGN 嵌入相加再检索。[^4] 图 5 展示多种图像+文本查询结果。这些例子不仅显示 ALIGN 嵌入跨视觉与语言域的良好可组合性，也说明一种用纯文本或纯图像查询难以实现的「多模态查询搜索」新范式可行。例如，现在可以寻找熊猫的「澳大利亚」或「马达加斯加」等价物，或把一双黑色鞋子变成外观相同但颜色为「米色」的鞋子。最后，如图 5 末三行所示，在嵌入空间做减法即可从场景中移除物体/属性。
8 Multilingual ALIGN Model
8 多语言 ALIGN 模型
One advantage of ALIGN is that the model is trained on noisy web image text data with very simple filters, and none of the filters are language specific. Given that, we further lift the language constraint of the conceptual caption data processing pipeline to extend the dataset to multilingual (covering 100+ languages) and match its size to the English dataset (1.8B image-text pairs). A multilingual model ALIGNmling is trained using this data. We created a new mutlilingual wordpiece vocabulary with size 250k to cover all languages. Model training follows the exact English configuration.
ALIGN 的一个优点是：模型在仅经极简过滤的嘈杂网页图文上训练，且过滤均非语言相关。据此，我们进一步取消 Conceptual Captions 数据处理流水线中的语言约束，将数据扩展到多语言（覆盖 100+ 种语言），规模与英文数据对齐（18 亿图文对）。用该数据训练多语言模型 ALIGNmling：新建覆盖全部语言、规模 25 万的多语言 wordpiece 词表；训练配置与英文完全一致。
We test the multilingual model on Multi30k, a multilingual image text retrieval dataset extends Flickr30K (Plummer et al. 2015) to German (de) (Elliott et al. 2016), French (fr) (Elliott et al. 2017) and Czech (cs) (Barrault et al. 2018). The dataset consists of 31,783 images with 5 captions per image in English and German and 1 caption per image in French and Czech. The train/dev/test splits are defined in Young et al. 2014. We evaluate the zero-shot model performance of ALIGN and compare it with M3P (Huang et al. 2020a) and UC2 (Zhou et al. 2021). The evaluation metric is mean Recall (mR), which computes the average score of Recall@1, Recall@5 and Recall@10 on image-to-text retrieval and text-to-image retrieval tasks.
我们在 Multi30k 上测试多语言模型——该多语言图文检索集把 Flickr30K（Plummer et al. 2015）扩展到德语（de）（Elliott et al. 2016）、法语（fr）（Elliott et al. 2017）与捷克语（cs）（Barrault et al. 2018）。数据含 31783 张图，英/德每图 5 条标题，法/捷每图 1 条；划分见 Young et al. 2014。我们评估 ALIGN 的零样本表现，并与 M3P（Huang et al. 2020a）、UC2（Zhou et al. 2021）比较。指标为平均召回（mR）：图搜文与文搜图上 Recall@1/5/10 的平均。
Table 11 shows that the zero-shot performance of ALIGNmling outperforms M3P on all languages by a large margin, with the largest +57.8 absolution mR improvement on fr. The zero-shot performance of ALIGNmling is even comparable to the fine-tuned (w/ training splits) M3P and UC2 except on cs. On en, ALIGNmling performs slightly worse on its counterpart ALIGNEN (trained on EN-only data.)
表 11 显示：ALIGNmling 的零样本表现在全部语言上大幅超过 M3P，法语上绝对 mR 提升最大达 +57.8。ALIGNmling 零样本甚至可与经训练划分微调的 M3P、UC2 相比（捷克语除外）。英语上，ALIGNmling 略弱于仅用英文数据训练的 ALIGNEN。
| Model | en | de | fr | cs |
|---|---|---|---|---|
| zero-shot | ||||
| M3P | 57.9 | 36.8 | 27.1 | 20.4 |
| ALIGNEN | 92.2 | - | - | - |
| ALIGNmling | 90.2 | 84.1 | 84.9 | 63.2 |
| w/ fine-tuning | ||||
| M3P | 87.7 | 82.7 | 73.9 | 72.2 |
| UC2 | 88.2 | 84.5 | 83.9 | 81.2 |
Table 11: Multimodal retrieval performance on Multi30K dataset. The metric is the mean Recall (mR).
| Model | en | de | fr | cs |
|---|---|---|---|---|
| zero-shot | ||||
| M3P | 57.9 | 36.8 | 27.1 | 20.4 |
| ALIGNEN | 92.2 | - | - | - |
| ALIGNmling | 90.2 | 84.1 | 84.9 | 63.2 |
| w/ fine-tuning | ||||
| M3P | 87.7 | 82.7 | 73.9 | 72.2 |
| UC2 | 88.2 | 84.5 | 83.9 | 81.2 |
表 11：Multi30K 上的多模态检索表现，指标为平均召回（mR）。
9 Conclusion
9 结论
We present a simple method of leveraging large-scale noisy image-text data to scale up visual and vision-language representation learning. Our method avoids heavy work on data curation and annotation, and only requires minimal frequency-based cleaning. On this dataset, we train a simple dual-encoder model using a contrastive loss. The resulting model, named ALIGN, is capable of cross-modal retrieval and significantly outperforms SOTA VSE and cross-attention vision-language models. In visual-only downstream tasks, ALIGN is also comparable to or outperforms SOTA models trained with large-scale labeled data.
我们提出一种利用大规模嘈杂图文数据放大视觉与视觉–语言表征学习的简单方法：避免重度数据策展与标注，只需基于频率的最小清洗。在该数据上，我们用对比损失训练简单双塔模型。所得模型 ALIGN 能做跨模态检索，显著超过 SOTA 的 VSE 与交叉注意力视觉–语言模型；在纯视觉下游任务上，也与用大规模标注数据训练的 SOTA 模型相当或更优。
10 Social Impacts and Future Work
10 社会影响与未来工作
While this work shows promising results from a methodology perspective with a simple data collection method, additional analysis of the data and the resulting model is necessary before the use of the model in practice. For instance, considerations should be made towards the potential for the use of harmful text data in alt-texts to reinforce such harms. On the fairness front, data balancing efforts may be required to prevent reinforcing stereotypes from the web data. Additional testing and training around sensitive religious or cultural items should be taken to understand and mitigate the impact from possibly mislabeled data.
尽管本工作从方法角度、结合简单数据采集展示了有前景的结果，在实践使用模型前仍有必要进一步分析数据与所得模型。例如，需考虑有害文本进入 alt-text 从而强化伤害的可能。公平性方面，可能需要数据平衡，以防强化来自网页数据的刻板印象。对敏感宗教或文化物品，应做额外测试与训练，以理解并缓解可能的错误标注影响。
Further analysis should also be taken to ensure that the demographic distribution of humans and related cultural items like clothing, food, and art do not cause model performance to be skewed. Analysis and balancing would be required if such models will be used in production.
还应进一步分析，确保人类及服装、食物、艺术等相关文化物品的人口统计分布不会导致模型表现偏斜。若此类模型将用于生产，需要分析与平衡。
Finally, unintended misuse of such models for surveillance or other nefarious purposes should be prohibited.
最后，应禁止将此类模型滥用于监控或其他恶意目的。
Acknowledgements
致谢
This work was done with invaluable help from colleagues from Google. We would like to thank Jan Dlabal and Zhe Li for continuous support in training infrastructure, Simon Kornblith for building the zero-shot & robustness model evaluation on ImageNet variants, Xiaohua Zhai for help on conducting VTAB evaluation, Mingxing Tan and Max Moroz for suggestions on EfficientNet training, Aleksei Timofeev for the early idea of multimodal query retrieval, Aaron Michelony and Kaushal Patel for their early work on data generation, and Sergey Ioffe, Jason Baldridge and Krishna Srinivasan for the insightful feedback and discussion.
本工作得到 Google 多位同事的宝贵帮助。我们感谢 Jan Dlabal 与 Zhe Li 对训练基础设施的持续支持，Simon Kornblith 构建 ImageNet 变体上的零样本与稳健性评估，Xiaohua Zhai 协助开展 VTAB 评估，Mingxing Tan 与 Max Moroz 对 EfficientNet 训练的建议，Aleksei Timofeev 提出多模态查询检索的早期想法，Aaron Michelony 与 Kaushal Patel 在数据生成上的早期工作，以及 Sergey Ioffe、Jason Baldridge 与 Krishna Srinivasan 富有洞见的反馈与讨论。
References
参考文献
Barrault et al. (2018) Barrault, L., Bougares, F., Specia, L., Lala, C., Elliott, D., and Frank, S. Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers , pp. 304–323, 2018.
Bossard et al. (2014) Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 – mining discriminative components with random forests. In European Conference on Computer Vision , 2014.
Caron et al. (2020) Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems , 2020.
Chen et al. (2020a) Chen, J., Hu, H., Wu, H., Jiang, Y., and Wang, C. Learning the best pooling strategy for visual semantic embedding. In arXiv preprint arXiv:2011.04305 , 2020a.
Chen et al. (2020b) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning , 2020b.
Chen et al. (2015) Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. In arXiv preprint arXiv:1504.00325 , 2015.
Chen et al. (2020c) Chen, Y.-C., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. Uniter: Universal image-text representation learning. In Proceedings of European Conference on Computer Vision , 2020c.
Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of Conference on Computer Vision and Pattern Recognition , 2009.
Desai & Johnson (2020) Desai, K. and Johnson, J. Virtex: Learning visual representations from textual annotations. In arXiv preprint arXiv:2006.06666 , 2020.
Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics , 2019.
Dosovitskiy et al. (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of International Conference on Learning Representations , 2021.
Elliott et al. (2016) Elliott, D., Frank, S., Sima’an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language , 2016.
Elliott et al. (2017) Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers , September 2017.
Faghri et al. (2018) Faghri, F., Fleet, D. J., Kiros, J. R., and Fidler, S. Vse++: Improving visual-semantic embeddings with hard negatives. In Proceedings of the British Machine Vision Conference , 2018.
Foret et al. (2021) Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.
Frome et al. (2013) Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T. Devise: A deep visual-semantic embedding model. In Proceedings of Neural Information Processing Systems , 2013.
Gan et al. (2020) Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., and Liu, J. Large-scale adversarial training for vision-and-language representation learning. In Proceedings of Neural Information Processing Systems , 2020.
Grill et al. (2020) Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 , 2020.
He et al. (2020) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , June 2020.
Hendrycks et al. (2020) Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006.16241 , 2020.
Hendrycks et al. (2021) Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. CVPR , 2021.
Hill et al. (2015) Hill, F., Reichart, R., and Korhonen, A. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics , 2015.
Huang et al. (2020a) Huang, H., Su, L., Qi, D., Duan, N., Cui, E., Bharti, T., Zhang, L., Wang, L., Gao, J., Liu, B., Fu, J., Zhang, D., Liu, X., and Zhou, M. M3p: Learning universal representations via multitask multilingual multimodal pre-training. arXiv , abs/2006.02635, 2020a.
Huang et al. (2020b) Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849 , 2020b.
Joulin et al. (2015) Joulin, A., van der Maaten, L., Jabri, A., and Vasilache, N. Learning visual features from large weakly supervised data. In European Conference on Computer Vision , 2015.
Juan et al. (2020) Juan, D.-C., Lu, C.-T., Li, Z., Peng, F., Timofeev, A., Chen, Y.-T., Gao, Y., Duerig, T., Tomkins, A., and Ravi, S. Graph-rise: Graph-regularized image semantic embedding. In Proceedings of ACM International Conference on Web Search and Data Mining , 2020.
Karpathy & Fei-Fei (2015) Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. In Proceedings of Conference on Computer Vision and Pattern Recognition , 2015.
Karpathy et al. (2014) Karpathy, A., Joulin, A., and Li, F. Deep fragment embeddings for bidirectional image sentence mapping. In Advances in Neural Information Processing Systems , 2014.
Kiros et al. (2018) Kiros, J., Chan, W., and Hinton, G. Illustrative language understanding: Large-scale visual grounding with image search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2018.
(30) Kiros, R., Salakhutdinov, R., and Zemel, R. S. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 .
Kolesnikov et al. (2020) Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In Proceedings of European Conference on Computer Vision , 2020.
Krause et al. (2013) Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In Proceedings of ICCV Workshop on 3D Representation and Recognition , 2013.
Krishna et al. (2016) Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., Bernstein, M., and Fei-Fei, L. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision , 2016.
Kuznetsova et al. (2020) Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., Duerig, T., and Ferrari, V. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision , 2020.
Li et al. (2017) Li, A., Jabri, A., Joulin, A., and van der Maaten, L. Learning visual n-grams from web data. In Proceedings of IEEE International Conference on Computer Vision , 2017.
Li et al. (2021) Li, J., Zhou, P., Xiong, C., and Hoi, S. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations , 2021.
Li et al. (2019) Li, K., Zhang, Y., Li, K., Li, Y., and Fu, Y. Visual semantic reasoning for image-text matching. In Proceedings of International Conference on Computer Vision , 2019.
Li et al. (2020) Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., and Gao, J. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Proceedings of European Conference on Computer Vision , 2020.
Liu et al. (2019a) Liu, F., Liu, Y., Ren, X., He, X., and Sun, X. Aligning visual regions and textual concepts for semantic-grounded image representations. In Advances in Neural Information Processing Systems , 2019a.
Liu et al. (2019b) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019b.
Lu et al. (2019) Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Proceedings of Neural Information Processing Systems , 2019.
Mahajan et al. (2018) Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and van der Maaten, L. Exploring the limits of weakly supervised pretraining. In Proceedings of European Conference on Computer Vision , 2018.
Messina et al. (2020) Messina, N., Amato, G., Esuli, A., Falchi, F., Gennaro, C., and Marchand-Maillet, S. Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders. ACM Transactions on Multimedia Computing, Communications, and Applications , 2020.
Mikolov et al. (2013a) Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013a.
Mikolov et al. (2013b) Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems , 2013b.
Misra & Maaten (2020) Misra, I. and Maaten, L. v. d. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , June 2020.
Musgrave et al. (2020) Musgrave, K., Belongie, S., and Lim, S.-N. A metric learning reality check. In Proceedings of European Conference on Computer Vision , 2020.
Nam et al. (2017) Nam, H., Ha, J.-W., and Kim, J. Dual attention networks for multimodal reasoning and matching. In Proceedings of Conference on Computer Vision and Pattern Recognition , 2017.
Nilsback & Zisserman (2008) Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing , Dec 2008.
Parekh et al. (2021) Parekh, Z., Baldridge, J., Cer, D., Waters, A., and Yang, Y. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for ms-coco. In Proceedings of Conference of the European Chapter of the Association for Computational Linguistics , 2021.
Parkhi et al. (2012) Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition , 2012.
Pennington et al. (2014) Pennington, J., Socher, R., and Manning, C. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2014.
Pham et al. (2020) Pham, H., Dai, Z., Xie, Q., Luong, M.-T., and Le, Q. V. Meta pseudo labels. In arXiv preprint arXiv:2003.10580 , 2020.
Plummer et al. (2015) Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the International Conference on Computer Vision , 2015.
Qi et al. (2020) Qi, D., Su, L., Song, J., Cui, E., Bharti, T., and Sacheti, A. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprintarXiv:2001.07966 , 2020.
Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019.
Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarawl, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. 2021.
Raffel et al. (2020) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020.
Recht et al. (2019) Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning , pp. 5389–5400, 2019.
Sariyildiz et al. (2020) Sariyildiz, M. B., Perez, J., and Larlus, D. Learning visual representations with caption annotations. arXiv preprint arXiv:2008.01392 , 2020.
Sharma et al. (2018) Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of Annual Meeting of the Association for Computational Linguistics , 2018.
Socher et al. (2014) Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., and Ng, A. Y. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics , 2014.
Sun et al. (2017) Sun, C., Shrivastava, A., Sigh, S., and Gupta, A. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the International Conference on Computer Vision , 2017.
Szegedy et al. (2015) Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of Conference on Computer Vision and Pattern Recognition , 2015.
Tian et al. (2020) Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. In European Conference on Computer Vision , 2020.
Touvron et al. (2019) Touvron, H., Vedaldi, A., Douze, M., and Jégou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems , 2019.
Wang et al. (2014) Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., and Wu, Y. Learning fine-grained image similarity with deep ranking. In Proceedings of Conference on Computer Vision and Pattern Recognition , 2014.
Xie et al. (2020) Xie, Q., Luong, M.-T., Hovy, E., and Le, Q. V. Self-training with noisy student improves imagenet classification. In Proceedings of Conference on Computer Vision and Pattern Recognition , 2020.
Yalniz et al. (2019) Yalniz, I. Z., Jégou, H., Chen, K., Paluri, M., and Mahajan, D. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546 , 2019.
Yang et al. (2019) Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems , 2019.
You et al. (2020) You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. In Proceedings of International Conference on Learning Representations , 2020.
Young et al. (2014) Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2014.
Yu et al. (2020) Yu, F., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H., and Wang, H. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934 , 2020.
Zhai & Wu (2019) Zhai, A. and Wu, H.-Y. Classification is a strong baseline for deep metric learning. In Proceedings of the British Machine Vision Conference , 2019.
Zhai et al. (2019) Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867 , 2019.
Zhang et al. (2020) Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747 , 2020.
Zhou et al. (2021) Zhou, M., Zhou, L., Wang, S., Cheng, Y., Li, L., Yu, Z., and Liu, J. UC2: Universal cross-lingual cross-modal vision-and-language pre-training. arXiv preprint arXiv:2104.00332 , 2021.
Zoph et al. (2020) Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. V. Rethinking pre-training and self-training. In Advances in Neural Information Processing Systems , 2020.
Appendix A Remove Near-Duplicate Test Images from Training Data
附录 A 从训练数据中移除近重复测试图像
To detect near-duplicate images, we first train a separate high-quality image embedding model following (Wang et al. 2014) with a large-scale labeled dataset as in (Juan et al. 2020), and then generate 4K clusters via k-means based on all training images of the embedding model. For each query image (from the ALIGN dataset) and index image (from test sets of downstream tasks), we find their top-10 nearest clusters based on the embedding distance. Each image is then assigned to buckets (all possible combinations of 3 clusters out of 10). For any query-index image pair that falls into the same bucket, we mark it as near-duplicated if their embedding cosine similarity is larger than 0.975. This threshold is trained on a large-scale dataset built with human rated data and synthesized data with random augmentation.
为检测近重复图像，我们先按 Wang et al. 2014、在 Juan et al. 2020 所用大规模标注数据上训练单独的高质量图像嵌入模型，再基于该嵌入模型全部训练图像经 k-means 生成 4K 聚类。对每个查询图像（来自 ALIGN 数据）与索引图像（来自下游任务测试集），按嵌入距离找 top-10 最近聚类；每张图再被赋给 个桶（10 个聚类中任选 3 个的全部组合）。落入同一桶的任意查询–索引对，若嵌入余弦相似度大于 0.975 则标为近重复。该阈值在由人工评分与随机增强合成数据构成的大规模集上训练得到。
Appendix B Evaluation on SimLex-999
附录 B 在 SimLex-999 上的评估
The image-text co-training could also help the natural language understanding as shown in Kiros et al. 2018. For instance, with language only, it is very hard to learn antonyms. In order to test this capability of ALIGN model, we also evaluate the word representation from ALIGN model[^5] on SimLex-999 (Hill et al. 2015), which is a task to compare word similarity for 999 word pairs. We follow Kiros et al. 2018 to report the results on 9 sub-tasks each contains a subset of word pairs: all, adjectives, nouns, verbs, concreteness quartiles (1-4), and hard.
如图文共训练也可帮助自然语言理解（Kiros et al. 2018）。例如仅用语言很难学到反义词。为测试 ALIGN 的这一能力，我们还在 SimLex-999（Hill et al. 2015）上评估 ALIGN 模型的词表征[^5]——该任务比较 999 个词对的相似度。我们按 Kiros et al. 2018 报告 9 个子任务结果，各含词对子集：all、adjectives、nouns、verbs、concreteness quartiles (1-4) 与 hard。
. GloVe Picturebook ALIGN all 40.8 37.3 39.8 adjs 62.2 11.7 49.8 nouns 42.8 48.2 45.9 verbs 19.6 17.3 16.6 conc-q1 43.3 14.4 23.9 conc-q2 41.6 27.5 41.7 conc-q3 42.3 46.2 47.6 conc-q4 40.2 60.7 57.8 hard 27.2 28.8 31.7
Table 12: SimLex-999 results (Spearman’s ).
. GloVe Picturebook ALIGN all 40.8 37.3 39.8 adjs 62.2 11.7 49.8 nouns 42.8 48.2 45.9 verbs 19.6 17.3 16.6 conc-q1 43.3 14.4 23.9 conc-q2 41.6 27.5 41.7 conc-q3 42.3 46.2 47.6 conc-q4 40.2 60.7 57.8 hard 27.2 28.8 31.7
表 12：SimLex-999 结果（Spearman ）。
The results are listed in the Table 12 compared to Picturebook (Kiros et al. 2018) and GloVe (Pennington et al. 2014) embeddings. Overall the learned ALIGN perform better than Picturebook but slightly worse than GloVe embeddings. What is interesting is that the ALIGN word embeddings has a similar trend of Picturebook embeddings, with better performance on nouns and most concrete categories but worse on adjs and less concrete categories compared to GloVe embeddings. ALIGN word embedding achieves the highest performance on the hard category, which similarity is difficult to distinguish from relatedness. This observation confirmed the hypothesis from Kiros et al. 2018 that image-based word embeddings are less likely to confuse similarity with relatedness than text learned distributional-based methods.
表 12 列出相对 Picturebook（Kiros et al. 2018）与 GloVe（Pennington et al. 2014）嵌入的结果。总体而言，学到的 ALIGN 优于 Picturebook，略弱于 GloVe。有趣的是，ALIGN 词嵌入与 Picturebook 趋势相似：相对 GloVe，在 nouns 与最具体类别上更好，在 adjs 与较不具体类别上更差。ALIGN 在 hard 类别上最高——该类别难以把相似度与相关性区分开。这一观察印证了 Kiros et al. 2018 的假设：基于图像的词嵌入比基于文本的分布方法更不易把相似度与相关性混淆。