LAION-5B: An open large-scale dataset for training next generation image-text models

LAION-5B: An open large-scale dataset for training next generation image-text models

LAION-5B：面向下一代图文模型训练的开放大规模数据集

Christoph Schuhmann §§°° Romain Beaumont §§°° Richard Vencu §§°° Cade Gordon §§°° Ross Wightman §§ Mehdi Cherti §§ Theo Coombes Aarush Katta Clayton Mullis Mitchell Wortsman Patrick Schramowski Srivatsa Kundurthy Katherine Crowson Ludwig Schmidt °° Robert Kaczmarczyk °° Jenia Jitsev °° LAION UC Berkeley Gentec Data TU Darmstadt Hessian.AI University of Washington Seattle Technical University of Munich Stability AI EleutherAI Juelich Supercomputing Center (JSC) Research Center Juelich (FZJ) Email: contact@laion.ai§§ Equal first contributions, °° Equal senior contributions

Abstract

摘要

Groundbreaking language-vision architectures like CLIP and DALL-E proved the utility of training on large amounts of noisy image-text data, without relying on expensive accurate labels used in standard vision unimodal supervised learning. The resulting models showed capabilities of strong text-guided image generation and transfer to downstream tasks, while performing remarkably at zero-shot classification with noteworthy out-of-distribution robustness. Since then, large-scale language-vision models like ALIGN, BASIC, GLIDE, Flamingo and Imagen made further improvements. Studying the training and capabilities of such models requires datasets containing billions of image-text pairs. Until now, no datasets of this size have been made openly available for the broader research community. To address this problem and democratize research on large-scale multi-modal models, we present LAION-5B - a dataset consisting of 5.85 billion CLIP-filtered image-text pairs, of which 2.32B contain English language. We show successful replication and fine-tuning of foundational models like CLIP, GLIDE and Stable Diffusion using the dataset, and discuss further experiments enabled with an openly available dataset of this scale. Additionally we provide several nearest neighbor indices, an improved web-interface for dataset exploration and subset generation, and detection scores for watermark, NSFW, and toxic content detection. [^1]

CLIP 和 DALL-E 这类开创性的语言—视觉架构表明，在大量带噪声的图文数据上训练是有用的，不必依赖传统单模态视觉监督学习里那些昂贵而精确的标注。由此得到的模型既能按文本生成图像、迁移到下游任务，也在零样本分类上表现突出，并对分布外数据相当稳健。此后，ALIGN、BASIC、GLIDE、Flamingo 和 Imagen 等大规模语言—视觉模型又做了进一步改进。要研究这类模型的训练和能力，需要包含数十亿图文对的数据集。在此之前，还没有如此规模的数据对更广泛的研究社区开放。为解决这一问题、让大规模多模态模型研究更容易参与，我们发布 LAION-5B：它包含 58.5 亿条经 CLIP 过滤的图文对，其中 23.2 亿条带有英语。我们展示了用该数据集成功复现并微调 CLIP、GLIDE 和 Stable Diffusion 等基础模型，并讨论了在如此规模的开放数据上还能做哪些实验。此外，我们还提供若干近邻索引、改进后的网页界面（用于浏览数据集和生成子集），以及水印、NSFW 和有害内容的检测分数。[^1]

1 Introduction

1 引言

Learning from multimodal data such as text, images, and audio is a longstanding research challenge in machine learning [51, 56, 83, 31, 86]. Recently, contrastive loss functions combined with large neural networks have led to breakthroughs in the generalization capabilities of vision and language models [58, 59, 66]. For instance, OpenAI’s CLIP models [58] achieved large gains in zero-shot classification on ImageNet [65], improving from the prior top-1 accuracy of 11.5% [41] to 76.2%. In addition, CLIP achieved unprecedented performance gains on multiple challenging distribution shifts [78, 61, 23, 82, 3, 70]. Inspired by CLIP’s performance, numerous groups have further improved image-text models by increasing the amount of computation and the training set size [28, 54, 94, 89]. Another recent success of multimodal learning is in image generation, where DALL-E [59] and later models [60, 66, 52, 90, 64] demonstrated the potential of text-guided image generation by producing high-quality images specific to the provided text.

从文本、图像和音频等多模态数据中学习，是机器学习里一个长期存在的研究课题[51, 56, 83, 31, 86]。近来，对比损失函数与大型神经网络结合，使视觉和语言模型的泛化能力出现了突破[58, 59, 66]。例如，OpenAI 的 CLIP 模型[58]把 ImageNet[65]上的零样本分类从先前的 11.5% top-1 准确率[41]提升到了 76.2%。CLIP 还在多个困难的分布偏移上取得了前所未有的进展[78, 61, 23, 82, 3, 70]。受 CLIP 表现的启发，不少团队通过增加计算量和训练集规模，进一步改进了图文模型[28, 54, 94, 89]。多模态学习的另一项近期进展是图像生成：DALL-E[59]以及后来的模型[60, 66, 52, 90, 64]表明，按文本生成与提示相对应的高质量图像是可行的。

A critical ingredient in this new generation of image-text models is the pre-training dataset. All of the aforementioned advances rely on large datasets containing hundreds of millions or even billions of image-text pairs, e.g., 400 million for CLIP [58] and 6.6 billion for BASIC [54]. However, none of these datasets are publicly available . While OpenAI still released the CLIP models publicly [58], later papers made neither the pre-training dataset nor the resulting models available to the wider research community [28, 54, 2, 90, 66, 52, 89]. As a result, research in this area has pooled into a small number of industrial research labs, limiting transparency and impeding research progress.

这一代图文模型的关键要素是预训练数据集。上述进展都依赖包含数亿甚至数十亿图文对的大规模数据，例如 CLIP 用了 4 亿对[58]，BASIC 用了 66 亿对[54]。可是， 这些数据集没有一个是公开的 。OpenAI 仍公开了 CLIP 模型本身[58]，但之后的论文既不公开预训练数据，也不公开训练得到的模型[28, 54, 2, 90, 66, 52, 89]。于是，这一方向的研究集中到少数工业实验室，透明度下降，研究进展也受到阻碍。

In this work, we address this challenge and make multimodal training more accessible by assembling a public dataset that is suitable for training large image-text models. Specifically, we introduce LAION-5B, the largest public image-text dataset containing over 5.8 billion examples (see Table 2 for a comparison). By starting from Common Crawl [1] and filtering this data source with an existing CLIP model, we derive a dataset consisting of three parts: 2.32 billion English image-text examples, 2.26 billion multilingual examples, and 1.27 billion examples that are not specific to a particular language (e.g., places, products, etc.). Beyond assembling the dataset, we also explore its ethical implications and flaws that emerge with large-scale data collection. By releasing LAION-5B publicly, we offer the first opportunity for the community to audit and refine a dataset of this magnitude.

本文要应对这一困难，通过整理一套适于训练大型图文模型的公开数据，让多模态训练更容易参与。具体来说，我们引入 LAION-5B，这是当时最大的公开图文数据集，包含超过 58 亿条样本（比较见表 2）。我们从 Common Crawl[1]出发，用现成的 CLIP 模型过滤，得到由三部分组成的数据集：23.2 亿英语图文样本、22.6 亿多语言样本，以及 12.7 亿并不对应某一特定语言的样本（例如地点、商品等）。除了整理数据，我们也讨论大规模采集带来的伦理问题和缺陷。把 LAION-5B 公开之后，社区第一次有机会审计并改进如此规模的数据集。

Figure 2: Zero-Shot Accuracy. CLIP models trained on LAION-400M (ours) [69], a previously released subset of LAION-5B, show competitive zero-shot accuracy compared to CLIP models trained on OpenAI’s original training set WIT when evaluated on ImageNet1k. Dataset # English Img-Txt Pairs Public Datasets MS-COCO 330K CC3M 3M Visual Genome 5.4M WIT 5.5M CC12M 12M RedCaps 12M YFCC100M 100M[^4] LAION-5B (Ours) 2.3B Private Datasets CLIP WIT (OpenAI) 400M ALIGN 1.8B BASIC 6.6B Table 3: Dataset Size. LAION-5B is more than 20 times larger than other public English image-text datasets. We extend the analysis from Desai et al. [14] and compare the sizes of public and private image-text datasets. Figure 2: Zero-Shot Accuracy. CLIP models trained on LAION-400M (ours) [69], a previously released subset of LAION-5B, show competitive zero-shot accuracy compared to CLIP models trained on OpenAI’s original training set WIT when evaluated on ImageNet1k. Dataset # English Img-Txt Pairs Public Datasets MS-COCO 330K CC3M 3M Visual Genome 5.4M WIT 5.5M CC12M 12M RedCaps 12M YFCC100M 100M[^4] LAION-5B (Ours) 2.3B Private Datasets CLIP WIT (OpenAI) 400M ALIGN 1.8B BASIC 6.6B Table 3: Dataset Size. LAION-5B is more than 20 times larger than other public English image-text datasets. We extend the analysis from Desai et al. [14] and compare the sizes of public and private image-text datasets.

图 2： 零样本准确率 。 用此前已发布的 LAION-5B 子集 LAION-400M（本文）[69]训练的 CLIP 模型，在 ImageNet1k 上的零样本准确率，与用 OpenAI 原始训练集 WIT 训练的 CLIP 模型相当。 数据集 英语图文对数 公开数据集 MS-COCO 330K、CC3M 3M、Visual Genome 5.4M、WIT 5.5M、CC12M 12M、RedCaps 12M、YFCC100M 100M[^4]、 LAION-5B（本文） 23 亿 。 私有数据集 CLIP WIT（OpenAI）4 亿、ALIGN 18 亿、BASIC 66 亿。表 3： 数据集规模 。 LAION-5B 比其他公开英语图文数据集大 20 倍以上。我们扩展了 Desai 等人[14]的分析，比较了公开与私有图文数据集的规模。

To validate that LAION-5B is indeed suitable for training large image-text models, we conduct multiple experiments. We focus on matching the performance of OpenAI’s CLIP models because they are the largest publicly released image-text models. OpenAI’s CLIP models were trained on 400 million image-text pairs, and hence we also train CLIP models on a subset of LAION-5B containing the same number of examples (“LAION-400M”). Across a diverse range of problem settings including ImageNet (zero-shot), distribution shifts, VTAB, retrieval, and fine-tuning, our models trained on LAION-400M match or come close to the performance of OpenAI’s CLIP models. Our ViT-L/14 models trained with OpenCLIP are the first open source reproductions of the largest CLIP models released by OpenAI.

为验证 LAION-5B 确实适于训练大型图文模型，我们做了多项实验。我们把目标定为追上 OpenAI CLIP 的表现，因为它们是当时公开的最大图文模型。OpenAI 的 CLIP 在 4 亿图文对上训练，因此我们也在包含同样数量样本的 LAION-5B 子集（“LAION-400M”）上训练 CLIP。在 ImageNet（零样本）、分布偏移、VTAB、检索和微调等多样设定上，用 LAION-400M 训练的模型达到或接近 OpenAI CLIP 的表现。我们用 OpenCLIP 训练的 ViT-L/14，是对 OpenAI 所发布最大 CLIP 模型的首次开源复现。

Despite these validation results, LAION-5B is not a finished data product. Due to the immense size of current image-text pre-training datasets, curating LAION-5B for widespread use goes beyond the scope of a single research paper. Hence we do not only release our dataset, but also our software stack we built for assembling LAION-5B. We view our initial data release and this paper as a first step on the way towards a widely applicable pre-training dataset for multimodal models. As a result, we strongly recommend that LAION-5B should only be used for academic research purposes in its current form. We advise against any applications in deployed systems without carefully investigating behavior and possible biases of models trained on LAION-5B.

尽管有这些验证结果，LAION-5B 并不是 一件完工的数据产品。当前图文预训练数据集规模极大，要把 LAION-5B 整理到可供广泛使用的程度，已经超出单篇研究论文的范围。因此我们不仅发布数据，也发布为组装 LAION-5B 而编写的软件栈。我们把这次初始数据发布和本文视为第一步，目标是逐步得到一套可广泛使用的多模态预训练数据。 以当前形态而言，我们强烈建议 LAION-5B 只用于学术研究。在未仔细考察行为以及可能偏见之前，不建议把在 LAION-5B 上训练的模型用于已部署系统 。

The remainder of the paper proceeds as follows. After reviewing related work, we present our data collection process for LAION-5B in Section 3. Section 4 then describes LAION-5B’s composition including its various subsets. To validate LAION-5B, we reproduce and evaluate different image-text models in Section 5. Before concluding, we discuss the technical limitations of LAION-5B in Section 6 and safety and ethics concerns in Section 7.

论文其余部分安排如下。回顾相关工作之后，第 3 节介绍 LAION-5B 的数据采集过程。第 4 节说明其构成及各个子集。为验证 LAION-5B，我们在第 5 节复现并评估不同的图文模型。结论之前，第 6 节讨论技术局限，第 7 节讨论安全与伦理问题。

2 Related Work

2 相关工作

Vision-Language Models. Radford et al. [58] made a large step forward in multimodal learning for image-text data with their CLIP (Contrastive Language-Image Pre-training) model. The authors proposed a contrastive learning scheme to embed both images and text into a shared representation space, which enabled unparalleled performance in zero-shot image classification. Moreover, CLIP made large progress on multiple challenging distribution shifts [78, 84].

视觉—语言模型 。 Radford 等人[58]用 CLIP（Contrastive Language-Image Pre-training）在图文多模态学习上迈出了一大步。作者提出一种对比学习方案，把图像和文本嵌入到共享表示空间，从而在零样本图像分类上取得了前所未有的表现。CLIP 还在多个困难的分布偏移上取得了显著进展[78, 84]。

After CLIP’s initial success, ALIGN and BASIC improved contrastive multimodal learning by increasing the training set size and the batch size used for training [28, 54]. LiT also increased training scale and experimented with a combination of pre-trained image representations and contrastive fine-tuning to connect frozen image representations to text [94]. Flamingo introduced the first large vision-language model with in-context learning [2]. Other papers have combined contrastive losses with image captioning to further improve performance [89, 43]. Beyond image classification and retrieval, the community later adapted CLIP to further vision tasks such as object navigation and visual question answering [50, 32, 72, 17].

CLIP 取得初步成功之后，ALIGN 和 BASIC 通过增大训练集和训练批量，改进了对比式多模态学习[28, 54]。LiT 同样扩大了训练规模，并尝试把预训练图像表示与对比微调结合起来，以便把冻结的图像表示接到文本上[94]。Flamingo 引入了第一个具备上下文学习能力的大型视觉—语言模型[2]。也有工作把对比损失与图像描述结合起来，以进一步提高表现[89, 43]。在图像分类和检索之外，社区后来还把 CLIP 用到目标导航和视觉问答等其他视觉任务上[50, 32, 72, 17]。

Another direction that has recently seen large progress in multimodal learning is text-guided image generation [47, 62, 95]. Specifically, DALL-E demonstrated diverse image generation capabilities for text prompts combining multiple concepts [59]. GLIDE, DALL-E 2, Imagen, Parti, and Stable Diffusion then improved visual fidelity and text-prompt correspondence [52, 66, 60, 90, 64].

多模态学习里最近进展很快的另一个方向是按文本生成图像[47, 62, 95]。DALL-E 展示了面对组合多个概念的文本提示时，也能生成多样的图像[59]。随后，GLIDE、DALL-E 2、Imagen、Parti 和 Stable Diffusion 提高了视觉保真度，也让生成结果与文本提示更加对应[52, 66, 60, 90, 64]。

Image-Text Datasets. Earlier dataset creation efforts such as MS-COCO and Visual Genome curated image and region labels through human annotation [44, 36]. While this resulted in high-quality labels, it also limited the scale of the datasets to only 330K and 5M examples, respectively. The web-harvested YFCC-100M dataset is substantially larger with about 99 million images and one million videos from Flickr, but only contains the user-generated metadata without additional annotations collected specifically for training computer vision models [79]. As a result, the text associated with an image sometimes has little to no correspondence with the actual image content.

图文数据集 。 早期的数据集工作，如 MS-COCO 和 Visual Genome，通过人工标注来整理图像和区域标签[44, 36]。标签质量很高，规模却分别只有 33 万和 500 万条。从网页采集的 YFCC-100M 要大得多，约有 9900 万张图片和 100 万段来自 Flickr 的视频，但只包含用户自己填写的元数据，并没有专门为训练计算机视觉模型而收集的额外标注[79]。因此，与图像对应的文本有时和图像内容几乎没有关系。

To address this shortcoming of web-harvested image-text data, the Conceptual Captions dataset (CC3M) started with images and alt-text collected from the web, but then performed additional data cleaning procedures [71]. To increase the size of the dataset, researchers later relaxed the filtering protocol to arrive at the subsequent CC12M dataset [11]. Building datasets from alt-text continued with ALT200M [26] and ALIGN [28], which increased the dataset size up to 1.8 billion image-text pairs. In contrast to relying on alt-text, RedCaps used the captions provided by Reddit users to collect higher quality captions [14].

为弥补网页采集图文数据的这一不足，Conceptual Captions（CC3M）同样从网上收集图像和 alt-text，但又做了额外的数据清洗[71]。为了扩大规模，研究者后来放宽过滤规则，得到后续的 CC12M[11]。继续用 alt-text 建库的还有 ALT200M[26]和 ALIGN[28]，规模一直扩到 18 亿图文对。与依赖 alt-text 不同，RedCaps 使用 Reddit 用户提供的说明文字，以收集质量更高的描述[14]。

Datasets with non-English image-text pairs are less common. As a result, researchers translated English captioning datasets to other languages such as Farsi, Korean, and Japanese [74, 73, 67]. To the best of our knowledge, the largest multilingual dataset before LAION-5B has around 36 million samples from Wikipedia Image Text [75]. With the release of LAION-5B, researchers now have access to roughly two orders of magnitude more multilingual samples, which provides new opportunities for research on low-resource languages and multilingual models.

非英语图文对数据集更少见。因此，研究者常把英语描述数据集翻译成波斯语、韩语和日语等其他语言[74, 73, 67]。据我们所知，在 LAION-5B 之前，最大的多语言数据集来自 Wikipedia Image Text，大约有 3600 万条样本[75]。LAION-5B 发布之后，研究者可以获得大约多两个数量级的多语言样本，为低资源语言和多语言模型研究提供了新的机会。

Scaling Behavior. Improving model performance by increasing data scale has been a theme in machine learning since at least the ImageNet dataset [13]. In the following decade, computer vision benefited from growth in model, data, and compute scale, in addition to advances in both convolutional and transformer architectures [33, 15, 81, 92]. Industrial research labs assembled large internal datasets such as Instagram-1B, JFT300M, and JFT3B to support image pre-training [46, 77, 93]. Natural language processing (NLP) demonstrated the beneficial effect of model, data, and compute scale on generalization through large language models such as GPT-3 [8] and associated experiments on scaling behavior [30]. Community efforts like the The Pile [18] and BigScience ROOTS [40] made large text datasets more accessible.

规模效应 。 靠扩大数据规模来提高模型表现，至少从 ImageNet 数据集起就是机器学习里的一条主线[13]。此后十年，计算机视觉得益于模型、数据和计算规模的增长，以及卷积和 Transformer 架构上的进展[33, 15, 81, 92]。工业实验室整理了 Instagram-1B、JFT300M 和 JFT3B 等大型内部数据集，用于图像预训练[46, 77, 93]。自然语言处理则通过 GPT-3[8]等大语言模型以及相关的规模实验[30]，表明模型、数据和计算规模有利于泛化。The Pile[18]和 BigScience ROOTS[40]这类社区工作，也让大规模文本数据更容易获得。

3 Collection Methodology

3 采集方法

We constructed LAION-5B starting from Common Crawl, a public web archive [1]. The Common Crawl organization crawls the web since 2008 and publishes the results in snapshots approximately every month. Recent snapshots each contain about 300 TiB of data for around 3 billion web pages. In the following, we introduce our pipeline to assemble and filter a vision-language dataset from images in Common Crawl and their associated HTML alt-text.

我们从公开网络存档 Common Crawl[1]出发构建 LAION-5B。Common Crawl 组织自 2008 年起抓取网页，大约每月发布一次快照。最近的快照每次约有 300 TiB 数据，对应大约 30 亿个网页。下面介绍我们如何从 Common Crawl 中的图像及其对应的 HTML alt-text 出发，组装并过滤得到一套视觉—语言数据集。

3.1 Dataset Assembly Pipeline

3.1 数据集聚成流水线

Our dataset assembly pipeline follows the flowchart of Figure 3. At a high level, the pipeline consists of three main components: (i) distributed filtering of the Common Crawl web pages, (ii) distributed downloading of image-text pairs, and (iii) content filtering. The code used for the dataset pipeline may be found on GitHub[^5]. We now describe each component in more detail.

我们的数据集聚成流水线如图 3 所示。整体上包含三个主要部分：（i）对 Common Crawl 网页做分布式过滤；（ii）分布式下载图文对；（iii）内容过滤。数据集流水线的代码可在 GitHub 上找到[^5]。下面分别说明各部分。

Figure 3: Overview of the acquisition pipeline: Files are downloaded, tracked, and undergo distributed inference to determine inclusion. Those above the specified CLIP threshold are saved.

图 3： 采集流水线概览 ： 文件被下载、跟踪，并经过分布式推理以决定是否纳入。超过指定 CLIP 阈值的样本会被保存。

Web page filtering. To extract image-text pairs from Common Crawl, we parse the HTML IMG (image) tags from Common Crawl’s WAT metadata files.[^6] Specifically, we focus on images with an alt-text so we can create image-text pairs. The alt-text is an HTML attribute of IMG tags containing alternative text for situations where the corresponding image cannot be rendered. For instance, screen reader software for a visually impaired person may read the alt-text in place of an image, or a search engine may use the alt-text to better index a web page without analyzing the actual image content.

网页过滤 。 为从 Common Crawl 中抽取图文对，我们解析其 WAT 元数据文件里的 HTML IMG（图像）标签。[^6]我们特别关注带有 alt-text 的图像，这样才能组成图文对。alt-text 是 IMG 标签的 HTML 属性，在对应图像无法显示时提供替代文本。例如，视障人士使用的屏幕阅读软件可能会朗读 alt-text 来代替图像，搜索引擎也可能用 alt-text 更好地索引网页，而不去分析图像本身。

After extracting the alt-text, we perform language detection using CLD3 [53] with three possible outputs: English, another language, or no detected language (i.e., all detections are below a confidence threshold [69]). Based on a manual inspection of a random sample, the “no language” set contains language-agnostic short form text such as the names of products and places.

抽出 alt-text 之后，我们用 CLD3[53]做语言检测，可能得到三种结果：英语、另一种语言，或未检出语言（也就是所有检测结果都低于置信阈值[69]）。对随机样本的人工检查表明，“无语言”集合里多是与具体语言无关的短文本，例如商品名和地名。

We stored the resulting data in a PostgreSQL server for processing in the next stages of the pipeline. We maintained about 500M image URLs in the server at all times.

我们把得到的数据存入 PostgreSQL 服务器，供流水线后续阶段处理。服务器中始终维持大约 5 亿个图像 URL。

Downloading Image-Text Pairs. In order to maximize resource utilization, we downloaded the raw images from the parsed URLs with asynchronous requests using the Trio and Asks Python libraries. To limit costs, we chose a small cloud node with 2 vCPUs, 1GB of RAM, and 10Mbps download bandwidth as a worker instance. Such a worker can process 10,000 links in about 10 - 15 minutes. We utilized roughly 300 workers in parallel and batched the workload into chunks of 10,000 links taken from the aforementioned PostgreSQL server.

下载图文对 。 为提高资源利用率，我们用 Trio 和 Asks 这两个 Python 库，以异步请求从解析出的 URL 下载原始图像。为控制成本，我们选择很小的云节点作为工作实例：2 个 vCPU、1GB 内存、10Mbps 下载带宽。这样一个工作节点大约 10 到 15 分钟可以处理 1 万条链接。我们并行使用约 300 个工作节点，并把任务按每批 1 万条链接划分，链接来自上述 PostgreSQL 服务器。

Post-Processing. After downloading the WAT files from Common Crawl, we removed data with less than 5 characters of text, less than 5 KB of image data, and potentially malicious, large, or redundant images. To conclude the pipeline, we filtered image-text pairs based on their content. Specifically, we computed cosine similarities between the image and text encodings with OpenAI’s ViT-B/32 CLIP model. For languages other than English, we utilized the multi-lingual CLIP ViT-B/32 from Carlsson et al. [10]. While OpenAI also released larger CLIP models later, these models were not available when we began to assemble LAION-5B. For consistency, we therefore relied on ViT-B/32 CLIP models for the entire dataset. We removed all English image-text pairs with cosine similarity below 0.28, and all other pairs with similarity below 0.26. This step removed around 90% of the original 50 billion images, leaving just short of 6 billion examples.

后处理 。 从 Common Crawl 下载 WAT 文件之后，我们去掉文本少于 5 个字符、图像数据少于 5 KB，以及可能有害、过大或重复的图像。流水线的最后一步按内容过滤图文对。具体来说，我们用 OpenAI 的 ViT-B/32 CLIP 模型计算图像编码与文本编码之间的余弦相似度。对于英语以外的语言，我们使用 Carlsson 等人[10]的多语言 CLIP ViT-B/32。OpenAI 后来也发布了更大的 CLIP 模型，但我们开始组装 LAION-5B 时这些模型还不可用。为保持一致，整套数据都依赖 ViT-B/32 CLIP。我们去掉余弦相似度低于 0.28 的英语图文对，以及其他相似度低于 0.26 的图文对。这一步去掉了原始约 500 亿张图像中约 90% 的样本，最后剩下略少于 60 亿条。

3.2 Safety During Collection

3.2 采集过程中的安全处理

Current automated filtering techniques are far from perfect: harmful images are likely to pass, and others are likely to be falsely removed. We make a best effort to identify, document, and tag such content. In the case of illegal content, we computed CLIP embeddings to filter out such samples. Furthermore, these images and texts could amplify the social bias of machine learning models, especially ones trained with no or weak supervision [76]. It is important to note that the above mentioned classifiers are not perfect, especially keeping the complexity of these tasks and the diverse opinions of different cultures in mind. Therefore, we advocate using these tags responsibly, not relying on them to create a truly safe, “production-ready” subset after removing all potentially problematic samples. For a detailed discussion in this regard, we refer to Sec. 7.

现有的自动过滤远谈不上完善：有害图像仍可能通过，另一些则可能被误删。我们尽力识别、记录并为这类内容打上标签。对于非法内容，我们计算 CLIP 嵌入来过滤相应样本。此外，这些图像和文本可能放大机器学习模型的社会偏见，尤其是在无监督或弱监督训练时[76]。需要强调的是，上述分类器并不完美，特别是考虑到这些任务本身的复杂性，以及不同文化中意见的多样性。因此，我们主张负责任地使用这些标签，不要以为删掉所有可能有问题的样本之后，就能得到真正安全、可直接用于生产的子集。更详细的讨论见第 7 节。

To encourage research in fields such as dataset curation, we refrain from removing potentially offensive samples and tag them instead. The user can decide whether to include content depending on their task. To this end, we also encourage model developers to state, e.g., in their model card [49] which subsets and tagged images are used.

为鼓励数据集整理等领域的研究，我们并不直接删除可能令人不适的样本，而是为它们打上标签。使用者可以根据自己的任务决定是否纳入这些内容。为此，我们也鼓励模型开发者在模型卡[49]中说明使用了哪些子集和带标签的图像。

We apply Q16 [68] and our own specialized pornographic and sexualized content classifier (here referred to as NSFW) to identify and document a broad range of inappropriate concepts displaying not only persons but also objects, symbols, and text, see cf. [68] and Appendix Sec. C.5 and Sec. C.6 for details. Both classifiers are based on CLIP embeddings. Following our main intention of a publicly available dataset, these two approaches, as with all other implementations related to LAION 5B, are open-sourced.

我们使用 Q16[68]以及自己专门训练的色情与性化内容分类器（下文称为 NSFW），来识别并记录一类较宽的不宜概念：它们不仅出现在人物上，也可能出现在物体、符号和文本中，详见[68]以及附录 C.5 和 C.6 节。两个分类器都基于 CLIP 嵌入。与公开数据集这一主旨一致，这两种方法以及与 LAION-5B 相关的其他实现都已开源。

We separate pornographic content and otherwise inappropriate content (e.g. harm, exploitation and degradation). Both can be dis- and enabled in the publicly available dataset exploration UI.[^7] With both together, the UI and the openly accessible code, we encourage users to explore and, subsequently, report further not yet detected content and thus contribute to the improvement of our and other existing approaches.

我们把色情内容和其他不宜内容（例如伤害、剥削和贬低）分开处理。两者都可以在公开的数据集浏览界面中打开或关闭。[^7]有了这一界面和公开代码，我们鼓励使用者继续浏览，并报告尚未检出的内容，从而帮助改进我们的方法以及其他已有方法。

4 Dataset Composition

4 数据集构成

(a)

(a)

We release LAION-5B as the following three subsets:

我们把 LAION-5B 按以下三个子集发布：

• 2.32 billion English image-text pairs. We refer to this subset as LAION-2B-en or LAION-2B if the language is clear from context.

• 23.2 亿条英语图文对。我们将该子集称为 LAION-2B-en；在上下文已能说明语言时，也简称为 LAION-2B。

• 2.26 billion image-text pairs from over 100 other languages. In the multilingual subset, the top-5 most frequent languages are Russian (10.6%), French (7.4%), German (6.6%), Spanish (6.6%), and Chinese (6.3%).

• 来自 100 多种其他语言的 22.6 亿条图文对。在多语言子集中，出现最多的 5 种语言是俄语（10.6%）、法语（7.4%）、德语（6.6%）、西班牙语（6.6%）和中文（6.3%）。

• 1.27 billion samples where a language could not be clearly detected. Based on visually inspecting a random subset of these low-confidence language samples, the corresponding images often depict products or places. The captions contain language with clear semantics, but might also include noise such as keywords for search engine optimiziation or product tags.

• 12.7 亿条无法清楚判定语言的样本。对其中低置信语言样本的随机子集做目视检查后可以看到，对应图像常常是商品或地点。说明文字本身往往有清楚的语义，但也可能夹杂搜索引擎优化关键词或商品标签这类噪声。

We provide metadata files in the Apache Parquet format that consist of the following attributes for each image-text pair:

我们以 Apache Parquet 格式提供元数据文件，每条图文对包含以下属性：

• A 64-bit integer identifier

• 一个 64 位整数标识符

• The URL of the image.

• 图像的 URL。

• The text string.

• 文本字符串。

• Height and width of the image.

• 图像的高度和宽度。

• Cosine similarity between the text and image embeddings.

• 文本嵌入与图像嵌入之间的余弦相似度。

• The output from our NSFW and watermark detectors (one score between 0 and 1 each).

• NSFW 检测器和水印检测器的输出（各一个介于 0 和 1 之间的分数）。

3% of images were detected as NSFW, which can be filtered out by a user with the NSFW tag.

约 3% 的图像被检测为 NSFW，使用者可以用 NSFW 标签把它们滤掉。

5 Experiments Validating LAION-5B

5 验证 LAION-5B 的实验

In this section, we showcase prior work using the LAION-400M [69] and other subsets as well as our CLIP reproduction studies to give quantitative and qualitative evidence of the dataset’s utility for training SOTA large scale language-vision models.

本节展示此前使用 LAION-400M[69]及其他子集的工作，以及我们复现 CLIP 的研究，从定量和定性两方面说明该数据集能够训练出当前一流的大规模语言—视觉模型。

5.1 Usage Examples

5.1 使用实例

Subdataset Generation. LAION-5B’s scale enables novel dataset curation for computer vision related tasks. Recently, researchers have utilized both LAION-5B and a subset, LAION-400M, as a data source in vision related tasks such as facial representation learning [96] and invasive species mitigation [38]. Within LAION, we have compiled from LAION-5B both LAION-High-Resolution[^8], a 170M subset for superresolution models, and LAION-Aesthetic[^9], a 120M subset of aesthetic images, as determined by a linear estimator on top of CLIP.

子数据集生成 。 LAION-5B 的规模使得面向计算机视觉相关任务的新型数据整理成为可能。近来已有研究者把 LAION-5B 及其子集 LAION-400M 用作视觉任务的数据来源，例如面部表示学习[96]和入侵物种治理[38]。在 LAION 内部，我们从 LAION-5B 中整理出 LAION-High-Resolution[^8]（约 1.7 亿条，用于超分辨率模型），以及 LAION-Aesthetic[^9]（约 1.2 亿条审美图像，由 CLIP 之上的线性估计器判定）。

CLIP Reproduction and Improvements. Gao et al. [19], trained an enhanced CLIP architecture on the LAION-400M subset, outperforming OpenAI’s CLIP on ImageNet zero-shot classification top-1 accuracy. See Sec. 5.2 for our CLIP reproduction experiments using models of different scales. Training on a LAION-5B subset, Li et al. [42] developed BLIP to unify understanding and generation for vision-language tasks via a novel Vision-Language Pretraining (VLP) framework. It has been shown that BLIP matched or outperformed comparable models as per CIDEr, SPICE, and BLEU@4 metrics. Eichenberg et al. [16] used a LAION subset for MAGMA, a model generating text “answers” for image-question pairs; MAGMA achieves state of the art results on OKVQA metrics and outperforming Frozen [80].

CLIP 的复现与改进 。 Gao 等人[19]在 LAION-400M 子集上训练了增强版 CLIP 结构，ImageNet 零样本分类的 top-1 准确率超过了 OpenAI 的 CLIP。我们用不同规模模型复现 CLIP 的实验见第 5.2 节。Li 等人[42]在 LAION-5B 的一个子集上提出 BLIP，通过一种新的视觉—语言预训练（VLP）框架，把理解与生成统一起来。结果显示，BLIP 在 CIDEr、SPICE 和 BLEU@4 上达到或超过了可比较的模型。Eichenberg 等人[16]用 LAION 的一个子集训练 MAGMA，该模型为图像—问题对生成文本“回答”；MAGMA 在 OKVQA 指标上取得了当时最好的结果，并超过了 Frozen [80]。

Image Generation. Rombach et al. [63] utilized a subset of LAION-5B in training latent diffusion models (LDM) that achieved state-of-the-art results on image inpainting and class-conditional image synthesis. The work was further extended into stable diffusion project that used subsets of LAION-5B (LAION-2B-en, laion-high-resolution and laion-aesthetics[^10]) for training a publicly available SOTA text-to-image generative model (see Appendix Sec. F.2). Furthermore, Gu et al. [21] used LAION-400M to train VQ diffusion text-to-image generation models, which have been shown to be more efficient, and are able to generate higher quality images. Moreover, Saharia et al. [66] showed an improved architecture of a diffusion model that was trained on a subset of LAION-400M that outperforms OpenAI’s recent DALLE-2 and achieves a new state-of-the-art COCO FID of 7.27.

图像生成 。 Rombach 等人[63]用 LAION-5B 的一个子集训练潜扩散模型（LDM），在图像修复和类别条件图像合成上取得了当时最好的结果。这项工作后来扩展为 Stable Diffusion 项目，使用 LAION-5B 的若干子集（LAION-2B-en、laion-high-resolution 和 laion-aesthetics[^10]）训练出公开的一流文生图模型（见附录 F.2）。此外，Gu 等人[21]用 LAION-400M 训练 VQ diffusion 文生图模型，这些模型效率更高，也能生成更高质量的图像。Saharia 等人[66]还展示了一种改进的扩散模型结构，在 LAION-400M 的一个子集上训练后超过了 OpenAI 当时的 DALL-E 2，并在 COCO 上取得 7.27 的新 FID 纪录。

5.2 Experiments on CLIP Reproduction

5.2 复现 CLIP 的实验

In an effort to reproduce the results of CLIP [58], and to validate the data collection pipeline we describe in Sec. 3, we trained several models on LAION-400M [69] and a model on LAION-2B-en, datasets which are both subsets of LAION-5B. As training such models require large compute due to dataset and model sizes that are considered in the experiments, the usage of supercomputers and large compute clusters is necessary in order to train the models efficiently.

为复现 CLIP[58]的结果，并验证第 3 节所述的数据采集流水线，我们在同为 LAION-5B 子集的 LAION-400M[69]上训练了多个模型，也在 LAION-2B-en 上训练了一个模型。由于实验中的数据和模型规模都很大，训练这些模型需要大量计算，因此必须使用超级计算机和大型计算集群才能高效完成。

We used OpenCLIP [27], an open source software for training CLIP-like models. After adapting OpenCLIP for distributed training and execution on JUWELS Booster supercomputer [29], we reproduced CLIP models of different size on the LAION-400M subset. We trained ViT-B/32, ViT-B/16, and ViT-L/14 following CLIP [58], and an additional model that we call ViT-B/16+, a slightly larger version of ViT-B/16. We followed the same hyper-parameter choices of the original CLIP models. We used between 128 and 400 NVIDIA A100 GPUs to train the models. All trained models may be found in the OpenCLIP repository[^11]. For more information about hyper-parameters and training details, see Appendix Sec. E.1.

我们使用开源软件 OpenCLIP[27]来训练类 CLIP 模型。在为分布式训练以及在 JUWELS Booster 超级计算机[29]上运行做了适配之后，我们在 LAION-400M 子集上复现了不同规模的 CLIP 模型。我们按 CLIP[58]训练了 ViT-B/32、ViT-B/16 和 ViT-L/14，另外还训练了一个略大于 ViT-B/16、称为 ViT-B/16+ 的模型。超参数选择与原始 CLIP 模型保持一致。训练使用了 128 到 400 块 NVIDIA A100 GPU。所有训练好的模型都可以在 OpenCLIP 仓库中找到[^11]。超参数和训练细节见附录 E.1。

5.2.1 Zero-Shot Classification and Robustness Performance Following CLIP [58] and subsequent works, we evaluate the models on zero-shot classification. For each downstream dataset, we use a set of pre-defined prompts for each class, which we collected from prior works [58, 94]. We compute the embeddings of each class by averaging over the embedding of the prompts, computed each using the text encoder. For each image, and for each class, we compute the cosine similarity between their embeddings, and classify each image as the class that have the largest cosine similarity with the image embedding. We evaluate the models using top-1 accuracy.

5.2.1 零样本分类与稳健性 沿用 CLIP[58]及后续工作，我们用零样本分类来评估模型。对每个下游数据集，我们为每个类别使用一组预先定义的提示，这些提示收集自已有工作[58, 94]。我们用文本编码器分别计算各条提示的嵌入，再取平均，得到每个类别的嵌入。对每张图像和每个类别，计算它们嵌入之间的余弦相似度，并把图像判为与图像嵌入余弦相似度最大的那个类别。评估指标是 top-1 准确率。

In Tab. 4, we show a comparison between models trained on LAION (400M, 2B) and original CLIP from [58]. We follow [94] and evaluate robustness performance on ImageNet distribution shift datasets [61, 23, 82, 25, 3]. Additionally, we construct a benchmark we call VTAB+, a superset of VTAB [91], on which we compute the average top-1 accuracy over 35 tasks[^12]. We can see that on ImageNet-1k (noted "INet" on the table), performance of LAION-400M models and original CLIP models (trained on a 400M private dataset) is matched well. On the four ImageNet distribution shift datasets, we observe some larger differences, notably on ObjNet (CLIP WIT is better) and INet-S (LAION is better), which allows us to conclude that in overall, CLIP models trained on LAION match in their robustness original CLIP. With ViT-B/32 and ViT-L/14, training on the larger LAION-2B-en improves over LAION-400M model everywhere.

表 4 比较了在 LAION（4 亿、20 亿）上训练的模型与文献[58]中的原始 CLIP。我们沿用[94]，在 ImageNet 分布偏移数据集上评估稳健性[61, 23, 82, 25, 3]。此外，我们构建了一个称为 VTAB+ 的基准，它是 VTAB[91]的超集，我们在其中 35 项任务上计算平均 top-1 准确率[^12]。可以看到，在 ImageNet-1k（表中记为 “INet”）上，LAION-400M 模型与在 4 亿私有数据上训练的原始 CLIP 表现相当接近。在四个 ImageNet 分布偏移数据集上，差异更大一些，尤其是 ObjNet（CLIP WIT 更好）和 INet-S（LAION 更好）；总体而言，可以认为在 LAION 上训练的 CLIP 在稳健性上与原始 CLIP 相当。对 ViT-B/32 和 ViT-L/14 来说，在更大的 LAION-2B-en 上训练，各处都优于 LAION-400M 模型。

To obtain an idea about how the zero-shot performance improves with scale, we show the relationship between the total compute and accuracy on VTAB+ on models trained on LAION (400M, 2B-en). In Figure 5, we see that accuracy on VTAB+ improves with compute (log-log plot). It would be interesting to study in future work if the relationship between compute and accuracy keeps showing the same trend or whether we start to see saturation, like it was observed in [93]. Here, we can report that increasing either model or data scale for CLIP pre-training results in improvement of zero-shot classification performance on various downstream transfer targets. For a full overview of zero-shot classification and retrieval results, view Sec. E.3 of the Appendix.

为了解零样本表现如何随规模提升，我们展示了在 LAION（4 亿、20 亿英语）上训练的模型中，总计算量与 VTAB+ 准确率之间的关系。图 5 显示，VTAB+ 准确率随计算量提高（双对数图）。未来值得研究这种关系是否会一直保持，还是会像[93]中观察到的那样开始饱和。就目前而言，增大 CLIP 预训练的模型规模或数据规模，都会提高在多种下游迁移目标上的零样本分类表现。零样本分类和检索的完整结果见附录 E.3。

To show that larger dataset scale matters for the performance of pre-trained models, we perform additional experiments using ViT-B/32 and ViT-L/14 on different LAION-5B and LAION-400M subsets, while varying the amount of training compute (samples seen). Our findings confirm that the effect of dataset scale is significant, given sufficient compute for training. For instance, for the same amount of compute (34B images seen), training ViT-L/14 on LAION-2B-en (75.4%) outperforms LAION-400M (73.9%) on ImageNet-1k zero-shot classification. Same effect is observed for smaller ViT-B/32 model. For more detailed results, see Fig. 15 and Tab. 8 in the Appendix.

为说明更大的数据规模对预训练模型表现很重要，我们用 ViT-B/32 和 ViT-L/14，在不同的 LAION-5B 与 LAION-400M 子集上做了额外实验，并改变训练计算量（见到的样本数）。结果证实，只要训练计算量足够，数据规模的影响是显著的。例如，在相同计算量（见到 340 亿张图像）下，ViT-L/14 在 LAION-2B-en 上训练后，ImageNet-1k 零样本分类达到 75.4%，超过在 LAION-400M 上的 73.9%。较小的 ViT-B/32 上也观察到同样的效应。更详细的结果见附录图 15 和表 8。

(a)

(a)

Model Pre-training INet INet-v2 INet-R INet-S ObjNet VTAB+ B/32 CLIP WIT 63.3 56.0 69.4 42.3 44.2 45.4 LAION-400M {62.9}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}} {55.1}^{\tiny\color[rgb]{1,0,0}\textbf{-0.9}} {73.4}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.0}} {49.4}^{\tiny\color[rgb]{0,0.88,0}\textbf{+7.1}} {43.9}^{\tiny\color[rgb]{1,0,0}\textbf{-0.3}} {45.6}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}} LAION-2B-en {65.7}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.4}} {57.4}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.4}} {75.9}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.5}} {52.9}^{\tiny\color[rgb]{0,0.88,0}\textbf{+10.6}} {48.7}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.5}} {47.9}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.5}} B/16 CLIP WIT 68.3 61.9 77.7 48.2 55.3 47.5 LAION-400M {67.0}^{\tiny\color[rgb]{1,0,0}\textbf{-1.3}} {59.6}^{\tiny\color[rgb]{1,0,0}\textbf{-2.3}} {77.9}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}} {52.4}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.2}} {51.5}^{\tiny\color[rgb]{1,0,0}\textbf{-3.8}} {48.3}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.8}} B/16+ LAION-400M 69.2 61.5 80.5 54.4 53.9 49.2 L/14 CLIP WIT 75.6 69.8 87.9 59.6 69.0 55.7 LAION-400M {72.8}^{\tiny\color[rgb]{1,0,0}\textbf{-2.8}} {65.4}^{\tiny\color[rgb]{1,0,0}\textbf{-4.4}} {84.7}^{\tiny\color[rgb]{1,0,0}\textbf{-3.2}} 59.6 {59.9}^{\tiny\color[rgb]{1,0,0}\textbf{-9.1}} {51.8}^{\tiny\color[rgb]{1,0,0}\textbf{-3.9}} LAION-2B-en {\color[rgb]{0,0,0}{75.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.3}} {\color[rgb]{0,0,0}{67.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.0}} {\color[rgb]{0,0,0}{87.4}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.5}} {\color[rgb]{0,0,0}{63.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+3.7}} {\color[rgb]{0,0,0}{65.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.6}} {\color[rgb]{0,0,0}{54.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.2}} Table 4: Comparison between CLIP models trained on LAION (400M, 2B) and the original CLIP models [58] trained on OpenAI’s WebImageText (WIT) dataset. We show zero-shot top-1 classification accuracy (%) on various datasets including ImageNet, four ImageNet distribution shift datasets, and a benchmark we call VTAB+, where we average performance over 35 tasks. See Appendix E.3 for more details about the datasets used for evaluation and the results.

Model	Pre-training	INet	INet-v2	INet-R	INet-S	ObjNet	VTAB+
B/32	CLIP WIT	63.3	56.0	69.4	42.3	44.2	45.4
	LAION-400M	${62.9}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	${55.1}^{\tiny\color[rgb]{1,0,0}\textbf{-0.9}}$	${73.4}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.0}}$	${49.4}^{\tiny\color[rgb]{0,0.88,0}\textbf{+7.1}}$	${43.9}^{\tiny\color[rgb]{1,0,0}\textbf{-0.3}}$	${45.6}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$
	LAION-2B-en	${65.7}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.4}}$	${57.4}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.4}}$	${75.9}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.5}}$	${52.9}^{\tiny\color[rgb]{0,0.88,0}\textbf{+10.6}}$	${48.7}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.5}}$	${47.9}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.5}}$
B/16	CLIP WIT	68.3	61.9	77.7	48.2	55.3	47.5
	LAION-400M	${67.0}^{\tiny\color[rgb]{1,0,0}\textbf{-1.3}}$	${59.6}^{\tiny\color[rgb]{1,0,0}\textbf{-2.3}}$	${77.9}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	${52.4}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.2}}$	${51.5}^{\tiny\color[rgb]{1,0,0}\textbf{-3.8}}$	${48.3}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.8}}$
B/16+	LAION-400M	69.2	61.5	80.5	54.4	53.9	49.2
L/14	CLIP WIT	75.6	69.8	87.9	59.6	69.0	55.7
	LAION-400M	${72.8}^{\tiny\color[rgb]{1,0,0}\textbf{-2.8}}$	${65.4}^{\tiny\color[rgb]{1,0,0}\textbf{-4.4}}$	${84.7}^{\tiny\color[rgb]{1,0,0}\textbf{-3.2}}$	59.6	${59.9}^{\tiny\color[rgb]{1,0,0}\textbf{-9.1}}$	${51.8}^{\tiny\color[rgb]{1,0,0}\textbf{-3.9}}$
	LAION-2B-en	${\color[rgb]{0,0,0}{75.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.3}}$	${\color[rgb]{0,0,0}{67.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.0}}$	${\color[rgb]{0,0,0}{87.4}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.5}}$	${\color[rgb]{0,0,0}{63.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+3.7}}$	${\color[rgb]{0,0,0}{65.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.6}}$	${\color[rgb]{0,0,0}{54.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.2}}$

模型 预训练 INet INet-v2 INet-R INet-S ObjNet VTAB+ B/32 CLIP WIT 63.3 56.0 69.4 42.3 44.2 45.4 LAION-400M {62.9}^{ iny\color[rgb]{1,0,0}\textbf{-0.4}} {55.1}^{ iny\color[rgb]{1,0,0}\textbf{-0.9}} {73.4}^{ iny\color[rgb]{0,0.88,0}\textbf{+4.0}} {49.4}^{ iny\color[rgb]{0,0.88,0}\textbf{+7.1}} {43.9}^{ iny\color[rgb]{1,0,0}\textbf{-0.3}} {45.6}^{ iny\color[rgb]{0,0.88,0}\textbf{+0.2}} LAION-2B-en {65.7}^{ iny\color[rgb]{0,0.88,0}\textbf{+2.4}} {57.4}^{ iny\color[rgb]{0,0.88,0}\textbf{+1.4}} {75.9}^{ iny\color[rgb]{0,0.88,0}\textbf{+6.5}} {52.9}^{ iny\color[rgb]{0,0.88,0}\textbf{+10.6}} {48.7}^{ iny\color[rgb]{0,0.88,0}\textbf{+4.5}} {47.9}^{ iny\color[rgb]{0,0.88,0}\textbf{+2.5}} B/16 CLIP WIT 68.3 61.9 77.7 48.2 55.3 47.5 LAION-400M {67.0}^{ iny\color[rgb]{1,0,0}\textbf{-1.3}} {59.6}^{ iny\color[rgb]{1,0,0}\textbf{-2.3}} {77.9}^{ iny\color[rgb]{0,0.88,0}\textbf{+0.2}} {52.4}^{ iny\color[rgb]{0,0.88,0}\textbf{+4.2}} {51.5}^{ iny\color[rgb]{1,0,0}\textbf{-3.8}} {48.3}^{ iny\color[rgb]{0,0.88,0}\textbf{+0.8}} B/16+ LAION-400M 69.2 61.5 80.5 54.4 53.9 49.2 L/14 CLIP WIT 75.6 69.8 87.9 59.6 69.0 55.7 LAION-400M {72.8}^{ iny\color[rgb]{1,0,0}\textbf{-2.8}} {65.4}^{ iny\color[rgb]{1,0,0}\textbf{-4.4}} {84.7}^{ iny\color[rgb]{1,0,0}\textbf{-3.2}} 59.6 {59.9}^{ iny\color[rgb]{1,0,0}\textbf{-9.1}} {51.8}^{ iny\color[rgb]{1,0,0}\textbf{-3.9}} LAION-2B-en {\color[rgb]{0,0,0}{75.2}}^{ iny\color[rgb]{1,0,0}\textbf{-0.3}} {\color[rgb]{0,0,0}{67.7}}^{ iny\color[rgb]{1,0,0}\textbf{-2.0}} {\color[rgb]{0,0,0}{87.4}}^{ iny\color[rgb]{1,0,0}\textbf{-0.5}} {\color[rgb]{0,0,0}{63.3}}^{ iny\color[rgb]{0,0.88,0}\textbf{+3.7}} {\color[rgb]{0,0,0}{65.5}}^{ iny\color[rgb]{1,0,0}\textbf{-3.6}} {\color[rgb]{0,0,0}{54.6}}^{ iny\color[rgb]{1,0,0}\textbf{-1.2}} 表 4：在 LAION（4 亿、20 亿）上训练的 CLIP 模型，与在 OpenAI 的 WebImageText（WIT）上训练的原始 CLIP 模型[58]的比较。我们报告 ImageNet、四个 ImageNet 分布偏移数据集，以及我们称为 VTAB+ 的基准（35 项任务的平均表现）上的零样本 top-1 分类准确率（%）。评估所用数据集及更多结果见附录 E.3。

模型	预训练	INet	INet-v2	INet-R	INet-S	ObjNet	VTAB+
B/32	CLIP WIT	63.3	56.0	69.4	42.3	44.2	45.4
	LAION-400M	${62.9}^{ iny\color[rgb]{1,0,0}\textbf{-0.4}}$	${55.1}^{ iny\color[rgb]{1,0,0}\textbf{-0.9}}$	${73.4}^{ iny\color[rgb]{0,0.88,0}\textbf{+4.0}}$	${49.4}^{ iny\color[rgb]{0,0.88,0}\textbf{+7.1}}$	${43.9}^{ iny\color[rgb]{1,0,0}\textbf{-0.3}}$	${45.6}^{ iny\color[rgb]{0,0.88,0}\textbf{+0.2}}$
	LAION-2B-en	${65.7}^{ iny\color[rgb]{0,0.88,0}\textbf{+2.4}}$	${57.4}^{ iny\color[rgb]{0,0.88,0}\textbf{+1.4}}$	${75.9}^{ iny\color[rgb]{0,0.88,0}\textbf{+6.5}}$	${52.9}^{ iny\color[rgb]{0,0.88,0}\textbf{+10.6}}$	${48.7}^{ iny\color[rgb]{0,0.88,0}\textbf{+4.5}}$	${47.9}^{ iny\color[rgb]{0,0.88,0}\textbf{+2.5}}$
B/16	CLIP WIT	68.3	61.9	77.7	48.2	55.3	47.5
	LAION-400M	${67.0}^{ iny\color[rgb]{1,0,0}\textbf{-1.3}}$	${59.6}^{ iny\color[rgb]{1,0,0}\textbf{-2.3}}$	${77.9}^{ iny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	${52.4}^{ iny\color[rgb]{0,0.88,0}\textbf{+4.2}}$	${51.5}^{ iny\color[rgb]{1,0,0}\textbf{-3.8}}$	${48.3}^{ iny\color[rgb]{0,0.88,0}\textbf{+0.8}}$
B/16+	LAION-400M	69.2	61.5	80.5	54.4	53.9	49.2
L/14	CLIP WIT	75.6	69.8	87.9	59.6	69.0	55.7
	LAION-400M	${72.8}^{ iny\color[rgb]{1,0,0}\textbf{-2.8}}$	${65.4}^{ iny\color[rgb]{1,0,0}\textbf{-4.4}}$	${84.7}^{ iny\color[rgb]{1,0,0}\textbf{-3.2}}$	59.6	${59.9}^{ iny\color[rgb]{1,0,0}\textbf{-9.1}}$	${51.8}^{ iny\color[rgb]{1,0,0}\textbf{-3.9}}$
	LAION-2B-en	${\color[rgb]{0,0,0}{75.2}}^{ iny\color[rgb]{1,0,0}\textbf{-0.3}}$	${\color[rgb]{0,0,0}{67.7}}^{ iny\color[rgb]{1,0,0}\textbf{-2.0}}$	${\color[rgb]{0,0,0}{87.4}}^{ iny\color[rgb]{1,0,0}\textbf{-0.5}}$	${\color[rgb]{0,0,0}{63.3}}^{ iny\color[rgb]{0,0.88,0}\textbf{+3.7}}$	${\color[rgb]{0,0,0}{65.5}}^{ iny\color[rgb]{1,0,0}\textbf{-3.6}}$	${\color[rgb]{0,0,0}{54.6}}^{ iny\color[rgb]{1,0,0}\textbf{-1.2}}$

5.3 Experiments with Generative Models

5.3 生成模型实验

To validate LAION-5B as a dataset for training strong text-to-image generation models, we fine-tuned OpenAI’s GLIDE [52] on LAION-5B data. The obtained results comparing generated samples from original OpenAI GLIDE and from our reproduction (LAIONIDE) are compiled into an interactive web demo[^13]. See Appendix Sec F for more technical details on experiments with GLIDE (F.1) and Stable Diffusion (F.2).

为验证 LAION-5B 能够训练出较强的文生图模型，我们在 LAION-5B 数据上微调了 OpenAI 的 GLIDE[52]。原始 OpenAI GLIDE 与我们复现版本（LAIONIDE）的生成样本比较，整理成了一个可交互的网页演示[^13]。关于 GLIDE（F.1）和 Stable Diffusion（F.2）实验的更多技术细节见附录 F。

6 Technical Limitations

6 技术局限

The large scale of current image-text datasets makes it infeasible to thoroughly investigate all aspects of a dataset in a single publication. Hence we now outline some potential technical limitations specifically affecting LAION-5B. These potential limitations are starting points for future work on analyzing and improving image-text datasets.

当前图文数据集规模很大，很难在单篇论文里彻底考察其所有方面。因此我们在此列出一些尤其可能影响 LAION-5B 的技术局限。这些潜在局限是未来分析并改进图文数据集的起点。

Data Overlap. Our experiments in Section 5.2 show that models trained on LAION-5B achieve good performance on a variety of downstream tasks. However, the LAION-5B training set may overlap with some of the downstream test sets if these test sets are also included in Common Crawl. If overlap is present, it may lead to incorrectly large test set accuracies that overstate the true generalization capabilities of models trained on LAION-5B.

数据重叠 。 第 5.2 节的实验表明，在 LAION-5B 上训练的模型在多种下游任务上表现良好。不过，如果这些下游测试集也包含在 Common Crawl 中，LAION-5B 训练集就可能与部分测试集重叠。若存在重叠，测试集准确率可能会被高估，从而夸大在 LAION-5B 上训练的模型的真实泛化能力。

Overall, we do not consider potential test set overlap to be a serious threat for the validity of results obtained with LAION-5B. OpenAI encountered the same question in the context of their pre-training dataset for CLIP and found only few examples of substantial performance difference due to data overlap on downstream target datasets [58]. Some datasets such as ObjectNet [3] are likely not contained in Common Crawl because ObjectNet was not assembled from web images. Instead, the authors of ObjectNet tasked MTurk workers to take new pictures in their own homes. Nevertheless, measuring the degree of overlap between LAION-5B and popular computer vision benchmarks is an important question for future work, which will include further de-duplication efforts.

总体而言，我们并不认为潜在的测试集重叠会严重威胁用 LAION-5B 得到的结果的有效性。OpenAI 在 CLIP 的预训练数据上也遇到过同样的问题，发现只有少数下游目标数据集因数据重叠而出现明显的表现差异[58]。ObjectNet[3]这类数据集很可能并不包含在 Common Crawl 中，因为它并不是从网络图像组装的，而是由作者请 MTurk 工作者在自己家中拍摄新照片。尽管如此，衡量 LAION-5B 与常见计算机视觉基准之间的重叠程度，仍是未来工作的重要问题，其中也包括进一步的去重。

Other text sources. Birhane et al. [6] described the shortcomings of alt-text and noted that alt-text is not necessarily a good description of the corresponding image. For instance, the alt-text may be search engine optimization (SEO) spam, an incoherent list of keywords, or overly corrupted otherwise. In such cases, the language in the text annotations may become less informative or entirely useless for training. For ImageNet zero-shot classification, BASIC [54] has demonstrated strong results when turning 5 billion of the 6.6 billion captions into the form of CLASS_1 and CLASS_2 and ... and CLASS_K, by using an internal multi-label classification dataset (JFT-3B). Thus, image captions formed by just concatenating class names may also serve as meaningful alternative of otherwise corrupted text. Such a finding adds a possibility of employing generated together with existing natural language captions for training contrastive image-language models with strong zero-shot performance.

其他文本来源 。 Birhane 等人[6]指出了 alt-text 的不足，并说明 alt-text 不一定是对相应图像的好描述。例如，alt-text 可能是搜索引擎优化（SEO）垃圾信息、一串没有连贯语义的关键词，或以其他方式严重损坏。在这些情形下，文本标注中的语言对训练可能信息量很低，甚至完全没有用处。在 ImageNet 零样本分类上，BASIC[54]表明，把 66 亿条说明中的 50 亿条改写成 CLASS_1 and CLASS_2 and ... and CLASS_K 这种形式（借助内部多标签分类数据集 JFT-3B），也能取得很强的结果。因此，仅由类别名拼接而成的图像说明，也可以作为损坏文本的一种有意义替代。这一发现意味着，把生成的说明与已有自然语言说明一起使用，也有可能训练出具有较强零样本能力的对比式图像—语言模型。

Filtering with CLIP. CLIP allows the curation and collection of this dataset to be low-cost and scalable. Such an automated process reduces dramatically necessity for the human control which would be otherwise intractable for such large scale collection. However, through curating with CLIP, we also incur its flaws and model biases. For additional discussion of CLIP filtering related to safety and ethics, see Appendix Sec. G.2.

用 CLIP 过滤 。 CLIP 让这套数据的整理和采集成本低、也容易扩展。这种自动化过程大大降低了对人工检查的需求，否则在如此规模下人工检查是不可行的。不过，用 CLIP 来整理，也会带进它的缺陷和模型偏见。与安全和伦理相关的 CLIP 过滤讨论见附录 G.2。

Filtering by a small scale CLIP ViT-B/32 may leave more image-text pairs with weak or no semantic connection in the dataset while also accidentally removing some high quality image-text pairs than filtering with stronger, larger scale models that were not available in the time of our experiments. The larger CLIP ViT-L/14 model may create a less noisy version of LAION datasets than what was possible with smaller scale CLIP ViT-B/32. We hypothesize that filtering Common Crawl with a CLIP ViT-L model will further increase the quality of our dataset. It is subject to our future work to create a CLIP ViT L/14 filtered version of LAION-400M and LAION-5B to test how this affects model training and downstream transfer performance.

用较小的 CLIP ViT-B/32 过滤，可能会在数据中留下更多语义联系很弱或没有联系的图文对，同时也可能误删一些高质量图文对；若改用我们做实验时尚不可用的更强、更大规模的模型，这种情况会少一些。较大的 CLIP ViT-L/14 有可能得到比小规模 CLIP ViT-B/32 噪声更少的 LAION 数据版本。我们推测，用 CLIP ViT-L 过滤 Common Crawl，会进一步提高数据集的质量。用 CLIP ViT-L/14 过滤得到新的 LAION-400M 和 LAION-5B 版本，并测试这会如何影响模型训练和下游迁移表现，是我们未来的工作。

7 Safety and Ethical Discussion

7 安全与伦理讨论

Recent developments in large-scale models, such as GPT-3 [9], CLIP [57], ALIGN [28], GLIDE [52], and DALLE-2 [60] have potential for far-reaching impact on society, both positive and negative, when deployed in applications such as image classification and generation, recommendation systems, or search engines. Besides model parameter scaling, the advances made so far also rely on the underlying large-scale datasets. Recent research [4, 5] described many potential negative societal implications that may arise due to careless use of vision-language models, e.g., the models perform worse for certain groups of users or reproduce discriminatory behavior.

GPT-3[9]、CLIP[57]、ALIGN[28]、GLIDE[52]和 DALL-E 2[60]等大规模模型的近期发展，在图像分类与生成、推荐系统或搜索引擎等应用中部署时，可能对社会产生广泛影响，既有正面的，也有负面的。除了模型参数规模，迄今的进展也依赖其背后的大规模数据集。近期研究[4, 5]描述了许多可能因不谨慎使用视觉—语言模型而产生的负面社会影响，例如模型对某些用户群体表现更差，或会再现歧视行为。

Unfortunately, only a minority of these models are publicly released, most of them are only accessible by an “input to output” interface. Importantly, the underlying large-scale datasets are also not often publicly available. While open-source efforts exist to re-implement model architectures and training, the closed nature of large-scale datasets used for model training makes any proper systematic investigation of model training and model behavior very hard or even impossible. Studying full training, comparison of different model architectures and progress in large-scale multi-modal learning becomes restricted to those institutions that were able to obtain their closed large-scale datasets. It also results in safety issues of creating and using such models, as broad research community does not get to test both model and the dataset used for its training for causes underlying undesired behaviours.

可惜的是，这些模型只有少数公开发布，大多数只能通过“输入到输出”的接口访问。更重要的是，其背后的大规模数据集也往往并不公开。尽管已有开源工作尝试重新实现模型结构和训练，但用于训练的大规模数据处于封闭状态，使得对模型训练和行为做任何系统调查都变得非常困难，甚至不可能。完整训练、不同模型结构的比较，以及大规模多模态学习的推进，于是被限制在那些已经获得封闭大规模数据的机构内部。这也会带来安全问题：更广泛的研究社区无法同时检查模型和用于训练的数据，从而难以查明不良行为背后的原因。

LAION-5B as an open large-scale dataset provides here not only a chance to make progress in careful studies of the trained models’ capabilities and replication but also to investigate how uncurated large-scale datasets impact various model biases and under which circumstances their usage may result in undesired safety issues. Such research can help to design automated ways to curate and create datasets from uncurated ones that alleviate the bias and safety issues. To this end, LAION also created a number of tools to aid researchers and other users in large-scale data handling and exploration. One such a tool uses pre-computed image embeddings to enable search of images guided either by text or image input via an easily and publically accessible web interface (CLIP retrieval tool[^14], see Appendix Sec. C.4). LAION made also source code for the tool and routines necessary to build an own version of it publicly available[^15] (see Appendix Sec C, C.2, C.3 for more details).

LAION-5B 作为开放的大规模数据集，不仅有助于仔细研究训练所得模型的能力并开展复现，也可以用来考察未加整理的大规模数据如何影响各类模型偏见，以及在何种情形下使用它们会带来不良安全问题。这类研究有助于设计自动化方法，从未加整理的数据中整理出能够减轻偏见和安全问题的数据集。为此，LAION 还制作了若干工具，帮助研究者和其他使用者处理并浏览大规模数据。其中一种工具使用预先计算的图像嵌入，通过易于访问的公开网页界面，按文本或图像输入检索图像（CLIP 检索工具[^14]，见附录 C.4）。LAION 也把该工具的源代码，以及搭建自有版本所需的程序公开了出来[^15]（更多细节见附录 C、C.2、C.3）。

After the release of LAION-400M, several groups (e.g., [6]) already used such tools and investigated potential problems arising from an unfiltered dataset. Motivated by these findings, with LAION-5B, we introduced an improved inappropriate content tagging (cf. Sec. 3.2) as well as a watermark filter, which can improve the safety and quality of the text-to-image models trained on the dataset.

LAION-400M 发布之后，已有若干团队（例如[6]）使用这类工具，考察未过滤数据集可能带来的问题。受这些发现启发，我们在 LAION-5B 中引入了改进后的不宜内容标签（见第 3.2 节）以及水印过滤器，这有助于提高在该数据上训练的文生图模型的安全性和质量。

Such development indicates that this dataset acts as a starting point, and is not the final endpoint, for creating further improved datasets to train models for various tasks. In our opinion, this process is not supposed to be a non-transparent closed-door avenue. It should be approached by broad research community, resulting in open and transparent datasets and procedures for model training. Towards meeting this challenge, the large-scale public image-text dataset of over 5.8 billion pairs and further annotations introduced here provides diversity that can be a starting point for ensuring balance and for selecting safe, curated subsets for corresponding target applications. We encourage everybody to participate in this exciting and important future journey.

这些进展表明，该数据集是一个起点，而不是终点：还需要继续整理出更好的数据，以训练面向各类任务的模型。在我们看来，这一过程不应是不透明的闭门工作，而应由更广泛的研究社区共同推进，从而得到开放、透明的数据集和模型训练流程。为应对这一挑战，本文引入的超过 58 亿对的大规模公开图文数据及其附加标注，提供了有助于保证平衡、并针对目标应用选出安全、经过整理的子集所需的多样性。我们鼓励所有人参与这一重要且令人期待的后续工作。

In the current form, we consider this dataset a research artefact and strongly advocate academic use-only and advise careful investigation of downstream model biases (Appendix Sec. G.2). Additionally, we encourage users to use the described tools and to transparently explore and, subsequently, report further not yet detected content and model behaviour to our dataset repository[^16], and help to further advance existing approaches for data curation using the real-world large dataset introduced here.

以当前形态而言，我们把该数据集视为一件研究产物，强烈主张 仅用于学术用途 ，并建议仔细考察下游模型的偏见（附录 G.2）。此外，我们鼓励使用者利用所述工具，公开地浏览数据，随后把尚未检出的内容和模型行为报告到我们的数据集仓库[^16]，并借助本文引入的真实大规模数据，进一步推进已有的数据整理方法。

Privacy. We comment on privacy issues arising from Common Crawl as source of links in LAION-5B and measures undertaken to handle those in the Appendix Sec. G.1

隐私 。 关于 Common Crawl 作为 LAION-5B 链接来源所带来的隐私问题，以及我们为此采取的措施，见附录 G.1。

8 Conclusion

8 结论

By releasing LAION-5B, a larger updated version of an openly available dataset that contains over 5 billion image-text pairs, we have further pushed the scale of open datasets for training and studying state-of-the-art language-vision models. This scale gives strong increases to zero-shot transfer and robustness.

通过发布 LAION-5B 这一包含超过 50 亿图文对的更新版开放数据集，我们进一步推高了用于训练和研究一流语言—视觉模型的开放数据规模。这一规模明显提高了零样本迁移能力和稳健性。

To validate the utility of LAION-5B, we demonstrated that a subset of our dataset can be used to train SOTA CLIP models of various scale that match the strong zero-shot and robustness performance of the original models trained on closed curated data, or to fine-tune generative models like GLIDE, producing samples of good quality. The dataset thus provides opportunities in multi-language large-scale training and research of language-vision models, that were previously restricted to those having access to proprietary large datasets, to the broader research community. Finally, thanks to its large scale, even a rather strict subset filtering (driven by various criterion like NSFW, watermark presence, resolution) provides high-quality datasets that are still large enough to provide sufficient scale for the training or fine-tuning of strong specialized language-vision models.

为验证 LAION-5B 的效用，我们表明其一个子集可以用来训练不同规模的一流 CLIP 模型，这些模型在零样本和稳健性上能够追上在封闭、经过整理的数据上训练的原始模型；也可以用来微调 GLIDE 这类生成模型，并得到质量较好的样本。该数据集因此把此前仅限于能够获得专有大规模数据者的多语言大规模训练与语言—视觉模型研究，开放给了更广泛的研究社区。最后，正因为规模很大，即便按 NSFW、是否含水印、分辨率等多种标准做较严格的子集过滤，仍然能得到足够大的高质量数据，用来训练或微调较强的专用语言—视觉模型。

Acknowledgments

致谢

We thank Phil Wang, the creator of the DALLE-pytorch github repository[^17], who inspired us and helped creating our open community. We also want to thank Aran Komatsuzaki, Andreas Köpf, Bokai Yu, John David Pressman, Natalie Parde, Gabriel Ilharco, Fredde Frallan (see also Appendix) and all the members of the LAION discord server[^18] for helping crawling image-text-pairs and run inference on their private computers. We want to thank Hugging Face and Stability AI for their continuous financial support and providing hosting space for open datasets and models. We would also like to thank openAI for making their pre-trained CLIP models publicly available, which allowed us to filter the LAION datasets. We would like to express gratitude to all the people who are working on making code, models and data publicly available, advancing community based research and making research more reproducible.

我们感谢 DALLE-pytorch GitHub 仓库的创建者 Phil Wang[^17]，他启发了我们，并帮助建立了这个开放社区。我们也要感谢 Aran Komatsuzaki、Andreas Köpf、Bokai Yu、John David Pressman、Natalie Parde、Gabriel Ilharco、Fredde Frallan（亦见附录）以及 LAION Discord 服务器[^18]的所有成员，他们帮助抓取图文对，并在自己的个人计算机上运行推理。我们感谢 Hugging Face 和 Stability AI 持续的资金支持，以及为开放数据集和模型提供托管空间。我们也感谢 OpenAI 公开其预训练的 CLIP 模型，使我们能够据此过滤 LAION 数据集。我们向所有致力于公开代码、模型和数据、推进基于社区的研究并提高研究可复现性的人表示感谢。

The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. [^19] for funding this work by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Booster [29] at Jülich Supercomputing Centre (JSC). We also acknowledge storage resources on JUST [20] granted and operated by JSC. Patrick Schramowski acknowledges the support by the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) cluster project “The Third Wave of AI”.

作者感谢 Gauss Centre for Supercomputing e.V.[^19] 通过约翰\cdot冯\cdot诺依曼计算研究所（NIC），在于利希超算中心（JSC）的 GCS 超级计算机 JUWELS Booster[29] 上提供计算时间，从而资助了本工作。我们也感谢 JSC 批准并运维的 JUST[20] 存储资源。Patrick Schramowski 感谢黑森州高等教育、研究、科学与艺术部（HMWK）集群项目 “The Third Wave of AI” 的支持。

References

参考文献

[91] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867 , 2019. [92] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. 2021a. doi: 10.48550/ARXIV.2106.04560 . URL https://arxiv.org/abs/2106.04560 . [93] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.04560 , 2021b. [94] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. arXiv preprint arXiv:2111.07991 , 2021c. [95] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision , pages 5907-5915, 2017. [96] Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learning in a visual-linguistic manner. CoRR , abs/2112.03109, 2021. URL https://arxiv.org/abs/2112.03109 .

Appendix (LAION-5B: An open large-scale dataset for training next generation image-text models)

附录（LAION-5B：面向下一代图文模型训练的开放大规模数据集）

Appendix A Datasheet for LAION-5B dataset

附录 A LAION-5B 数据集说明书

A.1 Motivation

A.1 动机

Q1 For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description. • LAION-5B was created as an open solution to training very large multimodal models such as CLIP or DALL-E. Before the curation of this dataset, the closest in size was YFCC with 100 million image/videos and associated metadata. OpenAI previously used a 15 million sample subset to train a publicly comparable CLIP model, but that pales in comparison to the private 400 million sample dataset they used to train the high-performant CLIP models. At the time of writing this, the ImageNet-1k zero-shot top-1 state-of-the-art, Google’s BASIC, used a dataset of 6.6 billion image-text pairs. With the release of LAION-5B, researchers no longer have to be part of a few selected institutions to study these problems.

Q1 该数据集是为了什么目的创建的 ？ 是否针对某一特定任务？是否要填补某一特定空白？请加以说明。• LAION-5B 是作为训练 CLIP 或 DALL-E 这类超大多模态模型的开放方案而创建的。在整理本数据集之前，规模最接近的是包含 1 亿张图像/视频及其元数据的 YFCC。OpenAI 此前用 1500 万条样本的子集训练过一个可公开比较的 CLIP 模型，但这远小于他们用来训练高性能 CLIP 的 4 亿条私有数据。在撰写本文时，ImageNet-1k 零样本 top-1 的最好成绩来自 Google 的 BASIC，所用数据集有 66 亿图文对。有了 LAION-5B，研究者不必再属于少数机构，也能研究这些问题。

Q2 Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? • This dataset is presented by LAION (Large-scale Artificial Intelligence Open Network), a non-profit research organization aiming to democratize access to large-scale open datasets and powerful machine learning models through the research and development of open-source resources. The communication and organization of this project took place on the open LAION discord server [^20].

Q2 谁创建了该数据集（例如哪个团队、研究组），又是代表哪个实体（例如公司、机构、组织） ？ • 本数据集由 LAION（Large-scale Artificial Intelligence Open Network）发布。这是一个非营利研究组织，旨在通过研发开源资源，让大规模开放数据集和强大的机器学习模型更容易被获取。本项目的交流与组织在开放的 LAION Discord 服务器上进行[^20]。

Q3 Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number. • This work was sponsored by Hugging Face and Stability AI.

Q3 谁资助了该数据集的创建 ？ 如有相关资助，请提供资助方名称以及资助项目的名称和编号。• 本工作由 Hugging Face 和 Stability AI 赞助。

Q4 Any other comments? • No. A.2 Composition

Q4 还有其他补充吗 ？ • 没有。A.2 构成

Q5 What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description. • We provide 5.8 billion image-text pairs. Each pair consists of the following: an image file url; text caption; width; height; the caption’s language; cosine similarity (CLIP ViT/B-32 for English and MCLIP for multiple and unknown languages); the probability of the image containing a watermark; the probability of a sample being NSFW. We made our models openly available on the LAION github page (https://github.com/LAION-AI/LAION-5B-WatermarkDetection, https://github.com/LAION-AI/CLIP-based-NSFW-Detector).

Q5 组成该数据集的实例代表什么（例如文档、照片、人物、国家） ？ 是否存在多种类型的实例（例如电影、用户及其评分；人物及其互动；节点与边）？请加以说明。 • 我们提供 58 亿条图文对。每一对包含：图像文件 URL；文本说明；宽度；高度；说明文字的语言；余弦相似度（英语使用 CLIP ViT/B-32，多语言和未知语言使用 MCLIP）；图像含水印的概率；样本属于 NSFW 的概率。相关模型已在 LAION 的 GitHub 页面公开（ https://github.com/LAION-AI/LAION-5B-WatermarkDetection，https://github.com/LAION-AI/CLIP-based-NSFW-Detector）。

Q6 How many instances are there in total (of each type, if appropriate)? • LAION-5B contains 2.3 billion English samples, 2.2 billion multilingual samples, and 1.2 billion unknown language samples. A further overview of the statistics may be seen in the announcement blog post .

Q6 总共有多少个实例（如适用，请按类型分别说明） ？ • LAION-5B 包含 23 亿条英语样本、22 亿条多语言样本，以及 12 亿条未知语言样本。更完整的统计可参见发布博客。

Q7 Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable). • Common Crawl is a public repository of crawled web pages. From this collection of web pages we filter the images and alt-text to derive LAION-5B. Of the existing 50+ billion images available in common crawl. We provide image url and alt-text pairings of only 5.8 billion images.

Q7 该数据集是否包含全部可能的实例，或者只是从更大集合中抽出的样本（不一定是随机的） ？ 如果是样本，更大的集合是什么？该样本是否能代表更大的集合（例如地理覆盖）？如果能，请说明这种代表性是如何验证的。如果不能，请说明原因（例如为了覆盖更多样的实例，或因为有些实例被保留或无法获得）。 • Common Crawl 是公开的网页抓取仓库。我们从这些网页中过滤图像和 alt-text，得到 LAION-5B。Common Crawl 中现有的图像超过 500 亿张，我们只提供其中 58 亿张图像的 URL 与 alt-text 配对。

Q8 What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features? In either case, please provide a description. • We provide raw urls and their associated alt-text.

Q8 每个实例由什么数据组成 ？ 是“原始”数据（例如未处理的文本或图像）还是特征？无论哪种情形，请加以说明。 • 我们提供原始 URL 及其对应的 alt-text。

Q9 Is there a label or target associated with each instance? If so, please provide a description. • There is no hard class label, but researchers will often formulate a mapping of the text to image or vice-versa.

Q9 每个实例是否带有标签或目标 ？ 如果有，请加以说明。 • 没有硬性的类别标签，但研究者常常会把文本映射到图像，或把图像映射到文本。

Q10 Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text. • No.

Q10 单个实例是否缺失某些信息 ？ 如果是，请说明缺失的原因（例如当时无法获得）。这里不包括有意删除的信息，但可能包括例如经过遮盖的文本。 • 没有。

Q11 Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit. • No.

Q11 各个实例之间的关系是否被明确标出（例如用户对电影的评分、社交网络链接） ？ 如果是，请说明这些关系是如何被明确标出的。 • 没有。

Q12 Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them. • No.

Q12 是否有推荐的数据划分（例如训练、开发/验证、测试） ？ 如果有，请说明这些划分及其理由。 • 没有。

Q13 Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description. • There exist near duplicate images which makes possible a many to one embedding in certain scenarios. CLIP embeddings may be used to remove more or less of them.

Q13 数据集中是否存在错误、噪声来源或冗余 ？ 如果有，请加以说明。 • 存在近似重复的图像，这在某些情形下会造成多对一的嵌入。可以用 CLIP 嵌入去掉或多或少的这类重复。

Q14 Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate. • This dataset is reliant on links to the World Wide Web. As such, we are unable to offer any guarantees of the existence of these samples. Due to the size we will also not be able to offer archives of the current state either. In order to rapidly and efficiently download images from URLs, we provide img2dataset. Depending on bandwidth, it’s feasible to download the entire LAION-5B dataset in 7 days using 10 nodes.

Q14 该数据集是自包含的，还是链接到或依赖于外部资源（例如网站、推文、其他数据集） ？ 如果链接到或依赖于外部资源：a) 是否能保证这些资源长期存在且保持不变；b) 是否存在完整数据集的官方存档版本（即包含创建时那些外部资源的当时状态）；c) 这些外部资源是否带有可能适用于未来使用者的限制（例如许可、费用）？请说明所有外部资源及其限制，并在适当时提供链接或其他访问入口。 • 本数据集依赖指向万维网的链接。因此，我们无法保证这些样本会一直存在。由于规模太大，我们也无法提供当前状态的存档。为了能快速、高效地从 URL 下载图像，我们提供 img2dataset。按带宽情况，使用 10 个节点大约 7 天即可下载完整的 LAION-5B。

Q15 Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals’ non-public communications)? If so, please provide a description. • This dataset was collected using openly available parts of the internet with the assumption that any data found was intended to be shared freely. However, it is possible that the parties crawled by Common Crawl may have publicly hosted confidential data.

Q15 数据集是否包含可能被视为机密的数据（例如受法律特权或医患保密保护的数据，或包含个人非公开通信内容的数据） ？ 如果是，请加以说明。 • 本数据集采集自互联网上公开可访问的部分，并假定所发现的数据本意是供自由分享的。不过，被 Common Crawl 抓取的站点仍有可能公开托管过机密数据。

Q16 Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why. • Since the dataset is scraped from Common Crawl, it is known to have instances of sexually explicit, racist, abusive or other discomforting or disturbing content. We choose to include these samples for the usage of safety researchers and further dataset curation surrounding these sensitive topics. • To address the existence of distressing content, we provide safety tags. Details on tagging potentially inappropriate content can be found in Sec. 3.2 in the main text and Appendix Sec. C.5 and Sec. C.6. During down-stream training tasks, users may check the sample’s boolean flags to determine whether or not the sample should be used. However, as we described in the main text, it is important to note that the safety tags are not perfect, especially keeping the complexity of these tasks and the diverse opinions of different cultures in mind. Therefore, we advocate using these tags responsibly, not relying on them to create a truly safe, “production-ready” subset after removing all potentially problematic samples.

Q16 数据集是否包含若直接查看可能令人反感、侮辱、威胁，或可能引起焦虑的数据 ？ 如果是，请说明原因。 • 由于数据抓取自 Common Crawl，其中已知存在色情露骨、种族主义、辱骂性或其他令人不适的内容。我们选择保留这些样本，供安全研究者使用，也便于围绕这些敏感主题继续整理数据。• 针对令人不安的内容，我们提供安全标签。如何为可能不宜的内容打标，见正文第 3.2 节以及附录 C.5 和 C.6。在下游训练任务中，使用者可以查看样本的布尔标记，以决定是否使用该样本。不过，正如正文所述，这些安全标签并不完美，尤其要考虑到这些任务本身的复杂性，以及不同文化中意见的多样性。因此，我们主张负责任地使用这些标签，不要以为删掉所有可能有问题的样本之后，就能得到真正安全、可直接用于生产的子集。

Q17 Does the dataset relate to people? If not, you may skip the remaining questions in this section. • People may be present in the images or textual descriptions, but people are not the sole focus of the dataset.

Q17 该数据集是否与人有关 ？ 如果无关，本节其余问题可以跳过。 • 图像或文本描述中可能出现人物，但人物并不是该数据集的唯一焦点。

Q18 Does the dataset identify any subpopulations (e.g., by age, gender)? • We do not provide any markers of subpopulation as attributes of the image-text pairs, but it may be possible to deduce this in some cases from the image and language pairing.

Q18 该数据集是否识别任何子群体（例如按年龄、性别） ？ • 我们并不把子群体标记作为图文对的属性提供，但在某些情形下，有可能从图像与语言的配对中推断出来。

Q19 Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how. • Yes it may be possible to identify people using face recognition. We do not provide any such means nor make attempts, but institutions owning large amounts of face identifiers may identify specific people in the dataset. Similarly, people may be identified through the associated text.

Q19 是否有可能从该数据集中直接或间接（即与其他数据结合）识别出个人（即一个或多个自然人） ？ 如果是，请说明如何识别。 • 是的，有可能通过人脸识别识别出人。我们不提供这类手段，也不做此类尝试，但拥有大量人脸标识的机构仍可能在数据集中认出特定的人。类似地，也可能通过对应文本识别出人。

Q20 Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description. • Yes the dataset contains sensitive content. Although, the dataset wasn’t created with the intention of obtaining samples fitting this criteria, it is possible that individuals might have hosted such items on a website that had been crawled by Common Crawl.

Q20 数据集是否包含在任何意义上可能被视为敏感的数据（例如揭示种族或民族出身、性取向、宗教信仰、政治观点或工会成员身份或地点的数据；财务或健康数据；生物特征或遗传数据；社会保障号等政府身份证明；犯罪记录） ？ 如果是，请加以说明。 • 是的，数据集包含敏感内容。尽管创建该数据集时并无意专门获取符合这些标准的样本，但个人仍有可能把这类内容托管在后来被 Common Crawl 抓取的网站上。

Q21 Any other comments? • We caution discretion on behalf of the user and call for responsible usage of the dataset for research purposes only . A.3 Collection Process

Q21 还有其他补充吗 ？ • 我们提醒使用者自行审慎判断，并呼吁负责任地、 仅 将该数据集用于研究目的。A.3 采集过程

Q22 How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how. • From the aforementioned Common Crawl, we filter images and their associated alt-text. Inclusion is determined by cosine similarity of the alt-text and the image as determined by OpenAI’s CLIP ViT-B/32 for english samples and MCLIP for all other samples. We include English samples with a cosine similarity score above 0.28, and we select all multilingual and unknown language samples with a 0.26 cosine similarity score or greater.

Q22 每个实例相关的数据是如何获得的 ？ 数据是直接可观察的（例如原始文本、电影评分），由被试报告的（例如问卷回答），还是从其他数据中间接推断/导出的（例如词性标签、基于模型的年龄或语言猜测）？如果数据由被试报告或间接推断/导出，是否经过验证？如果是，请说明如何验证。 • 我们从前述 Common Crawl 中过滤图像及其对应的 alt-text。是否纳入由 alt-text 与图像的余弦相似度决定：英语样本使用 OpenAI 的 CLIP ViT-B/32，其他样本使用 MCLIP。英语样本余弦相似度高于 0.28 才纳入；多语言和未知语言样本则选取余弦相似度为 0.26 及以上者。

Q23 What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated? • We ran a preprocessing script in python, over hundred of small CPU nodes, and few GPU nodes. They were validated by manual inspection of the results and post processing on them: computation of statistics on the width, height, size of captions, clip embeddings and indices

Q23 采集数据使用了哪些机制或程序（例如硬件装置或传感器、人工整理、软件程序、软件 API） ？ 这些机制或程序是如何验证的？ • 我们用 Python 预处理脚本，在上百个小型 CPU 节点和少量 GPU 节点上运行。验证方式是人工检查结果，并对宽度、高度、说明文字长度、CLIP 嵌入和索引做后处理统计。

Q24 If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? • The dataset was obtained by openAI CLIP ViT B/32 filtering of Common Crawl links using cosine similarity of the image and its text the links were referring to.

Q24 如果该数据集是从更大集合中抽出的样本，采样策略是什么（例如确定性采样，或带有特定采样概率的概率采样） ？ • 该数据集通过对 Common Crawl 链接做 OpenAI CLIP ViT-B/32 过滤得到，依据是链接所指向的图像与文本之间的余弦相似度。

Q25 Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? • No crowdworkers were used in the curation of the dataset. Open-source researchers and developers enabled its creation for no payment.

Q25 谁参与了数据采集过程（例如学生、众包工作者、承包商），他们如何获得报酬（例如众包工作者的报酬是多少） ？ • 整理该数据集时没有使用众包工作者。开源研究者和开发者无偿促成了它的创建。

Q26 Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created. • The data was filtered from September 2021 to January 2022, but those who created the sites might have included content from before then. It is impossible to know for certain how far back the data stretches.

Q26 数据采集覆盖了哪一段时间？这段时间是否与实例相关数据的创建时间一致（例如对旧新闻报道的近期抓取） ？ 如果不是，请说明实例相关数据是在哪段时间创建的。 • 过滤工作完成于 2021 年 9 月至 2022 年 1 月，但创建这些网站的人可能纳入了更早的内容。无法确切知道数据可以追溯到多早。

Q27 Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation. • We corresponded with the University of Washington’s Human Subject Division, and as we do not intervene with the people depicted in the data as well as the data being public, they stated that the work did not require IRB review. Furthermore, the NeurIPS ethics review determined that the work has no serious ethical issues.

Q27 是否开展过伦理审查（例如由机构审查委员会进行） ？ 如果是，请说明这些审查过程，包括结果，并提供任何支持文件的链接或其他访问入口。 • 我们与华盛顿大学人类受试者部门进行了沟通。由于我们并不干预数据中所描绘的人，且数据本身是公开的，对方表示这项工作不需要 IRB 审查。此外，NeurIPS 的伦理审查认定该工作没有严重的伦理问题。

Q28 Does the dataset relate to people? If not, you may skip the remaining questions in this section. • People may appear in the images and descriptions, although they are not the exclusive focus of the dataset.

Q28 该数据集是否与人有关 ？ 如果无关，本节其余问题可以跳过。 • 图像和描述中可能出现人物，尽管他们并不是该数据集的唯一焦点。

Q29 Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? • We retrieve the data from Common Crawl which contains almost all websites.

Q29 数据是直接从相关个人处采集的，还是通过第三方或其他来源（例如网站）获得的 ？ • 我们从几乎覆盖所有网站的 Common Crawl 中检索数据。

Q30 Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself. • Individuals were not notified about the data collection.

Q30 相关个人是否被告知数据采集一事 ？ 如果是，请说明（或用截图等信息展示）通知是如何提供的，并提供通知原文的链接或其他访问入口，或复述通知的准确措辞。 • 相关个人并未被告知数据采集一事。

Q31 Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented. • We follow Common Crawl’s practice of crawling the web and follow each site’s robots.txt file, thus users consent to their sites being crawled. However, those depicted in the photograph might not have given their consent to its upload.

Q31 相关个人是否同意采集并使用其数据 ？ 如果是，请说明（或用截图等信息展示）同意是如何请求并获得的，并提供个人所同意原文的链接或其他访问入口，或复述该原文。 • 我们遵循 Common Crawl 抓取网页的做法，并遵守各站点的 robots.txt，因此用户同意自己的站点被抓取。不过，照片中被拍摄的人未必同意过照片被上传。

Q32 If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate). • Users have a possibility to check for the presence of the links in our dataset leading to their data on public internet by using the search tool provided by LAION, accessible at https://knn5.laion.ai . If users wish to revoke their consent after finding sensitive data, they can contact the hosting party and request to delete the content from the underlying website - it will be automatically removed from LAION-5B since we distributed image-text pairs as URLs. Moreover, we provide a contact email contact@laion.ai and contact form https://laion.ai/dataset-requests/ to request removal of the links from the dataset. The actual content behind the links is out of our reach and will in that case remain accessible on the public internet for other crawlers.

Q33 Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation. • Birhane, Prabhu, and Kahembwe opened the discussion on the limitations and imminent biases that come with the creation of a weakly-curated dataset using CLIP. CLIP and its usage of cosine similarity offers a useful but imperfect heuristic for dataset inclusion that inherits various biases contained in the image-text pairs crawled from the web. In addition, the biases already existent within CLIP and the World Wide Web may become amplified when distilling original raw data and forming a filtered dataset. Using a model trained on this dataset without any further curation in production has the potential to reinforce harmful simplistic stereotypes against already marginalized communities. • However, the authors also note that this dataset posits currently the only openly available solution for studying multimodal models of this scale, examining their potential benefits and harms. Combining the aforementioned limitations and opportunities that this dataset provides, we agree with the authors and authorize the dataset for purely academic endeavors and strongly advice against any usage in end products.

Q33 是否分析过该数据集及其使用对数据主体的潜在影响（例如数据保护影响分析） ？ 如果是，请说明这一分析，包括结果，并提供任何支持文件的链接或其他访问入口。 • Birhane、Prabhu 和 Kahembwe 开启了关于用 CLIP 创建弱整理数据集时，其局限性和即将会出现的偏见的讨论。CLIP 及其对余弦相似度的使用，为是否纳入数据提供了一种有用但不完美的启发式规则，它会继承从网页抓取的图文对中所含的各种偏见。此外，CLIP 和万维网中本已存在的偏见，在从原始数据提炼并形成过滤后的数据集时，还可能被放大。若不加进一步整理就把在该数据上训练的模型用于生产，有可能强化针对本已处于边缘地位的社群的有害、简化刻板印象。• 不过，作者也指出，该数据集目前是公开研究如此规模多模态模型、考察其潜在益处与危害的唯一方案。综合上述局限与机会，我们同意这些作者的看法，授权该数据集仅用于纯粹的学术探索，并强烈建议不要把它用在面向最终用户的产品中。

Q34 Any other comments? • No. A.4 Preprocessing, Cleaning, and/or Labeling

Q34 还有其他补充吗 ？ • 没有。A.4 预处理、清洗和/或标注

Q35 Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section. • No preprocessing or labelling is done. Certain images were removed on the basis of safety, and others are tagged in the presence of NSFW content or a watermark.

Q35 是否对数据做过任何预处理/清洗/标注（例如离散化或分箱、分词、词性标注、SIFT 特征提取、删除实例、处理缺失值） ？ 如果是，请加以说明。如果没有，本节其余问题可以跳过。 • 没有做预处理或标注。部分图像因安全原因被移除，另一些则在存在 NSFW 内容或水印时被打上标签。

Q36 Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data. • We do not save the raw data.

Q36 除了经过预处理/清洗/标注的数据外，是否还保存了“原始”数据（例如以支持未预料到的未来用途） ？ 如果是，请提供“原始”数据的链接或其他访问入口。 • 我们不保存原始数据。

Q37 Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point. • To preprocess the data we used: - https://github.com/rvencu/crawlingathome-gpu-hcloud process common crawl into a laion5B-like dataset - http://github.com/rom1504/img2dataset A tool to easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine. - https://github.com/rom1504/clip-retrieval a tool to easily compute clip embeddings and build a clip retrieval system with them • For individuals to preprocess the data for training, we provide: - https://github.com/rom1504/laion-prepro

Q37 用于对实例做预处理/清洗/标注的软件是否可获得 ？ 如果是，请提供链接或其他访问入口。 • 预处理数据时我们使用了： - https://github.com/rvencu/crawlingathome-gpu-hcloud，把 Common Crawl 处理成类似 LAION-5B 的数据集 - http://github.com/rom1504/img2dataset，一种可以方便地把大量图像 URL 变成图像数据集的工具。可在一台机器上于 20 小时内下载、缩放并打包 1 亿个 URL。 - https://github.com/rom1504/clip-retrieval，一种可以方便地计算 CLIP 嵌入并据此构建 CLIP 检索系统的工具 • 供个人预处理训练数据，我们还提供： - https://github.com/rom1504/laion-prepro

Q38 Any other comments? • No. A.5 Uses

Q38 还有其他补充吗 ？ • 没有。A.5 用途

Q39 Has the dataset been used for any tasks already? If so, please provide a description. • LAION-5B (and the associated LAION-400M) has been used on a number of tasks such as CLIP Reproduction, BLIP Training, Glide Training, Cloob Training, and sub-dataset generation. For example, Gu et al. used LAION-400M to train VQ diffusion text-to-image generation models. Additionally, Rombach et al. applied a subset of LAION-400M in training Latent Diffusion Models that achieved state-of-the-art results on image inpainting and class-conditional image synthesis. The team behind open_CLIP demonstrated the capabilities of the 400M subset for CLIP reproduction, achieving performance on par with that of OpenAI. On the matter of subset generation and CLIP reproduction, Zheng et al. utilized LAION for facial representation learning. It should be noted that this example demonstrates the potential for users to misuse this dataset for the purpose of identification. Li et al. applied a subset of LAION for the purpose of image-captioning. Finally, Eichenberg et al. used a LAION subset for MAGMA, a model generating text “answers” for image-question pairs.

Q39 该数据集是否已经被用于某些任务 ？ 如果是，请加以说明。 • LAION-5B（以及相关的 LAION-400M）已被用于多项任务，例如复现 CLIP、训练 BLIP、训练 GLIDE、训练 CLOOB，以及生成子数据集。例如，Gu 等人用 LAION-400M 训练 VQ diffusion 文生图模型。Rombach 等人还把 LAION-400M 的一个子集用于训练潜扩散模型，在图像修复和类别条件图像合成上取得了当时最好的结果。open_CLIP 团队展示了 4 亿子集用于复现 CLIP 的能力，表现与 OpenAI 相当。在子数据集生成和 CLIP 复现方面，Zheng 等人把 LAION 用于面部表示学习。需要指出，这一例子也说明使用者可能把该数据集误用于身份识别。Li 等人把 LAION 的一个子集用于图像描述。最后，Eichenberg 等人用 LAION 的一个子集训练 MAGMA，该模型为图像—问题对生成文本“回答”。

Q40 Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point. • Yes, scientific publications and systems that use LAION datasets can be found on the LAION github page.

Q40 是否有仓库汇集了使用该数据集的部分或全部论文或系统 ？ 如果有，请提供链接或其他访问入口。 • 有，使用 LAION 数据集的科学出版物和系统可以在 LAION 的 GitHub 页面上找到。

Q41 What (other) tasks could the dataset be used for? • We encourage future researchers to curate LAION-5B for several tasks. Particularly, we see applications of the dataset in image and text representation learning, image to text generation, image captioning, and other common multimodal tasks. Due to the breadth of the data, it also offers a unique opportunity for safety and low resource language researchers. We hope for LAION-5B to serve under-represented projects as well.

Q41 该数据集还可以用于哪些（其他）任务 ？ • 我们鼓励未来的研究者为多项任务整理 LAION-5B。我们尤其看到它在图像与文本表示学习、图像到文本生成、图像描述以及其他常见多模态任务中的应用。由于数据覆盖面很广，它也为安全和低资源语言研究者提供了独特机会。我们希望 LAION-5B 也能服务于代表性不足的项目。

Q42 Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms? • As this data stems from the greater internet, it mirrors the broader biases of society in the period of its collection. Biases in subpopulation depiction (eg. correlation between gender and jobs), violence, and nudity (for which we provide safety tags) might create harmful outcomes for those a model might be applied to. For this reason this dataset should not be used to make a decision surrounding people.

Q42 数据集的构成，或它被采集和预处理/清洗/标注的方式，是否有可能影响未来用途 ？ 例如，未来使用者是否需要知道哪些事项，以避免造成对个人或群体的不公平对待（例如刻板印象、服务质量问题）或其他不良伤害（例如经济伤害、法律风险）？如果是，请加以说明。未来使用者可以做些什么来减轻这些不良伤害？ • 由于这些数据来自更广阔的互联网，它反映了采集时期社会中更广泛的偏见。子群体描绘中的偏见（例如性别与职业之间的相关性）、暴力以及裸体（我们为此提供了安全标签），可能给模型所作用的人带来有害结果。因此，该数据集不应用于围绕人做出决策。

Q43 Are there tasks for which the dataset should not be used? If so, please provide a description. • Due to the known biases of the dataset, under no circumstance should any models be put into production using the dataset as is. It is neither safe nor responsible. As it stands, the dataset should be solely used for research purposes in its uncurated state. • Likewise, this dataset should not be used to aid in military or surveillance tasks.

Q43 是否有不应当使用该数据集的任务 ？ 如果有，请加以说明。 • 鉴于该数据集已知存在偏见，在任何情形下都不应当把直接使用该数据集训练的模型投入生产。这样做既不安全，也不负责任。就目前未加整理的状态而言，该数据集应当只用于研究。• 同样，该数据集不应用于协助军事或监视任务。

Q44 Any other comments? • No. A.6 Distribution

Q44 还有其他补充吗 ？ • 没有。A.6 分发

Q45 Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description. • Yes, the dataset will be open-source.

Q45 该数据集是否会分发给创建该数据集所代表的实体（例如公司、机构、组织）之外的第三方 ？ 如果是，请加以说明。 • 会，该数据集将开源。

Q46 How will the dataset be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? • The data will be available through Huggingface datasets.

Q46 该数据集将如何分发（例如网站上的打包文件、API、GitHub） ？ 该数据集是否有数字对象标识符（DOI）？ • 数据将通过 Hugging Face datasets 提供。

Q47 When will the dataset be distributed? • 31/03/2022 and onward.

Q47 该数据集将于何时分发 ？ • 2022 年 3 月 31 日起持续提供。

Q48 Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions. • CC-BY-4.0

Q48 该数据集是否会在著作权或其他知识产权（IP）许可，和/或适用的使用条款（ToU）下分发 ？ 如果是，请说明该许可和/或使用条款，并提供相关许可条款或使用条款的链接或其他访问入口，或复述这些条款，以及与这些限制相关的任何费用。 • CC-BY-4.0

Q49 Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions. • LAION owns the metadata and release as CC-BY-4.0. • We do not own the copyright of the images or text.

Q49 是否有第三方对实例相关数据施加了基于知识产权或其他方面的限制 ？ 如果有，请说明这些限制，并提供相关许可条款的链接或其他访问入口，或复述这些条款，以及与这些限制相关的任何费用。 • 元数据归 LAION 所有，并以 CC-BY-4.0 发布。• 我们并不拥有图像或文本的著作权。

Q50 Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation. • No.

Q50 是否有任何出口管制或其他监管限制适用于该数据集或单个实例 ？ 如果有，请说明这些限制，并提供任何支持文件的链接或其他访问入口，或复述这些文件。 • 没有。

Q51 Any other comments? • No. A.7 Maintenance

Q51 还有其他补充吗 ？ • 没有。A.7 维护

Q52 Who will be supporting/hosting/maintaining the dataset? • Huggingface will support hosting of the metadata. • The Eye supports hosting of the embeddings and backups of the rest. • LAION will maintain the samples distributed.

Q52 谁将支持/托管/维护该数据集 ？ • Hugging Face 将支持元数据的托管。• The Eye 支持嵌入的托管以及其他内容的备份。• LAION 将维护所分发的样本。

Q53 How can the owner/curator/manager of the dataset be contacted (e.g., email address)? • https://laion.ai/dataset-requests/

Q53 如何联系该数据集的所有者/整理者/管理者（例如电子邮件地址） ？ • https://laion.ai/dataset-requests/

Q54 Is there an erratum? If so, please provide a link or other access point. • There is no erratum for our initial release. Errata will be documented as future releases on the dataset website.

Q54 是否有勘误表 ？ 如果有，请提供链接或其他访问入口。 • 首次发布没有勘误表。勘误将作为未来版本记录在数据集网站上。

Q55 Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)? • LAION-5B will not be updated. However a future LAION-streamed-from-CC may exist for updates. Specific samples can be removed on request.

Q55 该数据集是否会更新（例如纠正标注错误、增加新实例、删除实例） ？ 如果会，请说明更新频率、由谁更新，以及如何向使用者传达更新（例如邮件列表、GitHub）。 • LAION-5B 不会更新。不过将来可能会有从 Common Crawl 流式更新的 LAION 版本。特定样本可以根据请求移除。

Q56 If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced. • People may contact us at the LAION website to add specific samples to a blacklist.

Q56 如果该数据集与人有关，对实例相关数据的保留是否有适用的时限（例如，相关个人是否被告知其数据会保留固定时间然后删除） ？ 如果有，请说明这些时限以及将如何执行。 • 人们可以通过 LAION 网站联系我们，把特定样本加入黑名单。

Q57 Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to users. • We will continue to support LAION-400M.

Q57 较旧版本的数据集是否会继续得到支持/托管/维护 ？ 如果会，请说明如何进行。如果不会，请说明将如何向使用者传达其不再维护。 • 我们将继续支持 LAION-400M。

Q58 If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description. • Unless there are grounds for significant alteration to certain indexes, extension of the dataset will be carried out on an individual basis.

Q58 如果其他人希望扩展/增强/基于该数据集开展工作/做出贡献，是否有相应机制 ？ 如果有，请加以说明。这些贡献会经过验证吗？如果会，请说明如何验证；如果不会，请说明原因。是否有流程把这些贡献传达/分发给其他使用者？如果有，请加以说明。 • 除非有充分理由对某些索引做重大改动，对该数据集的扩展将按个案进行。

Q59 Any other comments? • No.

Q59 还有其他补充吗 ？ • 没有。

Appendix B Dataset Setup Procedure

附录 B 数据集准备步骤

After processing and filtering common crawl, 5B of image url/text samples are available. Here we provide an overview of all the steps necessary to combine the full dataset.

对 Common Crawl 进行处理和过滤之后，可以得到 50 亿条图像 URL/文本样本。这里概述把完整数据集组合起来所需的全部步骤。

Downloading the data as webdataset with distributed img2dataset

用分布式 img2dataset 把数据下载为 webdataset

Computing Vit-L/14 embeddings with distributed clip-inference

用分布式 clip-inference 计算 ViT-L/14 嵌入

Computing a KNN index from these embeddings using autofaiss

用 autofaiss 由这些嵌入计算 KNN 索引

Computing additional tags (NSFW and watermark) using CLIP embeddings

用 CLIP 嵌入计算额外标签（NSFW 和水印）

Appendix C Dataset Preparation and Curation Details

附录 C 数据集准备与整理细节

C.1 Distributed img2dataset

C.1 分布式 img2dataset

We developed img2dataset library to easily download, resize, and store images and captions in the webdataset format.[^21] This allows to download 100 million images from our list of URLs in 20 hours with a single node (1Gbps connection speed, 32GB of RAM, an i7 CPU with 16 cores), allowing anyone to obtain the whole dataset or a smaller subset.

我们开发了 img2dataset 库，以便方便地下载、缩放图像，并把图像与说明以 webdataset 格式存储。[^21] 这样，用单个节点（1Gbps 连接、32GB 内存、16 核 i7 CPU）即可在 20 小时内从我们的 URL 列表下载 1 亿张图像，从而使任何人都能获得完整数据集或较小子集。

For LAION-5B we introduced a distributed mode for this tool, allowing to download the 5B samples in a week using 10 nodes. see [^22] and [^23]

对于 LAION-5B，我们为该工具引入了分布式模式，用 10 个节点大约一周即可下载 50 亿条样本。见[^22]和[^23]。

C.2 Distributed CLIP inference

C.2 分布式 CLIP 推理

From these images, the CLIP retrieval inference tool [^24] was used to compute ViT-L/14 embeddings, allowing for a better analysis capacity of the data. In particular a distributed mode [^25] made it possible to compute these embeddings in a week using 32 NVIDIA A100s: this larger CLIP model can only be computed at a speed of 312 sample/s per gpu, compared to 1800 sample/s for ViT-B/32.

在这些图像之上，我们使用 CLIP retrieval 推理工具[^24]计算 ViT-L/14 嵌入，以便更好地分析数据。其中的分布式模式[^25]使得用 32 块 NVIDIA A100 在一周内完成这些嵌入的计算成为可能：这一更大的 CLIP 模型每块 GPU 只能以每秒 312 个样本的速度计算，而 ViT-B/32 可达每秒 1800 个样本。

The resulting embeddings are available for everyone to use for clustering, indexing, linear inference.

得到的嵌入对所有人开放，可用于聚类、建立索引和线性推理。

C.3 Distributed indexing

C.3 分布式索引

We then used these 9TB of image embeddings to build a large PQ128 knn index using the autofaiss tool [^26]. To make this run faster, a distributed mode is available [^27]

我们随后用这 9TB 图像嵌入，借助 autofaiss 工具构建了一个大型 PQ128 近邻索引[^26]。为加快运行，该工具提供了分布式模式[^27]。

C.4 Integration in the search UI

C.4 集成到检索界面

In order to demonstrate the value of this data, we integrated this index into the [^28] UI. It is powered by the code called clip back at [^29] The knn index is 800GB and the metadata (url and captions) as well, so memory mapping is used for both in order to use no RAM, only a SSD drive of that capacity is required.

为展示这些数据的价值，我们把该索引集成到了相应界面中[^28]。它由称为 clip back 的代码驱动[^29]。近邻索引有 800GB，元数据（URL 和说明）同样有 800GB，因此两者都使用内存映射，从而不占用内存，只需要同等容量的 SSD。

C.5 Specialized NSFW image content tagging

C.5 专门的 NSFW 图像内容标注

We applied various tagging to the content of LAION 5B. Among other contents, we tagged images with pornographic or sexualized content (referred to as NSFW). To ensure all implementations related to LAION-5B are open-source, we refrained from using existing commercial solutions.

我们对 LAION-5B 的内容做了多种标注。其中，我们为色情或性化内容的图像打上标签（称为 NSFW）。为确保与 LAION-5B 相关的所有实现都开源，我们没有使用已有的商业方案。

In particular, we first trained an EfficientNetV2-based classifier. However, then moved to a simple MLP based on OpenAI’s CLIP/L-14. To this end, we created a training dataset by retrieving images from the previous LAION-400M dataset which are close in the CLIP embedding space to various keywords related to the five categories: ‘‘neutral", ‘‘drawing’’, ‘‘porn’’, ‘‘hentai’’ or ‘‘sexy’’. Additionally, we added SFW images from the Wikiart [^30] and Danbooru datasets [^31] to the “drawing” category and NSFW images from Danbooru to the “hentai” category.

我们起初训练了一个基于 EfficientNetV2 的分类器，随后改用基于 OpenAI CLIP/L-14 的简单多层感知机。为此，我们从先前的 LAION-400M 中检索在 CLIP 嵌入空间里接近各类关键词的图像，这些关键词对应五个类别：“neutral”、“drawing”、“porn”、“hentai”或“sexy”。此外，我们把来自 Wikiart[^30]和 Danbooru[^31]的安全图像加入“drawing”类，并把来自 Danbooru 的 NSFW 图像加入“hentai”类。

Following this procedure, we obtained over 682K images from the five classes ‘‘drawing’’ (39026), ‘‘hentai’’ (28134), ‘‘neutral’’ (369507), ‘‘porn’’ (207969) and ‘‘sexy’’ (37914). Using this data we trained a detector for these five categories by finetuning an ImageNet-1k pretrained EfficientNet-V2-B02 model. [^32] To use this image classifier as a binary SFW - NSFW classifier, we consider images from the classes “drawing” and “neutral” as SFW and “hentai”, “porn” and “sexy” as NSFW. To measure the performance of this model, we created a test dataset with 1000 images from each category and manually inspected it, to make sure all test images where correctly annotated. Our EfficientNet-V2-B02 image classifier predicted 96,45% of the true NSFW correctly as NSFW and discards 7,96% of the SFW images incorrectly as NSFW.

按这一流程，我们从五个类别中获得了超过 68.2 万张图像：“drawing”（39026）、“hentai”（28134）、“neutral”（369507）、“porn”（207969）和“sexy”（37914）。我们用这些数据，通过微调在 ImageNet-1k 上预训练的 EfficientNet-V2-B02 模型，训练了针对这五个类别的检测器。[^32] 要把该图像分类器当作二分类的安全/不宜（SFW-NSFW）分类器使用时，我们将“drawing”和“neutral”视为 SFW，将“hentai”、“porn”和“sexy”视为 NSFW。为衡量该模型的表现，我们为每个类别建立了包含 1000 张图像的测试集，并人工检查以确保所有测试图像标注正确。我们的 EfficientNet-V2-B02 图像分类器把 96.45% 真正的 NSFW 正确预测为 NSFW，并把 7.96% 的 SFW 图像错误地判为 NSFW。

C.6 Further inappropriate content tagging

C.6 对其他不宜内容的标注

Further, we used the Q16 documentation pipeline [68] to document the broad range of identified potentially inappropriate concepts contained, cf. Sec. 3.2 for details. Fig. 6 shows the most frequent identified concepts following this procedure. One can see that in a lot of cases these images show humans (cf. concepts human, people, man, woman). Further, one main concept is pornographic content (e.g. porn, bondage, kinky, bdsm). Additionally, most frequent present concepts are, among other concepts, weapons, violence, terror, murder, slavery, racism and hate . Note that also content surrounding halloween (costume, halloween, zombie) and art or media such as movie s, game s and comic s are potentially tagged, depending on the displayed content. Further filtering depends highly on the use-case and users’ opinions.

此外，我们使用 Q16 记录流程[68]来记录所识别出的、范围较广的潜在不宜概念，细节见第 3.2 节。图 6 展示了按该流程识别出的最常见概念。可以看到，这些图像在很多情况下描绘的是人（参见 human、people、man、woman 等概念）。此外，一个主要概念是色情内容（例如 porn、bondage、kinky、bdsm ）。另外，最常出现的概念还包括 weapons、violence、terror、murder、slavery、racism 和 hate 等。需要注意的是，与 halloween 相关的内容（ costume、halloween、zombie ）以及 movie 、 game 和 comic 等艺术或媒体内容，也可能因所展示的画面而被打上标签。进一步过滤在很大程度上取决于具体用途和使用者的判断。

Figure 6: Word cloud based on [68] documenting the potentially inappropriate image content of the LAION-5B subset which contains text in English language. Provided alternative text is used as text description of the images. Word size is proportional to the word counts and rank in descriptions corresponding to the inappropriate image set.

图 6：基于[68]绘制的词云，用于记录 LAION-5B 英语文本子集中潜在不宜的图像内容。所提供的替代文本被用作这些图像的文字描述。词的大小与不宜图像集合相应描述中的词频和排序成正比。

C.7 Watermark and safety inference

C.7 水印与安全推理

Finally, we wanted to let user the ability to remove unsafe examples, and watermarked examples. To do that we collected training and test sets. The training set was augmented with examples retrieved from the KNN index, while the test set samples were selected to represent well the dataset distribution but were all manually annotated. 7

最后，我们希望让使用者能够去掉不安全的样本以及带水印的样本。为此我们收集了训练集和测试集。训练集用从近邻索引中检索到的样本做了增强，测试集样本则被选来较好地代表数据分布，并全部经过人工标注。

(a)

(a)

The inference is done using the embedding-reader[^33] module.

推理使用 embedding-reader[^33] 模块完成。

These tags were then integrated in the UI, allowing everyone to observe that the safety tags indeed filter out almost all the unsafe results, and giving confidence that training a generative model on this data will not result in unexpectedly unsafe images.

这些标签随后被集成到界面中，使所有人都能看到安全标签的确滤掉了几乎全部不安全结果，从而更有信心：在这些数据上训练生成模型，不会意外产生大量不安全图像。

Appendix D Dataset Samples and Statistics

附录 D 数据集样本与统计

Here, we present samples from the dataset and some distribution statistics to aid in understanding the dataset. In Figure 8, we randomly select 4 samples from each of the 3 LAION-5B subsets. As can be seen, the language classifier seems to have low confidence with names, identifying numbers, and short form text. An important future line of work will be to improve the language classifier.

这里展示数据集中的一些样本和分布统计，以帮助理解该数据集。在图 8 中，我们从 LAION-5B 的 3 个子集里各随机选取 4 条样本。可以看到，语言分类器对人名、识别编号和短文本的置信度似乎较低。改进语言分类器将是未来一项重要工作。

Figure 8: LAION-5B random examples from all subsets. We take the first 4 SFW samples from each of the 3 randomly shuffled LAION-5B subsets. We present the image and its associated caption.

图 8： 来自各个子集的 LAION-5B 随机样本 。 我们从 3 个经过随机打乱的 LAION-5B 子集中，各取前 4 条安全（SFW）样本，并展示图像及其对应说明。

Figure 10: Caption Character Length. Each of the LAION-5B subsets contains similar frequencies and exhibit a right skew. Figure 12: Multilingual Language Frequency. The 10 most frequent languages seem to be largely of European and East Asian origin. Figure 10: Caption Character Length. Each of the LAION-5B subsets contains similar frequencies and exhibit a right skew.

图 10： 说明文字的字符长度 。 LAION-5B 各个子集的频率分布相近，并呈右偏。图 12： 多语言的语言频率 。 出现最多的 10 种语言似乎主要以欧洲和东亚语言为主。

To comprehend the dataset beyond visual examples, we may look at statistics collected about the distribution. Figure 12 gives an overview of the caption length amongst all subsets. Additionally, Figure 12 describes the frequency of languages within the multilingual subset. The 10 most frequent languages compose 56% of the multilingual dataset.

除了视觉样本，我们还可以查看关于分布的统计，以便进一步理解该数据集。图 12 给出了各子集说明文字长度的概览。此外，图 12 描述了多语言子集中的语言频率。出现最多的 10 种语言占多语言数据集的 56%。

Appendix E Further Experimental Details and Results on CLIP reproduction

附录 E 复现 CLIP 的更多实验细节与结果

We provide details about experiments that were done to reproduce CLIP [58] using LAION (400M, 2B-en) subsets. In addition, we document all experimental results on both zero-shot classification using the VTAB+ suite and retrieval.

我们给出用 LAION（4 亿、20 亿英语）子集复现 CLIP[58]的实验细节，并记录 VTAB+ 套件上的全部零样本分类结果以及检索结果。

E.1 Training Details

E.1 训练细节

\beta_{1}=0.9

\beta_{1}=0.9

E.2 Distributed Training and InfoNCE Loss

E.2 分布式训练与 InfoNCE 损失

To properly deal with global batch for contrastive InfoNCE loss in distributed setting, we need additional communication between GPU workers to compute the loss and the gradients for all positive and negative sample pairs correctly. In each worker, we gather all image and text embeddings from the other workers, and use them as negative examples for each image-text pair in the mini-batch.

要在分布式设定下正确处理对比 InfoNCE 损失的全局批量，GPU 工作进程之间需要额外通信，才能正确计算所有正样本对和负样本对的损失与梯度。在每个工作进程中，我们汇集来自其他工作进程的全部图像嵌入和文本嵌入，并把它们当作小批量中每对图文的负例。

N\times N

N\times N

Name Width Embed Dim Depth Res. Acts. Params ViT-B/32 768 / 512 512 12 / 12 224x224 10 M 151 M ViT-B/16 768 / 512 512 12 / 12 224x224 29 M 150 M ViT-B/16+ 896 / 640 640 12 / 12 240x240 40 M 208 M ViT-L/14 1024 / 768 768 24 / 12 224x224 97 M 428 M Table 5: Hyper-parameters of different architectures we used for reproducing CLIP models. Acts refers to the number of activations in millions, while Params refers to the number of parameters in millions. All entries in the form of A / B denote image and text parameters respectively.

Name	Width	Embed Dim	Depth	Res.	Acts.	Params
ViT-B/32	768 / 512	512	12 / 12	224x224	10 M	151 M
ViT-B/16	768 / 512	512	12 / 12	224x224	29 M	150 M
ViT-B/16+	896 / 640	640	12 / 12	240x240	40 M	208 M
ViT-L/14	1024 / 768	768	24 / 12	224x224	97 M	428 M

名称 宽度 嵌入维 深度 分辨率 激活数 参数量 ViT-B/32 768 / 512 512 12 / 12 224x224 10 M 151 M ViT-B/16 768 / 512 512 12 / 12 224x224 29 M 150 M ViT-B/16+ 896 / 640 640 12 / 12 240x240 40 M 208 M ViT-L/14 1024 / 768 768 24 / 12 224x224 97 M 428 M 表 5：我们用于复现 CLIP 模型的不同结构超参数。 激活数 指以百万计的激活数量， 参数量 指以百万计的参数数量。形如 A / B 的条目分别表示图像侧和文本侧的参数。

名称	宽度	嵌入维	深度	分辨率	激活数	参数量
ViT-B/32	768 / 512	512	12 / 12	224x224	10 M	151 M
ViT-B/16	768 / 512	512	12 / 12	224x224	29 M	150 M
ViT-B/16+	896 / 640	640	12 / 12	240x240	40 M	208 M
ViT-L/14	1024 / 768	768	24 / 12	224x224	97 M	428 M

5\text{e-}4

Model (data size)	BS. (global)	#GPUs	LR.	Warm.	Ep.	Time (hrs.)
B/32 (400M)	256 (32768)	128	$5\text{e-}4$	2K	32	36
B/32 (2B)	416 (46592)	112	$5.5\text{e-}4$	10K	16	210
B/16 (400M)	192 (33792)	176	$5\text{e-}4$	2K	32	61
B/16+(400M)	160 (35840)	224	$7\text{e-}4$	5K	32	61
L/14 (400M)	96 (38400)	400	$6\text{e-}4$	5K	32	88

5\text{e-}4

模型（数据规模）	批量（全局）	GPU 数	学习率	预热	轮数	时间（小时）
B/32 (400M)	256 (32768)	128	$5\text{e-}4$	2K	32	36
B/32 (2B)	416 (46592)	112	$5.5\text{e-}4$	10K	16	210
B/16 (400M)	192 (33792)	176	$5\text{e-}4$	2K	32	61
B/16+(400M)	160 (35840)	224	$7\text{e-}4$	5K	32	61
L/14 (400M)	96 (38400)	400	$6\text{e-}4$	5K	32	88

E.3 Detailed Results & Further Analysis

E.3 详细结果与进一步分析

In this section we present all zero-shot classification results on VTAB+ as well as retrieval results. In Tab. 7, we describe the datasets that are used in VTAB+. For zero-shot classification, we collected prompts and class names from prior works [58, 94] and made them available in our benchmark repository[^34]. In Tab. 9, we show zero-shot top-1 classification accuracy (%) on VTAB+ datasets. Tables 10 and 11 depict retrieval results on Flickr30K[88] and MSCOCO [44].

本节给出 VTAB+ 上的全部零样本分类结果以及检索结果。表 7 说明了 VTAB+ 所用的数据集。对于零样本分类，我们从已有工作[58, 94]中收集提示和类别名，并把它们放在我们的基准仓库中[^34]。表 9 给出 VTAB+ 数据集上的零样本 top-1 分类准确率（%）。表 10 和表 11 展示 Flickr30K[88]和 MSCOCO[44]上的检索结果。

Dataset Abbr.(Tab. 4, 9) Test size #Classes ImageNet-1k INet 50,000 1,000 ImageNet-v2 INet-v2 10,000 1,000 ImageNet-R INet-R 30,000 200 ImageNet Sketch INet-S 50,889 1,000 ObjectNet ObjNet 18,574 113 ImageNet-A INet-A 7,500 200 CIFAR-10 - 10,000 10 CIFAR-100 - 10,000 100 MNIST - 10,000 10 Oxford Flowers 102 Flowers102 6,149 102 Stanford Cars Cars 8,041 196 SVHN - 26,032 10 Facial Emotion Recognition 2013 FER2013 7,178 7 RenderedSST2 - 1,821 2 Oxford-IIIT Pets Pets 3,669 37 Caltech-101 - 6,085 102 Pascal VOC 2007 Classification VOC2007-Cl 14,976 20 SUN397 - 108,754 397 FGVC Aircraft - 3,333 100 Country211 - 21,100 211 Describable Textures DTD 1,880 47 GTSRB - 12,630 43 STL10 - 8,000 10 Diabetic Retinopathy Retino 42,670 5 EuroSAT - 5,400 10 RESISC45 - 6,300 45 PatchCamelyon PCAM 32,768 2 CLEVR Counts - 15,000 8 CLEVR Object Distance CLEVR Dist 15,000 6 DSPRITES Orientation DSPRITES Orient 73,728 40 DSPRITES Position DSPRITES pos 73,728 32 SmallNORB Elevation SmallNORB Elv 12,150 9 SmallNORB Azimuth SmallNORB Azim 12,150 18 DMLAB - 22,735 6 KITTI closest vehicle distance KITTI Dist 7114 Table 7: Datasets used for zero-shot classification evaluation (VTAB+).

Dataset	Abbr.(Tab. 4, 9)	Test size	#Classes
ImageNet-1k	INet	50,000	1,000
ImageNet-v2	INet-v2	10,000	1,000
ImageNet-R	INet-R	30,000	200
ImageNet Sketch	INet-S	50,889	1,000
ObjectNet	ObjNet	18,574	113
ImageNet-A	INet-A	7,500	200
CIFAR-10	-	10,000	10
CIFAR-100	-	10,000	100
MNIST	-	10,000	10
Oxford Flowers 102	Flowers102	6,149	102
Stanford Cars	Cars	8,041	196
SVHN	-	26,032	10
Facial Emotion Recognition 2013	FER2013	7,178	7
RenderedSST2	-	1,821	2
Oxford-IIIT Pets	Pets	3,669	37
Caltech-101	-	6,085	102
Pascal VOC 2007 Classification	VOC2007-Cl	14,976	20
SUN397	-	108,754	397
FGVC Aircraft	-	3,333	100
Country211	-	21,100	211
Describable Textures	DTD	1,880	47
GTSRB	-	12,630	43
STL10	-	8,000	10
Diabetic Retinopathy	Retino	42,670	5
EuroSAT	-	5,400	10
RESISC45	-	6,300	45
PatchCamelyon	PCAM	32,768	2
CLEVR Counts	-	15,000	8
CLEVR Object Distance	CLEVR Dist	15,000	6
DSPRITES Orientation	DSPRITES Orient	73,728	40
DSPRITES Position	DSPRITES pos	73,728	32
SmallNORB Elevation	SmallNORB Elv	12,150	9
SmallNORB Azimuth	SmallNORB Azim	12,150	18
DMLAB	-	22,735	6
KITTI closest vehicle distance	KITTI Dist	711	4

数据集 缩写（表 4、表 9） 测试集规模 类别数 ImageNet-1k INet 50,000 1,000 ImageNet-v2 INet-v2 10,000 1,000 ImageNet-R INet-R 30,000 200 ImageNet Sketch INet-S 50,889 1,000 ObjectNet ObjNet 18,574 113 ImageNet-A INet-A 7,500 200 CIFAR-10 - 10,000 10 CIFAR-100 - 10,000 100 MNIST - 10,000 10 Oxford Flowers 102 Flowers102 6,149 102 Stanford Cars Cars 8,041 196 SVHN - 26,032 10 Facial Emotion Recognition 2013 FER2013 7,178 7 RenderedSST2 - 1,821 2 Oxford-IIIT Pets Pets 3,669 37 Caltech-101 - 6,085 102 Pascal VOC 2007 Classification VOC2007-Cl 14,976 20 SUN397 - 108,754 397 FGVC Aircraft - 3,333 100 Country211 - 21,100 211 Describable Textures DTD 1,880 47 GTSRB - 12,630 43 STL10 - 8,000 10 Diabetic Retinopathy Retino 42,670 5 EuroSAT - 5,400 10 RESISC45 - 6,300 45 PatchCamelyon PCAM 32,768 2 CLEVR Counts - 15,000 8 CLEVR Object Distance CLEVR Dist 15,000 6 DSPRITES Orientation DSPRITES Orient 73,728 40 DSPRITES Position DSPRITES pos 73,728 32 SmallNORB Elevation SmallNORB Elv 12,150 9 SmallNORB Azimuth SmallNORB Azim 12,150 18 DMLAB - 22,735 6 KITTI closest vehicle distance KITTI Dist 7114 表 7：用于零样本分类评估的数据集（VTAB+）。

数据集	缩写（表 4、表 9）	测试集规模	类别数
ImageNet-1k	INet	50,000	1,000
ImageNet-v2	INet-v2	10,000	1,000
ImageNet-R	INet-R	30,000	200
ImageNet Sketch	INet-S	50,889	1,000
ObjectNet	ObjNet	18,574	113
ImageNet-A	INet-A	7,500	200
CIFAR-10	-	10,000	10
CIFAR-100	-	10,000	100
MNIST	-	10,000	10
Oxford Flowers 102	Flowers102	6,149	102
Stanford Cars	Cars	8,041	196
SVHN	-	26,032	10
Facial Emotion Recognition 2013	FER2013	7,178	7
RenderedSST2	-	1,821	2
Oxford-IIIT Pets	Pets	3,669	37
Caltech-101	-	6,085	102
Pascal VOC 2007 Classification	VOC2007-Cl	14,976	20
SUN397	-	108,754	397
FGVC Aircraft	-	3,333	100
Country211	-	21,100	211
Describable Textures	DTD	1,880	47
GTSRB	-	12,630	43
STL10	-	8,000	10
Diabetic Retinopathy	Retino	42,670	5
EuroSAT	-	5,400	10
RESISC45	-	6,300	45
PatchCamelyon	PCAM	32,768	2
CLEVR Counts	-	15,000	8
CLEVR Object Distance	CLEVR Dist	15,000	6
DSPRITES Orientation	DSPRITES Orient	73,728	40
DSPRITES Position	DSPRITES pos	73,728	32
SmallNORB Elevation	SmallNORB Elv	12,150	9
SmallNORB Azimuth	SmallNORB Azim	12,150	18
DMLAB	-	22,735	6
KITTI closest vehicle distance	KITTI Dist	711	4

Effect of data scale. We observe similar or better results on most datasets when using the larger LAION-2B-en instead of LAION-400M. Exceptions are on some datasets with specialized domains (e.g., Diabetic Retinopathy, PatchCamelyon) or in structured tasks (see corresponding paragraph below). To demonstrate the importance of the data scale for the quality of the pre-trained models, we conduct a series of experiments where we vary both data scale (LAION-80M, LAION-400M and LAION-2B) and amount of training compute measured in samples seen (3B, 13B and 34B). We observe that when investing enough into training compute, seeing same number of samples on larger data scale leads consistently to better zero-shot transfer performance measured on ImageNet-1k. This is valid for both smaller B/32 and larger L/14 model scales. For instance, models pre-trained on LAION-2B outperform there significantly models pre-trained on LAION-400M, when using same large compute training budget of 34B samples seen (see Fig. 15 and Tab. 8). We conclude from these findings that extending dataset scale all the way up towards LAION-2B is indeed important for obtaining stronger zero-shot transfer performance, given sufficiently large compute for training.

数据规模的影响 改用更大的 LAION-2B-en 而不是 LAION-400M 时，我们在大多数数据集上观察到相近或更好的结果。例外出现在一些专门领域的数据集（例如 Diabetic Retinopathy、PatchCamelyon）或结构化任务上（见下一段）。为说明数据规模对预训练模型质量的重要性，我们做了一系列实验，同时改变数据规模（LAION-80M、LAION-400M 和 LAION-2B）以及以见到的样本数衡量的训练计算量（30 亿、130 亿和 340 亿）。我们观察到，只要在训练计算量上投入足够，在更大规模的数据上看到同样数量的样本，会稳定带来更好的 ImageNet-1k 零样本迁移表现。这对较小的 B/32 和较大的 L/14 都成立。例如，在同为见到 340 亿样本的大计算预算下，在 LAION-2B 上预训练的模型显著优于在 LAION-400M 上预训练的模型（见图 15 和表 8）。由此我们得出，在训练计算量足够大的前提下，把数据规模一直扩展到 LAION-2B，对获得更强的零样本迁移确实重要。

Few-shot transfer: comparison to CLIP and effect of scale. To examine the quality of the learned representations, we evaluate few-shot linear probe performance on seven datasets commonly used to benchmark transfer performance. The results are presented in Figures 13 and 14. Figure 13 displays few-shot performance on ImageNet [13] while Figure 14 displays few-shot performance on Food101 [7], Cars [35], CIFAR-10 & 100 [37], DTD [12] and SUN397 [85]. In addition to evaluating models trained on subsets of LAION, we also compare with the CLIP models of Radford et al. [58]. Overall we observe that the models trained on LAION achieve similar transfer performance to those trained by OpenAI. Moreover, we observe that performance increases with more data (i.e., B/32 2B outperforms B/32 400M) and larger models.

少样本迁移：与 CLIP 的比较以及规模的影响 为考察所学表示的质量，我们在常用于评测迁移表现的 7 个数据集上评估少样本线性探测。结果见图 13 和图 14。图 13 展示 ImageNet[13]上的少样本表现，图 14 展示 Food101[7]、Cars[35]、CIFAR-10 与 CIFAR-100[37]、DTD[12]和 SUN397[85]上的少样本表现。除了评估在 LAION 子集上训练的模型，我们还与 Radford 等人[58]的 CLIP 模型做了比较。总体来看，在 LAION 上训练的模型与 OpenAI 训练的模型迁移表现相近。此外，数据更多（即 B/32 20 亿优于 B/32 4 亿）以及模型更大时，表现都会提高。

ImageNet-A In ImageNet-A [24] (noted INet-A), we observe large differences between CLIP WIT and LAION models, e.g. a difference of 24.3% on ViT-L/14. We note that INet-A design and data collection is quite different from other ImageNet distribution shifts datasets, as the images were specifically selected to be adversarial for a ResNet-50 pre-trained on ImageNet-1k. Although we do not have yet an explanation for the observed discrepancies and it would be interesting to understand why LAION models are worse than CLIP WIT, it is not clear whether improvements in INet-A are generalizable, as the dataset is based on adversarial images specific to a pre-trained model (ResNet-50).

ImageNet-A 在 ImageNet-A[24]（记为 INet-A）上，我们观察到 CLIP WIT 与 LAION 模型之间存在很大差异，例如 ViT-L/14 上相差 24.3%。需要指出，INet-A 的设计与数据采集和其他 ImageNet 分布偏移数据集很不一样：这些图像是专门选来对抗在 ImageNet-1k 上预训练的 ResNet-50 的。尽管我们尚未解释所观察到的差异，也值得去理解为什么 LAION 模型比 CLIP WIT 更差，但 INet-A 上的改进是否具有普遍性并不清楚，因为该数据集建立在针对特定预训练模型（ResNet-50）的对抗图像之上。

Figure 13: Evaluating few-shot linear probe performance on ImageNet. We evaluate i) models trained on various LAION subsets and ii) the original CLIP models. Models trained on LAION show similar transfer performance to those trained by OpenAI. Also evident is clear effect of model or data scale on transfer across few-shot conditions.

图 13：评估 ImageNet 上的少样本线性探测表现。我们评估 i) 在不同 LAION 子集上训练的模型，以及 ii) 原始 CLIP 模型。在 LAION 上训练的模型显示出与 OpenAI 所训练模型相近的迁移表现。同时可以清楚看到，在各种少样本设定下，模型或数据规模都会影响迁移表现。

Figure 14: Evaluating few-shot linear probe performance on 6 datasets commonly used to benchmark transfer [34]. We evaluate i) models trained on various LAION subsets and ii) the original CLIP models. We evaluate performance on Food101 [7], Cars [35], CIFAR-10 & 100 [37], DTD [12] and SUN397 [85].

图 14：评估常用于评测迁移的 6 个数据集上的少样本线性探测表现[34]。我们评估 i) 在不同 LAION 子集上训练的模型，以及 ii) 原始 CLIP 模型。评估数据集包括 Food101[7]、Cars[35]、CIFAR-10 与 CIFAR-100[37]、DTD[12]和 SUN397[85]。

Figure 15: ViT-B/32 and ViT-L/14 additional experiments where we vary the amount compute (3B, 13B, and 34B images seen) and LAION subset size (80M, 400M, 2B). We evaluate the models on zero-shot Imagenet-1k classification. Seeing same number of samples on larger data scale leads consistently to better zero-shot transfer performance, when investing enough into training compute.

图 15：对 ViT-B/32 和 ViT-L/14 做的额外实验，其中我们改变计算量（见到 30 亿、130 亿和 340 亿张图像）以及 LAION 子集规模（8000 万、4 亿、20 亿）。我们在零样本 ImageNet-1k 分类上评估这些模型。只要在训练计算量上投入足够，在更大规模的数据上看到同样数量的样本，会稳定带来更好的零样本迁移表现。

Model Samples seen LAION-80M LAION-400M LAION-2B-en ViT-B/32 3B 51.93 58.73 57.60 13B 56.46 62.90 62.56 34B - 64.07 65.50 ViT-L/14 13B - 72.98 73.12 34B - 73.90 75.40 Table 8: ViT-B/32 and ViT-L/14 additional experiments where we vary the amount compute (3B, 13B, and 34B images seen) and LAION subset size (80M, 400M, 2B). We evaluate the models on zero-shot Imagenet-1k classification. When investing enough into training compute, seeing same number of samples on larger data scale leads consistently to better zero-shot transfer performance measured on ImageNet-1k.

Model	Samples seen	LAION-80M	LAION-400M	LAION-2B-en
ViT-B/32	3B	51.93	58.73	57.60
	13B	56.46	62.90	62.56
	34B	-	64.07	65.50
ViT-L/14	13B	-	72.98	73.12
	34B	-	73.90	75.40

模型 见到的样本数 LAION-80M LAION-400M LAION-2B-en ViT-B/32 3B 51.93 58.73 57.60 13B 56.46 62.90 62.56 34B - 64.07 65.50 ViT-L/14 13B - 72.98 73.12 34B - 73.90 75.40 表 8：对 ViT-B/32 和 ViT-L/14 做的额外实验，其中我们改变计算量（见到 30 亿、130 亿和 340 亿张图像）以及 LAION 子集规模（8000 万、4 亿、20 亿）。我们在零样本 ImageNet-1k 分类上评估这些模型。只要在训练计算量上投入足够，在更大规模的数据上看到同样数量的样本，会稳定带来更好的 ImageNet-1k 零样本迁移表现。

模型	见到的样本数	LAION-80M	LAION-400M	LAION-2B-en
ViT-B/32	3B	51.93	58.73	57.60
	13B	56.46	62.90	62.56
	34B	-	64.07	65.50
ViT-L/14	13B	-	72.98	73.12
	34B	-	73.90	75.40

	B/32	B/16	B/16+	L/14
Dataset	CLIP WIT	LAION-400M	LAION-2B	CLIP WIT	LAION-400M	LAION-400M	CLIP WIT	LAION-400M
INet	63.3	${\color[rgb]{0,0,0}{62.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	${\color[rgb]{0,0,0}{65.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.4}}$	68.3	${\color[rgb]{0,0,0}{67.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.3}}$	69.2	75.6	${\color[rgb]{0,0,0}{72.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.8}}$
INet-v2	56.0	${\color[rgb]{0,0,0}{55.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.9}}$	${\color[rgb]{0,0,0}{57.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.4}}$	61.9	${\color[rgb]{0,0,0}{59.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.3}}$	61.5	69.8	${\color[rgb]{0,0,0}{65.4}}^{\tiny\color[rgb]{1,0,0}\textbf{-4.4}}$
INet-R	69.4	${\color[rgb]{0,0,0}{73.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.0}}$	${\color[rgb]{0,0,0}{75.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.5}}$	77.7	${\color[rgb]{0,0,0}{77.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	80.5	87.9	${\color[rgb]{0,0,0}{84.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.2}}$
INet-S	42.3	${\color[rgb]{0,0,0}{49.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+7.1}}$	${\color[rgb]{0,0,0}{52.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+10.6}}$	48.2	${\color[rgb]{0,0,0}{52.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.2}}$	54.4	59.6	59.6
ObjNet	44.2	${\color[rgb]{0,0,0}{43.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.3}}$	${\color[rgb]{0,0,0}{48.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.5}}$	55.3	${\color[rgb]{0,0,0}{51.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.8}}$	53.9	69.0	${\color[rgb]{0,0,0}{59.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-9.1}}$
INet-A	31.6	${\color[rgb]{0,0,0}{21.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-9.9}}$	${\color[rgb]{0,0,0}{26.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-5.5}}$	49.9	${\color[rgb]{0,0,0}{33.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-16.7}}$	36.9	70.8	${\color[rgb]{0,0,0}{46.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-24.3}}$
CIFAR-10	89.8	${\color[rgb]{0,0,0}{90.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.9}}$	${\color[rgb]{0,0,0}{94.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.2}}$	90.8	${\color[rgb]{0,0,0}{91.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.9}}$	92.7	95.6	${\color[rgb]{0,0,0}{94.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.0}}$
CIFAR-100	64.2	${\color[rgb]{0,0,0}{70.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.1}}$	${\color[rgb]{0,0,0}{75.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+11.2}}$	66.9	${\color[rgb]{0,0,0}{71.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.3}}$	73.8	75.9	${\color[rgb]{0,0,0}{77.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.5}}$
MNIST	48.2	${\color[rgb]{0,0,0}{37.4}}^{\tiny\color[rgb]{1,0,0}\textbf{-10.8}}$	${\color[rgb]{0,0,0}{63.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+15.2}}$	51.8	${\color[rgb]{0,0,0}{66.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+14.5}}$	57.0	76.4	${\color[rgb]{0,0,0}{76.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$
Flowers102	66.5	${\color[rgb]{0,0,0}{68.1}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.6}}$	${\color[rgb]{0,0,0}{69.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.5}}$	71.2	${\color[rgb]{0,0,0}{69.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.9}}$	71.1	79.2	${\color[rgb]{0,0,0}{75.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.6}}$
Cars	59.6	${\color[rgb]{0,0,0}{79.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+19.7}}$	${\color[rgb]{0,0,0}{84.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+24.8}}$	64.7	${\color[rgb]{0,0,0}{83.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+19.0}}$	84.5	77.9	${\color[rgb]{0,0,0}{89.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+11.7}}$
SVHN	13.4	${\color[rgb]{0,0,0}{27.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+14.3}}$	${\color[rgb]{0,0,0}{38.8}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+25.4}}$	31.3	${\color[rgb]{0,0,0}{38.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+7.2}}$	36.2	57.0	${\color[rgb]{0,0,0}{38.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-19.0}}$
FER2013	41.4	${\color[rgb]{0,0,0}{43.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.6}}$	${\color[rgb]{0,0,0}{48.1}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.7}}$	46.3	${\color[rgb]{0,0,0}{43.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.1}}$	44.5	50.1	${\color[rgb]{0,0,0}{50.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$
RenderedSST2	58.6	${\color[rgb]{0,0,0}{52.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-6.3}}$	${\color[rgb]{0,0,0}{54.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-4.3}}$	60.5	${\color[rgb]{0,0,0}{54.4}}^{\tiny\color[rgb]{1,0,0}\textbf{-6.1}}$	57.9	68.9	${\color[rgb]{0,0,0}{56.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-12.9}}$
Pets	87.3	${\color[rgb]{0,0,0}{86.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	${\color[rgb]{0,0,0}{89.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.9}}$	89.0	${\color[rgb]{0,0,0}{89.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	90.3	93.3	${\color[rgb]{0,0,0}{91.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.4}}$
Caltech-101	81.6	${\color[rgb]{0,0,0}{83.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.6}}$	${\color[rgb]{0,0,0}{83.1}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.5}}$	82.2	${\color[rgb]{0,0,0}{83.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.4}}$	83.2	83.3	${\color[rgb]{0,0,0}{84.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.7}}$
VOC2007-Cl	76.4	${\color[rgb]{0,0,0}{75.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.6}}$	${\color[rgb]{0,0,0}{78.8}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.4}}$	78.3	${\color[rgb]{0,0,0}{76.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.5}}$	76.4	78.3	${\color[rgb]{0,0,0}{75.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.7}}$
SUN397	62.5	${\color[rgb]{0,0,0}{67.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.5}}$	${\color[rgb]{0,0,0}{68.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.0}}$	64.4	${\color[rgb]{0,0,0}{69.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+5.2}}$	69.8	67.6	${\color[rgb]{0,0,0}{72.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+5.0}}$
FGVC Aircraft	19.6	${\color[rgb]{0,0,0}{16.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.9}}$	${\color[rgb]{0,0,0}{23.1}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+3.5}}$	24.3	${\color[rgb]{0,0,0}{17.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-6.6}}$	18.5	31.8	${\color[rgb]{0,0,0}{25.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-6.8}}$
Country211	17.2	${\color[rgb]{0,0,0}{14.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.4}}$	${\color[rgb]{0,0,0}{16.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.7}}$	22.8	${\color[rgb]{0,0,0}{18.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-4.7}}$	18.9	31.9	${\color[rgb]{0,0,0}{23.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-8.9}}$
DTD	44.3	${\color[rgb]{0,0,0}{54.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+10.3}}$	${\color[rgb]{0,0,0}{53.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+9.6}}$	44.9	${\color[rgb]{0,0,0}{51.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.4}}$	55.5	55.3	${\color[rgb]{0,0,0}{60.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+5.2}}$
GTSRB	32.6	${\color[rgb]{0,0,0}{42.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+9.4}}$	${\color[rgb]{0,0,0}{36.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+3.9}}$	43.3	${\color[rgb]{0,0,0}{43.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	49.4	50.6	${\color[rgb]{0,0,0}{49.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.7}}$
STL10	97.1	${\color[rgb]{0,0,0}{95.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.5}}$	${\color[rgb]{0,0,0}{96.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.6}}$	98.2	${\color[rgb]{0,0,0}{97.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.2}}$	97.0	99.4	${\color[rgb]{0,0,0}{98.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.3}}$
Retino	45.5	${\color[rgb]{0,0,0}{24.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-21.3}}$	${\color[rgb]{0,0,0}{19.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-26.4}}$	3.3	${\color[rgb]{0,0,0}{7.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.1}}$	9.2	73.3	${\color[rgb]{0,0,0}{6.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-67.3}}$
EuroSAT	50.4	${\color[rgb]{0,0,0}{51.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.1}}$	${\color[rgb]{0,0,0}{50.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.1}}$	55.9	${\color[rgb]{0,0,0}{50.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-5.6}}$	58.2	62.6	${\color[rgb]{0,0,0}{62.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.3}}$
RESISC45	53.6	${\color[rgb]{0,0,0}{54.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.9}}$	${\color[rgb]{0,0,0}{61.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+8.3}}$	58.2	${\color[rgb]{0,0,0}{58.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.3}}$	61.4	63.4	${\color[rgb]{0,0,0}{67.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.0}}$
PCAM	62.3	${\color[rgb]{0,0,0}{55.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-6.4}}$	${\color[rgb]{0,0,0}{50.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-11.6}}$	50.7	${\color[rgb]{0,0,0}{59.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+8.9}}$	55.2	52.0	${\color[rgb]{0,0,0}{49.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.4}}$
CLEVR Counts	23.2	${\color[rgb]{0,0,0}{16.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-7.0}}$	${\color[rgb]{0,0,0}{19.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-4.0}}$	21.2	${\color[rgb]{0,0,0}{28.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+7.5}}$	23.9	19.4	${\color[rgb]{0,0,0}{24.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.8}}$
CLEVR Dist	16.3	${\color[rgb]{0,0,0}{15.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	${\color[rgb]{0,0,0}{16.8}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.5}}$	15.8	${\color[rgb]{0,0,0}{24.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+8.7}}$	15.9	16.1	${\color[rgb]{0,0,0}{14.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.2}}$
DSPRITES Orient	2.4	${\color[rgb]{0,0,0}{1.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.5}}$	${\color[rgb]{0,0,0}{2.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.1}}$	2.3	${\color[rgb]{0,0,0}{2.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.6}}$	2.7	2.3	${\color[rgb]{0,0,0}{2.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.3}}$
DSPRITES pos	3.6	${\color[rgb]{0,0,0}{2.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.8}}$	${\color[rgb]{0,0,0}{3.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.5}}$	3.0	${\color[rgb]{0,0,0}{3.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	4.3	3.2	${\color[rgb]{0,0,0}{3.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.2}}$
SmallNORB Elv	12.7	${\color[rgb]{0,0,0}{9.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.8}}$	${\color[rgb]{0,0,0}{11.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.7}}$	12.2	${\color[rgb]{0,0,0}{10.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.2}}$	11.0	11.5	${\color[rgb]{0,0,0}{11.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.5}}$
SmallNORB Azim	6.1	${\color[rgb]{0,0,0}{4.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.6}}$	${\color[rgb]{0,0,0}{5.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.9}}$	5.2	${\color[rgb]{0,0,0}{6.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.8}}$	5.5	4.5	${\color[rgb]{0,0,0}{5.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.8}}$
DMLAB	19.3	${\color[rgb]{0,0,0}{17.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.0}}$	${\color[rgb]{0,0,0}{18.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	15.5	${\color[rgb]{0,0,0}{15.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	14.8	16.3	${\color[rgb]{0,0,0}{18.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.4}}$
KITTI Dist	27.4	${\color[rgb]{0,0,0}{28.8}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.4}}$	${\color[rgb]{0,0,0}{17.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-9.8}}$	26.4	${\color[rgb]{0,0,0}{18.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-8.3}}$	28.1	21.8	${\color[rgb]{0,0,0}{20.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.7}}$
VTAB+(Avg.)	45.4	${\color[rgb]{0,0,0}{45.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	${\color[rgb]{0,0,0}{47.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.5}}$	47.5	${\color[rgb]{0,0,0}{48.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.8}}$	49.2	55.7	${\color[rgb]{0,0,0}{51.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.9}}$

	B/32	B/16	B/16+	L/14
数据集	CLIP WIT	LAION-400M	LAION-2B	CLIP WIT	LAION-400M	LAION-400M	CLIP WIT	LAION-400M
INet	63.3	${\color[rgb]{0,0,0}{62.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	${\color[rgb]{0,0,0}{65.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.4}}$	68.3	${\color[rgb]{0,0,0}{67.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.3}}$	69.2	75.6	${\color[rgb]{0,0,0}{72.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.8}}$
INet-v2	56.0	${\color[rgb]{0,0,0}{55.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.9}}$	${\color[rgb]{0,0,0}{57.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.4}}$	61.9	${\color[rgb]{0,0,0}{59.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.3}}$	61.5	69.8	${\color[rgb]{0,0,0}{65.4}}^{\tiny\color[rgb]{1,0,0}\textbf{-4.4}}$
INet-R	69.4	${\color[rgb]{0,0,0}{73.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.0}}$	${\color[rgb]{0,0,0}{75.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.5}}$	77.7	${\color[rgb]{0,0,0}{77.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	80.5	87.9	${\color[rgb]{0,0,0}{84.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.2}}$
INet-S	42.3	${\color[rgb]{0,0,0}{49.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+7.1}}$	${\color[rgb]{0,0,0}{52.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+10.6}}$	48.2	${\color[rgb]{0,0,0}{52.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.2}}$	54.4	59.6	59.6
ObjNet	44.2	${\color[rgb]{0,0,0}{43.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.3}}$	${\color[rgb]{0,0,0}{48.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.5}}$	55.3	${\color[rgb]{0,0,0}{51.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.8}}$	53.9	69.0	${\color[rgb]{0,0,0}{59.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-9.1}}$
INet-A	31.6	${\color[rgb]{0,0,0}{21.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-9.9}}$	${\color[rgb]{0,0,0}{26.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-5.5}}$	49.9	${\color[rgb]{0,0,0}{33.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-16.7}}$	36.9	70.8	${\color[rgb]{0,0,0}{46.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-24.3}}$
CIFAR-10	89.8	${\color[rgb]{0,0,0}{90.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.9}}$	${\color[rgb]{0,0,0}{94.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.2}}$	90.8	${\color[rgb]{0,0,0}{91.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.9}}$	92.7	95.6	${\color[rgb]{0,0,0}{94.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.0}}$
CIFAR-100	64.2	${\color[rgb]{0,0,0}{70.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.1}}$	${\color[rgb]{0,0,0}{75.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+11.2}}$	66.9	${\color[rgb]{0,0,0}{71.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.3}}$	73.8	75.9	${\color[rgb]{0,0,0}{77.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.5}}$
MNIST	48.2	${\color[rgb]{0,0,0}{37.4}}^{\tiny\color[rgb]{1,0,0}\textbf{-10.8}}$	${\color[rgb]{0,0,0}{63.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+15.2}}$	51.8	${\color[rgb]{0,0,0}{66.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+14.5}}$	57.0	76.4	${\color[rgb]{0,0,0}{76.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$
Flowers102	66.5	${\color[rgb]{0,0,0}{68.1}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.6}}$	${\color[rgb]{0,0,0}{69.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.5}}$	71.2	${\color[rgb]{0,0,0}{69.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.9}}$	71.1	79.2	${\color[rgb]{0,0,0}{75.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.6}}$
Cars	59.6	${\color[rgb]{0,0,0}{79.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+19.7}}$	${\color[rgb]{0,0,0}{84.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+24.8}}$	64.7	${\color[rgb]{0,0,0}{83.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+19.0}}$	84.5	77.9	${\color[rgb]{0,0,0}{89.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+11.7}}$
SVHN	13.4	${\color[rgb]{0,0,0}{27.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+14.3}}$	${\color[rgb]{0,0,0}{38.8}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+25.4}}$	31.3	${\color[rgb]{0,0,0}{38.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+7.2}}$	36.2	57.0	${\color[rgb]{0,0,0}{38.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-19.0}}$
FER2013	41.4	${\color[rgb]{0,0,0}{43.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.6}}$	${\color[rgb]{0,0,0}{48.1}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.7}}$	46.3	${\color[rgb]{0,0,0}{43.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.1}}$	44.5	50.1	${\color[rgb]{0,0,0}{50.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$
RenderedSST2	58.6	${\color[rgb]{0,0,0}{52.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-6.3}}$	${\color[rgb]{0,0,0}{54.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-4.3}}$	60.5	${\color[rgb]{0,0,0}{54.4}}^{\tiny\color[rgb]{1,0,0}\textbf{-6.1}}$	57.9	68.9	${\color[rgb]{0,0,0}{56.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-12.9}}$
Pets	87.3	${\color[rgb]{0,0,0}{86.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	${\color[rgb]{0,0,0}{89.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.9}}$	89.0	${\color[rgb]{0,0,0}{89.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	90.3	93.3	${\color[rgb]{0,0,0}{91.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.4}}$
Caltech-101	81.6	${\color[rgb]{0,0,0}{83.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.6}}$	${\color[rgb]{0,0,0}{83.1}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.5}}$	82.2	${\color[rgb]{0,0,0}{83.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.4}}$	83.2	83.3	${\color[rgb]{0,0,0}{84.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.7}}$
VOC2007-Cl	76.4	${\color[rgb]{0,0,0}{75.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.6}}$	${\color[rgb]{0,0,0}{78.8}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.4}}$	78.3	${\color[rgb]{0,0,0}{76.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.5}}$	76.4	78.3	${\color[rgb]{0,0,0}{75.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.7}}$
SUN397	62.5	${\color[rgb]{0,0,0}{67.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.5}}$	${\color[rgb]{0,0,0}{68.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.0}}$	64.4	${\color[rgb]{0,0,0}{69.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+5.2}}$	69.8	67.6	${\color[rgb]{0,0,0}{72.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+5.0}}$
FGVC Aircraft	19.6	${\color[rgb]{0,0,0}{16.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.9}}$	${\color[rgb]{0,0,0}{23.1}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+3.5}}$	24.3	${\color[rgb]{0,0,0}{17.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-6.6}}$	18.5	31.8	${\color[rgb]{0,0,0}{25.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-6.8}}$
Country211	17.2	${\color[rgb]{0,0,0}{14.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.4}}$	${\color[rgb]{0,0,0}{16.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.7}}$	22.8	${\color[rgb]{0,0,0}{18.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-4.7}}$	18.9	31.9	${\color[rgb]{0,0,0}{23.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-8.9}}$
DTD	44.3	${\color[rgb]{0,0,0}{54.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+10.3}}$	${\color[rgb]{0,0,0}{53.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+9.6}}$	44.9	${\color[rgb]{0,0,0}{51.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+6.4}}$	55.5	55.3	${\color[rgb]{0,0,0}{60.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+5.2}}$
GTSRB	32.6	${\color[rgb]{0,0,0}{42.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+9.4}}$	${\color[rgb]{0,0,0}{36.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+3.9}}$	43.3	${\color[rgb]{0,0,0}{43.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	49.4	50.6	${\color[rgb]{0,0,0}{49.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.7}}$
STL10	97.1	${\color[rgb]{0,0,0}{95.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.5}}$	${\color[rgb]{0,0,0}{96.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.6}}$	98.2	${\color[rgb]{0,0,0}{97.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.2}}$	97.0	99.4	${\color[rgb]{0,0,0}{98.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.3}}$
Retino	45.5	${\color[rgb]{0,0,0}{24.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-21.3}}$	${\color[rgb]{0,0,0}{19.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-26.4}}$	3.3	${\color[rgb]{0,0,0}{7.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.1}}$	9.2	73.3	${\color[rgb]{0,0,0}{6.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-67.3}}$
EuroSAT	50.4	${\color[rgb]{0,0,0}{51.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.1}}$	${\color[rgb]{0,0,0}{50.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.1}}$	55.9	${\color[rgb]{0,0,0}{50.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-5.6}}$	58.2	62.6	${\color[rgb]{0,0,0}{62.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.3}}$
RESISC45	53.6	${\color[rgb]{0,0,0}{54.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.9}}$	${\color[rgb]{0,0,0}{61.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+8.3}}$	58.2	${\color[rgb]{0,0,0}{58.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.3}}$	61.4	63.4	${\color[rgb]{0,0,0}{67.4}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.0}}$
PCAM	62.3	${\color[rgb]{0,0,0}{55.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-6.4}}$	${\color[rgb]{0,0,0}{50.7}}^{\tiny\color[rgb]{1,0,0}\textbf{-11.6}}$	50.7	${\color[rgb]{0,0,0}{59.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+8.9}}$	55.2	52.0	${\color[rgb]{0,0,0}{49.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.4}}$
CLEVR Counts	23.2	${\color[rgb]{0,0,0}{16.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-7.0}}$	${\color[rgb]{0,0,0}{19.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-4.0}}$	21.2	${\color[rgb]{0,0,0}{28.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+7.5}}$	23.9	19.4	${\color[rgb]{0,0,0}{24.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+4.8}}$
CLEVR Dist	16.3	${\color[rgb]{0,0,0}{15.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	${\color[rgb]{0,0,0}{16.8}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.5}}$	15.8	${\color[rgb]{0,0,0}{24.5}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+8.7}}$	15.9	16.1	${\color[rgb]{0,0,0}{14.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.2}}$
DSPRITES Orient	2.4	${\color[rgb]{0,0,0}{1.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.5}}$	${\color[rgb]{0,0,0}{2.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.1}}$	2.3	${\color[rgb]{0,0,0}{2.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.6}}$	2.7	2.3	${\color[rgb]{0,0,0}{2.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.3}}$
DSPRITES pos	3.6	${\color[rgb]{0,0,0}{2.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.8}}$	${\color[rgb]{0,0,0}{3.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.5}}$	3.0	${\color[rgb]{0,0,0}{3.2}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	4.3	3.2	${\color[rgb]{0,0,0}{3.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.2}}$
SmallNORB Elv	12.7	${\color[rgb]{0,0,0}{9.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.8}}$	${\color[rgb]{0,0,0}{11.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.7}}$	12.2	${\color[rgb]{0,0,0}{10.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.2}}$	11.0	11.5	${\color[rgb]{0,0,0}{11.0}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.5}}$
SmallNORB Azim	6.1	${\color[rgb]{0,0,0}{4.5}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.6}}$	${\color[rgb]{0,0,0}{5.2}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.9}}$	5.2	${\color[rgb]{0,0,0}{6.0}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.8}}$	5.5	4.5	${\color[rgb]{0,0,0}{5.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.8}}$
DMLAB	19.3	${\color[rgb]{0,0,0}{17.3}}^{\tiny\color[rgb]{1,0,0}\textbf{-2.0}}$	${\color[rgb]{0,0,0}{18.9}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	15.5	${\color[rgb]{0,0,0}{15.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-0.4}}$	14.8	16.3	${\color[rgb]{0,0,0}{18.7}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.4}}$
KITTI Dist	27.4	${\color[rgb]{0,0,0}{28.8}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+1.4}}$	${\color[rgb]{0,0,0}{17.6}}^{\tiny\color[rgb]{1,0,0}\textbf{-9.8}}$	26.4	${\color[rgb]{0,0,0}{18.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-8.3}}$	28.1	21.8	${\color[rgb]{0,0,0}{20.1}}^{\tiny\color[rgb]{1,0,0}\textbf{-1.7}}$
VTAB+(Avg.)	45.4	${\color[rgb]{0,0,0}{45.6}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.2}}$	${\color[rgb]{0,0,0}{47.9}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+2.5}}$	47.5	${\color[rgb]{0,0,0}{48.3}}^{\tiny\color[rgb]{0,0.88,0}\textbf{+0.8}}$	49.2	55.7	${\color[rgb]{0,0,0}{51.8}}^{\tiny\color[rgb]{1,0,0}\textbf{-3.9}}$

Flickr30K (1K test set) Model Pre-training Image \to Text Text \to Image R@1 R@5 R@10 R@1 R@5 R@10 ViT-B/32 CLIP WIT 77.5 94.7 98.2 58.8 83.3 89.7 LAION-400M 78.9 94.0 97.1 61.7 85.5 90.9 LAION-2B-en 84.3 96.3 98.4 66.3 88.2 93.2 ViT-B/16 CLIP WIT 81.9 96.2 98.8 81.9 96.2 98.8 LAION-400M 83.3 96.8 98.5 65.5 88.3 93.0 ViT-B/16+ LAION-400M 86.5 97.1 98.8 68.0 88.9 94.0 ViT-L/14 CLIP WIT 85.1 97.3 99.0 65.2 87.3 92.0 LAION-400M 87.6 97.7 99.5 70.3 90.9 94.6 Table 10: CLIP Zero-Shot retrieval results on the Flickr30K test set. We show retrieval performance at 1, 5, and 10 samples for both image to text and text to image.

		Flickr30K (1K test set)
Model	Pre-training	Image → Text	Text → Image
		R@1	R@5	R@10	R@1	R@5	R@10
ViT-B/32	CLIP WIT	77.5	94.7	98.2	58.8	83.3	89.7
	LAION-400M	78.9	94.0	97.1	61.7	85.5	90.9
	LAION-2B-en	84.3	96.3	98.4	66.3	88.2	93.2
ViT-B/16	CLIP WIT	81.9	96.2	98.8	81.9	96.2	98.8
	LAION-400M	83.3	96.8	98.5	65.5	88.3	93.0
ViT-B/16+	LAION-400M	86.5	97.1	98.8	68.0	88.9	94.0
ViT-L/14	CLIP WIT	85.1	97.3	99.0	65.2	87.3	92.0
	LAION-400M	87.6	97.7	99.5	70.3	90.9	94.6

Flickr30K (1K test set) 模型 预训练 图像 \to 文本 文本 \to 图像 R@1 R@5 R@10 R@1 R@5 R@10 ViT-B/32 CLIP WIT 77.5 94.7 98.2 58.8 83.3 89.7 LAION-400M 78.9 94.0 97.1 61.7 85.5 90.9 LAION-2B-en 84.3 96.3 98.4 66.3 88.2 93.2 ViT-B/16 CLIP WIT 81.9 96.2 98.8 81.9 96.2 98.8 LAION-400M 83.3 96.8 98.5 65.5 88.3 93.0 ViT-B/16+ LAION-400M 86.5 97.1 98.8 68.0 88.9 94.0 ViT-L/14 CLIP WIT 85.1 97.3 99.0 65.2 87.3 92.0 LAION-400M 87.6 97.7 99.5 70.3 90.9 94.6 表 10：CLIP 在 Flickr30K 测试集上的零样本检索结果。我们分别报告图像到文本和文本到图像在 1、5、10 个样本处的检索表现。

		Flickr30K (1K test set)
模型	预训练	图像 → 文本	文本 → 图像
		R@1	R@5	R@10	R@1	R@5	R@10
ViT-B/32	CLIP WIT	77.5	94.7	98.2	58.8	83.3	89.7
	LAION-400M	78.9	94.0	97.1	61.7	85.5	90.9
	LAION-2B-en	84.3	96.3	98.4	66.3	88.2	93.2
ViT-B/16	CLIP WIT	81.9	96.2	98.8	81.9	96.2	98.8
	LAION-400M	83.3	96.8	98.5	65.5	88.3	93.0
ViT-B/16+	LAION-400M	86.5	97.1	98.8	68.0	88.9	94.0
ViT-L/14	CLIP WIT	85.1	97.3	99.0	65.2	87.3	92.0
	LAION-400M	87.6	97.7	99.5	70.3	90.9	94.6

MSCOCO (5K test set) Model Pre-training Image \to Text Text \to Image R@1 R@5 R@10 R@1 R@5 R@10 ViT-B/32 CLIP WIT 50.0 75.0 83.3 30.4 54.8 66.1 LAION-400M 53.5 77.2 85.4 34.9 60.3 71.1 LAION-2B-en 56.4 79.6 87.4 38.7 64.1 74.4 ViT-B/16 CLIP WIT 51.7 76.8 84.3 32.7 57.8 68.2 LAION-400M 56.5 80.4 87.3 37.9 63.2 73.3 ViT-B/16+ LAION-400M 58.6 81.6 88.4 40.0 65.5 75.1 ViT-L/14 CLIP WIT 56.0 79.5 86.9 35.3 60.0 70.2 LAION-400M 59.3 81.9 89.0 42.0 67.2 76.6 Table 11: CLIP Zero-Shot retrieval results on the MSCOCO test set. We show retrieval performance at 1, 5, and 10 samples for both image to text and text to image.

		MSCOCO (5K test set)
Model	Pre-training	Image → Text	Text → Image
		R@1	R@5	R@10	R@1	R@5	R@10
ViT-B/32	CLIP WIT	50.0	75.0	83.3	30.4	54.8	66.1
	LAION-400M	53.5	77.2	85.4	34.9	60.3	71.1
	LAION-2B-en	56.4	79.6	87.4	38.7	64.1	74.4
ViT-B/16	CLIP WIT	51.7	76.8	84.3	32.7	57.8	68.2
	LAION-400M	56.5	80.4	87.3	37.9	63.2	73.3
ViT-B/16+	LAION-400M	58.6	81.6	88.4	40.0	65.5	75.1
ViT-L/14	CLIP WIT	56.0	79.5	86.9	35.3	60.0	70.2
	LAION-400M	59.3	81.9	89.0	42.0	67.2	76.6

MSCOCO (5K test set) 模型 预训练 图像 \to 文本 文本 \to 图像 R@1 R@5 R@10 R@1 R@5 R@10 ViT-B/32 CLIP WIT 50.0 75.0 83.3 30.4 54.8 66.1 LAION-400M 53.5 77.2 85.4 34.9 60.3 71.1 LAION-2B-en 56.4 79.6 87.4 38.7 64.1 74.4 ViT-B/16 CLIP WIT 51.7 76.8 84.3 32.7 57.8 68.2 LAION-400M 56.5 80.4 87.3 37.9 63.2 73.3 ViT-B/16+ LAION-400M 58.6 81.6 88.4 40.0 65.5 75.1 ViT-L/14 CLIP WIT 56.0 79.5 86.9 35.3 60.0 70.2 LAION-400M 59.3 81.9 89.0 42.0 67.2 76.6 表 11：CLIP 在 MSCOCO 测试集上的零样本检索结果。我们分别报告图像到文本和文本到图像在 1、5、10 个样本处的检索表现。

		MSCOCO (5K test set)
模型	预训练	图像 → 文本	文本 → 图像
		R@1	R@5	R@10	R@1	R@5	R@10
ViT-B/32	CLIP WIT	50.0	75.0	83.3	30.4	54.8	66.1
	LAION-400M	53.5	77.2	85.4	34.9	60.3	71.1
	LAION-2B-en	56.4	79.6	87.4	38.7	64.1	74.4
ViT-B/16	CLIP WIT	51.7	76.8	84.3	32.7	57.8	68.2
	LAION-400M	56.5	80.4	87.3	37.9	63.2	73.3
ViT-B/16+	LAION-400M	58.6	81.6	88.4	40.0	65.5	75.1
ViT-L/14	CLIP WIT	56.0	79.5	86.9	35.3	60.0	70.2
	LAION-400M	59.3	81.9	89.0	42.0	67.2	76.6

Diabetic Retinopathy We observe a large variation of performance on Diabetic Retinopathy [22] (noted Retino). Accuracy goes from 3% to 73.3% for CLIP WIT models, and from 7.4% to 24.2% for LAION models. Additionally, the difference between CLIP WIT and LAION models goes up to 67.3% (on L/14). After investigating, we found that on low accuracy models, performance on the majority class is very low (e.g., for ViT-B/16 LAION model, recall was 3.4% on the majority class), and given that the dataset is highly imbalanced (majority class constitutes 74% of the samples), accuracy is affected heavily. A possible reason for low performance could be the prompts that were used, thus tuning the prompts could alleviate the problem. We re-evaluated the models using mean per-class recall, and found that the performances are less disparate, with a maximum difference between CLIP WIT models and LAION models of 2.1%. Overall, the results remain quite low, best mean per-class recall was 25.4%, obtained with ViT-B/32 trained on LAION-400M.

糖尿病视网膜病变 我们在 Diabetic Retinopathy[22]（记为 Retino）上观察到很大的表现差异。CLIP WIT 模型的准确率从 3% 到 73.3% 不等，LAION 模型则从 7.4% 到 24.2%。此外，CLIP WIT 与 LAION 模型之间的差距最大可达 67.3%（在 L/14 上）。调查后我们发现，在准确率较低的模型上，多数类上的表现非常低（例如 ViT-B/16 的 LAION 模型在多数类上的召回率只有 3.4%）；而该数据集又高度不平衡（多数类占样本的 74%），因此准确率受到很大影响。表现偏低的一个可能原因是所用提示，因此调整提示或许能缓解这一问题。我们改用各类别平均召回率重新评估后发现，表现差异变小，CLIP WIT 模型与 LAION 模型之间的最大差距为 2.1%。总体结果仍然相当低，最好的各类别平均召回率是 25.4%，由在 LAION-400M 上训练的 ViT-B/32 获得。

Structured tasks Similarly to [94], we observe low accuracy on VTAB’s structured tasks [91] (CLEVR, DSPRITES, SmallNORB, DMLAB, KITTI) which involve counting, depth prediction, or position/angle prediction. Finding ways to improve accuracy on those tasks is an open research question [94] that would be interesting to investigate in future work.

结构化任务 与[94]类似，我们在 VTAB 的结构化任务[91]（CLEVR、DSPRITES、SmallNORB、DMLAB、KITTI）上观察到较低准确率，这些任务涉及计数、深度预测或位置/角度预测。如何提高这些任务上的准确率仍是一个开放问题[94]，值得在未来工作中研究。

Retrieval We observe consistent improvements of LAION models over CLIP WIT models on MSCOCO 5K test set (Tab. 11) across all metrics and model sizes. On Flickr30k (Tab 10), we observe similar or better results with LAION models, with the exception of image retrieval on ViT-B/16 where CLIP WIT model is better. It would be interesting to investigate why LAION models have an advantage, and whether the advantage is more general or specific to the datasets that are considered in this work. Overall, we obtain better results than the best reported results in [58], e.g. on MSCOCO text retrieval we obtain 59.3% vs 58.4% for CLIP WIT, and on image retrieval we obtain 42% vs 37.8% for CLIP WIT, both evaluated using the R@1 metric.

检索 在 MSCOCO 5K 测试集（表 11）上，我们观察到 LAION 模型在所有指标和模型规模上都稳定优于 CLIP WIT 模型。在 Flickr30k（表 10）上，LAION 模型取得相近或更好的结果，例外是 ViT-B/16 的图像检索，此时 CLIP WIT 更好。值得进一步研究 LAION 模型为何具有优势，以及这种优势是更普遍的，还是只针对本文考虑的数据集。总体而言，我们得到的结果优于[58]中报告的最好结果：例如在 MSCOCO 文本检索上，我们得到 59.3%，而 CLIP WIT 为 58.4%；在图像检索上，我们得到 42%，而 CLIP WIT 为 37.8%，两者都以 R@1 评估。

Appendix F Overview of Experiments and Results on Generative Models

附录 F 生成模型实验与结果概览

Here we provide overview about training experiments that were performed with generative models, GLIDE and Stable Diffusion, using subsets of LAION-5B.

这里概述我们用 LAION-5B 的若干子集，对生成模型 GLIDE 和 Stable Diffusion 所做的训练实验。

F.1 GLIDE

OpenAI released checkpoints for the GLIDE [52] architecture to the public, but only released checkpoints trained on a filtered dataset removing hate-symbols and humans. These models can do a lot, but are incapable of generating imagery of humans. To evaluate the LAION dataset and its generalization capabilities, we aim to re-introduce the ability to generate imagery of humans into these checkpoints by finetuning them on LAION-5B.

OpenAI 向公众发布了 GLIDE[52] 结构的检查点，但只发布了在过滤掉仇恨符号和人物后的数据上训练的检查点。这些模型能做很多事，却无法生成人物图像。为评估 LAION 数据集及其泛化能力，我们希望通过在 LAION-5B 上微调这些检查点，重新引入生成人物图像的能力。

We finetune the released GLIDE 64 pixel base (filtered) checkpoint from OpenAI on LAION-5B. For upscaling from 64x64 images to 256x256 images, we use the unmodified weights from OpenAI GLIDE-upsample-filtered. During training, captions were randomly replaced with the unconditional token 20% of the time. All code and checkpoints are provided in our GitHub repository [^35].

我们在 LAION-5B 上微调了 OpenAI 发布的 GLIDE 64 像素基础（过滤版）检查点。从 64\times64 图像放大到 256\times256 图像时，我们使用未经修改的 OpenAI GLIDE-upsample-filtered 权重。训练过程中，说明文字有 20% 的概率被随机替换为无条件标记。全部代码和检查点都放在我们的 GitHub 仓库中[^35]。

We finetune LAIONIDE-v1 first, using an NVIDIA RTX 2070 Super GPU. Due to the 8GB VRAM constrain posed by the RTX 2070, we only use a batch size of 1. This initial checkpoint is provided as LAIONIDE-v1.

我们首先微调得到 LAIONIDE-v1，使用的是一块 NVIDIA RTX 2070 Super GPU。由于 RTX 2070 只有 8GB 显存，我们只用批量大小 1。这一初始检查点作为 LAIONIDE-v1 提供。

To accelerate training, LAIONIDE-v2 is finetuned from LAIONIDE-v1 using an 8xA100 pod from Stability. LAIONIDE-v2 sees roughly 25 million shuffled text-image pairs from LAION-2B. Some data is filtered during finetuning: if a text-image pair’s ‘nsfw’ metadata has a value of ’NSFW’ or ’LIKELY’, we remove the sample. We remove any pairs where the language code is not ‘en’, to focus the model on english. We remove any images with an aspect ratio greater than 1.3 or less than 0.8. We remove all images where the smallest side is less than 256 pixels in length. Finally, we perform a sub-string search against a list of common slurs, and remove captions containing some slurs, although this is far from comprehensive.

为加快训练，LAIONIDE-v2 从 LAIONIDE-v1 出发，使用 Stability 的 8\timesA100 节点继续微调。LAIONIDE-v2 大约见到来自 LAION-2B 的 2500 万对打乱后的图文样本。微调时做了一些过滤：如果一条图文对的 “nsfw” 元数据值为 “NSFW” 或 “LIKELY”，我们就去掉该样本。我们去掉语言代码不是 “en” 的样本，以便让模型聚焦于英语。我们去掉宽高比大于 1.3 或小于 0.8 的图像。我们也去掉最短边小于 256 像素的全部图像。最后，我们用一份常见侮辱性词汇表做子字符串检索，并去掉包含其中一些词汇的说明；这远谈不上全面。

To reduce the number of watermarks output by LAIONIDE-v2, we finetune to create LAIONIDE-v3. It sees roughly 1 million text-image pairs from a shuffled mixture of datasets: COCO 2017’s training set (MS-COCO), Visual Genome, Open Images “Localized Annotations” and LAION-5B [44, 36, 39, 55]. We find this reduces the number of watermarks output compared to LAIONIDE-V2 during manual analysis.

为减少 LAIONIDE-v2 输出的水印，我们继续微调得到 LAIONIDE-v3。它大约见到 100 万对图文样本，这些样本来自打乱混合的多个数据集：COCO 2017 训练集（MS-COCO）、Visual Genome、Open Images 的 “Localized Annotations” 以及 LAION-5B[44, 36, 39, 55]。人工分析表明，与 LAIONIDE-v2 相比，这减少了输出中的水印数量。

To improve inference time we make use of the pseudo linear multi-step diffusion sampling method from Liu et al. [45] as implemented by Katherine Crowson.

为缩短推理时间，我们使用 Liu 等人[45]提出、由 Katherine Crowson 实现的伪线性多步扩散采样方法。

Figure 16: Comparison of GLIDE and LAIONIDE-v3 Generations. We compare the output of GLIDE and our LAIONIDE-v3 across three different prompts. The top row of each section depicts GLIDE’s results, while the bottom row depicts LAIONIDE-v3’s resutls.

图 16： GLIDE 与 LAIONIDE-v3 生成结果的比较 。 我们在三条不同提示上比较 GLIDE 与我们的 LAIONIDE-v3 的输出。每一部分的上一行是 GLIDE 的结果，下一行是 LAIONIDE-v3 的结果。

We compare some evaluations from OpenAI’s released filtered checkpoint and the one we train. Those can be found at the following link: https://wandb.ai/afiaka87/glide_compare/reports/laionide-v3-benchmark--VmlldzoxNTg3MTkz

我们比较了 OpenAI 发布的过滤版检查点与我们训练的检查点的若干评估。结果见以下链接： https://wandb.ai/afiaka87/glide_compare/reports/laionide-v3-benchmark--VmlldzoxNTg3MTkz

F.2 Stable Diffusion

Stable Diffusion is a generative latent diffusion model trained on various LAION-5B subsets:

Stable Diffusion 是在 LAION-5B 的多个子集上训练的生成式潜扩散模型：

• 237,000 steps at 256x256 on LAION-2B-en

• 在 LAION-2B-en 上以 256\times256 训练 237,000 步

• 194,000 steps at 512x512 on laion-high-resolution

• 在 laion-high-resolution 上以 512\times512 训练 194,000 步

• 515,000 steps at 512x512 on laion-improved-aesthetics

• 在 laion-improved-aesthetics 上以 512\times512 训练 515,000 步

• 390,000 steps at 512x512 on laion-improved-aesthetics with 10% dropping of the text conditioning

• 在 laion-improved-aesthetics 上以 512\times512 训练 390,000 步，同时以 10% 的概率丢弃文本条件

Here we show representative generated samples for an artistic (Fig. 17) and a photorealistic (Fig. 18) image. For more technical details, we refer to the Stable Diffusion github repository[^36].

这里展示有代表性的生成样本：一幅偏艺术的图像（图 17）和一幅偏写实的图像（图 18）。更多技术细节见 Stable Diffusion 的 GitHub 仓库[^36]。

Figure 17: **"The sigil of water by Gerardo Dottori, oil on canvas"**Generated by Stable Diffusion

图 17：“ The sigil of water by Gerardo Dottori, oil on canvas ”由 Stable Diffusion 生成

Figure 18: **"A wide river in the jungle, Provia, Velvia"**Generated by Stable Diffusion

图 18：“ A wide river in the jungle, Provia, Velvia ”由 Stable Diffusion 生成

Appendix G Further Discussion on Safety and Ethics

附录 G 关于安全与伦理的进一步讨论

G.1 Privacy

G.1 隐私

As any other dataset of links obtained from Common Crawl that gathers content from publicly available Internet, LAION-5B can contain links to images with personal information, like to photos of faces, medical images or other personal related content. Tools like CLIP retrieval (see Appendix Section C.4 for more details) provided by LAION make it possible for the users to find out by text or image prompt whether any of the links crawled for LAION-5B point to their personal data and if yes, where on the public internet the corresponding data is hosted. Thus, for the first time, the broad public can take a look inside of a typical large-scale crawled dataset and become aware of the possible content of datasets that can be used for model training. As most of institutions and companies use same crawling procedures to obtain their closed datasets, we thus also hope to increase awareness for the risks which publicly available data can be used and exploited by third parties who do not disclose their data collection and application procedures. At the same time, researchers can access LAION-5B to study privacy related issues in such data and develop measures that increase safety of applications arising from training models on data crawled from public internet.

与其他从 Common Crawl 获得、汇集公开互联网内容的链接数据集一样，LAION-5B 可能包含指向带有个人信息的图像的链接，例如人脸照片、医学图像或其他与个人相关的内容。LAION 提供的 CLIP 检索等工具（细节见附录 C.4）让使用者可以通过文本或图像提示，查明为 LAION-5B 抓取的链接是否指向其个人数据；如果是，对应数据又托管在公开互联网的何处。这样，更广泛的公众第一次可以查看一套典型的大规模抓取数据集的内部，并意识到这类可能用于模型训练的数据里会有什么。由于大多数机构和公司采用同样的抓取流程来获得其封闭数据集，我们也希望借此提高人们的警觉：公开可获得的数据可能被那些并不披露其采集和应用流程的第三方使用和利用。与此同时，研究者可以访问 LAION-5B，研究这类数据中的隐私问题，并发展相应措施，提高在从公开互联网抓取的数据上训练模型所产生的应用的安全性。

As LAION tools empower people to discover problematic personal or copyrighted content available in the public internet, the users can also initiate procedures of removing corresponding images from the public internet by contacting the responsible host providers that have published those images following the links provided in LAION-5B. In addition, we also provide a contact form on our website [^37] where requests for removal or blacklisting of the corresponding links from LAION-5B can be processed.

由于 LAION 的工具帮助人们发现公开互联网上存在问题的个人内容或受著作权保护的内容，使用者也可以联系负责托管这些图像的服务商，启动从公开互联网删除相应图像的程序；链接由 LAION-5B 提供。此外，我们还在网站上提供联系表单[^37]，用于处理从 LAION-5B 中移除或拉黑相应链接的请求。

Further, to mitigate privacy concerns, there exist methods that allow personal human attributes like faces to be obfuscated [87] or generated [48] and thus made anonymous, without hurting the quality and richness of learned representations. Especially generation based methods can be applied to open data like LAION-5B to create training datasets that do not contain any private facial data, while still allowing to learn proper face representation during training. This line of work is currently in progress in LAION community.

此外，为减轻隐私顾虑，已有方法可以对人脸等个人属性做模糊处理[87]或生成替换[48]，从而使其匿名，同时不损害所学表示的质量和丰富程度。尤其是基于生成的方法，可以应用到 LAION-5B 这类开放数据上，创建不含任何私人面部数据的训练数据集，同时仍允许模型在训练中学到恰当的面部表示。LAION 社区目前正在推进这一方向的工作。

G.2 Potential Biases Induced by CLIP Filtering

G.2 CLIP 过滤可能引入的偏见

Unknown initial dataset. The CLIP model in itself introduces a bias, which cannot be trivially assessed, as the underlying dataset on which the model was trained is not openly accessible. With the release of a large openly accessible image-text dataset, we offer a starting point in the open auditing of contrastive image-text models like CLIP.

未知的初始数据集 。 CLIP 模型本身会引入一种偏见，而这种偏见很难直接评估，因为该模型所训练的底层数据并未开放。通过发布大规模开放图文数据集，我们为公开审计 CLIP 这类对比式图文模型提供了一个起点。

Selection heuristic based on cosine similarity. As noted by [6], cosine similarity is only a heuristic that also may lead to suboptimal guidance for dataset filtering. The work showed examples in which captions with malignant descriptions obtain a higher similarity over a benign description. During CLIP’s training, the cosine similarity only acted as a logit to represent the likelihood of a given image-text pairing. It fails to encapsulate the nuance and rich semantic and contextual meaning that the image or language might contain. By using cosine similarity as a ground for filtering, the dataset might exacerbate those biases already contained by CLIP.

基于余弦相似度的选择启发式 。 正如[6]所指出的，余弦相似度只是一种启发式规则，用于指导数据过滤时也可能并非最优。该工作展示了一些例子：带有恶意描述的说明会比较为中性的描述获得更高的相似度。在 CLIP 的训练中，余弦相似度只是作为 logit，用来表示给定图文配对的可能性。它并不能概括图像或语言可能包含的细微差别，以及丰富的语义和上下文含义。把余弦相似度当作过滤依据，可能会加剧 CLIP 中本已存在的那些偏见。

Author contributions

作者贡献

• Christoph Schuhmann : He led this project and built POCs for most of its components including clip filtering, the safety model, the watermark model and the BLIP inference tuning project.

• Christoph Schuhmann ：他负责领导本项目，并为其中大多数组件构建了概念验证，包括 CLIP 过滤、安全模型、水印模型以及 BLIP 推理调参项目。

• Richard Vencu : System architecture and download script optimizations, GPU assisted filtering. Set up the AWS infrastructure.

• Richard Vencu ：系统架构与下载脚本优化，以及 GPU 辅助过滤。搭建了 AWS 基础设施。

• Romain Beaumont : Guidance on scaling for the Common Crawl filtering pipeline. Built and ran the dataset preparation pipeline: pyspark deduplication job, img2dataset, CLIP inference, autofaiss, safety tags.

• Romain Beaumont ：为 Common Crawl 过滤流水线的扩展提供指导。构建并运行了数据准备流水线：pyspark 去重任务、img2dataset、CLIP 推理、autofaiss、安全标签。

• Clayton Mullis : DALLE-pytorch training/analysis, WDS filtering, trained generative models (LAIONIDE) using LAION-5B.

• Clayton Mullis ：DALLE-pytorch 的训练与分析、WDS 过滤，并用 LAION-5B 训练生成模型（LAIONIDE）。

• Ludwig Schmidt : Provided advice on experiment design, scaling, ethical and social content, and paper writing.

• Ludwig Schmidt ：为实验设计、规模扩展、伦理与社会内容以及论文写作提供建议。

• Jenia Jitsev : scientific organization & manuscript writing, ethical and social content, experiments planning and design, compute and storage resource acquisition, general supervision.

• Jenia Jitsev ：科学组织与手稿撰写，伦理与社会内容，实验规划与设计，计算和存储资源争取，以及总体指导。

• Robert Kaczmarczyk : Established WDS architecture, performed DALL-E training runs, balancing calculation, sample (NSFW, watermark, caption quality) annotation, manuscript writing coordination, supervision and revision.

• Robert Kaczmarczyk ：建立 WDS 架构，执行 DALL-E 训练，做平衡计算，标注样本（NSFW、水印、说明质量），并协调、指导和修订手稿撰写。

• Theo Coombes : He was one of our first contributors & build the first versions of our worker swarm system. Without his enthusiasm this project might never have taken off.

• Theo Coombes ：他是我们最早的贡献者之一，并搭建了工作者集群系统的最初版本。没有他的热情，这个项目或许不会起步。

• Aarush Katta : Trained the watermark model.

• Aarush Katta ：训练了水印模型。

• Cade Gordon : Ran distributed inference for the watermark tags, trained the CLIP models on JUWELS Booster, and led the paper writing.

• Cade Gordon ：为水印标签运行分布式推理，在 JUWELS Booster 上训练 CLIP 模型，并主持论文撰写。

• Mehdi Cherti : Evaluated the CLIP-B/32, B/16, B/16+ and L/14 model, performed debugging of distributed training, executed experiments on JUWELS Booster, performed results collection, distillation and analysis, manuscript writing.

• Mehdi Cherti ：评估 CLIP-B/32、B/16、B/16+ 和 L/14 模型，调试分布式训练，在 JUWELS Booster 上执行实验，收集、提炼并分析结果，并参与手稿撰写。

• Ross Wightman : Ross debugged & trained the CLIP-B/32, B/16, B/16+ and L/14 model and executed experiments on JUWELS Booster.

• Ross Wightman ：调试并训练 CLIP-B/32、B/16、B/16+ 和 L/14 模型，并在 JUWELS Booster 上执行实验。

• Katherine Crowson : Contributed to development of latent diffusion and stable diffusion. Fine-tuned generative models on subsets of LAION-5B.

• Katherine Crowson ：参与潜扩散和 Stable Diffusion 的开发。在 LAION-5B 的子集上微调生成模型。

• Patrick Schramowski : Patrick helped with NSFW and otherwise inappropriate content tagging. Further, he wrote the corresponding parts as well as the ethical and social content.

• Patrick Schramowski ：帮助为 NSFW 及其他不宜内容打标，并撰写了相应部分以及伦理与社会内容。

• Srivatsa Kundurthy : Co-wrote the datasheet, researched usage cases & related works, trained face classifier and developed visualizations.

• Srivatsa Kundurthy ：参与撰写数据集说明书，调研用例和相关工作，训练人脸分类器并制作可视化。

• Mitchell Wortsman Initially created openCLIP, provided insights on scaling, performed experiments evaluating few-shot fine-tuning performance and robustness on ImageNet and other downstream datasets

• Mitchell Wortsman ：最初创建了 openCLIP，提供了关于规模扩展的见解，并开展实验，评估 ImageNet 及其他下游数据集上的少样本微调表现与稳健性。

Acknowledgments details

致谢细节

We want to thank our open community for their continuous efforts for openly available datasets and models. Without the broad support from the community, especially in the early crawling days with decentralized compute support, this project would not have been possible.

我们要感谢开放社区为可公开获取的数据集和模型持续付出的努力。没有社区的广泛支持，尤其是在早期抓取阶段提供的去中心化计算支持，这个项目不可能完成。

Moreover, the following organizations and persons contributed to this project:

此外，以下组织和个人为本项目做出了贡献：

• Aran Komatsuzaki : He led the initial crawling@home image-text-pair dataset building project (the predecessor of LAION-400M).

• Aran Komatsuzaki ：他领导了最初的 crawling@home 图文对数据集构建项目（LAION-400M 的前身）。

• Andreas Köpf : He conducted the hyperparameter search for the inference strategies with the BLIP image-captioning model.

• Andreas Köpf ：他为 BLIP 图像描述模型的推理策略做了超参数搜索。

• Bokai Yu : Accomplished most of the work to make the knn index building tool autofaiss work in a distributed setting.

• Bokai Yu ：完成了使近邻索引构建工具 autofaiss 能够在分布式环境中工作的大部分工作。

• John David Pressman : Provided aestethic dataset for creating aestethic LAION subset to fine-tune GLIDE.

• John David Pressman ：提供了审美数据集，用于创建审美 LAION 子集以微调 GLIDE。

• Natalie Parde Assisted in manuscript revisions.

• Natalie Parde 协助修订手稿。

• Gabriel Ilharco Initially created OpenCLIP and gave valuable insights on scaling.

• Gabriel Ilharco 最初创建了 OpenCLIP，并就规模扩展提供了有价值的见解。

• Fredde Frallan Provided zero-shot retrieval results.

• Fredde Frallan 提供了零样本检索结果。

• Hugging Face : provided financial and computing support, helped hosting LAION-5B as well as related subsets.

• Hugging Face ：提供资金和计算支持，并帮助托管 LAION-5B 及相关子集。

• Emad Mostaque (Stability AI) : provided financial and computation support for open-source datasets and models.

• Emad Mostaque（Stability AI） ：为开源数据集和模型提供资金和计算支持。