Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Colin Raffel craffel@gmail.com Thanks: Equal contribution. A description of each author’s contribution is available in Appendix A . Correspondence to craffel@gmail.com . Noam Shazeer noam@google.com Adam Roberts adarob@google.com Katherine Lee katherinelee@google.comSharan Narang sharannarang@google.comMichael Matena mmatena@google.comYanqi Zhou yanqiz@google.comWei Li mweili@google.comPeter J. Liu peterjliu@google.comGoogle, Mountain View, CA 94043, USA

Abstract

摘要

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ‘‘Colossal Clean Crawled Corpus’’, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.[^1]

迁移学习先在数据丰富的任务上预训练模型，再在下游任务上微调，在几乎所有自然语言处理（NLP）任务上已成为主流。得益于现代深度神经网络中层数与参数量的提升，以及计算与可用文本语料规模的扩大，迁移学习在 NLP 中的有效性近年来显著提高。本文探索这一系列的边界：通过引入统一框架，把每个语言问题都转换成文本输入到文本输出的格式，系统研究并比较迁移学习的预训练目标、架构、无标注数据语料、迁移方法与其他因素。通过把我们的发现推到极限，我们在涵盖问题回答、文本摘要、文本分类等许多基准上达到最优结果，同时对 NLP 迁移学习的现状做了新的综述。为便于后续研究，我们开源了为本研究开发的数据集、预训练模型与代码。

1 Introduction

1 引言

keywords Training a machine learning model to perform natural language processing (NLP) tasks often requires that the model can process text in a way that is amenable to downstream learning. This can be loosely viewed as developing general-purpose knowledge that allows the model to “understand” text. This knowledge can range from low-level (e.g. the spelling or meaning of words) to high-level (e.g. that a tuba is too large to fit in most backpacks). In modern machine learning practice, providing this knowledge is rarely done explicitly; instead, it is often learned as part of an auxiliary task. For example, a historically common approach is to use word vectors (Mikolov et al. 2013b; Mikolov et al. 2013a; Pennington et al. 2014) to map word identities to a continuous representation where, ideally, similar words map to similar vectors. These vectors are often learned through an objective that, for example, encourages co-occurring words to be positioned nearby in the continuous space (Mikolov et al. 2013b).

关键词 训练机器学习模型完成自然语言处理（NLP）任务，过去通常是为每项任务从零收集标注数据。这带来两大挑战。第一，对许多任务（尤其是低资源语言上的任务）很难获得大量标注数据。第二，为每项任务单独训练模型，会忽略可从其他任务与无标注数据中学到的共性。迁移学习的兴起旨在缓解这些问题：先在数据丰富的任务上预训练模型，再在感兴趣的下游任务上微调。

Recently, it has become increasingly common to pre-train the entire model on a data-rich task. Ideally, this pre-training causes the model to develop general-purpose abilities and knowledge that can then be transferred to downstream tasks. In applications of transfer learning to computer vision (Oquab et al. 2014; Jia et al. 2014; Huh et al. 2016; Yosinski et al. 2014), pre-training is typically done via supervised learning on a large labeled data set like ImageNet (Russakovsky et al. 2015; Deng et al. 2009). In contrast, modern techniques for transfer learning in NLP often pre-train using unsupervised learning on unlabeled data. This approach has recently been used to obtain state-of-the-art results in many of the most common NLP benchmarks (Devlin et al. 2018; Yang et al. 2019; Dong et al. 2019; Liu et al. 2019c; Lan et al. 2019). Beyond its empirical strength, unsupervised pre-training for NLP is particularly attractive because unlabeled text data is available en masse thanks to the Internet---for example, the Common Crawl project[^2] produces about 20TB of text data extracted from web pages each month. This is a natural fit for neural networks, which have been shown to exhibit remarkable scalability, i.e. it is often possible to achieve better performance simply by training a larger model on a larger data set (Hestness et al. 2017; Shazeer et al. 2017; Jozefowicz et al. 2016; Mahajan et al. 2018; Radford et al. 2019; Shazeer et al. 2018; Huang et al. 2018b; Keskar et al. 2019a).

近年来，越来越常见的做法是在大量无标注文本上预训练整个模型（或绝大部分），再用下游任务标注数据微调。这一范式的成功，部分来自现代深度神经网络可扩展的层数与参数量，以及可扩展的计算与可用文本语料。结果是：在几乎所有 NLP 基准上，迁移学习方法的表现都显著提升。

This synergy has resulted in a great deal of recent work developing transfer learning methodology for NLP, which has produced a wide landscape of pre-training objectives (Howard and Ruder 2018; Devlin et al. 2018; Yang et al. 2019; Dong et al. 2019), unlabeled data sets (Yang et al. 2019; Liu et al. 2019c; Zellers et al. 2019), benchmarks (Wang et al. 2019b; Wang et al. 2018; Conneau and Kiela 2018), fine-tuning methods (Howard and Ruder 2018; Houlsby et al. 2019; Peters et al. 2019), and more. The rapid rate of progress and diversity of techniques in this burgeoning field can make it difficult to compare different algorithms, tease apart the effects of new contributions, and understand the space of existing methods for transfer learning. Motivated by a need for more rigorous understanding, we leverage a unified approach to transfer learning that allows us to systematically study different approaches and push the current limits of the field.

这种协同催生了大量开发迁移学习方法与模型的近期工作，彼此在架构、预训练目标、无标注数据语料、微调方法等方面存在差异。由此自然提出一个问题：哪些因素对迁移学习在 NLP 中的成功最为重要？本文通过引入统一框架，把每项任务都转换成文本输入到文本输出的格式，系统研究并比较这些因素。

Figure 1: A diagram of our text-to-text framework. Every task we consider—including translation, question answering, and classification—is cast as feeding our model text as input and training it to generate some target text. This allows us to use the same model, loss function, hyperparameters, etc. across our diverse set of tasks. It also provides a standard testbed for the methods included in our empirical survey. “T5” refers to our model, which we dub the “ T ext- t o- T ext T ransfer T ransformer”.

图 1：我们的文本到文本框架示意图。我们考虑的每项任务——包括翻译、问题回答与分类——都以输入文本喂给模型，训练其生成目标文本。图中展示了翻译、共指消解与语义蕴含任务的输入与目标文本例子。模型本身为编码器—解码器 Transformer。

The basic idea underlying our work is to treat every text processing problem as a “text-to-text” problem, i.e. taking text as input and producing new text as output. This approach is inspired by previous unifying frameworks for NLP tasks, including casting all text problems as question answering (McCann et al. 2018), language modeling (Radford et al. 2019), or span extraction Keskar et al. 2019b tasks. Crucially, the text-to-text framework allows us to directly apply the same model, objective, training procedure, and decoding process to every task we consider. We leverage this flexibility by evaluating performance on a wide variety of English-based NLP problems, including question answering, document summarization, and sentiment classification, to name a few. With this unified approach, we can compare the effectiveness of different transfer learning objectives, unlabeled data sets, and other factors, while exploring the limits of transfer learning for NLP by scaling up models and data sets beyond what has previously been considered.

我们工作的基本想法是把每个文本处理问题都当作「文本到文本」问题——即以文本为输入、生成新文本作为输出。这一框架使我们能直接对翻译、问题回答、文本分类等任何文本任务应用同一模型、目标、训练流程与解码过程。我们称之为 T5，即「Text-to-Text Transfer Transformer」。

11

我们强调：目标不是提出新方法，而是对已有技术做全面研究，并在可能时把规模推到极限。因此，我们的大部分实验都在比较先前提出的不同方法。我们将比较预训练目标、架构、无标注数据语料、迁移方法以及其他因素。最终我们把发现推到极限，在许多基准上达到最优结果。

The remainder of the paper is structured as follows: In the following section, we discuss our base model and its implementation, our procedure for formulating every text processing problem as a text-to-text task, and the suite of tasks we consider. In Section 3, we present a large set of experiments that explore the field of transfer learning for NLP. At the end of the section (Section 3.7), we combine insights from our systematic study to obtain state-of-the-art results on a wide variety of benchmarks. Finally, we provide a summary of our results and wrap up with a look towards the future in Section 4.

本文其余部分结构如下：下一节介绍我们的文本到文本框架与实验所用模型结构。第 3 节给出大规模实证研究结果。第 4 节把发现推到极限，在许多基准上达到最优。我们在第 5 节讨论相关工作，并在第 6 节总结。

2 Setup

2 设定

Before presenting the results from our large-scale empirical study, we review the necessary background topics required to understand our results, including the Transformer model architecture and the downstream tasks we evaluate on. We also introduce our approach for treating every problem as a text-to-text task and describe our “Colossal Clean Crawled Corpus” (C4), the Common Crawl-based data set we created as a source of unlabeled text data. We refer to our model and framework as the “ T ext- t o- T ext T ransfer T ransformer” (T5).

在给出大规模实证研究结果之前，我们先描述实验所用基本成分：模型结构、用于无监督预训练的数据语料、下游任务集合，以及把每项任务都转换成文本到文本格式的方式。

2.1 Model

2.1 模型

Early results on transfer learning for NLP leveraged recurrent neural networks (Peters et al. 2018; Howard and Ruder 2018), but it has recently become more common to use models based on the “Transformer” architecture (Vaswani et al. 2017). The Transformer was initially shown to be effective for machine translation, but it has subsequently been used in a wide variety of NLP settings (Radford et al. 2018; Devlin et al. 2018; McCann et al. 2018; Yu et al. 2018). Due to its increasing ubiquity, all of the models we study are based on the Transformer architecture. Apart from the details mentioned below and the variants we explore in Section 3.2, we do not deviate significantly from this architecture as originally proposed. Instead of providing a comprehensive definition of this model, we refer the interested reader to the original paper (Vaswani et al. 2017) or follow-up tutorials[^3],[^4] for a more detailed introduction.

早期 NLP 迁移学习结果利用了循环神经网络（Sutskever et al. 2014；Cho et al. 2014）与注意力机制（Bahdanau et al. 2015）。近期工作则主要使用 Transformer 架构（Vaswani et al. 2017），它在机器翻译上达到当时最优，并迅速成为其他 NLP 任务的主流模型。

The primary building block of the Transformer is self-attention (Cheng et al. 2016). Self-attention is a variant of attention (Graves 2013; Bahdanau et al. 2015) that processes a sequence by replacing each element by a weighted average of the rest of the sequence. The original Transformer consisted of an encoder-decoder architecture and was intended for sequence-to-sequence (Sutskever et al. 2014; Kalchbrenner et al. 2014) tasks. It has recently also become common to use models consisting of a single Transformer layer stack, with varying forms of self-attention used to produce architectures appropriate for language modeling (Radford et al. 2018; Al-Rfou et al. 2019) or classification and span prediction tasks (Devlin et al. 2018; Yang et al. 2019). We empirically explore these architectural variants in Section 3.2.

Transformer 的主要积木块是自注意力（Cheng et al. 2016；Parikh et al. 2016；Lin et al. 2017）。自注意力是一种注意力机制，把同一序列的不同位置关联起来，以计算该序列的表示。Transformer 由堆叠的自注意力层与逐位置前馈网络组成，并配合残差连接（He et al. 2016）与层归一化（Ba et al. 2016）。

Overall, our encoder-decoder Transformer implementation closely follows its originally-proposed form (Vaswani et al. 2017). First, an input sequence of tokens is mapped to a sequence of embeddings, which is then passed into the encoder. The encoder consists of a stack of “blocks”, each of which comprises two subcomponents: a self-attention layer followed by a small feed-forward network. Layer normalization (Ba et al. 2016) is applied to the input of each subcomponent. We use a simplified version of layer normalization where the activations are only rescaled and no additive bias is applied. After layer normalization, a residual skip connection (He et al. 2016) adds each subcomponent’s input to its output. Dropout (Srivastava et al. 2014) is applied within the feed-forward network, on the skip connection, on the attention weights, and at the input and output of the entire stack. The decoder is similar in structure to the encoder except that it includes a standard attention mechanism after each self-attention layer that attends to the output of the encoder. The self-attention mechanism in the decoder also uses a form of autoregressive or causal self-attention, which only allows the model to attend to past outputs. The output of the final decoder block is fed into a dense layer with a softmax output, whose weights are shared with the input embedding matrix. All attention mechanisms in the Transformer are split up into independent “heads” whose outputs are concatenated before being further processed.

总体而言，我们的编码器—解码器 Transformer 实现紧密遵循最初的实现（Vaswani et al. 2017）。细节见附录。一个主要差异是我们移除了 Layer Norm 中的偏置，并把 Layer Norm 放在每个子块的输入处，再与残差连接结合使用（即「pre-norm」形式）。我们还使用相对位置嵌入（Shaw et al. 2018；Huang et al. 2019），而非原始 Transformer 中的正弦位置信号或学习到的位置嵌入。

32

由于自注意力与顺序无关（即它是集合上的操作），模型对序列中 token 顺序没有内置概念。因此，常见做法是向输入加入位置信号。我们使用简化的相对位置嵌入形式，其中每个嵌入是标量，加到对应的 logit 上，用于计算注意力权重。为高效处理任意长度的输入，我们把输入相对偏移量分到 32 个桶中，桶在对数尺度上增长直至偏移 128，之后所有更大偏移共用同一嵌入。与 Huang et al. (2019) 不同，我们在所有层共享位置嵌入参数，总体发现这带来相近性能，同时减少需要存储的参数量。

1{,}024

作为研究的一部分，我们实验这些模型的可扩展性。具体地，我们考虑编码器与解码器各有 6、12、18 或 24 层的模型，以及「base」「large」「3B」「11B」参数量变体。除非另有说明，所有实验使用「base」模型，编码器与解码器各 12 层。

2.2 The Colossal Clean Crawled Corpus

2.2 超大规模清洁爬取语料（C4）

Much of the previous work on transfer learning for NLP makes use of large unlabeled data sets for unsupervised learning. In this paper, we are interested in measuring the effect of the quality, characteristics, and size of this unlabeled data. To generate data sets that satisfy our needs, we leverage Common Crawl as a source of text scraped from the web. Common Crawl has previously been used as a source of text data for NLP, for example to train an n-gram language model (Buck et al. 2014), as training data for commonsense reasoning (Trinh and Le 2018), for mining parallel texts for machine translation (Smith et al. 2013), as a pre-training data set (Grave et al. 2018; Zellers et al. 2019; Liu et al. 2019c), and even simply as a giant text corpus for testing optimizers (Anil et al. 2019).

先前 NLP 迁移学习工作大量使用无标注文本数据。常见做法是在 Wikipedia、图书语料等相对干净的数据源上预训练。我们也实验这些数据源，但主要关注从 Common Crawl 网络爬取得到的数据——其规模可比其他语料大约两个数量级。

Common Crawl is a publicly-available web archive that provides “web extracted text” by removing markup and other non-text content from the scraped HTML files. This process produces around 20TB of scraped text data each month. Unfortunately, the majority of the resulting text is not natural language. Instead, it largely comprises gibberish or boiler-plate text like menus, error messages, or duplicate text. Furthermore, a good deal of the scraped text contains content that is unlikely to be helpful for any of the tasks we consider (offensive language, placeholder text, source code, etc.). To address these issues, we used the following heuristics for cleaning up Common Crawl’s web extracted text:

Common Crawl 是公开可用的网络存档，提供「网络提取文本」语料。原始 Common Crawl 数据包含大量不可用文本（样板、菜单、错误信息、重复内容等）。因此，我们用启发式方法过滤数据，得到我们称为 Colossal Clean Crawled Corpus（C4）的语料。过滤步骤如下：

• We only retained lines that ended in a terminal punctuation mark (i.e. a period, exclamation mark, question mark, or end quotation mark).

• 只保留以终端标点（即句号、感叹号、问号或结尾引号）结尾的行。

• We discarded any page with fewer than 3 sentences and only retained lines that contained at least 5 words.

• 丢弃句子少于 3 句的页面，且每行只保留至少 5 个词的句子。

• We removed any page that contained any word on the ‘‘List of Dirty, Naughty, Obscene or Otherwise Bad Words’’.[^6]

• 移除包含「List of Dirty, Naughty, and Other Bad Words」（污言秽语词表）中任一词的页面。

• Many of the scraped pages contained warnings stating that Javascript should be enabled so we removed any line with the word Javascript.

• 许多抓取页面含有声明 Javascript 应开启的警告；我们移除提及「javascript」的页面。

• Some pages had placeholder “lorem ipsum” text; we removed any page where the phrase “lorem ipsum” appeared.

• 有些页面含占位「lorem ipsum」文本；我们移除提及「lorem ipsum」的页面。

• Some pages inadvertently contained code. Since the curly bracket “{” appears in many programming languages (such as Javascript, widely used on the web) but not in natural text, we removed any pages that contained a curly bracket.

• 有些页面无意中包含代码。由于花括号「{」在典型英文散文中极少出现，我们移除任何含「{」的页面。

• Since some of the scraped pages were sourced from Wikipedia and had citation markers (e.g. [1], [citation needed], etc.), we removed any such markers.

• 由于部分抓取页面来自 Wikipedia 并含政策声明，我们移除以「Wikipedia」开头或含「Terms of Use」等常见样板短语的行。

• Many pages had boilerplate policy notices, so we removed any lines containing the strings “terms of use”, “privacy policy”, “cookie policy”, “uses cookies”, “use of cookies”, or “use cookies”.

• 许多页面有样板政策声明，因此我们移除含「cookies」「terms of use」「privacy policy」等短语的行。

• To deduplicate the data set, we discarded all but one of any three-sentence span occurring more than once in the data set.

• 为去重，我们丢弃在三句滑动窗口内完全重复的内容，只保留一份。

Additionally, since most of our downstream tasks are focused on English-language text, we used langdetect[^7] to filter out any pages that were not classified as English with a probability of at least 0.99. Our heuristics are inspired by past work on using Common Crawl as a source of data for NLP: For example, Grave et al. 2018 also filter text using an automatic language detector and discard short lines and Smith et al. 2013; Grave et al. 2018 both perform line-level deduplication. However, we opted to create a new data set because prior data sets use a more limited set of filtering heuristics, are not publicly available, and/or are different in scope (e.g. are limited to News data (Zellers et al. 2019; Liu et al. 2019c), comprise only Creative Commons content (Habernal et al. 2016), or are focused on parallel training data for machine translation (Smith et al. 2013)).

此外，由于多数下游任务聚焦英语，我们用 langdetect 做语言检测，并过滤掉非英语页面。应用这些过滤器后，我们得到约 750 GB 干净英文文本。我们开源了 C4 以及用于创建它的代码。

To assemble our base data set, we downloaded the web extracted text from April 2019 and applied the aforementioned filtering. This produces a collection of text that is not only orders of magnitude larger than most data sets used for pre-training (about 750 GB) but also comprises reasonably clean and natural English text. We dub this data set the “ C olossal C lean C rawled C orpus’’ (or C4 for short) and release it as part of TensorFlow Datasets.[^8] We consider the impact of using various alternative versions of this data set in Section 3.4.

为组装基础数据集，我们下载了 2019 年 4 月的网络提取文本，并应用上述启发式。所得语料足够大，使我们能在预训练中几乎不重复地喂数据——对评估模型是否从更多数据中受益很重要。

2.3 Downstream Tasks

2.3 下游任务

Our goal in this paper is to measure general language learning abilities. As such, we study downstream performance on a diverse set of benchmarks, including machine translation, question answering, abstractive summarization, and text classification. Specifically, we measure performance on the GLUE and SuperGLUE text classification meta-benchmarks; CNN/Daily Mail abstractive summarization; SQuAD question answering; and WMT English to German, French, and Romanian translation. All data was sourced from TensorFlow Datasets.[^9]

本文目标是测量一般语言学习能力，因此我们在广泛的英语 NLP 任务上评估：自然语言理解基准 GLUE 与 SuperGLUE、CNN/Daily Mail 摘要、SQuAD 问题回答，以及 WMT 英德、英法、英罗翻译。

GLUE (Wang et al. 2018) and SuperGLUE (Wang et al. 2019b) each comprise a collection of text classification tasks meant to test general language understanding abilities:

GLUE（Wang et al. 2018）与 SuperGLUE（Wang et al. 2019b）各包含一组需理解自然语言的文本分类任务。具体包括：

• Sentence acceptability judgment (CoLA (Warstadt et al. 2018))

• 句子可接受性判断（CoLA（Warstadt et al. 2018））

• Sentiment analysis (SST-2 (Socher et al. 2013))

• 情感分析（SST-2（Socher et al. 2013））

• Paraphrasing/sentence similarity (MRPC (Dolan and Brockett 2005), STS-B (Cer et al. 2017), QQP (Iyer et al. 2017))

• 释义/句子相似度（MRPC（Dolan and Brockett 2005）、STS-B（Cer et al. 2017）、QQP）

• Natural language inference (MNLI (Williams et al. 2017), QNLI (Rajpurkar et al. 2016), RTE (Dagan et al. 2005), CB (De Marneff et al. 2019))

• 自然语言推断（MNLI（Williams et al. 2017）、QNLI（Rajpurkar et al. 2016）、RTE（Dagan et al. 2005 等）、CB（De Marneffe et al. 2019））

• Coreference resolution (WNLI and WSC (Levesque et al. 2012))

• 共指消解（WNLI 与 WSC（Levesque et al. 2012））

• Sentence completion (COPA (Roemmele et al. 2011))

• 句子补全（COPA（Roemmele et al. 2011））

• Word sense disambiguation (WIC (Pilehvar and Camacho-Collados 2018))

• 词义消歧（WIC（Pilehvar and Camacho-Collados 2018））

• Question answering (MultiRC (Khashabi et al. 2018), ReCoRD (Zhang et al. 2018), BoolQ (Clark et al. 2019))

• 问题回答（MultiRC（Khashabi et al. 2018）、ReCoRD（Zhang et al. 2018）、BoolQ（Clark et al. 2019））

We use the data sets as distributed by the GLUE and SuperGLUE benchmarks. For simplicity, when fine-tuning we treat all of the tasks in the GLUE benchmark (and similarly for SuperGLUE) as a single task by concatenating all of the constituent data sets. As suggested by Kocijan et al. 2019 we also include the Definite Pronoun Resolution (DPR) data set (Rahman and Ng 2012) in the combined SuperGLUE task.

我们使用 GLUE 与 SuperGLUE 基准发布的数据集。对 GLUE，我们报告除 WNLI 外所有任务的平均分（按该基准惯例）。对 SuperGLUE，我们报告所有任务平均分。

The CNN/Daily Mail (Hermann et al. 2015) data set was introduced as a question-answering task but was adapted for text summarization by Nallapati et al. 2016; we use the non-anonymized version from See et al. 2017 as an abstractive summarization task. SQuAD (Rajpurkar et al. 2016) is a common question-answering benchmark. In our experiments, the model is fed the question and its context and asked to generate the answer token-by-token. For WMT English to German, we use the same training data as (Vaswani et al. 2017) (i.e. News Commentary v13, Common Crawl, Europarl v7) and newstest2013 as a validation set (Bojar et al. 2014). For English to French, we use the standard training data from 2015 and newstest2014 as a validation set (Bojar et al. 2015). For English to Romanian, which is a standard lower-resource machine translation benchmark, we use the train and validation sets from WMT 2016 (Bojar et al. 2016). Note that we only pre-train on English data, so in order to learn to translate a given model will need to learn to generate text in a new language.

CNN/Daily Mail（Hermann et al. 2015）数据集最初作为阅读理解任务引入，现常用作摘要基准（Nallapati et al. 2016；See et al. 2017）。SQuAD（Rajpurkar et al. 2016）是阅读理解数据集，包含维基百科文章段落与对应问题，答案是段落中的文本跨度。我们还评估 WMT 英德、英法与英罗翻译（Bojar et al. 2014；2015；2016）。

2.4 Input and Output Format

2.4 输入与输出格式

In order to train a single model on the diverse set of tasks described above, we cast all of the tasks we consider into a “text-to-text” format—that is, a task where the model is fed some text for context or conditioning and is then asked to produce some output text. This framework provides a consistent training objective both for pre-training and fine-tuning. Specifically, the model is trained with a maximum likelihood objective (using “teacher forcing” (Williams and Zipser 1989)) regardless of the task. To specify which task the model should perform, we add a task-specific (text) prefix to the original input sequence before feeding it to the model.

为在上述多样任务上训练单一模型，我们把每项任务都转换成文本到文本格式。也就是说，模型以文本为输入，训练其生成目标文本。这使我们能对每项任务使用完全相同的模型、损失函数、超参与训练流程。

As an example, to ask the model to translate the sentence “That is good.” from English to German, the model would be fed the sequence “translate English to German: That is good.” and would be trained to output “Das ist gut.” For text classification tasks, the model simply predicts a single word corresponding to the target label. For example, on the MNLI benchmark (Williams et al. 2017) the goal is to predict whether a premise implies (“entailment”), contradicts (“contradiction”), or neither (“neutral”) a hypothesis. With our preprocessing, the input sequence becomes “mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons are filled with animosity.” with the corresponding target word “entailment”. Note that an issue arises if our model outputs text on a text classification task that does not correspond to any of the possible labels (for example if the model outputs “hamburger” when the only possible labels for a task were “entailment”, “neutral”, or “contradiction”). In this case, we always count the model’s output as wrong, though we never observed this behavior in any of our trained models. Note that the choice of text prefix used for a given task is essentially a hyperparameter; we found that changing the exact wording of the prefix had limited impact and so did not perform extensive experiments into different prefix choices. A diagram of our text-to-text framework with a few input/output examples is shown in Figure 1. We provide full examples of preprocessed inputs for every task we studied in Appendix D.

例如，若要让模型把句子「That is good.」从英语翻译成德语，模型输入为「translate English to German: That is good.」，目标文本为「Das ist gut.」。对分类任务，模型只需预测对应类别的单个词。例如对 MNLI（Williams et al. 2017）——给定前提与假设，预测蕴含、矛盾或中立——输入为「mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons are indifferent.」，目标为「contradiction」。完整例子见附录。

Our text-to-text framework follows previous work that casts multiple NLP tasks into a common format: McCann et al. 2018 propose the “Natural Language Decathlon”, a benchmark that uses a consistent question-answering format for a suite of ten NLP tasks. The Natural Language Decathlon also stipulates that all models must be multi-task, i.e. are able to simultaneously tackle all of the tasks at once. We instead allow for separately fine-tuning the model on each individual task and use short task prefixes instead of an explicit question-answer format. Radford et al. 2019 evaluate the zero-shot learning capabilities of language models by feeding some input to the model as a prefix and then autoregressively sampling an output. For example, automatic summarization is done by feeding in a document followed by the text “TL;DR:” (short for “too long, didn’t read”, a common abbreviation) and then the summary is predicted via autoregressive decoding. We mainly consider models that explicitly process an input with an encoder before generating an output with a separate decoder and we focus on transfer learning rather than zero-shot learning. Finally, Keskar et al. 2019b unify many NLP tasks as “span extraction”, where text corresponding to possible output choices are appended to the input and the model is trained to extract the input span corresponding to the correct choice. In contrast, our framework also allows for generative tasks like machine translation and abstractive summarization where it is not possible to enumerate all possible output choices.

我们的文本到文本框架沿袭把多项 NLP 任务统一为单一格式的先前工作。McCann et al. (2018) 用问答格式；Keskar et al. (2019) 用多项选择格式。我们选择文本到文本，因为它自然适用于生成任务（如翻译与摘要），也适用于分类任务。

1

我们能把考虑的全部任务都直接写成文本到文本格式，唯一例外是 STSB 回归任务（Cer et al. 2017），其标签是 1 到 5 的连续分数。为此我们简单把分数四舍五入到最近的 0.2 增量，并转换成字符串，使模型预测例如「2.4」这样的字符串。训练时用教师强制，推理时用贪心解码。对分类任务，若模型输出不是有效标签，则计为错误。

Separately, we also convert the Winograd tasks (WNLI from GLUE, WSC from SuperGLUE, and the DPR data set we add to SuperGLUE) into a simpler format that is more amenable to the text-to-text framework. Examples from the Winograd tasks consist of a text passage containing an ambiguous pronoun that could refer to more than one of the noun phrases in the passage. For example, the passage might be “The city councilmen refused the demonstrators a permit because they feared violence.”, which contains the ambiguous pronoun “they” that could refer to “city councilmen” or “demonstrators”. We cast the WNLI, WSC, and DPR tasks as text-to-text problems by highlighting the ambiguous pronoun in the text passage and asking the model to predict the noun that it refers to. The example mentioned above would be transformed to the input “The city councilmen refused the demonstrators a permit because they feared violence.” and the model would be trained to predict the target text “The city councilmen”.

另外，我们还将 Winograd 任务（GLUE 的 WNLI、SuperGLUE 的 WSC）转换成不同格式。这些任务给出含歧义代词的句子，模型须判断候选先行词是否正确。标准格式用二元分类；我们发现把它写成文本生成任务更有效：输入提供句子与候选，模型输出候选文本本身（若正确）或特殊「false」token。

1{,}000

对 WSC，例子包含段落、歧义代词、候选先行词，以及候选是否正确的标签。我们把输入写成「wsc: [段落] * [候选] *」，其中候选在段落中用星号标出；若候选正确，目标为候选文本，否则为「false」。

The WNLI training and validation sets have a significant overlap with the WSC training set. To avoid leaking validation examples into our training data (a particular issue in the multi-task experiments of Section 3.5.2), we therefore never train on WNLI and never report results on the WNLI validation set. Omitting results on the WNLI validation set is standard practice (Devlin et al. 2018) due to the fact that it is “adversarial” with respect to the training set, i.e. validation examples are all slightly-perturbed versions of training examples with the opposite label. As such, we do not include WNLI in the average GLUE score whenever we report on the validation set (all sections except Section 3.7 where results are presented on the test sets). Converting examples from WNLI to the “referent noun prediction” variant described above is a little more involved; we describe this process in Appendix B.

WNLI 训练与验证集与 WSC 有显著重叠。此外，WNLI 训练集中的标签有已知问题。因此，对 GLUE 分数我们省略 WNLI，并在实验中用 DPR（Rahman and Ng 2012）与 WSC 作为 Winograd 风格任务的替代。我们在附录中提供这些任务的额外细节。

3 Experiments

3 实验

Recent advances in transfer learning for NLP have come from a wide variety of developments, such as new pre-training objectives, model architectures, unlabeled data sets, and more. In this section, we carry out an empirical survey of these techniques in hopes of teasing apart their contribution and significance. We then combine the insights gained to attain state-of-the-art in many of the tasks we consider. Since transfer learning for NLP is a rapidly growing area of research, it is not feasible for us to cover every possible technique or idea in our empirical study. For a broader literature review, we recommend a recent survey by Ruder et al. 2019.

NLP 迁移学习的近期进展来自广泛发展，例如新的预训练目标、模型架构、无标注数据集等。本节对这些技术做经验综述，希望理清各自贡献与重要性。随后我们把所得洞见结合起来，在所考虑的许多任务上达到最优。由于 NLP 迁移学习是快速增长的研究领域，经验研究无法覆盖每一种可能技术或想法。更广的文献综述，建议参阅 Ruder et al. 2019 的近期综述。

We systematically study these contributions by taking a reasonable baseline (described in Section 3.1) and altering one aspect of the setup at a time. For example, in Section 3.3 we measure the performance of different unsupervised objectives while keeping the rest of our experimental pipeline fixed. This “coordinate ascent” approach might miss second-order effects (for example, some particular unsupervised objective may work best on a model larger than our baseline setting), but performing a combinatorial exploration of all of the factors in our study would be prohibitively expensive. In future work, we expect it could be fruitful to more thoroughly consider combinations of the approaches we study.

我们通过取一个合理基线（见第 3.1 节），并每次只改动设定的一个方面，系统研究这些贡献。例如在第 3.3 节，我们在保持实验管线其余部分固定的情况下测量不同无监督目标的表现。这种「坐标上升」做法可能错过二阶效应（例如某个无监督目标可能在比基线更大的模型上效果最好），但组合探索本研究中全部因素的成本高得不可接受。未来工作中，更彻底地考虑我们所研究方法的组合，预计会有收获。

Our goal is to compare a variety of different approaches on a diverse set of tasks while keeping as many factors fixed as possible. In order to satisfy this aim, in some cases we do not exactly replicate existing approaches. For example, “encoder-only” models like BERT (Devlin et al. 2018) are designed to produce a single prediction per input token or a single prediction for an entire input sequence. This makes them applicable for classification or span prediction tasks but not for generative tasks like translation or abstractive summarization. As such, none of the model architectures we consider are identical to BERT or consist of an encoder-only structure. Instead, we test approaches that are similar in spirit—for example, we consider an analogous objective to BERT’s “masked language modeling” objective in Section 3.3 and we consider a model architecture that behaves similarly to BERT on text classification tasks in Section 3.2.

我们的目标是在多样任务集合上比较多种不同方法，同时尽可能固定其余因素。为满足这一目标，某些情形下我们并不精确复现已有方法。例如，像 BERT（Devlin et al. 2018）这样的「仅编码器」模型设计为对每个输入 token 产生单个预测，或对整个输入序列产生单个预测。这使它们适用于分类或跨度预测任务，但不适用于翻译或抽象摘要等生成任务。因此，我们考虑的模型架构无一与 BERT 完全相同，也不含仅编码器结构。相反，我们测试精神上相近的方法——例如，在第 3.3 节考虑与 BERT「掩码语言建模」目标类似的目标，在第 3.2 节考虑在文本分类任务上行为类似 BERT 的模型架构。

After outlining our baseline experimental setup in the following subsection, we undertake an empirical comparison of model architectures (Section 3.2), unsupervised objectives (Section 3.3), pre-training data sets (Section 3.4), transfer approaches (Section 3.5), and scaling (Section 3.6). At the culmination of this section, we combine insights from our study with scale to obtain state-of-the-art results in many tasks we consider (Section 3.7).

在下一小节概述基线实验设定后，我们对模型架构（第 3.2 节）、无监督目标（第 3.3 节）、预训练数据集（第 3.4 节）、迁移方法（第 3.5 节）与规模（第 3.6 节）做经验对照。本节收束处，我们把研究洞见与规模结合，在许多任务上取得最优结果（第 3.7 节）。

3.1 Baseline

3.1 基线

Our goal for our baseline is to reflect typical, modern practice. We pre-train a standard Transformer (described in Section 2.1) using a simple denoising objective and then separately fine-tune on each of our downstream tasks. We describe the details of this experimental setup in the following subsections.

基线的目标是反映典型的现代实践。我们用简单的去噪目标预训练标准 Transformer（见第 2.1 节），再分别在各下游任务上微调。细节见以下小节。

3.1.1 Model For our model, we use a standard encoder-decoder Transformer as proposed by Vaswani et al. 2017. While many modern approaches to transfer learning for NLP use a Transformer architecture consisting of only a single “stack” (e.g. for language modeling (Radford et al. 2018; Dong et al. 2019) or classification and span prediction (Devlin et al. 2018; Yang et al. 2019)), we found that using a standard encoder-decoder structure achieved good results on both generative and classification tasks. We explore the performance of different model architectures in Section 3.2.

3.1.1 模型 模型方面，我们使用 Vaswani et al. 2017 提出的标准编码器—解码器 Transformer。尽管许多现代 NLP 迁移学习方法使用仅含单一「堆栈」的 Transformer 架构（例如用于语言建模（Radford et al. 2018；Dong et al. 2019）或分类与跨度预测（Devlin et al. 2018；Yang et al. 2019）），我们发现标准编码器—解码器结构在生成与分类任务上都取得良好结果。第 3.2 节探索不同模型架构的表现。

12

12

3.1.2 Training As described in Section 2.4, all tasks are formulated as text-to-text tasks. This allows us to always train using standard maximum likelihood, i.e. using teacher forcing (Williams and Zipser 1989) and a cross-entropy loss. For optimization, we use AdaFactor (Shazeer and Stern 2018). At test time, we use greedy decoding (i.e. choosing the highest-probability logit at every timestep).

3.1.2 训练 如第 2.4 节所述，全部任务都表述为文本到文本任务。这使我们总能用标准最大似然训练，即使用教师强制（Williams and Zipser 1989）与交叉熵损失。优化用 AdaFactor（Shazeer and Stern 2018）。测试时用贪心解码（即每步选择最高概率 logit）。

2^{19}=524{,}288

2^{19}=524{,}288

1\big/\sqrt{\max(n,k)}

1\big/\sqrt{\max(n,k)}

2^{18}=262{,}144

2^{18}=262{,}144

32{,}000

32{,}000

\times

\times

15\%

15\%

10

5

When reporting results in the main text, we only report a subset of the scores across all the benchmarks to conserve space and ease interpretation. For GLUE and SuperGLUE, we report the average score across all subtasks (as stipulated by the official benchmarks) under the headings “GLUE” and “SGLUE”. For all translation tasks, we report the BLEU score (Papineni et al. 2002) as provided by SacreBLEU v1.3.0 (Post 2018) with “exp” smoothing and “intl” tokenization. We refer to scores for WMT English to German, English to French, and English to Romanian as EnDe, EnFr, and EnRo, respectively. For CNN/Daily Mail, we find the performance of models on the ROUGE-1-F, ROUGE-2-F, and ROUGE-L-F metrics (Lin 2004) to be highly correlated so we report the ROUGE-2-F score alone under the heading “CNNDM”. Similarly, for SQuAD we find the performance of the “exact match” and “F1” scores to be highly correlated so we report the “exact match” score alone. We provide every score achieved on every task for all experiments in Table 16, Appendix E.

正文报告结果时，为节省篇幅并便于解读，我们只报告全部基准分数的一个子集。对 GLUE 与 SuperGLUE，我们报告平均分；对 SQuAD，报告精确匹配分数；对 CNN/Daily Mail，报告 ROUGE-2-F；对翻译任务，报告 BLEU。全部任务的完整结果见附录。

\bigstar

结果表的格式均为：每行对应特定实验配置，各列给出各基准分数。我们将纳入基线配置（第 3.1 节）的平均表现以便对照。凡与基线有显著差异的分数以粗体标出。

\bigstar\,

	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ Baseline average	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
Baseline standard deviation	$0.235$	$0.065$	$0.343$	$0.416$	$0.112$	$0.090$	$0.108$
No pre-training	$66.22$	$17.60$	$50.31$	$53.04$	$25.86$	$\mathbf{39.77}$	$24.04$

\bigstar\,

	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ 基线平均	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
Baseline standard deviation	$0.235$	$0.065$	$0.343$	$0.416$	$0.112$	$0.090$	$0.108$
No pre-training	$66.22$	$17.60$	$50.31$	$53.04$	$25.86$	$\mathbf{39.77}$	$24.04$

80.8

80.8

1\%

1\%

3.2 Architectures

3.2 架构

While the Transformer was originally introduced with an encoder-decoder architecture, much modern work on transfer learning for NLP uses alternative architectures. In this section, we review and compare these architectural variants.

尽管 Transformer 最初以编码器—解码器架构引入，现代 NLP 迁移学习工作大量使用替代架构。本节回顾并比较这些架构变体。

x

x

y_{i}

y_{i}

x

图 4：我们考虑的 Transformer 架构变体示意。图中方块表示序列元素，线表示注意力可见性。不同颜色的方块组表示不同 Transformer 层堆栈。深灰线对应全可见掩码，浅灰线对应因果掩码。我们用「.」表示特殊的序列结束 token，代表一次预测结束。输入与输出序列分别表示为彩色与白色方块。

The first model structure we consider is an an encoder-decoder Transformer, which consists of two layer stacks: The encoder, which is fed an input sequence, and the decoder, which produces a new output sequence. A schematic of this architectural variant is shown in the left panel of Figure 4.

我们考虑的第一种模型结构是编码器—解码器 Transformer，由两个层堆栈组成：喂入输入序列的编码器，以及产生新输出序列的解码器。该架构变体示意见图 4 左面板。

The encoder uses a “fully-visible” attention mask. Fully-visible masking allows a self-attention mechanism to attend to any entry of the input when producing each entry of its output. We visualize this masking pattern in Figure 3, left. This form of masking is appropriate when attending over a “prefix”, i.e. some context provided to the model that is later used when making predictions. BERT (Devlin et al. 2018) also uses a fully-visible masking pattern and appends a special “classification” token to the input. BERT’s output at the timestep corresponding to the classification token is then used to make a prediction for classifying the input sequence.

编码器使用「全可见」注意力掩码。全可见掩码允许自注意力在产生每个输出条目时关注输入的任意条目。我们在图 3 左可视化该掩码模式。这种掩码适合对「前缀」做注意力，即提供给模型、随后用于做预测的上下文。BERT（Devlin et al. 2018）也使用全可见掩码模式，并在输入附加特殊「分类」token，其最终表示用于分类。

i

i

The decoder in an encoder-decoder Transformer is used to autoregressively produce an output sequence. That is, at each output timestep, a token is sampled from the model’s predicted distribution and the sample is fed back into the model to produce a prediction for the next output timestep, and so on. As such, a Transformer decoder (without an encoder) can be used as a language model (LM), i.e. a model trained solely for next-step prediction (Liu et al. 2018; Radford et al. 2018; Al-Rfou et al. 2019). This constitutes the second model structure we consider. A schematic of this architecture is shown in Figure 4, middle. In fact, early work on transfer learning for NLP used this architecture with a language modeling objective as a pre-training method (Radford et al. 2018).

编码器—解码器 Transformer 中的解码器用于自回归产生输出序列。也就是说，在每个输出时间步，从模型预测分布中采样一个 token，再把样本喂回模型以产生下一输出时间步的预测，如此反复。因此，（不带编码器的）Transformer 解码器可用作语言模型（LM），即仅训练做下一步预测的模型（Liu et al. 2018；Radford et al. 2018；Al-Rfou et al. 2019）。

Language models are typically used for compression or sequence generation (Graves 2013). However, they can also be used in the text-to-text framework simply by concatenating the inputs and targets. As an example, consider the case of English to German translation: If we have a training datapoint with input sentence “That is good.” and target “Das ist gut.”, we would simply train the model on next-step prediction over the concatenated input sequence “translate English to German: That is good. target: Das ist gut.” If we wanted to obtain the model’s prediction for this example, the model would be fed the prefix “translate English to German: That is good. target:” and would be asked to generate the remainder of the sequence autoregressively. In this way, the model can predict an output sequence given an input, which satisfies the needs of text-to-text tasks. This approach was recently used to show that language models can learn to perform some text-to-text tasks without supervision (Radford et al. 2019).

语言模型通常用于压缩或序列生成（Graves 2013）。但它们也可用于文本到文本框架，只需拼接输入与目标。例如英德翻译：若训练数据点输入句为「That is good.」、目标为「Das ist gut.」，则只需在拼接输入序列「translate English to German: That is good. target: Das ist gut.」上训练下一步预测，并要求模型预测「target:」之后的目标。

i

i

This issue can be avoided in a Transformer-based language model simply by changing the masking pattern. Instead of using a causal mask, we use fully-visible masking during the prefix portion of the sequence. This masking pattern and a schematic of the resulting “prefix LM” (the third model structure we consider) are illustrated in the rightmost panels of Figures 3 and 4, respectively. In the English to German translation example mentioned above, fully-visible masking would be applied to the prefix “translate English to German: That is good. target:” and causal masking would be used during training for predicting the target “Das ist gut.” Using a prefix LM in the text-to-text framework was originally proposed by Liu et al. 2018. More recently, Dong et al. 2019 showed that this architecture is effective on a wide variety of text-to-text tasks. This architecture is similar to an encoder-decoder model with parameters shared across the encoder and decoder and with the encoder-decoder attention replaced with full attention across the input and target sequence.

在基于 Transformer 的语言模型中，只需改变掩码模式即可避免该问题。我们不对序列使用因果掩码，而在序列的前缀部分使用全可见掩码。该掩码模式以及所得「前缀 LM」（我们考虑的第三种模型结构）的示意分别见图 3 与图 4 最右面板。在上述英德翻译例子中，全可见掩码会应用到前缀「translate English to German: That is good. target:」，因果掩码应用到目标「Das ist gut.」。

We note that when following our text-to-text framework, the prefix LM architecture closely resembles BERT (Devlin et al. 2018) for classification tasks. To see why, consider an example from the MNLI benchmark where the premise is “I hate pigeons.”, the hypothesis is “My feelings towards pigeons are filled with animosity.” and the correct label is “entailment”. To feed this example into a language model, we would transform it into the sequence “mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons are filled with animosity. target: entailment”. In this case, the fully-visible prefix would correspond to the entire input sequence up to the word “target:”, which can be seen as being analogous to the “classification” token used in BERT. So, our model would have full visibility over the entire input, and then would be tasked with making a classification by outputting the word “entailment”. It is easy for the model to learn to output one of the valid class labels given the task prefix (“mnli” in this case). As such, the main difference between a prefix LM and the BERT architecture is that the classifier is simply integrated into the output layer of the Transformer decoder in the prefix LM.

我们指出：在遵循文本到文本框架时，前缀 LM 架构在分类任务上与 BERT（Devlin et al. 2018）非常相似。例如 MNLI 基准中前提为「I hate pigeons.」、假设为「My feelings towards pigeons are filled with animosity.」、正确标签为「entailment」。要喂给语言模型，会把它变成序列「mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons are filled with animosity. target: entailment」，并对前缀用全可见掩码。这在精神上类似 BERT：对整个输入用全可见掩码，并训练模型预测分类标签。

L

3.2.2 比较不同模型结构 为实验比较这些架构变体，我们希望所考虑的每个模型在某种有意义的方式上等价。可以说两个模型等价，若它们参数量相同，或处理给定（输入序列、目标序列）对所需计算量大致相同。遗憾的是，不可能在同时匹配参数量与计算成本的情况下比较编码器—解码器模型与仅解码器语言模型架构——编码器—解码器模型有两个堆栈，而语言模型只有一个。因此我们考虑多种配置。

L

L

L

L

P

P

L/2

L/2

L

L

• A decoder-only prefix LM with the same architecture (and thus the same number of parameters and computational cost), but with fully-visible self-attention over the input.

• 相同架构（因而相同参数量与计算成本）的仅解码器前缀 LM，但对输入使用全可见自注意力。

3.2.3 Objectives As an unsupervised objective, we will consider both a basic language modeling objective as well as our baseline denoising objective described in Section 3.1.4. We include the language modeling objective due to its historic use as a pre-training objective (Dai and Le 2015; Ramachandran et al. 2016; Howard and Ruder 2018; Radford et al. 2018; Peters et al. 2018) as well as its natural fit for the language model architectures we consider. For models that ingest a prefix before making predictions (the encoder-decoder model and prefix LM), we sample a span of text from our unlabeled data set and choose a random point to split it into prefix and target portions. For the standard language model, we train the model to predict the entire span from beginning to end. Our unsupervised denoising objective is designed for text-to-text models; to adapt it for use with a language model we concatenate the inputs and targets as described in Section 3.2.1.

3.2.3 目标 作为无监督目标，我们将同时考虑基本语言建模目标以及第 3.1.4 节描述的基线去噪目标。纳入语言建模目标是因其作为预训练目标的历史使用（Dai and Le 2015；Ramachandran et al. 2016；Howard and Ruder 2018；Radford et al. 2018；Peters et al. 2018），以及它与我们所考虑语言模型架构的自然契合。对在做预测前摄入前缀的模型，语言建模目标对应于预测前缀之后的目标 token。

\bigstar\,

Architecture	Objective	Params	Cost	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ Encoder-decoder	Denoising	$2P$	$M$	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
Enc-dec, shared	Denoising	$P$	$M$	$82.81$	$18.78$	$\mathbf{80.63}$	$\mathbf{70.73}$	$26.72$	$39.03$	$\mathbf{27.46}$
Enc-dec, 6 layers	Denoising	$P$	$M/2$	$80.88$	$18.97$	$77.59$	$68.42$	$26.38$	$38.40$	$26.95$
Language model	Denoising	$P$	$M$	$74.70$	$17.93$	$61.14$	$55.02$	$25.09$	$35.28$	$25.86$
Prefix LM	Denoising	$P$	$M$	$81.82$	$18.61$	$78.94$	$68.11$	$26.43$	$37.98$	$27.39$
Encoder-decoder	LM	$2P$	$M$	$79.56$	$18.59$	$76.02$	$64.29$	$26.27$	$39.17$	$26.86$
Enc-dec, shared	LM	$P$	$M$	$79.60$	$18.13$	$76.35$	$63.50$	$26.62$	$39.17$	$27.05$
Enc-dec, 6 layers	LM	$P$	$M/2$	$78.67$	$18.26$	$75.32$	$64.06$	$26.13$	$38.42$	$26.89$
Language model	LM	$P$	$M$	$73.78$	$17.54$	$53.81$	$56.51$	$25.23$	$34.31$	$25.38$
Prefix LM	LM	$P$	$M$	$79.68$	$17.84$	$76.87$	$64.86$	$26.28$	$37.51$	$26.76$

\bigstar\,

架构	目标	参数	成本	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ 编码器—解码器	去噪	$2P$	$M$	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
编解码共享	去噪	$P$	$M$	$82.81$	$18.78$	$\mathbf{80.63}$	$\mathbf{70.73}$	$26.72$	$39.03$	$\mathbf{27.46}$
Enc-dec, 6 layers	去噪	$P$	$M/2$	$80.88$	$18.97$	$77.59$	$68.42$	$26.38$	$38.40$	$26.95$
Language model	去噪	$P$	$M$	$74.70$	$17.93$	$61.14$	$55.02$	$25.09$	$35.28$	$25.86$
Prefix LM	去噪	$P$	$M$	$81.82$	$18.61$	$78.94$	$68.11$	$26.43$	$37.98$	$27.39$
编码器—解码器	LM	$2P$	$M$	$79.56$	$18.59$	$76.02$	$64.29$	$26.27$	$39.17$	$26.86$
编解码共享	LM	$P$	$M$	$79.60$	$18.13$	$76.35$	$63.50$	$26.62$	$39.17$	$27.05$
Enc-dec, 6 layers	LM	$P$	$M/2$	$78.67$	$18.26$	$75.32$	$64.06$	$26.13$	$38.42$	$26.89$
Language model	LM	$P$	$M$	$73.78$	$17.54$	$53.81$	$56.51$	$25.23$	$34.31$	$25.38$
Prefix LM	LM	$P$	$M$	$79.68$	$17.84$	$76.87$	$64.86$	$26.28$	$37.51$	$26.76$

2P

2P

3.3 Unsupervised Objectives

3.3 无监督目标

The choice of unsupervised objective is of central importance as it provides the mechanism through which the model gains general-purpose knowledge to apply to downstream tasks. This has led to the development of a wide variety of pre-training objectives (Dai and Le 2015; Ramachandran et al. 2016; Radford et al. 2018; Devlin et al. 2018; Yang et al. 2019; Liu et al. 2019b; Wang et al. 2019a; Song et al. 2019; Dong et al. 2019; Joshi et al. 2019). In this section, we perform a procedural exploration of the space of unsupervised objectives. In many cases, we will not replicate an existing objective exactly—some will be modified to fit our text-to-text encoder-decoder framework and, in other cases, we will use objectives that combine concepts from multiple common approaches.

无监督目标的选择至关重要，因为它提供模型获得可应用于下游任务的通用知识的机制。这催生了种类繁多的预训练目标（Dai and Le 2015；Ramachandran et al. 2016；Radford et al. 2018；Devlin et al. 2018；Yang et al. 2019；Liu et al. 2019b；Wang et al. 2019a；Song et al. 2019；Dong et al. 2019；Joshi et al. 2019）。本节对无监督目标空间做程序性探索。许多情形下我们不会精确复现已有目标——有的会修改以适配文本到文本编码器—解码器框架，另一些则组合多种常见做法中的概念。

Overall, all of our objectives ingest a sequence of token IDs corresponding to a tokenized span of text from our unlabeled text data set. The token sequence is processed to produce a (corrupted) input sequence and a corresponding target. Then, the model is trained as usual with maximum likelihood to predict the target sequence. We provide illustrative examples of many of the objectives we consider in Table 3.

总体而言，我们的全部目标都摄入对应无标注文本数据集中一段已分词文本的 token ID 序列。该 token 序列经处理产生（被腐蚀的）输入序列与对应目标。然后模型照常以最大似然训练以预测目标序列。表 3 给出许多所考虑目标的示意例子。

Objective Inputs Targets Prefix language modeling Thank you for inviting me to your party last week . BERT-style Devlin et al. 2018 Thank you me to your party apple week . (original text) Deshuffling party me for your to . last fun you inviting week Thank (original text) MASS-style Song et al. 2019 Thank you me to your party week . (original text) I.i.d. noise, replace spans Thank you me to your party week . for inviting last I.i.d. noise, drop tokens Thank you me to your party week . for inviting last Random spans Thank you to week . for inviting me your party last Table 3: Examples of inputs and targets produced by some of the unsupervised objectives we consider applied to the input text “Thank you for inviting me to your party last week .” Note that all of our objectives process tokenized text. For this particular sentence, all words were mapped to a single token by our vocabulary. We write (original text) as a target to denote that the model is tasked with reconstructing the entire input text. denotes a shared mask token and , , and denote sentinel tokens that are assigned unique token IDs. The BERT-style objective (second row) includes a corruption where some tokens are replaced by a random token ID; we show this via the greyed-out word apple.

Objective	Inputs	Targets
Prefix language modeling	Thank you for inviting	me to your party last week .
BERT-style Devlin et al. 2018	Thank you me to your party apple week .	(original text)
Deshuffling	party me for your to . last fun you inviting week Thank	(original text)
MASS-style Song et al. 2019	Thank you me to your party week .	(original text)
I.i.d. noise, replace spans	Thank you me to your party week .	for inviting last
I.i.d. noise, drop tokens	Thank you me to your party week .	for inviting last
Random spans	Thank you to week .	for inviting me your party last

目标 输入 目标序列 前缀语言建模 Thank you for inviting me to your party last week . BERT 风格 Devlin et al. 2018 Thank you me to your party apple week . (original text) 重排 party me for your to . last fun you inviting week Thank (original text) MASS 风格 Song et al. 2019 Thank you me to your party week . (original text) I.i.d. noise, replace spans Thank you me to your party week . for inviting last I.i.d. noise, drop tokens Thank you me to your party week . for inviting last Random spans Thank you to week . for inviting me your party last Table 3: Examples of inputs and targets produced by some of the unsupervised objectives we consider applied to the input text “Thank you for inviting me to your party last week .” Note that all of our objectives process tokenized text. For this particular sentence, all words were mapped to a single token by our vocabulary. We write (original text) as a target to denote that the model is tasked with reconstructing the entire input text. denotes a shared mask token and , , and denote sentinel tokens that are assigned unique token IDs. The BERT 风格 objective (second row) includes a corruption where some tokens are replaced by a random token ID; we show this via the greyed-out word apple. 表 3：对输入文本「Thank you for inviting me to your party last week .」应用部分无监督目标所产生的输入与目标例子。注意全部目标都处理 已分词 文本。 表示共享掩码 token，// 表示哨兵 token。

目标	输入	目标序列
前缀语言建模	Thank you for inviting	me to your party last week .
BERT 风格 Devlin et al. 2018	Thank you me to your party apple week .	(original text)
重排	party me for your to . last fun you inviting week Thank	(original text)
MASS 风格 Song et al. 2019	Thank you me to your party week .	(original text)
I.i.d. noise, replace spans	Thank you me to your party week .	for inviting last
I.i.d. noise, drop tokens	Thank you me to your party week .	for inviting last
Random spans	Thank you to week .	for inviting me your party last

15\%

15\%

80.69

Objective	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
Prefix language modeling	$80.69$	$18.94$	$77.99$	$65.27$	$\mathbf{26.86}$	$39.73$	$\mathbf{27.49}$
BERT-style (Devlin et al. 2018)	$\mathbf{82.96}$	$\mathbf{19.17}$	$\mathbf{80.65}$	$\mathbf{69.85}$	$\mathbf{26.78}$	$\mathbf{40.03}$	$\mathbf{27.41}$
Deshuffling	$73.17$	$18.59$	$67.61$	$58.47$	$26.11$	$39.30$	$25.62$

80.69

目标	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
前缀语言建模	$80.69$	$18.94$	$77.99$	$65.27$	$\mathbf{26.86}$	$39.73$	$\mathbf{27.49}$
BERT 风格 (Devlin et al. 2018)	$\mathbf{82.96}$	$\mathbf{19.17}$	$\mathbf{80.65}$	$\mathbf{69.85}$	$\mathbf{26.78}$	$\mathbf{40.03}$	$\mathbf{27.41}$
重排	$73.17$	$18.59$	$67.61$	$58.47$	$26.11$	$39.30$	$25.62$

The performance of these three objectives is shown in Table 4. Overall, we find that the BERT-style objective performs best, though the prefix language modeling objective attains similar performance on the translation tasks. Indeed, the motivation for the BERT objective was to outperform language model-based pre-training. The deshuffling objective performs considerably worse than both prefix language modeling and the BERT-style objective.

这三种目标的表现见表 4。总体而言，BERT 风格目标表现最好，不过前缀语言建模目标在翻译任务上达到相近表现。的确，BERT 目标的动机就是超越基于语言模型的预训练。重排目标明显弱于前缀语言建模与 BERT 风格目标。

3.3.2 Simplifying the BERT Objective Based on the results in the prior section, we will now focus on exploring modifications to the BERT-style denoising objective. This objective was originally proposed as a pre-training technique for an encoder-only model trained for classification and span prediction. As such, it may be possible to modify it so that it performs better or is more efficient in our encoder-decoder text-to-text setup.

3.3.2 简化 BERT 目标 基于上一节结果，我们接下来聚焦探索对 BERT 风格去噪目标的修改。该目标最初是为训练做分类与跨度预测的仅编码器模型提出的预训练技术。因此，有可能修改它，使其在我们的编码器—解码器文本到文本设定中表现更好或更高效。

15\%

15\%

82.96

Objective	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
BERT-style (Devlin et al. 2018)	$82.96$	$19.17$	$\mathbf{80.65}$	$69.85$	$26.78$	$\mathbf{40.03}$	$27.41$
MASS-style (Song et al. 2019)	$82.32$	$19.16$	$80.10$	$69.28$	$26.79$	$\mathbf{39.89}$	$27.55$
$\bigstar\,$ Replace corrupted spans	$83.28$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$39.82$	$\mathbf{27.65}$
Drop corrupted tokens	$\mathbf{84.44}$	$\mathbf{19.31}$	$\mathbf{80.52}$	$68.67$	$\mathbf{27.07}$	$39.76$	$\mathbf{27.82}$

82.96

目标	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
BERT 风格 (Devlin et al. 2018)	$82.96$	$19.17$	$\mathbf{80.65}$	$69.85$	$26.78$	$\mathbf{40.03}$	$27.41$
MASS 风格 (Song et al. 2019)	$82.32$	$19.16$	$80.10$	$69.28$	$26.79$	$\mathbf{39.89}$	$27.55$
$\bigstar\,$ 替换腐蚀跨度	$83.28$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$39.82$	$\mathbf{27.65}$
丢弃腐蚀 token	$\mathbf{84.44}$	$\mathbf{19.31}$	$\mathbf{80.52}$	$68.67$	$\mathbf{27.07}$	$39.76$	$\mathbf{27.82}$

60.04

60.04

10\%

15\%

10\%

Corruption rate	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$10\%$	$\mathbf{82.82}$	$19.00$	$\mathbf{80.38}$	$69.55$	$\mathbf{26.87}$	$39.28$	$\mathbf{27.44}$
$\bigstar\,$ $15\%$	$\mathbf{83.28}$	$19.24$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
$25\%$	$\mathbf{83.00}$	$\mathbf{19.54}$	$\mathbf{80.96}$	$70.48$	$\mathbf{27.04}$	$\mathbf{39.83}$	$\mathbf{27.47}$
$50\%$	$81.27$	$19.32$	$79.80$	$70.33$	$\mathbf{27.01}$	$\mathbf{39.90}$	$\mathbf{27.49}$

10\%

腐蚀率	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$10\%$	$\mathbf{82.82}$	$19.00$	$\mathbf{80.38}$	$69.55$	$\mathbf{26.87}$	$39.28$	$\mathbf{27.44}$
$\bigstar\,$ $15\%$	$\mathbf{83.28}$	$19.24$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
$25\%$	$\mathbf{83.00}$	$\mathbf{19.54}$	$\mathbf{80.96}$	$70.48$	$\mathbf{27.04}$	$\mathbf{39.83}$	$\mathbf{27.47}$
$50\%$	$81.27$	$19.32$	$79.80$	$70.33$	$\mathbf{27.01}$	$\mathbf{39.90}$	$\mathbf{27.49}$

3.3.4 Corrupting Spans We now turn towards the goal of speeding up training by predicting shorter targets. The approach we have used so far makes an i.i.d. decision for each input token as to whether to corrupt it or not. When multiple consecutive tokens have been corrupted, they are treated as a “span” and a single unique mask token is used to replace the entire span. Replacing entire spans with a single token results in unlabeled text data being processed into shorter sequences. Since we are using an i.i.d. corruption strategy, it is not always the case that a significant number of corrupted tokens appear consecutively. As a result, we might obtain additional speedup by specifically corrupting spans of tokens rather than corrupting individual tokens in an i.i.d. manner. Corrupting spans was also previously considered as a pre-training objective for BERT, where it was found to improve performance (Joshi et al. 2019).

3.3.4 腐蚀跨度 我们现在转向通过预测更短目标来加速训练的目标。目前所用做法对每个输入 token 做独立同分布决定是否腐蚀。当多个连续 token 被腐蚀时，它们被视为一个「跨度」，并用单个唯一掩码 token 替换整个跨度。用单个 token 替换整个跨度使无标注文本被处理成更短序列。由于使用独立同分布腐蚀策略，并不总有大量被腐蚀 token 连续出现。因此，通过专门腐蚀 token 跨度而非独立同分布腐蚀单个 token，可能获得额外加速。腐蚀跨度先前也作为 BERT 的预训练目标被考虑，并发现可提升表现（Joshi et al. 2019）。

500

500

\bigstar\,

Span length	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ Baseline (i.i.d.)	$\mathbf{83.28}$	$19.24$	$80.88$	$71.36$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
$2$	$\mathbf{83.54}$	$19.39$	$\mathbf{82.09}$	$\mathbf{72.20}$	$\mathbf{26.76}$	$\mathbf{39.99}$	$\mathbf{27.63}$
$3$	$\mathbf{83.49}$	$\mathbf{19.62}$	$\mathbf{81.84}$	$\mathbf{72.53}$	$\mathbf{26.86}$	$39.65$	$\mathbf{27.62}$
$5$	$\mathbf{83.40}$	$19.24$	$\mathbf{82.05}$	$\mathbf{72.23}$	$\mathbf{26.88}$	$39.40$	$\mathbf{27.53}$
$10$	$82.85$	$19.33$	$\mathbf{81.84}$	$70.44$	$\mathbf{26.79}$	$39.49$	$\mathbf{27.69}$

\bigstar\,

跨度长度	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ 基线（独立同分布）	$\mathbf{83.28}$	$19.24$	$80.88$	$71.36$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
$2$	$\mathbf{83.54}$	$19.39$	$\mathbf{82.09}$	$\mathbf{72.20}$	$\mathbf{26.76}$	$\mathbf{39.99}$	$\mathbf{27.63}$
$3$	$\mathbf{83.49}$	$\mathbf{19.62}$	$\mathbf{81.84}$	$\mathbf{72.53}$	$\mathbf{26.86}$	$39.65$	$\mathbf{27.62}$
$5$	$\mathbf{83.40}$	$19.24$	$\mathbf{82.05}$	$\mathbf{72.23}$	$\mathbf{26.88}$	$39.40$	$\mathbf{27.53}$
$10$	$82.85$	$19.33$	$\mathbf{81.84}$	$70.44$	$\mathbf{26.79}$	$39.49$	$\mathbf{27.69}$

15\%

15\%

Figure 5: A flow chart of our exploration of unsupervised objectives. We first consider a few disparate approaches in Section 3.3.1 and find that a BERT-style denoising objective performs best. Then, we consider various methods for simplifying the BERT objective so that it produces shorter target sequences in Section 3.3.2. Given that replacing dropped-out spans with sentinel tokens performs well and results in short target sequences, in Section 3.3.3 we experiment with different corruption rates. Finally, we evaluate an objective that intentionally corrupts contiguous spans of tokens in Section 3.3.4.

图 5：我们探索无监督目标的流程图。首先在第 3.3.1 节考虑几种差异显著的方法，发现 BERT 风格去噪目标最好。然后在第 3.3.2 节考虑多种简化 BERT 目标以使目标序列更短的方法。鉴于用哨兵 token 替换被丢弃跨度表现良好且产生短目标序列，第 3.3.3 节实验不同腐蚀率。最后在第 3.3.4 节评估有意腐蚀连续 token 跨度的目标。

3.3.5 Discussion Figure 5 shows a flow chart of the choices made during our exploration of unsupervised objectives. Overall, the most significant difference in performance we observed was that denoising objectives outperformed language modeling and deshuffling for pre-training. We did not observe a remarkable difference across the many variants of the denoising objectives we explored. However, different objectives (or parameterizations of objectives) can lead to different sequence lengths and thus different training speeds. This implies that choosing among the denoising objectives we considered here should mainly be done according to their computational cost. Our results also suggest that additional exploration of objectives similar to the ones we consider here may not lead to significant gains for the tasks and model we consider. Instead, it may be fortuitous to explore entirely different ways of leveraging unlabeled data.

3.3.5 讨论 图 5 展示探索无监督目标过程中所作选择的流程图。总体而言，我们观察到的最显著表现差异是去噪目标优于用于预训练的语言建模与重排。在所探索的众多去噪目标变体之间，我们未观察到显著差别。然而，不同目标（或目标的参数化）可导致不同序列长度，从而不同训练速度。这意味着在此处考虑的去噪目标之间做选择，主要应按其计算成本。我们的结果也表明，对所考虑的任务与模型，继续探索与此处类似的目标可能不会带来显著收益。相反，探索利用无标注数据的全新方式或许更有收获。

3.4 Pre-training Data set

3.4 预训练数据集

Like the unsupervised objective, the pre-training data set itself is a crucial component of the transfer learning pipeline. However, unlike objectives and benchmarks, new pre-training data sets are usually not treated as significant contributions on their own and are often not released alongside pre-trained models and code. Instead, they are typically introduced in the course of presenting a new method or model. As a result, there has been relatively little comparison of different pre-training data sets as well as a lack of a “standard” data set used for pre-training. Some recent notable exceptions (Baevski et al. 2019; Liu et al. 2019c; Yang et al. 2019) have compared pre-training on a new large (often Common Crawl-sourced) data set to using a smaller preexisting data set (often Wikipedia). To probe more deeply into the impact of the pre-training data set on performance, in this section we compare variants of our C4 data set and other potential sources of pre-training data. We release all of the C4 data set variants we consider as part of TensorFlow Datasets.[^11]

与无监督目标一样，预训练数据集本身是迁移学习管线的关键成分。但与目标不同，对替代数据集做受控实验更难，因为公开可用的大型文本语料有限。本节比较若干替代数据源，并测量预训练数据集规模的影响。

3.4.1 Unlabeled Data Sets In creating C4, we developed various heuristics to filter the web-extracted text from Common Crawl (see Section 2.2 for a description). We are interested in measuring whether this filtering results in improved performance on downstream tasks, in addition to comparing it to other filtering approaches and common pre-training data sets. Towards this end, we compare the performance of our baseline model after pre-training on the following data sets:

3.4.1 无标注数据集 创建 C4 时，我们开发了多种启发式以过滤来自 Common Crawl 的网络提取文本（见第 2.2 节）。我们也考虑用其他过滤策略或数据源创建的预训练数据集。

As a baseline, we first consider pre-training on our proposed unlabeled data set as described in Section 2.2.

作为基线，我们首先考虑在第 2.2 节所述的无标注数据集上预训练。

To measure the effect of the heuristic filtering we used in creating C4 (deduplication, removing bad words, only retaining sentences, etc.), we also generate an alternate version of C4 that forgoes this filtering. Note that we still use langdetect to extract English text. As a result, our “unfiltered” variant still includes some filtering because langdetect sometimes assigns a low probability to non-natural English text.

为度量创建 C4 时所用启发式过滤（去重、移除不良词、只保留句子等）的影响，我们考虑在未过滤的 Common Crawl 网络提取文本上预训练。

Recent work has used text data extracted from news websites (Zellers et al. 2019; Baevski et al. 2019). To compare to this approach, we generate another unlabeled data set by additionally filtering C4 to only include content from one of the domains used in the “RealNews” data set (Zellers et al. 2019). Note that for ease of comparison, we retain the heuristic filtering methods used in C4; the only difference is that we have ostensibly omitted any non-news content.

近期工作使用从新闻网站提取的文本数据（Zellers et al. 2019；Baevski et al. 2019）。为与此对照，我们生成一个只含来自已知新闻域名内容的数据集。

Similarly, the WebText data set (Radford et al. 2019) only uses content from webpages that were submitted to the content aggregation website Reddit and received a ‘‘score’’ of at least 3. The score for a webpage submitted to Reddit is computed based on the proportion of users who endorse (upvote) or oppose (downvote) the webpage. The idea behind using the Reddit score as a quality signal is that users of the site would only upvote high-quality text content. To generate a comparable data set, we first tried removing all content from C4 that did not originate from a URL that appeared in the list prepared by the OpenWebText effort.[^12] However, this resulted in comparatively little content—only about 2 GB—because most pages never appear on Reddit. Recall that C4 was created based on a single month of Common Crawl data. To avoid using a prohibitively small data set, we therefore downloaded 12 months of data from Common Crawl from August 2018 to July 2019, applied our heuristic filtering for C4, then applied the Reddit filter. This produced a 17 GB WebText-like data set, which is of comparable size to the original 40GB WebText data set (Radford et al. 2019).

类似地，WebText 数据集（Radford et al. 2019）只使用被提交到内容聚合网站 Reddit 且获得至少 3 个「赞」的网页内容。我们也创建一个类似数据集以对照。

The website Wikipedia consists of millions of encyclopedia articles written collaboratively. The content on the site is subject to strict quality guidelines and therefore has been used as a reliable source of clean and natural text. We use the English Wikipedia text data from TensorFlow Datasets,[^13] which omits any markup or reference sections from the articles.

维基百科网站由协作撰写的数百万百科文章组成。站点内容受严格质量控制，并已被证明是有用的预训练数据源（Devlin et al. 2018；Yang et al. 2019；Liu et al. 2019c）。

A drawback of using pre-training data from Wikipedia is that it represents only one possible domain of natural text (encyclopedia articles). To mitigate this, BERT (Devlin et al. 2018) combined data from Wikipedia with the Toronto Books Corpus (TBC) (Zhu et al. 2015). TBC contains text extracted from eBooks, which represents a different domain of natural language. BERT’s popularity has led to the Wikipedia + TBC combination being used in many subsequent works.

使用来自维基百科的预训练数据的缺点是它只代表自然语言文本的一个可能领域（百科文章）。为缓解这一问题，我们考虑把维基百科与来自 TBC（Toronto Books Corpus）（Zhu et al. 2015）的图书文本混合，后者也被 BERT 使用。

\bigstar\,

Data set	Size	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ C4	745GB	$83.28$	$\mathbf{19.24}$	$80.88$	$71.36$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
C4, unfiltered	6.1TB	$81.46$	$19.14$	$78.78$	$68.04$	$26.55$	$39.34$	$27.21$
RealNews-like	35GB	$\mathbf{83.83}$	$\mathbf{19.23}$	$80.39$	$72.38$	$\mathbf{26.75}$	$\mathbf{39.90}$	$\mathbf{27.48}$
WebText-like	17GB	$\mathbf{84.03}$	$\mathbf{19.31}$	$\mathbf{81.42}$	$71.40$	$\mathbf{26.80}$	$\mathbf{39.74}$	$\mathbf{27.59}$
Wikipedia	16GB	$81.85$	$\mathbf{19.31}$	$81.29$	$68.01$	$\mathbf{26.94}$	$39.69$	$\mathbf{27.67}$
Wikipedia + TBC	20GB	$83.65$	$\mathbf{19.28}$	$\mathbf{82.08}$	$\mathbf{73.24}$	$\mathbf{26.77}$	$39.63$	$\mathbf{27.57}$

\bigstar\,

数据集	规模	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ C4	745GB	$83.28$	$\mathbf{19.24}$	$80.88$	$71.36$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
C4, unfiltered	6.1TB	$81.46$	$19.14$	$78.78$	$68.04$	$26.55$	$39.34$	$27.21$
RealNews-like	35GB	$\mathbf{83.83}$	$\mathbf{19.23}$	$80.39$	$72.38$	$\mathbf{26.75}$	$\mathbf{39.90}$	$\mathbf{27.48}$
WebText-like	17GB	$\mathbf{84.03}$	$\mathbf{19.31}$	$\mathbf{81.42}$	$71.40$	$\mathbf{26.80}$	$\mathbf{39.74}$	$\mathbf{27.59}$
Wikipedia	16GB	$81.85$	$\mathbf{19.31}$	$81.29$	$68.01$	$\mathbf{26.94}$	$39.69$	$\mathbf{27.67}$
Wikipedia + TBC	20GB	$83.65$	$\mathbf{19.28}$	$\mathbf{82.08}$	$\mathbf{73.24}$	$\mathbf{26.77}$	$39.63$	$\mathbf{27.57}$

73.24

在这些数据集各自上预训练后达到的结果见表 8。一个明显结论是：移除启发式过滤（即使用未过滤的 Common Crawl）会显著损害表现。使用新闻、WebText 类或维基百科数据在多数任务上给出与 C4 相近或略好的结果；维基百科+TBC 在 GLUE 上略好。总体而言，过滤后的网络文本（C4）提供强劲的通用表现，同时规模远大于领域更窄的语料。

40\times

只在单一领域预训练的缺点是所得数据集往往显著更小。类似地，尽管 WebText 类过滤可产生高质量文本，其规模也受可用于过滤的「赞」信号限制。C4 的启发式过滤在保留很大一部分 Common Crawl 数据的同时移除明显低质量文本。

3.4.2 Pre-training Data set Size The pipeline we use to create C4 was designed to be able to create extremely large pre-training data sets. The access to so much data allows us to pre-train our models without repeating examples. It is not clear whether repeating examples during pre-training would be helpful or harmful to downstream performance because our pre-training objective is itself stochastic and can help prevent the model from seeing the same exact data multiple times.

3.4.2 预训练数据集规模 我们用于创建 C4 的管线旨在能创建极大的预训练数据集。这使我们能研究预训练数据集规模对下游表现的影响，而不必重复数据。

2^{35}\approx 34\mathrm{B}

2^{35}

\bigstar\,

Number of tokens	Repeats	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ Full data set	$0$	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
$2^{29}$	$64$	$\mathbf{82.87}$	$\mathbf{19.19}$	$\mathbf{80.97}$	$\mathbf{72.03}$	$\mathbf{26.83}$	$\mathbf{39.74}$	$\mathbf{27.63}$
$2^{27}$	$256$	$82.62$	$\mathbf{19.20}$	$79.78$	$69.97$	$\mathbf{27.02}$	$\mathbf{39.71}$	$27.33$
$2^{25}$	$1{,}024$	$79.55$	$18.57$	$76.27$	$64.76$	$26.38$	$39.56$	$26.80$
$2^{23}$	$4{,}096$	$76.34$	$18.33$	$70.92$	$59.29$	$26.37$	$38.84$	$25.81$

\bigstar\,

token 数	重复	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ Full data set	$0$	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
$2^{29}$	$64$	$\mathbf{82.87}$	$\mathbf{19.19}$	$\mathbf{80.97}$	$\mathbf{72.03}$	$\mathbf{26.83}$	$\mathbf{39.74}$	$\mathbf{27.63}$
$2^{27}$	$256$	$82.62$	$\mathbf{19.20}$	$79.78$	$69.97$	$\mathbf{27.02}$	$\mathbf{39.71}$	$27.33$
$2^{25}$	$1{,}024$	$79.55$	$18.57$	$76.27$	$64.76$	$26.38$	$39.56$	$26.80$
$2^{23}$	$4{,}096$	$76.34$	$18.33$	$70.92$	$59.29$	$26.37$	$38.84$	$25.81$

The resulting downstream performance is shown in Table 9. As expected, performance degrades as the data set size shrinks. We suspect this may be due to the fact that the model begins to memorize the pre-training data set. To measure if this is true, we plot the training loss for each of these data set sizes in Figure 6. Indeed, the model attains significantly smaller training losses as the size of the pre-training data set shrinks, suggesting possible memorization. Baevski et al. 2019 similarly observed that truncating the pre-training data set size can degrade downstream task performance.

所得下游表现见表 9。如预期，随数据集缩小，表现下降。我们怀疑这主要因为过拟合无标注数据——当模型多次看到同一例子时，它可能开始记忆而非学习一般语言能力。

64

64

4

4

3.5 Training Strategy

3.5 训练策略

So far we have considered the setting where all parameters of a model are pre-trained on an unsupervised task before being fine-tuned on individual supervised tasks. While this approach is straightforward, various alternative methods for training the model on downstream/supervised tasks have been proposed. In this section, we compare different schemes for fine-tuning the model in addition to the approach of training the model simultaneously on multiple tasks.

到目前为止，我们考虑的设定是：模型全部参数先在无监督任务上预训练，再分别在各下游任务上微调。本节探索替代微调方法以及多任务学习。

3.5.1 Fine-tuning Methods It has been argued that fine-tuning all of the model’s parameters can lead to suboptimal results, particularly on low-resource tasks (Peters et al. 2019). Early results on transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier that was fed sentence embeddings produced by a fixed pre-trained model (Subramanian et al. 2018; Kiros et al. 2015; Logeswaran and Lee 2018; Hill et al. 2016; Conneau et al. 2017). This approach is less applicable to our encoder-decoder model because the entire decoder must be trained to output the target sequences for a given task. Instead, we focus on two alternative fine-tuning approaches that update only a subset of the parameters of our encoder-decoder model.

3.5.1 微调方法 有人主张微调模型全部参数可能导致次优结果，尤其在下游数据集较小时。因此我们比较若干替代微调方法。

d

第一种「适配器层」（Houlsby et al. 2019；Bapna et al. 2019）的动机是在微调时保持原模型大部分固定。额外的小型神经网络（适配器）被插入 Transformer 的每块中，微调期间只更新适配器参数与 Layer Norm 参数。

12

我们考虑的第二种替代微调方法是「逐步解冻」（Howard and Ruder 2018）。逐步解冻中，越来越多的模型层在微调过程中被解冻（即可训练）。具体地，我们开始时只更新最后一层，然后按固定间隔解冻更早的层。

d

d=32

\bigstar\,

Fine-tuning method	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ All parameters	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
Adapter layers, $d=32$	$80.52$	$15.08$	$79.32$	$60.40$	$13.84$	$17.88$	$15.54$
Adapter layers, $d=128$	$81.51$	$16.62$	$79.47$	$63.03$	$19.83$	$27.50$	$22.63$
Adapter layers, $d=512$	$81.54$	$17.78$	$79.18$	$64.30$	$23.45$	$33.98$	$25.81$
Adapter layers, $d=2048$	$81.51$	$16.62$	$79.47$	$63.03$	$19.83$	$27.50$	$22.63$
Gradual unfreezing	$82.50$	$18.95$	$79.17$	$\mathbf{70.79}$	$26.71$	$39.02$	$26.93$

\bigstar\,

微调方法	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ 全部参数	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
适配器层, $d=32$	$80.52$	$15.08$	$79.32$	$60.40$	$13.84$	$17.88$	$15.54$
适配器层, $d=128$	$81.51$	$16.62$	$79.47$	$63.03$	$19.83$	$27.50$	$22.63$
适配器层, $d=512$	$81.54$	$17.78$	$79.18$	$64.30$	$23.45$	$33.98$	$25.81$
适配器层, $d=2048$	$81.51$	$16.62$	$79.47$	$63.03$	$19.83$	$27.50$	$22.63$
逐步解冻	$82.50$	$18.95$	$79.17$	$\mathbf{70.79}$	$26.71$	$39.02$	$26.93$

3.5.2 Multi-task Learning So far, we have been pre-training our model on a single unsupervised learning task before fine-tuning it individually on each downstream task. An alternative approach, called “multi-task learning” (Ruder 2017; Caruana 1997), is to train the model on multiple tasks at a time. This approach typically has the goal of training a single model that can simultaneously perform many tasks at once, i.e. the model and most of its parameters are shared across all tasks. We relax this goal somewhat and instead investigate methods for training on multiple tasks at once in order to eventually produce separate parameter settings that perform well on each individual task. For example, we might train a single model on many tasks, but when reporting performance we are allowed to select a different checkpoint for each task. This loosens the multi-task learning framework and puts it on more even footing compared to the pre-train-then-fine-tune approach we have considered so far. We also note that in our unified text-to-text framework, “multi-task learning” simply corresponds to mixing data sets together. It follows that we can still train on unlabeled data when using multi-task learning by treating the unsupervised task as one of the tasks being mixed together. In contrast, most applications of multi-task learning to NLP add task-specific classification networks or use different loss functions for each task (Liu et al. 2019b).

3.5.2 多任务学习 到目前为止，我们一直在单一无监督学习任务上预训练模型，再分别微调到每个下游任务。替代方案是同时在多个任务上训练模型（即多任务学习）（Caruana 1997）。在我们的文本到文本框架中，多任务学习很直接——我们只需把来自不同任务的例子混合进同一批次。

As pointed out by Arivazhagan et al. 2019, an extremely important factor in multi-task learning is how much data from each task the model should be trained on. Our goal is to not under- or over-train the model—that is, we want the model to see enough data from a given task that it can perform the task well, but not to see so much data that it memorizes the training set. How exactly to set the proportion of data coming from each task can depend on various factors including data set sizes, the “difficulty” of learning the task (i.e. how much data the model must see before being able to perform the task effectively), regularization, etc. An additional issue is the potential for “task interference” or “negative transfer”, where achieving good performance on one task can hinder performance on another. Given these concerns, we begin by exploring various strategies for setting the proportion of data coming from each task. A similar exploration was performed by Wang et al. 2019a.

如 Arivazhagan et al. 2019 所指出，多任务学习中一个极其重要的因素是模型应看到来自每项任务多少数据。由于我们的任务规模差异巨大（从数百到数百万例子），混合策略对结果有显著影响。

N

模型对给定任务过拟合有多快的一个主要因素是该任务的数据集规模。因此，设定混合比例的自然方式是按数据集规模比例混合例子，即「按例子比例」混合。我们也考虑用温度参数平滑这些比例。

T

T

In this case, we sample examples from each task with equal probability. Specifically, each example in each batch is sampled uniformly at random from one of the data sets we train on. This is most likely a suboptimal strategy, as the model will overfit quickly on low-resource tasks and underfit on high-resource tasks. We mainly include it as a point of reference of what might go wrong when the proportions are set suboptimally.

此时我们从每项任务以相等概率采样例子。具体地，批中每个例子从全部任务的并集中均匀随机采样。

2^{19}+2^{18}=786{,}432

为在与基线「先预训练再微调」结果同等基础上比较这些混合策略，我们按与基线预训练加微调相同的总步数训练多任务模型。

K

总体而言，我们发现多任务训练在多数任务上弱于先预训练再微调。「均等」混合策略对低资源任务帮助最大，但损害高资源任务。按例子比例混合对高资源任务更好，但对低资源任务会导致快速过拟合。

\bigstar\,

Mixing strategy	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ Baseline (pre-train/fine-tune)	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
Equal	$76.13$	$19.02$	$76.51$	$63.37$	$23.89$	$34.31$	$26.78$
Examples-proportional, $K=2^{16}$	$80.45$	$19.04$	$77.25$	$69.95$	$24.35$	$34.99$	$27.10$
Examples-proportional, $K=2^{17}$	$81.56$	$19.12$	$77.00$	$67.91$	$24.36$	$35.00$	$27.25$
Examples-proportional, $K=2^{18}$	$81.67$	$19.07$	$78.17$	$67.94$	$24.57$	$35.19$	$27.39$
Examples-proportional, $K=2^{19}$	$81.42$	$\mathbf{19.24}$	$79.78$	$67.30$	$25.21$	$36.30$	$\mathbf{27.76}$
Examples-proportional, $K=2^{20}$	$80.80$	$\mathbf{19.24}$	$\mathbf{80.36}$	$67.38$	$25.66$	$36.93$	$\mathbf{27.68}$
Examples-proportional, $K=2^{21}$	$79.83$	$18.79$	$79.50$	$65.10$	$25.82$	$37.22$	$27.13$
Temperature-scaled, $T=2$	$81.90$	$\mathbf{19.28}$	$79.42$	$69.92$	$25.42$	$36.72$	$27.20$
Temperature-scaled, $T=4$	$80.56$	$\mathbf{19.22}$	$77.99$	$69.54$	$25.04$	$35.82$	$27.45$
Temperature-scaled, $T=8$	$77.21$	$19.10$	$77.14$	$66.07$	$24.55$	$35.35$	$27.17$

\bigstar\,

混合策略	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ 基线 (pre-train/fine-tune)	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
均等	$76.13$	$19.02$	$76.51$	$63.37$	$23.89$	$34.31$	$26.78$
按例子比例, $K=2^{16}$	$80.45$	$19.04$	$77.25$	$69.95$	$24.35$	$34.99$	$27.10$
按例子比例, $K=2^{17}$	$81.56$	$19.12$	$77.00$	$67.91$	$24.36$	$35.00$	$27.25$
按例子比例, $K=2^{18}$	$81.67$	$19.07$	$78.17$	$67.94$	$24.57$	$35.19$	$27.39$
按例子比例, $K=2^{19}$	$81.42$	$\mathbf{19.24}$	$79.78$	$67.30$	$25.21$	$36.30$	$\mathbf{27.76}$
按例子比例, $K=2^{20}$	$80.80$	$\mathbf{19.24}$	$\mathbf{80.36}$	$67.38$	$25.66$	$36.93$	$\mathbf{27.68}$
按例子比例, $K=2^{21}$	$79.83$	$18.79$	$79.50$	$65.10$	$25.82$	$37.22$	$27.13$
温度缩放, $T=2$	$81.90$	$\mathbf{19.28}$	$79.42$	$69.92$	$25.42$	$36.72$	$27.20$
温度缩放, $T=4$	$80.56$	$\mathbf{19.22}$	$77.99$	$69.54$	$25.04$	$35.82$	$27.45$
温度缩放, $T=8$	$77.21$	$19.10$	$77.14$	$66.07$	$24.55$	$35.35$	$27.17$

K=2^{19}

3.5.3 将多任务学习与微调结合 回顾我们研究的是多任务学习的宽松版本：仍可对每项任务单独微调。自然的做法是先在多任务混合上预训练，再在各下游任务上微调。我们也考虑先无监督预训练，再多任务微调，以及先多任务预训练再微调等变体。

\bigstar\,

Training strategy	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ Unsupervised pre-training + fine-tuning	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$39.82$	$27.65$
Multi-task training	$81.42$	$\mathbf{19.24}$	$79.78$	$67.30$	$25.21$	$36.30$	$27.76$
Multi-task pre-training + fine-tuning	$\mathbf{83.11}$	$\mathbf{19.12}$	$\mathbf{80.26}$	$\mathbf{71.03}$	$\mathbf{27.08}$	$39.80$	$\mathbf{28.07}$
Leave-one-out multi-task training	$81.98$	$19.05$	$79.97$	$\mathbf{71.68}$	$\mathbf{26.93}$	$39.79$	$\mathbf{27.87}$
Supervised multi-task pre-training	$79.93$	$18.96$	$77.38$	$65.36$	$26.81$	$\mathbf{40.13}$	$\mathbf{28.04}$

\bigstar\,

训练策略	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ 无监督预训练 + fine-tuning	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$39.82$	$27.65$
多任务 training	$81.42$	$\mathbf{19.24}$	$79.78$	$67.30$	$25.21$	$36.30$	$27.76$
多任务 pre-training + fine-tuning	$\mathbf{83.11}$	$\mathbf{19.12}$	$\mathbf{80.26}$	$\mathbf{71.03}$	$\mathbf{27.08}$	$39.80$	$\mathbf{28.07}$
Leave-one-out multi-task training	$81.98$	$19.05$	$79.97$	$\mathbf{71.68}$	$\mathbf{26.93}$	$39.79$	$\mathbf{27.87}$
Supervised multi-task pre-training	$79.93$	$18.96$	$77.38$	$65.36$	$26.81$	$\mathbf{40.13}$	$\mathbf{28.04}$

K=2^{19}

我们在表 12 对照这些方法的结果。为对照，也纳入基线（先预训练再微调）与纯多任务训练结果。总体而言，先无监督预训练再微调仍然强劲；在监督与无监督任务混合上预训练再微调也可达到相近表现。

3.6 Scaling

3.6 缩放

4\times

机器学习研究的「苦涩教训」认为，能利用额外计算的一般方法最终会胜过依赖人类知识的方法（Sutton 2019）。因此，我们研究几种利用额外计算的方式如何影响表现。

220\mathrm{M}

220\mathrm{M}

4\times

4\times

4

4

\bigstar\,

Scaling strategy	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ Baseline	$83.28$	$19.24$	$80.88$	$71.36$	$26.98$	$39.82$	$27.65$
$1\times$ size, $4\times$ training steps	$85.33$	$19.33$	$82.45$	$74.72$	$27.08$	$40.66$	$27.93$
$1\times$ size, $4\times$ batch size	$84.60$	$19.42$	$82.52$	$74.64$	$27.07$	$40.60$	$27.84$
$2\times$ size, $2\times$ training steps	$\mathbf{86.18}$	$19.66$	$\mathbf{84.18}$	$77.18$	$27.52$	$\mathbf{41.03}$	$28.19$
$4\times$ size, $1\times$ training steps	$\mathbf{85.91}$	$19.73$	$\mathbf{83.86}$	$\mathbf{78.04}$	$27.47$	$40.71$	$28.10$
$4\times$ ensembled	$84.77$	$\mathbf{20.10}$	$83.09$	$71.74$	$\mathbf{28.05}$	$40.53$	$\mathbf{28.57}$
$4\times$ ensembled, fine-tune only	$84.05$	$19.57$	$82.36$	$71.55$	$27.55$	$40.22$	$28.09$

\bigstar\,

缩放策略	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ 基线	$83.28$	$19.24$	$80.88$	$71.36$	$26.98$	$39.82$	$27.65$
$1\times$ size, $4\times$ training steps	$85.33$	$19.33$	$82.45$	$74.72$	$27.08$	$40.66$	$27.93$
$1\times$ size, $4\times$ batch size	$84.60$	$19.42$	$82.52$	$74.64$	$27.07$	$40.60$	$27.84$
$2\times$ size, $2\times$ training steps	$\mathbf{86.18}$	$19.66$	$\mathbf{84.18}$	$77.18$	$27.52$	$\mathbf{41.03}$	$28.19$
$4\times$ size, $1\times$ training steps	$\mathbf{85.91}$	$19.73$	$\mathbf{83.86}$	$\mathbf{78.04}$	$27.47$	$40.71$	$28.10$
$4\times$ ensembled	$84.77$	$\mathbf{20.10}$	$83.09$	$71.74$	$\mathbf{28.05}$	$40.53$	$\mathbf{28.57}$
$4\times$ ensembled, fine-tune only	$84.05$	$19.57$	$82.36$	$71.55$	$27.55$	$40.22$	$28.09$

4\times

应用这些缩放方法后达到的表现见表 13。不出所料，增加训练时间与模型规模都会改善结果。更大模型通常给出最大收益。集成也有帮助，但收益小于训练单一更大模型。

N

我们注意到不同缩放方法有与表现无关的不同权衡。例如，使用更大模型会增加记忆占用与每步时间；更长训练增加总计算；集成增加解码成本。

3.7 Putting It All Together

3.7 汇总

We now leverage the insights from our systematic study to determine how far we can push performance on popular NLP benchmarks. We are also interested in exploring the current limits of transfer learning for NLP by training larger models on large amounts of data. We start with our baseline training approach and make the following changes:

我们现在利用系统研究的洞见，确定能在流行 NLP 基准上把表现推到多远。我们也有兴趣看到规模化能带来多大收益。

3

3

\nicefrac{{1}}{{4}}

\nicefrac{{1}}{{4}}

In Section 3.6 we also showed how scaling up the baseline model size improved performance. However, using smaller models can be helpful in settings where limited computational resources are available for fine-tuning or inference. Based on these factors, we train models with a wide range of sizes:

第 3.6 节也表明放大基线模型规模可改善表现。然而在某些情形下使用较小模型也有帮助。因此我们训练多种规模的模型：

220

220

d_{\mathrm{model}}=512

d_{\mathrm{model}}=512

d_{\mathrm{model}}=1{,}024

d_{\mathrm{model}}=1{,}024

d_{\mathrm{model}}=1024

d_{\mathrm{model}}=1{,}024

710{,}000

第 3.5.3 节表明，在微调前于无监督与监督任务的多任务混合上预训练，与仅无监督预训练后再微调效果相当。由于多任务预训练使模型在微调前暴露于下游任务，我们采用该方法。

2^{11}

到目前为止，在 GLUE 与 SuperGLUE 上微调时，我们把每个基准中的全部数据集拼接，从而只微调两个模型（各一个基准）。对最终结果，我们改为对每个数据集单独微调模型。

4

4

Since this is our final set of experiments, we report results on the test set rather than the validation set. For CNN/Daily Mail, we use the standard test set distributed with the data set. For the WMT tasks, this corresponds to using newstest2014 for English-German, newstest2015 for English-French, and newstest2016 for English-Romanian. For GLUE and SuperGLUE, we used the benchmark evaluation servers to compute official test set scores.[^15],[^16] For SQuAD, evaluating on the test set requires running inference on a benchmark server. Unfortunately, the computational resources on this server are insufficient for obtaining predictions from our largest models. As a result, we instead continue to report performance on the SQuAD validation set. Fortunately, the model with the highest performance on the SQuAD test set also reported results on the validation set, so we can still compare to what is ostensibly the state-of-the-art.

由于这是最终一组实验，我们在测试集而非验证集上报告结果。对 CNN/Daily Mail，使用非匿名版本并报告 ROUGE 分数。对 SQuAD，报告精确匹配与 F1。对翻译任务，用 sacreBLEU 报告 BLEU。

Apart from those changes mentioned above, we use the same training procedure and hyperparameters as our baseline (AdaFactor optimizer, inverse square root learning rate schedule for pre-training, constant learning rate for fine-tuning, dropout regularization, vocabulary, etc.). For reference, these details are described in Section 2.

除上述改动外，我们使用与基线相同的训练流程与超参（AdaFactor 优化器、逆平方根学习率日程等）。

89.4

	GLUE	CoLA	SST-2	MRPC	MRPC	STS-B	STS-B
Model	Average	Matthew’s	Accuracy	F1	Accuracy	Pearson	Spearman
Previous best	$89.4$ a	$69.2$ b	$97.1$ a	$\mathbf{93.6}$ b	$\mathbf{91.5}$ b	$92.7$ b	$92.3$ b
T5-Small	$77.4$	$41.0$	$91.8$	$89.7$	$86.6$	$85.6$	$85.0$
T5-Base	$82.7$	$51.1$	$95.2$	$90.7$	$87.5$	$89.4$	$88.6$
T5-Large	$86.4$	$61.2$	$96.3$	$92.4$	$89.9$	$89.9$	$89.2$
T5-3B	$88.5$	$67.1$	$97.4$	$92.5$	$90.0$	$90.6$	$89.8$
T5-11B	$\mathbf{90.3}$	$\mathbf{71.6}$	$\mathbf{97.5}$	$92.8$	$90.4$	$\mathbf{93.1}$	$\mathbf{92.8}$

	QQP	QQP	MNLI-m	MNLI-mm	QNLI	RTE	WNLI
Model	F1	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy
Previous best	$74.8$ c	$\mathbf{90.7}$ b	$91.3$ a	$91.0$ a	$\mathbf{99.2}$ a	$89.2$ a	$91.8$ a
T5-Small	$70.0$	$88.0$	$82.4$	$82.3$	$90.3$	$69.9$	$69.2$
T5-Base	$72.6$	$89.4$	$87.1$	$86.2$	$93.7$	$80.1$	$78.8$
T5-Large	$73.9$	$89.9$	$89.9$	$89.6$	$94.8$	$87.2$	$85.6$
T5-3B	$74.4$	$89.7$	$91.4$	$91.2$	$96.3$	$91.1$	$89.7$
T5-11B	$\mathbf{75.1}$	$90.6$	$\mathbf{92.2}$	$\mathbf{91.9}$	$96.9$	$\mathbf{92.8}$	$\mathbf{94.5}$

	SQuAD	SQuAD	SuperGLUE	BoolQ	CB	CB	COPA
Model	EM	F1	Average	Accuracy	F1	Accuracy	Accuracy
Previous best	$90.1$ a	$95.5$ a	$84.6$ d	$87.1$ d	$90.5$ d	$95.2$ d	$90.6$ d
T5-Small	$79.10$	$87.24$	$63.3$	$76.4$	$56.9$	$81.6$	$46.0$
T5-Base	$85.44$	$92.08$	$76.2$	$81.4$	$86.2$	$94.0$	$71.2$
T5-Large	$86.66$	$93.79$	$82.3$	$85.4$	$91.6$	$94.8$	$83.4$
T5-3B	$88.53$	$94.95$	$86.4$	$89.9$	$90.3$	$94.4$	$92.0$
T5-11B	$\mathbf{91.26}$	$\mathbf{96.22}$	$\mathbf{88.9}$	$\mathbf{91.2}$	$\mathbf{93.9}$	$\mathbf{96.8}$	$\mathbf{94.8}$

	MultiRC	MultiRC	ReCoRD	ReCoRD	RTE	WiC	WSC
Model	F1a	EM	F1	Accuracy	Accuracy	Accuracy	Accuracy
Previous best	$84.4$ d	$52.5$ d	$90.6$ d	$90.0$ d	$88.2$ d	$69.9$ d	$89.0$ d
T5-Small	$69.3$	$26.3$	$56.3$	$55.4$	$73.3$	$66.9$	$70.5$
T5-Base	$79.7$	$43.1$	$75.0$	$74.2$	$81.5$	$68.3$	$80.8$
T5-Large	$83.3$	$50.7$	$86.8$	$85.9$	$87.8$	$69.3$	$86.3$
T5-3B	$86.8$	$58.3$	$91.2$	$90.4$	$90.7$	$72.1$	$90.4$
T5-11B	$\mathbf{88.1}$	$\mathbf{63.3}$	$\mathbf{94.1}$	$\mathbf{93.4}$	$\mathbf{92.5}$	$\mathbf{76.9}$	$\mathbf{93.8}$

	WMT EnDe	WMT EnFr	WMT EnRo	CNN/DM	CNN/DM	CNN/DM
Model	BLEU	BLEU	BLEU	ROUGE-1	ROUGE-2	ROUGE-L
Previous best	$\mathbf{33.8}$ e	$\mathbf{43.8}$ e	$\mathbf{38.5}$ f	$43.47$ g	$20.30$ g	$40.63$ g
T5-Small	$26.7$	$36.0$	$26.8$	$41.12$	$19.56$	$38.35$
T5-Base	$30.9$	$41.2$	$28.0$	$42.05$	$20.34$	$39.40$
T5-Large	$32.0$	$41.5$	$28.1$	$42.50$	$20.68$	$39.75$
T5-3B	$31.8$	$42.6$	$28.2$	$42.72$	$21.02$	$39.94$
T5-11B	$32.1$	$43.4$	$28.1$	$\mathbf{43.52}$	$\mathbf{21.55}$	$\mathbf{40.69}$

89.4

	GLUE	CoLA	SST-2	MRPC	MRPC	STS-B	STS-B
Model	Average	Matthew’s	Accuracy	F1	Accuracy	Pearson	Spearman
Previous best	$89.4$ a	$69.2$ b	$97.1$ a	$\mathbf{93.6}$ b	$\mathbf{91.5}$ b	$92.7$ b	$92.3$ b
T5-Small	$77.4$	$41.0$	$91.8$	$89.7$	$86.6$	$85.6$	$85.0$
T5-Base	$82.7$	$51.1$	$95.2$	$90.7$	$87.5$	$89.4$	$88.6$
T5-Large	$86.4$	$61.2$	$96.3$	$92.4$	$89.9$	$89.9$	$89.2$
T5-3B	$88.5$	$67.1$	$97.4$	$92.5$	$90.0$	$90.6$	$89.8$
T5-11B	$\mathbf{90.3}$	$\mathbf{71.6}$	$\mathbf{97.5}$	$92.8$	$90.4$	$\mathbf{93.1}$	$\mathbf{92.8}$

	QQP	QQP	MNLI-m	MNLI-mm	QNLI	RTE	WNLI
Model	F1	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy
Previous best	$74.8$ c	$\mathbf{90.7}$ b	$91.3$ a	$91.0$ a	$\mathbf{99.2}$ a	$89.2$ a	$91.8$ a
T5-Small	$70.0$	$88.0$	$82.4$	$82.3$	$90.3$	$69.9$	$69.2$
T5-Base	$72.6$	$89.4$	$87.1$	$86.2$	$93.7$	$80.1$	$78.8$
T5-Large	$73.9$	$89.9$	$89.9$	$89.6$	$94.8$	$87.2$	$85.6$
T5-3B	$74.4$	$89.7$	$91.4$	$91.2$	$96.3$	$91.1$	$89.7$
T5-11B	$\mathbf{75.1}$	$90.6$	$\mathbf{92.2}$	$\mathbf{91.9}$	$96.9$	$\mathbf{92.8}$	$\mathbf{94.5}$

	SQuAD	SQuAD	SuperGLUE	BoolQ	CB	CB	COPA
Model	EM	F1	Average	Accuracy	F1	Accuracy	Accuracy
Previous best	$90.1$ a	$95.5$ a	$84.6$ d	$87.1$ d	$90.5$ d	$95.2$ d	$90.6$ d
T5-Small	$79.10$	$87.24$	$63.3$	$76.4$	$56.9$	$81.6$	$46.0$
T5-Base	$85.44$	$92.08$	$76.2$	$81.4$	$86.2$	$94.0$	$71.2$
T5-Large	$86.66$	$93.79$	$82.3$	$85.4$	$91.6$	$94.8$	$83.4$
T5-3B	$88.53$	$94.95$	$86.4$	$89.9$	$90.3$	$94.4$	$92.0$
T5-11B	$\mathbf{91.26}$	$\mathbf{96.22}$	$\mathbf{88.9}$	$\mathbf{91.2}$	$\mathbf{93.9}$	$\mathbf{96.8}$	$\mathbf{94.8}$

	MultiRC	MultiRC	ReCoRD	ReCoRD	RTE	WiC	WSC
Model	F1a	EM	F1	Accuracy	Accuracy	Accuracy	Accuracy
Previous best	$84.4$ d	$52.5$ d	$90.6$ d	$90.0$ d	$88.2$ d	$69.9$ d	$89.0$ d
T5-Small	$69.3$	$26.3$	$56.3$	$55.4$	$73.3$	$66.9$	$70.5$
T5-Base	$79.7$	$43.1$	$75.0$	$74.2$	$81.5$	$68.3$	$80.8$
T5-Large	$83.3$	$50.7$	$86.8$	$85.9$	$87.8$	$69.3$	$86.3$
T5-3B	$86.8$	$58.3$	$91.2$	$90.4$	$90.7$	$72.1$	$90.4$
T5-11B	$\mathbf{88.1}$	$\mathbf{63.3}$	$\mathbf{94.1}$	$\mathbf{93.4}$	$\mathbf{92.5}$	$\mathbf{76.9}$	$\mathbf{93.8}$

	WMT EnDe	WMT EnFr	WMT EnRo	CNN/DM	CNN/DM	CNN/DM
Model	BLEU	BLEU	BLEU	ROUGE-1	ROUGE-2	ROUGE-L
Previous best	$\mathbf{33.8}$ e	$\mathbf{43.8}$ e	$\mathbf{38.5}$ f	$43.47$ g	$20.30$ g	$40.63$ g
T5-Small	$26.7$	$36.0$	$26.8$	$41.12$	$19.56$	$38.35$
T5-Base	$30.9$	$41.2$	$28.0$	$42.05$	$20.34$	$39.40$
T5-Large	$32.0$	$41.5$	$28.1$	$42.50$	$20.68$	$39.75$
T5-3B	$31.8$	$42.6$	$28.2$	$42.72$	$21.02$	$39.94$
T5-11B	$32.1$	$43.4$	$28.1$	$\mathbf{43.52}$	$\mathbf{21.55}$	$\mathbf{40.69}$

18

24

90.3

90.3

82.30

对 SQuAD，我们在精确匹配分数上超出先前最优（ALBERT（Lan et al. 2019））超过一分。SQuAD 是长期存在的基准，持续改进表明扩展文本到文本模型仍然有效。

84.6

84.6

We did not achieve state-of-the-art performance on any of the WMT translation tasks. This may be in part due to our use of an English-only unlabeled data set. We also note that most of the best results on these tasks use backtranslation (Edunov et al. 2018; Lample and Conneau 2019), which is a sophisticated data augmentation scheme. The state of the art on the low-resource English to Romanian benchmark also uses additional forms of cross-lingual unsupervised training (Lample and Conneau 2019). Our results suggest that scale and English-language pre-training may be insufficient to match the performance of these more sophisticated methods. On a more specific note, the best results on English to German newstest2014 set use the much larger training set from WMT 2018 (Edunov et al. 2018), making direct comparison to our results difficult.

我们未能在任一 WMT 翻译任务上达到最优。这部分可能因为我们使用仅英语的无监督预训练数据，尽管词表覆盖了目标语言。多语言预训练可能有帮助。

Finally, on CNN/Daily Mail we attain state-of-the-art performance, though only by a significant amount on the ROUGE-2-F score. It has been shown that improvements to the ROUGE score do not necessarily correspond to more coherent summaries (Paulus et al. 2017). Furthermore, while CNN/Daily Mail is posed as an abstractive summarization benchmark, purely extractive approaches have been shown to work well (Liu 2019). It has also been argued that generative models trained with maximum likelihood are prone to producing repetitive summaries (See et al. 2017). Despite these potential issues, we find that our models do generate coherent and largely correct summaries. We provide some non-cherry-picked validation set examples in Appendix C.

最后，在 CNN/Daily Mail 上我们达到最优，不过仅在 ROUGE-2-F 分数上有显著优势。有人指出 ROUGE 分数不一定反映摘要质量，但该结果表明我们的模型也能胜任生成任务。

2^{35}\approx 34\mathrm{B}

为取得强劲结果，T5 把实验研究的洞见与前所未有的规模结合起来。注意第 3.6 节发现放大模型是利用额外计算的最有效方式之一。我们的 11B 参数模型显著大于先前用于这些任务的模型。

The performance of these three model configurations is shown in Table 15. Consistent with the findings in Section 3.6, we find that additional pre-training improves performance over the baseline. Nevertheless, T5-Base substantially outperforms baseline-1T on all downstream tasks. This suggests that scale is not the only factor that contributes to T5’s success. We hypothesize that the larger models benefit not only from their increased size but also from these non-scaling factors.

这三种模型配置的表现见表 15。与第 3.6 节发现一致，我们看到额外参数持续改善下游表现。

\bigstar\,

Model	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ Baseline	$83.28$	$19.24$	$80.88$	$71.36$	$26.98$	$39.82$	$27.65$
Baseline-1T	$84.80$	$19.62$	$83.01$	$73.90$	$27.46$	$40.30$	$28.34$
T5-Base	$\mathbf{85.97}$	$\mathbf{20.90}$	$\mathbf{85.44}$	$\mathbf{75.64}$	$\mathbf{28.37}$	$\mathbf{41.37}$	$\mathbf{28.98}$

\bigstar\,

Model	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$ Baseline	$83.28$	$19.24$	$80.88$	$71.36$	$26.98$	$39.82$	$27.65$
Baseline-1T	$84.80$	$19.62$	$83.01$	$73.90$	$27.46$	$40.30$	$28.34$
T5-Base	$\mathbf{85.97}$	$\mathbf{20.90}$	$\mathbf{85.44}$	$\mathbf{75.64}$	$\mathbf{28.37}$	$\mathbf{41.37}$	$\mathbf{28.98}$

4 Reflection

4 反思

Having completed our systematic study, we wrap up by first recapping some of our most significant findings. Our results provide some high-level perspective on which avenues of research might be more or less promising. To conclude, we outline some topics we think might provide effective approaches for further progressing the field.

完成系统研究后，我们先回顾一些最重要的发现作为收束。我们的结果为迁移学习管线的哪些部分最重要提供了一些高层洞见。然后讨论对未来工作的展望。

4.1 Takeaways

4.1 要点

Our text-to-text framework provides a simple way to train a single model on a wide variety of text tasks using the same loss function and decoding procedure. We showed how this approach can be successfully applied to generative tasks like abstractive summarization, classification tasks like natural language inference, and even regression tasks like STS-B. In spite of its simplicity, we found the text-to-text framework obtained comparable performance to task-specific architectures and ultimately produced state-of-the-art results when combined with scale.

我们的文本到文本框架提供了一种简单方式：用相同损失函数与解码过程，在广泛文本任务上训练单一模型。除提供标准化方法外，该框架也使我们能探索编码器—解码器生成模型在通常不视为生成问题的任务（如分类）上的使用。

While some work on transfer learning for NLP has considered architectural variants of the Transformer, we found the original encoder-decoder form worked best in our text-to-text framework. Though an encoder-decoder model uses twice as many parameters as “encoder-only” (e.g. BERT) or “decoder-only” (language model) architectures, it has a similar computational cost. We also showed that sharing the parameters in the encoder and decoder did not result in a substantial performance drop while halving the total parameter count.

尽管一些 NLP 迁移学习工作考虑了 Transformer 的架构变体，我们发现原始编码器—解码器形式在分类与生成任务上都表现良好。

Overall, we found that most “denoising” objectives, which train the model to reconstruct randomly corrupted text, performed similarly in the text-to-text setup. As a result, we suggest using objectives that produce short target sequences so that unsupervised pre-training is more computationally efficient.

总体而言，我们发现多数「去噪」目标——训练模型重建随机腐蚀的文本——在我们的设定下表现相近。跨度腐蚀提供了良好的表现与效率折中。

We introduced the “Colossal Clean Crawled Corpus” (C4), which comprises heuristically-cleaned text from the Common Crawl web dump. When comparing C4 to data sets that use additional filtering, we found that training on in-domain unlabeled data could boost performance in a few downstream tasks. However, constraining to a single domain typically results in a smaller data set. We separately showed that performance can degrade when an unlabeled data set is small enough that it is repeated many times over the course of pre-training. This motivates the use of a large and diverse data set like C4 for generic language understanding tasks.

我们引入了「超大规模清洁爬取语料」（C4），它包含经启发式清洗的 Common Crawl 网络转储文本。与未过滤的网络文本相比，我们的清洗流水线显著改善下游表现。较大的预训练数据集也有帮助，因为它们避免了在预训练期间重复数据。

We found that the basic approach of updating all of a pre-trained model’s parameters during fine-tuning outperformed methods that are designed to update fewer parameters, although updating all parameters is most expensive. We also experimented with various approaches for training the model on multiple tasks at once, which in our text-to-text setting simply corresponds to mixing examples from different data sets when constructing batches. The primary concern in multi-task learning is setting the proportion of each task to train on. We ultimately did not find a strategy for setting mixing proportions that matched the performance of the basic approach of unsupervised pre-training followed by supervised fine-tuning. However, we found that fine-tuning after pre-training on a mixture of tasks produced comparable performance to unsupervised pre-training.

我们发现微调期间更新预训练模型全部参数的基本做法，优于为参数效率或缓解过拟合而设计的方法。多任务训练有前景，但我们尚未找到稳定优于「先预训练再微调」的混合策略。

We compared various strategies for taking advantage of additional compute, including training the model on more data, training a larger model, and using an ensemble of models. We found each approach conferred a significant boost in performance, though training a smaller model on more data was often outperformed by training a larger model for fewer steps. We also showed an ensemble of models can provide substantially better results than a single model, which provides an orthogonal means of leveraging additional computation. Ensembling models that were fine-tuned from the same base pre-trained model performed worse than pre-training and fine-tuning all models completely separately, though fine-tune-only ensembling still substantially outperformed a single model.

我们比较了利用额外计算的多种策略，包括在更多数据上训练模型、训练更大模型、使用更大批次以及模型集成。总体而言，更大模型通常给出最佳表现。

11

11

4.2 Outlook

4.2 展望

An unsurprising but important result from our study is that larger models tend to perform better. The fact that the hardware used for running these models is continually getting cheaper and more powerful suggests that scaling up may continue to be a promising way to achieve better performance (Sutton 2019). However, it will always be the case that there are applications and scenarios where using a smaller or less expensive model is helpful, for example when performing client-side inference or federated learning (Konečnỳ et al. 2015; Konečnỳ et al. 2016). Relatedly, one beneficial use of transfer learning is the possibility of attaining good performance on low-resource tasks. Low-resource tasks often occur (by definition) in settings where one lacks the assets to label more data. It follows that low-resource applications often also have limited access to computational resources which can incur additional costs. As a result, we advocate for research on methods that achieve stronger performance with cheaper models so that transfer learning can be applied where it will have the most impact. Some current work along these lines include distillation (Hinton et al. 2015; Sanh et al. 2019; Jiao et al. 2019), parameter sharing (Lan et al. 2019), and conditional computation (Shazeer et al. 2017).

我们研究中一个不意外但重要的结果是：更大模型往往表现更好。用于运行这些模型的硬件持续改进，表明扩展规模仍有空间。然而，开发能在受限计算预算下表现良好的方法，对让这些技术进步更广泛可用也很重要。

1

回顾预训练的目标之一是（宽泛地说）为模型提供改善下游任务表现的通用「知识」。除扩展规模外，开发更好的方法以从无标注数据抽取知识，可能带来进一步收益。

We observed that pre-training on unlabeled in-domain data can improve performance on downstream tasks (Section 3.4). This finding mostly relies on basic observations like the fact that SQuAD was created using data from Wikipedia. It would be useful to formulate a more rigorous notion of the “similarity” between the pre-training and downstream tasks, so that we could make more principled choices about what source of unlabeled data to use. There is some early empirical work along these lines in the field of computer vision (Huh et al. 2016; Kornblith et al. 2018; He et al. 2018). A better notion of the relatedness of tasks could also help choose supervised pre-training tasks, which has been shown to be helpful for the GLUE benchmark (Phang et al. 2018).

我们观察到在无标注领域内数据上预训练可改善下游任务表现（第 3.4 节）。该发现主要适用于有大量未标注领域内文本可用的情形。对低资源任务，开发能更好利用有限数据的方法仍然重要。

We were disappointed to find that English-only pre-training did not achieve state-of-the-art results on the translation tasks we studied. We also are interested in avoiding the logistical difficulty of needing to specify which languages a vocabulary can encode ahead of time. To address these issues, we are interested in further investigating language-agnostic models, i.e. models that can perform a given NLP task with good performance regardless of the text’s language. This is an especially pertinent issue given that English is not the native language for the majority of the world’s population.

我们失望地发现仅英语预训练未能在所研究的翻译任务上达到最优。我们怀疑多语言预训练目标与数据对改善机器翻译表现必要。

The motivation for this paper was the flurry of recent work on transfer learning for NLP. Before we began this work, these advances had already enabled breakthroughs in settings where learning-based methods had not yet been shown to be effective. We are happy to be able to continue this trend, for example by nearly matching human-level performance on the SuperGLUE benchmark, a task specifically designed to be difficult for modern transfer-learning pipelines. Our results stem from the combination of a straightforward and unified text-to-text framework, our new C4 data set, and insights from our systematic study. Additionally, we provided an empirical overview of the field and a perspective on where it stands. We are excited to see continued work using transfer learning towards the goal of general language understanding.

本文的动机是近期 NLP 迁移学习工作的热潮。在开始这项工作之前，这些进展已推动几乎所有流行英语 NLP 基准上的最优结果。我们希望我们的结果与洞见能帮助该领域继续前进。

Appendix A Contributions

附录 A 贡献

acknowledgments-disclosure-of-funding. Colin designed the scope of this project and wrote this paper, ran all the experiments in Sections 3.1, 3.2, 3.3, 3.4, 3.5 and 3.6, and contributed a large portion of our codebase. Noam contributed many of the ideas, including the text-to-text framework, unsupervised objectives, and data set mixing strategies; implemented our base Transformer model and its architectural variants; and ran the experiments in Section 3.7. Adam oversaw all engineering aspects for this project, created the C4 data set, implemented our data set pipeline, and added various benchmark data sets. Katherine coordinated experiments, wrote and updated documentation, ran experiments to help design our baseline, and contributed to many parts of our codebase. Sharan contributed some of the required data sets and preprocessors, and ran assorted preliminary experiments, in addition to co-leading the open-sourcing of our codebase. Michael owned all aspects of the Winograd data sets, ingested many of the data sets we used, contributed various improvements and fixes to our infrastructure, and ran some preliminary experiments. Yanqi ran experiments and implemented methods to help settle on a reasonable baseline and helped with the final fine-tuning of the models in Section 3.7. Wei also helped with final fine-tuning and improved some of our preprocessors. Peter prototyped an early version of the pre-training data set and resolved issues pertaining to the SQuAD and CNN/DM tasks. All authors helped set the scope and research direction we followed in this work.

致谢与资助披露。 Colin 设计了本项目范围并撰写本文。Noam 设计了用于高效训练的 Mesh TensorFlow Transformer 实现，并提供了宝贵指导。Adam 构建了基础设施，用于在 C4、下游数据集与词表上训练模型并跑实验。Katherine 清理了 Common Crawl 数据以产生 C4，并跑了数据相关实验。Sharan 实现并实验了不同预训练目标，并收集了相关工作结果。Michael 实现并实验了架构变体。Yanqi 与 Wei 实验了不同微调策略。Peter 提供了项目管理与指导。 我们感谢 Google Brain 团队成员的讨论与反馈。本研究未接受第三方资助。

Appendix B Converting WNLI to Our Text-to-Text Format

附录 B 将 WNLI 转换为我们的文本到文本格式

Note that as discussed in Section 2.4, we do not train on any of the data from WNLI. Instead, when evaluating on the WNLI test set (for the results in Section 3.7), we convert the WNLI test set to the “referent noun prediction” text-to-text format so that we can evaluate using a model trained on WSC and DPR. Our WNLI preprocessor is inspired by the one proposed by He et al. 2019. Recall that examples from WNLI consist of a premise, a hypothesis, and a label that indicates whether the hypothesis is True or False. Using the example from Section 2.4, the hypothesis would be “The city councilmen refused the demonstrators a permit because they feared violence.” with the premise “The demonstrators feared violence.” and the label False. We first find the location of all pronouns in the premise (“they” in our example). Then, we find the maximum number of words that precede or follow each pronoun that are a substring in the hypothesis (“feared violence” in our example), ignoring case and punctuation. When the premise contains multiple candidate pronouns, we choose the pronoun that is preceded or followed by the largest substring of the hypothesis. We then highlight the pronoun in the premise by surrounding it with asterisks. For the candidate noun (which is compared to our model’s prediction to obtain a True or False label), we remove the matching substring from the hypothesis and optionally make it non-possessive (resulting in “the demonstrators”).

注意如第 2.4 节所述，我们不在 WNLI 的任何数据上训练。相反，评估 GLUE 时，我们用在 DPR 上微调的模型对 WNLI 例子打分。为把 WNLI 转换成文本到文本格式，我们遵循与 WSC 类似的方案。

Appendix C Example Predictions on CNN/Daily Mail

附录 C CNN/Daily Mail 上的预测例子

To show that our model is generating fluent summaries, we include a few example decodes from our best model (T5-11B) on the validation set along with the ground-truth summaries. These examples selected at random and were not cherry-picked.

为表明我们的模型在生成流畅摘要，我们纳入来自最佳模型的若干解码例子。对每个例子，我们展示真实摘要与模型预测。

Ground-truth: leopard gave up after spiky creature refused to back down in fight in kruger national park, south africa . wildlife enthusiast lisl moolman, 41, caught the bizarre battle while out on the road and armed with her camera . the leopard and porcupine tumbled out of the bushes and began to fight by roadside - watched by ms moolman . Prediction: leopard tried to make lunch out of a plucky porcupine in kruger national park, south africa . but the predator was put firmly in its place after the spiky creature refused to back down during a fight . wildlife enthusiast lisl moolman, 41, caught the bizarre battle while out on the road and armed with her camera .

真实摘要 ： leopard gave up after spiky creature refused to back down in fight in kruger national park, south africa . wildlife enthusiast lisl moolman, 41, caught the bizarre battle while out on the road and armed with her camera . the leopard and porcupine tumbled out of the bushes and began to fight by roadside - watched by ms moolman . 预测 ： leopard tried to make lunch out of a plucky porcupine in kruger national park, south africa . but the predator was put firmly in its place after the spiky creature refused to back down during a fight . wildlife enthusiast lisl moolman, 41, caught the bizarre battle while out on the road and armed with her camera .

Ground-truth: researchers say homes are a hotbed of undiscovered species . study of 40 north carolina homes found 8,000 bacterial and archaeal taxa . 11 houses in california revealed a great variety of fungus among us . Prediction: researchers estimate that the indoor biome makes up about 0.5 percent of ice-free land, or about 247,000 square miles, almost the size of texas . a study of 40 houses in north carolina used genomic technologies to document more than 8,000 bacterial and archaeal taxa . another study of 50 houses in the same state described more than 750 types of arthropods, including insects and spiders . an examination of 11 houses in california found a great variety of fungus among us .

真实摘要 ： researchers say homes are a hotbed of undiscovered species . study of 40 north carolina homes found 8,000 bacterial and archaeal taxa . 11 houses in california revealed a great variety of fungus among us . 预测 ： researchers estimate that the indoor biome makes up about 0.5 percent of ice-free land, or about 247,000 square miles, almost the size of texas . a study of 40 houses in north carolina used genomic technologies to document more than 8,000 bacterial and archaeal taxa . another study of 50 houses in the same state described more than 750 types of arthropods, including insects and spiders . an examination of 11 houses in california found a great variety of fungus among us .

Ground-truth: executions have been temporarily halted in the state after the drug used for lethal injections appeared faulty on monday night . kelly gissendaner had her death delayed for a second time when it was found that the lethal pentobarbital dose appeared cloudy . state officials dithered over the decision , changing their mind three times before deciding not to take the risk that it wouldn’t work . it is ’essential that executions are carried out in a constitutional manner,’ said georgia attorney general sam olens . Prediction: the execution of kelly gissendaner, the only woman on georgia’s death row, was delayed for a second time on monday . the execution team found that the lethal pentobarbital dose appeared cloudy . the cloudy drug bolstered death penalty opponents, who have been vocal in their opposition after three botched executions in other parts of the country .

真实摘要 ： executions have been temporarily halted in the state after the drug used for lethal injections appeared faulty on monday night . kelly gissendaner had her death delayed for a second time when it was found that the lethal pentobarbital dose appeared cloudy . state officials dithered over the decision , changing their mind three times before deciding not to take the risk that it wouldn’t work . it is ’essential that executions are carried out in a constitutional manner,’ said georgia attorney general sam olens . 预测 ： the execution of kelly gissendaner, the only woman on georgia’s death row, was delayed for a second time on monday . the execution team found that the lethal pentobarbital dose appeared cloudy . the cloudy drug bolstered death penalty opponents, who have been vocal in their opposition after three botched executions in other parts of the country .

Ground-truth: dani alves was not selected for the brazil squad to face france and chile . barcelona defender was filmed serving up hot dogs to people on saturday . this week alves released a charity single with former team-mat jose pinto . alves looks set to leave barcelona on a free transfer this summer . reports in spanish press claim he will sign for paris saint-germain . Prediction: dani alves was not selected for brazil’s friendlies against france and chile . the barcelona right back has released a charity single with jose pinto . alves was filmed serving up snacks from behind the counter of a hot dog bar on saturday . alves is yet to be offered a new deal at the nou camp . click here for all the latest barcelona news .

真实摘要 ： dani alves was not selected for the brazil squad to face france and chile . barcelona defender was filmed serving up hot dogs to people on saturday . this week alves released a charity single with former team-mat jose pinto . alves looks set to leave barcelona on a free transfer this summer . reports in spanish press claim he will sign for paris saint-germain . 预测 ： dani alves was not selected for brazil’s friendlies against france and chile . the barcelona right back has released a charity single with jose pinto . alves was filmed serving up snacks from behind the counter of a hot dog bar on saturday . alves is yet to be offered a new deal at the nou camp . click here for all the latest barcelona news .

Appendix D Preprocessed Examples

附录 D 预处理例子

In this section, we provide examples of our preprocessing for each of the data sets we consider.

本节给出我们所考虑的每个数据集的预处理例子。

D.1 CoLA

John made Bill master of himself.

cola sentence: John made Bill master of himself.

1

acceptable

D.2 RTE

A smaller proportion of Yugoslavia’s Italians were settled in Slovenia (at the 1991 national census, some 3000 inhabitants of Slovenia declared themselves as ethnic Italians).

Slovenia has 3,000 inhabitants.

rte sentence1: A smaller proportion of Yugoslavia’s Italians were settled in Slovenia (at the 1991 national census, some 3000 inhabitants of Slovenia declared themselves as ethnic Italians). sentence2: Slovenia has 3,000 inhabitants.

1

not_entailment

D.3 MNLI

The St. Louis Cardinals have always won.

yeah well losing is i mean i’m i’m originally from Saint Louis and Saint Louis Cardinals when they were there were uh a mostly a losing team but

mnli hypothesis: The St. Louis Cardinals have always won. premise: yeah well losing is i mean i’m i’m originally from Saint Louis and Saint Louis Cardinals when they were there were uh a mostly a losing team but

2

contradiction

D.4 MRPC

We acted because we saw the existing evidence in a new light , through the prism of our experience on 11 September , " Rumsfeld said .

Rather , the US acted because the administration saw " existing evidence in a new light , through the prism of our experience on September 11 " .

mrpc sentence1: We acted because we saw the existing evidence in a new light , through the prism of our experience on 11 September , " Rumsfeld said . sentence2: Rather , the US acted because the administration saw " existing evidence in a new light , through the prism of our experience on September 11 " .

1

equivalent

D.5 QNLI

Where did Jebe die?

Genghis Khan recalled Subutai back to Mongolia soon afterwards, and Jebe died on the road back to Samarkand.

qnli question: Where did Jebe die? sentence: Genghis Khan recalled Subutai back to Mongolia soon afterwards, and Jebe died on the road back to Samarkand.

0

entailment

D.6 QQP

What attributes would have made you highly desirable in ancient Rome?

How I GET OPPERTINUTY TO JOIN IT COMPANY AS A FRESHER?

qqp question1: What attributes would have made you highly desirable in ancient Rome? question2: How I GET OPPERTINUTY TO JOIN IT COMPANY AS A FRESHER?

0

not_duplicate

D.7 SST2

it confirms fincher ’s status as a film maker who artfully bends technical know-how to the service of psychological insight .

sst2 sentence: it confirms fincher ’s status as a film maker who artfully bends technical know-how to the service of psychological insight .

1

positive

D.8 STSB

Representatives for Puretunes could not immediately be reached for comment Wednesday.

Puretunes representatives could not be located Thursday to comment on the suit.

stsb sentence1: Representatives for Puretunes could not immediately be reached for comment Wednesday. sentence2: Puretunes representatives could not be located Thursday to comment on the suit.

3.25

3.2

D.9 CB

Valence was helping

Valence the void-brain, Valence the virtuous valet. Why couldn’t the figger choose his own portion of titanic anatomy to shaft? Did he think he was helping?

cb hypothesis: Valence was helping premise: Valence the void-brain, Valence the virtuous valet. Why couldn’t the figger choose his own portion of titanic anatomy to shaft? Did he think he was helping?

1

contradiction

D.10 COPA

effect

Political violence broke out in the nation.

Many citizens relocated to the capitol.

Many citizens took refuge in other territories.

copa choice1: Many citizens relocated to the capitol. choice2: Many citizens took refuge in other territories. premise: Political violence broke out in the nation. question: effect

1

True

D.11 MultiRC

There was only pie to eat, rather than traditional breakfast foods

Sent 1: Once upon a time, there was a squirrel named Joey.Sent 2: Joey loved to go outside and play with his cousin Jimmy.Sent 3: Joey and Jimmy played silly games together, and were always laughing.Sent 4: One day, Joey and Jimmy went swimming together at their Aunt Julie’s pond.Sent 5: Joey woke up early in the morning to eat some food before they left.Sent 6: He couldn’t find anything to eat except for pie!Sent 7: Usually, Joey would eat cereal, fruit (a pear), or oatmeal for breakfast.Sent 8: After he ate, he and Jimmy went to the pond.Sent 9: On their way there they saw their friend Jack Rabbit.Sent 10: They dove into the water and swam for several hours.Sent 11: The sun was out, but the breeze was cold.Sent 12: Joey and Jimmy got out of the water and started walking home.Sent 13: Their fur was wet, and the breeze chilled them.Sent 14: When they got home, they dried off, and Jimmy put on his favorite purple shirt.Sent 15: Joey put on a blue shirt with red and green dots.Sent 16: The two squirrels ate some food that Joey’s mom, Jasmine, made and went off to bed.

Why was Joey surprised the morning he woke up for breakfast?

multirc question: Why was Joey surprised the morning he woke up for breakfast? answer: There was only pie to eat, rather than traditional breakfast foods paragraph: Sent 1: Once upon a time, there was a squirrel named Joey.Sent 2: Joey loved to go outside and play with his cousin Jimmy.Sent 3: Joey and Jimmy played silly games together, and were always laughing.Sent 4: One day, Joey and Jimmy went swimming together at their Aunt Julie’s pond.Sent 5: Joey woke up early in the morning to eat some food before they left.Sent 6: He couldn’t find anything to eat except for pie!Sent 7: Usually, Joey would eat cereal, fruit (a pear), or oatmeal for breakfast.Sent 8: After he ate, he and Jimmy went to the pond.Sent 9: On their way there they saw their friend Jack Rabbit.Sent 10: They dove into the water and swam for several hours.Sent 11: The sun was out, but the breeze was cold.Sent 12: Joey and Jimmy got out of the water and started walking home.Sent 13: Their fur was wet, and the breeze chilled them.Sent 14: When they got home, they dried off, and Jimmy put on his favorite purple shirt.Sent 15: Joey put on a blue shirt with red and green dots.Sent 16: The two squirrels ate some food that Joey’s mom, Jasmine, made and went off to bed.

1

True

D.12 WiC

N

It was the deliberation of his act that was insulting .

The deliberations of the jury .

deliberation

wic pos: N sentence1: It was the deliberation of his act that was insulting . sentence2: The deliberations of the jury . word: deliberation

0

False

D.13 WSC and DPR

D.13 WSC 与 DPR

it

stable

20

1

The stable was very roomy, with four good stalls; a large swinging window opened into the yard , which made it pleasant and airy.

wsc: The stable was very roomy, with four good stalls; a large swinging window opened into the yard , which made it pleasant and airy.

1

stable

D.14 CNN/Daily Mail

marouane fellaini and adnan januzaj continue to show the world they are not just teammates but also best mates. the manchester united and belgium duo both posted pictures of themselves out at a restaurant on monday night ahead of their game against newcastle on wednesday . januzaj poses in the middle of fellaini and a friend looking like somebody who failed to receive the memo about it being a jackson 5 themed night. premier league duo adnan januzaj and marouane fellaini pose with a friend on the dance floor . manchester united and belgium duo fellaini and januzaj are good friends both on and off the pitch . manchester united ace fellaini runs over to the bench to celebrate his goal against qpr with friend januzaj . the disco effect in the background adds to the theory, but januzaj doesn’t seem to mind as they later pose on the dance floor with other friends. united haven’t had too many reasons to have a song and dance this season so it seems they may be hitting the discotheques as another form of release. however, victory against newcastle on wednesday would leave manager louis van gaal at least tapping his toes as they continue to fight for a champions league spot this season. januzaj and robin van persie join fellaini in celebrating in front of the manchester united fans at west brom . januzaj receives some words of wisdom from manchester united’s dutch manager louis van gaal . januzaj and fellaini are joined by some friends as they take to the dance floor ahead of the newcastle game .

summarize: marouane fellaini and adnan januzaj continue to show the world they are not just teammates but also best mates. the manchester united and belgium duo both posted pictures of themselves out at a restaurant on monday night ahead of their game against newcastle on wednesday . januzaj poses in the middle of fellaini and a friend looking like somebody who failed to receive the memo about it being a jackson 5 themed night. premier league duo adnan januzaj and marouane fellaini pose with a friend on the dance floor . manchester united and belgium duo fellaini and januzaj are good friends both on and off the pitch . manchester united ace fellaini runs over to the bench to celebrate his goal against qpr with friend januzaj . the disco effect in the background adds to the theory, but januzaj doesn’t seem to mind as they later pose on the dance floor with other friends. united haven’t had too many reasons to have a song and dance this season so it seems they may be hitting the discotheques as another form of release. however, victory against newcastle on wednesday would leave manager louis van gaal at least tapping his toes as they continue to fight for a champions league spot this season. januzaj and robin van persie join fellaini in celebrating in front of the manchester united fans at west brom . januzaj receives some words of wisdom from manchester united’s dutch manager louis van gaal . januzaj and fellaini are joined by some friends as they take to the dance floor ahead of the newcastle game .

the belgian duo took to the dance floor on monday night with some friends . manchester united face newcastle in the premier league on wednesday . red devils will be looking for just their second league away win in seven . louis van gaal’s side currently sit two points clear of liverpool in fourth .

the belgian duo took to the dance floor on monday night with some friends . manchester united face newcastle in the premier league on wednesday . red devils will be looking for just their second league away win in seven . louis van gaal’s side currently sit two points clear of liverpool in fourth .

D.15 SQuAD

What does increased oxygen concentrations in the patient’s lungs displace?

Hyperbaric (high-pressure) medicine uses special oxygen chambers to increase the partial pressure of O 2 around the patient and, when needed, the medical staff. Carbon monoxide poisoning, gas gangrene, and decompression sickness (the ’bends’) are sometimes treated using these devices. Increased O 2 concentration in the lungs helps to displace carbon monoxide from the heme group of hemoglobin. Oxygen gas is poisonous to the anaerobic bacteria that cause gas gangrene, so increasing its partial pressure helps kill them. Decompression sickness occurs in divers who decompress too quickly after a dive, resulting in bubbles of inert gas, mostly nitrogen and helium, forming in their blood. Increasing the pressure of O 2 as soon as possible is part of the treatment.

question: What does increased oxygen concentrations in the patient’s lungs displace? context: Hyperbaric (high-pressure) medicine uses special oxygen chambers to increase the partial pressure of O 2 around the patient and, when needed, the medical staff. Carbon monoxide poisoning, gas gangrene, and decompression sickness (the ’bends’) are sometimes treated using these devices. Increased O 2 concentration in the lungs helps to displace carbon monoxide from the heme group of hemoglobin. Oxygen gas is poisonous to the anaerobic bacteria that cause gas gangrene, so increasing its partial pressure helps kill them. Decompression sickness occurs in divers who decompress too quickly after a dive, resulting in bubbles of inert gas, mostly nitrogen and helium, forming in their blood. Increasing the pressure of O 2 as soon as possible is part of the treatment.

carbon monoxide

carbon monoxide

D.16 WMT English to German

D.16 WMT 英译德

"Luigi often said to me that he never wanted the brothers to end up in court," she wrote.

translate English to German: "Luigi often said to me that he never wanted the brothers to end up in court," she wrote.

"Luigi sagte oft zu mir, dass er nie wollte, dass die Brüder vor Gericht landen", schrieb sie.

"Luigi sagte oft zu mir, dass er nie wollte, dass die Brüder vor Gericht landen", schrieb sie.

D.17 WMT English to French

D.17 WMT 英译法

This image section from an infrared recording by the Spitzer telescope shows a "family portrait" of countless generations of stars: the oldest stars are seen as blue dots, while more difficult to identify are the pink-coloured "new-borns" in the star delivery room.

translate English to French: This image section from an infrared recording by the Spitzer telescope shows a "family portrait" of countless generations of stars: the oldest stars are seen as blue dots, while more difficult to identify are the pink-coloured "new-borns" in the star delivery room.

Ce détail d’une photographie infrarouge prise par le télescope Spitzer montre un "portrait de famille" des innombrables générations d’étoiles: les plus vieilles étoiles sont en bleu et les points roses, plus difficiles à identifier, sont les "nouveau-nés" dans la salle d’accouchement de l’univers.

Ce détail d’une photographie infrarouge prise par le télescope Spitzer montre un "portrait de famille" des innombrables générations d’étoiles: les plus vieilles étoiles sont en bleu et les points roses, plus difficiles à identifier, sont les "nouveau-nés" dans la salle d’accouchement de l’univers.

D.18 WMT English to Romanian

D.18 WMT 英译罗

Taco Bell said it plans to add 2,000 locations in the US by 2022.

translate English to Romanian: Taco Bell said it plans to add 2,000 locations in the US by 2022.

Taco Bell a afirmat că, p^ană ^in 2022, inten t , ionează să deschidă 2000 de restaurante ^in SUA.

Taco Bell a afirmat că, p^ană ^in 2022, inten t , ionează să deschidă 2000 de restaurante ^in SUA.

Appendix E Scores on Every Task for All Experiments

附录 E 全部实验在每项任务上的分数

The following table lists the scores achieved on every task in the experiments described in Sections 3.2, 3.3, 3.4, 3.5 and 3.6.

下表列出第 3 节所述实验在每项任务上达到的分数。我们纳入该详细分解，以便对特定任务结果感兴趣的读者查阅。

\bigstar\,

\bigstar\,

References

Bojar et al. (2016) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation , 2016. Bowman et al. (2015) Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349 , 2015. Buck et al. (2014) Christian Buck, Kenneth Heafield, and Bas Van Ooyen. N-gram counts and language models from the common crawl. In LREC , 2014. Caruana (1997) Rich Caruana. Multitask learning. Machine learning , 28(1), 1997. Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055 , 2017. Cheng et al. (2016) Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733 , 2016. Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 , 2019. Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 , 2020. Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. SentEval: An evaluation toolkit for universal sentence representations. arXiv preprint arXiv:1803.05449 , 2018. Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364 , 2017.

Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop , 2005. Dai and Le (2015) Andrew M. Dai and Quoc V. Le. Semi-supervised sequence learning. In Advances in neural information processing systems , 2015. De Marneff et al. (2019) Marie-Catherine De Marneff, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. In Sinn und Bedeutung 23 , 2019. Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , 2009. Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. Dolan and Brockett (2005) William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005) , 2005. Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197 , 2019. Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381 , 2018. Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. arXiv preprint arXiv:1802.06893 , 2018. Graves (2013) Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013.

Kudo and Richardson (2018) Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226 , 2018. Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291 , 2019. Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning , 2012. Li (2012) Qi Li. Literature survey: domain adaptation algorithms for natural language processing. 2012. Lin (2004) Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out , 2004. Liu et al. (2018) Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating Wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198 , 2018. Liu et al. (2019a) Peter J. Liu, Yu-An Chung, and Jie Ren. SummAE: Zero-shot abstractive text summarization using length-agnostic auto-encoders. arXiv preprint arXiv:1910.00998 , 2019a. Liu et al. (2015) Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2015. Liu et al. (2019b) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504 , 2019b.

Paulus et al. (2017) Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 , 2017. Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014. Peters et al. (2019) Matthew Peters, Sebastian Ruder, and Noah A. Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv:1903.05987 , 2019. Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365 , 2018. Phang et al. (2018) Jason Phang, Thibault Févry, and Samuel R. Bowman. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088 , 2018. Pilehvar and Camacho-Collados (2018) Mohammad Taher Pilehvar and Jose Camacho-Collados. WIC: 10,000 example pairs for evaluating context-sensitive representations. arXiv preprint arXiv:1808.09121 , 2018. Post (2018) Matt Post. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771 , 2018. Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. Rahman and Ng (2012) Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: the Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning . Association for Computational Linguistics, 2012.

Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885 , 2018. Zhu et al. (2019) Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Thomas Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for language understanding. arXiv preprint arXiv:1909.11764 , 2019. Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision , 2015.

		GLUE						SuperGLUE	WMT
		Score	CoLA	SST-2	MRPC	MRPC	STSB	STSB	QQP	QQP	MNLIm	MNLImm	QNLI	RTE	CNN/DM	SQuAD	Score	BoolQ	CB	CB	COPA	MultiRC	MultiRC	ReCoRD	ReCoRD	RTE	WiC	WSC	EnDe	EnFr	EnRo
Table	Experiment	Average	MCC	Acc	F1	Acc	PCC	SCC	F1	Acc	Acc	Acc	Acc	Acc	R-1-F	R-2-F	R-L-F	EM	F1	Average	Acc	F1	Acc	Acc	F1	EM	F1	EM	Acc	Acc	Acc	BLEU	BLEU	BLEU
1	$\bigstar\,$ Baseline average	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
1	Baseline standard deviation	$0.235$	$1.111$	$0.569$	$0.729$	$1.019$	$0.374$	$0.418$	$0.108$	$0.070$	$0.291$	$0.231$	$0.361$	$1.393$	$0.065$	$0.065$	$0.058$	$0.343$	$0.226$	$0.416$	$0.365$	$3.237$	$2.560$	$2.741$	$0.716$	$1.011$	$0.370$	$0.379$	$1.228$	$0.850$	$2.029$	$0.112$	$0.090$	$0.108$
1	No pre-training	$66.22$	$12.29$	$80.62$	$81.42$	$73.04$	$72.58$	$72.97$	$81.94$	$86.62$	$68.02$	$67.98$	$75.69$	$58.84$	$39.19$	$17.60$	$36.69$	$50.31$	$61.97$	$53.04$	$65.38$	$71.61$	$76.79$	$62.00$	$59.10$	$0.84$	$20.33$	$17.95$	$54.15$	$54.08$	$65.38$	$25.86$	$39.77$	$24.04$
2	$\bigstar\,$ Enc/dec, denoising	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
2	Enc/dec, shared, denoising	$82.81$	$55.24$	$91.86$	$91.58$	$88.24$	$87.43$	$87.58$	$88.69$	$91.60$	$83.88$	$84.01$	$90.23$	$73.65$	$41.11$	$18.78$	$38.48$	$80.63$	$88.49$	$70.73$	$77.13$	$95.04$	$96.43$	$65.00$	$66.16$	$22.98$	$68.95$	$68.09$	$70.76$	$68.18$	$75.96$	$26.72$	$39.03$	$27.46$
2	Enc/dec, 6 layers, denoising	$80.88$	$46.26$	$92.09$	$91.51$	$87.99$	$87.01$	$86.76$	$87.93$	$90.97$	$82.20$	$82.41$	$88.83$	$71.48$	$40.83$	$18.97$	$38.31$	$77.59$	$86.07$	$68.42$	$73.79$	$91.70$	$92.86$	$67.00$	$61.02$	$19.62$	$61.26$	$60.33$	$72.20$	$65.99$	$75.00$	$26.38$	$38.40$	$26.95$
2	Language model, denoising	$74.70$	$24.50$	$90.60$	$86.08$	$78.92$	$85.22$	$85.42$	$85.40$	$88.99$	$76.72$	$77.05$	$86.02$	$64.62$	$39.49$	$17.93$	$36.91$	$61.14$	$71.37$	$55.02$	$65.47$	$60.08$	$71.43$	$58.00$	$43.03$	$2.94$	$53.35$	$52.31$	$53.07$	$58.62$	$63.46$	$25.09$	$35.28$	$25.86$
2	Prefix LM, denoising	$81.82$	$49.99$	$92.43$	$91.43$	$88.24$	$87.20$	$86.98$	$88.41$	$91.39$	$82.32$	$82.93$	$88.71$	$74.01$	$40.46$	$18.61$	$37.90$	$78.94$	$87.31$	$68.11$	$75.50$	$93.37$	$91.07$	$60.00$	$63.43$	$21.20$	$65.03$	$64.11$	$71.48$	$65.67$	$73.08$	$26.43$	$37.98$	$27.39$
2	Enc/dec, LM	$79.56$	$42.03$	$91.86$	$91.64$	$88.24$	$87.13$	$87.00$	$88.21$	$91.15$	$81.68$	$81.66$	$88.54$	$65.70$	$40.67$	$18.59$	$38.13$	$76.02$	$84.85$	$64.29$	$72.23$	$85.74$	$89.29$	$57.00$	$60.53$	$16.26$	$59.28$	$58.30$	$65.34$	$64.89$	$70.19$	$26.27$	$39.17$	$26.86$
2	Enc/dec, shared, LM	$79.60$	$44.83$	$92.09$	$90.20$	$85.78$	$86.03$	$85.87$	$87.77$	$91.02$	$81.74$	$82.29$	$89.16$	$65.34$	$40.16$	$18.13$	$37.59$	$76.35$	$84.86$	$63.50$	$70.49$	$91.41$	$87.50$	$55.00$	$60.21$	$16.89$	$57.83$	$56.73$	$63.54$	$63.48$	$70.19$	$26.62$	$39.17$	$27.05$
2	Enc/dec, 6 layers, LM	$78.67$	$38.72$	$91.40$	$90.40$	$86.52$	$86.82$	$86.49$	$87.87$	$91.03$	$80.99$	$80.92$	$88.05$	$65.70$	$40.29$	$18.26$	$37.70$	$75.32$	$84.06$	$64.06$	$71.38$	$85.25$	$89.29$	$60.00$	$57.56$	$16.79$	$55.22$	$54.30$	$66.79$	$63.95$	$71.15$	$26.13$	$38.42$	$26.89$
2	Language model, LM	$73.78$	$28.53$	$89.79$	$85.23$	$78.68$	$84.22$	$84.00$	$84.88$	$88.70$	$74.94$	$75.77$	$84.84$	$58.84$	$38.97$	$17.54$	$36.37$	$53.81$	$64.55$	$56.51$	$64.22$	$59.92$	$71.43$	$64.00$	$53.04$	$1.05$	$46.81$	$45.78$	$58.84$	$56.74$	$69.23$	$25.23$	$34.31$	$25.38$
2	Prefix LM, LM	$79.68$	$41.26$	$92.09$	$90.11$	$86.27$	$86.82$	$86.32$	$88.35$	$91.35$	$81.71$	$82.02$	$89.04$	$68.59$	$39.66$	$17.84$	$37.13$	$76.87$	$85.39$	$64.86$	$71.47$	$93.37$	$91.07$	$57.00$	$58.67$	$16.89$	$59.25$	$58.16$	$64.26$	$66.30$	$71.15$	$26.28$	$37.51$	$26.76$
4	Language modeling with prefix	$80.69$	$44.22$	$93.00$	$91.68$	$88.48$	$87.20$	$87.18$	$88.39$	$91.41$	$82.66$	$83.09$	$89.29$	$68.95$	$40.71$	$18.94$	$38.15$	$77.99$	$86.43$	$65.27$	$73.55$	$83.95$	$87.50$	$55.00$	$59.65$	$18.89$	$61.76$	$60.76$	$68.59$	$65.67$	$73.08$	$26.86$	$39.73$	$27.49$
4	BERT-style (Devlin et al. 2018)	$82.96$	$52.49$	$92.55$	$92.79$	$89.95$	$87.68$	$87.66$	$88.47$	$91.44$	$83.60$	$84.05$	$90.33$	$75.45$	$41.27$	$19.17$	$38.72$	$80.65$	$88.24$	$69.85$	$76.48$	$94.37$	$94.64$	$61.00$	$63.29$	$25.08$	$66.76$	$65.85$	$72.20$	$69.12$	$75.00$	$26.78$	$40.03$	$27.41$
4	Deshuffling	$73.17$	$22.82$	$87.16$	$86.88$	$81.13$	$84.03$	$83.82$	$86.38$	$89.90$	$76.30$	$76.34$	$84.18$	$58.84$	$40.75$	$18.59$	$38.10$	$67.61$	$76.76$	$58.47$	$69.17$	$63.70$	$78.57$	$56.00$	$59.85$	$12.70$	$45.52$	$44.36$	$57.04$	$64.89$	$68.27$	$26.11$	$39.30$	$25.62$
5	BERT-style (Devlin et al. 2018)	$82.96$	$52.49$	$92.55$	$92.79$	$89.95$	$87.68$	$87.66$	$88.47$	$91.44$	$83.60$	$84.05$	$90.33$	$75.45$	$41.27$	$19.17$	$38.72$	$80.65$	$88.24$	$69.85$	$76.48$	$94.37$	$94.64$	$61.00$	$63.29$	$25.08$	$66.76$	$65.85$	$72.20$	$69.12$	$75.00$	$26.78$	$40.03$	$27.41$
5	MASS-style (Song et al. 2019)	$82.32$	$47.01$	$91.63$	$92.53$	$89.71$	$88.21$	$88.18$	$88.58$	$91.44$	$82.96$	$83.67$	$90.02$	$77.26$	$41.16$	$19.16$	$38.55$	$80.10$	$88.07$	$69.28$	$75.08$	$84.98$	$89.29$	$63.00$	$64.46$	$23.50$	$66.71$	$65.91$	$72.20$	$67.71$	$78.85$	$26.79$	$39.89$	$27.55$
5	$\bigstar\,$ Replace corrupted spans	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
5	Drop corrupted tokens	$84.44$	$60.04$	$92.89$	$92.79$	$89.95$	$87.28$	$86.85$	$88.56$	$91.54$	$83.94$	$83.92$	$90.74$	$79.42$	$41.27$	$19.31$	$38.70$	$80.52$	$88.28$	$68.67$	$75.90$	$96.02$	$94.64$	$56.00$	$65.06$	$23.92$	$65.54$	$64.60$	$71.12$	$67.40$	$74.04$	$27.07$	$39.76$	$27.82$
6	Corruption rate = $10\%$	$82.82$	$52.71$	$92.09$	$91.55$	$88.24$	$88.19$	$88.15$	$88.47$	$91.40$	$83.50$	$84.51$	$90.33$	$75.45$	$41.05$	$19.00$	$38.53$	$80.38$	$88.36$	$69.55$	$74.98$	$92.37$	$92.86$	$62.00$	$66.04$	$24.66$	$67.93$	$67.09$	$70.76$	$67.24$	$75.96$	$26.87$	$39.28$	$27.44$
6	$\bigstar\,$ Corruption rate = $15\%$	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
6	Corruption rate = $25\%$	$83.00$	$53.47$	$93.00$	$92.44$	$89.46$	$87.36$	$87.36$	$88.68$	$91.53$	$84.44$	$84.15$	$90.77$	$74.01$	$41.69$	$19.54$	$39.14$	$80.96$	$88.61$	$70.48$	$76.39$	$93.02$	$92.86$	$68.00$	$65.46$	$24.66$	$68.20$	$67.39$	$73.65$	$67.87$	$72.12$	$27.04$	$39.83$	$27.47$
6	Corruption rate = $50\%$	$81.27$	$46.26$	$91.63$	$91.11$	$87.99$	$87.87$	$87.64$	$88.70$	$91.57$	$83.64$	$84.10$	$90.24$	$70.76$	$41.51$	$19.32$	$38.89$	$79.80$	$87.76$	$70.33$	$75.02$	$93.05$	$92.86$	$68.00$	$62.97$	$24.13$	$64.94$	$64.13$	$72.20$	$68.50$	$77.88$	$27.01$	$39.90$	$27.49$
7	$\bigstar\,$ Baseline (i.i.d.)	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
7	Average span length = $2$	$83.54$	$53.82$	$92.20$	$93.05$	$90.44$	$87.85$	$87.71$	$88.42$	$91.40$	$84.28$	$84.46$	$90.88$	$77.62$	$41.23$	$19.39$	$38.69$	$82.09$	$89.69$	$72.20$	$77.06$	$90.43$	$91.07$	$70.00$	$66.28$	$26.13$	$71.34$	$70.61$	$75.45$	$68.34$	$78.85$	$26.76$	$39.99$	$27.63$
7	Average span length = $3$	$83.49$	$53.90$	$92.43$	$92.25$	$89.46$	$87.49$	$87.53$	$88.72$	$91.51$	$84.85$	$84.84$	$90.99$	$77.26$	$41.50$	$19.62$	$38.94$	$81.84$	$89.66$	$72.53$	$76.85$	$94.37$	$94.64$	$70.00$	$67.64$	$28.75$	$70.84$	$69.90$	$74.73$	$67.71$	$77.88$	$26.86$	$39.65$	$27.62$
7	Average span length = $5$	$83.40$	$52.12$	$93.12$	$92.63$	$89.71$	$88.70$	$88.47$	$88.84$	$91.64$	$84.32$	$84.29$	$90.79$	$76.90$	$41.39$	$19.24$	$38.82$	$82.05$	$89.79$	$72.23$	$77.06$	$83.06$	$89.29$	$69.00$	$68.16$	$30.12$	$71.36$	$70.53$	$75.81$	$69.91$	$79.81$	$26.88$	$39.40$	$27.53$
7	Average span length = $10$	$82.85$	$50.11$	$92.09$	$91.95$	$88.97$	$88.45$	$88.22$	$88.86$	$91.63$	$84.34$	$84.28$	$91.07$	$76.17$	$41.38$	$19.33$	$38.80$	$81.84$	$89.39$	$70.44$	$76.45$	$87.40$	$89.29$	$65.00$	$66.87$	$29.59$	$69.82$	$68.94$	$72.56$	$67.55$	$75.96$	$26.79$	$39.49$	$27.69$
8	$\bigstar\,$ C4	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
8	C4, unfiltered	$81.46$	$48.01$	$91.63$	$92.72$	$89.95$	$87.79$	$87.60$	$88.31$	$91.27$	$82.30$	$82.34$	$88.71$	$72.20$	$41.09$	$19.14$	$38.54$	$78.78$	$87.04$	$68.04$	$75.75$	$89.17$	$91.07$	$62.00$	$65.52$	$25.60$	$62.42$	$61.58$	$69.68$	$67.08$	$72.12$	$26.55$	$39.34$	$27.21$
8	RealNews-like	$83.83$	$56.55$	$92.66$	$92.06$	$88.97$	$87.71$	$87.37$	$88.51$	$91.49$	$84.35$	$84.46$	$90.61$	$78.34$	$41.38$	$19.23$	$38.84$	$80.39$	$88.50$	$72.38$	$77.00$	$93.09$	$94.64$	$66.00$	$65.92$	$23.82$	$74.56$	$73.72$	$75.81$	$66.61$	$80.77$	$26.75$	$39.90$	$27.48$
8	WebText-like	$84.03$	$56.38$	$93.12$	$92.31$	$89.22$	$88.69$	$88.68$	$88.65$	$91.56$	$84.70$	$84.84$	$90.83$	$77.62$	$41.23$	$19.31$	$38.70$	$81.42$	$89.15$	$71.40$	$76.88$	$83.08$	$89.29$	$66.00$	$64.10$	$24.24$	$72.24$	$71.36$	$75.45$	$68.03$	$82.69$	$26.80$	$39.74$	$27.59$
8	Wikipedia	$81.85$	$45.53$	$92.32$	$91.67$	$88.24$	$85.62$	$86.40$	$88.37$	$91.34$	$82.61$	$83.25$	$90.96$	$77.26$	$41.39$	$19.31$	$38.81$	$81.29$	$89.18$	$68.01$	$76.12$	$56.03$	$80.36$	$67.00$	$65.01$	$25.92$	$69.03$	$68.06$	$74.73$	$67.08$	$76.92$	$26.94$	$39.69$	$27.67$
8	Wikipedia + TBC	$83.65$	$55.53$	$92.78$	$92.41$	$89.22$	$86.67$	$86.27$	$89.47$	$92.29$	$84.38$	$83.45$	$91.94$	$76.90$	$41.22$	$19.28$	$38.67$	$82.08$	$89.70$	$73.24$	$76.22$	$95.40$	$92.86$	$69.00$	$51.59$	$50.93$	$69.53$	$68.51$	$77.62$	$66.93$	$81.73$	$26.77$	$39.63$	$27.57$
9	$\bigstar\,$ Full data set	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
9	$2^{29}$ ( $64$ repeats)	$82.87$	$53.82$	$92.78$	$91.79$	$88.73$	$87.56$	$87.58$	$88.73$	$91.54$	$84.07$	$84.21$	$90.59$	$73.65$	$41.18$	$19.19$	$38.67$	$80.97$	$88.90$	$72.03$	$76.76$	$92.96$	$92.86$	$66.00$	$65.11$	$26.76$	$69.35$	$68.49$	$75.81$	$67.24$	$82.69$	$26.83$	$39.74$	$27.63$
9	$2^{27}$ ( $256$ repeats)	$82.62$	$50.60$	$92.32$	$92.07$	$88.73$	$87.83$	$87.60$	$88.65$	$91.54$	$83.43$	$84.37$	$90.12$	$75.81$	$41.24$	$19.20$	$38.70$	$79.78$	$87.63$	$69.97$	$75.29$	$93.42$	$91.07$	$63.00$	$61.82$	$23.61$	$66.27$	$65.39$	$73.65$	$66.30$	$80.77$	$27.02$	$39.71$	$27.33$
9	$2^{25}$ ( $1{,}024$ repeats)	$79.55$	$43.84$	$91.28$	$89.32$	$85.05$	$85.92$	$85.74$	$88.05$	$91.09$	$81.29$	$81.72$	$87.90$	$69.31$	$40.66$	$18.57$	$38.13$	$76.27$	$84.58$	$64.76$	$72.63$	$83.97$	$82.14$	$64.00$	$59.39$	$17.94$	$56.94$	$56.04$	$64.98$	$65.20$	$73.08$	$26.38$	$39.56$	$26.80$
9	$2^{23}$ ( $4{,}096$ repeats)	$76.34$	$32.68$	$89.45$	$89.84$	$86.03$	$83.49$	$83.42$	$87.18$	$90.61$	$77.80$	$78.69$	$85.47$	$64.62$	$40.16$	$18.33$	$37.66$	$70.92$	$80.20$	$59.29$	$69.85$	$73.48$	$73.21$	$56.00$	$57.66$	$14.38$	$46.69$	$45.79$	$59.57$	$65.05$	$68.27$	$26.37$	$38.84$	$25.81$
10	$\bigstar\,$ All parameters	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
10	Adapter layers, $d=32$	$80.52$	$45.33$	$91.63$	$90.59$	$86.76$	$88.38$	$88.06$	$86.99$	$90.26$	$83.63$	$83.94$	$90.72$	$67.15$	$34.50$	$15.08$	$32.15$	$79.32$	$87.70$	$60.40$	$65.32$	$50.87$	$73.21$	$52.00$	$58.61$	$19.41$	$65.50$	$64.58$	$62.09$	$64.58$	$73.08$	$13.84$	$17.88$	$15.54$
10	Adapter layers, $d=128$	$81.51$	$45.35$	$92.89$	$91.49$	$88.24$	$87.73$	$87.65$	$87.73$	$90.93$	$83.64$	$84.09$	$90.52$	$72.56$	$36.71$	$16.62$	$34.37$	$79.47$	$87.61$	$63.03$	$69.20$	$52.21$	$75.00$	$56.00$	$61.08$	$18.05$	$67.94$	$66.97$	$68.59$	$66.77$	$73.08$	$19.83$	$27.50$	$22.63$
10	Adapter layers, $d=512$	$81.54$	$44.25$	$93.35$	$91.00$	$87.25$	$88.74$	$88.44$	$88.02$	$91.15$	$83.08$	$83.80$	$89.62$	$74.37$	$38.63$	$17.78$	$36.25$	$79.18$	$87.32$	$64.30$	$73.18$	$59.86$	$71.43$	$56.00$	$62.94$	$18.57$	$66.56$	$65.74$	$70.76$	$67.87$	$74.04$	$23.45$	$33.98$	$25.81$
10	Adapter layers, $d=2048$	$82.62$	$49.86$	$92.55$	$91.30$	$87.99$	$88.46$	$88.35$	$88.36$	$91.40$	$83.63$	$83.18$	$90.66$	$76.53$	$39.44$	$18.30$	$37.06$	$79.40$	$87.36$	$68.61$	$74.53$	$88.00$	$91.07$	$58.00$	$61.10$	$18.89$	$66.73$	$66.06$	$73.29$	$71.16$	$75.96$	$25.64$	$36.92$	$26.93$
10	Gradual Unfreezing	$82.50$	$51.74$	$91.97$	$92.61$	$89.71$	$87.27$	$86.90$	$88.26$	$91.35$	$83.42$	$83.49$	$89.71$	$75.09$	$40.88$	$18.95$	$38.40$	$79.17$	$87.30$	$70.79$	$75.51$	$93.09$	$94.64$	$70.00$	$62.03$	$21.51$	$65.69$	$64.79$	$72.92$	$69.12$	$77.89$	$26.71$	$39.02$	$26.93$
11	$\bigstar\,$ Baseline (pre-train/fine-tune)	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
11	Equal	$76.13$	$39.47$	$90.94$	$82.90$	$75.74$	$78.83$	$78.44$	$86.45$	$89.71$	$82.08$	$82.92$	$90.13$	$59.93$	$40.95$	$19.02$	$38.39$	$76.51$	$85.61$	$63.37$	$73.06$	$82.37$	$83.93$	$65.00$	$60.89$	$17.52$	$60.51$	$59.70$	$61.01$	$60.03$	$65.38$	$23.89$	$34.31$	$26.78$
11	Examples-proportional, $K=2^{16}$	$80.45$	$42.07$	$91.97$	$90.97$	$87.50$	$85.41$	$85.04$	$86.89$	$90.10$	$83.01$	$83.66$	$90.74$	$72.56$	$41.16$	$19.04$	$38.59$	$77.25$	$85.72$	$69.95$	$76.67$	$86.38$	$89.29$	$70.00$	$65.93$	$27.91$	$62.78$	$61.95$	$76.90$	$65.83$	$73.08$	$24.35$	$34.99$	$27.10$
11	Examples-proportional, $K=2^{17}$	$81.56$	$47.35$	$91.40$	$91.55$	$88.24$	$86.15$	$85.93$	$86.94$	$90.06$	$82.76$	$84.12$	$90.79$	$75.09$	$41.06$	$19.12$	$38.47$	$77.00$	$85.87$	$67.91$	$77.89$	$77.54$	$85.71$	$57.00$	$67.78$	$27.07$	$61.51$	$60.54$	$79.06$	$65.20$	$74.04$	$24.36$	$35.00$	$27.25$
11	Examples-proportional, $K=2^{18}$	$81.67$	$46.85$	$91.63$	$91.99$	$88.73$	$87.68$	$87.20$	$86.93$	$90.35$	$83.30$	$84.01$	$91.47$	$73.29$	$40.96$	$19.07$	$38.43$	$78.17$	$86.74$	$67.94$	$76.57$	$78.88$	$87.50$	$62.00$	$67.70$	$30.85$	$63.43$	$62.54$	$76.53$	$65.67$	$67.31$	$24.57$	$35.19$	$27.39$
11	Examples-proportional, $K=2^{19}$	$81.42$	$45.94$	$91.63$	$92.20$	$89.22$	$88.44$	$88.32$	$86.84$	$90.10$	$83.73$	$84.29$	$91.84$	$70.40$	$41.26$	$19.24$	$38.71$	$79.78$	$88.15$	$67.30$	$75.66$	$75.59$	$87.50$	$59.00$	$68.22$	$30.64$	$65.32$	$64.29$	$73.65$	$65.05$	$69.23$	$25.21$	$36.30$	$27.76$
11	Examples-proportional, $K=2^{20}$	$80.80$	$42.55$	$92.78$	$91.27$	$87.99$	$88.36$	$88.10$	$86.10$	$89.62$	$84.15$	$84.26$	$92.20$	$68.95$	$41.05$	$19.24$	$38.46$	$80.36$	$88.27$	$67.38$	$73.21$	$76.18$	$83.93$	$62.00$	$67.57$	$26.86$	$66.12$	$65.22$	$76.90$	$64.73$	$69.23$	$25.66$	$36.93$	$27.68$
11	Examples-proportional, $K=2^{21}$	$79.83$	$44.45$	$91.28$	$89.00$	$84.31$	$87.54$	$87.40$	$84.93$	$88.53$	$82.54$	$84.16$	$90.85$	$67.87$	$40.51$	$18.79$	$37.92$	$79.50$	$87.48$	$65.10$	$71.16$	$68.88$	$85.71$	$57.00$	$62.75$	$23.40$	$64.50$	$63.65$	$72.92$	$64.11$	$71.15$	$25.82$	$37.22$	$27.13$
11	Temperature-scaled, $T=2$	$81.90$	$54.00$	$91.74$	$90.56$	$86.76$	$85.11$	$84.60$	$86.40$	$89.74$	$83.47$	$84.15$	$91.51$	$72.56$	$41.09$	$19.28$	$38.54$	$79.42$	$87.77$	$69.92$	$76.73$	$92.37$	$92.86$	$57.00$	$69.80$	$31.90$	$66.65$	$65.74$	$72.92$	$67.08$	$75.96$	$25.42$	$36.72$	$27.20$
11	Temperature-scaled, $T=4$	$80.56$	$45.38$	$91.97$	$89.68$	$85.78$	$83.13$	$82.76$	$86.39$	$90.00$	$82.78$	$84.19$	$91.16$	$73.65$	$41.09$	$19.22$	$38.51$	$77.99$	$86.81$	$69.54$	$76.76$	$97.36$	$96.43$	$59.00$	$68.10$	$31.48$	$64.26$	$63.27$	$74.73$	$64.26$	$71.15$	$25.04$	$35.82$	$27.45$
11	Temperature-scaled, $T=8$	$77.21$	$40.07$	$91.06$	$88.11$	$83.33$	$79.20$	$79.06$	$86.60$	$89.90$	$83.05$	$83.56$	$90.21$	$59.93$	$41.01$	$19.10$	$38.40$	$77.14$	$85.99$	$66.07$	$73.94$	$93.70$	$94.64$	$60.00$	$66.36$	$26.86$	$63.46$	$62.60$	$62.09$	$63.32$	$65.38$	$24.55$	$35.35$	$27.17$
12	$\bigstar\,$ Unsupervised pre-training + fine-tuning	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
12	Multi-task training	$81.42$	$45.94$	$91.63$	$92.20$	$89.22$	$88.44$	$88.32$	$86.84$	$90.10$	$83.73$	$84.29$	$91.84$	$70.40$	$41.26$	$19.24$	$38.71$	$79.78$	$88.15$	$67.30$	$75.66$	$75.59$	$87.50$	$59.00$	$68.22$	$30.64$	$65.32$	$64.29$	$73.65$	$65.05$	$69.23$	$25.21$	$36.30$	$27.76$
12	Multi-task pre-training + fine-tuning	$83.11$	$51.42$	$92.66$	$91.73$	$88.73$	$88.06$	$87.70$	$88.61$	$91.61$	$84.09$	$84.31$	$91.85$	$76.53$	$41.15$	$19.12$	$38.59$	$80.26$	$88.50$	$71.03$	$79.54$	$81.69$	$87.50$	$65.00$	$70.72$	$31.48$	$65.94$	$65.03$	$81.23$	$68.18$	$73.08$	$27.08$	$39.80$	$28.07$
12	Leave-one-out multi-task training	$81.98$	$48.00$	$93.23$	$91.72$	$88.24$	$87.76$	$87.32$	$88.61$	$91.44$	$84.00$	$84.11$	$90.79$	$72.20$	$41.34$	$19.05$	$38.77$	$79.97$	$88.10$	$71.68$	$78.35$	$86.76$	$89.29$	$66.00$	$68.09$	$29.49$	$66.23$	$65.27$	$79.06$	$68.65$	$78.85$	$26.93$	$39.79$	$27.87$
12	Supervised multi-task pre-training	$79.93$	$36.60$	$92.43$	$91.58$	$88.24$	$87.03$	$86.78$	$88.15$	$91.20$	$82.87$	$83.16$	$90.13$	$70.76$	$41.12$	$18.96$	$38.49$	$77.38$	$85.65$	$65.36$	$75.66$	$68.87$	$83.93$	$58.00$	$64.81$	$21.93$	$55.37$	$54.61$	$71.12$	$67.40$	$75.96$	$26.81$	$40.13$	$28.04$
13	$\bigstar\,$ Baseline	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
13	$1\times$ size, $4\times$ training steps	$85.33$	$60.29$	$93.81$	$94.06$	$91.67$	$89.42$	$89.25$	$89.15$	$91.87$	$86.01$	$85.70$	$91.63$	$78.34$	$41.52$	$19.33$	$38.96$	$82.45$	$90.19$	$74.72$	$79.17$	$94.75$	$92.86$	$71.00$	$67.34$	$29.70$	$72.63$	$71.59$	$78.34$	$72.10$	$82.69$	$27.08$	$40.66$	$27.93$
13	$1\times$ size, $4\times$ batch size	$84.60$	$56.08$	$93.12$	$92.31$	$89.22$	$88.85$	$88.84$	$89.35$	$92.07$	$85.98$	$86.13$	$91.07$	$80.14$	$41.70$	$19.42$	$39.08$	$82.52$	$90.21$	$74.64$	$78.78$	$93.69$	$94.64$	$72.00$	$68.09$	$30.95$	$74.73$	$73.90$	$76.53$	$70.06$	$81.73$	$27.07$	$40.60$	$27.84$
13	$2\times$ size, $2\times$ training steps	$86.18$	$62.04$	$93.69$	$93.36$	$90.69$	$89.18$	$89.23$	$89.35$	$92.05$	$87.23$	$87.05$	$92.68$	$81.95$	$41.74$	$19.66$	$39.14$	$84.18$	$91.29$	$77.18$	$80.98$	$97.36$	$96.43$	$74.00$	$71.34$	$35.68$	$77.11$	$76.34$	$80.51$	$69.28$	$85.58$	$27.52$	$41.03$	$28.19$
13	$4\times$ size, $1\times$ training steps	$85.91$	$57.58$	$94.38$	$92.67$	$89.95$	$89.60$	$89.60$	$89.44$	$92.14$	$87.05$	$87.12$	$93.12$	$83.39$	$41.60$	$19.73$	$39.08$	$83.86$	$91.32$	$78.04$	$81.38$	$89.09$	$94.64$	$73.00$	$73.74$	$40.40$	$78.25$	$77.40$	$81.59$	$70.22$	$91.35$	$27.47$	$40.71$	$28.10$
13	$4\times$ ensembled	$84.77$	$56.14$	$93.46$	$93.31$	$90.67$	$89.71$	$89.60$	$89.62$	$92.24$	$86.22$	$86.53$	$91.60$	$77.98$	$42.10$	$20.10$	$39.56$	$83.09$	$90.40$	$71.74$	$77.58$	$89.85$	$91.07$	$66.00$	$69.32$	$29.49$	$72.67$	$71.94$	$76.90$	$69.12$	$72.12$	$28.05$	$40.53$	$28.09$
13	$4\times$ ensembled, fine-tune only	$84.05$	$54.78$	$92.78$	$93.15$	$90.44$	$88.34$	$88.12$	$89.27$	$91.97$	$85.33$	$85.88$	$90.98$	$77.62$	$41.66$	$19.57$	$39.12$	$82.36$	$89.86$	$71.56$	$77.43$	$90.07$	$92.86$	$69.00$	$67.31$	$26.34$	$70.47$	$69.64$	$75.45$	$68.18$	$74.04$	$27.55$	$40.22$	$28.09$

		GLUE						SuperGLUE	WMT
		Score	CoLA	SST-2	MRPC	MRPC	STSB	STSB	QQP	QQP	MNLIm	MNLImm	QNLI	RTE	CNN/DM	SQuAD	Score	BoolQ	CB	CB	COPA	MultiRC	MultiRC	ReCoRD	ReCoRD	RTE	WiC	WSC	EnDe	EnFr	EnRo
Table	Experiment	Average	MCC	Acc	F1	Acc	PCC	SCC	F1	Acc	Acc	Acc	Acc	Acc	R-1-F	R-2-F	R-L-F	EM	F1	Average	Acc	F1	Acc	Acc	F1	EM	F1	EM	Acc	Acc	Acc	BLEU	BLEU	BLEU
1	$\bigstar\,$ Baseline average	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
1	Baseline standard deviation	$0.235$	$1.111$	$0.569$	$0.729$	$1.019$	$0.374$	$0.418$	$0.108$	$0.070$	$0.291$	$0.231$	$0.361$	$1.393$	$0.065$	$0.065$	$0.058$	$0.343$	$0.226$	$0.416$	$0.365$	$3.237$	$2.560$	$2.741$	$0.716$	$1.011$	$0.370$	$0.379$	$1.228$	$0.850$	$2.029$	$0.112$	$0.090$	$0.108$
1	No pre-training	$66.22$	$12.29$	$80.62$	$81.42$	$73.04$	$72.58$	$72.97$	$81.94$	$86.62$	$68.02$	$67.98$	$75.69$	$58.84$	$39.19$	$17.60$	$36.69$	$50.31$	$61.97$	$53.04$	$65.38$	$71.61$	$76.79$	$62.00$	$59.10$	$0.84$	$20.33$	$17.95$	$54.15$	$54.08$	$65.38$	$25.86$	$39.77$	$24.04$
2	$\bigstar\,$ Enc/dec, denoising	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
2	Enc/dec, shared, denoising	$82.81$	$55.24$	$91.86$	$91.58$	$88.24$	$87.43$	$87.58$	$88.69$	$91.60$	$83.88$	$84.01$	$90.23$	$73.65$	$41.11$	$18.78$	$38.48$	$80.63$	$88.49$	$70.73$	$77.13$	$95.04$	$96.43$	$65.00$	$66.16$	$22.98$	$68.95$	$68.09$	$70.76$	$68.18$	$75.96$	$26.72$	$39.03$	$27.46$
2	Enc/dec, 6 layers, denoising	$80.88$	$46.26$	$92.09$	$91.51$	$87.99$	$87.01$	$86.76$	$87.93$	$90.97$	$82.20$	$82.41$	$88.83$	$71.48$	$40.83$	$18.97$	$38.31$	$77.59$	$86.07$	$68.42$	$73.79$	$91.70$	$92.86$	$67.00$	$61.02$	$19.62$	$61.26$	$60.33$	$72.20$	$65.99$	$75.00$	$26.38$	$38.40$	$26.95$
2	Language model, denoising	$74.70$	$24.50$	$90.60$	$86.08$	$78.92$	$85.22$	$85.42$	$85.40$	$88.99$	$76.72$	$77.05$	$86.02$	$64.62$	$39.49$	$17.93$	$36.91$	$61.14$	$71.37$	$55.02$	$65.47$	$60.08$	$71.43$	$58.00$	$43.03$	$2.94$	$53.35$	$52.31$	$53.07$	$58.62$	$63.46$	$25.09$	$35.28$	$25.86$
2	Prefix LM, denoising	$81.82$	$49.99$	$92.43$	$91.43$	$88.24$	$87.20$	$86.98$	$88.41$	$91.39$	$82.32$	$82.93$	$88.71$	$74.01$	$40.46$	$18.61$	$37.90$	$78.94$	$87.31$	$68.11$	$75.50$	$93.37$	$91.07$	$60.00$	$63.43$	$21.20$	$65.03$	$64.11$	$71.48$	$65.67$	$73.08$	$26.43$	$37.98$	$27.39$
2	Enc/dec, LM	$79.56$	$42.03$	$91.86$	$91.64$	$88.24$	$87.13$	$87.00$	$88.21$	$91.15$	$81.68$	$81.66$	$88.54$	$65.70$	$40.67$	$18.59$	$38.13$	$76.02$	$84.85$	$64.29$	$72.23$	$85.74$	$89.29$	$57.00$	$60.53$	$16.26$	$59.28$	$58.30$	$65.34$	$64.89$	$70.19$	$26.27$	$39.17$	$26.86$
2	Enc/dec, shared, LM	$79.60$	$44.83$	$92.09$	$90.20$	$85.78$	$86.03$	$85.87$	$87.77$	$91.02$	$81.74$	$82.29$	$89.16$	$65.34$	$40.16$	$18.13$	$37.59$	$76.35$	$84.86$	$63.50$	$70.49$	$91.41$	$87.50$	$55.00$	$60.21$	$16.89$	$57.83$	$56.73$	$63.54$	$63.48$	$70.19$	$26.62$	$39.17$	$27.05$
2	Enc/dec, 6 layers, LM	$78.67$	$38.72$	$91.40$	$90.40$	$86.52$	$86.82$	$86.49$	$87.87$	$91.03$	$80.99$	$80.92$	$88.05$	$65.70$	$40.29$	$18.26$	$37.70$	$75.32$	$84.06$	$64.06$	$71.38$	$85.25$	$89.29$	$60.00$	$57.56$	$16.79$	$55.22$	$54.30$	$66.79$	$63.95$	$71.15$	$26.13$	$38.42$	$26.89$
2	Language model, LM	$73.78$	$28.53$	$89.79$	$85.23$	$78.68$	$84.22$	$84.00$	$84.88$	$88.70$	$74.94$	$75.77$	$84.84$	$58.84$	$38.97$	$17.54$	$36.37$	$53.81$	$64.55$	$56.51$	$64.22$	$59.92$	$71.43$	$64.00$	$53.04$	$1.05$	$46.81$	$45.78$	$58.84$	$56.74$	$69.23$	$25.23$	$34.31$	$25.38$
2	Prefix LM, LM	$79.68$	$41.26$	$92.09$	$90.11$	$86.27$	$86.82$	$86.32$	$88.35$	$91.35$	$81.71$	$82.02$	$89.04$	$68.59$	$39.66$	$17.84$	$37.13$	$76.87$	$85.39$	$64.86$	$71.47$	$93.37$	$91.07$	$57.00$	$58.67$	$16.89$	$59.25$	$58.16$	$64.26$	$66.30$	$71.15$	$26.28$	$37.51$	$26.76$
4	Language modeling with prefix	$80.69$	$44.22$	$93.00$	$91.68$	$88.48$	$87.20$	$87.18$	$88.39$	$91.41$	$82.66$	$83.09$	$89.29$	$68.95$	$40.71$	$18.94$	$38.15$	$77.99$	$86.43$	$65.27$	$73.55$	$83.95$	$87.50$	$55.00$	$59.65$	$18.89$	$61.76$	$60.76$	$68.59$	$65.67$	$73.08$	$26.86$	$39.73$	$27.49$
4	BERT-style (Devlin et al. 2018)	$82.96$	$52.49$	$92.55$	$92.79$	$89.95$	$87.68$	$87.66$	$88.47$	$91.44$	$83.60$	$84.05$	$90.33$	$75.45$	$41.27$	$19.17$	$38.72$	$80.65$	$88.24$	$69.85$	$76.48$	$94.37$	$94.64$	$61.00$	$63.29$	$25.08$	$66.76$	$65.85$	$72.20$	$69.12$	$75.00$	$26.78$	$40.03$	$27.41$
4	Deshuffling	$73.17$	$22.82$	$87.16$	$86.88$	$81.13$	$84.03$	$83.82$	$86.38$	$89.90$	$76.30$	$76.34$	$84.18$	$58.84$	$40.75$	$18.59$	$38.10$	$67.61$	$76.76$	$58.47$	$69.17$	$63.70$	$78.57$	$56.00$	$59.85$	$12.70$	$45.52$	$44.36$	$57.04$	$64.89$	$68.27$	$26.11$	$39.30$	$25.62$
5	BERT-style (Devlin et al. 2018)	$82.96$	$52.49$	$92.55$	$92.79$	$89.95$	$87.68$	$87.66$	$88.47$	$91.44$	$83.60$	$84.05$	$90.33$	$75.45$	$41.27$	$19.17$	$38.72$	$80.65$	$88.24$	$69.85$	$76.48$	$94.37$	$94.64$	$61.00$	$63.29$	$25.08$	$66.76$	$65.85$	$72.20$	$69.12$	$75.00$	$26.78$	$40.03$	$27.41$
5	MASS-style (Song et al. 2019)	$82.32$	$47.01$	$91.63$	$92.53$	$89.71$	$88.21$	$88.18$	$88.58$	$91.44$	$82.96$	$83.67$	$90.02$	$77.26$	$41.16$	$19.16$	$38.55$	$80.10$	$88.07$	$69.28$	$75.08$	$84.98$	$89.29$	$63.00$	$64.46$	$23.50$	$66.71$	$65.91$	$72.20$	$67.71$	$78.85$	$26.79$	$39.89$	$27.55$
5	$\bigstar\,$ Replace corrupted spans	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
5	Drop corrupted tokens	$84.44$	$60.04$	$92.89$	$92.79$	$89.95$	$87.28$	$86.85$	$88.56$	$91.54$	$83.94$	$83.92$	$90.74$	$79.42$	$41.27$	$19.31$	$38.70$	$80.52$	$88.28$	$68.67$	$75.90$	$96.02$	$94.64$	$56.00$	$65.06$	$23.92$	$65.54$	$64.60$	$71.12$	$67.40$	$74.04$	$27.07$	$39.76$	$27.82$
6	Corruption rate = $10\%$	$82.82$	$52.71$	$92.09$	$91.55$	$88.24$	$88.19$	$88.15$	$88.47$	$91.40$	$83.50$	$84.51$	$90.33$	$75.45$	$41.05$	$19.00$	$38.53$	$80.38$	$88.36$	$69.55$	$74.98$	$92.37$	$92.86$	$62.00$	$66.04$	$24.66$	$67.93$	$67.09$	$70.76$	$67.24$	$75.96$	$26.87$	$39.28$	$27.44$
6	$\bigstar\,$ Corruption rate = $15\%$	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
6	Corruption rate = $25\%$	$83.00$	$53.47$	$93.00$	$92.44$	$89.46$	$87.36$	$87.36$	$88.68$	$91.53$	$84.44$	$84.15$	$90.77$	$74.01$	$41.69$	$19.54$	$39.14$	$80.96$	$88.61$	$70.48$	$76.39$	$93.02$	$92.86$	$68.00$	$65.46$	$24.66$	$68.20$	$67.39$	$73.65$	$67.87$	$72.12$	$27.04$	$39.83$	$27.47$
6	Corruption rate = $50\%$	$81.27$	$46.26$	$91.63$	$91.11$	$87.99$	$87.87$	$87.64$	$88.70$	$91.57$	$83.64$	$84.10$	$90.24$	$70.76$	$41.51$	$19.32$	$38.89$	$79.80$	$87.76$	$70.33$	$75.02$	$93.05$	$92.86$	$68.00$	$62.97$	$24.13$	$64.94$	$64.13$	$72.20$	$68.50$	$77.88$	$27.01$	$39.90$	$27.49$
7	$\bigstar\,$ Baseline (i.i.d.)	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
7	Average span length = $2$	$83.54$	$53.82$	$92.20$	$93.05$	$90.44$	$87.85$	$87.71$	$88.42$	$91.40$	$84.28$	$84.46$	$90.88$	$77.62$	$41.23$	$19.39$	$38.69$	$82.09$	$89.69$	$72.20$	$77.06$	$90.43$	$91.07$	$70.00$	$66.28$	$26.13$	$71.34$	$70.61$	$75.45$	$68.34$	$78.85$	$26.76$	$39.99$	$27.63$
7	Average span length = $3$	$83.49$	$53.90$	$92.43$	$92.25$	$89.46$	$87.49$	$87.53$	$88.72$	$91.51$	$84.85$	$84.84$	$90.99$	$77.26$	$41.50$	$19.62$	$38.94$	$81.84$	$89.66$	$72.53$	$76.85$	$94.37$	$94.64$	$70.00$	$67.64$	$28.75$	$70.84$	$69.90$	$74.73$	$67.71$	$77.88$	$26.86$	$39.65$	$27.62$
7	Average span length = $5$	$83.40$	$52.12$	$93.12$	$92.63$	$89.71$	$88.70$	$88.47$	$88.84$	$91.64$	$84.32$	$84.29$	$90.79$	$76.90$	$41.39$	$19.24$	$38.82$	$82.05$	$89.79$	$72.23$	$77.06$	$83.06$	$89.29$	$69.00$	$68.16$	$30.12$	$71.36$	$70.53$	$75.81$	$69.91$	$79.81$	$26.88$	$39.40$	$27.53$
7	Average span length = $10$	$82.85$	$50.11$	$92.09$	$91.95$	$88.97$	$88.45$	$88.22$	$88.86$	$91.63$	$84.34$	$84.28$	$91.07$	$76.17$	$41.38$	$19.33$	$38.80$	$81.84$	$89.39$	$70.44$	$76.45$	$87.40$	$89.29$	$65.00$	$66.87$	$29.59$	$69.82$	$68.94$	$72.56$	$67.55$	$75.96$	$26.79$	$39.49$	$27.69$
8	$\bigstar\,$ C4	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
8	C4, unfiltered	$81.46$	$48.01$	$91.63$	$92.72$	$89.95$	$87.79$	$87.60$	$88.31$	$91.27$	$82.30$	$82.34$	$88.71$	$72.20$	$41.09$	$19.14$	$38.54$	$78.78$	$87.04$	$68.04$	$75.75$	$89.17$	$91.07$	$62.00$	$65.52$	$25.60$	$62.42$	$61.58$	$69.68$	$67.08$	$72.12$	$26.55$	$39.34$	$27.21$
8	RealNews-like	$83.83$	$56.55$	$92.66$	$92.06$	$88.97$	$87.71$	$87.37$	$88.51$	$91.49$	$84.35$	$84.46$	$90.61$	$78.34$	$41.38$	$19.23$	$38.84$	$80.39$	$88.50$	$72.38$	$77.00$	$93.09$	$94.64$	$66.00$	$65.92$	$23.82$	$74.56$	$73.72$	$75.81$	$66.61$	$80.77$	$26.75$	$39.90$	$27.48$
8	WebText-like	$84.03$	$56.38$	$93.12$	$92.31$	$89.22$	$88.69$	$88.68$	$88.65$	$91.56$	$84.70$	$84.84$	$90.83$	$77.62$	$41.23$	$19.31$	$38.70$	$81.42$	$89.15$	$71.40$	$76.88$	$83.08$	$89.29$	$66.00$	$64.10$	$24.24$	$72.24$	$71.36$	$75.45$	$68.03$	$82.69$	$26.80$	$39.74$	$27.59$
8	Wikipedia	$81.85$	$45.53$	$92.32$	$91.67$	$88.24$	$85.62$	$86.40$	$88.37$	$91.34$	$82.61$	$83.25$	$90.96$	$77.26$	$41.39$	$19.31$	$38.81$	$81.29$	$89.18$	$68.01$	$76.12$	$56.03$	$80.36$	$67.00$	$65.01$	$25.92$	$69.03$	$68.06$	$74.73$	$67.08$	$76.92$	$26.94$	$39.69$	$27.67$
8	Wikipedia + TBC	$83.65$	$55.53$	$92.78$	$92.41$	$89.22$	$86.67$	$86.27$	$89.47$	$92.29$	$84.38$	$83.45$	$91.94$	$76.90$	$41.22$	$19.28$	$38.67$	$82.08$	$89.70$	$73.24$	$76.22$	$95.40$	$92.86$	$69.00$	$51.59$	$50.93$	$69.53$	$68.51$	$77.62$	$66.93$	$81.73$	$26.77$	$39.63$	$27.57$
9	$\bigstar\,$ Full data set	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
9	$2^{29}$ ( $64$ repeats)	$82.87$	$53.82$	$92.78$	$91.79$	$88.73$	$87.56$	$87.58$	$88.73$	$91.54$	$84.07$	$84.21$	$90.59$	$73.65$	$41.18$	$19.19$	$38.67$	$80.97$	$88.90$	$72.03$	$76.76$	$92.96$	$92.86$	$66.00$	$65.11$	$26.76$	$69.35$	$68.49$	$75.81$	$67.24$	$82.69$	$26.83$	$39.74$	$27.63$
9	$2^{27}$ ( $256$ repeats)	$82.62$	$50.60$	$92.32$	$92.07$	$88.73$	$87.83$	$87.60$	$88.65$	$91.54$	$83.43$	$84.37$	$90.12$	$75.81$	$41.24$	$19.20$	$38.70$	$79.78$	$87.63$	$69.97$	$75.29$	$93.42$	$91.07$	$63.00$	$61.82$	$23.61$	$66.27$	$65.39$	$73.65$	$66.30$	$80.77$	$27.02$	$39.71$	$27.33$
9	$2^{25}$ ( $1{,}024$ repeats)	$79.55$	$43.84$	$91.28$	$89.32$	$85.05$	$85.92$	$85.74$	$88.05$	$91.09$	$81.29$	$81.72$	$87.90$	$69.31$	$40.66$	$18.57$	$38.13$	$76.27$	$84.58$	$64.76$	$72.63$	$83.97$	$82.14$	$64.00$	$59.39$	$17.94$	$56.94$	$56.04$	$64.98$	$65.20$	$73.08$	$26.38$	$39.56$	$26.80$
9	$2^{23}$ ( $4{,}096$ repeats)	$76.34$	$32.68$	$89.45$	$89.84$	$86.03$	$83.49$	$83.42$	$87.18$	$90.61$	$77.80$	$78.69$	$85.47$	$64.62$	$40.16$	$18.33$	$37.66$	$70.92$	$80.20$	$59.29$	$69.85$	$73.48$	$73.21$	$56.00$	$57.66$	$14.38$	$46.69$	$45.79$	$59.57$	$65.05$	$68.27$	$26.37$	$38.84$	$25.81$
10	$\bigstar\,$ All parameters	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
10	Adapter layers, $d=32$	$80.52$	$45.33$	$91.63$	$90.59$	$86.76$	$88.38$	$88.06$	$86.99$	$90.26$	$83.63$	$83.94$	$90.72$	$67.15$	$34.50$	$15.08$	$32.15$	$79.32$	$87.70$	$60.40$	$65.32$	$50.87$	$73.21$	$52.00$	$58.61$	$19.41$	$65.50$	$64.58$	$62.09$	$64.58$	$73.08$	$13.84$	$17.88$	$15.54$
10	Adapter layers, $d=128$	$81.51$	$45.35$	$92.89$	$91.49$	$88.24$	$87.73$	$87.65$	$87.73$	$90.93$	$83.64$	$84.09$	$90.52$	$72.56$	$36.71$	$16.62$	$34.37$	$79.47$	$87.61$	$63.03$	$69.20$	$52.21$	$75.00$	$56.00$	$61.08$	$18.05$	$67.94$	$66.97$	$68.59$	$66.77$	$73.08$	$19.83$	$27.50$	$22.63$
10	Adapter layers, $d=512$	$81.54$	$44.25$	$93.35$	$91.00$	$87.25$	$88.74$	$88.44$	$88.02$	$91.15$	$83.08$	$83.80$	$89.62$	$74.37$	$38.63$	$17.78$	$36.25$	$79.18$	$87.32$	$64.30$	$73.18$	$59.86$	$71.43$	$56.00$	$62.94$	$18.57$	$66.56$	$65.74$	$70.76$	$67.87$	$74.04$	$23.45$	$33.98$	$25.81$
10	Adapter layers, $d=2048$	$82.62$	$49.86$	$92.55$	$91.30$	$87.99$	$88.46$	$88.35$	$88.36$	$91.40$	$83.63$	$83.18$	$90.66$	$76.53$	$39.44$	$18.30$	$37.06$	$79.40$	$87.36$	$68.61$	$74.53$	$88.00$	$91.07$	$58.00$	$61.10$	$18.89$	$66.73$	$66.06$	$73.29$	$71.16$	$75.96$	$25.64$	$36.92$	$26.93$
10	Gradual Unfreezing	$82.50$	$51.74$	$91.97$	$92.61$	$89.71$	$87.27$	$86.90$	$88.26$	$91.35$	$83.42$	$83.49$	$89.71$	$75.09$	$40.88$	$18.95$	$38.40$	$79.17$	$87.30$	$70.79$	$75.51$	$93.09$	$94.64$	$70.00$	$62.03$	$21.51$	$65.69$	$64.79$	$72.92$	$69.12$	$77.89$	$26.71$	$39.02$	$26.93$
11	$\bigstar\,$ Baseline (pre-train/fine-tune)	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
11	Equal	$76.13$	$39.47$	$90.94$	$82.90$	$75.74$	$78.83$	$78.44$	$86.45$	$89.71$	$82.08$	$82.92$	$90.13$	$59.93$	$40.95$	$19.02$	$38.39$	$76.51$	$85.61$	$63.37$	$73.06$	$82.37$	$83.93$	$65.00$	$60.89$	$17.52$	$60.51$	$59.70$	$61.01$	$60.03$	$65.38$	$23.89$	$34.31$	$26.78$
11	Examples-proportional, $K=2^{16}$	$80.45$	$42.07$	$91.97$	$90.97$	$87.50$	$85.41$	$85.04$	$86.89$	$90.10$	$83.01$	$83.66$	$90.74$	$72.56$	$41.16$	$19.04$	$38.59$	$77.25$	$85.72$	$69.95$	$76.67$	$86.38$	$89.29$	$70.00$	$65.93$	$27.91$	$62.78$	$61.95$	$76.90$	$65.83$	$73.08$	$24.35$	$34.99$	$27.10$
11	Examples-proportional, $K=2^{17}$	$81.56$	$47.35$	$91.40$	$91.55$	$88.24$	$86.15$	$85.93$	$86.94$	$90.06$	$82.76$	$84.12$	$90.79$	$75.09$	$41.06$	$19.12$	$38.47$	$77.00$	$85.87$	$67.91$	$77.89$	$77.54$	$85.71$	$57.00$	$67.78$	$27.07$	$61.51$	$60.54$	$79.06$	$65.20$	$74.04$	$24.36$	$35.00$	$27.25$
11	Examples-proportional, $K=2^{18}$	$81.67$	$46.85$	$91.63$	$91.99$	$88.73$	$87.68$	$87.20$	$86.93$	$90.35$	$83.30$	$84.01$	$91.47$	$73.29$	$40.96$	$19.07$	$38.43$	$78.17$	$86.74$	$67.94$	$76.57$	$78.88$	$87.50$	$62.00$	$67.70$	$30.85$	$63.43$	$62.54$	$76.53$	$65.67$	$67.31$	$24.57$	$35.19$	$27.39$
11	Examples-proportional, $K=2^{19}$	$81.42$	$45.94$	$91.63$	$92.20$	$89.22$	$88.44$	$88.32$	$86.84$	$90.10$	$83.73$	$84.29$	$91.84$	$70.40$	$41.26$	$19.24$	$38.71$	$79.78$	$88.15$	$67.30$	$75.66$	$75.59$	$87.50$	$59.00$	$68.22$	$30.64$	$65.32$	$64.29$	$73.65$	$65.05$	$69.23$	$25.21$	$36.30$	$27.76$
11	Examples-proportional, $K=2^{20}$	$80.80$	$42.55$	$92.78$	$91.27$	$87.99$	$88.36$	$88.10$	$86.10$	$89.62$	$84.15$	$84.26$	$92.20$	$68.95$	$41.05$	$19.24$	$38.46$	$80.36$	$88.27$	$67.38$	$73.21$	$76.18$	$83.93$	$62.00$	$67.57$	$26.86$	$66.12$	$65.22$	$76.90$	$64.73$	$69.23$	$25.66$	$36.93$	$27.68$
11	Examples-proportional, $K=2^{21}$	$79.83$	$44.45$	$91.28$	$89.00$	$84.31$	$87.54$	$87.40$	$84.93$	$88.53$	$82.54$	$84.16$	$90.85$	$67.87$	$40.51$	$18.79$	$37.92$	$79.50$	$87.48$	$65.10$	$71.16$	$68.88$	$85.71$	$57.00$	$62.75$	$23.40$	$64.50$	$63.65$	$72.92$	$64.11$	$71.15$	$25.82$	$37.22$	$27.13$
11	Temperature-scaled, $T=2$	$81.90$	$54.00$	$91.74$	$90.56$	$86.76$	$85.11$	$84.60$	$86.40$	$89.74$	$83.47$	$84.15$	$91.51$	$72.56$	$41.09$	$19.28$	$38.54$	$79.42$	$87.77$	$69.92$	$76.73$	$92.37$	$92.86$	$57.00$	$69.80$	$31.90$	$66.65$	$65.74$	$72.92$	$67.08$	$75.96$	$25.42$	$36.72$	$27.20$
11	Temperature-scaled, $T=4$	$80.56$	$45.38$	$91.97$	$89.68$	$85.78$	$83.13$	$82.76$	$86.39$	$90.00$	$82.78$	$84.19$	$91.16$	$73.65$	$41.09$	$19.22$	$38.51$	$77.99$	$86.81$	$69.54$	$76.76$	$97.36$	$96.43$	$59.00$	$68.10$	$31.48$	$64.26$	$63.27$	$74.73$	$64.26$	$71.15$	$25.04$	$35.82$	$27.45$
11	Temperature-scaled, $T=8$	$77.21$	$40.07$	$91.06$	$88.11$	$83.33$	$79.20$	$79.06$	$86.60$	$89.90$	$83.05$	$83.56$	$90.21$	$59.93$	$41.01$	$19.10$	$38.40$	$77.14$	$85.99$	$66.07$	$73.94$	$93.70$	$94.64$	$60.00$	$66.36$	$26.86$	$63.46$	$62.60$	$62.09$	$63.32$	$65.38$	$24.55$	$35.35$	$27.17$
12	$\bigstar\,$ Unsupervised pre-training + fine-tuning	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
12	Multi-task training	$81.42$	$45.94$	$91.63$	$92.20$	$89.22$	$88.44$	$88.32$	$86.84$	$90.10$	$83.73$	$84.29$	$91.84$	$70.40$	$41.26$	$19.24$	$38.71$	$79.78$	$88.15$	$67.30$	$75.66$	$75.59$	$87.50$	$59.00$	$68.22$	$30.64$	$65.32$	$64.29$	$73.65$	$65.05$	$69.23$	$25.21$	$36.30$	$27.76$
12	Multi-task pre-training + fine-tuning	$83.11$	$51.42$	$92.66$	$91.73$	$88.73$	$88.06$	$87.70$	$88.61$	$91.61$	$84.09$	$84.31$	$91.85$	$76.53$	$41.15$	$19.12$	$38.59$	$80.26$	$88.50$	$71.03$	$79.54$	$81.69$	$87.50$	$65.00$	$70.72$	$31.48$	$65.94$	$65.03$	$81.23$	$68.18$	$73.08$	$27.08$	$39.80$	$28.07$
12	Leave-one-out multi-task training	$81.98$	$48.00$	$93.23$	$91.72$	$88.24$	$87.76$	$87.32$	$88.61$	$91.44$	$84.00$	$84.11$	$90.79$	$72.20$	$41.34$	$19.05$	$38.77$	$79.97$	$88.10$	$71.68$	$78.35$	$86.76$	$89.29$	$66.00$	$68.09$	$29.49$	$66.23$	$65.27$	$79.06$	$68.65$	$78.85$	$26.93$	$39.79$	$27.87$
12	Supervised multi-task pre-training	$79.93$	$36.60$	$92.43$	$91.58$	$88.24$	$87.03$	$86.78$	$88.15$	$91.20$	$82.87$	$83.16$	$90.13$	$70.76$	$41.12$	$18.96$	$38.49$	$77.38$	$85.65$	$65.36$	$75.66$	$68.87$	$83.93$	$58.00$	$64.81$	$21.93$	$55.37$	$54.61$	$71.12$	$67.40$	$75.96$	$26.81$	$40.13$	$28.04$
13	$\bigstar\,$ Baseline	$83.28$	$53.84$	$92.68$	$92.07$	$88.92$	$88.02$	$87.94$	$88.67$	$91.56$	$84.24$	$84.57$	$90.48$	$76.28$	$41.33$	$19.24$	$38.77$	$80.88$	$88.81$	$71.36$	$76.62$	$91.22$	$91.96$	$66.20$	$66.13$	$25.78$	$69.05$	$68.16$	$75.34$	$68.04$	$78.56$	$26.98$	$39.82$	$27.65$
13	$1\times$ size, $4\times$ training steps	$85.33$	$60.29$	$93.81$	$94.06$	$91.67$	$89.42$	$89.25$	$89.15$	$91.87$	$86.01$	$85.70$	$91.63$	$78.34$	$41.52$	$19.33$	$38.96$	$82.45$	$90.19$	$74.72$	$79.17$	$94.75$	$92.86$	$71.00$	$67.34$	$29.70$	$72.63$	$71.59$	$78.34$	$72.10$	$82.69$	$27.08$	$40.66$	$27.93$
13	$1\times$ size, $4\times$ batch size	$84.60$	$56.08$	$93.12$	$92.31$	$89.22$	$88.85$	$88.84$	$89.35$	$92.07$	$85.98$	$86.13$	$91.07$	$80.14$	$41.70$	$19.42$	$39.08$	$82.52$	$90.21$	$74.64$	$78.78$	$93.69$	$94.64$	$72.00$	$68.09$	$30.95$	$74.73$	$73.90$	$76.53$	$70.06$	$81.73$	$27.07$	$40.60$	$27.84$
13	$2\times$ size, $2\times$ training steps	$86.18$	$62.04$	$93.69$	$93.36$	$90.69$	$89.18$	$89.23$	$89.35$	$92.05$	$87.23$	$87.05$	$92.68$	$81.95$	$41.74$	$19.66$	$39.14$	$84.18$	$91.29$	$77.18$	$80.98$	$97.36$	$96.43$	$74.00$	$71.34$	$35.68$	$77.11$	$76.34$	$80.51$	$69.28$	$85.58$	$27.52$	$41.03$	$28.19$
13	$4\times$ size, $1\times$ training steps	$85.91$	$57.58$	$94.38$	$92.67$	$89.95$	$89.60$	$89.60$	$89.44$	$92.14$	$87.05$	$87.12$	$93.12$	$83.39$	$41.60$	$19.73$	$39.08$	$83.86$	$91.32$	$78.04$	$81.38$	$89.09$	$94.64$	$73.00$	$73.74$	$40.40$	$78.25$	$77.40$	$81.59$	$70.22$	$91.35$	$27.47$	$40.71$	$28.10$
13	$4\times$ ensembled	$84.77$	$56.14$	$93.46$	$93.31$	$90.67$	$89.71$	$89.60$	$89.62$	$92.24$	$86.22$	$86.53$	$91.60$	$77.98$	$42.10$	$20.10$	$39.56$	$83.09$	$90.40$	$71.74$	$77.58$	$89.85$	$91.07$	$66.00$	$69.32$	$29.49$	$72.67$	$71.94$	$76.90$	$69.12$	$72.12$	$28.05$	$40.53$	$28.09$
13	$4\times$ ensembled, fine-tune only	$84.05$	$54.78$	$92.78$	$93.15$	$90.44$	$88.34$	$88.12$	$89.27$	$91.97$	$85.33$	$85.88$	$90.98$	$77.62$	$41.66$	$19.57$	$39.12$	$82.36$	$89.86$	$71.56$	$77.43$	$90.07$	$92.86$	$69.00$	$67.31$	$26.34$	$70.47$	$69.64$	$75.45$	$68.18$	$74.04$	$27.55$	$40.22$	$28.09$

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Abstract

摘要

1 Introduction

1 引言

keywords

关键词

2 Setup

2 设定

2.1 Model

2.1 模型

2.2 The Colossal Clean Crawled Corpus

2.2 超大规模清洁爬取语料（C4）

2.3 Downstream Tasks

2.3 下游任务

2.4 Input and Output Format

2.4 输入与输出格式

3 Experiments

3 实验

3.1 Baseline

3.1 基线

3.1.1 Model

3.1.1 模型

3.1.2 Training

3.1.2 训练

3.1.3 Vocabulary

3.1.3 词表

3.1.4 Unsupervised Objective

3.1.4 无监督目标

3.1.5 Baseline Performance

3.1.5 基线表现

3.2 Architectures

3.2 架构

3.2.1 Model Structures

3.2.1 模型结构

3.2.2 Comparing Different Model Structures

3.2.2 比较不同模型结构

3.2.3 Objectives

3.2.3 目标

3.2.4 Results

3.2.4 Results

3.3 Unsupervised Objectives

3.3 无监督目标

3.3.1 Disparate High-Level Approaches

3.3.1 差异显著的高层方法

3.3.2 Simplifying the BERT Objective

3.3.2 简化 BERT 目标

3.3.3 Varying the Corruption Rate

3.3.3 改变腐蚀率

3.3.4 Corrupting Spans

3.3.4 腐蚀跨度

3.3.5 Discussion

3.3.5 讨论

3.4 Pre-training Data set

3.4 预训练数据集

3.4.1 Unlabeled Data Sets

3.4.1 无标注数据集

3.4.2 Pre-training Data set Size

3.4.2 预训练数据集规模

3.5 Training Strategy

3.5 训练策略

3.5.1 Fine-tuning Methods

3.5.1 微调方法

3.5.2 Multi-task Learning

3.5.2 多任务学习

3.5.3 Combining Multi-Task Learning with Fine-Tuning

3.5.3 将多任务学习与微调结合

3.6 Scaling

3.6 缩放

3.7 Putting It All Together

3.7 汇总

4 Reflection

4 反思

4.1 Takeaways

4.1 要点

4.2 Outlook

4.2 展望

Appendix A Contributions

附录 A 贡献

acknowledgments-disclosure-of-funding.