Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu

Colin Raffel、Noam Shazeer、Adam Roberts、Katherine Lee、Sharan Narang、Michael Matena、Yanqi Zhou、Wei Li、Peter J. Liu

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

用统一的文本到文本 Transformer 探索迁移学习的极限（Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer）

Colin Raffel craffel@gmail.com Thanks:  Equal contribution. A description of each author’s contribution is available in Appendix   A . Correspondence to craffel@gmail.com .    Noam Shazeer noam@google.com    Adam Roberts adarob@google.com    Katherine Lee katherinelee@google.comSharan Narang sharannarang@google.comMichael Matena mmatena@google.comYanqi Zhou yanqiz@google.comWei Li mweili@google.comPeter J. Liu peterjliu@google.comGoogle, Mountain View, CA 94043, USA

Colin Raffel craffel@gmail.com Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu
Google
作者贡献说明见附录 A；通信请致 craffel@gmail.com

Abstract

摘要

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ‘‘Colossal Clean Crawled Corpus’’, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.[^1]

迁移学习——先在数据丰富的任务上预训练，再在下游任务上微调——已成为自然语言处理（NLP）中的有力技术。迁移学习的有效性催生了多样的方法与实践。本文引入统一框架，把所有基于文本的语言问题都转成文本到文本（text-to-text）格式，借此系统梳理 NLP 迁移学习的版图。我们在数十个语言理解任务上比较预训练目标、架构、无标注数据集、迁移方式及其他因素。把探索所得洞见与规模以及新的「超大规模清洁爬取语料」（Colossal Clean Crawled Corpus，C4）结合起来，我们在摘要、问答、文本分类等多项基准上达到当时最优。为方便后续工作，我们公开数据集、预训练模型与代码。

1 Introduction

1 引言

keywords

Training a machine learning model to perform natural language processing (NLP) tasks often requires that the model can process text in a way that is amenable to downstream learning. This can be loosely viewed as developing general-purpose knowledge that allows the model to “understand” text. This knowledge can range from low-level (e.g. the spelling or meaning of words) to high-level (e.g. that a tuba is too large to fit in most backpacks). In modern machine learning practice, providing this knowledge is rarely done explicitly; instead, it is often learned as part of an auxiliary task. For example, a historically common approach is to use word vectors (Mikolov et al. 2013b; Mikolov et al. 2013a; Pennington et al. 2014) to map word identities to a continuous representation where, ideally, similar words map to similar vectors. These vectors are often learned through an objective that, for example, encourages co-occurring words to be positioned nearby in the continuous space (Mikolov et al. 2013b).

关键词

训练机器学习模型来完成自然语言处理(NLP)任务往往要求模型能够以适合下游学习的方式处理文本. 这可被松散地视为发展通用知识,使模型能够“理解”案文。 这种知识可以从低层次(如词的拼写或意思)到高层次(如大管太大,无法融入大多数背包). 在现代机器学习实践中,提供这种知识很少被明确做到;相反,它常常被作为辅助任务的一部分来学习. 例如,一个历史上常见的方法是使用单词矢量(Mikolov等人,2013年b;Mikolov等人,2013年a;Pennington等人,2014年)将单词身份映射到一个连续的表示,理想的情况是将类似单词映射到类似矢量的地方. 这些矢量通常是通过一个目标来学习的,例如,这个目标鼓励在连续空间中共同使用词语的位置(Mikolov等人,2013年b)。

Recently, it has become increasingly common to pre-train the entire model on a data-rich task. Ideally, this pre-training causes the model to develop general-purpose abilities and knowledge that can then be transferred to downstream tasks. In applications of transfer learning to computer vision (Oquab et al. 2014; Jia et al. 2014; Huh et al. 2016; Yosinski et al. 2014), pre-training is typically done via supervised learning on a large labeled data set like ImageNet (Russakovsky et al. 2015; Deng et al. 2009). In contrast, modern techniques for transfer learning in NLP often pre-train using unsupervised learning on unlabeled data. This approach has recently been used to obtain state-of-the-art results in many of the most common NLP benchmarks (Devlin et al. 2018; Yang et al. 2019; Dong et al. 2019; Liu et al. 2019c; Lan et al. 2019). Beyond its empirical strength, unsupervised pre-training for NLP is particularly attractive because unlabeled text data is available en masse thanks to the Internet---for example, the Common Crawl project[^2] produces about 20TB of text data extracted from web pages each month. This is a natural fit for neural networks, which have been shown to exhibit remarkable scalability, i.e. it is often possible to achieve better performance simply by training a larger model on a larger data set (Hestness et al. 2017; Shazeer et al. 2017; Jozefowicz et al. 2016; Mahajan et al. 2018; Radford et al. 2019; Shazeer et al. 2018; Huang et al. 2018b; Keskar et al. 2019a).

最近,在数据丰富的任务上预先培训整个模型已变得越来越普遍。 理想的情况是,这种预训导致模式发展通用能力和知识,然后可以转移到下游任务. 在计算机视觉转移学习应用方面(Oquab等人,2014年;Jia等人,2014年;Huh等人,2016年;Yosinski等人,2014年),培训前通常通过在ImageNet(Russakovsky等人,2015年;Deng等人,2009年)等大型标注数据集的监督下学习进行。 相比之下,NLP中现代的转移学习技术往往使用无标签数据上的无监督学习进行预训. 这种方法最近被用于获得许多最常见的NLP基准的最新结果(Devlin等人,2018年;杨等人,2019年;董等人,2019年;刘等人,2019年c;兰等人,2019年). 除了经验力量之外,无监督的NLP预训特别有吸引力,因为由于互联网,无标签的文本数据可以大量获取——例如,共同爬行项目[^2]每月从网页中提取大约20TB的文本数据. 这自然适合神经网络,已经证明神经网络表现出显著的可伸缩性,即只要在更大的数据集上培训一个更大的模型,就往往有可能取得更好的性能(Hestness等人,2017年;Shazeer等人,2017年;Jozefowicz等人,2016年;Mahajan等人,2018年;Radford等人,2019年;Shazeer等人,2018年;Huang等人,2018年b;Keskar等人,2019年a).

This synergy has resulted in a great deal of recent work developing transfer learning methodology for NLP, which has produced a wide landscape of pre-training objectives (Howard and Ruder 2018; Devlin et al. 2018; Yang et al. 2019; Dong et al. 2019), unlabeled data sets (Yang et al. 2019; Liu et al. 2019c; Zellers et al. 2019), benchmarks (Wang et al. 2019b; Wang et al. 2018; Conneau and Kiela 2018), fine-tuning methods (Howard and Ruder 2018; Houlsby et al. 2019; Peters et al. 2019), and more. The rapid rate of progress and diversity of techniques in this burgeoning field can make it difficult to compare different algorithms, tease apart the effects of new contributions, and understand the space of existing methods for transfer learning. Motivated by a need for more rigorous understanding, we leverage a unified approach to transfer learning that allows us to systematically study different approaches and push the current limits of the field.

这一协同关系导致近期为NLP制定大量转移学习方法,产生了广泛的培训前目标(Howard和Ruder 2018;Devlin等人2018;Yang等人2019;Dong等人2019),未贴标签的数据集(Yang等人2019;Liu等人2019c;Zellers等人2019),基准(Wang等人2019b;Wang等人2018;Conneau和Kiela2018),微调方法(Howard和Ruder2018;Houlsby等人2019;Peters等人2019),等等. 在这个蓬勃发展的领域中技术的迅速进步和多样化,可能使得难以比较不同的算法,嘲笑新贡献的影响,并理解现有转学方法的空间. 由于需要更严格的理解,我们利用一种统一的方法来转移学习,使我们能够系统地研究不同的方法,并推进该领域目前的极限。

Figure 1: A diagram of our text-to-text framework. Every task we consider—including translation, question answering, and classification—is cast as feeding our model text as input and training it to generate some target text. This allows us to use the same model, loss function, hyperparameters, etc. across our diverse set of tasks. It also provides a standard testbed for the methods included in our empirical survey. “T5” refers to our model, which we dub the “Text-to-Text Transfer Transformer”.

图1:我们文本对文本框架图。 我们所考虑的每一项任务——包括翻译、问答和分类——都是为了提供我们的示范文本,作为投入和培训,以产生一些目标文本。 这使得我们能使用同样的模型,损失功能,超参数等,跨越我们不同的一组任务. 它还为我们的经验性调查所包含的方法提供了一个标准测试床。 “T5”是指我们的型号,我们把它称为证实了“Text-to-TextTransferTransformer”。

The basic idea underlying our work is to treat every text processing problem as a “text-to-text” problem, i.e. taking text as input and producing new text as output. This approach is inspired by previous unifying frameworks for NLP tasks, including casting all text problems as question answering (McCann et al. 2018), language modeling (Radford et al. 2019), or span extraction Keskar et al. 2019b tasks. Crucially, the text-to-text framework allows us to directly apply the same model, objective, training procedure, and decoding process to every task we consider. We leverage this flexibility by evaluating performance on a wide variety of English-based NLP problems, including question answering, document summarization, and sentiment classification, to name a few. With this unified approach, we can compare the effectiveness of different transfer learning objectives, unlabeled data sets, and other factors, while exploring the limits of transfer learning for NLP by scaling up models and data sets beyond what has previously been considered.

我们工作的基本思想是将每个文本处理问题视为一个“文本对文本”问题,即把文本当作投入和产生新的文本作为产出。 这种方法的灵感来自以往NLP任务的统一框架,包括将所有文本问题铸成问答(McCann等人,2018年),语言建模(Radford等人,2019年),或横跨提取Keskar等人,2019年b任务. 关键是,文本对文本的框架使我们能够直接将同样的模式、客观、培训程序和编解码程序应用于我们考虑的每一项任务。 我们利用这种灵活性,评估各种基于英语的NLP问题的业绩,包括回答问题、文件摘要和情绪分类等等。 通过这种统一的方法,我们可以比较不同的转移学习目标,无标签数据集和其他因素的有效性,同时通过扩大模型和数据集的范围,超越之前考虑的范围,探索NLP的转移学习的局限性.

We emphasize that our goal is not to propose new methods but instead to provide a comprehensive perspective on where the field stands. As such, our work primarily comprises a survey, exploration, and empirical comparison of existing techniques. We also explore the limits of current approaches by scaling up the insights from our systematic study (training models up to $11$ billion parameters) to obtain state-of-the-art results in many of the tasks we consider. In order to perform experiments at this scale, we introduce the “Colossal Clean Crawled Corpus” (C4), a data set consisting of hundreds of gigabytes of clean English text scraped from the web. Recognizing that the main utility of transfer learning is the possibility of leveraging pre-trained models in data-scarce settings, we release our code, data sets, and pre-trained models.1

我们强调,我们的目标不是提出新的方法,而是就实地状况提供全面的看法。 因此,我们的工作主要包括对现有技术的调查、探索和经验比较。 我们还通过扩大我们系统研究(培训模式高达XXMATH0.20亿参数)的洞察力,探索目前做法的局限性,以便在我们考虑的许多任务中获得最新成果。 为了进行这种规模的实验,我们引入了“Colossal Clean Crawled Corpus”(C4),这是一个由数百千兆字节从网络上刮掉的干净英文文本组成的数据集。 认识到转移学习的主要用途是在数据稀缺的设置中利用预先训练过的模型的可能性,我们释放了我们的代码,数据集和预先训练过的模型. 页:1

The remainder of the paper is structured as follows: In the following section, we discuss our base model and its implementation, our procedure for formulating every text processing problem as a text-to-text task, and the suite of tasks we consider. In Section 3, we present a large set of experiments that explore the field of transfer learning for NLP. At the end of the section (Section 3.7), we combine insights from our systematic study to obtain state-of-the-art results on a wide variety of benchmarks. Finally, we provide a summary of our results and wrap up with a look towards the future in Section 4.

本文件其余部分的结构如下: 在下一节中,我们讨论我们的基本模式及其执行情况,我们作为文本对文本任务拟定每个文本处理问题的程序,以及我们所考虑的一系列任务。 在第三节中,我们提出了一套大型的实验,探索NLP的转移学习领域. 在本节(第3.7节)末尾,我们综合了我们系统研究的见解,以便在各种基准方面取得最新成果。 最后,我们总结了我们的成果,最后在第4节中展望未来。

2 Setup

2 设置

Before presenting the results from our large-scale empirical study, we review the necessary background topics required to understand our results, including the Transformer model architecture and the downstream tasks we evaluate on. We also introduce our approach for treating every problem as a text-to-text task and describe our “Colossal Clean Crawled Corpus” (C4), the Common Crawl-based data set we created as a source of unlabeled text data. We refer to our model and framework as the “Text-to-Text Transfer Transformer” (T5).

在介绍我们大规模经验研究的结果之前,我们审查了解我们的结果所需的必要背景议题,包括Transformer模型架构和我们评价的下游任务。 我们还介绍了我们把每个问题作为文本对文本的任务处理的方法,并描述了我们作为未加标签的文本数据来源而创建的“Clossal Clean Crawled Corpus”(C4),即以Crawl为基础的共同数据集。 我们把我们的模型和框架称为“Text-to-TextTransfer” (T5)。

2.1 Model

2.1 模型

Early results on transfer learning for NLP leveraged recurrent neural networks (Peters et al. 2018; Howard and Ruder 2018), but it has recently become more common to use models based on the “Transformer” architecture (Vaswani et al. 2017). The Transformer was initially shown to be effective for machine translation, but it has subsequently been used in a wide variety of NLP settings (Radford et al. 2018; Devlin et al. 2018; McCann et al. 2018; Yu et al. 2018). Due to its increasing ubiquity, all of the models we study are based on the Transformer architecture. Apart from the details mentioned below and the variants we explore in Section 3.2, we do not deviate significantly from this architecture as originally proposed. Instead of providing a comprehensive definition of this model, we refer the interested reader to the original paper (Vaswani et al. 2017) or follow-up tutorials[^3],[^4] for a more detailed introduction.

NLP早期的转移学习结果利用了经常性神经网络(Peters等人,2018年;Howard和Ruder 2018年),但最近更常用的是使用基于"Transformer"架构的模型(Vaswani等人,2017年). "Transformer"最初被证明对机器翻译有效,但后来被用于多种NLP设置(拉德福德等人,2018年;德夫林等人,2018年;麦坎等人,2018年;尤等人,2018年). 由于它越来越普遍,我们研究的所有模型都基于Transformer架构. 除了下文提到的细节和我们在第3.2节中探讨的各种变体外,我们没有按照原先的建议,大大偏离这一结构。 我们没有为这一模式提供一个全面的定义,而是请感兴趣的读者参考原论文(Vaswani等人,2017年)或后续教程[^3],[^4],作为更详细的介绍.

The primary building block of the Transformer is self-attention (Cheng et al. 2016). Self-attention is a variant of attention (Graves 2013; Bahdanau et al. 2015) that processes a sequence by replacing each element by a weighted average of the rest of the sequence. The original Transformer consisted of an encoder-decoder architecture and was intended for sequence-to-sequence (Sutskever et al. 2014; Kalchbrenner et al. 2014) tasks. It has recently also become common to use models consisting of a single Transformer layer stack, with varying forms of self-attention used to produce architectures appropriate for language modeling (Radford et al. 2018; Al-Rfou et al. 2019) or classification and span prediction tasks (Devlin et al. 2018; Yang et al. 2019). We empirically explore these architectural variants in Section 3.2.

"变 你为什么要这样做"的主要构件是自觉的(Cheng等人,2016年)。 自觉是注意的一个变体(Graves 2013;Bahdanau等人,2015年),它通过将每个元素替换为其余序列的加权平均值来处理一个序列. 原Transformer由编码器-解码器架构组成,用于序列对序(Sutskever等人,2014年;Kalchbrenner等人,2014年)任务. 最近也常采用由单个Transformer层堆栈组成的模型,使用不同形式的自注意力来制作适合语言模型的架构(拉德福德等人,2018年;Al-Rfou等人,2019年)或分类和跨度预测任务(Devlin等人,2018年;杨等人,2019年). 我们在3.2节中以实证方式探索这些建筑变体.

Overall, our encoder-decoder Transformer implementation closely follows its originally-proposed form (Vaswani et al. 2017). First, an input sequence of tokens is mapped to a sequence of embeddings, which is then passed into the encoder. The encoder consists of a stack of “blocks”, each of which comprises two subcomponents: a self-attention layer followed by a small feed-forward network. Layer normalization (Ba et al. 2016) is applied to the input of each subcomponent. We use a simplified version of layer normalization where the activations are only rescaled and no additive bias is applied. After layer normalization, a residual skip connection (He et al. 2016) adds each subcomponent’s input to its output. Dropout (Srivastava et al. 2014) is applied within the feed-forward network, on the skip connection, on the attention weights, and at the input and output of the entire stack. The decoder is similar in structure to the encoder except that it includes a standard attention mechanism after each self-attention layer that attends to the output of the encoder. The self-attention mechanism in the decoder also uses a form of autoregressive or causal self-attention, which only allows the model to attend to past outputs. The output of the final decoder block is fed into a dense layer with a softmax output, whose weights are shared with the input embedding matrix. All attention mechanisms in the Transformer are split up into independent “heads” whose outputs are concatenated before being further processed.

总的来说,我们的编码器-解码器Transformer的采用与最初提议的形式(Vaswani等人,2017年)密切相仿。 首先,将一个输入序列的符号映射到一个嵌入序列,然后传入编码器. 编码器由一叠“块”组成,每个块由两个子部分组成:自留层,然后是小的向导网。 层态化(Ba等人,2016年)适用于每个子组件的输入. 我们使用简化版的层层正常化,其中的激活只进行重新标定,没有应用添加剂偏差. 在层态化后,一个剩余跳转连接(He等人,2016年)将每个子组件的输入加入到其输出中. 退学(Srivastava等人,2014年)在向导网络内应用,在跳过连接上应用,在注意力重量上应用,在整个堆栈的输入和输出上应用. 解码器在结构上与编码器相似,只是它包括一个标准注意机制,在每一个自注意力层后,都关注编码器的输出. 解码器中的自注意力机制也使用一种自注意力或因果自注意力的形式,只允许模型关注过去的输出. 最终解码器块的输出被装入一个具有软max输出的密集层,其重量与输入嵌入矩阵共享. " Transformer " 中的所有注意机制都分成独立的 " 头 " ,其产出在进一步处理之前就被压缩。

Since self-attention is order-independent (i.e. it is an operation on sets), it is common to provide an explicit position signal to the Transformer. While the original Transformer used a sinusoidal position signal or learned position embeddings, it has recently become more common to use relative position embeddings (Shaw et al. 2018; Huang et al. 2018a). Instead of using a fixed embedding for each position, relative position embeddings produce a different learned embedding according to the offset between the “key” and “query” being compared in the self-attention mechanism. We use a simplified form of position embeddings where each “embedding” is simply a scalar that is added to the corresponding logit used for computing the attention weights. For efficiency, we also share the position embedding parameters across all layers in our model, though within a given layer each attention head uses a different learned position embedding. Typically, a fixed number of embeddings are learned, each corresponding to a range of possible key-query offsets. In this work, we use $32$ embeddings for all of our models with ranges that increase in size logarithmically up to an offset of $128$ beyond which we assign all relative positions to the same embedding. Note that a given layer is insensitive to relative position beyond $128$ tokens, but subsequent layers can build a sensitivity to larger offsets by combining local information from previous layers. To summarize, our model is roughly equivalent to the original Transformer proposed by Vaswani et al. 2017 with the exception of removing the Layer Norm bias, placing the layer normalization outside the residual path, and using a different position embedding scheme. Since these architectural changes are orthogonal to the experimental factors we consider in our empirical survey of transfer learning, we leave the ablation of their impact for future work.

由于自我意向是命令独立的(即它是在sets上的操作),因此通常向Transformer提供明确的位置信号. 虽然最初的Transformer使用了鼻音位置信号或学习到的位置嵌入,但最近更常用的是相对位置嵌入(Shaw等人,2018年;Huang等人,2018年a). 相对位置嵌入法不是为每个位置使用固定嵌入法,而是根据自留置机制中比较的“钥匙”和“钥匙”之间的抵消,产生不同的已学习到的嵌入法。 我们使用一种简化的嵌入式位置形式,其中每个“嵌入”只是用于计算注意力权重的相应记录中添加的标尺。 为了提高效率,我们还分享我们模型中所有层的嵌入参数的位置,尽管在一个特定的层内,每个注意力头都使用不同的学习到的位置嵌入. 通常情况下,会学会固定的嵌入数,每个嵌入数对应一系列可能的密钥-query抵消. 在这项工作中,我们对所有模型使用$32$嵌入法,其范围在大小对数上增加,最多可抵消$128$,除此之外,我们将所有相对位置分配给同一嵌入法。 注意,一个给定的层对$128$令牌以外的相对位置不敏感,但后续的层可以通过结合以前层的局部信息来建立对较大抵消的敏感性. 概括而言,我们的模型大致相当于瓦斯瓦尼等人2017年提出的原Transformer,除了去除层诺姆偏差,将层正常化置于剩余路径之外,以及使用不同位置嵌入方案. 由于这些建筑变化与我们在对转移学习的实证调查中考虑的实验因素是正统的,所以我们把其影响的减退留给今后的工作.

As part of our study, we experiment with the scalability of these models, i.e. how their performance changes as they are made to have more parameters or layers. Training large models can be non-trivial since they might not fit on a single machine and require a great deal of computation. As a result, we use a combination of model and data parallelism and train models on ‘‘slices’’ of Cloud TPU Pods.[^5] TPU pods are are multi-rack ML supercomputers that contain $1{,}024$ TPU v3 chips connected via a high-speed 2D mesh interconnect with supporting CPU host machines. We leverage the Mesh TensorFlow library (Shazeer et al. 2018) for ease of implementation of both model parallelism and data parallelism (Krizhevsky 2014).

作为我们研究的一部分,我们试验这些模型的可伸缩性,即它们的性能如何随着它们被制成更多的参数或层而改变. 训练大型模型可能是非三角型的,因为它们可能不适合单机,需要大量的计算. 因此,我们在Cloud TPU Pods的 QQslices 上采用了模型和数据并行论以及列车模型的组合. [^5] TPU pods是多轨ML超级计算机,包含$1{,}024$ TPU v3芯片,通过高速2D网格连接与支持的CPU主机连接. 我们利用Mesh TensorFlow图书馆(Shazeer等人,2018年)方便实施模式并行主义和数据并行主义(Krizhevsky,2014年)。

2.2 The Colossal Clean Crawled Corpus

2.2 超大规模清洁爬取语料（C4）

Much of the previous work on transfer learning for NLP makes use of large unlabeled data sets for unsupervised learning. In this paper, we are interested in measuring the effect of the quality, characteristics, and size of this unlabeled data. To generate data sets that satisfy our needs, we leverage Common Crawl as a source of text scraped from the web. Common Crawl has previously been used as a source of text data for NLP, for example to train an n-gram language model (Buck et al. 2014), as training data for commonsense reasoning (Trinh and Le 2018), for mining parallel texts for machine translation (Smith et al. 2013), as a pre-training data set (Grave et al. 2018; Zellers et al. 2019; Liu et al. 2019c), and even simply as a giant text corpus for testing optimizers (Anil et al. 2019).

先前为NLP进行的转移学习工作,大多使用大型无标签数据集进行无监督学习. 在本文中,我们有兴趣测量这一无标签数据的质量、特点和大小的影响。 为了生成能满足我们需要的数据集,我们利用Common Crawl作为从网络中刮去的文本的来源. 常见的Crawl此前被用作NLP的文本数据源,例如用于培训n-gram语言模型(Buck等人,2014年),作为常识推理的培训数据(Trinh and Le 2018),用于为机器翻译挖掘平行文本(Smith等人,2013年),作为培训前数据集(Grave等人,2018年;Zellers等人,2019年;Liu等人,2019年c),甚至只是作为测试优化器的巨型文本体(Anil等人,2019年).

Common Crawl is a publicly-available web archive that provides “web extracted text” by removing markup and other non-text content from the scraped HTML files. This process produces around 20TB of scraped text data each month. Unfortunately, the majority of the resulting text is not natural language. Instead, it largely comprises gibberish or boiler-plate text like menus, error messages, or duplicate text. Furthermore, a good deal of the scraped text contains content that is unlikely to be helpful for any of the tasks we consider (offensive language, placeholder text, source code, etc.). To address these issues, we used the following heuristics for cleaning up Common Crawl’s web extracted text:

Crawl是一个可公开查阅的网页档案,它通过从已删除的 HTML 文件中删除标记和其他非文本内容来提供“web提取的文本”。 这个过程每月产生大约20TB的废文本数据. 遗憾的是,由此产生的案文大多数不是自然语言。 相反,它主要包括诸如菜单、错误信息或重复文本等粗略或锅炉板文本。 此外 当我们考虑任何任务时(攻击性语言、占位符文本、源代码等), 为了解决这些问题,我们用以下热度来清理共同爬行网摘录的文本:

• • We only retained lines that ended in a terminal punctuation mark (i.e. a period, exclamation mark, question mark, or end quotation mark).
• • We discarded any page with fewer than 3 sentences and only retained lines that contained at least 5 words.
• • We removed any page that contained any word on the ‘‘List of Dirty, Naughty, Obscene or Otherwise Bad Words’’.[^6]
• • Many of the scraped pages contained warnings stating that Javascript should be enabled so we removed any line with the word Javascript.
• • Some pages had placeholder “lorem ipsum” text; we removed any page where the phrase “lorem ipsum” appeared.
• • Some pages inadvertently contained code. Since the curly bracket “{” appears in many programming languages (such as Javascript, widely used on the web) but not in natural text, we removed any pages that contained a curly bracket.
• • Since some of the scraped pages were sourced from Wikipedia and had citation markers (e.g. [1], [citation needed], etc.), we removed any such markers.
• • Many pages had boilerplate policy notices, so we removed any lines containing the strings “terms of use”, “privacy policy”, “cookie policy”, “uses cookies”, “use of cookies”, or “use cookies”.
• • To deduplicate the data set, we discarded all but one of any three-sentence span occurring more than once in the data set.

• 我们只保留以终点标注结束的线路(即一段期、感叹号、问号或最后引号)。
• 我们删除了任何小于3句的页面,只保留至少5个字的行。
• 我们删除了任何含有“肮脏、猥亵、淫秽或其他坏话清单”任何词的页面。
• 许多被刮掉的页中含有警告,说明应该启用Javascript,所以我们删除了任何与Javascript一词的行.
• 有些页有“自然”的占位词;我们删除了出现“自然”一词的任何页。
• 一些网页无意中载有代码。 由于卷轴括号“{”出现在许多编程语言中(如在网络上广泛使用的Javascript),但不出现在自然文本中,因此我们删除了含有卷轴括号的任何页面。
• 由于一些被刮掉的页面来自维基百科,并有引用标记(如[1],[需要引用]等),我们删除了任何此类标记.
• 许多页都有锅炉板政策通知,因此,我们删除了包含“使用条件”、“隐私政策”、“饼干政策”、“使用饼干”、“使用饼干”或“使用饼干”的任何线条。
• 为了简化数据集,我们放弃了数据集中不止一次发生的除一个之外的所有三刑期。

Additionally, since most of our downstream tasks are focused on English-language text, we used langdetect[^7] to filter out any pages that were not classified as English with a probability of at least 0.99. Our heuristics are inspired by past work on using Common Crawl as a source of data for NLP: For example, Grave et al. 2018 also filter text using an automatic language detector and discard short lines and Smith et al. 2013; Grave et al. 2018 both perform line-level deduplication. However, we opted to create a new data set because prior data sets use a more limited set of filtering heuristics, are not publicly available, and/or are different in scope (e.g. are limited to News data (Zellers et al. 2019; Liu et al. 2019c), comprise only Creative Commons content (Habernal et al. 2016), or are focused on parallel training data for machine translation (Smith et al. 2013)).

此外,由于我们下游的任务大多集中在英语文本上,我们使用langdetect[^7]来过滤任何未归类为英语的页面,其概率至少为0.99. 我们的heuristic受到过去使用Common Crawl作为NLP数据来源的工作的启发: 例如,Grave等人2018也使用自动语言探测器过滤文本,并丢弃短线;Smith等人2013年;Grave等人2018都进行线级疏解. 然而,我们选择建立一套新的数据集,因为以前的数据集使用比较有限的一套过滤heuristics,不公开,和/或范围不同(例如仅限于News数据(Zellers等人,2019年;Liu等人,2019年c),只包含Creative Commons内容(Habernal等人,2016年),或侧重于机器翻译的平行培训数据(Smith等人,2013年))。

To assemble our base data set, we downloaded the web extracted text from April 2019 and applied the aforementioned filtering. This produces a collection of text that is not only orders of magnitude larger than most data sets used for pre-training (about 750 GB) but also comprises reasonably clean and natural English text. We dub this data set the “Colossal Clean Crawled Corpus’’ (or C4 for short) and release it as part of TensorFlow Datasets.[^8] We consider the impact of using various alternative versions of this data set in Section 3.4.

为了组装我们的基础数据集,我们下载了2019年4月的网络提取文本,并应用了上述过滤. 由此收集的文本不仅比大多数用于预训的数据集(约750 GB)的数量级要大,而且包含相当干净和自然的英文文本. 我们将这一数据集编为“COlossalCLear”CLearedCorpus’’(或简称C4),并作为TensorFlow数据集的一部分发布。 我们考虑使用第3.4节中这一数据集的各种替代版本的影响。

2.3 Downstream Tasks

2.3 下游任务

Our goal in this paper is to measure general language learning abilities. As such, we study downstream performance on a diverse set of benchmarks, including machine translation, question answering, abstractive summarization, and text classification. Specifically, we measure performance on the GLUE and SuperGLUE text classification meta-benchmarks; CNN/Daily Mail abstractive summarization; SQuAD question answering; and WMT English to German, French, and Romanian translation. All data was sourced from TensorFlow Datasets.[^9]

我们在本文中的目标是衡量一般语言学习能力. 因此,我们研究了下游的一套不同基准的业绩,包括机器翻译、问答、抽象总结和文本分类。 具体来说,我们测量GLUE和SuperGLUE文本分类元基准;CNN/Daily Mail抽象总结;SQuAD问题回答;WMT英语到德语、法语和罗马尼亚语翻译。 所有数据都来自 TensorFlow 数据集。 [^9]

GLUE (Wang et al. 2018) and SuperGLUE (Wang et al. 2019b) each comprise a collection of text classification tasks meant to test general language understanding abilities:

GLUE(Wang等人,2018年)和SuperGLUE(Wang等人,2019年b)各包含一批文字分类任务,旨在测试一般语言理解能力: 语言理解能力:

• • Sentence acceptability judgment (CoLA (Warstadt et al. 2018))
• • Sentiment analysis (SST-2 (Socher et al. 2013))
• • Paraphrasing/sentence similarity (MRPC (Dolan and Brockett 2005), STS-B (Cer et al. 2017), QQP (Iyer et al. 2017))
• • Natural language inference (MNLI (Williams et al. 2017), QNLI (Rajpurkar et al. 2016), RTE (Dagan et al. 2005), CB (De Marneff et al. 2019))
• • Coreference resolution (WNLI and WSC (Levesque et al. 2012))
• • Sentence completion (COPA (Roemmele et al. 2011))
• • Word sense disambiguation (WIC (Pilehvar and Camacho-Collados 2018))
• • Question answering (MultiRC (Khashabi et al. 2018), ReCoRD (Zhang et al. 2018), BoolQ (Clark et al. 2019))

• 判刑可接受性判决(CoLA(Warstadt等人,2018年))
  • 感官分析(SST-2(Socher等人,2013年))
  • 审案/判决相似性(MRPC(Dolan和Brockett,2005年)、STS-B(Cer等人,2017年)、QQP(Iyer等人,2017年))
• 自然语言推论(MNLI(Williams等人,2017年)、QNLI(Rajpurkar等人,2016年)、RTE(Dagan等人,2005年)、CB(De Marneff等人,2019年))
  • 参照决议(WNLI和WSC(Levesque等人,2012年))
  • 判决完成情况(COPA(Roemmele等人,2011年))
  · 语言感知分裂(WIC(Pilehvar和Camacho-Collados 2018))
• 回答问题(多资源中心(Khashabi等人,2018年)、ReCORD(Zhang等人,2018年)、BoolQ(Clark等人,2019年))

We use the data sets as distributed by the GLUE and SuperGLUE benchmarks. For simplicity, when fine-tuning we treat all of the tasks in the GLUE benchmark (and similarly for SuperGLUE) as a single task by concatenating all of the constituent data sets. As suggested by Kocijan et al. 2019 we also include the Definite Pronoun Resolution (DPR) data set (Rahman and Ng 2012) in the combined SuperGLUE task.

我们使用由GLUE和SuperGLUE基准分配的数据集. 简而言之,微调时,我们将GLUE基准中的所有任务(对于SuperGLUE来说也是类似的)作为一个单一任务,通过调整所有组成数据集. 正如Kocijan等人2019年建议的那样,我们还把Definite Pronoun分辨率(DPR)数据集(Rahman和Ng 2012)纳入合并的SuperGLUE任务中.

The CNN/Daily Mail (Hermann et al. 2015) data set was introduced as a question-answering task but was adapted for text summarization by Nallapati et al. 2016; we use the non-anonymized version from See et al. 2017 as an abstractive summarization task. SQuAD (Rajpurkar et al. 2016) is a common question-answering benchmark. In our experiments, the model is fed the question and its context and asked to generate the answer token-by-token. For WMT English to German, we use the same training data as (Vaswani et al. 2017) (i.e. News Commentary v13, Common Crawl, Europarl v7) and newstest2013 as a validation set (Bojar et al. 2014). For English to French, we use the standard training data from 2015 and newstest2014 as a validation set (Bojar et al. 2015). For English to Romanian, which is a standard lower-resource machine translation benchmark, we use the train and validation sets from WMT 2016 (Bojar et al. 2016). Note that we only pre-train on English data, so in order to learn to translate a given model will need to learn to generate text in a new language.

CNN/Daily Mail(Hermann et al. 2015)数据集作为问答任务推出,但由Nallapati et al. 2016进行文字总结修改;我们使用来自See et al. 2017的非匿名版本作为抽象总结任务. SQuAD(Rajpurkar等人,2016年)是一个常见的问答基准. 在我们的实验中,模型被输入了问题及其背景,并被要求逐个生成答案符号. 对于WMT英语到德语,我们使用与(Vaswani等人,2017年)相同的训练数据(即新闻评论v13,Common Crawl,Europal v7)和新闻test2013作为验证集(Bojar等人,2014年). 对于英语到法语,我们使用2015年的标准培训数据以及Newstest2014作为验证集(Bojar等人,2015年). 对于英语到罗马尼亚语,这是一个标准的低资源机器翻译基准,我们使用来自WMT 2016(Bojar等人,2016年)的列车和验证集. 请注意,我们只进行英语数据的预训,所以为了学习翻译一个特定的模型,需要学习用新语言生成文本.

2.4 Input and Output Format

2.4 输入与输出格式

In order to train a single model on the diverse set of tasks described above, we cast all of the tasks we consider into a “text-to-text” format—that is, a task where the model is fed some text for context or conditioning and is then asked to produce some output text. This framework provides a consistent training objective both for pre-training and fine-tuning. Specifically, the model is trained with a maximum likelihood objective (using “teacher forcing” (Williams and Zipser 1989)) regardless of the task. To specify which task the model should perform, we add a task-specific (text) prefix to the original input sequence before feeding it to the model.

为了就上述各种任务培训一个单一的模型,我们将我们所考虑的所有任务投入到一种 " 文字对文字 " 的格式中,即一种任务,即向模型提供一些文字供上下文或条件,然后要求它产生一些产出文本。 这一框架为培训前和调整提供了一致的培训目标。 具体来说,无论任务如何,该模式都以最大可能性目标(使用“教师强迫”(Williams和Zipser,1989年))加以培训。 要指定模型应该执行哪项任务,我们先在原始输入序列中添加一个任务特定(文本)前缀,然后再将其输入模型.

As an example, to ask the model to translate the sentence “That is good.” from English to German, the model would be fed the sequence “translate English to German: That is good.” and would be trained to output “Das ist gut.” For text classification tasks, the model simply predicts a single word corresponding to the target label. For example, on the MNLI benchmark (Williams et al. 2017) the goal is to predict whether a premise implies (“entailment”), contradicts (“contradiction”), or neither (“neutral”) a hypothesis. With our preprocessing, the input sequence becomes “mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons are filled with animosity.” with the corresponding target word “entailment”. Note that an issue arises if our model outputs text on a text classification task that does not correspond to any of the possible labels (for example if the model outputs “hamburger” when the only possible labels for a task were “entailment”, “neutral”, or “contradiction”). In this case, we always count the model’s output as wrong, though we never observed this behavior in any of our trained models. Note that the choice of text prefix used for a given task is essentially a hyperparameter; we found that changing the exact wording of the prefix had limited impact and so did not perform extensive experiments into different prefix choices. A diagram of our text-to-text framework with a few input/output examples is shown in Figure 1. We provide full examples of preprocessed inputs for every task we studied in Appendix D.

例如,请模型将“这是好的 ” 句子从英文翻译成德文,该模型将输入“把英文翻译成德文:这是好的 ” 的序列,并将接受培训以输出“Das ist gut ” 。 对于文本分类任务,模型只是预测一个对应目标标签的单词. 例如,在MNLI基准(Williams 等人,2017年)上,目标是预测一个前提是否意味着“尾声”),是否与一个假设相矛盾,或者两个假设都不相矛盾。 随着我们的预处理,输入序列成为“mnli前提:我讨厌鸽子。 假设:我对鸽子的感情充满敌意。” 请注意,如果我们的关于文本分类任务的示范产出文本与任何可能的标签不符,就会产生一个问题(例如,当一项任务的唯一可能标签是“贴近”、“中性”或“约束”时,示范产出“汉堡”)。 在这种情况下,我们总是将模型的输出算作错误,尽管我们从未在任何经过训练的模型中观察到过这种行为. 注意特定任务使用的文本前缀的选择本质上是一个超参数;我们发现修改前缀的准确措辞影响有限,因此没有进行广泛的实验,改为不同的前缀选择. 图1显示了我们文本对文本框架的图表,其中有一些投入/产出实例。 我们为附录D中研究的每一项任务提供了经过预先处理的投入的全部例子。

Our text-to-text framework follows previous work that casts multiple NLP tasks into a common format: McCann et al. 2018 propose the “Natural Language Decathlon”, a benchmark that uses a consistent question-answering format for a suite of ten NLP tasks. The Natural Language Decathlon also stipulates that all models must be multi-task, i.e. are able to simultaneously tackle all of the tasks at once. We instead allow for separately fine-tuning the model on each individual task and use short task prefixes instead of an explicit question-answer format. Radford et al. 2019 evaluate the zero-shot learning capabilities of language models by feeding some input to the model as a prefix and then autoregressively sampling an output. For example, automatic summarization is done by feeding in a document followed by the text “TL;DR:” (short for “too long, didn’t read”, a common abbreviation) and then the summary is predicted via autoregressive decoding. We mainly consider models that explicitly process an input with an encoder before generating an output with a separate decoder and we focus on transfer learning rather than zero-shot learning. Finally, Keskar et al. 2019b unify many NLP tasks as “span extraction”, where text corresponding to possible output choices are appended to the input and the model is trained to extract the input span corresponding to the correct choice. In contrast, our framework also allows for generative tasks like machine translation and abstractive summarization where it is not possible to enumerate all possible output choices.

我们的文本对文本框架沿用了之前将多个NLP任务置于一个共同格式的工作:McCann 等人 2018年提出"自然语言Decathlon",这个基准为一套NLP任务使用一致的问答格式. "自然语言"(Natural Language Decathlon)也规定,所有模型都必须是多任务,即能够同时处理所有任务. 相反,我们允许对每个单独任务的模型分别进行微调,并使用简短的任务前缀而不是明确的问答格式。 Radford等人2019评价了语言模型的零弹射学习能力,将一些投入作为前缀提供给模型,然后自动对一个输出进行采样. 例如,自动归纳是通过在“TL;DR:”之后的文档中输入(简写为“太长,没有读过”,一个常用缩写),然后通过自动递归解码来预测汇总。 我们主要考虑的是,在用单独的解码器生成一个输出之前,用编码器明确处理输入的模型,我们侧重于转移学习而不是零镜头学习. 最后,Keskar 等人 2019b 将许多 NLP 任务统一为 “ span 提取 ” , 其中与可能输出选择对应的文本附在输入后,并且该模型经过培训以提取与正确选择相对应的输入跨度。 相比之下,我们的框架也允许诸如机器翻译和抽象归纳等遗传性任务,因为不可能列举所有可能的输出选择。

We were able to straightforwardly cast all of the tasks we considered into a text-to-text format with the exception of STS-B, which is a regression task where the goal is to predict a similarity score between $1$ and $5$. We found that most of these scores were annotated in increments of $0.2$, so we simply rounded any score to the nearest increment of $0.2$ and converted the result to a literal string representation of the number (e.g. the floating-point value $2.57$ would be mapped to the string “2.6”). At test time, if the model outputs a string corresponding to a number between $1$ and $5$, we convert it to a floating-point value; otherwise, we treat the model’s prediction as incorrect. This effectively recasts the STS-B regression problem as a 21-class classification problem.

我们能够直接将我们所审议的所有任务都转化为文本对文本格式,但STS-B除外,这是一个回归式任务,目标是预测$1$和$5$之间的相似性分数。 我们发现,这些分数大多以$0.2$的增量注出,因此我们简单地将任何分数四舍五入到最近的$0.2$增量,并将结果转换为数字的字串表示(例如,浮动点值$2.57$将映射到字符串“2.26”). 在测试时,如果模型输出一个与$1$和$5$之间的数字相对应的字符串,我们把它转换成浮点值;否则,我们将模型的预测视为不正确. 这有效地将STS-B回归问题改写为21级分类问题.

Separately, we also convert the Winograd tasks (WNLI from GLUE, WSC from SuperGLUE, and the DPR data set we add to SuperGLUE) into a simpler format that is more amenable to the text-to-text framework. Examples from the Winograd tasks consist of a text passage containing an ambiguous pronoun that could refer to more than one of the noun phrases in the passage. For example, the passage might be “The city councilmen refused the demonstrators a permit because they feared violence.”, which contains the ambiguous pronoun “they” that could refer to “city councilmen” or “demonstrators”. We cast the WNLI, WSC, and DPR tasks as text-to-text problems by highlighting the ambiguous pronoun in the text passage and asking the model to predict the noun that it refers to. The example mentioned above would be transformed to the input “The city councilmen refused the demonstrators a permit because they feared violence.” and the model would be trained to predict the target text “The city councilmen”.

另外,我们还将W作了答复的Winograd任务(WNLI从GLUE,WSC从SuperGLUE,以及我们添加到SuperGLUE的DPR数据集)转换为更适合文本对文本框架的更简单格式. 维诺格勒任务的例子包括一个文本段落中包含一个模糊的代词,可以指该段落中不止一个名词短语. 例如,这段话可能是“市议员因害怕暴力而拒绝示威者的许可”,其中含有含糊的“他们”一词,可以指“市议员”或“示威者”。 我们把WNLI,WSC和DPR任务作为文本对文本的问题,在文本段落中强调模棱两可的代词,并要求模型预测它所提到的名词. 上面提到的例子将变为“市议员拒绝给示威者发放许可证,因为他们* 害怕暴力。” 模型将接受培训,以预测目标文字“市议员”。

For WSC, examples contain the passage, the ambiguous pronoun, a candidate noun, and a True/False label reflecting whether the candidate matches the pronoun (ignoring any articles). We only train on examples with a “True” label since we do not know the correct noun targets for examples with a “False” label. For evaluation, we assign a “True” label if the words in the model’s output are a subset of the words in the candidate noun phrase (or vice versa) and assign a “False” label otherwise. This removes roughly half of the WSC training set, but the DPR data set adds about $1{,}000$ pronoun resolution examples. Examples from DPR are annotated with the correct referent noun, making it easy to use this data set in the format listed above.

对于WSC来说,例子包括段落,模棱两可的代词,候选名词,以及反映候选名词是否与代词匹配的True/False标签(在任何文章上发声). 我们只是用“真”标签来培训例子,因为我们不知道有“假”标签的例子的正确名词目标。 对于评价,如果模型输出中的单词是候选名词短语(反之亦然)中单词的子集,我们就会指定一个 " True " 标签,并另指定一个 " False " 标签。 这移除了大约一半的WSC训练集,但DPR数据集增加了大约$1{,}000$代名词解析实例. DPR中的实例以正确的参考名词注解,使得使用上述格式的本数据集变得容易.

The WNLI training and validation sets have a significant overlap with the WSC training set. To avoid leaking validation examples into our training data (a particular issue in the multi-task experiments of Section 3.5.2), we therefore never train on WNLI and never report results on the WNLI validation set. Omitting results on the WNLI validation set is standard practice (Devlin et al. 2018) due to the fact that it is “adversarial” with respect to the training set, i.e. validation examples are all slightly-perturbed versions of training examples with the opposite label. As such, we do not include WNLI in the average GLUE score whenever we report on the validation set (all sections except Section 3.7 where results are presented on the test sets). Converting examples from WNLI to the “referent noun prediction” variant described above is a little more involved; we describe this process in Appendix B.

WNLI的培训和验证装置与WSC的训练装置有很大的重叠. 因此,为了避免将验证实例泄露到我们的培训数据中(第3.5.2节的多任务实验中的一个具体问题),我们从未对WNLI进行培训,也从未报告WNLI验证集的结果。 WNLI验证集的观测结果是标准做法(Devlin等人,2018年),因为它对训练集是"对抗性"的,即验证实例都是带有相反标签的训练实例的轻微扰动版本. 因此,我们每次报告验证组(除第3.7节在测试组上提交结果)时,都不将WNLI列入平均GLUE分数。 将例子从WNLI转换为上文所述的“参考名词预测”变体,参与程度略高;我们在附录B中描述了这一过程。

3 Experiments

3 实验

Recent advances in transfer learning for NLP have come from a wide variety of developments, such as new pre-training objectives, model architectures, unlabeled data sets, and more. In this section, we carry out an empirical survey of these techniques in hopes of teasing apart their contribution and significance. We then combine the insights gained to attain state-of-the-art in many of the tasks we consider. Since transfer learning for NLP is a rapidly growing area of research, it is not feasible for us to cover every possible technique or idea in our empirical study. For a broader literature review, we recommend a recent survey by Ruder et al. 2019.

近期NLP的转移学习进展来自多种发展,如新的培训前目标,模型架构,无标签数据集等等. 在本节中,我们对这些技术进行了实证调查,希望它们能够发挥各自的贡献和意义。 然后,我们综合了我们所考虑的许多任务中为达到最新水平而取得的见解。 由于NLP的转学学习是一个迅速增长的研究领域,因此我们无法在经验研究中涵盖每一种可能的技术或想法. 为了进行更广泛的文献审查,我们推荐鲁德等人最近的一项调查. 2019.

We systematically study these contributions by taking a reasonable baseline (described in Section 3.1) and altering one aspect of the setup at a time. For example, in Section 3.3 we measure the performance of different unsupervised objectives while keeping the rest of our experimental pipeline fixed. This “coordinate ascent” approach might miss second-order effects (for example, some particular unsupervised objective may work best on a model larger than our baseline setting), but performing a combinatorial exploration of all of the factors in our study would be prohibitively expensive. In future work, we expect it could be fruitful to more thoroughly consider combinations of the approaches we study.

我们系统地研究这些贡献,采用合理的基线(在第3.1节中说明),并一次改变设置的一个方面。 例如,在第3.3节中,我们衡量不同无监督目标的绩效,同时维持我们试验管道的其余部分。 这种“协调上升”的办法可能忽略了二阶效应(例如,一些特定不受监督的目标在超出我们基准设定的模型上可能最有效),但对我们的研究中的所有因素进行组合性探讨将极为昂贵。 在今后的工作中,我们期望更彻底地考虑我们所研究的各种办法的结合,是能够取得成果的。

Our goal is to compare a variety of different approaches on a diverse set of tasks while keeping as many factors fixed as possible. In order to satisfy this aim, in some cases we do not exactly replicate existing approaches. For example, “encoder-only” models like BERT (Devlin et al. 2018) are designed to produce a single prediction per input token or a single prediction for an entire input sequence. This makes them applicable for classification or span prediction tasks but not for generative tasks like translation or abstractive summarization. As such, none of the model architectures we consider are identical to BERT or consist of an encoder-only structure. Instead, we test approaches that are similar in spirit—for example, we consider an analogous objective to BERT’s “masked language modeling” objective in Section 3.3 and we consider a model architecture that behaves similarly to BERT on text classification tasks in Section 3.2.

我们的目标是在尽可能多地确定因素的同时,就一套不同的任务比较各种不同的办法。 为了实现这一目标,在某些情况下,我们并不完全照搬现有的办法。 例如,BERT(Devlin等人,2018年)等“仅编码器”模型旨在为整个输入序列产生每个输入符的单一预测或单一预测。 这使得它们适用于分类或跨度预测任务,但不适用于翻译或抽象归纳等遗传性任务. 因此,我们认为的模型架构中没有一个与BERT完全相同,或者由一个只编码器的结构组成. 相反,我们检验了精神上相似的方法——例如,我们认为,与BERT在第3.3节中的 " 假冒语言模型 " 目标类似的目标,我们认为,在3.2节的案文分类任务上,示范结构的表现与BERT相似。

After outlining our baseline experimental setup in the following subsection, we undertake an empirical comparison of model architectures (Section 3.2), unsupervised objectives (Section 3.3), pre-training data sets (Section 3.4), transfer approaches (Section 3.5), and scaling (Section 3.6). At the culmination of this section, we combine insights from our study with scale to obtain state-of-the-art results in many tasks we consider (Section 3.7).

在以下小节中概述了我们的基线实验设置之后,我们对模型结构(第3.2节)、不受监督的目标(第3.3节)、培训前数据集(第3.4节)、转移方法(第3.5节)和规模(第3.6节)进行了实证比较。 在本节的高潮中,我们将我们研究的见解与规模结合起来,以便在我们考虑的许多任务中取得最新成果(第3.7节)。

3.1 Baseline

3.1 基线

Our goal for our baseline is to reflect typical, modern practice. We pre-train a standard Transformer (described in Section 2.1) using a simple denoising objective and then separately fine-tune on each of our downstream tasks. We describe the details of this experimental setup in the following subsections.

我们的基线目标是反映典型的现代做法。 我们预先训练一个标准Transformer(在第2.1节中描述),使用一个简单的去名目标,然后对我们的每一项下游任务分别进行微调. 我们在以下各小节中描述了这一实验设置的细节.

3.1.1 Model

For our model, we use a standard encoder-decoder Transformer as proposed by Vaswani et al. 2017. While many modern approaches to transfer learning for NLP use a Transformer architecture consisting of only a single “stack” (e.g. for language modeling (Radford et al. 2018; Dong et al. 2019) or classification and span prediction (Devlin et al. 2018; Yang et al. 2019)), we found that using a standard encoder-decoder structure achieved good results on both generative and classification tasks. We explore the performance of different model architectures in Section 3.2.

3.1.1 型号

对于我们的模型,我们使用由Vaswani等人2017年提出的标准编码器-解码器Transformer. 虽然许多用于NLP的现代转学方法使用只有单个"链条"(如语言建模(Radford等人,2018年;Dong等人,2019年))或分类和跨度预测(Devlin等人,2018年;Yang等人,2019年)组成的Transformer架构,但我们发现,使用标准编码器-解码器结构在基因和分类任务上都取得了良好的成果. 我们探索3.2节不同模型建筑的性能.

Our baseline model is designed so that the encoder and decoder are each similar in size and configuration to a “BERTBASE” (Devlin et al. 2018) stack. Specifically, both the encoder and decoder consist of $12$ blocks (each block comprising self-attention, optional encoder-decoder attention, and a feed-forward network). The feed-forward networks in each block consist of a dense layer with an output dimensionality of $d_{\mathrm{ff}}=3072$ followed by a ReLU nonlinearity and another dense layer. The “key” and “value” matrices of all attention mechanisms have an inner dimensionality of $d_{\mathrm{kv}}=64$ and all attention mechanisms have $12$ heads. All other sub-layers and embeddings have a dimensionality of $d_{\mathrm{model}}=768$. In total, this results in a model with about $220$ million parameters. This is roughly twice the number of parameters of BERTBASE since our baseline model contains two layer stacks instead of one. For regularization, we use a dropout probability of $0.1$ everywhere dropout is applied in the model.

我们的基准模型的设计使得编码器和解码器在大小和配置上都与一个“BERTBASE”(Devlin等人,2018年)堆栈类似。 具体来说,编码器和解码器都由$12$区块(每个区块包括自注意力,可选的编码器解码器注意,以及一个向导网络)组成. 每个区块的向导网络由一个输出维度为$d_{\mathrm{ff}}=3072$的密集层组成,然后是ReLU非线性层和另一个密集层。 所有关注机制的“关键”和“价值”矩阵具有$d_{\mathrm{kv}}=64$的内在维度,所有关注机制都有$12$的头。 所有其他子层和嵌入层的维度为$d_{\mathrm{model}}=768$. 总的来说,这形成了一个拥有大约$220$百万参数的模型. 这大约是BERTBASE参数的两倍,因为我们的基准模型包含两个层叠而不是一个。 在正规化方面,模式中采用了每处$0.1$辍学概率。

3.1.2 Training

As described in Section 2.4, all tasks are formulated as text-to-text tasks. This allows us to always train using standard maximum likelihood, i.e. using teacher forcing (Williams and Zipser 1989) and a cross-entropy loss. For optimization, we use AdaFactor (Shazeer and Stern 2018). At test time, we use greedy decoding (i.e. choosing the highest-probability logit at every timestep).

3.1.2 培训

如第2.4节所述,所有任务都作为文本对文本的任务拟订。 这使得我们总是可以使用标准的最大可能性来进行训练,即使用教师强迫(Williams和Zipser 1989)和交叉性的损失. 为了优化,我们使用AdaFactor(Shazeer和Stern 2018). 在测试时,我们使用贪婪的解码(即在每个时步选择最高概率的登录).

We pre-train each model for $2^{19}=524{,}288$ steps on C4 before fine-tuning. We use a maximum sequence length of $512$ and a batch size of $128$ sequences. Whenever possible, we ‘‘pack’’ multiple sequences into each entry of the batch[^10] so that our batches contain roughly $2^{16}=65{,}536$ tokens. In total, this batch size and number of steps corresponds to pre-training on $2^{35}\approx 34\mathrm{B}$ tokens. This is considerably less than BERT (Devlin et al. 2018), which used roughly $137\mathrm{B}$ tokens, or RoBERTa (Liu et al. 2019c), which used roughly $2.2\mathrm{T}$ tokens. Using only $2^{35}$ tokens results in a reasonable computational budget while still providing a sufficient amount of pre-training for acceptable performance. We consider the effect of pre-training for more steps in Sections 3.6 and 3.7. Note that $2^{35}$ tokens only covers a fraction of the entire C4 data set, so we never repeat any data during pre-training.

我们在微调前先对每款型号进行XMATH0XX步骤在C4上的预训. 我们使用最大序列长度为$512$,批量大小为$128$序列. 只要有可能,我们就会在批[^10]的每个条目中装入多个序列,使我们的批次包含大约$2^{16}=65{,}536$令牌。 总的来说,这种批量大小和步数对应$2^{35}\approx 34\mathrm{B}$令牌的预训. 这大大低于BERT(Devlin等人,2018年),后者大约使用$137\mathrm{B}$令牌,或RoBERTa(Liu等人,2019年c),后者大约使用$2.2\mathrm{T}$令牌. 仅使用$2^{35}$令牌就产生了合理的计算预算,同时仍然为可接受的业绩提供足够数量的预训. 我们认为预训对第3.6节和第3.7节中更多步骤的影响。 请注意,$2^{35}$令牌只涵盖整个C4数据集的一小部分,因此我们在预训期间从未重复过任何数据.

During pre-training, we use an “inverse square root” learning rate schedule: $1\big/\sqrt{\max(n,k)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps (set to $10^{4}$ in all of our experiments). This sets a constant learning rate of $0.01$ for the first $10^{4}$ steps, then exponentially decays the learning rate until pre-training is over. We also experimented with using a triangular learning rate (Howard and Ruder 2018), which produced slightly better results but requires knowing the total number of training steps ahead of time. Since we will be varying the number of training steps in some of our experiments, we opt for the more generic inverse square root schedule.

在预训期间,我们使用“反方根”学习率表:$1\big/\sqrt{\max(n,k)}$,其中$n$是目前的训练迭代,$k$是热身步骤的数量(我们所有实验中设定为$10^{4}$)。 这为第一个$10^{4}$步骤设定了$0.01$的恒定学习速率,然后指数化地衰减学习速率直至预训结束. 我们还试用了三角学习率(Howard和Ruder 2018),取得了略好的结果,但需要提前了解培训步骤的总数。 由于我们将在有些实验中改变训练步骤的数量,我们选择比较通用的反方根表.

Our models are fine-tuned for $2^{18}=262{,}144$ steps on all tasks. This value was chosen as a trade-off between the high-resource tasks (i.e. those with large data sets), which benefit from additional fine-tuning, and low-resource tasks (smaller data sets), which overfit quickly. During fine-tuning, we continue using batches with $128$ length-$512$ sequences (i.e. $2^{16}$ tokens per batch). We use a constant learning rate of $0.001$ when fine-tuning. We save a checkpoint every $5{,}000$ steps and report results on the model checkpoint corresponding to the highest validation performance. For models fine-tuned on multiple tasks, we choose the best checkpoint for each task independently. For all of the experiments except those in Section 3.7, we report results in the validation set to avoid performing model selection on the test set.

我们的模型被精细调整,用于所有任务的$2^{18}=262{,}144$步骤. 这一数值被选作高资源任务(即那些拥有大数据集的任务)与低资源任务(更小的数据集)之间的权衡,后者得益于额外的微调,而后者很快的过于匹配. 在微调期间,我们继续使用带有$128$长度-$512$序列的批次(即每批$2^{16}$令牌). 我们在微调时使用$0.001$的恒定学习率. 我们每个$5{,}000$步骤都省下一个检查站,并报告模型检查站对应于最高验证性能的结果. 对于对多个任务进行微调的模型,我们独立选择每个任务的最佳检查点. 除了3.7节的实验外,我们报告验证组的结果,以避免在测试组上进行模型选择。

3.1.3 Vocabulary

We use SentencePiece (Kudo and Richardson 2018) to encode text as WordPiece tokens (Sennrich et al. 2015; Kudo 2018). For all experiments, we use a vocabulary of $32{,}000$ wordpieces. Since we ultimately fine-tune our model on English to German, French, and Romanian translation, we also require that our vocabulary covers these non-English languages. To address this, we classified pages from the Common Crawl scrape used in C4 as German, French, and Romanian. Then, we trained our SentencePiece model on a mixture of $10$ parts of English C4 data with $1$ part each of data classified as German, French or Romanian. This vocabulary was shared across both the input and output of our model. Note that our vocabulary makes it so that our model can only process a predetermined, fixed set of languages.

3.1.3 词汇

我们使用PriedPiece(Kudo and Richardson 2018)将文本编码为WordPiece符(Sennrich et al. 2015;Kudo 2018). 对所有实验,我们使用$32{,}000$词组的词汇. 由于我们最终微调了英语到德语、法语和罗马尼亚语的翻译模式,我们也要求我们的词汇涵盖这些非英语语言。 为了解决这个问题,我们把C4中使用的Crawl scrape中的网页归类为德语、法语和罗马尼亚语。 然后,我们训练了我们的PrisonPi spiderce模型, 将英国C4数据的$10$部分与$1$部分混合在一起, 每个数据被归类为德国,法国或罗马尼亚. 我们的模型投入和产出都分享了这一词汇。 请注意,我们的词汇使得我们的模型只能处理一套预定的,固定的语言.

3.1.4 Unsupervised Objective

Figure 2: Schematic of the objective we use in our baseline model. In this example, we process the sentence “Thank you for inviting me to your party last week.” The words “for”, “inviting” and “last” (marked with an $\times$) are randomly chosen for corruption. Each consecutive span of corrupted tokens is replaced by a sentinel token (shown as and ) that is unique over the example. Since “for” and “inviting” occur consecutively, they are replaced by a single sentinel . The output sequence then consists of the dropped-out spans, delimited by the sentinel tokens used to replace them in the input plus a final sentinel token .

3.1.4 无监督的目标

图2:我们在基线模型中使用的目标的示意图。 在这个例子中,我们处理“感谢你上周邀请我参加你的聚会”这句话。 随意选用“用于”、“邀请”和“最后”(标注为$\times$)等词作为腐败罪。 每个连续的损坏的代币都被一个哨音代币(shown as QQ和)所取代,该代币在实例上是独一无二的. 由于“为”和“邀请”是连续发生的,它们被单一的哨兵所取代。 然后,输出序列由退出的跨度组成,由用于在输入中替换的哨点符号加上最后哨点符号所划分。

Leveraging unlabeled data to pre-train our model necessitates an objective that does not require labels but (loosely speaking) teaches the model generalizable knowledge that will be useful in downstream tasks. Preliminary work that applied the transfer learning paradigm of pre-training and fine-tuning all of the model’s parameters to NLP problems used a causal language modeling objective for pre-training (Dai and Le 2015; Peters et al. 2018; Radford et al. 2018; Howard and Ruder 2018). However, it has recently been shown that “denoising” objectives (Devlin et al. 2018; Taylor 1953) (also called “masked language modeling”) produce better performance and as a result they have quickly become standard. In a denoising objective, the model is trained to predict missing or otherwise corrupted tokens in the input. Inspired by BERT’s “masked language modeling” objective and the “word dropout” regularization technique (Bowman et al. 2015), we design an objective that randomly samples and then drops out $15\%$ of tokens in the input sequence. All consecutive spans of dropped-out tokens are replaced by a single sentinel token. Each sentinel token is assigned a token ID that is unique to the sequence. The sentinel IDs are special tokens which are added to our vocabulary and do not correspond to any wordpiece. The target then corresponds to all of the dropped-out spans of tokens, delimited by the same sentinel tokens used in the input sequence plus a final sentinel token to mark the end of the target sequence. Our choices to mask consecutive spans of tokens and only predict dropped-out tokens were made to reduce the computational cost of pre-training. We perform thorough investigation into pre-training objectives in Section 3.3. An example of the transformation resulting from applying this objective is shown in Figure 2. We empirically compare this objective to many other variants in Section 3.3.

利用未加标签的数据来预先训练我们的模型,需要有一个不需要标签的目标,但(轻而易举地)教导在下游任务中有用的模型一般知识。 对NLP问题应用了前期培训的转移学习范式,并对模型的所有参数进行了微调的初步工作使用了前期培训的因果语言模型化目标(Dai和Le 2015;Peters等人,2018;Radford等人,2018;Howard和Ruder 2018). 然而,最近有证据表明,“贬低”目标(Devlin等人,2018年;Taylor 1953年)(也称为“大规模语言建模”)能产生更好的性能,并因此迅速成为标准。 在打点目标中,该模型被训练来预测输入中缺失或以其他方式腐蚀的符号. 在BERT的“假冒语言模型”目标和“辞退”规范化技术(Bowman等人,2015年)的启发下,我们设计了一个随机抽样的目标,然后在输入序列中丢掉$15\%$。 所有连续跨度的退学令牌被一个单一的哨兵令牌所取代. 每个哨音令牌被指定一个令牌ID,这是序列独有的. 哨兵的ID是特殊标志 加入我们的词汇 与任何文字都不符 目标然后对应所有丢弃的代号跨度,由输入序列中使用的同一代号外加最终的代号以标记目标序列的结束. 我们选择掩盖连续的象征,只预测退学的象征,以减少预训的计算费用。 我们对第3.3节的培训前目标进行彻底调查。 应用这一目标所产生的转变实例见图2。 我们从经验上将这一目标与第3.3节中许多其他变体进行比较。

3.1.5 Baseline Performance

In this section, we present results using the baseline experimental procedure described above to get a sense of what kind of performance to expect on our suite of downstream tasks. Ideally, we would repeat every experiment in our study multiple times to get a confidence interval on our results. Unfortunately, this would be prohibitively expensive due to the large number of experiments we run. As a cheaper alternative, we train our baseline model $10$ times from scratch (i.e. with different random initializations and data set shuffling) and assume that the variance over these runs of the base model also applies to each experimental variant. We don’t expect most of the changes we make to have a dramatic effect on the inter-run variance, so this should provide a reasonable indication of the significance of different changes. Separately, we also measure the performance of training our model for $2^{18}$ steps (the same number we use for fine-tuning) on all downstream tasks without pre-training. This gives us an idea of how much pre-training benefits our model in the baseline setting.

3.1.5 基线业绩

在本节中,我们利用上述基线实验程序介绍结果,以了解我们一套下游任务需要什么样的业绩。 理想的情况是,我们将在研究中重复多次每一个实验,以获得对结果的信任间隔。 不幸的是,由于我们进行的实验数量众多,这费用太高了。 作为更便宜的替代品,我们从零开始训练我们的基线模型$10$倍(即采用不同的随机初始化和数据组散射),并假设基模型在这些运行上的差异也适用于每个实验变体. 我们并不期望我们所做的大多数改变会对运行间差异产生巨大影响, 另外,我们还测量了在不经过预先培训的情况下对所有下游任务进行$2^{18}$步骤(与我们用于微调的数量相同)培训的绩效。 这让我们知道,在基线设定中,培训前的训练对我们的模式产生了多少好处。

When reporting results in the main text, we only report a subset of the scores across all the benchmarks to conserve space and ease interpretation. For GLUE and SuperGLUE, we report the average score across all subtasks (as stipulated by the official benchmarks) under the headings “GLUE” and “SGLUE”. For all translation tasks, we report the BLEU score (Papineni et al. 2002) as provided by SacreBLEU v1.3.0 (Post 2018) with “exp” smoothing and “intl” tokenization. We refer to scores for WMT English to German, English to French, and English to Romanian as EnDe, EnFr, and EnRo, respectively. For CNN/Daily Mail, we find the performance of models on the ROUGE-1-F, ROUGE-2-F, and ROUGE-L-F metrics (Lin 2004) to be highly correlated so we report the ROUGE-2-F score alone under the heading “CNNDM”. Similarly, for SQuAD we find the performance of the “exact match” and “F1” scores to be highly correlated so we report the “exact match” score alone. We provide every score achieved on every task for all experiments in Table 16, Appendix E.

在报告正文结果时,我们只报告所有基准的分数,以节省空间和方便解释。 就GLUE和SuperGLUE而言,我们在“GLUE”和“SGLUE”的标题下报告所有子任务的平均得分(按照官方基准的规定)。 对所有翻译任务,我们报告SacreBLEU v1.3.0(2018年后)提供的BLEU分数(Papineni等人,2002年),并用“exp”平滑和“intl”表示。 我们分别将WMT英语对德语,英语对法语,英语对罗马尼亚语的分数称为EnDe,EnFr和EnRo. 就CNN/Daily Mail而言,我们发现ROUGE-1-F、ROUGE-2-F和ROUGE-L-F(Lin 2004)的模型性能密切相关,因此我们在“CNNDM”标题下仅报告ROUGE-2-F的得分。 同样,对于SQuAD来说,我们发现“精确匹配”和“F1”的分数表现高度相关,因此我们仅报告“精确匹配”分数。 我们为附录E表16中的所有实验提供每项任务所取得的每一分数。

Our results tables are all formatted so that each row corresponds to a particular experimental configuration with columns giving the scores for each benchmark. We will include the mean performance of the baseline configuration in most tables. Wherever a baseline configuration appears, we will mark it with a $\bigstar$ (as in the first row of Table 1). We also will boldface any score that is within two standard deviations of the maximum (best) in a given experiment.

我们的成果表都是格式化的,以便每一行都对应一个特定的实验配置,各栏给出每个基准的分数。 我们将把基线配置的平均业绩列入大多数表格。 在出现基线配置的地方,我们将用$\bigstar$(如表1第一行)标记。 我们还将在某一试验中,在最大(最大)的两个标准偏差内的任何分数。

	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$Baseline average	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
Baseline standard deviation	$0.235$	$0.065$	$0.343$	$0.416$	$0.112$	$0.090$	$0.108$
No pre-training	$66.22$	$17.60$	$50.31$	$53.04$	$25.86$	$\mathbf{39.77}$	$24.04$

Table 1: Average and standard deviation of scores achieved by our baseline model and training procedure. For comparison, we also report performance when training on each task from scratch (i.e. without any pre-training) for the same number of steps used to fine-tune the baseline model. All scores in this table (and every table in our paper except Table 14) are reported on the validation sets of each data set.

	格吕	CNNDM( CNNDM ) : 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题	SQAD 状态	英格兰	定义	定义Fr	恩罗
$\bigstar\,$ 基线平均数	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
基线标准差	$0.235$	$0.065$	$0.343$	$0.416$	$0.112$	$0.090$	$0.108$
没有 不用了	$66.22$	$17.60$	$50.31$	$53.04$	$25.86$	$\mathbf{39.77}$	$24.04$

表1:我国基准模式和培训程序实现的分数平均和标准差。 为了进行比较,我们还报告从头到尾(即没有任何培训前)就用于微调基准模式的相同步骤进行培训的情况。 本表(除表14外,我们文件中的每个表格)的所有分数都报告在每个数据集的验证组上.

Our baseline results are shown in Table 1. Overall, our results are comparable to existing models of similar size. For example, BERTBASE achieved an exact match score of $80.8$ on SQuAD and an accuracy of $84.4$ on MNLI-matched, whereas we achieve $80.88$ and $84.24$, respectively (see Table 16). Note that we cannot directly compare our baseline to BERTBASE because ours is an encoder-decoder model and was pre-trained for roughly $\nicefrac{{1}}{{4}}$ as many steps. Unsurprisingly, we find that pre-training provides significant gains across almost all benchmarks. The only exception is WMT English to French, which is a large enough data set that gains from pre-training tend to be marginal. We include this task in our experiments to test the behavior of transfer learning in the high-resource regime. Since we perform early stopping by selecting the best-performing checkpoint, the large disparity between our baseline and “no pre-training” emphasize how much pre-training improves performance on tasks with limited data. While we do not explicitly measure improvements in data efficiency in this paper, we emphasize that this is one of the primary benefits of the transfer learning paradigm.

我们的基线结果见表1。 总体而言,我们的结果与规模类似的现有模型相当。 例如,BERTBASE在SQuAD上实现了$80.8$的精确比分,在MNLI上实现了$84.4$的精确比分,而我们分别实现了$80.88$和$84.24$(见表16). 请注意,我们无法直接将基线与BERTBASE进行比较,因为我们的编码器-解码器模型,并且经过了大约$\nicefrac{{1}}{{4}}$的预先训练,可以达到许多步骤. 不足为奇的是,我们发现,在几乎所有基准中,预先培训都取得了显著进展。 唯一的例外是WMT英语到法语,这是一套足够大的数据集,从预训中获得的收益往往微不足道. 我们将此任务纳入实验中,以测试高资源体系中的转移学习行为. 由于我们通过选择表现最好的检查站来进行早期检查,我们的基线与“无培训前”之间的巨大差距突出表明,培训前工作能用有限的数据改善各项任务的业绩。 虽然我们没有在本文中明确衡量数据效率的提高,但我们强调这是转移学习模式的主要好处之一。

As for inter-run variance, we find that for most tasks the standard deviation across runs is smaller than $1\%$ of the task’s baseline score. Exceptions to this rule include CoLA, CB, and COPA, which are all low-resource tasks from the GLUE and SuperGLUE benchmarks. For example, on CB our baseline model had an average F1 score of $91.22$ with a standard deviation of $3.237$ (see Table 16), which may be partly due to the fact that CB’s validation set contains only $56$ examples. Note that the GLUE and SuperGLUE scores are computed as the average of scores across the tasks comprising each benchmark. As a result, we caution that the high inter-run variance of CoLA, CB, and COPA can make it harder to compare models using the GLUE and SuperGLUE scores alone.

至于运行间的差异,我们发现对于大多数任务来说,跨运行的标准差小于任务基准分数的$1\%$。 这一规则的例外包括CoLA,CB,和COPA,它们都是GLUE和SuperGLUE基准中的低资源任务. 例如,在CB中,我们的基线模型的平均F1分为$91.22$,标准差为$3.237$(见表16),部分原因可能是CB的验证集只包含$56$的例子。 请注意,GLUE和SuperGLUE的分数是按构成每个基准的任务的平均分数计算的。 因此,我们告诫说,CoLA,CB和COPA的高运行间差异可能使得单用GLUE和SuperGLUE的得分比较模型变得更加困难.

3.2 Architectures

3.2 架构

While the Transformer was originally introduced with an encoder-decoder architecture, much modern work on transfer learning for NLP uses alternative architectures. In this section, we review and compare these architectural variants.

虽然"Transformer"最初 小心采用编码器-解码器架构,但许多用于NLP的转移学习的现代工作都采用了替代架构. 在本节中,我们续审查和比较这些建筑变体.

3.2.1 Model Structures

Figure 3: Matrices representing different attention mask patterns. The input and output of the self-attention mechanism are denoted $x$ and $y$ respectively. A dark cell at row $i$ and column $j$ indicates that the self-attention mechanism is allowed to attend to input element $j$ at output timestep $i$. A light cell indicates that the self-attention mechanism is not allowed to attend to the corresponding $i$ and $j$ combination. Left: A fully-visible mask allows the self-attention mechanism to attend to the full input at every output timestep. Middle: A causal mask prevents the $i$th output element from depending on any input elements from “the future”. Right: Causal masking with a prefix allows the self-attention mechanism to use fully-visible masking on a portion of the input sequence.

3.2.1 模型结构

图3:代表不同关注面罩图案的图案. 自觉机制的输入和输出分别表示$x$和$y$. 第$i$栏和第$j$栏的暗单元格表示,允许自觉机制在输出时间步$i$. 一个轻细胞表示,自觉机制是不允许处理相应的$i$和$j$组合. 左侧:一个全可见的面具可以让自觉机制在每一个输出时段都注意全输入. 中间:一个因果面具使$i$第2个输出元素无法依赖来自"未来"的任何输入元素. 右侧:带有前缀的causal masking允许自觉机制在部分输入序列上使用完全可见的面具.

A major distinguishing factor for different architectures is the “mask” used by different attention mechanisms in the model. Recall that the self-attention operation in a Transformer takes a sequence as input and outputs a new sequence of the same length. Each entry of the output sequence is produced by computing a weighted average of entries of the input sequence. Specifically, let $y_{i}$ refer to the $i$th element of the output sequence and $x_{j}$ refer to the $j$th entry of the input sequence. $y_{i}$ is computed as $\sum_{j}w_{i,j}x_{j}$, where $w_{i,j}$ is the scalar weight produced by the self-attention mechanism as a function of $x_{i}$ and $x_{j}$. The attention mask is then used to zero out certain weights in order to constrain which entries of the input can be attended to at a given output timestep. Diagrams of the masks we will consider are shown in Figure 3. For example, the causal mask (Figure 3, middle) sets any $w_{i,j}$ to zero if $j>i$.

不同结构的一个主要区别因素是模型中不同关注机制使用的“任务”。 回顾Transformer中的自注意力操作以一个序列作为输入,输出一个相同长度的新序列. 输出序列的每个条目都是通过计算输入序列条目的加权平均值生成的. 具体来说,让$y_{i}$指输出序列的$i$第十一元素,让$x_{j}$指输入序列的$j$第十一条目. $y_{i}$被计算为$\sum_{j}w_{i,j}x_{j}$,其中$w_{i,j}$是自觉机制作为$x_{i}$和$x_{j}$的一种函数产生的卡片重量. 然后使用注意面罩去除一定的重量,以限制在特定输出时段可以处理哪些输入的条目. 我们将考虑的口罩图见图3。 例如,因果面罩(图3,中间)将任何$w_{i,j}$设定为零,如果是$j>i$.

Figure 4: Schematics of the Transformer architecture variants we consider. In this diagram, blocks represent elements of a sequence and lines represent attention visibility. Different colored groups of blocks indicate different Transformer layer stacks. Dark grey lines correspond to fully-visible masking and light grey lines correspond to causal masking. We use “.” to denote a special end-of-sequence token that represents the end of a prediction. The input and output sequences are represented as $x$ and $y$ respectively. Left: A standard encoder-decoder architecture uses fully-visible masking in the encoder and the encoder-decoder attention, with causal masking in the decoder. Middle: A language model consists of a single Transformer layer stack and is fed the concatenation of the input and target, using a causal mask throughout. Right: Adding a prefix to a language model corresponds to allowing fully-visible masking over the input.

图4:我们考虑的Transformer结构变体的示意图. 在这个图中,块代表一个序列的元素,线代表注意的能见度. 不同颜色的块组表示不同的Transformer层堆栈. 暗灰色线对应全可见的遮盖,浅灰色线对应因果遮盖. 我们使用“.”来表示一种特殊的序列结束信号,它代表着预测的结束。 输入和输出序列分别以$x$和$y$表示. 左:一个标准的编码器-解码器架构在编码器和编码器-解码器的注意力中采用完全可见的遮罩,在解码器中带有因果遮罩. 中间:一种语言模型由单个Transformer层堆栈组成,并被输入和目标的调和,在整个过程中使用因果面具. 右:在语言模型中添加前缀对应允许在输入上完全可见的遮盖.

The first model structure we consider is an an encoder-decoder Transformer, which consists of two layer stacks: The encoder, which is fed an input sequence, and the decoder, which produces a new output sequence. A schematic of this architectural variant is shown in the left panel of Figure 4.

我们考虑的第一个模型结构是编码器-解码器Transformer,它由两层堆叠组成: 被输入序列的编码器和生成新输出序列的解码器. 该建筑变体的示意图见图4的左面板.

The encoder uses a “fully-visible” attention mask. Fully-visible masking allows a self-attention mechanism to attend to any entry of the input when producing each entry of its output. We visualize this masking pattern in Figure 3, left. This form of masking is appropriate when attending over a “prefix”, i.e. some context provided to the model that is later used when making predictions. BERT (Devlin et al. 2018) also uses a fully-visible masking pattern and appends a special “classification” token to the input. BERT’s output at the timestep corresponding to the classification token is then used to make a prediction for classifying the input sequence.

编码器使用“完全可见”的注意罩。 完全可见的遮罩可以使一个自觉机制在生产其输出的每个条目时能够处理输入的任何条目. 我们在图3中将这个蒙面图案描绘为左侧. 这种蒙面形式在“前缀”上,即向后来在预测时使用的模型提供某些上下文时是适当的。 BERT(Devlin等人,2018年)也使用全可见的遮盖图案,并在输入中附加一个特殊的"分类"符号. BERT在与分类符对应的时序输出,然后用于预测输入序列的分类.

The self-attention operations in the Transformer’s decoder use a “causal” masking pattern. When producing the $i$th entry of the output sequence, causal masking prevents the model from attending to the $j$th entry of the input sequence for $j>i$. This is used during training so that the model can’t “see into the future” as it produces its output. An attention matrix for this masking pattern is shown in Figure 3, middle.

Transformer解码器中的自觉操作使用一种"因果"遮掩模式. 在生成输出序列的及其它输入序列的$i$第2个条目时,因果遮罩使模型无法关注$j$第2个输入序列的$j>i$. 这在培训期间使用,以便模型在产生其产出时不能 " 看到未来 " 。 这种蒙面图案的注意矩阵如图3中所示.

The decoder in an encoder-decoder Transformer is used to autoregressively produce an output sequence. That is, at each output timestep, a token is sampled from the model’s predicted distribution and the sample is fed back into the model to produce a prediction for the next output timestep, and so on. As such, a Transformer decoder (without an encoder) can be used as a language model (LM), i.e. a model trained solely for next-step prediction (Liu et al. 2018; Radford et al. 2018; Al-Rfou et al. 2019). This constitutes the second model structure we consider. A schematic of this architecture is shown in Figure 4, middle. In fact, early work on transfer learning for NLP used this architecture with a language modeling objective as a pre-training method (Radford et al. 2018).

编码器-解码器Transformer中的解码器用于自动递归生成输出序列. 也就是说,在每个输出时间步上,从模型的预测分布中抽取一个符号,并将样本反馈到模型中,为下一个输出时间步作出预测等等. 因此,Transformer解码器(没有编码器)可以用作语言模型(LM),即完全为下一步预测而训练的模型(Liu等人,2018年;Radford等人,2018年;Al-Rfou等人,2019年). 这是我们认为的第二个模式结构。 这种建筑的图示如图4,中间部分所示. 事实上,NLP早期的转学学习工作将这个带有语言建模目标的架构作为预训方法(拉德福德等人,2018年).

Language models are typically used for compression or sequence generation (Graves 2013). However, they can also be used in the text-to-text framework simply by concatenating the inputs and targets. As an example, consider the case of English to German translation: If we have a training datapoint with input sentence “That is good.” and target “Das ist gut.”, we would simply train the model on next-step prediction over the concatenated input sequence “translate English to German: That is good. target: Das ist gut.” If we wanted to obtain the model’s prediction for this example, the model would be fed the prefix “translate English to German: That is good. target:” and would be asked to generate the remainder of the sequence autoregressively. In this way, the model can predict an output sequence given an input, which satisfies the needs of text-to-text tasks. This approach was recently used to show that language models can learn to perform some text-to-text tasks without supervision (Radford et al. 2019).

语言模型通常用于压缩或序列生成(Graves 2013). 然而,它们也可以仅仅通过压缩投入和目标在文本对文本框架中使用。 例如,将英文译为德文的情况: 如果我们有一个带有“很好”和“Das 这是很好的。 目标: " Das ist gut " 。 如果我们想获得模型的预测,这个模型会被输入前缀 " 将英语翻译成德语: 这是很好的。 目标:“并将要求自动生成序列的其余部分。 这样,模型可以预测给定输入的输出序列,这满足了文本到文本任务的需要. 这种方法最近被用来表明语言模型可以学会在没有监督的情况下执行一些文本对文本的任务(Radford等人,2019年).

A fundamental and frequently cited drawback of using a language model in the text-to-text setting is that causal masking forces the model’s representation of the $i$th entry of the input sequence to only depend on the entries up until $i$. To see why this is potentially disadvantageous, consider the text-to-text framework where the model is provided with a prefix/context before being asked to make predictions (e.g., the prefix is an English sentence and the model is asked to predict the German translation). With fully causal masking, the model’s representation of a prefix state can only depend on prior entries of the prefix. So, when predicting an entry of the output, the model will attend to a representation of the prefix that is unnecessarily limited. Similar arguments have been made against using a unidirectional recurrent neural network encoder in sequence-to-sequence models (Bahdanau et al. 2015).

在文本对文本设置中使用一种语言模型的一个基本和经常提到的缺点是,因果遮掩迫使该模型代表输入序列的$i$第2个条目,只取决于直到$i$为止的条目。 为了了解这有潜在劣势的原因,在被要求作出预测(例如前缀是英文句子,要求模型预测德文翻译)之前,先考虑给模型提供前缀/语句的文本对文本框架. 在完全因果遮盖的情况下,模型对前缀状态的表示只能取决于前缀的先前条目. 因此,在预测输出的条目时,模型将处理一个不必要限制的前缀的表示. 也有人提出类似的论点,反对在序列到序列模型中使用单向反复神经网络编码器(Bahdanau等人,2015年)。

This issue can be avoided in a Transformer-based language model simply by changing the masking pattern. Instead of using a causal mask, we use fully-visible masking during the prefix portion of the sequence. This masking pattern and a schematic of the resulting “prefix LM” (the third model structure we consider) are illustrated in the rightmost panels of Figures 3 and 4, respectively. In the English to German translation example mentioned above, fully-visible masking would be applied to the prefix “translate English to German: That is good. target:” and causal masking would be used during training for predicting the target “Das ist gut.” Using a prefix LM in the text-to-text framework was originally proposed by Liu et al. 2018. More recently, Dong et al. 2019 showed that this architecture is effective on a wide variety of text-to-text tasks. This architecture is similar to an encoder-decoder model with parameters shared across the encoder and decoder and with the encoder-decoder attention replaced with full attention across the input and target sequence.

这个问题可以在基于Transformer的语言模型中简单地通过改变遮掩模式来避免. 与其使用因果口罩,不如在序列的前缀部分使用全可见口罩. 图3和图4最右侧的板块分别说明了这种掩盖模式和由此产生的“前缀LM”(我们认为是第三个模型结构)。 在上文提到的英文到德文翻译例子中,完全可见的面具将适用于前缀“将英文译为德文: 这是很好的。 目标:“和因果遮罩将用于训练,以预测目标“Das ist gut”。 在文本对文本框架中使用前缀LM最初是由刘等人在2018年提出的. 最近,董等人的2019年显示,这种架构在各种各样的文本对文本任务上是有效的. 这种架构类似于一个编码器-解码器模型,其参数在编码器和解码器之间共享,而编码器-解码器的注意力则在输入和目标序列之间被完全的注意所取代.

We note that when following our text-to-text framework, the prefix LM architecture closely resembles BERT (Devlin et al. 2018) for classification tasks. To see why, consider an example from the MNLI benchmark where the premise is “I hate pigeons.”, the hypothesis is “My feelings towards pigeons are filled with animosity.” and the correct label is “entailment”. To feed this example into a language model, we would transform it into the sequence “mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons are filled with animosity. target: entailment”. In this case, the fully-visible prefix would correspond to the entire input sequence up to the word “target:”, which can be seen as being analogous to the “classification” token used in BERT. So, our model would have full visibility over the entire input, and then would be tasked with making a classification by outputting the word “entailment”. It is easy for the model to learn to output one of the valid class labels given the task prefix (“mnli” in this case). As such, the main difference between a prefix LM and the BERT architecture is that the classifier is simply integrated into the output layer of the Transformer decoder in the prefix LM.

我们注意到,在遵循我们的文本对文本框架时,前缀LM架构与分类任务的BERT(Devlin等人,2018年)非常相似. 为了了解原因,请从MNLI基准中以“我讨厌鸽子”为前提,假设是“我对鸽子的感情充满敌意”。 为了将这个例子输入语言模型,我们将把它变成“mnli前提:我讨厌鸽子”。 假设:我对鸽子的感情充满敌意. 目标:产生”。 在这种情况下,完全可见的前缀将对应整个输入序列,直至“目标:”一词,这可被视为类似于BERT中使用的“分类”符号。 因此,我们的模型将对整个投入具有充分的可见度,然后负责通过输出“尾注”一词来进行分类。 模型很容易在任务前缀(在此情况下为“mnli”)中学习输出一个有效的类标签。 因此,前缀LM和BERT架构的主要区别在于,分类器只是被整合到前缀LM中的Transformer解码器的输出层中.

3.2.2 Comparing Different Model Structures

In the interest of experimentally comparing these architectural variants, we would like each model we consider to be equivalent in some meaningful way. We might say that two models are equivalent if they either have the same number of parameters or they require roughly the same amount of computation to process a given (input-sequence, target-sequence) pair. Unfortunately, it is not possible to compare an encoder-decoder model to a language model architecture (comprising a single Transformer stack) according to both of these criteria at the same time. To see why, first note an encoder-decoder model with $L$ layers in the encoder and $L$ layers in the decoder has approximately the same number of parameters as a language model with $2L$ layers. However, the same $L+L$ encoder-decoder model will have approximately the same computational cost as a language model with only $L$ layers. This is a consequence of the fact that the $L$ layers in the language model must be applied to both the input and output sequence, while the encoder is only applied to the input sequence and the decoder is only applied to the output sequence. Note that these equivalences are approximate—there are some extra parameters in the decoder due to the encoder-decoder attention and there are also some computational costs in the attention layers that are quadratic in the sequence lengths. In practice, however, we observed nearly identical step times for $L$-layer language models versus $L+L$-layer encoder-decoder models, suggesting a roughly equivalent computational cost. Further, for the model sizes we consider, the number of parameters in the encoder-decoder attention layers is about 10% of the total parameter count, so we make the simplifying assumption that an $L+L$-layer encoder-decoder model has the same number of parameters as an $2L$-layer language model.

3.2.2 比较不同的模型结构

为了实验性地比较这些建筑变体,我们希望我们所认为的每一个模型都以某种有意义的方式等同起来. 我们可以说,如果两个模型的参数数目相同,或者它们需要大致相同的计算量来处理一个给定的对(输入-序列,目标-序列),这两个模型是等效的. 不幸的是,无法同时根据这两个标准将编码器-解码器模型与语言模型架构(包含一个单Transformer堆栈)进行比较. 要了解为什么,首先注意一个编码器-解码器模型,其编码器中的$L$层和解码器中的$L$层的参数数量与具有$2L$层的语言模型大致相同. 然而,同样的$L+L$编码器-解码器模型的计算成本与语言模型大致相同,只有**$L$层. 这是由于语言模型中的$L$层必须应用于*双 *输入和输出序列,而编码器仅适用于输入序列,解码器仅适用于输出序列. 注意这些等效物是近似的——由于编码器-解码器的注意,解码器中有一些额外的参数,在注意层中也有一些计算成本,在序列长度上是四极的. 然而,在实践中,我们看到$L$-层语言模型与$L+L$-层编码器-解码器模型的步数几乎相同,表明计算成本大致相当。 此外,对于我们考虑的模型大小,编码器-解码器关注层中的参数数大约是总参数数的10%,所以我们做了简化的假设,即$L+L$-层编码器-解码器模型的参数数与$2L$-层语言模型相同.

To provide a reasonable means of comparison, we consider multiple configurations for our encoder-decoder model. We will refer to the number of layers and parameters in a BERTBASE-sized layer stack as $L$ and $P$, respectively. We will use $M$ to refer to the number of FLOPs required for an $L+L$-layer encoder-decoder model or $L$-layer decoder-only model to process a given input-target pair. In total, we will compare:

为了提供合理的比较手段,我们考虑我们的编码器-解码器模型的多种配置. 我们将分别以$L$和$P$来指代BERTBASE大小的层堆栈中的层数和参数. 我们将使用$M$来指代一个$L+L$ - 层编码器-解码器模型或$L$ - 层解码器-只处理一个给定输入目标对所需的FLOP数量. 总的来说,我们将比较:

• • An encoder-decoder model with $L$ layers in the encoder and $L$ layers in the decoder. This model has $2P$ parameters and a computation cost of $M$ FLOPs.
• • An equivalent model, but with parameters shared across the encoder and decoder, resulting in $P$ parameters and an $M$-FLOP computational cost.
• • An encoder-decoder model with $L/2$ layers each in the encoder and decoder, giving $P$ parameters and an $M/2$-FLOP cost.
• • A decoder-only language model with $L$ layers and $P$ parameters and a resulting computational cost of $M$ FLOPs.
• • A decoder-only prefix LM with the same architecture (and thus the same number of parameters and computational cost), but with fully-visible self-attention over the input.

• 编码器-解码器模型,其中编码器为$L$层,解码器为$L$层。 该模型有$2P$参数和$M$ FLOP的计算成本.
• 等效模型,但参数在编码器和解码器之间共享,导致$P$参数和$M$-FLOP计算成本。
• 一个编码器-解码器模型,在编码器和解码器中各有$L/2$层,给出$P$参数和$M/2$-FLOP成本.
• 一个只有解码器的语言模型,有$L$层和$P$参数,并由此产生$M$ FLOP的计算成本.
• 只有一个解码器的前缀LM,其结构相同(因此参数和计算成本相同),但在输入上完全可见。

3.2.3 Objectives

As an unsupervised objective, we will consider both a basic language modeling objective as well as our baseline denoising objective described in Section 3.1.4. We include the language modeling objective due to its historic use as a pre-training objective (Dai and Le 2015; Ramachandran et al. 2016; Howard and Ruder 2018; Radford et al. 2018; Peters et al. 2018) as well as its natural fit for the language model architectures we consider. For models that ingest a prefix before making predictions (the encoder-decoder model and prefix LM), we sample a span of text from our unlabeled data set and choose a random point to split it into prefix and target portions. For the standard language model, we train the model to predict the entire span from beginning to end. Our unsupervised denoising objective is designed for text-to-text models; to adapt it for use with a language model we concatenate the inputs and targets as described in Section 3.2.1.

3.2.3 目标

作为一项不受监督的目标,我们将既考虑基本的语文建模目标,也考虑第3.1.4节所述的基线去名目标。 我们包括了语言建模目标,因为其作为培训前目标的历史用途(Dai和Le 2015;Ramachandran等人,2016年;Howard和Ruder 2018;Radford等人,2018年;Peters等人,2018年)及其与我们认为的语言建模架构的自然适合性. 对于在作出预测前摄入前缀的模型(编码器-解码器模型和前缀LM),我们从我们无标签的数据集中抽取了一段文字,并选择随机点将其分为前缀和目标部分. 对于标准语言模型,我们训练模型来预测从开始到结束的整个跨度. 我们不受监督的去名目标是为文本对文本模式设计的;为了使其与语言模式相适应,我们将第3.2.1节所述的投入和目标结合起来。

3.2.4 Results

Architecture	Objective	Params	Cost	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$Encoder-decoder	Denoising	$2P$	$M$	$\mathbf{83.28}$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
Enc-dec, shared	Denoising	$P$	$M$	$82.81$	$18.78$	$\mathbf{80.63}$	$\mathbf{70.73}$	$26.72$	$39.03$	$\mathbf{27.46}$
Enc-dec, 6 layers	Denoising	$P$	$M/2$	$80.88$	$18.97$	$77.59$	$68.42$	$26.38$	$38.40$	$26.95$
Language model	Denoising	$P$	$M$	$74.70$	$17.93$	$61.14$	$55.02$	$25.09$	$35.28$	$25.86$
Prefix LM	Denoising	$P$	$M$	$81.82$	$18.61$	$78.94$	$68.11$	$26.43$	$37.98$	$27.39$
Encoder-decoder	LM	$2P$	$M$	$79.56$	$18.59$	$76.02$	$64.29$	$26.27$	$39.17$	$26.86$
Enc-dec, shared	LM	$P$	$M$	$79.60$	$18.13$	$76.35$	$63.50$	$26.62$	$39.17$	$27.05$
Enc-dec, 6 layers	LM	$P$	$M/2$	$78.67$	$18.26$	$75.32$	$64.06$	$26.13$	$38.42$	$26.89$
Language model	LM	$P$	$M$	$73.78$	$17.54$	$53.81$	$56.51$	$25.23$	$34.31$	$25.38$
Prefix LM	LM	$P$	$M$	$79.68$	$17.84$	$76.87$	$64.86$	$26.28$	$37.51$	$26.76$

Table 2: Performance of the different architectural variants described in Section 3.2.2. We use $P$ to refer to the number of parameters in a 12-layer base Transformer layer stack and $M$ to refer to the FLOPs required to process a sequence using the encoder-decoder model. We evaluate each architectural variant using a denoising objective (described in Section 3.1.4) and an autoregressive objective (as is commonly used to train language models).

3.2.4 结果煞费苦心

EnFr = EnRo ===
唵(引)|日啰(二合引)贺(引)贺.
$\bigstar\,$ Encoder-decoder$2P$$2P$$M$$\mathbf{83.28}$$\mathbf{19.24}$$\mathbf{80.88}$$\mathbf{71.36}$$\mathbf{26.98}$$\mathbf{39.82}$$\mathbf{27.65}$
XXMAT的三个月$82.81$$18.78$$\mathbf{80.63}$$\mathbf{70.73}$$26.72$$39.03$ $\mathbf{27.46}$
$80.88$$18.97$$18.97$$77.59$$68.42$$26.38$$38.40$$26.95$$26.95$
语言模型$P$$M$$74.70$$17.93$$61.14$$55.02$$25.09$$35.28$ $25.86$
$P$$M$$81.82$$18.61$$78.94$$68.11$$26.43$$37.98$$27.39$ $27.39$
$79.56$$79.56$$18.59$$76.02$$64.29$$26.27$$39.17$$26.86$$26.86$
$M$$79.60$$18.13$$76.35$$63.50$$26.62$$39.17$$27.05$$27.05$
$M/2$$18.26$$18.26$$75.32$$64.06$$26.13$$38.42$$26.89$$26.89$
LM$P$$M$$73.78$$17.54$$53.81$$56.51$$25.23$$34.31$$25.38$$25.38$
LM 前缀 LM$P$$M$$79.68$$17.84$$76.87$$64.86$$26.28$$37.51$ $26.76$

表2:第3.2.2节所述不同建筑变体的性能。 我们使用$P$来指代一个12层基底Transformer层堆栈中的参数数,而$M$来指代使用编码器-解码器模型处理一个序列所需的FLOP. 我们使用一个去名目标(在第3.1.4节中描述)和一个自动递归目标(通常用于培训语言模型)来评价每个建筑变体。

The scores achieved by each of the architectures we compare are shown in Table 2. For all tasks, the encoder-decoder architecture with the denoising objective performed best. This variant has the highest parameter count ($2P$) but the same computational cost as the $P$-parameter decoder-only models. Surprisingly, we found that sharing parameters across the encoder and decoder performed nearly as well. In contrast, halving the number of layers in the encoder and decoder stacks significantly hurt performance. Concurrent work (Lan et al. 2019) also found that sharing parameters across Transformer blocks can be an effective means of lowering the total parameter count without sacrificing much performance. XLNet also bears some resemblance to the shared encoder-decoder approach with a denoising objective (Yang et al. 2019). We also note that the shared parameter encoder-decoder outperforms the decoder-only prefix LM, suggesting that the addition of an explicit encoder-decoder attention is beneficial. Finally, we confirm the widely-held conception that using a denoising objective always results in better downstream task performance compared to a language modeling objective. This observation has been previously made by Devlin et al. 2018, Voita et al. 2019, and Lample and Conneau 2019 among others. We undertake a more detailed exploration of unsupervised objectives in the following section.

我们所比较的每个架构所获得的分数见表2。 对所有任务来说,带有解密目标的编码器-解码器架构表现最好. 此变体拥有最高参数计数($2P$),但与$P$-参数解码器唯一模式的计算成本相同. 令人惊讶的是,我们发现共享参数 跨编码器和解码器几乎也完成了。 相比之下,将编码器和解码器堆栈的层数减半,对性能有很大的伤害. 同时期工作(Lan et al. 2019)也发现跨变形星块共享参数可以成为降低总参数计数而不牺牲很多性能的有效手段. XLNet也与具有去诺目标的共同编码器-解码器方法有些相似(Yang等人,2019年). 我们还注意到,共享参数编码器-解码器超越了解码器唯一的前缀LM,建议添加一个明确的编码器-解码器的注意是有好处的. 最后,我们确认一个普遍持有的概念,即使用去名目标总是比语言建模目标更能完成下游任务。 此观测此前由Devlin等人2018年,Voita等人2019年,Lample和Conneau2019等人提出. 我们将在下一节更详细地探讨不受监督的目标。

3.3 Unsupervised Objectives

3.3 无监督目标

The choice of unsupervised objective is of central importance as it provides the mechanism through which the model gains general-purpose knowledge to apply to downstream tasks. This has led to the development of a wide variety of pre-training objectives (Dai and Le 2015; Ramachandran et al. 2016; Radford et al. 2018; Devlin et al. 2018; Yang et al. 2019; Liu et al. 2019b; Wang et al. 2019a; Song et al. 2019; Dong et al. 2019; Joshi et al. 2019). In this section, we perform a procedural exploration of the space of unsupervised objectives. In many cases, we will not replicate an existing objective exactly—some will be modified to fit our text-to-text encoder-decoder framework and, in other cases, we will use objectives that combine concepts from multiple common approaches.

选择不受监督的目标至关重要,因为它提供了一种机制,通过这一机制,该模式获得普通用途知识,用于下游任务。 这导致了各种培训前目标的发展(Dai和Le 2015;Ramachandran等人,2016年;Radford等人,2018年;Devlin等人,2018年;Yang等人,2019年;Liu等人,2019年b;Wang等人,2019年a;Song等人,2019年;Dong等人,2019年;Dong等人,2019年9月20日;Dong等人,2019年9月20日;Devford等人,2018年;Devlin等人,2018年;Yang等人,2019年9月20日;Liu等人,2019年b;Wang等人,2019年a;Song等人,2019年19年;Dong等人,2019年9月2019日;Dong等人,2019年1月2019日)。 在本节中,我们从程序上探讨不受监督的目标的空间。 在许多情况下,我们不会完全照搬一个现有目标——有些目标将加以修改,以适应我们从文本到文本的编码器-解码器框架,而在另一些情况下,我们将使用从多种共同办法中综合概念的目标。

Overall, all of our objectives ingest a sequence of token IDs corresponding to a tokenized span of text from our unlabeled text data set. The token sequence is processed to produce a (corrupted) input sequence and a corresponding target. Then, the model is trained as usual with maximum likelihood to predict the target sequence. We provide illustrative examples of many of the objectives we consider in Table 3.

总的来说,我们所有的目标 都摄取了一系列的代号标识 与我们未加标签的文本数据集的代号长度相对应 符号序列经过处理,生成一个(被腐蚀)输入序列和相应的目标. 之后,该模型被照常训练,最有可能预测目标序列. 我们举例说明了我们在表3中考虑的许多目标。

Objective	Inputs	Targets
Prefix language modeling	Thank you for inviting	me to your party last week .
BERT-style Devlin et al. 2018	Thank you me to your party apple week .	(original text)
Deshuffling	party me for your to . last fun you inviting week Thank	(original text)
MASS-style Song et al. 2019	Thank you me to your party week .	(original text)
I.i.d. noise, replace spans	Thank you me to your party week .	for inviting last
I.i.d. noise, drop tokens	Thank you me to your party week .	for inviting last
Random spans	Thank you to week .	for inviting me your party last

Table 3: Examples of inputs and targets produced by some of the unsupervised objectives we consider applied to the input text “Thank you for inviting me to your party last week .” Note that all of our objectives process tokenized text. For this particular sentence, all words were mapped to a single token by our vocabulary. We write (original text) as a target to denote that the model is tasked with reconstructing the entire input text. denotes a shared mask token and , , and denote sentinel tokens that are assigned unique token IDs. The BERT-style objective (second row) includes a corruption where some tokens are replaced by a random token ID; we show this via the greyed-out word apple.

目标	输入	目标
前缀语言建模	谢谢你的邀请	我上周参加你的派对
BERT风格Devlin等人 2018年	感谢您"M" "M" 我为您的派对苹果周。	(原文)
脱衣舞	请你为我狂欢吧,最后的乐趣	践行(原文)
MASS风格的宋等 2019	感谢你的"M" "M" 我对你的"M"周。	(原文)
I.i.d. 噪声,替换跨度	谢谢你的派对 "Y"周。	QQ 用于邀请 最后一个
I.i.d. 噪声, 投币	谢谢你的派对周。	用于最后一次邀请
随机跨度	感谢您 我不想在"Y"周。	QQ 因邀请我 "Y " 您的晚会最后一次"Z"

表3:一些未经监督的目标所产生的投入和目标的例子,我们认为这些投入适用于“感谢你上周邀请我参加你的聚会”。 请注意,我们的所有目标都经过了kenized文本。 对于这个特别的句子,所有词都用我们的词汇图成一个符号. 我们写 * (原始文本) * 作为目标,以表示该模型的任务是重建整个输入文本。 表示一个共享的掩码, +, , 表示被指定为唯一代号的哨点标识。 BERT风格的目标(第二行)包括一种腐败,即一些令牌被随机令牌ID所取代;我们通过灰外字苹果来显示这一点.

3.3.1 Disparate High-Level Approaches

To begin with, we compare three techniques that are inspired by commonly-used objectives but differ significantly in their approach. First, we include a basic “prefix language modeling” objective as was used in Section 3.2.3. This technique splits a span of text into two components, one to use as inputs to the encoder and the other to use as a target sequence to be predicted by the decoder. Second, we consider an objective inspired by the “masked language modeling” (MLM) objective used in BERT (Devlin et al. 2018). MLM takes a span of text and corrupts $15\%$ of the tokens. $90\%$ of the corrupted tokens are replaced with a special mask token and $10\%$ are replaced with a random token. Since BERT is an encoder-only model, its goal during pre-training is to reconstruct masked tokens at the output of the encoder. In the encoder-decoder case, we simply use the entire uncorrupted sequence as the target. Note that this differs from our baseline objective, which uses only the corrupted tokens as targets; we compare these two approaches in Section 3.3.2. Finally, we also consider a basic deshuffling objective as used e.g. in (Liu et al. 2019a) where it was applied to a denoising sequential autoencoder. This approach takes a sequence of tokens, shuffles it, and then uses the original deshuffled sequence as a target. We provide examples of the inputs and targets for these three methods in the first three rows of Table 3.

3.3.1 不同高级别办法

首先,我们比较了三种技术,它们受常用目标的启发,但其方法差别很大。 首先,我们列入了第3.2.3节中使用的基本“前缀语文模式”目标。 这一技术将一段文字分成两个部分,一个是用作编码器的输入,另一个是用作解码器预测的目标序列. 其次,我们认为一个目标受到BERT(Devlin等人,2018年)中使用的“假冒语言建模”(MLM)目标的启发。 MLM需要一段文字,并损坏了符号的$15\%$. 被腐蚀的代币的$90\%$被替换为特殊的面具代币,$10\%$被替换为随机代币. 由于BERT是一个只编码器的模型,它在预训期间的目标是在编码器输出时重建蒙面符号. 在编码器-解码器的情况下,我们只是使用整个未腐蚀的序列作为目标. 请注意,这与我们的基线目标不同,因为基线目标只使用腐败的标志作为目标;我们在第3.3.2节中比较了这两种方法。 最后,我们也考虑在制定一个基本解析目标,例如(Liu等人,2019年a)中,它被应用到自动编码器中。 这种方法采用一系列的符号,洗牌,然后使用原来的解牌序列作为目标. 我们举例说明表3前三行对这三种方法的投入和目标。

Objective	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
Prefix language modeling	$80.69$	$18.94$	$77.99$	$65.27$	$\mathbf{26.86}$	$39.73$	$\mathbf{27.49}$
BERT-style (Devlin et al. 2018)	$\mathbf{82.96}$	$\mathbf{19.17}$	$\mathbf{80.65}$	$\mathbf{69.85}$	$\mathbf{26.78}$	$\mathbf{40.03}$	$\mathbf{27.41}$
Deshuffling	$73.17$	$18.59$	$67.61$	$58.47$	$26.11$	$39.30$	$25.62$

Table 4: Performance of the three disparate pre-training objectives described in Section 3.3.1.

目标	格吕	CNNDM( CNNDM ) : 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题	SQAD 状态	英格兰	定义	定义Fr	安围绕着
前缀语言建模	$80.69$	$18.94$	$77.99$	$65.27$	$\mathbf{26.86}$	$39.73$	$\mathbf{27.49}$
BERT风格(Devlin等人,2018年).	$\mathbf{82.96}$	$\mathbf{19.17}$	$\mathbf{80.65}$	$\mathbf{69.85}$	$\mathbf{26.78}$	$\mathbf{40.03}$	$\mathbf{27.41}$
脱衣舞	$73.17$	$18.59$	$67.61$	$58.47$	$26.11$	$39.30$	$25.62$

表 Rainbow4:第3.3.1节所述三个不同的培训前目标的执行情况。

The performance of these three objectives is shown in Table 4. Overall, we find that the BERT-style objective performs best, though the prefix language modeling objective attains similar performance on the translation tasks. Indeed, the motivation for the BERT objective was to outperform language model-based pre-training. The deshuffling objective performs considerably worse than both prefix language modeling and the BERT-style objective.

这三个目标的执行情况见表4。 总的来说,我们发现BERT式的目标表现最好,尽管前缀语言模型的目标在翻译任务上也取得了类似的表现. 事实上,BERT目标的动机是超越基于语言模式的预训。 拆解目标的表现比前缀语言模型和BERT风格的目标都差得多.

3.3.2 Simplifying the BERT Objective

Based on the results in the prior section, we will now focus on exploring modifications to the BERT-style denoising objective. This objective was originally proposed as a pre-training technique for an encoder-only model trained for classification and span prediction. As such, it may be possible to modify it so that it performs better or is more efficient in our encoder-decoder text-to-text setup.

3.3.2 简化BERT目标

根据上一节中的结果,我们现在将集中探讨修改BERT式的去名目标。 这一目标最初是作为培训前技术提出的,用于一种只经过分类和跨度预测培训的编码器模型。 因此,也许可以修改它,以便在我们的编码器-解码器文本到文本设置中更好地或更有效地执行它.

First, we consider a simple variant of the BERT-style objective where we don’t include the random token swapping step. The resulting objective simply replaces $15\%$ of the tokens in the input with a mask token and the model is trained to reconstruct the original uncorrupted sequence. A similar masking objective was used by Song et al. 2019 where it was referred to as “MASS”, so we call this variant the “MASS-style” objective. Second, we were interested to see if it was possible to avoid predicting the entire uncorrupted text span since this requires self-attention over long sequences in the decoder. We consider two strategies to achieve this: First, instead of replacing each corrupted token with a mask token, we replace the entirety of each consecutive span of corrupted tokens with a unique mask token. Then, the target sequence becomes the concatenation of the “corrupted” spans, each prefixed by the mask token used to replace it in the input. This is the pre-training objective we use in our baseline, described in Section 3.1.4. Second, we also consider a variant where we simply drop the corrupted tokens from the input sequence completely and task the model with reconstructing the dropped tokens in order. Examples of these approaches are shown in the fifth and sixth rows of Table 3.

首先,我们考虑BERT式目标的一个简单变体,我们并不包括随机代币互换步骤。 由此产生的目标只是将输入中的符号的$15\%$替换成一个面具符号,模型被训练来重建原始的未腐蚀序列. Song et al. 2019使用了类似的遮掩目标,它被称为“MASS”,因此,我们称这个变体为“MASS风格”目标。 第二,我们想了解是否能够避免预测整个未腐蚀文本的跨度,因为这需要自己注意解码器中的长序列。 我们考虑两种策略来达到这个目的:第一,我们不是用面具标志取代每一个腐烂的标志,而是用一个独特的面具标志来取代每一个连续的腐烂标志的全部. 然后,目标序列成为“腐蚀”跨度的调和,每个跨度的前缀都是用于替换输入中的面具符号。 这是我们在第3.1.4节所述基线中采用的训练前目标。 其次,我们也考虑一种变体,我们简单地从输入序列中完全放下腐烂的符号,并让模型按照顺序重建掉的符号. 这些方法的例子见表3第五和第六行。

Objective	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
BERT-style (Devlin et al. 2018)	$82.96$	$19.17$	$\mathbf{80.65}$	$69.85$	$26.78$	$\mathbf{40.03}$	$27.41$
MASS-style (Song et al. 2019)	$82.32$	$19.16$	$80.10$	$69.28$	$26.79$	$\mathbf{39.89}$	$27.55$
$\bigstar\,$Replace corrupted spans	$83.28$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$39.82$	$\mathbf{27.65}$
Drop corrupted tokens	$\mathbf{84.44}$	$\mathbf{19.31}$	$\mathbf{80.52}$	$68.67$	$\mathbf{27.07}$	$39.76$	$\mathbf{27.82}$

Table 5: Comparison of variants of the BERT-style pre-training objective. In the first two variants, the model is trained to reconstruct the original uncorrupted text segment. In the latter two, the model only predicts the sequence of corrupted tokens.

目标	格 当我们是...	CNNDM( CNNDM ) : 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题	SQAD 状态	英格兰	定义	定义Fr	恩罗
BERT风格(Devlin等人,2018年).	$82.96$	$19.17$	$\mathbf{80.65}$	$69.85$	$26.78$	$\mathbf{40.03}$	$27.41$
MASS型(宋等人,2019年)	$82.32$	$19.16$	$80.10$	$69.28$	$26.79$	$\mathbf{39.89}$	$27.55$
第二十届 替换已损坏的跨度	$83.28$	$\mathbf{19.24}$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$39.82$	$\mathbf{27.65}$
丢弃已损坏的令牌	$\mathbf{84.44}$	$\mathbf{19.31}$	$\mathbf{80.52}$	$68.67$	$\mathbf{27.07}$	$39.76$	$\mathbf{27.82}$

表5:BERT式预训目标的变种比较. 在前两个变体中,该模型被训练来重建原始的未腐蚀文本段. 在后两者中,模型只预测腐烂的符的序列.

An empirical comparison of the original BERT-style objective to these three alternatives is shown in Table 5. We find that in our setting, all of these variants perform similarly. The only exception was that dropping corrupted tokens completely produced a small improvement in the GLUE score thanks to a significantly higher score on CoLA ($60.04$, compared to our baseline average of $53.84$, see Table 16). This may be due to the fact that CoLA involves classifying whether a given sentence is grammatically and syntactically acceptable, and being able to determine when tokens are missing is closely related to detecting acceptability. However, dropping tokens completely performed worse than replacing them with sentinel tokens on SuperGLUE. The two variants that do not require predicting the full original sequence (“replace corrupted spans” and “drop corrupted spans”) are both potentially attractive since they make the target sequences shorter and consequently make training faster. Going forward, we will explore variants where we replace corrupted spans with sentinel tokens and only predict the corrupted tokens (as in our baseline objective).

表5对最初的BERT式目标与这三种选择进行了实证比较。 我们发现,在我们所处的环境里,所有这些变体都表现得相似。 唯一的例外是,由于COLA的得分明显较高,降下腐烂的代币完全导致GLUE分数略有改善($60.04$,与我们的基线平均值$53.84$相比,见表16)。 这可能是因为CoLA涉及对某一句子在语法上和在战术上是否可接受进行分类,并且能够确定是否缺少符号与检测可接受性密切相关. 然而,投放令牌的表现完全不如SuperGLUE上的哨兵令牌取代. 不需要预测全部原始序列的两个变体(“替换腐败的跨度”和“下降腐败的跨度”)都具有潜在吸引力,因为这些变体使目标序列缩短,从而使培训更快。 展望未来,我们将探索各种变体,在其中我们用监控符代替腐败的跨度,并且只预测腐败的标志(如我们的基线目标).

3.3.3 Varying the Corruption Rate

So far, we have been corrupting 15% of the tokens, the value used in BERT (Devlin et al. 2018). Again, since our text-to-text framework differs from BERT’s, we are interested to see if a different corruption rate works better for us. We compare corruption rates of $10\%$, $15\%$, $25\%$, and $50\%$ in Table 6. Overall, we find that the corruption rate had a limited effect on the model’s performance. The only exception is that the largest corruption rate we consider ($50\%$) results in a significant degradation of performance on GLUE and SQuAD. Using a larger corruption rate also results in longer targets, which can potentially slow down training. Based on these results and the historical precedent set by BERT, we will use a corruption rate of $15\%$ going forward.

3.3.3 腐败率的变化

目前为止,我们一直在腐蚀15%的代币,BERT中使用的值(Devlin等人,2018年). 我们想知道不同的腐败率对我们是否更好。 我们比较表6中的$10\%$、$15\%$、$25\%$和$50\%$的腐败率。 总体来说,我们发现腐败率对模型的性能影响有限。 唯一的例外是,我们认为最大的腐败率($50\%$)导致GLUE和SQuAD的业绩严重下降。 使用较高的腐败率也会导致目标更长,这有可能减缓培训。 根据这些结果和BERT开创的历史先例,我们将使用$15\%$的腐败率向前迈进。

Corruption rate	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$10\%$	$\mathbf{82.82}$	$19.00$	$\mathbf{80.38}$	$69.55$	$\mathbf{26.87}$	$39.28$	$\mathbf{27.44}$
$\bigstar\,$$15\%$	$\mathbf{83.28}$	$19.24$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
$25\%$	$\mathbf{83.00}$	$\mathbf{19.54}$	$\mathbf{80.96}$	$70.48$	$\mathbf{27.04}$	$\mathbf{39.83}$	$\mathbf{27.47}$
$50\%$	$81.27$	$19.32$	$79.80$	$70.33$	$\mathbf{27.01}$	$\mathbf{39.90}$	$\mathbf{27.49}$

Table 6: Performance of the i.i.d. corruption objective with different corruption rates.

腐败率	格吕	CNNDM( CNNDM ) : 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题	SQAD 状态	英格兰	定义	定义Fr	恩罗
$10\%$	$\mathbf{82.82}$	$19.00$	$\mathbf{80.38}$	$69.55$	$\mathbf{26.87}$	$39.28$	$\mathbf{27.44}$
$\bigstar\,$$15\%$	$\mathbf{83.28}$	$19.24$	$\mathbf{80.88}$	$\mathbf{71.36}$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
$25\%$	$\mathbf{83.00}$	$\mathbf{19.54}$	$\mathbf{80.96}$	$70.48$	$\mathbf{27.04}$	$\mathbf{39.83}$	$\mathbf{27.47}$
$50\%$	$81.27$	$19.32$	$79.80$	$70.33$	$\mathbf{27.01}$	$\mathbf{39.90}$	$\mathbf{27.49}$

表6:不同腐败率的腐败目标的业绩。

3.3.4 Corrupting Spans

We now turn towards the goal of speeding up training by predicting shorter targets. The approach we have used so far makes an i.i.d. decision for each input token as to whether to corrupt it or not. When multiple consecutive tokens have been corrupted, they are treated as a “span” and a single unique mask token is used to replace the entire span. Replacing entire spans with a single token results in unlabeled text data being processed into shorter sequences. Since we are using an i.i.d. corruption strategy, it is not always the case that a significant number of corrupted tokens appear consecutively. As a result, we might obtain additional speedup by specifically corrupting spans of tokens rather than corrupting individual tokens in an i.i.d. manner. Corrupting spans was also previously considered as a pre-training objective for BERT, where it was found to improve performance (Joshi et al. 2019).

3.3.4 腐蚀西班牙人

我们现在通过预测更短的目标来实现加快培训的目标。 我们迄今采用的方法使每个输入符号都能够就是否腐败问题作出决定。 当多个连续的符号被损坏时,它们被当作一个“span”处理,并使用一个单一的独特面具符号来取代整个长度。 以单个符号取代整个跨度,导致未加标签的文本数据被处理成较短的序列。 因为我们正在采用 " i.d. " 的腐败策略,并非总能连续出现大量腐败的标志。 因此,我们可能会通过具体腐蚀象征物的跨度而获得额外的加速,而不是以某种方式腐蚀个别象征物。 腐败期以前也被认为是BERT的培训前目标,在那里发现它可以提高性能(Joshi等人,2019年).

To test this idea, we consider an objective that specifically corrupts contiguous, randomly-spaced spans of tokens. This objective can be parametrized by the proportion of tokens to be corrupted and the total number of corrupted spans. The span lengths are then chosen randomly to satisfy these specified parameters. For example, if we are processing a sequence of $500$ tokens and we have specified that $15\%$ of tokens should be corrupted and that there should be $25$ total spans, then the total number of corrupted tokens would be $500\times 0.15=75$ and the average span length would be $75/25=3$. Note that given the original sequence length and corruption rate, we can equivalently parametrize this objective either by the average span length or the total number of spans.

为了检验这个想法,我们考虑一个目标,它具体地腐蚀了毗连的,随机间隔的代币. 这一目标可以被破坏的代币比例和腐败的跨度总数所压倒. 然后随机选择长度来满足这些指定的参数. 例如,如果我们正在处理一个$500$令牌的序列,而且我们已经规定$15\%$令牌应当被损坏,并且应该有$25$总跨度,那么被损坏令牌的总数将是 $500\times 0.15=75$,平均跨度将是$75/25=3$。 请注意,鉴于最初的序列长度和腐败率,我们可以以平均长度或总长度等同地使这一目标瘫痪。

Span length	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$Baseline (i.i.d.)	$\mathbf{83.28}$	$19.24$	$80.88$	$71.36$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
$2$	$\mathbf{83.54}$	$19.39$	$\mathbf{82.09}$	$\mathbf{72.20}$	$\mathbf{26.76}$	$\mathbf{39.99}$	$\mathbf{27.63}$
$3$	$\mathbf{83.49}$	$\mathbf{19.62}$	$\mathbf{81.84}$	$\mathbf{72.53}$	$\mathbf{26.86}$	$39.65$	$\mathbf{27.62}$
$5$	$\mathbf{83.40}$	$19.24$	$\mathbf{82.05}$	$\mathbf{72.23}$	$\mathbf{26.88}$	$39.40$	$\mathbf{27.53}$
$10$	$82.85$	$19.33$	$\mathbf{81.84}$	$70.44$	$\mathbf{26.79}$	$39.49$	$\mathbf{27.69}$

Table 7: Performance of the span-corruption objective (inspired by Joshi et al. 2019) for different average span lengths. In all cases, we corrupt 15% of the original text sequence.

宽度	格吕	CNNDM( CNNDM ) : 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题	SQAD 状态	英格兰	定义	定义Fr	恩罗
$\bigstar\,$ 基线(一.d.)	$\mathbf{83.28}$	$19.24$	$80.88$	$71.36$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
$2$	$\mathbf{83.54}$	$19.39$	$\mathbf{82.09}$	$\mathbf{72.20}$	$\mathbf{26.76}$	$\mathbf{39.99}$	$\mathbf{27.63}$
$3$	$\mathbf{83.49}$	$\mathbf{19.62}$	$\mathbf{81.84}$	$\mathbf{72.53}$	$\mathbf{26.86}$	$39.65$	$\mathbf{27.62}$
$5$	$\mathbf{83.40}$	$19.24$	$\mathbf{82.05}$	$\mathbf{72.23}$	$\mathbf{26.88}$	$39.40$	$\mathbf{27.53}$
$10$	$82.85$	$19.33$	$\mathbf{81.84}$	$70.44$	$\mathbf{26.79}$	$39.49$	$\mathbf{27.69}$

表7:跨度腐败目标(Joshi等人,2019年)不同平均跨度的绩效。 在所有情况下,我们腐败了15%的原始文本序列.

We compare the span-corruption objective to the i.i.d-corruption objective in Table 7. We use a corruption rate of $15\%$ in all cases and compare using average span lengths of $2$, $3$, $5$ and $10$. Again, we find a limited difference between these objectives, though the version with an average span length of $10$ slightly underperforms the other values in some cases. We also find in particular that using an average span length of $3$ slightly (but significantly) outperforms the i.i.d. objective on most non-translation benchmarks. Fortunately, the span-corruption objective also provides some speedup during training compared to the i.i.d. noise approach because span corruption produces shorter sequences on average.

我们比较了表7中的跨腐败目标与即腐败目标。 我们在所有案件中都采用$15\%$的腐败率,并使用XX墅道1XX、$3$、$5$和$10$的平均长度进行比较。 我们再次发现,这些目标之间的差别有限,尽管平均跨度为$10$的版本在某些情况下略低于其他数值。 我们还特别发现,使用$3$的平均跨度略微(但显著)超过大多数非翻译基准的目标。 所幸的是,在培训期间,跨度腐败目标也比 " 噪音 " 方法提供了一些速度,因为跨度腐败平均产生较短的序列。

Figure 5: A flow chart of our exploration of unsupervised objectives. We first consider a few disparate approaches in Section 3.3.1 and find that a BERT-style denoising objective performs best. Then, we consider various methods for simplifying the BERT objective so that it produces shorter target sequences in Section 3.3.2. Given that replacing dropped-out spans with sentinel tokens performs well and results in short target sequences, in Section 3.3.3 we experiment with different corruption rates. Finally, we evaluate an objective that intentionally corrupts contiguous spans of tokens in Section 3.3.4.

图5:我们探索无监督目标的流程图. 我们首先在第3.3.1节中考虑了几种不同的做法,认为BERT式的去名化目标最能发挥效用。 然后,我们考虑简化BERT目标的各种方法,以便它在第3.3.2节产生较短的目标序列。 鉴于以哨兵代号取代退学的期间表现良好,结果目标序列短,在第3.3节中,我们试验了不同的腐败率。 最后,我们评价第3.3.4节中故意破坏毗连的标志范围的目标。

3.3.5 Discussion

Figure 5 shows a flow chart of the choices made during our exploration of unsupervised objectives. Overall, the most significant difference in performance we observed was that denoising objectives outperformed language modeling and deshuffling for pre-training. We did not observe a remarkable difference across the many variants of the denoising objectives we explored. However, different objectives (or parameterizations of objectives) can lead to different sequence lengths and thus different training speeds. This implies that choosing among the denoising objectives we considered here should mainly be done according to their computational cost. Our results also suggest that additional exploration of objectives similar to the ones we consider here may not lead to significant gains for the tasks and model we consider. Instead, it may be fortuitous to explore entirely different ways of leveraging unlabeled data.

3.3.5 讨论

图5显示了我们探索无监督目标期间所作的选择流程图。 人所共知, 人所共知。 我们没有看到我们所探讨的诽谤目标的许多变体之间有显著的差异。 然而,不同的目标(或目标参数化)可能导致不同的序列长度,从而导致不同的训练速度. 这意味着,在本文所考虑的取消提名目标中作出选择应主要根据其计算成本进行。 我们的结果还表明,进一步探讨与我们在这里审议的目标相似的目标,可能不会使我们审议的任务和模式取得重大进展。 相反,探索利用无标签数据的完全不同的方法可能是偶然的。

3.4 Pre-training Data set

3.4 预训练数据集

Like the unsupervised objective, the pre-training data set itself is a crucial component of the transfer learning pipeline. However, unlike objectives and benchmarks, new pre-training data sets are usually not treated as significant contributions on their own and are often not released alongside pre-trained models and code. Instead, they are typically introduced in the course of presenting a new method or model. As a result, there has been relatively little comparison of different pre-training data sets as well as a lack of a “standard” data set used for pre-training. Some recent notable exceptions (Baevski et al. 2019; Liu et al. 2019c; Yang et al. 2019) have compared pre-training on a new large (often Common Crawl-sourced) data set to using a smaller preexisting data set (often Wikipedia). To probe more deeply into the impact of the pre-training data set on performance, in this section we compare variants of our C4 data set and other potential sources of pre-training data. We release all of the C4 data set variants we consider as part of TensorFlow Datasets.[^11]

与未受监督的目标一样,培训前数据集本身也是转移学习程序的重要组成部分。 然而,与目标和基准不同的是,新的培训前数据集通常不单独作为重大贡献对待,而且往往不与培训前的模型和代码一起发布。 相反,它们通常是在提出新方法或模式的过程中提出的。 因此,不同的培训前数据集比较较少,而且缺乏用于培训前的“标准”数据集。 最近一些显著的例外(Baevski等人,2019年;Liu等人,2019年c;Yang等人,2019年)将新大型(常为常见的爬行源)数据集的预训比作使用更小的原有数据集(常为维基百科). 为了更深入地探索培训前数据集对性能的影响,我们在本节中比较了我们的C4数据集的变体和其他潜在的培训前数据来源. 我们发布了我们视为TensorFlow数据集的一部分的所有C4数据集变体. [^11]

3.4.1 Unlabeled Data Sets

In creating C4, we developed various heuristics to filter the web-extracted text from Common Crawl (see Section 2.2 for a description). We are interested in measuring whether this filtering results in improved performance on downstream tasks, in addition to comparing it to other filtering approaches and common pre-training data sets. Towards this end, we compare the performance of our baseline model after pre-training on the following data sets:

3.4.1 无标签数据集

在创建C4时,我们开发了各种heuristics来过滤从Common Crawl中提取的网络文本(描述见第2.2节). 我们有兴趣衡量这种过滤是否有助于提高 43. 下游任务的业绩,以及将其与其他过滤方法和共同的培训前数据集进行比较。 为此,我们在对下列数据集进行预先培训后,比较了基线模型的性能:

As a baseline, we first consider pre-training on our proposed unlabeled data set as described in Section 2.2.

作为基线,我们 未如第2.2节所述,首先考虑就拟议的无标签数据集进行预先培训。

To measure the effect of the heuristic filtering we used in creating C4 (deduplication, removing bad words, only retaining sentences, etc.), we also generate an alternate version of C4 that forgoes this filtering. Note that we still use langdetect to extract English text. As a result, our “unfiltered” variant still includes some filtering because langdetect sometimes assigns a low probability to non-natural English text.

为了衡量我们在创建C4(引出,删除坏字,只保留句子等)时使用的热度过滤的效果,我们还生成了C4的替代版本,以放弃这种过滤. 请注意,我们仍然使用langdetect来提取英文文本. 因此,我们的“未过滤”变体仍然包括一些过滤,因为langdetect有时将低概率分配给非自然英文文本。

Recent work has used text data extracted from news websites (Zellers et al. 2019; Baevski et al. 2019). To compare to this approach, we generate another unlabeled data set by additionally filtering C4 to only include content from one of the domains used in the “RealNews” data set (Zellers et al. 2019). Note that for ease of comparison, we retain the heuristic filtering methods used in C4; the only difference is that we have ostensibly omitted any non-news content.

最近的工作使用了从新闻网站提取的文本数据(Zellers等人,2019年;Baevski等人,2019年). 为了与这种方法进行比较,我们通过另外过滤C4生成另一个无标签数据集,只包括“RealNews”数据集中使用的一个域的内容(Zellers等人,2019年)。 请注意,为了便于比较,我们保留了C4中使用的热度过滤方法;唯一的区别是,我们表面上省略了任何非新闻内容.

Similarly, the WebText data set (Radford et al. 2019) only uses content from webpages that were submitted to the content aggregation website Reddit and received a ‘‘score’’ of at least 3. The score for a webpage submitted to Reddit is computed based on the proportion of users who endorse (upvote) or oppose (downvote) the webpage. The idea behind using the Reddit score as a quality signal is that users of the site would only upvote high-quality text content. To generate a comparable data set, we first tried removing all content from C4 that did not originate from a URL that appeared in the list prepared by the OpenWebText effort.[^12] However, this resulted in comparatively little content—only about 2 GB—because most pages never appear on Reddit. Recall that C4 was created based on a single month of Common Crawl data. To avoid using a prohibitively small data set, we therefore downloaded 12 months of data from Common Crawl from August 2018 to July 2019, applied our heuristic filtering for C4, then applied the Reddit filter. This produced a 17 GB WebText-like data set, which is of comparable size to the original 40GB WebText data set (Radford et al. 2019).

同样,WebText数据集(Radford等人,2019年)只使用提交给内容汇总网站Reddit的网页内容,并收到至少3页的QQ分数. 提交给Reddit的网页的分数是根据赞同(上)或反对(下)网页的用户比例计算的。 将Reddit分数作为质量信号背后的想法是网站的用户只会提升高质量的文本内容. 为了生成一个可比较的数据集,我们首先尝试从C4中删除所有并非来自OpenWebText努力编写的列表中出现的URL的内容. [^12] 然而,这导致内容相对较少,只有大约2GB,因为大多数页面从未出现在Reddit上. 回顾C4是根据一个月的普通爬行数据创建的. 为了避免使用令人望而却步的小数据集,我们因此从2018年8月至2019年7月从Common Crawl下载了12个月的数据,对C4应用了我们的heuristic滤波器,然后应用了Reddit滤波器. 这产生了17GB的类似WebText的数据集,其大小与原40GB的WebText数据集(拉德福德等人,2019年)相当.

The website Wikipedia consists of millions of encyclopedia articles written collaboratively. The content on the site is subject to strict quality guidelines and therefore has been used as a reliable source of clean and natural text. We use the English Wikipedia text data from TensorFlow Datasets,[^13] which omits any markup or reference sections from the articles.

网站维基百科由数百万篇合作撰写的百科全书文章组成. 网站内容遵循严格的质量准则,因此被用作清洁和自然文本的可靠来源。 我们使用来自TensorFlow Datas集的英文维基百科文本数据,[^13] 其中省略了文章中的任何标注或参考部分.

A drawback of using pre-training data from Wikipedia is that it represents only one possible domain of natural text (encyclopedia articles). To mitigate this, BERT (Devlin et al. 2018) combined data from Wikipedia with the Toronto Books Corpus (TBC) (Zhu et al. 2015). TBC contains text extracted from eBooks, which represents a different domain of natural language. BERT’s popularity has led to the Wikipedia + TBC combination being used in many subsequent works.

使用维基百科的预训数据的一个缺点是,它只代表自然文本的一个可能领域(百科全书文章). 为缓解这种情况,BERT(Devlin等人,2018年)将维基百科中的相关数据与多伦多图书公司(TBC)合并(Zhu等人,2015年). TBC包含从eBooks中提取的文本,这代表着自然语言的不同领域. BERT的流行导致维基百科+TBC组合在之后的许多作品中使用.

Data set	Size	GLUE	CNNDM	SQuAD	SGLUE	EnDe	EnFr	EnRo
$\bigstar\,$C4	745GB	$83.28$	$\mathbf{19.24}$	$80.88$	$71.36$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
C4, unfiltered	6.1TB	$81.46$	$19.14$	$78.78$	$68.04$	$26.55$	$39.34$	$27.21$
RealNews-like	35GB	$\mathbf{83.83}$	$\mathbf{19.23}$	$80.39$	$72.38$	$\mathbf{26.75}$	$\mathbf{39.90}$	$\mathbf{27.48}$
WebText-like	17GB	$\mathbf{84.03}$	$\mathbf{19.31}$	$\mathbf{81.42}$	$71.40$	$\mathbf{26.80}$	$\mathbf{39.74}$	$\mathbf{27.59}$
Wikipedia	16GB	$81.85$	$\mathbf{19.31}$	$81.29$	$68.01$	$\mathbf{26.94}$	$39.69$	$\mathbf{27.67}$
Wikipedia + TBC	20GB	$83.65$	$\mathbf{19.28}$	$\mathbf{82.08}$	$\mathbf{73.24}$	$\mathbf{26.77}$	$39.63$	$\mathbf{27.57}$

Table 8: Performance resulting from pre-training on different data sets. The first four variants are based on our new C4 data set.

数据集	大小	格吕	CNNDM( CNNDM ) : 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题 美国主题	SQAD 状态	英格兰	定义	定义Fr	恩罗
$\bigstar\,$C4	745GB (英语).	$83.28$	$\mathbf{19.24}$	$80.88$	$71.36$	$\mathbf{26.98}$	$\mathbf{39.82}$	$\mathbf{27.65}$
C4,未过滤	6.1TB (中文(简体) ).	$81.46$	$19.14$	$78.78$	$68.04$	$26.55$	$39.34$	$27.21$
真新闻类	35GB (英语).	$\mathbf{83.83}$	$\mathbf{19.23}$	$80.39$	$72.38$	$\mathbf{26.75}$	$\mathbf{39.90}$	$\mathbf{27.48}$
类似 WebText 的	17GB (英语).	$\mathbf{84.03}$	$\mathbf{19.31}$	$\mathbf{81.42}$	$71.40$	$\mathbf{26.80}$	$\mathbf{39.74}$	$\mathbf{27.59}$
维基百科	16GB (英语).	$81.85$	$\mathbf{19.31}$	$81.29$	$68.01$	$\mathbf{26.94}$	$39.69$	$\mathbf{27.67}$
维基百科+TBC	20GB (英语).	$83.65$	$\mathbf{19.28}$	$\mathbf{82.08}$	$\mathbf{73.24}$	$\mathbf{26.77}$	$39.63$	$\mathbf{27.57}$

表8:不同数据集培训前的绩效。 前四个变体是基于我们新的C4数据集.

The results achieved after pre-training on each of these data sets is shown in Table 8. A first obvious takeaway is that removing the heuristic filtering from C4 uniformly degrades performance and makes the unfiltered variant perform the worst in every task. Beyond this, we found that in some cases a pre-training data set with a more constrained domain outperformed the diverse C4 data set. For example, using the Wikipedia + TBC corpus produced a SuperGLUE score of $73.24$, beating our baseline’s score (using C4) of $71.36$. This is almost entirely attributable to a boost in performance from $25.78$ (baseline, C4) to $50.93$ (Wikipedia + TBC) on the Exact Match score for MultiRC (see Table 16). MultiRC is a reading comprehension data set whose largest source of data comes from fiction books, which is exactly the domain covered by TBC. Similarly, using the RealNews-like data set for pre-training conferred an increase from $68.16$ to $73.72$ on the Exact Match score for ReCoRD, a data set that measures reading comprehension on news articles. As a final example, using data from Wikipedia produced significant (but less dramatic) gains on SQuAD, which is a question-answering data set with passages sourced from Wikipedia. Similar observations have been made in prior work, e.g. Beltagy et al. 2019 found that pre-training BERT on text from research papers improved its performance on scientific tasks. The main lesson behind these findings is that pre-training on in-domain unlabeled data can improve performance on downstream tasks. This is unsurprising but also unsatisfying if our goal is to pre-train a model that can rapidly adapt to language tasks from arbitrary domains. Liu et al. 2019c also observed that pre-training on a more diverse data set yielded improvements on downstream tasks. This observation also motivates the parallel line of research on domain adaptation for natural language processing; for surveys of this field see e.g. Ruder 2019; Li 2012.