Language Models are Unsupervised Multitask Learners

Language Models are Unsupervised Multitask Learners

语言模型是无监督多任务学习器

Alec Radford * 1 Jeffrey Wu * 1 Rewon Child 1 David Luan 1 Dario Amodei ** 1 Ilya Sutskever ** 1

Abstract

摘要

Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on taskspeciﬁc datasets. We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText. When conditioned on a document plus questions, the answers generated by the language model reach 55 F1 on the CoQA dataset - matching or exceeding the performance of 3 out of 4 baseline systems without using the 127,000+ training examples. The capacity of the language model is essential to the success of zero-shot task transfer and increasing it improves performance in a log-linear fashion across tasks. Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting but still underﬁts WebText. Samples from the model reﬂect these improvements and contain coherent paragraphs of text. These ﬁndings suggest a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.

问答、机器翻译、阅读理解与摘要等自然语言处理任务，通常在任务专用数据集上用监督学习来做。我们表明：当语言模型在名为 WebText 的、由数百万网页构成的新数据集上训练时，即使没有任何显式监督，也开始学会这些任务。在以文档加问题为条件时，语言模型生成的答案在 CoQA 数据集上达到 55 F1——在不使用 127,000+ 训练样例的情况下，达到或超过 4 个基线系统中的 3 个。语言模型容量对零样本任务迁移的成功至关重要，增大容量会跨任务以近似对数线性的方式改进表现。我们最大的模型 GPT-2 是一个 15 亿参数的 Transformer，在零样本设置下于 8 个测试的语言建模数据集中的 7 个上达到最优，但仍对 WebText 欠拟合。模型样本反映出这些改进，并包含连贯的文本段落。这些发现表明：构建能从自然语言中自然出现的示范里学会执行任务的语言处理系统，是一条有希望的道路。

1 Introduction Machine learning systems now excel

1 引言 机器学习系统如今已能出色完成

(in expectation) at tasks they are trained for by using a combination of large datasets, high-capacity models, and supervised learning (Krizhevsky et al., 2012) (Sutskever et al., 2014) (Amodei et al., 2016). Yet these systems are brittle and sensitive to slight changes in the data distribution (Recht et al., 2018) and task speciﬁcation (Kirkpatrick et al., 2017). Current systems are better characterized as narrow experts rather than *, **Equal contribution 1OpenAI, San Francisco, California, United States. Correspondence to: Alec Radford alec@openai.com. competent generalists. We would like to move towards more general systems which can perform many tasks - eventually without the need to manually create and label a training dataset for each one.

The dominant approach to creating ML systems is to collect a dataset of training examples demonstrating correct behavior for a desired task, train a system to imitate these behaviors, and then test its performance on independent and identically distributed (IID) held-out examples. This has served well to make progress on narrow experts. But the often erratic behavior of captioning models (Lake et al., 2017), reading comprehension systems (Jia & Liang, 2017), and image classiﬁers (Alcorn et al., 2018) on the diversity and variety of possible inputs highlights some of the shortcomings of this approach.

创建 ML 系统的主流做法是：收集一组展示期望任务正确行为的训练样例，训练系统模仿这些行为，再在独立同分布（IID）的留出样例上测试其表现。这对推进狭窄专家很有效。但字幕模型（Lake 等，2017）、阅读理解系统（Jia & Liang，2017）与图像分类器（Alcorn 等，2018）在多样输入上常常表现飘忽，凸显了这一做法的一些短板。

Our suspicion is that the prevalence of single task training on single domain datasets is a major contributor to the lack of generalization observed in current systems. Progress towards robust systems with current architectures is likely to require training and measuring performance on a wide range of domains and tasks. Recently, several benchmarks have been proposed such as GLUE (Wang et al., 2018) and decaNLP (McCann et al., 2018) to begin studying this.

我们怀疑：在单一领域数据集上做单任务训练的普遍做法，是当前系统缺乏泛化的主因之一。要以现有架构推进稳健系统，很可能需要在广泛领域与任务上训练并度量表现。近来已有人提出 GLUE（Wang 等，2018）与 decaNLP（McCann 等，2018）等基准，开始研究这一点。

Multitask learning (Caruana, 1997) is a promising framework for improving general performance. However, multitask training in NLP is still nascent. Recent work reports modest performance improvements (Yogatama et al., 2019) and the two most ambitious efforts to date have trained on a total of 10 and 17 (dataset, objective) pairs respectively (McCann et al., 2018) (Bowman et al., 2018). From a meta-learning perspective, each (dataset, objective) pair is a single training example sampled from the distribution of datasets and objectives. Current ML systems need hundreds to thousands of examples to induce functions which generalize well. This suggests that multitask training many need just as many effective training pairs to realize its promise with current approaches. It will be very difﬁcult to continue to scale the creation of datasets and the design of objectives to the degree that may be required to brute force our way there with current techniques. This motivates exploring additional setups for performing multitask learning.

多任务学习（Caruana，1997）是改进总体表现的有希望框架。然而 NLP 中的多任务训练仍处于早期。近期工作报告的性能提升有限（Yogatama 等，2019）；迄今最雄心勃勃的两次努力分别只在总共 10 与 17 个（数据集，目标）对上训练（McCann 等，2018）（Bowman 等，2018）。从元学习视角看，每个（数据集，目标）对只是从数据集与目标分布中采样的单个训练样例。当前 ML 系统需要成百上千个样例才能归纳出泛化良好的函数。这意味着多任务训练可能同样需要同样多的有效训练对，才能用当前方法兑现其承诺。继续按当前技术靠蛮力扩展数据集创建与目标设计，会非常困难。这促使我们探索执行多任务学习的额外设定。

The current best performing systems on language tasks

当前在语言任务上表现最好的系统

Figure 1. Zero-shot task performance of WebText LMs as a function of model size on many NLP tasks. Reading Comprehension results are on CoQA (Reddy et al., 2018), translation on WMT-14 Fr-En (Artetxe et al., 2017), summarization on CNN and Daily Mail (See et al., 2017), and Question Answering on Natural Questions (Kwiatkowski et al., 2019). Section 3 contains detailed descriptions of each result. utilize a combination of pre-training and supervised ﬁnetuning. This approach has a long history with a trend towards more ﬂexible forms of transfer. First, word vectors were learned and used as inputs to task-speciﬁc architectures (Mikolov et al., 2013) (Collobert et al., 2011), then the contextual representations of recurrent networks were transferred (Dai & Le, 2015) (Peters et al., 2018), and recent work suggests that task-speciﬁc architectures are no longer necessary and transferring many self-attention blocks is sufﬁcient (Radford et al., 2018) (Devlin et al., 2018).

图 1. WebText 语言模型零样本任务表现随模型规模的变化，覆盖多项 NLP 任务。阅读理解结果来自 CoQA（Reddy 等，2018），翻译来自 WMT-14 Fr-En（Artetxe 等，2017），摘要来自 CNN 与 Daily Mail（See 等，2017），问答来自 Natural Questions（Kwiatkowski 等，2019）。第 3 节给出每项结果的详细描述。利用预训练与监督微调的组合。这一做法历史悠久，并趋向更灵活的迁移形式。首先，词向量被学习并用作任务专用架构的输入（Mikolov 等，2013）（Collobert 等，2011），然后循环网络的上下文表征被迁移（Dai & Le，2015）（Peters 等，2018），近期工作表明任务专用架构不再必要，迁移许多自注意力块已足够（Radford 等，2018）（Devlin 等，2018）。

These methods still require supervised training in order to perform a task. When only minimal or no supervised data is available, another line of work has demonstrated the promise of language models to perform speciﬁc tasks, such as commonsense reasoning (Schwartz et al., 2017) and sentiment analysis (Radford et al., 2017).

这些方法仍需要监督学习才能执行任务。当只有极少或没有监督数据时，另一条工作线已证明语言模型有望执行特定任务，例如常识推理（Schwartz 等，2017）与情感分析（Radford 等，2017）。

In this paper, we connect these two lines of work and continue the trend of more general methods of transfer. We demonstrate language models can perform down-stream tasks in a zero-shot setting - without any parameter or architecture modiﬁcation. We demonstrate this approach shows potential by highlighting the ability of language models to perform a wide range of tasks in a zero-shot setting. We achieve promising, competitive, and state of the art results depending on the task.

本文将这两条工作线联系起来，并延续更通用迁移方法的趋势。我们表明：语言模型可在零样本设置下执行下游任务——无需任何参数或架构修改。我们通过突出语言模型在零样本设置下执行广泛任务的能力，展示这一方法的潜力。视任务而定，我们取得有希望的、有竞争力的乃至最优的结果。

2 Approach At the core of our approach is language

2 方法

modeling. Language modeling is usually framed as unsupervised distribution estimation from a set of examples (x1, x2, ..., xn) each composed of variable length sequences of symbols (s1, s2, ..., sn). Since language has a natural sequential ordering, it is common to factorize the joint probabilities over symbols as the product of conditional probabilities (Jelinek & Mercer, 1980) (Bengio et al., 2003): n Y p(sn|s1, ..., sn-1) (1) p(x) = i=1 This approach allows for tractable sampling from and estimation of p(x) as well as any conditionals of the form p(sn-k, ..., sn|s1, ..., sn-k-1). In recent years, there have been signiﬁcant improvements in the expressiveness of models that can compute these conditional probabilities, such as self-attention architectures like the Transformer (Vaswani et al., 2017).

(x_1,x_2,...,x_n)

Learning to perform a single task can be expressed in a probabilistic framework as estimating a conditional distribution p(output|input). Since a general system should be able to perform many different tasks, even for the same input, it should condition not only on the input but also on the task to be performed. That is, it should model p(output|input, task). This has been variously formalized in multitask and meta-learning settings. Task conditioning is often implemented at an architectural level, such as the task speciﬁc encoders and decoders in (Kaiser et al., 2017) or at an algorithmic level such as the inner and outer loop optimization framework of MAML (Finn et al., 2017). But as exempliﬁed in McCann et al. (2018), language provides a ﬂexible way to specify tasks, inputs, and outputs all as a sequence of symbols. For example, a translation training example can be written as the sequence (translate to french, english text, french text). Likewise, a reading comprehension training example can be written as (answer the question, document, question, answer). McCann et al. (2018) demonstrated it was possible to train a single model, the MQAN,

p(\mathrm{output}|\mathrm{input})

to infer and perform many different tasks on examples with this type of format.

在具有这种格式的样例上推断并执行许多不同任务。

Language modeling is also able to, in principle, learn the tasks of McCann et al. (2018) without the need for explicit supervision of which symbols are the outputs to be predicted. Since the supervised objective is the the same as the unsupervised objective but only evaluated on a subset of the sequence, the global minimum of the unsupervised objective is also the global minimum of the supervised objective. In this slightly toy setting, the concerns with density estimation as a principled training objective discussed in (Sutskever et al., 2015) are side stepped. The problem instead becomes whether we are able to, in practice, optimize the unsupervised objective to convergence. Preliminary experiments conﬁrmed that sufﬁciently large language models are able to perform multitask learning in this toy-ish setup but learning is much slower than in explicitly supervised approaches.

原则上，语言建模也能学会 McCann 等（2018）的任务，而无需显式监督哪些符号是要预测的输出。因为监督目标与无监督目标相同，只是只在序列的一个子集上评估，所以无监督目标的全局最小值也是监督目标的全局最小值。在这一略带玩具色彩的设定中，（Sutskever 等，2015）讨论的、把密度估计作为有原则训练目标的担忧被绕开了。问题变成：我们在实践中能否把无监督目标优化到收敛。初步实验证实：足够大的语言模型能在这种偏玩具的设定中做多任务学习，但学习远慢于显式监督方法。

While it is a large step from the well-posed setup described above to the messiness of “language in the wild”, Weston (2016) argues, in the context of dialog, for the need to develop systems capable of learning from natural language directly and demonstrated a proof of concept - learning a QA task without a reward signal by using forward prediction of a teacher’s outputs. While dialog is an attractive approach, we worry it is overly restrictive. The internet contains a vast amount of information that is passively available without the need for interactive communication. Our speculation is that a language model with sufﬁcient capacity will begin to learn to infer and perform the tasks demonstrated in natural language sequences in order to better predict them, regardless of their method of procurement. If a language model is able to do this it will be, in effect, performing unsupervised multitask learning. We test whether this is the case by analyzing the performance of language models in a zero-shot setting on a wide variety of tasks. 2.1. Training Dataset Most prior work trained language models on a single domain of text, such as news articles (Jozefowicz et al., 2016), Wikipedia (Merity et al., 2016), or ﬁction books (Kiros et al., 2015). Our approach motivates building as large and diverse a dataset as possible in order to collect natural language demonstrations of tasks in as varied of domains and contexts as possible.

从上述表述良好的设定跨到「野外语言」的杂乱是很大一步。Weston（2016）在对话语境中论证需要发展能直接从自然语言学习的系统，并给出概念验证——通过对教师输出做前向预测、在没有奖励信号的情况下学习 QA 任务。尽管对话是有吸引力的路径，我们担心它过于受限。互联网包含海量信息，

A promising source of diverse and nearly unlimited text is web scrapes such as Common Crawl. While these archives are many orders of magnitude larger than current language modeling datasets, they have signiﬁcant data quality issues. Trinh & Le (2018) used Common Crawl in their work on commonsense reasoning but noted a large amount of documents “whose content are mostly unintelligible”. We observed similar data issues in our initial experiments with ”I’m not the cleverest man in the world, but like they say in French: Je ne suis pas un imbecile [I’m not a fool].

一个有希望的、多样且近乎无限的文本来源，是 Common Crawl 一类网页抓取。这些归档比当前语言建模数据集大许多个数量级，但存在显著的数据质量问题。Trinh & Le（2018）在常识推理工作中使用了 Common Crawl，但指出大量文档「内容大多难以理解」。我们在初始实验中也观察到类似数据问题。

In a now-deleted post from Aug. 16, Soheil Eid, Tory candidate in the riding of Joliette, wrote in French: ”Mentez mentez, il en restera toujours quelque chose,” which translates as, ”Lie lie and something will always remain.” “I hate the word ‘perfume,”’ Burr says. ‘It’s somewhat better in French: ‘parfum.’ If listened carefully at 29:55, a conversation can be heard between two guys in French: “-Comment on fait pour aller de l’autre cot´e? -Quel autre cot´e?”, which means “- How do you get to the other side? - What side?”.

在一则现已删除的、日期为 8 月 16 日的帖子中，Joliette 选区的保守党候选人 Soheil Eid 用法语写道：»Mentez mentez, il en restera toujours quelque chose,« 译为「谎言谎言，总会留下点什么。」

If this sounds like a bit of a stretch, consider this question in French: As-tu aller au cin´ema?, or Did you go to the movies?, which literally translates as Have-you to go to movies/theater? “Brevet Sans Garantie Du Gouvernement”, translated to English: “Patented without government warranty”.

若这听起来有些牵强，不妨看法语问题：As-tu aller au cinéma?，即「你去看电影了吗？」，字面直译为 Have-you to go to movies/theater?。「Brevet Sans Garantie Du Gouvernement」译为英文：»Patented without government warranty«。

Table 1. Examples of naturally occurring demonstrations of English to French and French to English translation found throughout the WebText training set.

表 1. WebText 训练集中随处可见的、英译法与法译英的自然示范例子。

Common Crawl. Trinh & Le (2018)’s best results were achieved using a small subsample of Common Crawl which included only documents most similar to their target dataset, the Winograd Schema Challenge. While this is a pragmatic approach to improve performance on a speciﬁc task, we want to avoid making assumptions about the tasks to be performed ahead of time.

Common Crawl。Trinh & Le（2018）的最好结果来自 Common Crawl 的一个小子集，其中只包含与他们目标数据集（Winograd Schema Challenge）最相似的文档。虽是提升特定任务表现的务实做法，但我们希望事先避免对将要执行的任务做假设。

Instead, we created a new web scrape which emphasizes document quality. To do this we only scraped web pages which have been curated/ﬁltered by humans. Manually ﬁltering a full web scrape would be exceptionally expensive so as a starting point, we scraped all outbound links from Reddit, a social media platform, which received at least 3 karma. This can be thought of as a heuristic indicator for whether other users found the link interesting, educational, or just funny.

相反，我们创建了一个强调文档质量的新网页抓取。为此我们只抓取经过人类策展/过滤的网页。手工过滤完整网页抓取会异常昂贵，因此作为起点，我们抓取了 Reddit（一个社交媒体平台）上所有至少获得 3 个 karma 的外链。这可视为其他用户是否觉得该链接有趣、有教育意义或只是好笑的启发式指标。

The resulting dataset, WebText, contains the text subset of these 45 million links. To extract the text from HTML responses we use a combination of the Dragnet (Peters & Lecocq, 2013) and Newspaper1 content extractors. All results presented in this paper use a preliminary version of WebText which does not include links created after Dec 2017 and which after de-duplication and some heuristic based cleaning contains slightly over 8 million documents for a total of 40 GB of text. We removed all Wikipedia documents from WebText since it is a common data source for other datasets and could complicate analysis due to over1 https://github.com/codelucas/newspaper

得到的数据集 WebText 包含这 4500 万链接中的文本子集。为从 HTML 响应中提取文本，我们结合使用 Dragnet（Peters & Lecocq，2013）与 Newspaper 内容提取器。本文报告的所有结果使用 WebText 的初步版本：不含 2017 年 12 月之后创建的链接；经去重与一些启发式清理后，略多于 800 万文档，共约 40 GB 文本。我们移除了所有维基百科内容，因为

lapping training data with test evaluation tasks. 2.2. Input Representation A general language model (LM) should be able to compute the probability of (and also generate) any string. Current large scale LMs include pre-processing steps such as lowercasing, tokenization, and out-of-vocabulary tokens which restrict the space of model-able strings. While processing Unicode strings as a sequence of UTF-8 bytes elegantly fulﬁlls this requirement as exempliﬁed in work such as Gillick et al. (2015), current byte-level LMs are not competitive with word-level LMs on large scale datasets such as the One Billion Word Benchmark (Al-Rfou et al., 2018). We observed a similar performance gap in our own attempts to train standard byte-level LMs on WebText.

它与测试评估任务的训练数据重叠。2.2. 输入表示 通用语言模型（LM）应能计算（并生成）任意字符串的概率。当前大规模 LM 包含小写化、分词与未登录词标记等预处理步骤，从而限制了可建模字符串空间。虽如 Gillick 等（2015）所示，把 Unicode 字符串处理为 UTF-8 字节序列能优雅满足该要求，但当前

Byte Pair Encoding (BPE) (Sennrich et al., 2015) is a practical middle ground between character and word level language modeling which effectively interpolates between word level inputs for frequent symbol sequences and character level inputs for infrequent symbol sequences. Despite its name, reference BPE implementations often operate on Unicode code points and not byte sequences. These implementations would require including the full space of Unicode symbols in order to model all Unicode strings. This would result in a base vocabulary of over 130,000 before any multi-symbol tokens are added. This is prohibitively large compared to the 32,000 to 64,000 token vocabularies often used with BPE. In contrast, a byte-level version of BPE only requires a base vocabulary of size 256. However, directly applying BPE to the byte sequence results in suboptimal merges due to BPE using a greedy frequency based heuristic for building the token vocabulary. We observed BPE including many versions of common words like dog since they occur in many variations such as dog. dog! dog? . This results in a sub-optimal allocation of limited vocabulary slots and model capacity. To avoid this, we prevent BPE from merging across character categories for any byte sequence. We add an exception for spaces which signiﬁcantly improves the compression efﬁciency while adding only minimal fragmentation of words across multiple vocab tokens.

字节对编码（BPE）（Sennrich 等，2015）是字符级与词级语言建模之间的实用折中，对频繁符号序列有效地插值到词级输入，对罕见符号序列插值到字符级输入。尽管名称如此，参考 BPE 实现往往在 Unicode 码点而非字节序列上操作。这些实现若要建模所有 Unicode 字符串，需要纳入完整 Unicode 符号空间。这会因 BPE 用基于贪心频率的启发式构建词表而导致次优合并。我们观察到 BPE 会为 dog、dog.、dog!、dog? 等常见词的许多变体分别建词，造成有限词表槽位与模型容量的次优分配。为避免这一点，我们禁止 BPE 跨任意字节序列的字符类别做合并。我们对空格添加例外，这显著改进压缩效率，同时只带来极少的跨多个词表词元的词碎片化。

This input representation allows us to combine the empirical beneﬁts of word-level LMs with the generality of byte-level approaches. Since our approach can assign a probability to any Unicode string, this allows us to evaluate our LMs on any dataset regardless of pre-processing, tokenization, or vocab size. 2.3. Model We use a Transformer (Vaswani et al., 2017) based architecture for our LMs. The model largely follows the details of the OpenAI GPT model (Radford et al., 2018) with a Parameters Layers dmodel 117M 12 768 345M 24 1024 762M 36 1280 1542M 48 1600

该输入表示使我们能把词级 LM 的经验收益与字节级方法的通用性结合起来。由于我们的方法可为任意 Unicode 字符串赋概率，我们可以在任意数据集上评估 LM，而不论其预处理、分词或词表大小。2.3. 模型 我们为 LM 使用基于 Transformer（Vaswani 等，2017）的架构。模型大体遵循 OpenAI GPT 模型（Radford 等，2018）的细节，参数规模为：Parameters Layers dmodel：117M 12 768；345M 24 1024；762M 36 1280；1542M 48 1600。

Table 2. Architecture hyperparameters for the 4 model sizes. few modiﬁcations. Layer normalization (Ba et al., 2016) was moved to the input of each sub-block, similar to a pre-activation residual network (He et al., 2016) and an additional layer normalization was added after the ﬁnal selfattention block. A modiﬁed initialization which accounts for the accumulation on the residual path with model depth is used. We scale the weights of residual layers at initialization by a factor of 1/ \sqrt N where N is the number of residual layers. The vocabulary is expanded to 50,257. We also increase the context size from 512 to 1024 tokens and a larger batchsize of 512 is used.

1/\sqrt{N}

Experiments We trained and benchmarked four LMs with approximately log-uniformly spaced sizes. The architectures are summarized in Table 2. The smallest model is equivalent to the original GPT, and the second smallest equivalent to the largest model from BERT (Devlin et al., 2018). Our largest model, which we call GPT-2, has over an order of magnitude more parameters than GPT. The learning rate of each model was manually tuned for the best perplexity on a 5% held-out sample of WebText. All models still underﬁt WebText and held-out perplexity has as of yet improved given more training time. 3.1. Language Modeling As an initial step towards zero-shot task transfer, we are interested in understanding how WebText LM’s perform at zero-shot domain transfer on the primary task they are trained for - language modeling. Since our model operates on a byte level and does not require lossy pre-processing or tokenization, we can evaluate it on any language model benchmark. Results on language modeling datasets are commonly reported in a quantity which is a scaled or exponentiated version of the average negative log probability per canonical prediction unit - usually a character, a byte, or a word. We evaluate the same quantity by computing the log-probability of a dataset according to a WebText LM and dividing by the number of canonical units. For many of these datasets, WebText LMs would be tested signiﬁcantly outof-distribution, having to predict aggressively standardized text, tokenization artifacts such as disconnected punctuation and contractions, shufﬂed sentences, and even the string

3 实验 我们训练并评测了四个规模近似对数均匀间隔的语言模型。架构总结见表 2。最小模型等价于原始 GPT，次小模型等价于 BERT（Devlin 等，2018）中的最大模型。我们最大的模型称为 GPT-2，参数量比 GPT 多一个数量级以上。每个模型的学习率按 WebText 留出 5% 样本上的最佳困惑度手工调节。所有模型仍对 WebText 欠拟合，留出困惑度在更多训练时间下仍持续改进。3.1. 语言建模 作为迈向零样本任务迁移的第一步，我们想了解 WebText LM 在它们所受训的主任务——语言建模——上的零样本领域迁移表现如何。由于模型在字节级运行且不需要有损预处理或分词，我们可以在任意语言建模基准上评估它。语言建模数据集上的结果通常报告为：每个规范预测单位（通常是字符、字节或词）平均负对数概率的缩放或指数化版本。我们通过按 WebText LM 计算数据集的对数概率并除以规范单位数来评估同一量。对许多这些数据集，WebText LM 会被显著地测在分布外：要预测被强标准化的文本、断开的标点与缩写等分词伪影、打乱的句子，甚至字符串

SOTA 99.8 59.23 85.7 82.3 39.14 46.54 0.99 1.08 18.3 21.8 117M 35.13 45.99 87.65 83.4 29.41 65.85 1.16 1.17 37.50 75.20 345M 15.60 55.48 92.35 87.1 22.76 47.33 1.01 1.06 26.37 55.72 762M 10.87 60.12 93.45 88.0 19.93 40.31 0.97 1.02 22.05 44.575 1542M 8.63 63.24 93.30 89.05 18.34 35.76 0.93 0.98 17.48 42.16

Table 3. Zero-shot results on many datasets. No training or ﬁne-tuning was performed for any of these results. PTB and WikiText-2 results are from (Gong et al., 2018). CBT results are from (Bajgar et al., 2016). LAMBADA accuracy result is from (Hoang et al., 2018) and LAMBADA perplexity result is from (Grave et al., 2016). Other results are from (Dai et al., 2019). which is extremely rare in WebText - occurring only 26 times in 40 billion bytes. We report our main results in Table 3 using invertible de-tokenizers which remove as many of these tokenization / pre-processing artifacts as possible. Since these de-tokenizers are invertible, we can still calculate the log probability of a dataset and they can be thought of as a simple form of domain adaptation. We observe gains of 2.5 to 5 perplexity for GPT-2 with these de-tokenizers.

表 3. 多个数据集上的零样本结果。这些结果均未做任何训练或微调。PTB 与 WikiText-2 结果来自（Gong 等，2018）。CBT 结果来自（Bajgar 等，2016）。LAMBADA 精度结果来自（Hoang 等，2018），LAMBADA 困惑度结果来自（Grave 等，2016）。其他结果来自（Dai 等，2019）。 在 WebText 中极罕见——在 400 亿字节中只出现 26 次。我们用可逆去分词器报告表 3 的主要结果，以尽可能去除这些分词/预处理伪影。由于这些去分词器可逆，我们仍能计算数据集的对数概率，可将其视为一种简单的领域自适应。我们观察到 GPT-2 使用这些去分词器可获得 2.5 到 5 个困惑度的收益。

WebText LMs transfer well across domains and datasets, improving the state of the art on 7 out of the 8 datasets in a zero-shot setting. Large improvements are noticed on small datasets such as Penn Treebank and WikiText-2 which have only 1 to 2 million training tokens. Large improvements are also noticed on datasets created to measure long-term dependencies like LAMBADA (Paperno et al., 2016) and the Children’s Book Test (Hill et al., 2015). Our model is still signiﬁcantly worse than prior work on the One Billion Word Benchmark (Chelba et al., 2013). This is likely due to a combination of it being both the largest dataset and having some of the most destructive pre-processing - 1BW’s sentence level shufﬂing removes all long-range structure. 3.2. Children’s Book Test

WebText LM 跨领域与数据集迁移良好，在零样本设置下于 8 个数据集中的 7 个上改进了最优结果。在只有 100 万到 200 万训练词元的小数据集（如 Penn Treebank 与 WikiText-2）上改进显著。在为度量长期依赖而创建的数据集（如 LAMBADA（Paperno 等，2016）与 Children’s Book Test（Hill 等，2015））上改进同样显著。我们的模型在 One Billion Word Benchmark（Chelba 等，2013）上仍显著差于先前工作。这很可能因为它既是最大的数据集，又有一些最具破坏性的预处理——1BW 的句子级打乱去掉了所有长程结构。3.2. Children’s Book Test

Figure 2. Performance on the Children’s Book Test as a function of model capacity. Human performance are from Bajgar et al. (2016), instead of the much lower estimates from the original paper.

图 2. Children’s Book Test 表现随模型容量的变化。人类表现来自 Bajgar 等（2016），而非原论文中低得多的估计。

Children’s Book Test（CBT）（Hill 等，2015）旨在检验 LM 在不同词类上的表现：命名实体、名词、动词与介词。CBT 不报告困惑度，而是报告在自动构建的完形填空测试上的精度：任务是从省略词的 10 个可能选项中预测正确者。遵循原论文引入的 LM 方法，我们按 LM 计算每个选项及其余句子在该选项条件下的概率，并预测概率最高者。如图 2 所示，表现随模型规模稳步提升，并在该测试上缩小了与人类表现的大部分差距。数据重叠分析显示 CBT 测试集中的一本书——Rudyard Kipling 的 The Jungle Book ——在 WebText 中，因此我们在没有显著重叠的验证集上报告结果。GPT-2 在普通名词上达到 93.3%、在命名实体上达到 89.1% 的新最优。对 CBT 应用了去分词器以去除 PTB 风格分词伪影。3.3. LAMBADA LAMBADA 数据集（Paperno 等，2016）测试系统建模文本长程依赖的能力。任务是预测句子的最后一个词，而这些句子对人类而言至少需要 50 个词元的上下文才能成功预测。GPT-2 把最优从 99.8（Grave 等，2016）改进到 8.6 困惑度，并把 LM 在该测试上的精度从 19%（Dehghani 等，2018）提高到 52.66%。调查 GPT-2 的错误发现：多数预测是句子的有效延续，但不是有效的最终词。这表明 LM 没有利用「该词必须是句末」这一额外有用约束。加入停用词过滤作为近似后，精度进一步提高到 63.24%，把该任务的总体最优再提升 4%。先前最优（Hoang 等，2018）使用不同的受限预测设置，把模型输出约束为只出现在上下文中的词。对 GPT-2，这一限制有害而非有益，

since 19% of answers are not in context. We use a version of the dataset without preprocessing. 3.4. Winograd Schema Challenge

因为 19% 的答案不在上下文中。我们使用未经预处理的数据集版本。3.4. Winograd Schema Challenge

Figure 3. Performance on the Winograd Schema Challenge as a function of model capacity.

图 3. Winograd Schema Challenge 表现随模型容量的变化。

The Winograd Schema challenge (Levesque et al., 2012) was constructed to measure the capability of a system to perform commonsense reasoning by measuring its ability to resolve ambiguities in text. Recently Trinh & Le (2018) demonstrated signiﬁcant progress on this challenge using LMs, by predicting the resolution of the ambiguity with higher probability. We follow their problem formulation and visualize the performance of our models with both full and partial scoring techniques in Figure 3. GPT-2 improves state of the art accuracy by 7%, achieving 70.70%. The dataset is quite small with only 273 examples so we recommend reading Trichelair et al. (2018) to help contextualize this result. 3.5. Reading Comprehension The Conversation Question Answering dataset (CoQA) Reddy et al. (2018) consists of documents from 7 different domains paired with natural language dialogues between a question asker and a question answerer about the document. CoQA tests reading comprehension capabilities and also the ability of models to answer questions that depend on conversation history (such as “Why?”).

Winograd Schema Challenge（Levesque 等，2012）旨在通过度量系统消解文本歧义的能力，衡量其做常识推理的能力。近来 Trinh & Le（2018）用 LM 在该挑战上取得显著进展：用更高概率预测歧义的消解。我们遵循他们的问题形式，并在图 3 中可视化完整与部分打分技术下我们模型的表现。GPT-2 把最优精度提高 7%，达到 70.70%。数据集很小，只有 273 个样例，因此建议阅读 Trichelair 等（2018）以帮助理解该结果的语境。3.5. 阅读理解 Conversation Question Answering 数据集（CoQA）Reddy 等（2018）由 7 个不同领域的文档组成，并配有提问者与回答者关于文档的自然语言对话。CoQA 测试阅读理解能力，以及模型回答依赖对话历史的问题（例如「为什么？」）的能力。

Greedy decoding from GPT-2 when conditioned on a document, the history of the associated conversation, and a ﬁnal token A: achieves 55 F1 on the development set. This matches or exceeds the performance of 3 out of 4 baseline systems without using the 127,000+ manually collected question answer pairs those baselines were trained on. The supervised SOTA, a BERT based system (Devlin et al., R-1 R-2 R-L R-AVG Bottom-Up Sum 41.22 18.68 38.34 32.75 Lede-3 40.38 17.66 36.62 31.55 Seq2Seq + Attn 31.33 11.81 28.83 23.99 GPT-2 TL;DR: 29.34 8.27 26.58 21.40 Random-3 28.78 8.63 25.52 20.98 GPT-2 no hint 21.58 4.03 19.47 15.03

当以文档、相关对话历史以及最终词元 A: 为条件时，从 GPT-2 做贪心解码在开发集上达到 55 F1。这在不使用那些基线所训练的 127,000+ 人工收集问答对的情况下，达到或超过 4 个基线系统中的 3 个。监督最优是基于 BERT 的系统（Devlin 等， R-1 R-2 R-L R-AVG Bottom-Up Sum 41.22 18.68 38.34 32.75 Lede-3 40.38 17.66 36.62 31.55 Seq2Seq + Attn 31.33 11.81 28.83 23.99 GPT-2 TL;DR: 29.34 8.27 26.58 21.40 Random-3 28.78 8.63 25.52 20.98 GPT-2 no hint 21.58 4.03 19.47 15.03

	R-1	R-2	R-L	R-AVG
Bottom-Up Sum	41.22	18.68	38.34	32.75
Lede-3	40.38	17.66	36.62	31.55
Seq2Seq + Attn	31.33	11.81	28.83	23.99
GPT-2 TL;DR:	29.34	8.27	26.58	21.40
Random-3	28.78	8.63	25.52	20.98
GPT-2 no hint	21.58	4.03	19.47	15.03

表 4. 在 CNN 与 Daily Mail 数据集上用 ROUGE F1 度量的摘要表现。

表 5. GPT-2 在 Natural Questions 开发集上生成的 30 个最高置信答案。

Dataset train 2.67% 0.66% 7.50% 2.34% 9.09% 13.19% WebText train 0.88% 1.63% 6.31% 3.94% 2.42% 3.75%

Table 6. Percentage of test set 8 grams overlapping with training sets.

表 6. 测试集 8-gram 与训练集重叠的百分比。

Generalization vs Memorization Recent work in computer vision has shown that common image datasets contain a non-trivial amount of near-duplicate images. For instance CIFAR-10 has 3.3% overlap between train and test images (Barz & Denzler, 2019). This results in an over-reporting of the generalization performance of machine learning systems. As the size of datasets increases this issue becomes increasingly likely which suggests a similar phenomena could be happening with WebText. Therefore it is important to analyze how much test data also shows up in the training data.

4 泛化与记忆 计算机视觉领域的近期工作表明，常见图像分类数据集存在大量重叠。语言任务是否也存在类似问题？WebText 与评估任务之间的重叠可能夸大零样本表现。

To study this we created Bloom ﬁlters containing 8-grams of WebText training set tokens. To improve recall, strings were normalized to contain only lower-cased alphanumeric words with a single space as a delimiter. The Bloom ﬁlters were constructed such that the false positive rate is upper bounded by 1 108 . We further veriﬁed the low false positive rate by generating 1M strings, of which zero were found by the ﬁlter.

为研究这一点，我们创建了包含 WebText 训练集词元 8-gram 的 Bloom 过滤器。为改进召回，我们对文本做规范化处理后再计算 8-gram。

These Bloom ﬁlters let us calculate, given a dataset, the percentage of 8-grams from that dataset that are also found in the WebText training set. Table 6 shows this overlap analysis for the test sets of common LM benchmarks. Common LM datasets’ test sets have between 1-6% overlap with WebText train, with an average of overlap of 3.2%. Somewhat surprisingly, many datasets have larger overlaps with their own training splits, with an average of 5.9% overlap.

这些 Bloom 过滤器让我们能够计算：给定一个数据集，其中来自该数据集的 8-gram 有多大比例也出现在 WebText 训练集中。

Our approach optimizes for recall, and while manual inspection of the overlaps shows many common phrases, there are many longer matches that are due to duplicated data. This is not unique to WebText. For instance, we discovered that the test set of WikiText-103 has an article which is also in the training dataset. Since there are only 60 articles in the test set there is at least an overlap of 1.6%.4 Potentially more worryingly, 1BW has an overlap of nearly 13.2% with its own training set according to our procedure.

我们的方法优化召回；虽然对重叠的人工检查显示许多常见短语，但也可能包含噪声匹配。

For the Winograd Schema Challenge, we found only 10 schemata which had any 8-gram overlaps with the WebText training set. Of these, 2 were spurious matches. Of the remaining 8, only 1 schema appeared in any contexts that 4A signiﬁcant portion of additional overlap is due to editors reusing some paragraphs across multiple articles with a shared theme such as various battles in the Korean War.

对 Winograd Schema Challenge，我们只找到 10 个与训练数据有任何 8-gram 重叠的图式。去掉这些后，表现几乎不变。

PTB WikiText-2 enwik8 text8 Wikitext-103 1BW gave away the answer.

For CoQA, about 15% of documents in the news domain are already in WebText and the model performs about 3 F1 better on these. CoQA’s development set metric reports the average performance over 5 different domains and we measure a gain of about 0.5-1.0 F1 due to overlap across the various domains. However, no actual training questions or answers are in WebText since CoQA was released after the cutoff date for links in WebText.

对 CoQA，新闻领域中约 15% 的文档已在 WebText 中，模型在这些文档上的表现约高 5 F1。

On LAMBADA, the average overlap is 1.2%. GPT-2 performs about 2 perplexity better on examples with greater than 15% overlap. Recalculating metrics when excluding all examples with any overlap shifts results from 8.6 to 8.7 perplexity and reduces accuracy from 63.2% to 62.9%. This very small change in overall results is likely due to only 1 in 200 examples having signiﬁcant overlap.

在 LAMBADA 上，平均重叠为 1.2%。GPT-2 在有较大重叠的样例上困惑度约好 2 点。

Overall, our analysis suggests that data overlap between WebText training data and speciﬁc evaluation datasets provides a small but consistent beneﬁt to reported results. However, for most datasets we do not notice signiﬁcantly larger overlaps than those already existing between standard training and test sets, as Table 6 highlights.

总体而言，我们的分析表明：WebText 训练数据与特定评估集之间的数据重叠，无法解释 GPT-2 在这些任务上的大部分收益。

Understanding and quantifying how highly similar text impacts performance is an important research question. Better de-duplication techniques such as scalable fuzzy matching could also help better answer these questions. For now, we recommend the use of n-gram overlap based de-duplication as an important veriﬁcation step and sanity check during the creation of training and test splits for new NLP datasets.

理解并量化高度相似文本如何影响表现，是一项重要研究问题，超出本文范围。

Another potential way of determining whether the performance of WebText LMs is attributable to memorization is inspecting their performance on their own held-out set. As shown in Figure 4, performance on both the training and test sets of WebText are similar and improve together as model size is increased. This suggests even GPT-2 is still underﬁtting on WebText in many ways.

另一种判断 WebText LM 的表现是否可归因于记忆的潜在方式，是看模型在明显分布外的上下文上的条件生成。

GPT-2 is also able to write news articles about the discovery of talking unicorns. An example is provided in Table 13.

GPT-2 也能撰写关于发现会说话独角兽的新闻文章。附录给出了一个例子。

5 Related Work A

5 相关工作

signiﬁcant portion of this work measured the performance of larger language models trained on larger datasets. This

本工作的相当一部分度量了在更大数据集上训练的更大语言模型的表现。图 4 显示 WebText 上训练的 LM 表现随模型规模的变化。

Figure 4. The performance of LMs trained on WebText as a function of model size. is similar to the work of Jozefowicz et al. (2016) which scaled RNN based language models on the 1 Billion Word Benchmark. Bajgar et al. (2016) also previously improved results on the Children’s Book Test by creating a much larger training dataset out of Project Gutenberg to supplement the standard training dataset. Hestness et al. (2017) conducted a thorough analysis of how the performance of various deep learning models changes as a function of both model capacity and dataset size. Our experiments, while much noisier across tasks, suggest similar trends hold for sub-tasks of an objective and continue into the 1B+ parameter regime.

图 4. 在 WebText 上训练的 LM 表现随模型规模的变化。与先前观察到的对数线性趋势类似。

Interesting learned functionality in generative models has been documented before such as the cells in an RNN language model performing line-width tracking and quote/comment detection Karpathy et al. (2015). More inspirational to our work was the observation of Liu et al. (2018) that a model trained to generate Wikipedia articles also learned to translate names between languages.

生成模型中有趣的习得功能此前已被记录，例如字符级 RNN 中追踪括号与引号的单元。

Previous work has explored alternative approaches to ﬁltering and constructing a large text corpus of web pages, such as the iWeb Corpus (Davies, 2018).

先前工作探索了过滤与构建大规模文本语料的替代方法，包括基于相似性的子采样。

There has been extensive work on pre-training methods for language tasks. In addition to those mentioned in the introduction, GloVe (Pennington et al., 2014) scaled word vector representation learning to all of Common Crawl. An inﬂuential early work on deep representation learning for text was Skip-thought Vectors (Kiros et al., 2015). McCann et al. (2017) explored the use of representations derived from machine translation models and Howard & Ruder (2018) improved the RNN based ﬁne-tuning approaches of (Dai & Le, 2015). (Conneau et al., 2017a) studied the transfer performance of representations learned by natural language inference models and (Subramanian et al., 2018) explored large-scale multitask training. (Ramachandran et al., 2016) demonstrated that seq2seq models beneﬁt from being initialized with pre-trained language models as encoders and decoders. More recent work has shown that LM pre-training is helpful when ﬁne-tuned for difﬁcult generation tasks like chit-chat dialog and dialog based question answering systems as well (Wolf et al., 2019) (Dinan et al., 2018).

语言任务的预训练方法已有大量工作。除引言中提到的之外，还有多种监督与半监督迁移设定。

Discussion Much research has been dedicated to learning (Hill et al., 2016), understanding (Levy & Goldberg, 2014), and critically evaluating (Wieting & Kiela, 2019) the representations of both supervised and unsupervised pre-training methods. Our results suggest that unsupervised task learning is an additional promising area of research to explore. These ﬁndings potentially help explain the widespread success of pre-training techniques for down-stream NLP tasks as we show that, in the limit, one of these pre-training techniques begins to learn to perform tasks directly without the need for supervised adaption or modiﬁcation.

6 讨论 大量研究致力于学习（Hill 等，2016）、理解（Levy & Goldberg，2014）与改进语言表征。我们的结果表明：在足够大且多样的文本上训练的单向语言模型，已能以零样本方式表现出许多任务行为。

On reading comprehension the performance of GPT-2 is competitive with supervised baselines in a zero-shot setting. However, on other tasks such as summarization, while it is qualitatively performing the task, its performance is still only rudimentary according to quantitative metrics. While suggestive as a research result, in terms of practical applications, the zero-shot performance of GPT-2 is still far from use-able.

在阅读理解上，GPT-2 的零样本表现可与若干监督基线竞争，但仍显著落后于最强的监督系统。

We have studied the zero-shot performance of WebText LMs on many canonical NLP tasks, but there are many additional tasks that could be evaluated. There are undoubtedly many practical tasks where the performance of GPT-2 is still no better than random. Even on common tasks that we evaluated on, such as question answering and translation, language models only begin to outperform trivial baselines when they have sufﬁcient capacity.

我们研究了 WebText LM 在许多经典 NLP 任务上的零样本表现，但仍有许多任务与设定未覆盖。

While zero-shot performance establishes a baseline of the potential performance of GPT-2 on many tasks, it is not clear where the ceiling is with ﬁnetuning. On some tasks, GPT-2’s fully abstractive output is a signiﬁcant departure from the extractive pointer network (Vinyals et al., 2015) based outputs which are currently state of the art on many question answering and reading comprehension datasets. Given the prior success of ﬁne-tuning GPT, we plan to investigate ﬁne-tuning on benchmarks such as decaNLP and GLUE, especially since it is unclear whether the additional

尽管零样本表现确立了 GPT-2 在许多任务上的潜力基线，微调、提示设计与少样本学习可能会进一步释放能力。

training data and capacity of GPT-2 is sufﬁcient to overcome the inefﬁciencies of uni-directional representations demonstrated by BERT (Devlin et al., 2018).

GPT-2 的训练数据与容量足以克服单向表示的低效，并在多项基准上达到强零样本结果。

Conclusion When a large language model is trained on a sufﬁciently large and diverse dataset it is able to perform well across many domains and datasets. GPT-2 zero-shots to state of the art performance on 7 out of 8 tested language modeling datasets. The diversity of tasks the model is able to perform in a zero-shot setting suggests that high-capacity models trained to maximize the likelihood of a sufﬁciently varied text corpus begin to learn how to perform a surprising amount of tasks without the need for explicit supervision.5

7 结论 当在足够大且多样的数据集上训练大型语言模型时，它能够在许多领域与数据集上习得执行任务的能力。GPT-2 展示了这一趋势的早期证据，但仍对 WebText 欠拟合，表明继续扩大数据与模型仍有空间。

Acknowledgements Thanks to everyone who wrote the text, shared the links, and upvoted the content in WebText. Many millions of people were involved in creating the data that GPT-2 was trained on. Also thanks to all the Googlers who helped us with training infrastructure, including Zak Stone, JS Riehl, Jonathan Hseu, Russell Power, Youlong Cheng, Noam Shazeer, Solomon Boulos, Michael Banﬁeld, Aman Gupta, Daniel Sohn, and many more. Finally thanks to the people who gave feedback on drafts of the paper: Jacob Steinhardt, Sam Bowman, Geoffrey Irving, and Madison May.

References

Al-Rfou, R., Choe, D., Constant, N., Guo, M., and Jones, L.

Character-level language modeling with deeper self-attention. arXiv preprint arXiv:1808.04444, 2018.

Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q., Chen, G., et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In International Conference on Machine Learning, pp. 173-182, 2016.

Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137-1155, 2003.

Caruana, R. Multitask learning. Machine learning, 28(1):41-75, 1997.

Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493- 2537, 2011.

Conneau, A., Lample, G., Ranzato, M., Denoyer, L., and J´egou,

Alberti, C., Lee, K., and Collins, M. A bert baseline for the natural questions. arXiv preprint arXiv:1901.08634, 2019. Alcorn, M. A., Li, Q., Gong, Z., Wang, C., Mai, L., Ku, W.-S., and Nguyen, A. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. arXiv preprint arXiv:1811.11553, 2018. Artetxe, M., Labaka, G., Agirre, E., and Cho, K. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041, 2017. Artetxe, M., Labaka, G., and Agirre, E. An effective approach to unsupervised machine translation. arXiv preprint arXiv:1902.01313, 2019. 5Preliminary code for downloading and using the small model is available at https://github.com/openai/gpt-2 Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. Bajgar, O., Kadlec, R., and Kleindienst, J. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:1610.00956, 2016. Barz, B. and Denzler, J. Do we train on test data? purging cifar of near-duplicates. arXiv preprint arXiv:1902.00423, 2019. Bowman, S. R., Pavlick, E., Grave, E., Van Durme, B., Wang, A., Hula, J., Xia, P., Pappagari, R., McCoy, R. T., Patel, R., et al. Looking for elmo’s friends: Sentence-level pretraining beyond language modeling. arXiv preprint arXiv:1812.10860, 2018. Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364, 2017a.

H. Word translation without parallel data. arXiv preprint arXiv:1710.04087, 2017

Dai, A. M. and Le, Q. V. Semi-supervised sequence learning. In Advances in neural information processing systems, pp. 3079- 3087, 2015.

Dai, Z., Yang, Z., Yang, Y., Cohen, W. W., Carbonell, J., Le,

Q. V., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a

Davies, M. The 14 billion word iweb corpus. https://corpus.byu.edu/iWeb/, 2018.

Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., and Weston,

Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819, 2018. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

J. Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint

Gong, C., He, D., Tan, X., Qin, T., Wang, L., and Liu, T.-Y. Frage: frequency-agnostic word representation. In Advances in Neural Information Processing Systems, pp. 1341-1352, 2018.

He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.

Howard, J. and Ruder, S. Universal language model ﬁne-tuning for text classiﬁcation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 328-339, 2018.

Jelinek, F. and Mercer, R. L. Interpolated estimation of markov source parameters from sparse data. In Proceedings of the Workshop on Pattern Recognition in Practice, Amsterdam, The Netherlands: North-Holland, May., 1980.

Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., and Wu,

Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833, 2018. Finn, C., Abbeel, P., and Levine, S. Model-agnostic metalearning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400, 2017. Gehrmann, S., Deng, Y., and Rush, A. M. Bottom-up abstractive summarization. arXiv preprint arXiv:1808.10792, 2018. Gillick, D., Brunk, C., Vinyals, O., and Subramanya, A. Multilingual language processing from bytes. arXiv preprint arXiv:1512.00103, 2015. Grave, E., Joulin, A., and Usunier, N. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426, 2016. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M., Ali, M., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. Hill, F., Bordes, A., Chopra, S., and Weston, J. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301, 2015. Hill, F., Cho, K., and Korhonen, A. Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483, 2016. Hoang, L., Wiseman, S., and Rush, A. M. Entity tracking improves cloze-style reading comprehension. arXiv preprint arXiv:1810.02891, 2018. Jia, R. and Liang, P. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328, 2017.

Y. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2

Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., GrabskaBarwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, pp. 201611835, 2017.

Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. Skip-thought vectors. In Advances in neural information processing systems, pp. 3294-3302, 2015.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classiﬁcation with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.

Kwiatkowski, T., Palomaki, J., Rhinehart, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., et al. Natural questions: a benchmark for question answering research. 2019.

Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J.

Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017.

Levesque, H., Davis, E., and Morgenstern, L. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.

Levy, O. and Goldberg, Y. Neural word embedding as implicit matrix factorization. In Advances in neural information processing systems, pp. 2177-2185, 2014.

McCann, B., Bradbury, J., Xiong, C., and Socher, R. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pp. 6294-6305, 2017.

McCann, B., Keskar, N. S., Xiong, C., and Socher, R. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730, 2018.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean,

Kaiser, L., Gomez, A. N., Shazeer, N., Vaswani, A., Parmar, N., Jones, L., and Uszkoreit, J. One model to learn them all. arXiv preprint arXiv:1706.05137, 2017. Karpathy, A., Johnson, J., and Fei-Fei, L. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078, 2015. Lample, G., Conneau, A., Denoyer, L., and Ranzato, M. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043, 2017. Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198, 2018. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.

J. Distributed representations of words and phrases and their compositionality.

Pennington, J., Socher, R., and Manning, C. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.

Peters, M. E. and Lecocq, D. Content extraction using diverse feature sets. In Proceedings of the 22nd International Conference on World Wide Web, pp. 89-90. ACM, 2013.

Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I.

Improving language understanding by generative pre-training. 2018.

Schwartz, R., Sap, M., Konstas, I., Zilles, L., Choi, Y., and Smith,

Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B., et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023, 2016. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern´andez, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. Radford, A., Jozefowicz, R., and Sutskever, I. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444, 2017. Ramachandran, P., Liu, P. J., and Le, Q. V. Unsupervised pretraining for sequence to sequence learning. arXiv preprint arXiv:1611.02683, 2016. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do cifar-10 classiﬁers generalize to cifar-10? arXiv preprint arXiv:1806.00451, 2018. Reddy, S., Chen, D., and Manning, C. D. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042, 2018.

N. A. Story cloze task: Uw nlp system. In Proceedings of the 2nd Workshop on Lin

Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.

Vinyals, O., Fortunato, M., and Jaitly, N. Pointer networks. In Advances in Neural Information Processing Systems, pp. 2692- 2700, 2015.

Weston, J. E. Dialog-based language learning. In Advances in Neural Information Processing Systems, pp. 829-837, 2016.

See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368, 2017. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. Subramanian, S., Trischler, A., Bengio, Y., and Pal, C. J. Learning general purpose distributed sentence representations via large scale multi-task learning. arXiv preprint arXiv:1804.00079, 2018. Sutskever, I., Jozefowicz, R., Gregor, K., Rezende, D., Lillicrap, T., and Vinyals, O. Towards principled unsupervised learning. arXiv preprint arXiv:1511.06440, 2015. Trichelair, P., Emami, A., Cheung, J. C. K., Trischler, A., Suleman, K., and Diaz, F. On the evaluation of common-sense reasoning in natural language understanding. arXiv preprint arXiv:1811.01778, 2018. Trinh, T. H. and Le, Q. V. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847, 2018. Vinyals, O. and Le, Q. A neural conversational model. arXiv preprint arXiv:1506.05869, 2015. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. Wieting, J. and Kiela, D. No training required: Exploring random encoders for sentence classiﬁcation. arXiv preprint arXiv:1901.10444, 2019. Wolf, T., Sanh, V., Chaumond, J., and Delangue, C. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149, 2019.

Yogatama, D., d’Autume, C. d. M., Connor, J., Kocisky, T., Chrzanowski, M., Kong, L., Lazaridou, A., Ling, W., Yu, L., Dyer, C., et al. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373, 2019.

8 Appendix

8 附录

A: Samples 8.1. Model capacity To complement the reported perplexity gains of bigger LMs on WebText show in Figure 4, Tables 7 through 11 show side-by-side completions of the smallest WebText LM and GPT-2 on random unseen WebText test set articles. 8.2. Text Memorization We observe some memorizing behavior in GPT-2 on longer strings that are repeated many times in the dataset such as famous quotes or speeches. For example, when conditioned on the ﬁrst sentence and a half of the Gettysburg Address (which occurs approximately 40 times throughout WebText), an argmax decode from GPT-2 recovers the speech. Even when sampling without truncation, we ﬁnd that the model copies the speech for awhile before drifting, albeit in a similar style. It typically drifts within 100-200 tokens, and displays widening diversity once it drifts.

为补充更大 LM 在 WebText 上报告的困惑度收益，我们展示了条件生成样本。

To quantify how often exact memorization shows up in samples, we generated samples from GPT-2 conditioned on WebText test set articles and compared the overlap rates of GPT-2’s generations to the overlap rates of the ground-truth completions. The results of this analysis are shown below and suggest that GPT-2 repeats text from the training set less often then the baseline rate of held-out articles.

为量化精确记忆在样本中出现的频率，我们从以 WebText 测试集上下文为条件的 GPT-2 生成样本。

Figure 5. CDF of percentage 8-gram overlap with WebText training set, for both WebText test set and samples (conditioned on WebText test set, with top-k truncated random sampling with k = 40). Most samples have less than 1% overlap, including over 30% of samples with no overlap, whereas the median for test set is 2.6% overlap. 8.3. Diversity

图 5. WebText 测试集与模型样本相对 WebText 训练集的 8-gram 重叠百分比 CDF。

Table 12 shows multiple completions of the same random WebText test set context, showing the diversity of completions with standard sampling settings. 8.4. Robustness

表 12. 由同一上下文（来自 WebText 测试集）生成的、未经精选的 GPT-2 补全。

Table 13 shows the previously mentioned talking unicorns news article. We ﬁnd the model to be capable of handling out of distribution contexts, but the quality of these samples is generally lower.

表 13. GPT-2 在分布外上下文上的条件生成。从 10 个样本中精选。

Context (WebText test) Outﬁt 1: Typical This pairing was the ﬁrst outﬁt I thought of when I bought the shoes. It’s like a summer version of this Jake Grantham outﬁt; in fact, my shoes are close to the colors of his Nike Racers! Instead of a heavy Harris Tweed jacket and denim shirt, I’m wearing a cotton DB jacket and and a linen shirt. Both fabrics (in these colors) are an absolute must for summer, as they go with both dark and and light pants! As you can see, they pair wonderfully with the dark jeans and shoes. It’s a pseudo menswear/prep outﬁt.

Overall, this is a very casual outﬁt which is why I paired my sneakers with it. I’m not about wearing a full wool suit with sneakers (as GQ shows a lot) but I’m deﬁnitely open to keeping things casual, like this cotton DB. Casual fabrics are key to pulling off your sneakers in a dressed down menswear outﬁt. I’d even suggest to wear these sneakers with a khaki chino suit or a white linen suit. Just be sure to ditch the tie or wear a tee or polo; wearing a tie with sneakers is a bit too much Small model completion .

Outﬁt 2: High End This outﬁt was just in the eye of the storm. It’s a hybrid of what ASICS does with streetwear (like J. Crew) and what J. Crew does with casual pieces that are a bit more traditional. Again, as I mentioned before, it’s a low budget item and they can be very expensive. These will deﬁnitely be an issue if you’re trying to work as a casual sneakermaker, and that’s OK but don’t hesitate if you want to try the lower price versions of their outﬁts!

This version features a very low cut-cut neckline (which is not a bad thing), slightly longer legs, and a pair of long legs to keep things nice. My preferred color of color these items are blue-gray (which is my ﬁrst colors, as they are a light white), or black (which is my favorite of my designs). As always, my pick of colors from this store have nothing to do with that other than the design. For a high end, I would recommend this to everyone!

Outﬁt 3: High Light This was a ”high light” look while I had a chance to get some color. I think the color is based on

Table 7. Random unseen contexts (top), and non-cherry-picked completions from both the smallest (left) and largest (right) models.

表 7. 随机未见过的上下文（上），以及最小模型（左）与 GPT-2（右）的非精选补全。

Contexts are 768 tokens, with approximately 256 tokens worth of paragraphs shown. Completions are 256 tokens and fully shown. Top-k random sampling with k = 40 was used for generation.

上下文为 768 个词元，图中约展示 256 个词元的段落。补全为 256 个词元。

GPT-2 completion for me!

Outﬁt 2: Casual Jacket I know that some people might be opposed to wearing sneakers in a turtleneck, but I wanted to be true to myself, so I went with a slim ﬁtting turtleneck in a color more similar to my favorite color of the day. Overall, it’s a classic turtleneck, with nothing too ﬂashy. I did swap my shoes for these ASICS Tiger Killshots and I wish I hadn’t did that. The shoe itself is very comfortable, however, I found that wearing them in these sneakers made my feet look a bit larger. I’d wear them with denim, chinos, whatever, but would never recommend wearing them alone. There are just too many variables involved.

Outﬁt 3: Dress This is the ﬁnal outﬁt I wore the ASICS Tiger Killshots. This version of the ASICS is a much more modern and streamlined version. The shoe has the most modern and minimal design that I can think of! With a lot of black, these sneakers look very cool. They’ll deﬁnitely be very comfortable for summer.

To me, wearing them with a suit is the easiest way to have a good pair of sneaker shoes. That’s pretty much it! If

Context (WebText test) Some of the most glorious historical attractions in Spain date from the period of Muslim rule, including The Mezquita, built as the Great Mosque of C´ordoba and the Medina Azahara, also in C´ordoba and now in ruins but still visitable as such and built as the Madinat al-Zahra, the Palace of al-Andalus; and the Alhambra in Granada, a splendid, intact palace. There are also two synagogues still standing that were built during the era of Muslim Spain: Santa Mar´ıa la Blanca in Toledo and the Synagogue of C´ordoba, in the Old City.

Reconquista and Imperial era [ edit ] Small model completion The era of Muslim Spain is very different from that of other parts of the world, with a rather extensive medieval history. While the Spanish Empire existed during the second half of the 9th century BC it was quite slowly transformed to a small, secularized empire by 1150 A.D., in a much more prosperous and diverse state. In 1050 CE the emperor of Naples, at least partially in characterised by a series of wars and invasions from the north-western coast to the western coast of the region (e.g., on the Danube River which extended all the way from Gibraltar into the south of Spain, but was quickly destroyed on the same day by the Roman army), assumed the title of Emperor of Naples. He made a large contribution over the last few decades to restore and maintain the empire, the ﬁrst of which to the west. His successors were Ferdinand, who took power from the Papal Throne in 1148, while the ﬁrst imperial successor was King Alfred II of Sweden. The empire was again made one of the few independent states in the European Continent and although no more than 200 million people lived on its territory during this period the people had much to say to the Emperor. From 1720 on the Muslims made up half the population of Spain.

Table 8. Random unseen contexts (top), and non-cherry-picked completions from both the smallest (left) and largest (right) models.

表 8. 随机未见过的上下文（上），以及最小模型（左）与 GPT-2（右）的非精选补全。

Contexts are 768 tokens, with approximately 256 tokens worth of paragraphs shown. Completions are 256 tokens and fully shown. Top-k random sampling with k = 40 was used for generation.

上下文为 768 个词元，图中约展示 256 个词元的段落。补全为 256 个词元。

GPT-2 completion The Reconquista (literally, ”The return to the country”) of 1092 and 1095 saw the establishment in the Iberian peninsula of a united kingdom of Castile and Aragon in which a Muslim dynasty would exist. This was the last time in Europe where Muslim rule would be considered ”native” and the last occasion where Christian-inﬂamed and organized conﬂicts would make themselves known in Europe.

The reign of Muslim Moors in Spain is often referred to, especially among non-Muslims, as the ”Conquest of Granada” and Spain is sometimes referred to as Spain in ”Granada”. The last years of Muslim rule were followed by the Spanish Reconquista where the Muslims were driven or compelled out of the Iberian Peninsula with the victory of Ferdinand IX. in the Battle of Poitiers in March 1189 and the conquest of Jerusalem in 1187 by Saladin.

The Reconquista period in Spain came to an end when the Moors retreated north. However the Moorish rule would not be fully replaced by the Christian rule until the Reconquista era resumed when the Reconquista lasted from 1229 to 1492.

The last Muslim state in Spain, Granada, was conquered by the

Context (WebText test) Pinky The pink ghost’s AI is designed to ”feel” opposite of the red ghost’s behavior. Pinky actually attempts to get out in front of Pac-Man. This is accomplished by setting the target 4 tiles ahead of Pac-Man’s current location in the direction that Pac-Man is travelling. One exception to this is when Pac-Man is traveling up. Due to an overﬂow bug in the code, the calculation includes a left offset equal to the expected up offset.

The use of Pac-Man’s expected position allows the player to exploit the navigation code for Pinky. This was most often seen in situations where Pac-Man and Pinky are in close proximity to each other. For example, if they were headed toward one another, once they are less than 4 tiles apart, the target tile is now behind Pinky. If he encounters an intersection while in this conﬁguration, he will attempt to double-back (by turning) in order to reach the target tile.

Player’s would often perform what was referred to as a ”head fake” by changing directions rapidly just as Pinky was approaching an intersection. By displacing the target tile just as Pinky Small model completion was coming to the left, they would get the ”correct” direction.

If the location is actually farther from what PacMan was at, the player’s will not respond to the ghost’s movement, and the ghost will instead return to the player’s current location. This is because Pac-Man does not move by turning.

The ghost will have to go to another location of some kind to return where he began, and this is achieved in many ways. It will often simply leave the location, as for example if Pac-Man did jump off a cliff, he would return to the spot where he was before vanishing into space. There is a good number of possibilities these ghost paths may be possible, but the most obvious ones are if the player wants to avoid getting caught up in the game, with a goal as easy as keeping Pac-Man in the game for 1/3 as long as he was.

Chasing In Pac-Man’s case, this would be similar to how Pac-Man can ”catch up” in a ﬁght.

A ghost could chase through enemies. This is useful for many reasons, both for Pac-Man and his AI. Unlike most Ghost games, this ghost simply travels in the direction from

Table 9. Random unseen contexts (top), and non-cherry-picked completions from both the smallest (left) and largest (right) models.

表 9. 随机未见过的上下文（上），以及最小模型（左）与 GPT-2（右）的非精选补全。

Contexts are 768 tokens, with approximately 256 tokens worth of paragraphs shown. Completions are 256 tokens and fully shown. Top-k random sampling with k = 40 was used for generation.

上下文为 768 个词元，图中约展示 256 个词元的段落。补全为 256 个词元。

GPT-2 completion was about to cross the intersection, Pac-Man would be able to dodge the ghosts’ projectiles and return to the safe location of the safe house.

Buster The yellow ghost’s AI is speciﬁcally designed to try to avoid Pac-Man from leaving the safe house. The player must press the SPACE bar to reveal the safe house’s doors, then go into them to ﬁnd the yellow ghost hiding in the next room.

The player must also note that the ghost will not attempt to exit the safe house if it sees Pac-Man already there. In fact, the ghost will not attempt an exit in the ﬁrst place unless they’re already in the safe house.

These two behaviors are based on the fact that the player does not always attempt to reach the safe house before Pac-Man (or Blinky).

The player must also beware of this glitch if a ghost (or another player) is already in the safe house and attempting to get out as soon as Pac-Man or Blinky is gone. At this point, the ghost will attempt to exit the safe house in a similar manner as it is in ”Chase” mode.

Pac-Man There are some ghosts that follow Pac-Man. These ghosts are

Context (WebText test) Corporal Michael P. Goeldin was an unskilled laborer from Ireland when he enlisted in Company A in November 1860. Goldein survived the war. Corporal Patrick O’Neal, also from Ireland, ﬁrst enlisted in 1854 and served with Company L, 3d U.S. Artillery, in Oregon. He returned to the East Coast and enlisted in the company in 1860. O’Neal served until 1874, when he was named superintendent of the National Cemetery at Willets Point, New York. Corporal Benjamin Browne was a shoemaker from Orange County, New York. In August 1862, he enlisted in the newly formed 124th New York Volunteers, and was one of sixty-one men who transferred into Company A that October. Browne reenlisted in the company in February 1864 while it was camped at Brandy Station. He returned to civilian life after completing his enlistment in 1867.

On 10 June, Artiﬁcer William Collins was promoted to corporal, probably to ﬁll a combat leadership void for the crossing of the James River. Collins’s service record does not reﬂect the qualities he demonstrated to earn this promotion, but he had obviously overcome some serious problems. Born in Sacketts Harbor, New York, Collins enlisted in the company in December 1853 at the age of twenty-two, and reenlisted in December 1858. Just a month before the war began in April 1861, Collins went ”over the hill” and was not caught until three years later. Returned to the company on 22 March 1864, he was tried Small model completion on military charges and was discharged from the U.S. Military. He returned to the company, and was assigned to Division of Internal Affairs during the Civil War.

As an engineer, Collins also had training in engineering from his native New York. After graduate studies in the Union College of Engineering, he was employed in the United States Army for two years at the same time. There he worked under Captain Daniel Pfeiffer and was promoted to colonel in May 1870, three years after Captain Pfeiffer’s enlistment. Collins returned home to England after completing his ﬁrst year in the First Division to serve the next two years in the Army Corps of Engineers, and by his second year, he began to experience his service duty.¡—endoftext—¿MOSCOW, July 5 (Reuters) - U.K.-based Russian state media and government sources and experts say Moscow has sought to downplay the ”military buildup” of the Syrian opposition while maintaining support for President Donald Trump’s plan to build a wall along the U.S.-Mexico border, calling it a ”military coup”.

U.S.-led Syria war veteran and Syria envoy Michael Flynn (R), right, gestures with U.S.-based Syrian political analyst Ahmet Koryat, with

Table 10. Random unseen contexts (top), and non-cherry-picked completions from both the smallest (left) and largest (right) models.

表 10. 随机未见过的上下文（上），以及最小模型（左）与 GPT-2（右）的非精选补全。

Contexts are 768 tokens, with approximately 256 tokens worth of paragraphs shown. Completions are 256 tokens and fully shown. Top-k random sampling with k = 40 was used for generation.

上下文为 768 个词元，图中约展示 256 个词元的段落。补全为 256 个词元。

GPT-2 completion in June of that year. Sentenced to be shot for desertion, he remained on active duty.

Captain James J. Dall was born in Virginia in 1829 during the Revolution and joined his father in the Army in 1836. He served with the 2d Virginia Infantry until 1845, when he went to the Western Reserve Regiment; and, on his discharge, he reenlisted with Company I, 25th New York Volunteers (which in 1845 numbered about 4,500 men). One of his brothers, James, served with his company in the same capacity in the same brigade. While the regiment remained in Connecticut through 1846, Captain Dall served with Company I at various times before his discharge. A veteran of the Mexican War, he had served two years on the staff of Brigadier General John J. Sullivan at San Antonio. During the Peninsula Campaign of 1863, he commanded a company of twenty-three and was in charge of the battery of ﬁfty-four heavy mortars and the gunboat Pinta of the Battery, a gunboat and several other boats. Captain Dall was on active duty with Company I in the fall of 1865. Two years later, he was promoted to second lieutenant.

Captain John A. O’

Context (WebText test) Xmonad, Xresources, and Fonts Xresources can be an absolute pain (they were for me). So, this a bit of help: First of all, the way that fonts are generally handled in Xmonad is through XFT . Xft is the X FreeType Interface library and is a library that handles the actual rendering of fonts in Xmonad/urxvt (Xresources speciﬁes a bunch of resources to things that launch under the X window manager. Settings for individual programs are speciﬁed in Xresources like so: Urxvt.background:background settins here ).

Anyway, fonts are set using this syntax in Xresources : urxvt*font:xft:your font name goes here:size=the size of the font goes here,xft:fall-back font 1 goes here So, thats the most basic part of fonts in urxvt speciﬁed via Xresources.

Fixing /.ssh/ Permissions in Cygwin In Cygwin on Windows I found that I could not set the permissions of my /.ssh/ folder to be 0600, as is required for ssh to allow you to use keys. The symptom I had was that I no matter what I did, it always modﬁed the owner and the group Small model completion was set to /.ssh/ , which was the way things were. This could not be ﬁxed completely. I found I could make that much more workable with just typing the username with your mouse then trying to set permissions to be ”0600” for the group, which is what my system is now conﬁgured to do.

However, as of Cygwin 7 on Linux I have found that doing so is now ”unnecessary” for my system! I also installed a package called chroot. It does that by allowing you to install/upgrade to your system directory.

So, as of Windows 8 and up, installing that can require chroot, so I tried and tried and found that it wasn’t so bad when installing the package on Linux and then uninstalling it. As of my last write-up on Cygwin, the script just does its job (the sudo commands do the rest), if you try it you will have to log the change.

If that wasn’t confusing enough, this is what I tried and just went back to my /.ssh/ directory and the chroot script. I found I was ﬁnally running out of permissions (because i forgot to create a symlink to ssh from the ﬁle system and

Table 11. Random unseen contexts (top), and non-cherry-picked completions from both the smallest (left) and largest (right) models.

表 11. 随机未见过的上下文（上），以及最小模型（左）与 GPT-2（右）的非精选补全。

Contexts are 768 tokens, with approximately 256 tokens worth of paragraphs shown. Completions are 256 tokens and fully shown. Top-k random sampling with k = 40 was used for generation.

上下文为 768 个词元，图中约展示 256 个词元的段落。补全为 256 个词元。

GPT-2 completion , making these ﬁles read-only. I tried just starting ssh and using a program like passwd which modiﬁes the owner/group to 0644 permissions, but it didn’t work. It was annoying but not nearly as bad as it could be, just annoying to have to ﬁddle with permissions.

So I decided to do this as the reverse: For each .ssh folder in your Cygwin directory, set the permission of each ﬁle and directory to 2644. Then start ssh with: ssh-add /.ssh/ The ﬁrst time you connect, your permissions of the /.ssh folder that you just had in Cygwin will be set to 0600. Then every time you connect again, your permissions will change to 0644.

Makeﬁle for Xresources as used in /.xsessionrc These are made in my .xsessionrc : .xproﬁle: .xsession-errors: color=”” For example: .xproﬁle: color=blue As the colors are stored in a separate ﬁle, the color in your /.xsession-errors will look like cyan instead of blue.

This is mostly because, as of now, only the color is assigned to your

Context (WebText test) Aaron loves mint chocolate cake, but he requires that it be paired with mini chocolate chips, so I threw some of those in between the layers. I also had a few Peppermint Jo Jos on hand so I crushed them up and threw some of those in along with some crushed meringue cookies because, why not? It’s a total smorgasbord of minty chocolate chippy cookie crunchy goodness. I didn’t measure how much of each topping I used, but after I tasted the ﬁnished product, I wish I had added more. You can add anything you wantcrushed candy canes, peppermint bark, etc. And don’t be afraid to use a heavy hand. Texture = good.

If you don’t have 7-inch cake pans, you can get 3 shorter 8-inch layers out of this GPT-2 Completion 1 recipe. I made the longer version, then used the pan for a layer of buttercream frosting and stuck 1/3 of it into the fridge for a frosting topper. Just make sure you have all of this frosting prepared earlier than you make this cake and you won’t have to wait on it as your egg whites sit on the bottom of your mixing bowl.

GPT-2 Completion 2 recipe. If you do, and you know how to make that cake, then your life will never be the same.

I have two very simple rules when it comes to cookie crumbs or chocolate chips. 1) When I buy raw sugar, I do not use it until it’s cooled down and has at least half a teaspoon of sugar on it. That’s my rule. And 2) If I have a recipe that calls for something and I want to use it in a recipe, I do not substitute it with something else on-line. Like this.

You want the egg whites whipped to stiff peaks? Then put your cake dish over a pot of simmering water and heat it up until it’s just barely warm. It doesn’t take more than 3-5 minutes; just enough to get it started I found this really cute card with cute little kittens on GPT-2 Completion 4 recipe.

GPT-2 Completion 5 recipe.

Meringue Cookies My 2 failures at Christmas morning were not due to poor frosting skills. That was another success story. The cake was beautifully moist, and well, just a beautiful cake. It’s got very little air in it or moisture, but all the chocolate and mint was perfectly blended together. 1 3/4 cup butter, softened 1 cup sugar 1 egg yolk 3 T. heavy cream 3 1/2-4 cups ﬂour pinch of salt Peppermint-Jojo Topping 1 1/4 cups powdered sugar 1 cup chopped pecans 1/2 cup ﬁnely chopped mint leaves 1/2 cup chopped fresh mint (about 1/2 sheet) 1 tsp. peppermint extract 1/4 tsp. sea salt pinch of sea One thing that made me cringe a bit was that all those ingredientsegg yolks, sugar, melted butter, chocolate chips, melted chocolate, and minthad to be mixed right into the cake batter, which is super time-consuming. A friend suggested that if I put the batter in a separate bowl, I

Table 12. Non-cherry-picked completions from GPT-2 generated from the same context (from WebText test). Context is 384 tokens (shown truncated), and generations are 128 tokens. Top-k random sampling with k = 40 was used for generation.

表 12. 由同一上下文（来自 WebText 测试集）生成的、未经精选的 GPT-2 补全。

GPT-2 Completion 3 recipe, which will likely yield about the same amount of cake.

Chocolate Buttermilk Cake Print Chunks of chocolate cake, coated in buttermilk and topped with crunchy chopped chocolate, drizzled with chocolate sauce. Simple but a nice change from chocolate cake! Author: Momma Made Ingredients For the chocolate buttermilk cake 1 1/2 cup unsalted butter , room temperature 1 1/2 cup granulated sugar 3 large eggs plus 1 egg yolk For the chocolate glaze 1/3 cup cocoa powder 1 3/4 cups powdered sugar 6 ounces semis GPT-2 Completion 6 recipe, which will give you about 6-9 servings of delicious chocolate cake. To prevent your layers from sticking together, I used my 8-inch round tart pan, lined it with parchment paper, then cut up 4 7-inch round cakes from it and sandwiched the layers between them (this also worked well for the 1-layer chocolate cake as well). When you’re ready to bake this cake, remove the parchment paper, preheat oven to 350 degrees and bake for about 30 - 40 minutes, until the top is golden and a tester comes out clean.

Chocolate Buttermilk Layer Cake with Peppermint Jo

Context (human-written): In a shocking ﬁnding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

GPT-2: The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.

Now, after almost two centuries, the mystery of what sparked this odd phenomenon is ﬁnally solved.

Dr. Jorge P´erez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. P´erez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.

P´erez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said P´erez.

P´erez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them - they were so close they could touch their horns.

While examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. P´erez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.” Dr. P´erez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America.

While their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to P´erez, “In South America, such incidents seem to be quite common.” However, P´erez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist.

Table 13. Conditional generation on an out-of-distribution context by GPT-2. Cherry pick of 10 samples generated with k = 40.

表 13. GPT-2 在分布外上下文上的条件生成。从 10 个样本中精选。

Article: Prehistoric man sketched an incredible array of prehistoric beasts on the rough limestone walls of a cave in modern day France 36,000 years ago. Now, with the help of cutting-edge technology, those works of art in the Chauvet-Pont-d’Arc Cave have been reproduced to create the biggest replica cave in the world. The manmade cavern named the Caverne du Pont-d’Arc has been built a few miles from the original site in Vallon-Pont-D’arc in Southern France and contains 1,000 painstakingly-reproduced drawings as well as around 450 bones and other features... Cavemen and women sketched an incredible array of prehistoric beasts on the rough limestone walls of a cave 36,000 years ago and now a replica has been created (pictured) ...

GPT-2: The original site in Vallon-Pont-D’arc in Southern France is a Unesco World Heritage site and is the oldest known and the best preserved cave decorated by man. The replica cave was built a few miles from the original site in Vallon-Pont-D’Arc in Southern France. The cave contains images of 14 different species of animals including woolly rhinoceros, mammoths, and big cats.

Article: ‘If you can ﬁnd my submarine, it’s yours,’ Russian oil billionaire Roman Abramovich once said. And it seems the oligarch and Chelsea FC owner, whose submarine is just one of the extras that came with his £300million superyacht Eclipse (perfect for getting to shore undetected), is not the only wealthy businessman splashing out on underwater exploration. Dubbed Earth’s real ‘ﬁnal frontier’, the oceans are still so little-explored that billionaires are queuing up to buy vessels that give them a glimpse of the dramatic seascapes and incredible wildlife of the world’s oceans. So if you have a spare few million in the bank and want some holiday snaps and Instagram posts that will really trump everyone else, you may want to snap up one of these... Whale of a time: The OrcaSub takes you 2000 feet beneath the surface in two pressurised Perspex viewing domes for optimum exploration ...

GPT-2: The U-Boat Worx submarine can dive to 984 feet (300 metres) and is capable of diving to the bottom of the ocean. It can be used for diving, snorkelling and even snorkel diving. The submarine is also equipped with a snorkelling platform.

Article: Amina Ali Qassim is sitting with her youngest grandchild on her lap, wiping away tears with her headscarf. Only a few months old, this is the baby girl whose ears she desperately tried to cover the night the aerial bombardment started. She lay awake, she says, in a village mosque on the Yemeni island of Birim, counting explosions as the baby cried. It could have been worse though. They could have still been in their house when the ﬁrst missile landed. ”Our neighbor shouted to my husband ’you have to leave, they’re coming.’ And we just ran. As soon as we left the house, the ﬁrst missile fell right by it and then a second on it. It burned everything to the ground,” Qassim tells us ...

GPT-2: Yemen is in the middle of a civil war. Saudi Arabia is leading the coalition bombing campaign. It’s been bombing Yemen for more than two months now.

Table 14. Summaries generated by GPT-2 and reference summaries on the test set of the CNN and Daily Mail dataset.

表 14. GPT-2 生成的摘要与 CNN 和 Daily Mail 测试集上的参考摘要。

Reference: Cave mimics famous Caverne du Pont-d’Arc in France, the oldest cave decorated by man and the best preserved. The replica contains all 1,000 paintings which include 425 such as a woolly rhinoceros and mammoths. Minute details were copied using 3D modelling and anamorphic techniques, often used to shoot widescreen images. The modern cave also includes replica paw prints of bears, bones and details preserved in the original cave.

Reference: Feel like a James Bond villain with the most advanced submerged ocean vehicles. Some convert from a yacht into a submarine, and others detached off for a speedy exploration. Oliver’s Travels offers a Mile Low package on its submarine, complete with chef and butler for £175,000 a night.

Reference: Amina Ali Qassim’s family sought shelter in a mosque before ﬂeeing Yemen. Thousands like them are boarding boats to sail to Djibouti. Saudi Arabia has been pounding Yemen in a bid to defeat Houthi rebels.

English reference GPT-2 French translation This re-release, titled The Next Day Extra, was presented in the form of three disks: the original album, unpublished studio sessions and remixes, plus a DVD containing the four clips that have already been unveiled.

French reference GPT-2 English translation Un homme expliquait que le fonctionnement de la hernia fonctionnelle qu’il avait reconnaˆıt avant de faire, le fonctionnement de la hernia fonctionnelle que j’ai r´eussi, j’ai r´eussi.

English reference GPT-2 French translation One man explained that the free hernia surgery he’d received will allow him to work again.

French reference GPT-2 English translation Un homme a expliqu´e que l’op´eration gratuite qu’il avait subie pour soigner une hernie lui permettrait de travailler `a nouveau.

English reference GPT-2 French translation Speaking to an open government conference in London via video link, Mr Kerry said: ”There is no question that the President and I and others in government have actually learned of some things that had been happening on an automatic pilot because the ability has been there, going back to World War Two and to the very difﬁcult years of the Cold War, and then, of course, 9/11.” French reference GPT-2 English translation S’exprimant lors d’une conf´erence intergouvernementale a Londres par liaison vid´eo, M. Kerry a d´eclar´e: ”Il est ind´eniable que le Pr´esident, moi-mˆeme et d’autres membres du gouvernement avons pris connaissance de certaines choses en mode pilote automatique parce que nous en avions la possibilit´e, d es la Seconde guerre mondiale et jusqu’aux ann´ees difﬁciles de la Guerre froide, puis bien sˆur le 11 septembre.”

Table 15. English to French and French to English translations generated by GPT-2.

表 15. GPT-2 生成的英译法与法译英翻译。

Les nouvelles re-releases, tout en premier disc, n´ecessaire de l’album, un studio session et remixes, plus une DVD de l’´ecran de quelques clips qui ont ´et´e d´ej`a ´echapp´es.

This release, originally released on three discs: the original album, sessions studio in´edites and remixes, a DVD containing the four clips that were released on the previous day.

Un homme expliquait que le fonctionnement de la hernia fonctionnelle qu’il avait reconnaˆıt avant de faire, le fonctionnement de la hernia fonctionnelle que j’ai r´eussi, j’ai r´eussi.

A man told me that the operation gratuity he had been promised would not allow him to travel.

Le premier conference de l’´edition internationale en ligne sur la vide, le ministre de l’´Edition internationale a ´et´e vide, lui dit : ”Il n’y a qu’une question que le pr´esident et moi et tous les autres personnes d’une administration qui a ´et´e ´evident par la fois, par la part de la guerre, par la part de la guer In a conf´erence between the United States and London, Secretary of State John Kerry said: ”It is indeniable that the President, myself and others of the government have been aware of certain certain choices that have been made in the past in order to be able to do certain things in a more automated way.”

Context (passage and previous question/answer pairs) The 2008 Summer Olympics torch relay was run from March 24 until August 8, 2008, prior to the 2008 Summer Olympics, with the theme of “one world, one dream”. Plans for the relay were announced on April 26, 2007, in Beijing, China. The relay, also called by the organizers as the “Journey of Harmony”, lasted 129 days and carried the torch 137,000 km (85,000 mi) - the longest distance of any Olympic torch relay since the tradition was started ahead of the 1936 Summer Olympics.

After being lit at the birthplace of the Olympic Games in Olympia, Greece on March 24, the torch traveled to the Panathinaiko Stadium in Athens, and then to Beijing, arriving on March 31. From Beijing, the torch was following a route passing through six continents. The torch has visited cities along the Silk Road, symbolizing ancient links between China and the rest of the world. The relay also included an ascent with the ﬂame to the top of Mount Everest on the border of Nepal and Tibet, China from the Chinese side, which was closed specially for the event.

Q: What was the theme A: “one world, one dream”.

Q: What was the length of the race? A: 137,000 km Q: Was it larger than previous ones? A: No Q: Where did the race begin? A: Olympia, Greece Q: Is there anything notable about that place? A: birthplace of Olympic Games Q: Where did they go after? A: Athens Q: How many days was the race? A: seven Q: Did they visit any notable landmarks? A: Panathinaiko Stadium Q: And did they climb any mountains? A: Model answer: Everest Turker answers: unknown, yes, Yes, yes

Table 16. Selected CoQA completion.

表 16. 选定的 CoQA 补全。

Context (passage and previous question/answer pairs) Tom goes everywhere with Catherine Green, a 54-year-old secretary. He moves around her ofﬁce at work and goes shopping with her. ”Most people don’t seem to mind Tom,” says Catherine, who thinks he is wonderful. ”He’s my fourth child,” she says. She may think of him and treat him that way as her son. He moves around buying his food, paying his health bills and his taxes, but in fact Tom is a dog.

Catherine and Tom live in Sweden, a country where everyone is expected to lead an orderly life according to rules laid down by the government, which also provides a high level of care for its people. This level of care costs money.

People in Sweden pay taxes on everything, so aren’t surprised to ﬁnd that owning a dog means more taxes. Some people are paying as much as 500 Swedish kronor in taxes a year for the right to keep their dog, which is spent by the government on dog hospitals and sometimes medical treatment for a dog that falls ill. However, most such treatment is expensive, so owners often decide to offer health and even life for their dog.

In Sweden dog owners must pay for any damage their dog does. A Swedish Kennel Club ofﬁcial explains what this means: if your dog runs out on the road and gets hit by a passing car, you, as the owner, have to pay for any damage done to the car, even if your dog has been killed in the accident.

Q: How old is Catherine? A: 54 Q: where does she live? A: Model answer: Stockholm Turker answers: Sweden, Sweden, in Sweden, Sweden

Table 17. Selected CoQA completion.

表 17. 选定的 CoQA 补全。