Language Models are Few-Shot Learners

语言模型是少样本学习者（Language Models are Few-Shot Learners）

Tom B. Brown † † thanks: Equal contribution    Benjamin Mann 1 1 footnotemark: 1    Nick Ryder 1 1 footnotemark: 1    Melanie Subbiah 1 1 footnotemark: 1    Jared Kaplan † † thanks: Johns Hopkins University, OpenAI Author contributions listed at end of paper .    Prafulla Dhariwal    Arvind Neelakantan    Pranav Shyam    Girish Sastry    Amanda Askell    Sandhini Agarwal    Ariel Herbert-Voss    Gretchen Krueger    Tom Henighan    Rewon Child    Aditya Ramesh    Daniel M. Ziegler    Jeffrey Wu    Clemens Winter    Christopher Hesse    Mark Chen    Eric Sigler    Mateusz Litwin    Scott Gray    Benjamin Chess    Jack Clark    Christopher Berner    Sam McCandlish    Alec Radford    Ilya Sutskever    Dario Amodei    OpenAI

Tom B. Brown † † thanks: 同等贡献    Benjamin Mann 1 1 footnotemark: 1    Nick Ryder 1 1 footnotemark: 1    Melanie Subbiah 1 1 footnotemark: 1    Jared Kaplan † † thanks: Johns Hopkins University, OpenAI 作者贡献见文末。    Prafulla Dhariwal    Arvind Neelakantan    Pranav Shyam    Girish Sastry    Amanda Askell    Sandhini Agarwal    Ariel Herbert-Voss    Gretchen Krueger    Tom Henighan    Rewon Child    Aditya Ramesh    Daniel M. Ziegler    Jeffrey Wu    Clemens Winter    Christopher Hesse    Mark Chen    Eric Sigler    Mateusz Litwin    Scott Gray    Benjamin Chess    Jack Clark    Christopher Berner    Sam McCandlish    Alec Radford    Ilya Sutskever    Dario Amodei    OpenAI

Abstract

摘要（Abstract）

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

近期工作表明，在大规模文本语料上预训练、再针对具体任务微调，可在众多 NLP 任务与基准上取得显著提升。尽管架构通常与任务无关，该方法仍需要数千乃至数万条任务专用微调数据。相比之下，人类往往只需少量示例或简单指令就能完成新的语言任务——而这正是当前 NLP 系统仍普遍难以做到的。本文表明，放大语言模型可大幅提升与任务无关的少样本（few-shot）表现，有时甚至可与此前最先进的微调方法相竞争。具体地，我们训练了 GPT-3，一个拥有 1750 亿参数的自回归语言模型，规模约为此前任何非稀疏语言模型的 10 倍，并在少样本设定下测试其表现。对所有任务，GPT-3 均不进行任何梯度更新或微调，任务与少样本演示纯粹通过与模型的文本交互来指定。GPT-3 在许多 NLP 数据集上表现强劲，包括翻译、问答与完形填空，以及若干需要即时推理或领域适应的任务，例如字母重组、在句子中使用新词，或完成三位数算术。与此同时，我们也指出了一些 GPT-3 少样本学习仍然吃力的数据集，以及一些因在大型网络语料上训练而带来方法论问题的数据集。最后，我们发现 GPT-3 能生成新闻文章样本，人类评估者难以将其与真人撰写的文章区分开。我们讨论了这一发现以及 GPT-3 本身更广泛的社会影响。

1 Introduction

1 引言（Introduction）

Contents

目录

Recent years have featured a trend towards pre-trained language representations in NLP systems, applied in increasingly flexible and task-agnostic ways for downstream transfer. First, single-layer representations were learned using word vectors [82, 102] and fed to task-specific architectures, then RNNs with multiple layers of representations and contextual state were used to form stronger representations [24, 81, 100] (though still applied to task-specific architectures), and more recently pre-trained recurrent or transformer language models [134] have been directly fine-tuned, entirely removing the need for task-specific architectures [112, 20, 43].

近年来，自然语言处理系统中出现了向预训练语言表示的趋势，这些表示以越来越灵活、与任务无关的方式应用于下游迁移。首先，使用词向量 [82, 102] 学习了单层表示，并将其输入到特定任务的架构中，然后使用具有多层表示和上下文状态的 RNN 来形成更强的表示 [24, 81, 100]（尽管仍然应用于特定任务的架构），最近，预训练的循环或 Transformer 语言模型 [134] 已经被直接微调，完全消除了对特定任务的需求fic 架构 [112, 20, 43]。

Figure 1.1: Language model meta-learning. During unsupervised pre-training, a language model develops a broad set of skills and pattern recognition abilities. It then uses these abilities at inference time to rapidly adapt to or recognize the desired task. We use the term “in-context learning” to describe the inner loop of this process, which occurs within the forward-pass upon each sequence. The sequences in this diagram are not intended to be representative of the data a model would see during pre-training, but are intended to show that there are sometimes repeated sub-tasks embedded within a single sequence.

图 1.1：语言模型元学习。 在无监督的预训练期间，语言模型会发展出广泛的技能和模式识别能力。然后，它在推理时使用这些能力快速适应或识别所需的任务。我们使用“上下文学习”一词来描述这一过程的内部循环，该循环发生在每个序列的前向传播过程中。此图中的序列并不旨在代表模型在预训练期间会看到的数据，而是旨在展示在单个序列中有时会嵌入重复的子任务。

This last paradigm has led to substantial progress on many challenging NLP tasks such as reading comprehension, question answering, textual entailment, and many others, and has continued to advance based on new architectures and algorithms [116, 74, 139, 62]. However, a major limitation to this approach is that while the architecture is task-agnostic, there is still a need for task-specific datasets and task-specific fine-tuning: to achieve strong performance on a desired task typically requires fine-tuning on a dataset of thousands to hundreds of thousands of examples specific to that task. Removing this limitation would be desirable, for several reasons.

这一最后的范式在许多具有挑战性的自然语言处理任务上取得了实质性进展，例如阅读理解、问答、文本蕴涵等，并且基于新的架构和算法持续发展 [116, 74, 139, 62]。然而，这种方法的一个主要限制是，尽管该架构是任务无关的，仍然需要针对特定任务的数据集和特定任务的微调：要在所需任务上取得优异表现，通常需要对该任务特定的成千上万甚至数十万的示例数据集进行微调。出于多种原因，消除这一限制是可取的。

First, from a practical perspective, the need for a large dataset of labeled examples for every new task limits the applicability of language models. There exists a very wide range of possible useful language tasks, encompassing anything from correcting grammar, to generating examples of an abstract concept, to critiquing a short story. For many of these tasks it is difficult to collect a large supervised training dataset, especially when the process must be repeated for every new task.

首先，从实际的角度来看，每个新任务都需要大量标注例子的需求限制了语言模型的适用性。存在非常广泛的可能有用的语言任务，涵盖从纠正语法，到生成抽象概念的示例，再到评论短篇小说的所有内容。对于许多这些任务来说，收集大量的监督训练数据集是困难的，尤其是在每个新任务都必须重复这一过程时。

Second, the potential to exploit spurious correlations in training data fundamentally grows with the expressiveness of the model and the narrowness of the training distribution. This can create problems for the pre-training plus fine-tuning paradigm, where models are designed to be large to absorb information during pre-training, but are then fine-tuned on very narrow task distributions. For instance [41] observe that larger models do not necessarily generalize better out-of-distribution. There is evidence that suggests that the generalization achieved under this paradigm can be poor because the model is overly specific to the training distribution and does not generalize well outside it [138, 88]. Thus, the performance of fine-tuned models on specific benchmarks, even when it is nominally at human-level, may exaggerate actual performance on the underlying task [36, 91].

其次，利用训练数据中虚假相关性的潜力，随着模型表达能力的增强和训练分布的狭窄而根本性地增加。这可能会对预训练加微调的范式造成问题，在该范式中，模型设计得很大以在预训练期间吸收信息，但随后在非常狭窄的任务分布上进行微调。例如，[41] 观察到更大的模型不一定在分布外泛化得更好。有证据表明，在这种范式下实现的泛化可能很差，因为模型对训练分布过于特定，且在训练分布之外泛化能力不佳 [138, 88]。因此，即使在名义上达到人类水平，微调模型在特定基准测试上的表现，也可能夸大其在基础任务上的实际表现 [36, 91]。

Third, humans do not require large supervised datasets to learn most language tasks – a brief directive in natural language (e.g. “please tell me if this sentence describes something happy or something sad”) or at most a tiny number of demonstrations (e.g. “here are two examples of people acting brave; please give a third example of bravery”) is often sufficient to enable a human to perform a new task to at least a reasonable degree of competence. Aside from pointing to a conceptual limitation in our current NLP techniques, this adaptability has practical advantages – it allows humans to seamlessly mix together or switch between many tasks and skills, for example performing addition during a lengthy dialogue. To be broadly useful, we would someday like our NLP systems to have this same fluidity and generality.

第三，人类在学习大多数语言任务时不需要大量的监督数据集——只需简短的自然语言指令（例如，“请告诉我这句话描述的是快乐的事情还是悲伤的事情”）或最多少量示范（例如。“‘这里有两个勇敢行为的例子；请给出第三个勇敢的例子’通常足以让人类以至少合理的能力执行一项新任务。”除了指出我们当前自然语言处理技术的概念限制之外，这种适应性还有实际的优势——它允许人类无缝地混合或切换多种任务和技能，例如在漫长的对话中进行加法运算。为了广泛有用，我们希望有一天我们的自然语言处理系统也能拥有这种流动性和通用性。

One potential route towards addressing these issues is meta-learning[^1] – which in the context of language models means the model develops a broad set of skills and pattern recognition abilities at training time, and then uses those abilities at inference time to rapidly adapt to or recognize the desired task (illustrated in Figure 1.1). Recent work [117] attempts to do this via what we call “in-context learning”, using the text input of a pretrained language model as a form of task specification: the model is conditioned on a natural language instruction and/or a few demonstrations of the task and is then expected to complete further instances of the task simply by predicting what comes next.

解决这些问题的一个潜在途径是元学习[^1]——在语言模型的背景下，这意味着模型在训练时发展出一整套技能和模式识别能力，然后在推理时利用这些能力快速适应或识别所需的任务（如图1.1所示）。最近的工作 [117] 尝试通过我们称之为“上下文学习”的方法来实现这一点，使用预训练语言模型的文本输入作为任务规范的一种形式：模型基于自然语言指令和/或该任务的几个示例进行条件训练，然后期望仅通过预测接下来会发生的内容来完成该任务的进一步实例。

While it has shown some initial promise, this approach still achieves results far inferior to fine-tuning – for example [117] achieves only 4% on Natural Questions, and even its 55 F1 CoQa result is now more than 35 points behind the state of the art. Meta-learning clearly requires substantial improvement in order to be viable as a practical method of solving language tasks.

虽然这一方法显示出了一些初步的前景，但其结果仍远不及微调——例如，[117] 在 Natural Questions 上仅达到 4%，即使其在 CoQa 上的 F1 为 55，现在也比最先进水平落后超过 35 分。元学习显然需要显著改进，才能作为解决语言任务的可行方法。

Another recent trend in language modeling may offer a way forward. In recent years the capacity of transformer language models has increased substantially, from 100 million parameters [112], to 300 million parameters [20], to 1.5 billion parameters [117], to 8 billion parameters [125], 11 billion parameters [116], and finally 17 billion parameters [132]. Each increase has brought improvements in text synthesis and/or downstream NLP tasks, and there is evidence suggesting that log loss, which correlates well with many downstream tasks, follows a smooth trend of improvement with scale [57]. Since in-context learning involves absorbing many skills and tasks within the parameters of the model, it is plausible that in-context learning abilities might show similarly strong gains with scale.

语言建模的另一个最新趋势可能提供了一条前进的道路。近年来，变压器语言模型的容量显著增加，从1亿参数[112]，到3亿参数[20]，到15亿参数[117]，到80亿参数[125]，110亿参数[116]，最终达到170亿参数[132]。每一次增加都带来了文本合成和/或下游 NLP 任务的改进，并且有证据表明，对许多下游任务相关性良好的对数损失（log loss）随着规模的增加呈现平稳的改进趋势 [57]。由于上下文学习涉及在模型参数中吸收许多技能和任务，因此上下文学习能力可能会随着规模的增加而显示出同样显著的提升。

Figure 1.2: Larger models make increasingly efficient use of in-context information.   We show in-context learning performance on a simple task requiring the model to remove random symbols from a word, both with and without a natural language task description (see Sec. 3.9.2). The steeper “in-context learning curves” for large models demonstrate improved ability to learn a task from contextual information. We see qualitatively similar behavior across a wide range of tasks.

图 1.2：更大的模型能够更高效地利用上下文信息。 我们展示了在一个简单任务上的上下文学习性能，该任务要求模型从单词中移除随机符号，同时分别考虑有无自然语言任务描述的情况（参见第 3.9.2 节）。较大的模型具有更陡的“上下文学习曲线”，这表明它们从上下文信息中学习任务的能力有所提高。我们在各种任务中都看到定性上相似的行为。

Figure 1.3: Aggregate performance for all 42 accuracy-denominated benchmarks   While zero-shot performance improves steadily with model size, few-shot performance increases more rapidly, demonstrating that larger models are more proficient at in-context learning. See Figure 3.8 for a more detailed analysis on SuperGLUE, a standard NLP benchmark suite.

图 1.3：所有 42 个以准确率为标准的基准汇总性能   随着模型规模的增加，零样本性能稳步提升，而少样本性能增长更快，这表明较大的模型在上下文学习方面更为娴熟。有关标准 NLP 基准套件 SuperGLUE 的更详细分析，请参见图 3.8。

In this paper, we test this hypothesis by training a 175 billion parameter autoregressive language model, which we call GPT-3, and measuring its in-context learning abilities. Specifically, we evaluate GPT-3 on over two dozen NLP datasets, as well as several novel tasks designed to test rapid adaptation to tasks unlikely to be directly contained in the training set. For each task, we evaluate GPT-3 under 3 conditions: (a) “few-shot learning”, or in-context learning where we allow as many demonstrations as will fit into the model’s context window (typically 10 to 100), (b) “one-shot learning”, where we allow only one demonstration, and (c) “zero-shot” learning, where no demonstrations are allowed and only an instruction in natural language is given to the model. GPT-3 could also in principle be evaluated in the traditional fine-tuning setting, but we leave this to future work.

在本文中，我们通过训练一个拥有1750亿参数的自回归语言模型——我们称之为GPT-3——并测量其上下文学习能力来测试这一假设。具体来说，我们在二十多个自然语言处理数据集上评估GPT-3，以及几个旨在测试对不太可能直接包含在训练集中的任务的快速适应能力的新任务。对于每个任务，我们在三种条件下评估 GPT-3：（a）“少量示例学习”，即上下文学习，我们允许尽可能多的示例以适应模型的上下文窗口（通常为 10 到 100 个）；（b）“单次示例学习”，我们只允许一个示例；以及（c）“零示例学习”，不允许任何示例，只向模型提供自然语言指令。原则上，GPT-3也可以在传统的微调设置中进行评估，但我们将此留待未来工作。

Figure 1.2 illustrates the conditions we study, and shows few-shot learning of a simple task requiring the model to remove extraneous symbols from a word. Model performance improves with the addition of a natural language task description, and with the number of examples in the model’s context, $K$. Few-shot learning also improves dramatically with model size. Though the results in this case are particularly striking, the general trends with both model size and number of examples in-context hold for most tasks we study. We emphasize that these “learning” curves involve no gradient updates or fine-tuning, just increasing numbers of demonstrations given as conditioning.

图 1.2 展示了我们研究的条件，并显示了简单任务的小样本学习，该任务要求模型从一个单词中删除多余的符号。随着自然语言任务描述的加入以及模型上下文中示例数量的增加 $K$，模型性能有所提升。小样本学习的效果也随着模型规模的增大而显著提高。尽管在这种情况下结果尤其引人注目，但我们研究的大多数任务的模型规模和上下文示例数量的一般趋势仍然成立。我们强调，这些“学习”曲线不涉及任何梯度更新或微调，仅仅是增加作为条件输入的示范数量。

Broadly, on NLP tasks GPT-3 achieves promising results in the zero-shot and one-shot settings, and in the the few-shot setting is sometimes competitive with or even occasionally surpasses state-of-the-art (despite state-of-the-art being held by fine-tuned models). For example, GPT-3 achieves 81.5 F1 on CoQA in the zero-shot setting, 84.0 F1 on CoQA in the one-shot setting, 85.0 F1 in the few-shot setting. Similarly, GPT-3 achieves 64.3% accuracy on TriviaQA in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting, the last of which is state-of-the-art relative to fine-tuned models operating in the same closed-book setting.

总体而言，在自然语言处理任务中，GPT-3 在零样本和单样本设定下取得了令人满意的结果，在少样本设定下有时能够与最先进技术竞争，甚至偶尔超过最先进技术（尽管最先进技术是由微调模型保持的）。例如，GPT-3 在零样本设定下在 CoQA 上取得 81.5 的 F1 分数，在单样本设定下取得 84.0 的 F1 分数，在少样本设定下取得 85.0 的 F1 分数。同样，GPT-3 在 TriviaQA 数据集上在零样本情况下达到了 64.3% 的准确率，在单样本情况下达到了 68.0%，在少样本情况下达到了 71.2%，后者在与之在相同闭卷设置下操作的微调模型相比时是最先进的水平。

GPT-3 also displays one-shot and few-shot proficiency at tasks designed to test rapid adaption or on-the-fly reasoning, which include unscrambling words, performing arithmetic, and using novel words in a sentence after seeing them defined only once. We also show that in the few-shot setting, GPT-3 can generate synthetic news articles which human evaluators have difficulty distinguishing from human-generated articles.

GPT-3 在旨在测试快速适应或即时推理的任务中，也表现出一次性学习和少量学习的能力，这些任务包括重组单词、进行算术运算，以及在仅看到一次定义后将新词用于句子中。我们还表明，在少量学习的情况下，GPT-3 可以生成合成新闻文章，而人类评估者很难将其与人类创作的文章区分开来。

At the same time, we also find some tasks on which few-shot performance struggles, even at the scale of GPT-3. This includes natural language inference tasks like the ANLI dataset, and some reading comprehension datasets like RACE or QuAC. By presenting a broad characterization of GPT-3’s strengths and weaknesses, including these limitations, we hope to stimulate study of few-shot learning in language models and draw attention to where progress is most needed.

与此同时，我们还发现一些任务即使在 GPT-3 的规模下，少量示范的性能也表现不佳。这包括像 ANLI 数据集这样的自然语言推理任务，以及一些阅读理解数据集，如 RACE 或 QuAC。通过对GPT-3的优势和劣势进行广泛的描述，包括这些限制，我们希望能促进对语言模型中少样本学习的研究，并引起人们对最需要改进的地方的关注。

A heuristic sense of the overall results can be seen in Figure 1.3, which aggregates the various tasks (though it should not be seen as a rigorous or meaningful benchmark in itself).

图 1.3 中可以看到对整体结果的启发式认知，它汇总了各种任务（尽管本身不应被视为严格或有意义的基准）。

We also undertake a systematic study of “data contamination” – a growing problem when training high capacity models on datasets such as Common Crawl, which can potentially include content from test datasets simply because such content often exists on the web. In this paper we develop systematic tools to measure data contamination and quantify its distorting effects. Although we find that data contamination has a minimal effect on GPT-3’s performance on most datasets, we do identify a few datasets where it could be inflating results, and we either do not report results on these datasets or we note them with an asterisk, depending on the severity.

我们还对“数据污染”进行了系统研究——当在诸如 Common Crawl 这样的数据集上训练高容量模型时，这是一个日益严重的问题，因为测试数据集中的内容可能会被包含在内，仅仅因为这些内容通常存在于网络上。在本文中，我们开发了系统工具来测量数据污染并量化其扭曲效应。虽然我们发现数据污染对 GPT-3 在大多数数据集上的性能影响很小，但我们确实发现一些数据集可能会导致结果被高估，对于这些数据集，我们要么不报告结果，要么根据严重程度用星号标注它们。

In addition to all the above, we also train a series of smaller models (ranging from 125 million parameters to 13 billion parameters) in order to compare their performance to GPT-3 in the zero, one and few-shot settings. Broadly, for most tasks we find relatively smooth scaling with model capacity in all three settings; one notable pattern is that the gap between zero-, one-, and few-shot performance often grows with model capacity, perhaps suggesting that larger models are more proficient meta-learners.

除了上述所有内容之外，我们还训练了一系列较小的模型（从1.25亿参数到130亿参数不等），以便在零样本、一样本和少样本设定中将它们的性能与GPT-3进行比较。总体而言，对于大多数任务，我们发现在所有三种设置下，模型容量与性能的扩展相对平滑；一个显著的模式是，零-shot、单-shot 和少-shot 性能之间的差距通常随着模型容量的增加而增大，这或许表明更大的模型是更有能力的元学习者。

Finally, given the broad spectrum of capabilities displayed by GPT-3, we discuss concerns about bias, fairness, and broader societal impacts, and attempt a preliminary analysis of GPT-3’s characteristics in this regard.

最后，鉴于 GPT-3 展示的广泛能力，我们讨论了关于偏见、公平性以及更广泛社会影响的担忧，并尝试对 GPT-3 在这方面的特性进行初步分析。

The remainder of this paper is organized as follows. In Section 2, we describe our approach and methods for training GPT-3 and evaluating it. Section 3 presents results on the full range of tasks in the zero-, one- and few-shot settings. Section 4.1 addresses questions of data contamination (train-test overlap). Section 5 discusses limitations of GPT-3. Section 6 discusses broader impacts. Section 7 reviews related work and Section 8 concludes.

本文其余部分的组织结构如下。第2节，我们描述了训练GPT-3的方法以及评估方法。第3节呈现了在零样本、一样本和少样本设定下的全任务范围的结果。第4.1节讨论了数据污染（训练-测试重叠）的问题。第5节讨论了GPT-3的局限性。第6节讨论了更广泛的影响。第7节回顾相关工作，第8节总结全文。

2 Approach

2 方法（Approach）

Figure 2.1: Zero-shot, one-shot and few-shot, contrasted with traditional fine-tuning. The panels above show four methods for performing a task with a language model – fine-tuning is the traditional method, whereas zero-, one-, and few-shot, which we study in this work, require the model to perform the task with only forward passes at test time. We typically present the model with a few dozen examples in the few shot setting. Exact phrasings for all task descriptions, examples and prompts can be found in Appendix G.

图 2.1：零样本、单样本和少样本，与传统微调的对比。上图展示了使用语言模型完成任务的四种方法——微调是传统方法，而零样本、单样本和少样本（我们在本研究中探讨的方法）则只需在测试时进行前向传播即可完成任务。在少样本设定下，我们通常会向模型展示几十个示例。所有任务描述、示例和提示的确切表述可以在附录 G 中找到。

Our basic pre-training approach, including model, data, and training, is similar to the process described in [117], with relatively straightforward scaling up of the model size, dataset size and diversity, and length of training. Our use of in-context learning is also similar to [117], but in this work we systematically explore different settings for learning within the context. Therefore, we start this section by explicitly defining and contrasting the different settings that we will be evaluating GPT-3 on or could in principle evaluate GPT-3 on. These settings can be seen as lying on a spectrum of how much task-specific data they tend to rely on. Specifically, we can identify at least four points on this spectrum (see Figure 2.1 for an illustration):

我们的基本预训练方法，包括模型、数据和训练，类似于[117]中描述的过程，只是在模型规模、数据集规模和多样性以及训练时间上进行了相对直接的扩展。我们对上下文学习的使用也类似于[117]，但在这项工作中，我们系统地探索了在上下文学习的不同设置。因此，我们从明确地定义并对比将用于评估 GPT-3 或原则上可以用于评估 GPT-3 的不同设置开始本节。这些设置可以被看作位于依赖任务特定数据程度的光谱上。具体来说，我们至少可以在这个光谱上识别四个点（见图 2.1 的示意图）:

• • Fine-Tuning (FT) has been the most common approach in recent years, and involves updating the weights of a pre-trained model by training on a supervised dataset specific to the desired task. Typically thousands to hundreds of thousands of labeled examples are used. The main advantage of fine-tuning is strong performance on many benchmarks. The main disadvantages are the need for a new large dataset for every task, the potential for poor generalization out-of-distribution [88], and the potential to exploit spurious features of the training data [36, 91], potentially resulting in an unfair comparison with human performance. In this work we do not fine-tune GPT-3 because our focus is on task-agnostic performance, but GPT-3 can be fine-tuned in principle and this is a promising direction for future work.
• • Few-Shot (FS) is the term we will use in this work to refer to the setting where the model is given a few demonstrations of the task at inference time as conditioning [117], but no weight updates are allowed. As shown in Figure 2.1, for a typical dataset an example has a context and a desired completion (for example an English sentence and the French translation), and few-shot works by giving $K$ examples of context and completion, and then one final example of context, with the model expected to provide the completion. We typically set $K$ in the range of 10 to 100 as this is how many examples can fit in the model’s context window ($n_{\mathrm{ctx}}=2048$). The main advantages of few-shot are a major reduction in the need for task-specific data and reduced potential to learn an overly narrow distribution from a large but narrow fine-tuning dataset. The main disadvantage is that results from this method have so far been much worse than state-of-the-art fine-tuned models. Also, a small amount of task specific data is still required. As indicated by the name, few-shot learning as described here for language models is related to few-shot learning as used in other contexts in ML [45, 133] – both involve learning based on a broad distribution of tasks (in this case implicit in the pre-training data) and then rapidly adapting to a new task.
• • One-Shot (1S) is the same as few-shot except that only one demonstration is allowed, in addition to a natural language description of the task, as shown in Figure 1. The reason to distinguish one-shot from few-shot and zero-shot (below) is that it most closely matches the way in which some tasks are communicated to humans. For example, when asking humans to generate a dataset on a human worker service (for example Mechanical Turk), it is common to give one demonstration of the task. By contrast it is sometimes difficult to communicate the content or format of a task if no examples are given.
• • Zero-Shot (0S) is the same as one-shot except that no demonstrations are allowed, and the model is only given a natural language instruction describing the task. This method provides maximum convenience, potential for robustness, and avoidance of spurious correlations (unless they occur very broadly across the large corpus of pre-training data), but is also the most challenging setting. In some cases it may even be difficult for humans to understand the format of the task without prior examples, so this setting is in some cases “unfairly hard”. For example, if someone is asked to “make a table of world records for the 200m dash”, this request can be ambiguous, as it may not be clear exactly what format the table should have or what should be included (and even with careful clarification, understanding precisely what is desired can be difficult). Nevertheless, for at least some settings zero-shot is closest to how humans perform tasks – for example, in the translation example in Figure 2.1, a human would likely know what to do from just the text instruction.

• • 微调（FT） 是近年来最常见的方法，它通过在特定任务的监督数据集上训练来更新预训练模型的权重。通常使用数千到数十万标注示例。微调的主要优点是在许多基准测试中表现出色。主要的缺点是每项任务都需要新的大型数据集、在分布外可能出现的泛化能力差 [88]，以及可能利用训练数据的虚假特征 [36, 91]，这可能导致与人类表现的不公平比较。在这项工作中，我们没有对 GPT-3 进行微调，因为我们的重点是任务无关的性能，但原则上 GPT-3 是可以微调的，这是未来工作的一个有前途的方向。- • 少量示例(FS) 是我们在这项工作中将使用的术语，用于指模型在推理时收到任务的一些示例作为条件 [117]，但不允许进行权重更新。如图2.1所示，对于一个典型的数据集，一个示例具有上下文和期望的完成（例如一个英语句子及其法语翻译），而少量示例学习通过给出$K$个上下文和完成的示例，然后给出最后一个上下文示例，模型需要提供完成内容。我们通常将 $K$ 设置在 10 到 100 的范围内，因为这是模型上下文窗口（$n_{\mathrm{ctx}}=2048$）中可以容纳的示例数量。少样本学习的主要优势是显著减少对特定任务数据的需求，并降低从大而窄的微调数据集中学习到过于狭窄分布的可能性。主要的缺点是这种方法的结果迄今为止远远不如最先进的微调模型。此外，仍然需要少量特定任务的数据。顾名思义，这里描述的用于语言模型的小样本学习与在机器学习其他上下文中使用的小样本学习相关 [45, 133]——两者都涉及基于广泛任务分布（在本例中隐含于预训练数据中）进行学习，然后快速适应新任务。- • 一次性学习（1S） 与少量学习相同，只是只允许一个示例，外加任务的自然语言描述，如图1所示。将一次性学习与少量学习和零样本学习（如下）区分开的原因是，它最接近某些任务传达给人类的方式。例如，当要求人类在一个人工工作者服务平台（例如 Mechanical Turk）上生成数据集时，通常会给出一个任务示例。相比之下，如果没有给出示例，有时很难传达任务的内容或格式。 - • 零样本（0S） 与单样本相同，只是不允许提供示例，模型仅给出描述任务的自然语言指令。这种方法提供了最大的便利性、潜在的稳健性以及避免虚假相关性（除非它们在大规模预训练数据语料库中广泛出现），但它也是最具挑战性的配置。在某些情况下，如果没有先前的示例，人类甚至可能难以理解任务的格式，因此在某些情况下，这种设置是“极其困难的”。例如，如果有人被要求“制作一张200米短跑的世界纪录表格”，这个请求可能是模糊的，因为可能不清楚表格应该采用什么格式或应包含什么内容（即使经过仔细澄清，也很难准确理解所需的具体内容）。然而，至少在某些设置下，零样本最接近人类执行任务的方式——例如，在图2.1的翻译示例中，人类很可能仅从文本指令就知道该怎么做。

Figure 2.1 shows the four methods using the example of translating English to French. In this paper we focus on zero-shot, one-shot and few-shot, with the aim of comparing them not as competing alternatives, but as different problem settings which offer a varying trade-off between performance on specific benchmarks and sample efficiency. We especially highlight the few-shot results as many of them are only slightly behind state-of-the-art fine-tuned models. Ultimately, however, one-shot, or even sometimes zero-shot, seem like the fairest comparisons to human performance, and are important targets for future work.

图 2.1 展示了以英语翻译为法语为例的四种方法。在本文中，我们重点讨论零样本、一样本和少样本，目的是将它们进行比较时，不是作为相互竞争的选择，而是作为不同的问题设置，这些设置在特定基准的性能和样本效率之间提供了不同的权衡。我们特别强调少量示例的结果，因为其中许多仅略低于最先进的微调模型。然而，最终来看，一次示例甚至有时零示例，似乎才是与人类表现最公平的比较，并且是未来研究的重要目标。

Sections 2.1-2.3 below give details on our models, training data, and training process respectively. Section 2.4 discusses the details of how we do few-shot, one-shot, and zero-shot evaluations.

下面的第2.1至2.3节分别介绍了我们的模型、训练数据和训练过程的详细信息。第2.4节讨论了我们如何进行少样本、一样本和零样本评估的详细方法。

2.1 Model and Architectures

2.1 模型与架构（Model and Architectures）

We use the same model and architecture as GPT-2 [117], including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer [15]. To study the dependence of ML performance on model size, we train 8 different sizes of model, ranging over three orders of magnitude from 125 million parameters to 175 billion parameters, with the last being the model we call GPT-3. Previous work [57] suggests that with enough training data, scaling of validation loss should be approximately a smooth power law as a function of size; training models of many different sizes allows us to test this hypothesis both for validation loss and for downstream language tasks.

我们使用与 GPT-2 [117] 相同的模型和架构，包括其中描述的修改初始化、预归一化和可逆分词，唯一的例外是我们在变换器的各层中使用交替的密集和局部带状稀疏注意模式，类似于稀疏变换器 [15]。为了研究机器学习性能对模型规模的依赖性，我们训练了8种不同规模的模型，参数数量跨越三个数量级，从1.25亿到1750亿，最后一个是我们称之为GPT-3的模型。先前的工作 [57] 表明，拥有足够的训练数据时，验证损失的缩放应大致作为大小的平滑幂律函数；训练不同规模的模型可以让我们测试这一假设，无论是对于验证损失还是下游语言任务。

Model Name	$n_{\mathrm{params}}$	$n_{\mathrm{layers}}$	$d_{\mathrm{model}}$	$n_{\mathrm{heads}}$	$d_{\mathrm{head}}$	Batch Size	Learning Rate
GPT-3 Small	125M	12	768	12	64	0.5M	$6.0\times 10^{-4}$
GPT-3 Medium	350M	24	1024	16	64	0.5M	$3.0\times 10^{-4}$
GPT-3 Large	760M	24	1536	16	96	0.5M	$2.5\times 10^{-4}$
GPT-3 XL	1.3B	24	2048	24	128	1M	$2.0\times 10^{-4}$
GPT-3 2.7B	2.7B	32	2560	32	80	1M	$1.6\times 10^{-4}$
GPT-3 6.7B	6.7B	32	4096	32	128	2M	$1.2\times 10^{-4}$
GPT-3 13B	13.0B	40	5140	40	128	2M	$1.0\times 10^{-4}$
GPT-3 175B or “GPT-3”	175.0B	96	12288	96	128	3.2M	$0.6\times 10^{-4}$

Table 2.1: Sizes, architectures, and learning hyper-parameters (batch size in tokens and learning rate) of the models which we trained. All models were trained for a total of 300 billion tokens.

模型名称	$n_{\mathrm{params}}$	$n_{\mathrm{layers}}$	$d_{\mathrm{model}}$	$n_{\mathrm{heads}}$	$d_{\mathrm{head}}$	批量大小	学习率
GPT-3 小型	125M	12	768	12	64	0.5M	$6.0\times 10^{-4}$
GPT-3 中型	350M	24	1024	16	64	0.5M	$3.0\times 10^{-4}$
GPT-3 大型	760M	24	1536	16	96	0.5M	$2.5\times 10^{-4}$
GPT-3 XL	1.3B	24	2048	24	128	1M	$2.0\times 10^{-4}$
GPT-3 2.7B	2.7B	32	2560	32	80	1M	$1.6\times 10^{-4}$
GPT-3 6.7B	6.7B	32	4096	32	128	2M	$1.2\times 10^{-4}$
GPT-3 13B	13.0B	40	5140	40	128	2M	$1.0\times 10^{-4}$
GPT-3 175B 或 “GPT-3”	175.0B	96	12288	96	128	3.2M	$0.6\times 10^{-4}$
表 2.1：我们训练的模型的规模、架构和学习超参数（批次大小以令牌计，学习率）。所有模型的训练总量为 3000 亿令牌。

Table 2.1 shows the sizes and architectures of our 8 models. Here $n_{\mathrm{params}}$ is the total number of trainable parameters, $n_{\mathrm{layers}}$ is the total number of layers, $d_{\mathrm{model}}$ is the number of units in each bottleneck layer (we always have the feedforward layer four times the size of the bottleneck layer, $d_{\mathrm{ff}}$ $=4\ast d_{\mathrm{model}}$), and $d_{\mathrm{head}}$ is the dimension of each attention head. All models use a context window of $n_{\mathrm{ctx}}=2048$ tokens. We partition the model across GPUs along both the depth and width dimension in order to minimize data-transfer between nodes. The precise architectural parameters for each model are chosen based on computational efficiency and load-balancing in the layout of models across GPU’s. Previous work [57] suggests that validation loss is not strongly sensitive to these parameters within a reasonably broad range.

表2.1展示了我们8个模型的规模和架构。这里$n_{\mathrm{params}}$是可训练参数的总数，$n_{\mathrm{layers}}$是总层数，$d_{\mathrm{model}}$是每个瓶颈层的单元数（我们总是让前馈层的大小是瓶颈层的四倍，$d_{\mathrm{ff}}$ $=4\ast d_{\mathrm{model}}$），$d_{\mathrm{head}}$是每个注意力头的维度。所有模型使用 $n_{\mathrm{ctx}}=2048$ 个标记的上下文窗口。我们沿深度和宽度维度在 GPU 之间划分模型，以尽量减少节点间的数据传输。每个模型的具体架构参数是根据计算效率和在 GPU 上模型布局的负载均衡来选择的。之前的工作 [57] 表明，在合理宽的范围内，验证损失对这些参数并不敏感。

Figure 2.2: Total compute used during training. Based on the analysis in Scaling Laws For Neural Language Models [57] we train much larger models on many fewer tokens than is typical. As a consequence, although GPT-3 3B is almost 10x larger than RoBERTa-Large (355M params), both models took roughly 50 petaflop/s-days of compute during pre-training. Methodology for these calculations can be found in Appendix D.

图 2.2：训练期间使用的总计算量。根据《神经语言模型的规模定律》[57]中的分析，我们在远少于典型数量的标记上训练了更大的模型。因此，尽管 GPT-3 3B 的规模几乎是 RoBERTa-Large（3.55 亿参数）的 10 倍，但这两个模型在预训练期间所使用的计算量大约都是 50 petaflop/s-天。这些计算方法可以在附录 D 中找到。

2.2 Training Dataset

2.2 训练数据集（Training Dataset）

Datasets for language models have rapidly expanded, culminating in the Common Crawl dataset[^2] [116] constituting nearly a trillion words. This size of dataset is sufficient to train our largest models without ever updating on the same sequence twice. However, we have found that unfiltered or lightly filtered versions of Common Crawl tend to have lower quality than more curated datasets. Therefore, we took 3 steps to improve the average quality of our datasets: (1) we downloaded and filtered a version of CommonCrawl based on similarity to a range of high-quality reference corpora, (2) we performed fuzzy deduplication at the document level, within and across datasets, to prevent redundancy and preserve the integrity of our held-out validation set as an accurate measure of overfitting, and (3) we also added known high-quality reference corpora to the training mix to augment CommonCrawl and increase its diversity.

用于语言模型的数据集迅速扩展，最终形成了 Common Crawl 数据集[^2] [116]，其包含近一万亿个单词。这种规模的数据集足以训练我们最大的模型，而无需在同一序列上重复更新。然而，我们发现未经过滤或轻度过滤的 Common Crawl 版本的质量往往低于更精心策划的数据集。因此，我们采取了三步措施来提高数据集的平均质量：（1）我们根据与一系列高质量参考语料库的相似度下载并筛选了 CommonCrawl 的版本，（2）我们在文档级别对数据集内外进行了模糊去重，以防止冗余并保持我们保留的验证集的完整性，使其能够准确衡量过拟合情况，（3）我们还添加了已知的高质量参考资料将语料库加入训练混合中，以增强 CommonCrawl 并增加其多样性。

Details of the first two points (processing of Common Crawl) are described in Appendix A. For the third, we added several curated high-quality datasets, including an expanded version of the WebText dataset [117], collected by scraping links over a longer period of time, and first described in [57], two internet-based books corpora (Books1 and Books2) and English-language Wikipedia.

前两点（Common Crawl 的处理）的详细信息在附录 A 中描述。对于第三点，我们添加了几个精心挑选的高质量数据集，包括 WebText 数据集的扩展版本 [117]，该版本通过在更长时间内抓取链接收集，并首次在 [57] 中描述，以及两个基于互联网的图书语料库（Books1 和 Books2）和英文维基百科。

Table 2.2 shows the final mixture of datasets that we used in training. The CommonCrawl data was downloaded from 41 shards of monthly CommonCrawl covering 2016 to 2019, constituting 45TB of compressed plaintext before filtering and 570GB after filtering, roughly equivalent to 400 billion byte-pair-encoded tokens. Note that during training, datasets are not sampled in proportion to their size, but rather datasets we view as higher-quality are sampled more frequently, such that CommonCrawl and Books2 datasets are sampled less than once during training, but the other datasets are sampled 2-3 times. This essentially accepts a small amount of overfitting in exchange for higher quality training data.

表2.2显示了我们在训练中使用的最终数据集混合。CommonCrawl 数据来自 2016 到 2019 年的 41 个 CommonCrawl 月度分片，过滤前构成了 45TB 的压缩纯文本，过滤后为 570GB，大致相当于 4000 亿个字节对编码的标记。请注意，在训练过程中，数据集的采样不是按其大小成比例进行的，而是我们认为质量较高的数据集会被更频繁地采样，因此 CommonCrawl 和 Books2 数据集在训练过程中采样次数不到一次，而其他数据集则会被采样 2-3 次。这实际上是在接受少量过拟合的情况下，以换取更高质量的训练数据。

Dataset	Quantity (tokens)	Weight in training mix	Epochs elapsed when training for 300B tokens
Common Crawl (filtered)	410 billion	60%	0.44
WebText2	19 billion	22%	2.9
Books1	12 billion	8%	1.9
Books2	55 billion	8%	0.43
Wikipedia	3 billion	3%	3.4

Table 2.2: Datasets used to train GPT-3. “Weight in training mix” refers to the fraction of examples during training that are drawn from a given dataset, which we intentionally do not make proportional to the size of the dataset. As a result, when we train for 300 billion tokens, some datasets are seen up to 3.4 times during training while other datasets are seen less than once.

数据集	数量（标记）	训练混合中的权重	在训练 3000 亿标记时经过的周期数
Common Crawl（过滤后）	4100 亿	60%	0.44
WebText2	190 亿	22%	2.9
Books1	120 亿	8%	1.9
Books2	550 亿	8%	0.43
维基百科	30 亿	3%	3.4
表 2.2：用于训练 GPT-3 的数据集。“训练混合中的权重”是指训练期间从给定数据集中抽取的示例所占的比例，我们刻意不按数据集的大小来设置比例。因此，当我们训练 3000 亿个标记时，有些数据集在训练期间会被看到达 3.4 次，而其他数据集则不到一次。

A major methodological concern with language models pretrained on a broad swath of internet data, particularly large models with the capacity to memorize vast amounts of content, is potential contamination of downstream tasks by having their test or development sets inadvertently seen during pre-training. To reduce such contamination, we searched for and attempted to remove any overlaps with the development and test sets of all benchmarks studied in this paper. Unfortunately, a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model. In Section 4.1 we characterize the impact of the remaining overlaps, and in future work we will more aggressively remove data contamination.

对用广泛互联网数据预训练的语言模型而言，一个主要的方法学问题，尤其是具有记忆大量内容能力的大型模型，是在下游任务中可能受到污染，因为其测试集或开发集可能在预训练过程中被无意中见过。为了减少这种污染，我们搜索并尝试删除与本文研究的所有基准的开发集和测试集的任何重叠。不幸的是，过滤中的一个错误导致我们忽略了一些重叠，并且由于训练成本高昂，重新训练模型不可行。在第4.1节中，我们描述了剩余重叠的影响，并且在未来的工作中，我们将更积极地去除数据污染。

2.3 Training Process

2.3 训练过程（Training Process）

As found in [57, 85], larger models can typically use a larger batch size, but require a smaller learning rate. We measure the gradient noise scale during training and use it to guide our choice of batch size [85]. Table 2.1 shows the parameter settings we used. To train the larger models without running out of memory, we use a mixture of model parallelism within each matrix multiply and model parallelism across the layers of the network. All models were trained on V100 GPU’s on part of a high-bandwidth cluster provided by Microsoft. Details of the training process and hyperparameter settings are described in Appendix B.

如[57, 85]所示，更大的模型通常可以使用更大的批量大小，但需要更小的学习率。我们在训练过程中测量梯度噪声规模，并用它来指导我们选择批量大小[85]。表2.1显示了我们使用的参数设置。为了在训练更大模型时不耗尽内存，我们在每次矩阵乘法中使用模型并行，并在网络的各层之间使用模型并行。所有模型都在 V100 GPU 上训练，训练使用了微软提供的高带宽集群的一部分。训练过程和超参数设置的详细信息在附录 B 中描述。

2.4 Evaluation

2.4 评测（Evaluation）

For few-shot learning, we evaluate each example in the evaluation set by randomly drawing $K$ examples from that task’s training set as conditioning, delimited by 1 or 2 newlines depending on the task. For LAMBADA and Storycloze there is no supervised training set available so we draw conditioning examples from the development set and evaluate on the test set. For Winograd (the original, not SuperGLUE version) there is only one dataset, so we draw conditioning examples directly from it.

对于少样本学习，我们通过从该任务的训练集中随机抽取$K$个示例作为条件来评估评估集中的每个示例，根据任务的不同，用1个或2个换行符分隔。对于LAMBADA和Storycloze，没有可用的监督训练集，因此我们从开发集中抽取条件示例，并在测试集上进行评估。对于 Winograd（原版，而不是 SuperGLUE 版本），只有一个数据集，因此我们直接从中提取条件示例。

$K$ can be any value from 0 to the maximum amount allowed by the model’s context window, which is $n_{\mathrm{ctx}}=2048$ for all models and typically fits $10$ to $100$ examples. Larger values of $K$ are usually but not always better, so when a separate development and test set are available, we experiment with a few values of $K$ on the development set and then run the best value on the test set. For some tasks (see Appendix G) we also use a natural language prompt in addition to (or for $K=0$, instead of) demonstrations.

$K$ 可以是从 0 到模型上下文窗口允许的最大值的任何数值，对于所有模型，该最大值为 $n_{\mathrm{ctx}}=2048$，通常可以容纳 $10$ 到 $100$ 个示例。$K$ 的较大值通常但不总是更好，因此当有单独的开发集和测试集时，我们会在开发集上尝试 $K$ 的几个值，然后在测试集上运行表现最好的值。对于某些任务（见附录 G），我们还使用自然语言提示，作为示例的补充（或对于 $K=0$，替代示例使用）。

On tasks that involve choosing one correct completion from several options (multiple choice), we provide $K$ examples of context plus correct completion, followed by one example of context only, and compare the LM likelihood of each completion. For most tasks we compare the per-token likelihood (to normalize for length), however on a small number of datasets (ARC, OpenBookQA, and RACE) we gain additional benefit as measured on the development set by normalizing by the unconditional probability of each completion, by computing $\frac{P(\mathrm{completion}|\mathrm{context})}{P(\mathrm{completion}|\mathrm{answer\_context})}$, where $\mathrm{answer\_context}$ is the string "Answer: " or "A: " and is used to prompt that the completion should be an answer but is otherwise generic.

在涉及从多个选项中选择一个正确完成的任务（多项选择）中，我们提供$K$个包含上下文及正确完成的示例，随后提供一个仅包含上下文的示例，并比较语言模型对每个完成的可能性。对于大多数任务，我们比较每个标记的可能性（以对长度进行归一化），然而在少量数据集（ARC、OpenBookQA 和 RACE）上，我们通过对每个完成的无条件概率进行归一化，在开发集上获得了额外的好处，通过计算 $\frac{P(\mathrm{completion}|\mathrm{context})}{P(\mathrm{completion}|\mathrm{answer\_context})}$，其中 $\mathrm{answer\_context}$ 是字符串“Answer: ”或“A:”"" 用于提示完成内容应该是一个答案，但除此之外是通用的。

On tasks that involve binary classification, we give the options more semantically meaningful names (e.g. “True” or “False” rather than 0 or 1) and then treat the task like multiple choice; we also sometimes frame the task similar to what is done by [116] (see Appendix G) for details.

对于涉及二分类的任务，我们给选项赋予更有语义意义的名称（例如“真”或“假”而不是0或1），然后将任务视为多项选择；我们有时也将任务以类似于[116]所做的方式来框架（详情见附录G）。

On tasks with free-form completion, we use beam search with the same parameters as [116]: a beam width of 4 and a length penalty of $\alpha=0.6$. We score the model using F1 similarity score, BLEU, or exact match, depending on what is standard for the dataset at hand.

在自由形式完成的任务中，我们使用与[116]相同参数的束搜索：束宽为4，长度惩罚为$\alpha=0.6$。我们根据所使用数据集的标准，通过F1相似度评分、BLEU或完全匹配来对模型进行评分。

Final results are reported on the test set when publicly available, for each model size and learning setting (zero-, one-, and few-shot). When the test set is private, our model is often too large to fit on the test server, so we report results on the development set. We do submit to the test server on a small number of datasets (SuperGLUE, TriviaQA, PiQa) where we were able to make submission work, and we submit only the 200B few-shot results, and report development set results for everything else.

当测试集公开时，会报告每种模型规模和学习设置（零样本、单样本和少样本）下的最终结果。当测试集是私有的时，我们的模型通常太大而无法在测试服务器上运行，因此我们会报告开发集上的结果。我们确实在少量数据集（SuperGLUE、TriviaQA、PiQa）上提交到测试服务器，在这些数据集上我们能够完成提交工作，并且我们只提交了 200B 少样本的结果，对其他所有情况报告开发集结果。

3 Results

3 结果（Results）

Figure 3.1: Smooth scaling of performance with compute. Performance (measured in terms of cross-entropy validation loss) follows a power-law trend with the amount of compute used for training. The power-law behavior observed in [57] continues for an additional two orders of magnitude with only small deviations from the predicted curve. For this figure, we exclude embedding parameters from compute and parameter counts.

图 3.1：性能随计算量平滑增长。 性能（以交叉熵验证损失衡量）随训练所用计算量遵循幂律趋势。[57] 中观察到的幂律行为在额外两个数量级上继续存在，仅有与预测曲线的小偏差。对于本图，我们在计算量和参数计数中排除了嵌入参数。

In Figure 3.1 we display training curves for the 8 models described in Section 2. For this graph we also include 6 additional extra-small models with as few as 100,000 parameters. As observed in [57], language modeling performance follows a power-law when making efficient use of training compute. After extending this trend by two more orders of magnitude, we observe only a slight (if any) departure from the power-law. One might worry that these improvements in cross-entropy loss come only from modeling spurious details of our training corpus. However, we will see in the following sections that improvements in cross-entropy loss lead to consistent performance gains across a broad spectrum of natural language tasks.

在图 3.1 中，我们展示了第 2 节中描述的 8 个模型的训练曲线。对于此图，我们还包括了 6 个额外的小型模型，其参数数量少至 100,000 个。正如 [57] 所观察到的，当有效利用训练计算时，语言建模性能遵循幂律。在将这一趋势扩展两个数量级后，我们仅观察到与幂律略有（如果有的话）偏离。有人可能会担心，这些交叉熵损失的改进只是来自于对我们训练语料库中虚假细节的建模。然而，我们将在以下章节看到，交叉熵损失的改进会在广泛的自然语言任务中带来一致的性能提升。

Below, we evaluate the 8 models described in Section 2 (the 175 billion parameter parameter GPT-3 and 7 smaller models) on a wide range of datasets. We group the datasets into 9 categories representing roughly similar tasks.

下面，我们评估了第2节中描述的8个模型（1750亿参数的GPT-3和7个较小的模型）在各种数据集上的表现。我们将这些数据集分为9类，大致代表相似的任务。

In Section 3.1 we evaluate on traditional language modeling tasks and tasks that are similar to language modeling, such as Cloze tasks and sentence/paragraph completion tasks. In Section 3.2 we evaluate on “closed book” question answering tasks: tasks which require using the information stored in the model’s parameters to answer general knowledge questions. In Section 3.3 we evaluate the model’s ability to translate between languages (especially one-shot and few-shot). In Section 3.4 we evaluate the model’s performance on Winograd Schema-like tasks. In Section 3.5 we evaluate on datasets that involve commonsense reasoning or question answering. In Section 3.6 we evaluate on reading comprehension tasks, in Section 3.7 we evaluate on the SuperGLUE benchmark suite, and in 3.8 we briefly explore NLI. Finally, in Section 3.9, we invent some additional tasks designed especially to probe in-context learning abilities – these tasks focus on on-the-fly reasoning, adaptation skills, or open-ended text synthesis. We evaluate all tasks in the few-shot, one-shot, and zero-shot settings.

在第3.1节中，我们对传统的语言建模任务以及类似于语言建模的任务进行评估，例如完形填空任务和句子/段落完成任务。在第3.2节中，我们对“闭卷”问答任务进行评估：这些任务需要使用存储在模型参数中的信息来回答常识性问题。在第3.3节中，我们评估模型在语言之间翻译的能力（尤其是一-shot和少-shot）。在第3.4节中，我们评估模型在类似Winograd Schema任务中的表现。在第3.5节中，我们评估涉及常识推理或问答的数据集。在第3.6节中，我们评估阅读理解任务，在第3.7节中，我们评估SuperGLUE基准套件，在第3.8节中我们简要探讨自然语言推理（NLI）。最后，在第3.9节中，我们发明了一些专门设计用来探测上下文学习能力的额外任务——这些任务侧重于即时推理、适应能力或开放式文本生成。我们在少样本、单样本和零样本设定下评估所有任务。

3.1 Language Modeling, Cloze, and Completion Tasks

3.1 语言建模、完形与补全任务（Language Modeling, Cloze, and Completion Tasks）

In this section we test GPT-3’s performance on the traditional task of language modeling, as well as related tasks that involve predicting a single word of interest, completing a sentence or paragraph, or choosing between possible completions of a piece of text.

在本节中，我们测试 GPT-3 在传统语言建模任务上的表现，以及涉及预测一个感兴趣的单词、完成一句话或一段文字，或在文本的可能完成选项之间进行选择的相关任务。

3.1.1 Language Modeling

3.1.1 语言建模

We calculate zero-shot perplexity on the Penn Tree Bank (PTB) [86] dataset measured in [117]. We omit the 4 Wikipedia-related tasks in that work because they are entirely contained in our training data, and we also omit the one-billion word benchmark due to a high fraction of the dataset being contained in our training set. PTB escapes these issues due to predating the modern internet. Our largest model sets a new SOTA on PTB by a substantial margin of 15 points, achieving a perplexity of 20.50. Note that since PTB is a traditional language modeling dataset it does not have a clear separation of examples to define one-shot or few-shot evaluation around, so we measure only zero-shot.

我们在宾州树库（PTB）[86] 数据集上计算零样本困惑度，如 [117] 中所测量的。我们省略了该工作中的 4 个与维基百科相关的任务，因为它们完全包含在我们的训练数据中，我们还省略了十亿词基准测试，因为数据集中的很大一部分包含在我们的训练集中。PTB 避免了这些问题，因为它早于现代互联网的出现。我们最大的模型在 PTB 上以显著的 15 点优势创造了新的 SOTA，困惑度达到了 20.50。请注意，由于 PTB 是一个传统的语言建模数据集，它没有明确的样本分隔来定义单次或少次示例的评估，因此我们只进行零次示例评估。

Setting	PTB
SOTA (Zero-Shot)	35.8a
GPT-3 Zero-Shot	20.5

Table 3.1: Zero-shot results on PTB language modeling dataset. Many other common language modeling datasets are omitted because they are derived from Wikipedia or other sources which are included in GPT-3’s training data. a[117]

设置	PTB
SOTA（零样本）	35.8a
GPT-3 零样本	20.5

表 3.1：PTB 语言建模数据集的零样本结果。 许多其他常见的语言建模数据集被省略，因为它们来源于 Wikipedia 或其他包含在 GPT-3 训练数据中的来源。a[117]

3.1.2 LAMBADA

Setting	LAMBADA (acc)	LAMBADA (ppl)	StoryCloze (acc)	HellaSwag (acc)
SOTA	68.0a	8.63b	91.8c	85.6d
GPT-3 Zero-Shot	76.2	3.00	83.2	78.9
GPT-3 One-Shot	72.5	3.35	84.7	78.1
GPT-3 Few-Shot	86.4	1.92	87.7	79.3

Table 3.2: Performance on cloze and completion tasks. GPT-3 significantly improves SOTA on LAMBADA while achieving respectable performance on two difficult completion prediction datasets. a[132] b[117] c[64] d[63]

设置	LAMBADA (准确率)	LAMBADA (困惑度)	StoryCloze (准确率)	HellaSwag (准确率)
SOTA	68.0a	8.63b	91.8c	85.6d
GPT-3 零样本	76.2	3.00	83.2	78.9
GPT-3 一样本	72.5	3.35	84.7	78.1
GPT-3 少样本	86.4	1.92	87.7	79.3
表 3.2：完形填空和完成任务的性能。 GPT-3 在 LAMBADA 上显著提升了最新水平（SOTA），同时在两个困难的完成预测数据集上也取得了可观的表现。a[132] b[117] c[64] d[63]

Figure 3.2: On LAMBADA, the few-shot capability of language models results in a strong boost to accuracy. GPT-3 2.7B outperforms the SOTA 17B parameter Turing-NLG [132] in this setting, and GPT-3 175B advances the state of the art by 18%. Note zero-shot uses a different format from one-shot and few-shot as described in the text.

图 3.2：在 LAMBADA 数据集上，语言模型的少样本能力显著提升了准确率。在这种设置下，GPT-3 2.7B 的表现超过了拥有 170 亿参数的 SOTA Turing-NLG [132]，而 GPT-3 175B 将最新技术水平提高了 18%。请注意，零样本使用的格式与文中描述的一次样本和少样本不同。

The LAMBADA dataset [99] tests the modeling of long-range dependencies in text – the model is asked to predict the last word of sentences which require reading a paragraph of context. It has recently been suggested that the continued scaling of language models is yielding diminishing returns on this difficult benchmark. [9] reflect on the small 1.5% improvement achieved by a doubling of model size between two recent state of the art results ([125] and [132]) and argue that “continuing to expand hardware and data sizes by orders of magnitude is not the path forward”. We find that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of 8% over the previous state of the art.

LAMBADA 数据集 [99] 测试文本中长距离依赖的建模——模型需要在阅读一段上下文后预测句子的最后一个单词。最近有人提出，语言模型的持续扩展在这一困难基准测试上正产生收益递减。[9] 回顾了在两项近期最先进成果之间将模型规模翻倍所取得的小幅 1.5% 改进（[125] 和 [132]），并认为“继续按数量级增加硬件和数据规模并不是前进的道路”。我们发现这条道路仍然很有前景，在零样本设定中，GPT-3 在 LAMBADA 测试中取得了 76% 的成绩，比之前的最先进水平提高了 8%。

LAMBADA is also a demonstration of the flexibility of few-shot learning as it provides a way to address a problem that classically occurs with this dataset. Although the completion in LAMBADA is always the last word in a sentence, a standard language model has no way of knowing this detail. It thus assigns probability not only to the correct ending but also to other valid continuations of the paragraph. This problem has been partially addressed in the past with stop-word filters [117] (which ban “continuation” words). The few-shot setting instead allows us to “frame” the task as a cloze-test and allows the language model to infer from examples that a completion of exactly one word is desired. We use the following fill-in-the-blank format:

LAMBADA 也是少样本学习灵活性的一种展示，因为它提供了一种方法来解决在该数据集中典型出现的问题。尽管 LAMBADA 中的填空总是句子的最后一个词，标准的语言模型却无法知道这一细节。因此，它不仅为正确的结尾分配概率，还为段落的其他有效延续分配概率。这个问题在过去已经通过停用词过滤器[117]部分解决（它们禁止使用“连续”词）。而少样本设定则允许我们将任务“构造成”完形填空测试，并允许语言模型从示例中推断出所需的正好一个词的完成。我们使用以下填空格式：

Alice was friends with Bob. Alice went to visit her friend     . $\to$ Bob

爱丽丝和鲍勃是朋友。爱丽丝去看她的朋友鲍勃。

George bought some baseball equipment, a ball, a glove, and a     . $\to$

乔治买了一些棒球装备，一只球，一只手套，和一个    。$\to$

When presented with examples formatted this way, GPT-3 achieves 86.4% accuracy in the few-shot setting, an increase of over 18% from the previous state-of-the-art. We observe that few-shot performance improves strongly with model size. While this setting decreases the performance of the smallest model by almost 20%, for GPT-3 it improves accuracy by 10%. Finally, the fill-in-blank method is not effective one-shot, where it always performs worse than the zero-shot setting. Perhaps this is because all models still require several examples to recognize the pattern.

当呈现以这种方式格式化的示例时，GPT-3 在少量示例设定下取得了 86.4% 的准确率，比之前的最先进水平提高了超过 18%。我们观察到，少量示例的性能随着模型规模的增大而显著提升。虽然这种设定使最小模型的性能下降了近 20%，但对于 GPT-3 来说，准确率提高了 10%。最后，填空法并不是一次就有效的方法，它的表现总是比零样本设定要差。也许这是因为所有模型仍然需要几个示例来识别模式。

One note of caution is that an analysis of test set contamination identified that a significant minority of the LAMBADA dataset appears to be present in our training data – however analysis performed in Section 4.1 suggests negligible impact on performance.

需要注意的一点是，对测试集污染的分析发现，LAMBADA 数据集中的一小部分似乎出现在我们的训练数据中——然而，第 4.1 节的分析表明，这对性能影响可以忽略不计。

3.1.3 HellaSwag

The HellaSwag dataset [140] involves picking the best ending to a story or set of instructions. The examples were adversarially mined to be difficult for language models while remaining easy for humans (who achieve 95.6% accuracy). GPT-3 achieves 78.1% accuracy in the one-shot setting and 79.3% accuracy in the few-shot setting, outperforming the 75.4% accuracy of a fine-tuned 1.5B parameter language model [141] but still a fair amount lower than the overall SOTA of 85.6% achieved by the fine-tuned multi-task model ALUM.

HellaSwag 数据集 [140] 涉及选择故事或一组指令的最佳结局。这些示例是经过对抗性挖掘的，对语言模型来说很难，同时对人类来说仍然容易（人类的准确率达到 95.6%）。GPT-3 在单次示例设置中达到了 78.1% 的准确率，在少量示例设置中达到了 79.3% 的准确率，优于经过微调的 15 亿参数语言模型的 75.4% 准确率 [141]，但仍明显低于经过微调的多任务模型 ALUM 达到的整体 SOTA 85.6% 的准确率。

3.1.4 StoryCloze

We next evaluate GPT-3 on the StoryCloze 2016 dataset [83], which involves selecting the correct ending sentence for five-sentence long stories. Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot setting (with $K=70$). This is still 4.1% lower than the fine-tuned SOTA using a BERT based model [64] but improves over previous zero-shot results by roughly 10%.

接下来，我们在 StoryCloze 2016 数据集 [83] 上评估 GPT-3，该数据集涉及为五句长的故事选择正确的结尾句。在这里，GPT-3 在零样本设定下达到 83.2%，在少样本设定下（使用 $K=70$）达到 87.7%。这仍然比使用基于 BERT 的模型进行微调的 SOTA 低 4.1% [64]，但相比之前的零样本结果提高了约 10%。

3.2 Closed Book Question Answering

3.2 闭卷问答（Closed Book Question Answering）

Setting	NaturalQS	WebQS	TriviaQA
RAG (Fine-tuned, Open-Domain) [75]	44.5	45.5	68.0
T5-11B+SSM (Fine-tuned, Closed-Book) [115]	36.6	44.7	60.5
T5-11B (Fine-tuned, Closed-Book)	34.5	37.4	50.1
GPT-3 Zero-Shot	14.6	14.4	64.3
GPT-3 One-Shot	23.0	25.3	68.0
GPT-3 Few-Shot	29.9	41.5	71.2

Table 3.3: Results on three Open-Domain QA tasks. GPT-3 is shown in the few-, one-, and zero-shot settings, as compared to prior SOTA results for closed book and open domain settings. TriviaQA few-shot result is evaluated on the wiki split test server.

设置	NaturalQS	WebQS	TriviaQA
RAG（微调，开放领域）[75]	44.5	45.5	68.0
T5-11B+SSM（微调，闭卷）[115]	36.6	44.7	60.5
T5-11B（微调，闭卷）	34.5	37.4	50.1
GPT-3 零样本	14.6	14.4	64.3
GPT-3 单样本	23.0	25.3	68.0
GPT-3 少样本	29.9	41.5	71.2
表 3.3：三个开放域问答任务的结果。 GPT-3 在少样本、一样本和零样本设定下的表现，与之前封闭卷本和开放域设置的最新 SOTA 结果进行比较。TriviaQA 的少样本结果在 wiki 分割测试服务器上评估。

Figure 3.3: On TriviaQA GPT3’s performance grows smoothly with model size, suggesting that language models continue to absorb knowledge as their capacity increases. One-shot and few-shot performance make significant gains over zero-shot behavior, matching and exceeding the performance of the SOTA fine-tuned open-domain model, RAG [75]

图 3.3：在 TriviaQA 上，GPT3 的性能随着模型规模的增长而平滑提升，这表明语言模型随着其容量的增加继续吸收知识。一次性和少量示例的性能相较于零示例有显著提升，达到并超越了最先进的经过微调的开放领域模型 RAG [75] 的性能。

In this section we measure GPT-3’s ability to answer questions about broad factual knowledge. Due to the immense amount of possible queries, this task has normally been approached by using an information retrieval system to find relevant text in combination with a model which learns to generate an answer given the question and the retrieved text. Since this setting allows a system to search for and condition on text which potentially contains the answer it is denoted “open-book”. [115] recently demonstrated that a large language model can perform surprisingly well directly answering the questions without conditioning on auxilliary information. They denote this more restrictive evaluation setting as “closed-book”. Their work suggests that even higher-capacity models could perform even better and we test this hypothesis with GPT-3. We evaluate GPT-3 on the 3 datasets in [115]: Natural Questions [58], WebQuestions [5], and TriviaQA [49], using the same splits. Note that in addition to all results being in the closed-book setting, our use of few-shot, one-shot, and zero-shot evaluations represent an even stricter setting than previous closed-book QA work: in addition to external content not being allowed, fine-tuning on the Q&A dataset itself is also not permitted.

在本节中，我们评估 GPT-3 回答广泛事实性知识问题的能力。由于可能的查询数量庞大，这项任务通常通过使用信息检索系统来查找相关文本，并结合一个模型，该模型学习在给定问题和检索到的文本后生成答案。由于此设置允许系统搜索并以潜在包含答案的文本为条件，因此被称为“开卷”。[115] 最近展示了大型语言模型可以直接回答问题，而无需依赖辅助信息，且表现出令人惊讶的良好。他们将这种更严格的评估设置称为“闭卷”。他们的工作表明，即使是更高容量的模型也可能表现得更好，我们使用 GPT-3 来测试这一假设。我们在 [115] 中的 3 个数据集上评估 GPT-3：Natural Questions [58]、WebQuestions [5] 和 TriviaQA [49]，使用相同的数据划分。请注意，除了所有结果都基于闭卷环境之外，我们使用的少量示例、一示例和零示例评估表示了比以往闭卷问答研究更严格的设置：除了不允许使用外部内容之外，也不允许对问答数据集本身进行微调。

The results for GPT-3 are shown in Table 3.3. On TriviaQA, we achieve 64.3% in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting. The zero-shot result already outperforms the fine-tuned T5-11B by 14.2%, and also outperforms a version with Q&A tailored span prediction during pre-training by 3.8%. The one-shot result improves by 3.7% and matches the SOTA for an open-domain QA system which not only fine-tunes but also makes use of a learned retrieval mechanism over a 15.3B parameter dense vector index of 21M documents [75]. GPT-3’s few-shot result further improves performance another 3.2% beyond this.

GPT-3 的结果如表 3.3 所示。在 TriviaQA 数据集上，我们在零样本设定下取得了 64.3%，在单样本设定下取得了 68.0%，在少样本设定下取得了 71.2%。零样本结果已经比经过微调的 T5-11B 高出 14.2%，同时也比在预训练期间采用 Q&A 定制跨度预测的版本高出 3.8%。单样本结果提高了 3.7%，并达到了开放域问答系统的最新水平，该系统不仅进行了微调，还利用了对 21M 文档的 15.3B 参数密集向量索引的学习检索机制 [75]。GPT-3 的少量示例结果在此基础上进一步提高了 3.2% 的性能。

On WebQuestions (WebQs), GPT-3 achieves 14.4% in the zero-shot setting, 25.3% in the one-shot setting, and 41.5% in the few-shot setting. This compares to 37.4% for fine-tuned T5-11B, and 44.7% for fine-tuned T5-11B+SSM, which uses a Q&A-specific pre-training procedure. GPT-3 in the few-shot setting approaches the performance of state-of-the-art fine-tuned models. Notably, compared to TriviaQA, WebQS shows a much larger gain from zero-shot to few-shot (and indeed its zero-shot and one-shot performance are poor), perhaps suggesting that the WebQs questions and/or the style of their answers are out-of-distribution for GPT-3. Nevertheless, GPT-3 appears able to adapt to this distribution, recovering strong performance in the few-shot setting.

在 WebQuestions（WebQs）上，GPT-3 在零样本设定下达到 14.4%，在单样本设定下达到 25.3%，在少样本设定下达到 41.5%。相比之下，经过微调的 T5-11B 达到 37.4%，经过微调且使用问答特定预训练程序的 T5-11B+SSM 达到 44.7%。GPT-3 在少样本设定下的表现接近最先进的微调模型的性能。值得注意的是，与 TriviaQA 相比，WebQS 在从零样本到少样本的提升上要大得多（实际上其零样本和单样本表现较差），这或许表明 WebQS 的问题和/或答案风格对于 GPT-3 来说是分布外的。然而，GPT-3 似乎能够适应这种分布，在少样本设定中恢复强劲的表现。

On Natural Questions (NQs) GPT-3 achieves 14.6% in the zero-shot setting, 23.0% in the one-shot setting, and 29.9% in the few-shot setting, compared to 36.6% for fine-tuned T5 11B+SSM. Similar to WebQS, the large gain from zero-shot to few-shot may suggest a distribution shift, and may also explain the less competitive performance compared to TriviaQA and WebQS. In particular, the questions in NQs tend towards very fine-grained knowledge on Wikipedia specifically which could be testing the limits of GPT-3’s capacity and broad pretraining distribution.

在自然问题（NQs）上，GPT-3 在零样本设定下的表现为 14.6%，在单样本设定下为 23.0%，在少样本设定下为 29.9%，而经过微调的 T5 11B+SSM 表现为 36.6%。类似于 WebQS，从零样本到少样本的大幅提升可能表明存在分布偏移，也可能解释了其相比 TriviaQA 和 WebQS 表现不那么具有竞争力的原因。特别是，NQs中的问题倾向于对维基百科上的非常细粒度的知识，这可能在测试GPT-3的能力和广泛的预训练分布的极限。

Overall, on one of the three datasets GPT-3’s one-shot matches the open-domain fine-tuning SOTA. On the other two datasets it approaches the performance of the closed-book SOTA despite not using fine-tuning. On all 3 datasets, we find that performance scales very smoothly with model size (Figure 3.3 and Appendix H Figure H.7), possibly reflecting the idea that model capacity translates directly to more ‘knowledge’ absorbed in the parameters of the model.

总体而言，在三个数据集中的一个上，GPT-3 的一次性学习与开放领域微调的最先进水平相匹配。在另外两个数据集上，尽管没有使用微调，它的表现接近闭卷最先进水平。在所有三个数据集上，我们发现性能随着模型规模的增加而非常平滑地提升（图3.3和附录H图H.7），这可能反映了模型容量直接转化为模型参数中吸收了更多“知识”的这一观点。

3.3 Translation

3.3 翻译（Translation）

For GPT-2 a filter was used on a multilingual collection of documents to produce an English only dataset due to capacity concerns. Even with this filtering GPT-2 showed some evidence of multilingual capability and performed non-trivially when translating between French and English despite only training on 10 megabytes of remaining French text. Since we increase the capacity by over two orders of magnitude from GPT-2 to GPT-3, we also expand the scope of the training dataset to include more representation of other languages, though this remains an area for further improvement. As discussed in 2.2 the majority of our data is derived from raw Common Crawl with only quality-based filtering. Although GPT-3’s training data is still primarily English (93% by word count), it also includes 7% of text in other languages. These languages are documented in the supplemental material. In order to better understand translation capability, we also expand our analysis to include two additional commonly studied languages, German and Romanian.

对于 GPT-2，由于容量问题，对多语言文档集合使用了过滤器，以生成仅包含英文的数据集。即使经过这种过滤，GPT-2 仍显示出一定的多语言能力，并且在法语与英语之间的翻译中表现不凡，尽管它只在剩余的 10 兆字节法语文本上进行了训练。由于我们将容量从 GPT-2 提升了两个数量级以上到 GPT-3，我们也扩大了训练数据集的范围，以包含更多其他语言的表现，虽然这仍然是一个需要进一步改进的领域。如 2.2 所讨论，我们的大部分数据来自原始 Common Crawl，仅进行了基于质量的过滤。尽管 GPT-3 的训练数据仍然主要是英文（按单词数计占 93%），它也包括 7% 的其他语言文本。这些语言在补充材料中有记录。为了更好地理解翻译能力，我们还将分析扩展到包括两种常见研究的语言：德语和罗马尼亚语。

Existing unsupervised machine translation approaches often combine pretraining on a pair of monolingual datasets with back-translation [123] to bridge the two languages in a controlled way. By contrast, GPT-3 learns from a blend of training data that mixes many languages together in a natural way, combining them on a word, sentence, and document level. GPT-3 also uses a single training objective which is not customized or designed for any task in particular. However, our one / few-shot settings aren’t strictly comparable to prior unsupervised work since they make use of a small amount of paired examples (1 or 64). This corresponds to up to a page or two of in-context training data.

现有的无监督机器翻译方法通常将对一对单语数据集的预训练与回译 [123] 结合起来，以一种可控的方式桥接两种语言。相比之下，GPT-3 从混合了多种语言的训练数据中学习，并以自然的方式在单词、句子和文档层面上将它们结合起来。GPT-3 也使用单一的训练目标，这一目标并不是为任何特定任务定制或设计的。然而，我们的一次/少量示例设置并不能与之前的无监督工作严格比较，因为它们使用了一小部分配对示例（1 个或 64 个）。这相当于最多一两页的上下文训练数据。

Setting	En$\to$Fr	Fr$\to$En	En$\to$De	De$\to$En	En$\to$Ro	Ro$\to$En
SOTA (Supervised)	45.6a	35.0 b	41.2c	40.2d	38.5e	39.9e
XLM [61]	33.4	33.3	26.4	34.3	33.3	31.8
MASS [127]	37.5	34.9	28.3	35.2	35.2	33.1
mBART [66]	-	-	29.8	34.0	35.0	30.5
GPT-3 Zero-Shot	25.2	21.2	24.6	27.2	14.1	19.9
GPT-3 One-Shot	28.3	33.7	26.2	30.4	20.6	38.6
GPT-3 Few-Shot	32.6	39.2	29.7	40.6	21.0	39.5

Table 3.4: Few-shot GPT-3 outperforms previous unsupervised NMT work by 5 BLEU when translating into English reflecting its strength as an English LM. We report BLEU scores on the WMT’14 Fr$\leftrightarrow$En, WMT’16 De$\leftrightarrow$En, and WMT’16 Ro$\leftrightarrow$En datasets as measured by multi-bleu.perl with XLM’s tokenization in order to compare most closely with prior unsupervised NMT work. SacreBLEUf [101] results reported in Appendix H. Underline indicates an unsupervised or few-shot SOTA, bold indicates supervised SOTA with relative confidence. a[29] b[23] c[136] d[95] e[66] f[SacreBLEU signature: BLEU+case.mixed+numrefs.1+smooth.exp+tok.intl+version.1.2.20]

设置	英$\to$法	法$\to$英	英$\to$德	德$\to$英	英$\to$罗	罗$\to$英
SOTA（监督）	45.6a	35.0 b	41.2c	40.2d	38.5e	39.9e
XLM [61]	33.4	33.3	26.4	34.3	33.3	31.8
MASS [127]	37.5	34.9	28.3	35.2	35.2	33.1
mBART [66]	-	-	29.8	34.0	35.0	30.5
GPT-3 零样本	25.2	21.2	24.6	27.2	14.1	19.9
GPT-3 一次性	28.3	33.7	26.2	30.4	20.6	38.6
GPT-3 少量示例	32.6	39.2	29.7	40.6	21.0	39.5
表3.4：少量示例的GPT-3在翻译成英语时，比以往的无监督NMT工作高出5个BLEU分数，这反映了它作为英语语言模型的优势。 我们在WMT’14 Fr$\leftrightarrow$En、WMT’16 De$\leftrightarrow$En和WMT’16 Ro$\leftrightarrow$En数据集上报告了BLEU分数，使用多重bleu.perl结合XLM的分词方式，以便与之前的无监督NMT工作进行最接近的比较。SacreBLEUf [101] 结果见附录H。下划线表示无监督或少量示例的SOTA，加粗表示有监督SOTA且相对可靠。a[29] b[23] c[136] d[95] e[66] f[SacreBLEU签名：BLEU+case.mixed+numrefs.1+smooth.exp+tok.intl+version.1.2.20]

Figure 3.4: Few-shot translation performance on 6 language pairs as model capacity increases. There is a consistent trend of improvement across all datasets as the model scales, and as well as tendency for translation into English to be stronger than translation from English.

图 3.4：随着模型容量的增加，在 6 对语言对上的少样本翻译性能。随着模型规模的扩大，所有数据集都有一致的性能提升趋势，同时翻译成英语的性能往往比从英语翻译出的性能更强。

Results are shown in Table 3.4. Zero-shot GPT-3, which only receives on a natural language description of the task, still underperforms recent unsupervised NMT results. However, providing only a single example demonstration for each translation task improves performance by over 7 BLEU and nears competitive performance with prior work. GPT-3 in the full few-shot setting further improves another 4 BLEU resulting in similar average performance to prior unsupervised NMT work. GPT-3 has a noticeable skew in its performance depending on language direction. For the three input languages studied, GPT-3 significantly outperforms prior unsupervised NMT work when translating into English but underperforms when translating in the other direction. Performance on En-Ro is a noticeable outlier at over 10 BLEU worse than prior unsupervised NMT work. This could be a weakness due to reusing the byte-level BPE tokenizer of GPT-2 which was developed for an almost entirely English training dataset. For both Fr-En and De-En, few shot GPT-3 outperforms the best supervised result we could find but due to our unfamiliarity with the literature and the appearance that these are un-competitive benchmarks we do not suspect those results represent true state of the art. For Ro-En, few shot GPT-3 performs within 0.5 BLEU of the overall SOTA which is achieved by a combination of unsupervised pretraining, supervised finetuning on 608K labeled examples, and backtranslation [70].

结果如表3.4所示。零样本GPT-3仅接收任务的自然语言描述，但其表现仍不如最近的无监督NMT结果。然而，仅为每个翻译任务提供一个示例演示就能将性能提升超过7 BLEU，并接近与先前工作竞争的性能。在完整的少样本设定下，GPT-3 的 BLEU 又提高了 4 分，从而使其平均性能与之前的无监督 NMT 工作相似。GPT-3 的性能明显依赖于语言方向存在偏差。在研究的三种输入语言中，GPT-3 在翻译成英语时明显优于之前的无监督 NMT 工作，但在反向翻译时表现不佳。在英-罗（En-Ro）上的表现是一个明显的异常，比之前的无监督NMT工作低了超过10个BLEU分。这可能是由于重用了为几乎完全是英语训练数据开发的GPT-2字节级BPE分词器而导致的弱点。对于法英和德英，少量示例的 GPT-3 表现超过了我们能找到的最佳监督结果，但由于我们对相关文献不熟悉，并且这些基准似乎不具竞争力，我们不认为这些结果代表了真正的最新技术水平。对于罗马尼亚语到英语，少量示例的 GPT-3 性能与整体的最先进水平（SOTA）相差不到 0.5 BLEU，该最先进水平是通过无监督预训练、在 608K 标注示例上的监督微调以及回译相结合实现的 [70]。

Finally, across all language pairs and across all three settings (zero-, one-, and few-shot), there is a smooth trend of improvement with model capacity. This is shown in Figure 3.4 in the case of few-shot results, and scaling for all three settings is shown in Appendix H.

最后，在所有语言对以及所有三种设定（零样本、单样本和少样本）中，模型容量的增加都呈现出平滑的提高趋势。在少样本结果的情况下，这一点在图3.4中有所显示，而三种设定的规模变化则显示在附录H中。

3.4 Winograd-Style Tasks

3.4 Winograd 风格任务（Winograd-Style Tasks）

Setting	Winograd	Winogrande (XL)
Fine-tuned SOTA	90.1a	84.6b
GPT-3 Zero-Shot	88.3*	70.2
GPT-3 One-Shot	89.7*	73.2
GPT-3 Few-Shot	88.6*	77.7

Table 3.5: Results on the WSC273 version of Winograd schemas and the adversarial Winogrande dataset. See Section 4.1 for details on potential contamination of the Winograd test set. a[118] b[79]

设置	Winograd	Winogrande (XL)
微调后的SOTA	90.1a	84.6b
GPT-3 零样本	88.3*	70.2
GPT-3 一样本	89.7*	73.2
GPT-3 少样本	88.6*	77.7

表 3.5：在 WSC273 版本的 Winograd 模板和对抗性的 Winogrande 数据集上的结果。关于 Winograd 测试集潜在污染的详细信息，请参见第 4.1 节。a[118] b[79]

Figure 3.5: Zero-, one-, and few-shot performance on the adversarial Winogrande dataset as model capacity scales. Scaling is relatively smooth with the gains to few-shot learning increasing with model size, and few-shot GPT-3 175B is competitive with a fine-tuned RoBERTA-large.

图 3.5：随着模型容量的增加，在对抗性的 Winogrande 数据集上的零样本、一样本和少样本性能。随着模型大小的增加，扩展相对平稳，少样本学习的收益也随之增加，少样本 GPT-3 175B 的表现可与经过微调的 RoBERTA-large 相媲美。

The Winograd Schemas Challenge [65] is a classical task in NLP that involves determining which word a pronoun refers to, when the pronoun is grammatically ambiguous but semantically unambiguous to a human. Recently fine-tuned language models have achieved near-human performance on the original Winograd dataset, but more difficult versions such as the adversarially-mined Winogrande dataset [118] still significantly lag human performance. We test GPT-3’s performance on both Winograd and Winogrande, as usual in the zero-, one-, and few-shot setting.

Winograd 架构挑战 [65] 是自然语言处理中的一个经典任务，它涉及在代词在语法上存在歧义但对人类语义上没有歧义的情况下，确定代词所指的词。最近经过微调的语言模型在原始 Winograd 数据集上已达到接近人类的性能，但更困难的版本，如对抗性挖掘的 Winogrande 数据集 [118]，仍然明显落后于人类的表现。我们像往常一样，在零样本、一样本和少样本设定下测试 GPT-3 在 Winograd 和 Winogrande 上的表现。

On Winograd we test GPT-3 on the original set of 273 Winograd schemas, using the same “partial evaluation” method described in [117]. Note that this setting differs slightly from the WSC task in the SuperGLUE benchmark, which is presented as binary classification and requires entity extraction to convert to the form described in this section. On Winograd GPT-3 achieves 88.3%, 89.7%, and 88.6% in the zero-shot, one-shot, and few-shot settings, showing no clear in-context learning but in all cases achieving strong results just a few points below state-of-the-art and estimated human performance. We note that contamination analysis found some Winograd schemas in the training data but this appears to have only a small effect on results (see Section 4.1).

在 Winograd 上，我们使用原始的 273 个 Winograd 架构集合测试 GPT-3，并使用 [117] 中描述的相同“部分评估”方法。请注意，这种设置与 SuperGLUE 基准中的 WSC 任务略有不同，后者以二元分类形式呈现，并且需要实体提取以转换为本节所述的形式。在 Winograd 测试中，GPT-3 在零样本、一次样本和少样本设定下分别达到 88.3%、89.7% 和 88.6%，没有明显的上下文学习效果，但在所有情况下都取得了强劲的结果，仅比最先进水平和预估的人类表现低几个百分点。我们注意到污染分析发现训练数据中存在一些 Winograd 题目，但这似乎对结果的影响很小（见第 4.1 节）。

On the more difficult Winogrande dataset, we do find gains to in-context learning: GPT-3 achieves 70.2% in the zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot setting. For comparison a fine-tuned RoBERTA model achieves 79%, state-of-the-art is 84.6% achieved with a fine-tuned high capacity model (T5), and human performance on the task as reported by [118] is 94.0%.

在更困难的 Winogrande 数据集中，我们确实发现了上下文学习的提升：GPT-3 在零样本设定中达到 70.2%，在一次样本设置中达到 73.2%，在少样本设定中达到 77.7%。作为比较，经过微调的 RoBERTA 模型达到 79%，最先进的技术是通过微调高容量模型（T5）达到的 84.6%，而[118] 报告的人的任务表现为 94.0%。

3.5 Common Sense Reasoning

3.5 常识推理（Common Sense Reasoning）

Setting	PIQA	ARC (Easy)	ARC (Challenge)	OpenBookQA
Fine-tuned SOTA	79.4	92.0[55]	78.5[55]	87.2[55]
GPT-3 Zero-Shot	80.5*	68.8	51.4	57.6
GPT-3 One-Shot	80.5*	71.2	53.2	58.8
GPT-3 Few-Shot	82.8*	70.1	51.5	65.4

Table 3.6: GPT-3 results on three commonsense reasoning tasks, PIQA, ARC, and OpenBookQA. GPT-3 Few-Shot PIQA result is evaluated on the test server. See Section 4.1 for details on potential contamination issues on the PIQA test set.

设置	PIQA	ARC（简单）	ARC（挑战）	OpenBookQA
微调后的最先进模型	79.4	92.0[55]	78.5[55]	87.2[55]
GPT-3 零样本	80.5*	68.8	51.4	57.6
GPT-3 单样本	80.5*	71.2	53.2	58.8
GPT-3 少样本	82.8*	70.1	51.5	65.4
表 3.6：GPT-3 在三个常识推理任务 PIQA、ARC 和 OpenBookQA 上的结果。GPT-3 的少样本 PIQA 结果是在测试服务器上评估的。关于 PIQA 测试集潜在污染问题的详细信息，请参见第 4.1 节。

Figure 3.6: GPT-3 results on PIQA in the zero-shot, one-shot, and few-shot settings. The largest model achieves a score on the development set in all three conditions that exceeds the best recorded score on the task.

图 3.6：GPT-3 在 PIQA 上的零次、一次和少量示例设置下的结果。最大模型在开发集上的三种条件下的得分都超过了该任务的最佳记录得分。

Next we consider three datasets which attempt to capture physical or scientific reasoning, as distinct from sentence completion, reading comprehension, or broad knowledge question answering. The first, PhysicalQA (PIQA) [11], asks common sense questions about how the physical world works and is intended as a probe of grounded understanding of the world. GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot, and 82.8% accuracy few-shot (the last measured on PIQA’s test server). This compares favorably to the 79.4% accuracy prior state-of-the-art of a fine-tuned RoBERTa. PIQA shows relatively shallow scaling with model size and is still over 10% worse than human performance, but GPT-3’s few-shot and even zero-shot result outperform the current state-of-the-art. Our analysis flagged PIQA for a potential data contamination issue (despite hidden test labels), and we therefore conservatively mark the result with an asterisk. See Section 4.1 for details.

接下来我们考虑三个试图捕捉物理或科学推理的数据集，这些数据集不同于句子完形填空、阅读理解或广泛知识问答。第一个数据集，PhysicalQA (PIQA) [11]，提出关于物理世界如何运作的常识性问题，并旨在作为对世界具体理解的探测工具。GPT-3 在零样本时达到 81.0% 的准确率，在单样本时达到 80.5% 的准确率，在少样本时达到 82.8% 的准确率（最后一个是在 PIQA 的测试服务器上测量的）。这优于之前通过微调 RoBERTa 达到的 79.4% 的最先进准确率。PIQA 在模型规模上显示出相对浅显的扩展性，并且仍比人类表现低超过 10%，但 GPT-3 的少样本甚至零样本结果都超过了当前的最先进水平。我们的分析发现 PIQA 可能存在数据污染问题（尽管隐藏了测试标签），因此我们保守地用星号标记该结果。详情请参见第 4.1 节。

ARC [14] is a dataset of multiple-choice questions collected from 3rd to 9th grade science exams. On the “Challenge” version of the dataset which has been filtered to questions which simple statistical or information retrieval methods are unable to correctly answer, GPT-3 achieves 51.4% accuracy in the zero-shot setting, 53.2% in the one-shot setting, and 51.5% in the few-shot setting. This is approaching the performance of a fine-tuned RoBERTa baseline (55.9%) from UnifiedQA [55]. On the “Easy” version of the dataset (questions which either of the mentioned baseline approaches answered correctly), GPT-3 achieves 68.8%, 71.2%, and 70.1% which slightly exceeds a fine-tuned RoBERTa baseline from [55]. However, both of these results are still much worse than the overall SOTAs achieved by the UnifiedQA which exceeds GPT-3’s few-shot results by 27% on the challenge set and 22% on the easy set.

ARC [14] 是一个多项选择题数据集，收集自三年级到九年级的科学考试。在数据集的“挑战”版本中，该版本已被筛选为简单的统计或信息检索方法无法正确回答的问题，GPT-3 在零样本设定下的准确率为 51.4%，在一次示例设置下的准确率为 53.2%，在少量示例设置下的准确率为 51.5%.这已经接近 UnifiedQA [55] 的微调 RoBERTa 基线（55.9%）的性能。在数据集的“简单”版本（上述基线方法中任一方法回答正确的问题）上，GPT-3 分别达到了 68.8%、71.2% 和 70.1%，略微超过了 [55] 的微调 RoBERTa 基线。然而，这两个结果仍然远不及 UnifiedQA 达到的整体最先进水平，其在挑战集上的结果比 GPT-3 的少量示例结果高出 27%，在简单集上高出 22%。

On OpenBookQA [84], GPT-3 improves significantly from zero to few shot settings but is still over 20 points short of the overall SOTA. GPT-3’s few-shot performance is similar to a fine-tuned BERT Large baseline on the leaderboard.

在 OpenBookQA [84] 上，GPT-3 从零样本到少样本设定有显著提升，但仍比总体最先进水平低 20 多分。GPT-3 的少样本表现与排行榜上经过微调的 BERT Large 基线相似。

Overall, in-context learning with GPT-3 shows mixed results on commonsense reasoning tasks, with only small and inconsistent gains observed in the one and few-shot learning settings for both PIQA and ARC, but a significant improvement is observed on OpenBookQA. GPT-3 sets SOTA on the new PIQA dataset in all evaluation settings.

总体而言，GPT-3 在上下文学习中在常识推理任务上显示出喜忧参半的结果，在 PIQA 和 ARC 的一次学习和少量学习设置中只观察到小幅且不一致的提升，但在 OpenBookQA 上观察到显著的改进。GPT-3 在所有评估设置中在新的 PIQA 数据集上设定了最先进水平。

3.6 Reading Comprehension

3.6 阅读理解（Reading Comprehension）

Setting	CoQA	DROP	QuAC	SQuADv2	RACE-h	RACE-m
Fine-tuned SOTA	90.7a	89.1b	74.4c	93.0d	90.0e	93.1e
GPT-3 Zero-Shot	81.5	23.6	41.5	59.5	45.5	58.4
GPT-3 One-Shot	84.0	34.3	43.3	65.4	45.9	57.4
GPT-3 Few-Shot	85.0	36.5	44.3	69.8	46.8	58.1

Table 3.7: Results on reading comprehension tasks. All scores are F1 except results for RACE which report accuracy. a[53] b[50] c[2] d[103] e[125]

设置	CoQA	DROP	QuAC	SQuADv2	RACE-h	RACE-m
微调后的最先进	90.7a	89.1b	74.4c	93.0d	90.0e	93.1e
GPT-3 零样本	81.5	23.6	41.5	59.5	45.5	58.4
GPT-3 一样本	84.0	34.3	43.3	65.4	45.9	57.4
GPT-3 少样本	85.0	36.5	44.3	69.8	46.8	58.1
表 3.7：阅读理解任务结果。所有得分均为 F1，RACE 的结果使用准确率。a[53] b[50] c[2] d[103] e[125]

Figure 3.7: GPT-3 results on CoQA reading comprehension task. GPT-3 175B achieves 85 F1 in the few-shot setting, only a few points behind measured human performance and state-of-the-art fine-tuned models. Zero-shot and one-shot performance is a few points behind, with the gains to few-shot being largest for bigger models.

图 3.7：GPT-3 在 CoQA 阅读理解任务上的结果。GPT-3 175B 在少量示例设置下达到 85 的 F1 分数，仅比测得的人类表现和最先进的微调模型低几分。零样本和单样本表现低几分，其中少量示例设置带来的提升在更大模型上最大。

Next we evaluate GPT-3 on the task of reading comprehension. We use a suite of 5 datasets including abstractive, multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread in GPT-3’s performance across these datasets suggestive of varying capability with different answer formats. In general we observe GPT-3 is on par with initial baselines and early results trained using contextual representations on each respective dataset.

接下来，我们评估 GPT-3 在阅读理解任务上的表现。我们使用包含摘要式、选择题以及基于片段的答案格式的五个数据集套件，这些数据集涵盖了对话和单一问题的场景。我们观察到 GPT-3 在这些数据集上的表现差异很大，这表明它在不同答案格式上能力存在差异。总体而言，我们观察到 GPT-3 与最初的基线以及在各自数据集上使用上下文表示训练的早期结果相当。

GPT-3 performs best (within 3 points of the human baseline) on CoQA [106] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC [16] a dataset which requires modeling structured dialog acts and answer span selections of teacher-student interactions. On DROP [27], a dataset testing discrete reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the fine-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [110]. On SQuAD 2.0 [108], GPT-3 demonstrates its few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to slightly outperform the best fine-tuned result in the original paper. On RACE [78], a multiple choice dataset of middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with the earliest work utilizing contextual representations and is still 45% behind SOTA.

GPT-3 在 CoQA [106]（一个自由形式的对话数据集）上表现最佳（与人类基准相差不超过 3 分），而在 QuAC [16]（一个需要建模结构化对话行为和教师-学生互动答案跨度选择的数据集）上表现最差（比 ELMo 基准低 13 F1）。在 DROP [27] 上，这是一个在阅读理解背景下测试离散推理和数值能力的数据集，GPT-3 在少量示例设置下的表现超出了原论文中经过微调的 BERT 基线，但仍远低于人类表现以及使用符号系统增强神经网络的最先进方法 [110]。在 SQuAD 2.0 [108] 上，GPT-3 展示了其少量学习能力，与零样本设定相比，其 F1 分数提高了近 10（达到 69.8）。这使得它略微超过了原论文中最佳微调结果。在 RACE [78]——一个包含初中和高中英语考试的多项选择题数据集上，GPT-3 的表现相对较弱，仅能与最早使用上下文表示的工作竞争，并且仍落后于最先进水平 45% 。

	SuperGLUE	BoolQ	CB	CB	COPA	RTE
	Average	Accuracy	Accuracy	F1	Accuracy	Accuracy
Fine-tuned SOTA	89.0	91.0	96.9	93.9	94.8	92.5
Fine-tuned BERT-Large	69.0	77.4	83.6	75.7	70.6	71.7
GPT-3 Few-Shot	71.8	76.4	75.6	52.0	92.0	69.0

	WiC	WSC	MultiRC	MultiRC	ReCoRD	ReCoRD
	Accuracy	Accuracy	Accuracy	F1a	Accuracy	F1
Fine-tuned SOTA	76.1	93.8	62.3	88.2	92.5	93.3
Fine-tuned BERT-Large	69.6	64.6	24.1	70.0	71.3	72.0
GPT-3 Few-Shot	49.4	80.1	30.5	75.4	90.2	91.1

Table 3.8: Performance of GPT-3 on SuperGLUE compared to fine-tuned baselines and SOTA. All results are reported on the test set. GPT-3 few-shot is given a total of 32 examples within the context of each task and performs no gradient updates.

	SuperGLUE	BoolQ	CB	CB	COPA	RTE
	平均值	准确率	准确率	F1 值	准确率	准确率
微调后的 SOTA	89.0	91.0	96.9	93.9	94.8	92.5
微调后的 BERT-Large	69.0	77.4	83.6	75.7	70.6	71.7
GPT-3 少量示例	71.8	76.4	75.6	52.0	92.0	69.0

	WiC	WSC	MultiRC	MultiRC	ReCoRD	ReCoRD
	准确率	准确率	准确率	F1a	准确率	F1
微调的最先进模型	76.1	93.8	62.3	88.2	92.5	93.3
微调的 BERT-Large	69.6	64.6	24.1	70.0	71.3	72.0
GPT-3 少量示例	49.4	80.1	30.5	75.4	90.2	91.1
表 3.8：GPT-3 在 SuperGLUE 上的表现，与微调基线和最新技术（SOTA）进行比较。所有结果均在测试集上报告。GPT-3 少量示例学习（few-shot）在每个任务的上下文中提供总共 32 个示例，并且不进行梯度更新。

Figure 3.8: Performance on SuperGLUE increases with model size and number of examples in context. A value of $K=32$ means that our model was shown 32 examples per task, for 256 examples total divided across the 8 tasks in SuperGLUE. We report GPT-3 values on the dev set, so our numbers are not directly comparable to the dotted reference lines (our test set results are in Table 3.8). The BERT-Large reference model was fine-tuned on the SuperGLUE training set (125K examples), whereas BERT++ was first fine-tuned on MultiNLI (392K examples) and SWAG (113K examples) before further fine-tuning on the SuperGLUE training set (for a total of 630K fine-tuning examples). We find the difference in performance between the BERT-Large and BERT++ to be roughly equivalent to the difference between GPT-3 with one example per context versus eight examples per context.

图 3.8：在 SuperGLUE 上的性能随着模型规模和上下文中的示例数量增加而提高。 $K=32$ 的值表示我们的模型在每个任务中展示了 32 个示例，总共 256 个示例，分布在 SuperGLUE 的 8 个任务中。我们报告的是 GPT-3 在开发集上的数值，因此我们的数字不能直接与虚线参考线进行比较（我们的测试集结果见表 3.8）。BERT-Large参考模型是在SuperGLUE训练集（125K例子）上进行微调的，而BERT++首先在MultiNLI（392K例子）和SWAG（113K例子）上进行微调，然后再在SuperGLUE训练集上进一步微调（总共630K微调例子）。我们发现 BERT-Large 和 BERT++ 之间的性能差异大致相当于 GPT-3 在每个上下文提供一个示例与提供八个示例之间的差异。

3.7 SuperGLUE

In order to better aggregate results on NLP tasks and compare to popular models such as BERT and RoBERTa in a more systematic way, we also evaluate GPT-3 on a standardized collection of datasets, the SuperGLUE benchmark [135] [135] [17] [25] [105] [54] [142] [21] [8] [34] [6] [96] [98]. GPT-3’s test-set performance on the SuperGLUE dataset is shown in Table 3.8. In the few-shot setting, we used 32 examples for all tasks, sampled randomly from the training set. For all tasks except WSC and MultiRC, we sampled a new set of examples to use in the context for each problem. For WSC and MultiRC, we used the same set of randomly drawn examples from the training set as context for all of the problems we evaluated.

为了更好地汇总自然语言处理任务的结果，并以更系统的方式与流行模型如 BERT 和 RoBERTa 进行比较，我们还在一个标准化的数据集集合上评估 GPT-3，即 SuperGLUE 基准 [135] [135] [17] [25] [105] [54] [142] [21] [8] [34] [6] [96] [98]。GPT-3 在 SuperGLUE 数据集上的测试集表现如表 3.8 所示。在少量示例设置中，我们为所有任务使用了 32 个示例，这些示例是从训练集中随机抽取的。对于除 WSC 和 MultiRC 之外的所有任务，我们为每个问题在上下文中使用时，都抽取了一组新的示例。对于 WSC 和 MultiRC，我们对所有评估的问题使用了从训练集中随机抽取的相同示例集作为上下文。

We observe a wide range in GPT-3’s performance across tasks. On COPA and ReCoRD GPT-3 achieves near-SOTA performance in the one-shot and few-shot settings, with COPA falling only a couple points short and achieving second place on the leaderboard, where first place is held by a fine-tuned 11 billion parameter model (T5). On WSC, performance is still relatively strong, achieving 80.1% in the few-shot setting (note that GPT-3 achieves 88.6% on the original Winograd dataset as described in Section 3.4). On BoolQ, MultiRC, and RTE, performance is reasonable, roughly matching that of a fine-tuned BERT-Large. On CB, we see signs of life at 75.6% in the few-shot setting.

我们观察到 GPT-3 在各项任务中的表现差异很大。在 COPA 和 ReCoRD 上，GPT-3 在一次性学习和少样本学习情况下接近最先进水平，其中 COPA 仅落后几分，在排行榜上名列第二，第一名由一个经过微调的 110 亿参数模型（T5）保持。在 WSC 上，性能仍然相对较强，在少样本情况下达到了 80.1%（注意，GPT-3 在原始 Winograd 数据集上的表现为 88.6%，如第 3.4 节所述）。在 BoolQ、MultiRC 和 RTE 上，性能尚可，大致与微调后的 BERT-Large 相当。在 CB 上，我们在少样本情况下看到了 75.6% 的生命迹象。

WiC is a notable weak spot with few-shot performance at 49.4% (at random chance). We tried a number of different phrasings and formulations for WiC (which involves determining if a word is being used with the same meaning in two sentences), none of which was able to achieve strong performance. This hints at a phenomenon that will become clearer in the next section (which discusses the ANLI benchmark) – GPT-3 appears to be weak in the few-shot or one-shot setting at some tasks that involve comparing two sentences or snippets, for example whether a word is used the same way in two sentences (WiC), whether one sentence is a paraphrase of another, or whether one sentence implies another. This could also explain the comparatively low scores for RTE and CB, which also follow this format. Despite these weaknesses, GPT-3 still outperforms a fine-tuned BERT-large on four of eight tasks and on two tasks GPT-3 is close to the state-of-the-art held by a fine-tuned 11 billion parameter model.

WiC 是一个显著的弱点，少样本表现为 49.4%（接近随机机会）。我们尝试了许多不同的措辞和表述方式用于 WiC（这涉及判断一个单词在两个句子中是否被以相同的意思使用），但没有一种能够达到较强的表现。这暗示了一个现象，在下一节（讨论 ANLI 基准）中会变得更清楚——GPT-3 在一些涉及比较两个句子或片段的任务中，在少量示例或单示例设置下似乎表现较弱，例如判断一个词在两个句子中的用法是否相同（WiC）、判断一个句子是否是另一个句子的意译，或者判断一个句子是否意味着另一个句子。这也可以解释RTE和CB相对较低的分数，这两个任务也遵循这种格式。尽管存在这些弱点，GPT-3在八个任务中的四个任务上仍然优于经过微调的BERT-large，而在两个任务上，GPT-3接近由经过微调的110亿参数模型保持的最先进水平。

Finally, we note that the few-shot SuperGLUE score steadily improves with both model size and with number of examples in the context showing increasing benefits from in-context learning (Figure 3.8). We scale $K$ up to 32 examples per task, after which point additional examples will not reliably fit into our context. When sweeping over values of $K$, we find that GPT-3 requires less than eight total examples per task to outperform a fine-tuned BERT-Large on overall SuperGLUE score.

最后，我们注意到少样本的 SuperGLUE 分数随着模型规模的增大以及上下文中示例数量的增加而稳步提高，显示出从上下文学习中获得的益处不断增加（图 3.8）。我们将 $K$ 扩展到每个任务 32 个示例，此后更多的示例将无法可靠地适应我们的上下文。当对 $K$ 的值进行扫描时，我们发现 GPT-3 每个任务总共需要少于八个示例，就能在整体 SuperGLUE 得分上超过经过微调的 BERT-Large。

Figure 3.9: Performance of GPT-3 on ANLI Round 3. Results are on the dev-set, which has only 1500 examples and therefore has high variance (we estimate a standard deviation of 1.2%). We find that smaller models hover around random chance, while few-shot GPT-3 175B closes almost half the gap from random chance to SOTA. Results for ANLI rounds 1 and 2 are shown in the appendix.

图 3.9：GPT-3 在 ANLI 第三轮的表现。 结果基于开发集，该集仅有 1500 个样本，因此方差较高（我们估计标准差为 1.2%）。我们发现较小的模型表现接近随机水平，而少量示例的 GPT-3 175B 几乎填补了从随机水平到最先进水平的一半差距。ANLI 第一轮和第二轮的结果见附录。

3.8 NLI

3.8 自然语言推理（NLI）

Natural Language Inference (NLI) [31] concerns the ability to understand the relationship between two sentences. In practice, this task is usually structured as a two or three class classification problem where the model classifies whether the second sentence logically follows from the first, contradicts the first sentence, or is possibly true (neutral). SuperGLUE includes an NLI dataset, RTE, which evaluates the binary version of the task. On RTE, only the largest version of GPT-3 performs convincingly better than random (56%) in any evaluation setting, but in a few-shot setting GPT-3 performs similarly to a single-task fine-tuned BERT Large. We also evaluate on the recently introduced Adversarial Natural Language Inference (ANLI) dataset [94]. ANLI is a difficult dataset employing a series of adversarially mined natural language inference questions in three rounds (R1, R2, and R3). Similar to RTE, all of our models smaller than GPT-3 perform at almost exactly random chance on ANLI, even in the few-shot setting ($\sim 33\%$), whereas GPT-3 itself shows signs of life on Round 3. Results for ANLI R3 are highlighted in Figure 3.9 and full results for all rounds can be found in Appendix H. These results on both RTE and ANLI suggest that NLI is still a very difficult task for language models and they are only just beginning to show signs of progress.

自然语言推断（NLI）[31] 涉及理解两个句子之间关系的能力。在实践中，这个任务通常被构建为一个二分类或三分类问题，模型需要判断第二个句子是否逻辑上从第一个句子推导出、与第一个句子矛盾或可能为真（中性）。SuperGLUE 包含一个 NLI 数据集 RTE，用于评估该任务的二分类版本。在RTE上，只有最大版本的GPT-3在任何评估设置中表现明显优于随机（56%），但在少样本的设置下，GPT-3的表现与单任务微调的BERT Large类似。我们还在最近发布的对抗性自然语言推理（ANLI）数据集[94]上进行了评估。ANLI 是一个困难的数据集，包含三轮（R1、R2 和 R3）经过对抗性挖掘的自然语言推理问题。类似于 RTE，我们所有小于 GPT-3 的模型在 ANLI 上的表现几乎完全是随机的，即使在少样本设定下（$\sim 33\%$）也是如此，而 GPT-3 本身在第三轮表现出一定能力。ANLI R3 的结果在图 3.9 中突出显示，所有轮次的完整结果可以在附录 H 中找到。在RTE和ANLI上的这些结果表明，NLI对于语言模型来说仍然是一个非常困难的任务，它们才刚刚开始显示出进展的迹象。

3.9 Synthetic and Qualitative Tasks

3.9 合成与定性任务（Synthetic and Qualitative Tasks）

One way to probe GPT-3’s range of abilities in the few-shot (or zero- and one-shot) setting is to give it tasks which require it to perform simple on-the-fly computational reasoning, recognize a novel pattern that is unlikely to have occurred in training, or adapt quickly to an unusual task. We devise several tasks to test this class of abilities. First, we test GPT-3’s ability to perform arithmetic. Second, we create several tasks that involve rearranging or unscrambling the letters in a word, tasks which are unlikely to have been exactly seen during training. Third, we test GPT-3’s ability to solve SAT-style analogy problems few-shot. Finally, we test GPT-3 on several qualitative tasks, including using new words in a sentence, correcting English grammar, and news article generation. We will release the synthetic datasets with the hope of stimulating further study of test-time behavior of language models.

在少量示例（或零示例和单示例）环境下探测 GPT-3 能力范围的一种方法是给它一些任务，这些任务要求它进行简单的即时计算推理、识别在训练中不太可能出现的新模式，或快速适应不寻常的任务。我们设计了几个任务来测试这一类能力。首先，我们测试 GPT-3 进行算术运算的能力。其次，我们创建了几个涉及重新排列或解密单词中字母的任务，这些任务在训练期间不太可能被完全见过。第三，我们测试 GPT-3 以少量示例解决 SAT 风格类比问题的能力。最后，我们测试 GPT-3 在几个定性任务上的表现，包括在句子中使用新词、纠正英语语法以及新闻文章生成。我们将发布这些合成数据集，希望能进一步推动对语言模型在测试时行为的研究。

3.9.1 Arithmetic

3.9.1 算术

To test GPT-3’s ability to perform simple arithmetic operations without task-specific training, we developed a small battery of 10 tests that involve asking GPT-3 a simple arithmetic problem in natural language:

为了测试 GPT-3 在没有特定任务训练的情况下执行简单算术运算的能力，我们开发了一小组包含 10 个测试的测试集，这些测试涉及用自然语言向 GPT-3 提出一个简单的算术问题：

• • 2 digit addition (2D+) – The model is asked to add two integers sampled uniformly from $[0,100)$, phrased in the form of a question, e.g. “Q: What is 48 plus 76? A: 124.”
• • 2 digit subtraction (2D-) – The model is asked to subtract two integers sampled uniformly from $[0,100)$; the answer may be negative. Example: “Q: What is 34 minus 53? A: -19”.
• • 3 digit addition (3D+) – Same as 2 digit addition, except numbers are uniformly sampled from $[0,1000)$.
• • 3 digit subtraction (3D-) – Same as 2 digit subtraction, except numbers are uniformly sampled from $[0,1000)$.
• • 4 digit addition (4D+) – Same as 3 digit addition, except uniformly sampled from $[0,10000)$.
• • 4 digit subtraction (4D-) – Same as 3 digit subtraction, except uniformly sampled from $[0,10000)$.
• • 5 digit addition (5D+) – Same as 3 digit addition, except uniformly sampled from $[0,100000)$.
• • 5 digit subtraction (5D-) – Same as 3 digit subtraction, except uniformly sampled from $[0,100000)$.
• • 2 digit multiplication (2Dx) – The model is asked to multiply two integers sampled uniformly from $[0,100)$, e.g. “Q: What is 24 times 42? A: 1008”.
• • One-digit composite (1DC) – The model is asked to perform a composite operation on three 1 digit numbers, with parentheses around the last two. For example, “Q: What is 6+(48)? A: 38”. The three 1 digit numbers are selected uniformly on $[0,10)$ and the operations are selected uniformly from {+,-,}.

• • 两位数加法（2D+） – 模型需要对从 $[0,100)$ 中均匀抽取的两个整数进行加法运算，以问题的形式提出，例如：“问：48 加 76 等于多少？答：124。”
• • 两位数减法（2D-） – 模型需要对从 $[0,100)$ 中均匀抽取的两个整数进行减法运算；答案可能为负数。示例：“问：34 减 53 等于多少？答：-19”.- • 三位数加法 (3D+) – 与两位数加法相同，但数字是从 $[0,1000)$ 中均匀抽取的。
• • 三位数减法 (3D-) – 与两位数减法相同，但数字是从 $[0,1000)$ 中均匀抽取的。
• • 四位数加法 (4D+) – 与三位数加法相同，但数字是从 $[0,10000)$ 中均匀抽取的。
• • 四位数减法 (4D-) – 与三位数减法相同，但数字是从 $[0,10000)$ 中均匀抽取的。- • 五位数加法 (5D+) – 与三位数加法相同，只是从 $[0,100000)$ 中均匀抽样。- • 五位数减法 (5D-) – 与三位数减法相同，只是从 $[0,100000)$ 中均匀抽样。- • 两位数乘法 (2Dx) – 要求模型对从 $[0,100)$ 均匀抽样的两个整数进行相乘，例如：“问：24 乘以 42 等于多少？答：1008”。- • 一位合成数（1DC） – 模型被要求对三个一位数进行合成运算，其中最后两个数用括号括起来。例如，“问：6+(48) 等于多少？答：38”。三个一位数均从 $[0,10)$ 中均匀选择，运算符则从 {+,-,} 中均匀选择。

In all 10 tasks the model must generate the correct answer exactly. For each task we generate a dataset of 2,000 random instances of the task and evaluate all models on those instances.

在所有10个任务中，模型必须完全生成正确答案。对于每个任务，我们生成一个包含2,000个随机任务实例的数据集，并在这些实例上评估所有模型。

First we evaluate GPT-3 in the few-shot setting, for which results are shown in Figure 3.10. On addition and subtraction, GPT-3 displays strong proficiency when the number of digits is small, achieving 100% accuracy on 2 digit addition, 98.9% at 2 digit subtraction, 80.2% at 3 digit addition, and 94.2% at 3-digit subtraction. Performance decreases as the number of digits increases, but GPT-3 still achieves 25-26% accuracy on four digit operations and 9-10% accuracy on five digit operations, suggesting at least some capacity to generalize to larger numbers of digits. GPT-3 also achieves 29.2% accuracy at 2 digit multiplication, an especially computationally intensive operation. Finally, GPT-3 achieves 21.3% accuracy at single digit combined operations (for example, 9*(7+5)), suggesting that it has some robustness beyond just single operations.

首先，我们在少样本设定下评估 GPT-3，其结果如图 3.10 所示。在加法和减法方面，当数字位数较少时，GPT-3 展现出较强的熟练度，在两位数加法上达到 100% 的准确率，两位数减法为 98.9%，三位数加法为 80.2%，三位数减法为 94.2%。随着数字位数的增加，性能会下降，但 GPT-3 在四位数运算中仍能达到 25-26% 的准确率，在五位数运算中达到 9-10% 的准确率，这表明它至少有一定的能力将运算推广到更多位数。GPT-3 在两位数乘法中也能达到 29.2% 的准确率，这是一项特别计算密集的操作。最终，GPT-3 在单个数字的组合运算（例如，9*(7+5)）中达到了 21.3% 的准确率，这表明它在单一运算之外具有一定的稳健性。

Figure 3.10: Results on all 10 arithmetic tasks in the few-shot settings for models of different sizes. There is a significant jump from the second largest model (GPT-3 13B) to the largest model (GPT-3 175), with the latter being able to reliably accurate 2 digit arithmetic, usually accurate 3 digit arithmetic, and correct answers a significant fraction of the time on 4-5 digit arithmetic, 2 digit multiplication, and compound operations. Results for one-shot and zero-shot are shown in the appendix.

图 3.10：不同规模模型在少样本设定下所有 10 个算术任务的结果。从第二大模型（GPT-3 13B）到最大模型（GPT-3 175）有显著的提升，后者能够可靠地进行两位数算术运算，通常能够正确进行三位数算术运算，并且在四到五位数算术、两位数乘法以及复合运算中，正确答案的比例也相当可观。一次性学习和零次学习的结果显示在附录中。

Setting	2D+	2D-	3D+	3D-	4D+	4D-	5D+	5D-	2Dx	1DC
GPT-3 Zero-shot	76.9	58.0	34.2	48.3	4.0	7.5	0.7	0.8	19.8	9.8
GPT-3 One-shot	99.6	86.4	65.5	78.7	14.0	14.0	3.5	3.8	27.4	14.3
GPT-3 Few-shot	100.0	98.9	80.4	94.2	25.5	26.8	9.3	9.9	29.2	21.3

Table 3.9: Results on basic arithmetic tasks for GPT-3 175B. {2,3,4,5}D{+,-} is 2, 3, 4, and 5 digit addition or subtraction, 2Dx is 2 digit multiplication. 1DC is 1 digit composite operations. Results become progressively stronger moving from the zero-shot to one-shot to few-shot setting, but even the zero-shot shows significant arithmetic abilities.

设置	2D+	2D-	3D+	3D-	4D+	4D-	5D+	5D-	2Dx	1DC
GPT-3 零样本	76.9	58.0	34.2	48.3	4.0	7.5	0.7	0.8	19.8	9.8
GPT-3 一样本	99.6	86.4	65.5	78.7	14.0	14.0	3.5	3.8	27.4	14.3
GPT-3 少样本	100.0	98.9	80.4	94.2	25.5	26.8	9.3	9.9	29.2	21.3
表 3.9：GPT-3 175B 在基础算术任务上的结果。{2,3,4,5}D{+,-} 表示 2、3、4 和 5 位数的加法或减法，2Dx 表示 2 位数乘法，1DC 表示 1 位数复合运算。从零样本到一示例再到少量示例，结果逐渐增强，但即使是零样本也显示出显著的算术能力。

As Figure 3.10 makes clear, small models do poorly on all of these tasks – even the 13 billion parameter model (the second largest after the 175 billion full GPT-3) can solve 2 digit addition and subtraction only half the time, and all other operations less than 10% of the time.

正如图3.10所示，小型模型在所有这些任务上的表现都很差——即使是拥有130亿参数的模型（在1750亿完整GPT-3之后排第二大）也只能在一半的时间内解决两位数的加法和减法，而所有其他运算的成功率不到10%。

One-shot and zero-shot performance are somewhat degraded relative to few-shot performance, suggesting that adaptation to the task (or at the very least recognition of the task) is important to performing these computations correctly. Nevertheless, one-shot performance is still quite strong, and even zero-shot performance of the full GPT-3 significantly outperforms few-shot learning for all smaller models. All three settings for the full GPT-3 are shown in Table 3.9, and model capacity scaling for all three settings is shown in Appendix H.

相较于少样本（few-shot）表现，一次性（one-shot）和零样本（zero-shot）表现有所下降，这表明对任务的适应（或至少对任务的识别）对于正确执行这些计算是重要的。尽管如此，一次性表现仍然相当强，而完整的 GPT-3 的零样本表现甚至显著优于所有较小模型的少样本学习。完整 GPT-3 的三种设置均显示在表 3.9 中，所有三种设置的模型容量扩展显示在附录 H 中。

To spot-check whether the model is simply memorizing specific arithmetic problems, we took the 3-digit arithmetic problems in our test set and searched for them in our training data in both the forms " + =" and " plus ". Out of 2,000 addition problems we found only 17 matches (0.8%) and out of 2,000 subtraction problems we found only 2 matches (0.1%), suggesting that only a trivial fraction of the correct answers could have been memorized. In addition, inspection of incorrect answers reveals that the model often makes mistakes such as not carrying a “1”, suggesting it is actually attempting to perform the relevant computation rather than memorizing a table.

为了抽查模型是否仅仅在记忆特定的算术问题，我们将测试集中三位数的算术问题拿出来，并在训练数据中搜索它们的两种形式：“ + =” 和 “ plus ”。在 2,000 道加法题中，我们只发现了 17 个匹配（0.8%），在 2,000 道减法题中，我们只发现了 2 个匹配（0.1%），这表明只有极少数正确答案可能是被记住的。此外，对错误答案的检查显示，模型经常犯诸如未进一位的错误，这表明它实际上是在尝试执行相关计算，而不是记忆一个表格。

Overall, GPT-3 displays reasonable proficiency at moderately complex arithmetic in few-shot, one-shot, and even zero-shot settings.

总体而言，GPT-3 在少量示例、一示例甚至零示例的情况下，对中等复杂的算术表现出合理的熟练程度。

3.9.2 Word Scrambling and Manipulation Tasks

3.9.2 单词打乱与操作任务

To test GPT-3’s ability to learn novel symbolic manipulations from a few examples, we designed a small battery of 5 “character manipulation” tasks. Each task involves giving the model a word distorted by some combination of scrambling, addition, or deletion of characters, and asking it to recover the original word. The 5 tasks are:

为了测试 GPT-3 从少量示例中学习新符号操作的能力，我们设计了五个“小型字符操作”任务。每个任务都涉及给模型一个通过某种字符打乱、添加或删除组合而扭曲的单词，并要求它恢复原始单词。这五个任务是：

Setting	CL	A1	A2	RI	RW
GPT-3 Zero-shot	3.66	2.28	8.91	8.26	0.09
GPT-3 One-shot	21.7	8.62	25.9	45.4	0.48
GPT-3 Few-shot	37.9	15.1	39.7	67.2	0.44

Table 3.10: GPT-3 175B performance on various word unscrambling and word manipulation tasks, in zero-, one-, and few-shot settings. CL is “cycle letters in word”, A1 is anagrams of but the first and last letters, A2 is anagrams of all but the first and last two letters, RI is “Random insertion in word”, RW is “reversed words”.

设置	CL	A1	A2	RI	RW
GPT-3 零样本	3.66	2.28	8.91	8.26	0.09
GPT-3 一样本	21.7	8.62	25.9	45.4	0.48
GPT-3 少样本	37.9	15.1	39.7	67.2	0.44
表 3.10：GPT-3 175B 在各种单词重组和单词操作任务上的表现，在零样本、一样本和少样本设定下。CL 是“单词中字母循环”，A1 是除首尾字母外的字母重排，A2 是除首尾两个字母外的字母重排，RI 是“单词中随机插入”，RW 是“单词反序”。

Figure 3.11: Few-shot performance on the five word scrambling tasks for different sizes of model. There is generally smooth improvement with model size although the random insertion task shows an upward slope of improvement with the 175B model solving the task the majority of the time. Scaling of one-shot and zero-shot performance is shown in the appendix. All tasks are done with $K=100$.

图 3.11：不同规模模型在五个单词打乱任务上的少样本表现。通常随着模型规模的增大表现会平稳提升，尽管随机插入任务显示出向上的改善趋势，其中 175B 模型在大多数情况下能够解决该任务。单次和零次学习表现的规模化展示在附录中。所有任务均使用 $K=100$ 完成。

• • Cycle letters in word (CL) – The model is given a word with its letters cycled, then the “=” symbol, and is expected to generate the original word. For example, it might be given “lyinevitab” and should output “inevitably”.
• • Anagrams of all but first and last characters (A1) – The model is given a word where every letter except the first and last have been scrambled randomly, and must output the original word. Example: criroptuon = corruption.
• • Anagrams of all but first and last 2 characters (A2) – The model is given a word where every letter except the first 2 and last 2 have been scrambled randomly, and must recover the original word. Example: opoepnnt $\to$ opponent.
• • Random insertion in word (RI) – A random punctuation or space character is inserted between each letter of a word, and the model must output the original word. Example: s.u!c/c!e.s s i/o/n = succession.
• • Reversed words (RW) – The model is given a word spelled backwards, and must output the original word. Example: stcejbo $\to$ objects.

• • 单词字母循环（CL） – 模型会收到一个字母被循环后的单词，然后是“=”符号，模型需要生成原始单词。例如，可能会给出“lyinevitab”，模型应该输出“inevitably”。
• • 除首尾字母外的字母重排（A1） – 模型会收到一个单词，除了首尾字母外，其余字母已经随机打乱，模型必须输出原始单词。示例：criroptuon = corruption。- • 除首尾两个字符外所有字符的字母重排（A2） – 模型会得到一个单词，该单词除了前两个和后两个字母外，其余字母已被随机打乱，模型必须恢复原始单词。示例：opoepnnt $\to$ opponent。- • 单词中随机插入（RI） – 在单词的每个字母之间插入一个随机标点符号或空格字符，模型必须输出原始单词。示例：s.u!c/c!e.s s i/o/n = succession。- • 倒序单词（RW） – 模型会得到一个倒拼的单词，并且必须输出原始单词。示例：stcejbo $\to$ objects。

For each task we generate 10,000 examples, which we chose to be the top 10,000 most frequent words as measured by [92] of length more than 4 characters and less than 15 characters. The few-shot results are shown in Figure 3.11. Task performance tends to grow smoothly with model size, with the full GPT-3 model achieving 66.9% on removing random insertions, 38.6% on cycling letters, 40.2% on the easier anagram task, and 15.1% on the more difficult anagram task (where only the first and last letters are held fixed). None of the models can reverse the letters in a word.

对于每个任务，我们生成了 10,000 个示例，我们选择使用 [92] 测量的前 10,000 个最常用单词，这些单词长度大于 4 个字符且小于 15 个字符。少样本的结果显示在图 3.11 中。任务性能随着模型规模的增大而趋于平滑增长，完整的 GPT-3 模型在移除随机插入时达到 66.9%，在字母循环任务中达到 38.6%，在较简单的字谜任务中达到 40.2%，在较困难的字谜任务中达到 15.1%（其中仅固定首尾字母）。没有任何模型能够将单词中的字母倒转。

In the one-shot setting, performance is significantly weaker (dropping by half or more), and in the zero-shot setting the model can rarely perform any of the tasks (Table 3.10). This suggests that the model really does appear to learn these tasks at test time, as the model cannot perform them zero-shot and their artificial nature makes them unlikely to appear in the pre-training data (although we cannot confirm this with certainty).

在一次性学习（one-shot）设置下，性能显著下降（下降一半或更多），而在零次学习（zero-shot）设置下，模型几乎无法执行任何任务（表 3.10）。这表明模型在测试时确实似乎学会了这些任务，因为模型无法零次学习地完成它们，而且这些任务的人工性质使它们不太可能出现在预训练数据中（尽管我们不能完全确定这一点）。

We can further quantify performance by plotting “in-context learning curves”, which show task performance as a function of the number of in-context examples. We show in-context learning curves for the Symbol Insertion task in Figure 1.2. We can see that larger models are able to make increasingly effective use of in-context information, including both task examples and natural language task descriptions.

我们可以通过绘制“上下文学习曲线”来进一步量化性能，该曲线显示任务性能与上下文示例数量的关系。我们在图1.2中展示了符号插入任务的上下文学习曲线。我们可以看到，更大的模型能够越来越有效地利用上下文信息，包括任务示例和自然语言任务描述。

Finally, it is worth adding that solving these tasks requires character-level manipulations, whereas our BPE encoding operates on significant fractions of a word (on average $\sim 0.7$ words per token), so from the LM’s perspective succeeding at these tasks involves not just manipulating BPE tokens but understanding and pulling apart their substructure. Also, CL, A1, and A2 are not bijective (that is, the unscrambled word is not a deterministic function of the scrambled word), requiring the model to perform some search to find the correct unscrambling. Thus, the skills involved appear to require non-trivial pattern-matching and computation.

最后，值得补充的是，解决这些任务需要字符级别的操作，而我们的 BPE 编码是基于单词的较大部分进行的（平均每个 token $\sim 0.7$ 个单词），因此从语言模型的角度来看，完成这些任务不仅涉及操作 BPE token，还需要理解并拆解其子结构。此外，CL、A1 和 A2 不是双射的（也就是说，未打乱的单词不是打乱单词的确定性函数），因此模型需要进行一些搜索才能找到正确的解乱。因此，所涉及的技能似乎需要非平凡的模式匹配和计算能力。

3.9.3 SAT Analogies

3.9.3 SAT 类比

To test GPT-3 on another task that is somewhat unusual relative to the typical distribution of text, we collected a set of 374 “SAT analogy” problems [131]. Analogies are a style of multiple choice question that constituted a section of the SAT college entrance exam before 2005. A typical example is “audacious is to boldness as (a) sanctimonious is to hypocrisy, (b) anonymous is to identity, (c) remorseful is to misdeed, (d) deleterious is to result, (e) impressionable is to temptation”. The student is expected to choose which of the five word pairs has the same relationship as the original word pair; in this example the answer is “sanctimonious is to hypocrisy”. On this task GPT-3 achieves 65.2% in the few-shot setting, 59.1% in the one-shot setting, and 53.7% in the zero-shot setting, whereas the average score among college applicants was 57% [129] (random guessing yields 20%). As shown in Figure 3.12, the results improve with scale, with the the full 175 billion model improving by over 10% compared to the 13 billion parameter model.

为了在一个相对于典型文本分布来说有些不同的任务上测试 GPT-3，我们收集了一组 374 道“SAT 类比”问题 [131]。类比是一种选择题风格的问题，曾在 2005 年之前构成 SAT 大学入学考试的一部分。一个典型的例子是“audacious 与 boldness 的关系如同 (a) sanctimonious 与 hypocrisy 的关系，(b) anonymous 与 identity 的关系，(c) remorseful 与 misdeed 的关系，(d) deleterious 与 result 的关系，(e) impressionable 与 temptation 的关系”。学生需要选择这五组词中哪一组与原始词对具有相同的关系；在这个例子中，答案是“sanctimonious 与 hypocrisy”的关系。在这个任务中，GPT-3 在少量示例设置下达到 65.2%，在单示例设置下达到 59.1%，在零示例设置下达到 53.7%，而大学申请者的平均分为 57% [129]（随机猜测为 20%）。如图 3.12 所示，结果随着规模的增加而提高，完整的 1750 亿参数模型相比 130 亿参数模型提高了超过 10%。

Figure 3.12: Zero-, one-,and few-shot performance on SAT analogy tasks, for different sizes of model. The largest model achieves 65% accuracy in the few-shot setting, and also demonstrates significant gains to in-context learning which are not present in smaller models.

图 3.12：不同规模模型在 SAT 类比任务上的零样本、单样本和少样本表现。最大模型在少样本设定中达到 65% 的准确率，并且在上下文学习中也表现出显著提升，而较小模型中不存在这种提升。

3.9.4 News Article Generation

3.9.4 新闻文章生成

Previous work on generative language models qualitatively tested their ability to generate synthetic “news articles” by conditional sampling from the model given a human-written prompt consisting of a plausible first sentence for a news story [117]. Relative to [117], the dataset used to train GPT-3 is much less weighted towards news articles, so trying to generate news articles via raw unconditional samples is less effective – for example GPT-3 often interprets the proposed first sentence of a “news article” as a tweet and then posts synthetic responses or follow-up tweets. To solve this problem we employed GPT-3’s few-shot learning abilities by providing three previous news articles in the model’s context to condition it. With the title and subtitle of a proposed next article, the model is able to reliably generate short articles in the “news” genre.

以往关于生成式语言模型的研究通过定性测试它们生成合成“新闻文章”的能力，方法是给模型一个由人为撰写、作为新闻故事合理第一句的提示，并从模型中进行条件采样 [117]。相对于[117]，用于训练GPT-3的数据集在新闻文章上的权重要小得多，因此尝试通过原始无条件样本生成新闻文章的效果较差——例如，GPT-3经常将拟生成的“新闻文章”的第一句话解读为推文，然后发布合成的回复或后续推文。为了解决这个问题，我们通过在模型的上下文中提供三篇之前的新闻文章来利用 GPT-3 的少量示例学习能力，以对其进行条件化。通过提供拟定下一篇文章的标题和副标题，模型能够可靠地生成“新闻”类型的短篇文章。

To gauge the quality of news article generation from GPT-3 (which we believe is likely to be correlated with conditional sample generation quality in general), we decided to measure human ability to distinguish GPT-3-generated articles from real ones. Similar work has been carried out by Kreps et al. [56] and Zellers et al. [141]. Generative language models are trained to match the distribution of content generated by humans, so the (in)ability of humans to distinguish the two is a potentially important measure of quality.[^3]

为了评估GPT-3生成新闻文章的质量（我们认为这可能与条件样本生成质量总体相关），我们决定测量人类区分GPT-3生成的文章与真实文章的能力。类似的工作已由Kreps等人[56]和Zellers等人[141]进行。生成式语言模型经过训练以匹配人类生成内容的分布，因此人类区分两者的（不）能力可能是衡量质量的重要标准。[^3]

In order to see how well humans can detect model generated text, we arbitrarily selected 25 article titles and subtitles from the website newser.com (mean length: 215 words). We then generated completions of these titles and subtitles from four language models ranging in size from 125M to 175B (GPT-3) parameters (mean length: 200 words). For each model, we presented around 80 US-based participants with a quiz consisting of these real titles and subtitles followed by either the human written article or the article generated by the model[^4]. Participants were asked to select whether the article was “very likely written by a human”, “more likely written by a human”, “I don’t know”, “more likely written by a machine”, or “very likely written by a machine”.

为了看看人类在多大程度上能检测出模型生成的文本，我们随意从网站 newser.com 上选择了 25 个文章标题和副标题（平均长度：215 个单词）。然后，我们使用四个语言模型生成了这些标题和副标题的补全，这些模型的参数规模从 1.25 亿到 1750 亿（GPT-3）不等（平均长度：200 个单词）。对于每个模型，我们向大约80名美国参与者展示了一个问答测验，测验内容包括这些真实的标题和副标题，随后是人工撰写的文章或模型生成的文章[^4]。参与者需要选择文章是“很可能是人类写的”、“更可能是人类写的”、“我不知道”、“更可能是机器写的”还是“很可能是机器写的”。

The articles we selected were not in the models’ training data and the model outputs were formatted and selected programmatically to prevent human cherry-picking. All models used the same context to condition outputs on and were pre-trained with the same context size and the same article titles and subtitles were used as prompts for each model. However, we also ran an experiment to control for participant effort and attention that followed the same format but involved intentionally bad model generated articles. This was done by generating articles from a “control model”: a 160M parameter model with no context and increased output randomness.

我们选择的文章不在模型的训练数据中，模型的输出是通过程序化方式格式化和选择的，以防止人工挑选。所有模型使用相同的上下文来调节输出，并且都使用相同的上下文大小进行预训练，每个模型都使用相同的文章标题和副标题作为提示。然而，我们还进行了一项实验，以控制参与者的努力和注意力，该实验遵循相同的格式，但涉及故意生成质量较差的模型文章。这是通过使用“控制模型”生成文章来完成的：该模型具有1.6亿参数，不使用上下文，并增加输出的随机性。

Mean human accuracy (the ratio of correct assignments to non-neutral assignments per participant) at detecting that the intentionally bad articles were model generated was $\sim 86\%$  where 50% is chance level performance. By contrast, mean human accuracy at detecting articles that were produced by the 175B parameter model was barely above chance at $\sim 52\%$ (see Table 3.11).[^5] Human abilities to detect model generated text appear to decrease as model size increases: there appears to be a trend towards chance accuracy with model size, and human detection of GPT-3 is close to chance.[^6] This is true despite the fact that participants spend more time on each output as model size increases (see Appendix E).

检测故意编写的糟糕文章是由模型生成的人的平均准确率（每个参与者正确分配与非中性分配的比例）为$\sim 86\%$，其中50%为随机水平表现。相比之下，人类在检测由175B参数模型生成的文章时的平均准确率仅略高于偶然水平$\sim 52\%$（见表3.11）[^5]。人类检测模型生成文本的能力似乎随着模型规模的增大而下降：随着模型规模的增大，准确率趋向偶然水平，而人类对GPT-3生成文本的检测接近偶然水平[^6]。尽管参与者在每篇文章上花费的时间更多，这一情况仍然成立。随着模型规模的增加，产出也增加（见附录 E）。

	Mean accuracy	95% Confidence Interval (low, hi)	$t$ compared to control ($p$-value)	“I don’t know” assignments
Control (deliberately bad model)	86%	83%–90%	-	3.6 %
GPT-3 Small	76%	72%–80%	3.9 (2$e$-4)	4.9%
GPT-3 Medium	61%	58%–65%	10.3 (7$e$-21)	6.0%
GPT-3 Large	68%	64%–72%	7.3 (3$e$-11)	8.7%
GPT-3 XL	62%	59%–65%	10.7 (1$e$-19)	7.5%
GPT-3 2.7B	62%	58%–65%	10.4 (5$e$-19)	7.1%
GPT-3 6.7B	60%	56%–63%	11.2 (3$e$-21)	6.2%
GPT-3 13B	55%	52%–58%	15.3 (1$e$-32)	7.1%
GPT-3 175B	52%	49%–54%	16.9 (1$e$-34)	7.8%

Table 3.11: Human accuracy in identifying whether short ($\sim$200 word) news articles are model generated. We find that human accuracy (measured by the ratio of correct assignments to non-neutral assignments) ranges from 86% on the control model to 52% on GPT-3 175B. This table compares mean accuracy between five different models, and shows the results of a two-sample T-Test for the difference in mean accuracy between each model and the control model (an unconditional GPT-3 Small model with increased output randomness).

	平均准确率	95% 置信区间（低，高）	$t$ 与对照组比较（$p$值）	“我不知道” 分配比例
对照组（故意表现差的模型）	86%	83%–90%	-	3.6%
GPT-3 小型	76%	72%–80%	3.9 (2$e$-4)	4.9%
GPT-3 中型	61%	58%–65%	10.3 (7$e$-21)	6.0%
GPT-3 大型	68%	64%–72%	7.3 (3$e$-11)	8.7%
GPT-3 XL	62%	59%–65%	10.7 (1$e$-19)	7.5%
GPT-3 2.7B	62%	58%–65%	10.4 (5$e$-19)	7.1%
GPT-3 6.7B	60%	56%–63%	11.2 (3$e$-21)	6.2%
GPT-3 13B	55%	52%–58%	15.3 (1$e$-32)	7.1%
GPT-3 175B	52%	49%–54%	16.9 (1$e$-34)	7.8%
表 3.11：人类准确识别短篇（$\sim$200词）新闻文章是否由模型生成的准确率。我们发现，人类准确率（通过正确判断占非中性判断的比率来衡量）从对控制模型的 86% 到对 GPT-3 175B 的 52% 不等。本表比较了五种不同模型之间的平均准确率，并显示了每种模型与控制模型（一个增加输出随机性的无条件 GPT-3 Small 模型）之间平均准确率差异的双样本 T 检验结果。

Figure 3.13: People’s ability to identify whether news articles are model-generated (measured by the ratio of correct assignments to non-neutral assignments) decreases as model size increases. Accuracy on the outputs on the deliberately-bad control model (an unconditioned GPT-3 Small model with higher output randomness) is indicated with the dashed line at the top, and the random chance (50%) is indicated with the dashed line at the bottom. Line of best fit is a power law with 95% confidence intervals.

图 3.13：随着模型规模的增加，人们识别新闻文章是否由模型生成的能力（通过正确分配与非中性分配的比率衡量）下降。故意设置较差的控制模型（一个无条件的 GPT-3 Small 模型且输出随机性较高）的输出准确率用顶部的虚线表示，而随机机会（50%）用底部的虚线表示。最佳拟合线是具有95%置信区间的幂律。

Examples of synthetic articles from GPT-3 are given in Figures 3.14 and 3.15.[^7] Much of the text is—as indicated by the evaluations—difficult for humans to distinguish from authentic human content. Factual inaccuracies can be an indicator that an article is model generated since, unlike human authors, the models have no access to the specific facts that the article titles refer to or when the article was written. Other indicators include repetition, non sequiturs, and unusual phrasings, though these are often subtle enough that they are not noticed.

GPT-3 生成的合成文章示例见图 3.14 和 3.15。[ ^7 ] 正如评估所示，文本中的大部分内容对人类来说很难与真实的人类内容区分开。事实不准确可能是文章由模型生成的一个指标，因为与人类作者不同，模型无法获取文章标题所指的具体事实，也无法知道文章的写作时间。其他指标包括重复、非逻辑性言论和不寻常的措辞，尽管这些通常都很微妙，以至于不易被注意到。

Title: United Methodists Agree to Historic Split Subtitle: Those who oppose gay marriage will form their own denomination Article: After two days of intense debate, the United Methodist Church has agreed to a historic split - one that is expected to end in the creation of a new denomination, one that will be ”theologically and socially conservative,” according to The Washington Post. The majority of delegates attending the church’s annual General Conference in May voted to strengthen a ban on the ordination of LGBTQ clergy and to write new rules that will ”discipline” clergy who officiate at same-sex weddings. But those who opposed these measures have a new plan: They say they will form a separate denomination by 2020, calling their church the Christian Methodist denomination. The Post notes that the denomination, which claims 12.5 million members, was in the early 20th century the ”largest Protestant denomination in the U.S.,” but that it has been shrinking in recent decades. The new split will be the second in the church’s history. The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church. The Post notes that the proposed split ”comes at a critical time for the church, which has been losing members for years,” which has been ”pushed toward the brink of a schism over the role of LGBTQ people in the church.” Gay marriage is not the only issue that has divided the church. In 2016, the denomination was split over ordination of transgender clergy, with the North Pacific regional conference voting to ban them from serving as clergy, and the South Pacific regional conference voting to allow them.

Title: United Methodists Agree to Historic Split Subtitle: Those who oppose gay marriage will form their own denomination Article: After two days of intense debate, the United Methodist Church has agreed to a historic split - one that is expected to end in the creation of a new denomination, one that will be ”theologically and socially conservative,” according to The Washington Post. The majority of delegates attending the church’s annual General Conference in May voted to strengthen a ban on the ordination of LGBTQ clergy and to write new rules that will ”discipline” clergy who officiate at same-sex weddings. But those who opposed these measures have a new plan: They say they will form a separate denomination by 2020, calling their church the Christian Methodist denomination.

The Post notes that the denomination, which claims 12.5 million members, was in the early 20th century the ”largest Protestant denomination in the U.S.,” but that it has been shrinking in recent decades. The new split will be the second in the church’s history. The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church. The Post notes that the proposed split ”comes at a critical time for the church, which has been losing members for years,” which has been ”pushed toward the brink of a schism over the role of LGBTQ people in the church.” Gay marriage is not the only issue that has divided the church. In 2016, the denomination was split over ordination of transgender clergy, with the North Pacific regional conference voting to ban them from serving as clergy, and the South Pacific regional conference voting to allow them.

Figure 3.14: The GPT-3 generated news article that humans had the greatest difficulty distinguishing from a human written article (accuracy: 12%).

标题：联合卫理公会同意历史性分裂
副标题：反对同性婚姻的人将成立自己的教派
文章：经过两天的激烈辩论，联合卫理公会已同意进行一次历史性分裂——据《华盛顿邮报》报道，这预计将导致一个新的教派的成立，该教派将在“神学上和社会上保持保守”。五月份教会年度大会的大多数代表投票支持加强对LGBTQ牧师按立的禁令，并制定新规则，以“惩戒”主持同性婚礼的牧师。但是那些反对这些措施的人有了一个新计划：他们表示将于2020年前成立一个独立的教派，称他们的教会为基督教卫理公会教派。《邮报》指出，该教派声称拥有1250万会员，在20世纪初曾是“美国最大的基督教新教教派”，但近年来其规模一直在缩小。这次新的分裂将是教会历史上的第二次。第一次发生在1968年，当时大约10%的教派成员离开，成立了福音联合弟兄会教会。《华盛顿邮报》指出，拟议的分裂“正值教会处于关键时期，教会多年来一直在失去成员”，并且“因教会中LGBTQ人士的角色问题已被推向分裂的边缘”。同性婚姻并不是唯一分裂教会的问题。2016年，该教派因是否允许跨性别神职人员按立而发生分裂，北太平洋地区会议投票决定禁止他们担任神职人员，而南太平洋地区会议投票决定允许他们担任神职人员。

（以下为论文中的英文样本原文，保持原样以便对照评测设定。）

Title: United Methodists Agree to Historic Split Subtitle: Those who oppose gay marriage will form their own denomination Article: After two days of intense debate, the United Methodist Church has agreed to a historic split - one that is expected to end in the creation of a new denomination, one that will be ”theologically and socially conservative,” according to The Washington Post. The majority of delegates attending the church’s annual General Conference in May voted to strengthen a ban on the ordination of LGBTQ clergy and to write new rules that will ”discipline” clergy who officiate at same-sex weddings. But those who opposed these measures have a new plan: They say they will form a separate denomination by 2020, calling their church the Christian Methodist denomination. The Post notes that the denomination, which claims 12.5 million members, was in the early 20th century the ”largest Protestant denomination in the U.S.,” but that it has been shrinking in recent decades. The new split will be the second in the church’s history. The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church. The Post notes that the proposed split ”comes at a critical time for the church, which has been losing members for years,” which has been ”pushed toward the brink of a schism over the role of LGBTQ people in the church.” Gay marriage is not the only issue that has divided the church. In 2016, the denomination was split over ordination of transgender clergy, with the North Pacific regional conference voting to ban them from serving as clergy, and the South Pacific regional conference voting to allow them.

Title: United Methodists Agree to Historic Split Subtitle: Those who oppose gay marriage will form their own denomination Article: After two days of intense debate, the United Methodist Church has agreed to a historic split - one that is expected to end in the creation of a new denomination, one that will be ”theologically and socially conservative,” according to The Washington Post. The majority of delegates attending the church’s annual General Conference in May voted to strengthen a ban on the ordination of LGBTQ clergy and to write new rules that will ”discipline” clergy who officiate at same-sex weddings. But those who opposed these measures have a new plan: They say they will form a separate denomination by 2020, calling their church the Christian Methodist denomination.

The Post notes that the denomination, which claims 12.5 million members, was in the early 20th century the ”largest Protestant denomination in the U.S.,” but that it has been shrinking in recent decades. The new split will be the second in the church’s history. The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church. The Post notes that the proposed split ”comes at a critical time for the church, which has been losing members for years,” which has been ”pushed toward the brink of a schism over the role of LGBTQ people in the church.” Gay marriage is not the only issue that has divided the church. In 2016, the denomination was split over ordination of transgender clergy, with the North Pacific regional conference voting to ban them from serving as clergy, and the South Pacific regional conference voting to allow them.

Figure 3.14: The GPT-3 generated news article that humans had the greatest difficulty distinguishing from a human written article (accuracy: 12%).

Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, ”I am a shape-shifter. I can’t change the world. I can only change myself.” It was a promise to not change to fit into the Hollywood mold: ”I think that’s a really special thing, to not change yourself. I think it’s a really special thing to say, ‘This is what’s inside of me, I’m proud of it, and I’m not going to be ashamed because of the way that someone else thinks I should be.’” Now, it’s the Oscars, and Phoenix is at it again. But this time, his publicist is saying he’ll be wearing a tux no matter what. Megyn Kelly was not impressed, and she let him have it on The Tonight Show. ”You know, I feel like, I feel like you could have worn the tux,” she says. ”But you’re saying you’re a shape-shifter. I don’t know if you can change your tux, but you can change your mind. You can change your mind. You can change your mind.” Phoenix says he did, but it didn’t stick. ”I was like, ‘Okay, I’m going to wear a tuxedo to this thing.’ And then I thought, ‘I don’t want to wear a tuxedo to this thing.’” Kelly goes on to encourage him to change his mind again, but Phoenix says it’s too late: ”I’m committed to wearing this.”

Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm

Subtitle: Joaquin Phoenix pledged to not change for each awards event

Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, ”I am a shape-shifter. I can’t change the world. I can only change myself.” It was a promise to not change to fit into the Hollywood mold: ”I think that’s a really special thing, to not change yourself. I think it’s a really special thing to say, ‘This is what’s inside of me, I’m proud of it, and I’m not going to be ashamed because of the way that someone else thinks I should be.’” Now, it’s the Oscars, and Phoenix is at it again. But this time, his publicist is saying he’ll be wearing a tux no matter what.

Megyn Kelly was not impressed, and she let him have it on The Tonight Show. ”You know, I feel like, I feel like you could have worn the tux,” she says. ”But you’re saying you’re a shape-shifter. I don’t know if you can change your tux, but you can change your mind. You can change your mind. You can change your mind.” Phoenix says he did, but it didn’t stick. ”I was like, ‘Okay, I’m going to wear a tuxedo to this thing.’ And then I thought, ‘I don’t want to wear a tuxedo to this thing.’” Kelly goes on to encourage him to change his mind again, but Phoenix says it’s too late: ”I’m committed to wearing this.”

Figure 3.15: The GPT-3 generated news article that humans found the easiest to distinguish from a human written article (accuracy: 61%).

标题：星光的燕尾服承诺引来梅根·凯利的讽刺
副标题：华金·菲尼克斯承诺不会为每个颁奖活动改变
文章：**一年前，当华金·菲尼克斯在金球奖红毯上戴着印有“我是变形者。我无法改变世界”的纸袋亮相时，引起了媒体的广泛关注。“我只能改变自己。” 这是一个承诺，不去改变自己以迎合好莱坞的模式：“我认为，不改变自己是一件非常特别的事情。我认为说‘这是我的本心，我为此感到自豪，我不会因为别人认为我应该怎样而感到羞愧’是一件非常特别的事情。” 现在，又到了奥斯卡，菲尼克斯再次如此做了。但是这一次，他的公关人员表示无论如何他都会穿燕尾服。梅根·凯利并不觉得印象深刻，并且在《今夜秀》上直言不讳。“你知道，我觉得，我觉得你本可以穿燕尾服的，”她说。“但你说你是一个变形者。我不知道你是否能改变你的燕尾服，但你可以改变主意。你可以改变主意。”“你可以改变主意。”菲尼克斯说他确实改变了主意，但没有坚持下来。 “我当时想，‘好吧，我要穿礼服去这个活动。’然后我又想，‘我不想穿礼服去这个活动。’”凯利接着鼓励他再次改变主意，但菲尼克斯说已经太晚了：‘我已经决定穿这件了。’

（以下为论文中的英文样本原文，保持原样以便对照评测设定。）

Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, ”I am a shape-shifter. I can’t change the world. I can only change myself.” It was a promise to not change to fit into the Hollywood mold: ”I think that’s a really special thing, to not change yourself. I think it’s a really special thing to say, ‘This is what’s inside of me, I’m proud of it, and I’m not going to be ashamed because of the way that someone else thinks I should be.’” Now, it’s the Oscars, and Phoenix is at it again. But this time, his publicist is saying he’ll be wearing a tux no matter what. Megyn Kelly was not impressed, and she let him have it on The Tonight Show. ”You know, I feel like, I feel like you could have worn the tux,” she says. ”But you’re saying you’re a shape-shifter. I don’t know if you can change your tux, but you can change your mind. You can change your mind. You can change your mind.” Phoenix says he did, but it didn’t stick. ”I was like, ‘Okay, I’m going to wear a tuxedo to this thing.’ And then I thought, ‘I don’t want to wear a tuxedo to this thing.’” Kelly goes on to encourage him to change his mind again, but Phoenix says it’s too late: ”I’m committed to wearing this.”

Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm

Subtitle: Joaquin Phoenix pledged to not change for each awards event

Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, ”I am a shape-shifter. I can’t change the world. I can only change myself.” It was a promise to not change to fit into the Hollywood mold: ”I think that’s a really special thing, to not change yourself. I think it’s a really special thing to say, ‘This is what’s inside of me, I’m proud of it, and I’m not going to be ashamed because of the way that someone else thinks I should be.’” Now, it’s the Oscars, and Phoenix is at it again. But this time, his publicist is saying he’ll be wearing a tux no matter what.

Megyn Kelly was not impressed, and she let him have it on The Tonight Show. ”You know, I feel like, I feel like you could have worn the tux,” she says. ”But you’re saying you’re a shape-shifter. I don’t know if you can change your tux, but you can change your mind. You can change your mind. You can change your mind.” Phoenix says he did, but it didn’t stick. ”I was like, ‘Okay, I’m going to wear a tuxedo to this thing.’ And then I thought, ‘I don’t want to wear a tuxedo to this thing.’” Kelly goes on to encourage him to change his mind again, but Phoenix says it’s too late: ”I’m committed to wearing this.”

Figure 3.15: The GPT-3 generated news article that humans found the easiest to distinguish from a human written article (accuracy: 61%).

Related work on language model detection by Ippolito et al. [48] indicates that automatic discriminators like Grover [141] and GLTR [37] may have greater success at detecting model generated text than human evaluators. Automatic detection of these models may be a promising area of future research.

Ippolito 等人 [48] 关于语言模型检测的相关工作表明，像 Grover [141] 和 GLTR [37] 这样的自动判别器在检测模型生成的文本方面可能比人类评估者更成功。对这些模型的自动检测可能是未来研究的一个有前景的领域。

Ippolito et al. [48] also note that human accuracy at detecting model generated text increases as humans observe more tokens. To do a preliminary investigation of how good humans are at detecting longer news articles generated by GPT-3 175B, we selected 12 world news articles from Reuters with an average length of 569 words and generated completions of these articles from GPT-3 with an average length of 498 words (298 words longer than our initial experiments). Following the methodology above, we ran two experiments, each on around 80 US-based participants, to compare human abilities to detect the articles generated by GPT-3 and a control model.

Ippolito 等人 [48] 还指出，当人类观察更多的标记时，识别模型生成文本的准确性会提高。为了对人类在识别由 GPT-3 175B 生成的较长新闻文章方面的能力进行初步调查，我们从路透社选取了 12 篇世界新闻文章，平均长度为 569 个单词，并生成了这些文章的 GPT-3 补全内容，平均长度为 498 个单词（比我们最初的实验长 298 个单词）。按照上述方法，我们进行了两项实验，每项大约有80名美国参与者，以比较人类检测由GPT-3和一个控制模型生成的文章的能力。

We found that mean human accuracy at detecting the intentionally bad longer articles from the control model was $\sim 88\%$, while mean human accuracy at detecting the longer articles that were produced by GPT-3 175B was still barely above chance at $\sim 52\%$ (see Table 3.12). This indicates that, for news articles that are around 500 words long, GPT-3 continues to produce articles that humans find difficult to distinguish from human written news articles.

我们发现，人类在检测控制模型故意生成的较长糟糕文章时的平均准确率为$\sim 88\%$，而检测由GPT-3 175B生成的较长文章时的人类平均准确率仍仅略高于随机水平，为$\sim 52\%$（见表3.12）。这表明，对于大约500字的新闻文章，GPT-3 仍然能够生成人类难以分辨是否由人类撰写的文章。

	Mean accuracy	95% Confidence Interval (low, hi)	$t$ compared to control ($p$-value)	“I don’t know” assignments
Control	88%	84%–91%	-	2.7%
GPT-3 175B	52%	48%–57%	12.7 (3.2$e$-23)	10.6%

Table 3.12: People’s ability to identify whether $\sim 500$ word articles are model generated (as measured by the ratio of correct assignments to non-neutral assignments) was 88% on the control model and 52% on GPT-3 175B. This table shows the results of a two-sample T-Test for the difference in mean accuracy between GPT-3 175B and the control model (an unconditional GPT-3 Small model with increased output randomness).

	平均准确率	95% 置信区间（低，高）	$t$与对照组比较（$p$值）	“我不知道”分配率
对照组	88%	84%–91%	-	2.7%
GPT-3 175B	52%	48%–57%	12.7 (3.2$e$-23)	10.6%
表 3.12：人们判断$\sim 500$单词文章是否由模型生成的能力（以正确判断次数与非中性判断次数的比率衡量）在控制模型上为 88%，在 GPT-3 175B 上为 52%。本表显示了 GPT-3 175B 与控制模型（一个增加输出随机性的无条件 GPT-3 小模型）之间平均准确率差异的双样本 T 检验结果。

A "whatpu" is a small, furry animal native to Tanzania. An example of a sentence that uses the word whatpu is: We were traveling in Africa and we saw these very cute whatpus.   To do a "farduddle" means to jump up and down really fast. An example of a sentence that uses the word farduddle is: **One day when I was playing tag with my little sister, she got really excited and she started doing these crazy farduddles.**A "yalubalu" is a type of vegetable that looks like a big pumpkin. An example of a sentence that uses the word yalubalu is: I was on a trip to Africa and I tried this yalubalu vegetable that was grown in a garden there. It was delicious. A "Burringo" is a car with very fast acceleration. An example of a sentence that uses the word Burringo is: In our garage we have a Burringo that my father drives to work every day. A "Gigamuru" is a type of Japanese musical instrument. An example of a sentence that uses the word Gigamuru is: I have a Gigamuru that my uncle gave me as a gift. I love to play it at home. To "screeg" something is to swing a sword at it. An example of a sentence that uses the word screeg is: We screeghed at each other for several minutes and then we went outside and ate ice cream.

A "whatpu" is a small, furry animal native to Tanzania. An example of a sentence that uses the word whatpu is: We were traveling in Africa and we saw these very cute whatpus.

To do a "farduddle" means to jump up and down really fast. An example of a sentence that uses the word farduddle is: **One day when I was playing tag with my little sister, she got really excited and she started doing these crazy farduddles.**A "yalubalu" is a type of vegetable that looks like a big pumpkin. An example of a sentence that uses the word yalubalu is: I was on a trip to Africa and I tried this yalubalu vegetable that was grown in a garden there. It was delicious.

A "Burringo" is a car with very fast acceleration. An example of a sentence that uses the word Burringo is: In our garage we have a Burringo that my father drives to work every day.

A "Gigamuru" is a type of Japanese musical instrument. An example of a sentence that uses the word Gigamuru is: I have a Gigamuru that my uncle gave me as a gift. I love to play it at home.

To "screeg" something is to swing a sword at it. An example of a sentence that uses the word screeg is: We screeghed at each other for several minutes and then we went outside and ate ice cream.

Figure 3.16: Representative GPT-3 completions for the few-shot task of using a new word in a sentence. Boldface is GPT-3’s completions, plain text is human prompts. In the first example both the prompt and the completion are provided by a human; this then serves as conditioning for subsequent examples where GPT-3 receives successive additional prompts and provides the completions. Nothing task-specific is provided to GPT-3 other than the conditioning shown here.

“whatpu” 是一种原产于坦桑尼亚的小型毛茸茸动物。使用单词 whatpu 的句子示例是：我们正在非洲旅行，我们看到了这些非常可爱的 whatpu。做 “farduddle” 意味着快速上下跳动。

使用单词 farduddle 的句子例子是：有一天，当我和妹妹玩捉迷藏时，她非常兴奋，开始做这些疯狂的 farduddles。 “yalubalu” 是一种看起来像大南瓜的蔬菜。使用单词 yalubalu 的句子例子是：我去非洲旅行时，尝试了在那里花园里种植的这种 yalubalu 蔬菜。

很好吃。** “Burringo”是一种加速非常快的汽车。使用Burringo这个词的句子示例是：** 在我们的车库里有一辆Burringo，我父亲每天开它去上班。** “Gigamuru”是一种日本乐器。使用Gigamuru这个词的句子示例是：** 我有一个我叔叔送给我的Gigamuru作为礼物。我喜欢在家里弹奏它。** “screeg”某物是指用剑向它挥砍。

使用单词 'screeg' 的句子示例是：我们互相 screeg 了好几分钟，然后我们出去吃冰淇淋。

“Whatpu”是一种原产于坦桑尼亚的小型毛茸动物。使用“whatpu”这个词的句子示例是：我们在非洲旅行时，看到了这些非常可爱的whatpu。

做“farduddle”意味着快速地上下跳。使用单词 farduddle 的句子示例是：有一天，当我和妹妹玩捉迷藏时，她非常兴奋，开始做这些疯狂的 farduddle。“yalubalu”是一种看起来像大南瓜的蔬菜。

使用单词 yalubalu 的一句话示例是：我去非洲旅行时尝试了在那里的一个花园里种植的 yalubalu 菜。它非常美味。

“Burringo” 是一种加速非常快的汽车。使用 Burringo 这个词的句子示例是：在我们的车库里，我们有一辆 Burringo，我父亲每天开它上班。

“Gigamuru” 是一种日本乐器。使用 Gigamuru 这个词的句子示例是：我有一把 Gigamuru，是我叔叔送给我的礼物。我喜欢在家里演奏它。

“screeg” 某物的意思是用剑向它挥去。使用 screeg 这个词的句子例子是：我们互相 screeg 了好几分钟，然后我们出去吃了冰淇淋。

图 3.16：使用新词造句的少样本任务中 GPT-3 的代表性完成结果。加粗部分是 GPT-3 的完成内容，普通文本是人类提示。在第一个例子中，提示和完成内容均由人类提供；然后，这作为后续示例的条件，GPT-3 接收连续的额外提示并提供完成内容。

除了这里显示的条件外，没有向 GPT-3 提供任何特定任务的内容。

Poor English input: I eated the purple berries. Good English output: I ate the purple berries. Poor English input: Thank you for picking me as your designer. I’d appreciate it. Good English output: Thank you for choosing me as your designer. I appreciate it. Poor English input: The mentioned changes have done. or I did the alteration that you requested. or I changed things you wanted and did the modifications. Good English output: The requested changes have been made. or I made the alteration that you requested. or I changed things you wanted and made the modifications. Poor English input: I’d be more than happy to work with you in another project. Good English output: I’d be more than happy to work with you on another project. Poor English input: Please provide me with a short brief of the design you’re looking for and that’d be nice if you could share some examples or project you did before. Good English output: Please provide me with a brief description of the design you’re looking for and that would be nice if you could share some examples or projects you have done before. Poor English input: The patient was died. Good English output: The patient died. Poor English input: We think that Leslie likes ourselves. Good English output: We think that Leslie likes us. Poor English input: Janet broke Bill on the finger. Good English output: Janet broke Bill’s finger. Poor English input: Mary arranged for, in St. Louis, John to rent a house cheap. Good English output: Mary arranged for John to rent a house in St. Louis. Poor English input: Today I have went to the store to to buys some many bottle of water. Good English output: Today I went to the store to buy some bottles of water. Poor English input: I have tried to hit ball with bat, but my swing is has miss. Good English output: I tried to hit the ball with the bat, but my swing missed.

Poor English input: I eated the purple berries. Good English output: I ate the purple berries. Poor English input: Thank you for picking me as your designer. I’d appreciate it. Good English output: Thank you for choosing me as your designer. I appreciate it. Poor English input: The mentioned changes have done. or I did the alteration that you requested. or I changed things you wanted and did the modifications. Good English output: The requested changes have been made. or I made the alteration that you requested. or I changed things you wanted and made the modifications. Poor English input: I’d be more than happy to work with you in another project. Good English output: I’d be more than happy to work with you on another project.

Poor English input: Please provide me with a short brief of the design you’re looking for and that’d be nice if you could share some examples or project you did before. Good English output: Please provide me with a brief description of the design you’re looking for and that would be nice if you could share some examples or projects you have done before.

Poor English input: The patient was died. Good English output: The patient died.

Poor English input: We think that Leslie likes ourselves. Good English output: We think that Leslie likes us.

Poor English input: Janet broke Bill on the finger. Good English output: Janet broke Bill’s finger.

Poor English input: Mary arranged for, in St. Louis, John to rent a house cheap. Good English output: Mary arranged for John to rent a house in St. Louis.

Poor English input: Today I have went to the store to to buys some many bottle of water. Good English output: Today I went to the store to buy some bottles of water.

Poor English input: I have tried to hit ball with bat, but my swing is has miss. Good English output: I tried to hit the ball with the bat, but my swing missed.

Figure 3.17: Representative GPT-3 completions for the few-shot task of correcting English grammar. Boldface is GPT-3’s completions, plain text is human prompts. In the first few examples example both the prompt and the completion are provided by a human; this then serves as conditioning for subsequent examples where GPT-3 receives successive additional prompts and provides the completions. Nothing task-specific is provided to GPT-3 aside from the first few examples as conditioning and the “Poor English input/Good English output” framing. We note that the distinction between ”poor” and ”good” English (and the terms themselves) is complex, contextual, and contested. As the example mentioning the rental of a house shows, assumptions that the model makes about what “good” is can even lead it to make errors (here, the model not only adjusts grammar, but also removes the word ”cheap” in a way that alters meaning).

糟糕的英语输入：I eated the purple berries. 正确的英语输出：I ate the purple berries. 糟糕的英语输入：Thank you for picking me as your designer. I’d appreciate it. 正确的英语输出：Thank you for choosing me as your designer. I appreciate it. 糟糕的英语输入：The mentioned changes have done. 或者 I did the alteration that you requested.或者我修改了你想要的东西并进行了调整。良好的英语表达：已完成所请求的更改。或者我做了你要求的修改。或者我更改了你想要的东西并进行了调整。差的英语输入：我很乐意与你合作另一个项目。良好的英文输出：我非常乐意与您在另一个项目上合作。 英文糟糕输入：请向我提供您正在寻找的设计的简短说明，如果您能分享一些之前做过的案例或项目，那就太好了。良好的英语表达：请简要描述您所期望的设计，如果您能分享一些之前的例子或项目，那就更好了。 不良英语表达：The patient was died. 良好的英语表达：The patient died. 不良英语表达：We think that Leslie likes ourselves.良好的英语输出：我们认为 Leslie 喜欢我们。 不良的英语输入：Janet 把 Bill 的手指弄坏了。良好的英语输出：Janet 弄断了 Bill 的手指。 不良的英语输入：Mary 安排了，在圣路易斯，John 租一间便宜的房子。良好的英语输出：Mary 安排 John 在圣路易斯租一间房子。路易 差的英文输入：今天我去商店去买了很多瓶水。 正确的英文输出：今天我去商店买了一些瓶水。 差的英文输入：我试着用球棒打球，但我的挥棒没打到。 正确的英文输出：我试着用球棒打球，但我的挥棒漏打了。

（以下为论文中的英文样本原文，保持原样以便对照评测设定。）

Poor English input: I eated the purple berries. Good English output: I ate the purple berries. Poor English input: Thank you for picking me as your designer. I’d appreciate it. Good English output: Thank you for choosing me as your designer. I appreciate it. Poor English input: The mentioned changes have done. or I did the alteration that you requested. or I changed things you wanted and did the modifications. Good English output: The requested changes have been made. or I made the alteration that you requested. or I changed things you wanted and made the modifications. Poor English input: I’d be more than happy to work with you in another project. Good English output: I’d be more than happy to work with you on another project. Poor English input: Please provide me with a short brief of the design you’re looking for and that’d be nice if you could share some examples or project you did before. Good English output: Please provide me with a brief description of the design you’re looking for and that would be nice if you could share some examples or projects you have done before. Poor English input: The patient was died. Good English output: The patient died. Poor English input: We think that Leslie likes ourselves. Good English output: We think that Leslie likes us. Poor English input: Janet broke Bill on the finger. Good English output: Janet broke Bill’s finger. Poor English input: Mary arranged for, in St. Louis, John to rent a house cheap. Good English output: Mary arranged for John to rent a house in St. Louis. Poor English input: Today I have went to the store to to buys some many bottle of water. Good English output: Today I went to the store to buy some bottles of water. Poor English input: I have tried to hit ball with bat, but my swing is has miss. Good English output: I tried to hit the ball with the bat, but my swing missed.

Poor English input: I eated the purple berries. Good English output: I ate the purple berries. Poor English input: Thank you for picking me as your designer. I’d appreciate it. Good English output: Thank you for choosing me as your designer. I appreciate it. Poor English input: The mentioned changes have done. or I did the alteration that you requested. or I changed things you wanted and did the modifications. Good English output: The requested changes have been made. or I made the alteration that you requested. or I changed things you wanted and made the modifications. Poor English input: I’d be more than happy to work with you in another project. Good English output: I’d be more than happy to work with you on another project.

Poor English input: Please provide me with a short brief of the design you’re looking for and that’d be nice if you could share some examples or project you did before. Good English output: Please provide me with a brief description of the design you’re looking for and that would be nice if you could share some examples or projects you have done before.

Poor English input: The patient was died. Good English output: The patient died.

Poor English input: We think that Leslie likes ourselves. Good English output: We think that Leslie likes us.

Poor English input: Janet broke Bill on the finger. Good English output: Janet broke Bill’s finger.

Poor English input: Mary arranged for, in St. Louis, John to rent a house cheap. Good English output: Mary arranged for John to rent a house in St. Louis.

Poor English input: Today I have went to the store to to buys some many bottle of water. Good English output: Today I went to the store to buy some bottles of water.

Poor English input: I have tried to hit ball with bat, but my swing is has miss. Good English output: I tried to hit the ball with the bat, but my swing missed.

Figure 3.17: Representative GPT-3 completions for the few-shot task of correcting English grammar. Boldface is GPT-3’s completions, plain text is human prompts. In the first few examples example both the prompt and the completion are provided by a human; this then serves as conditioning for subsequent examples where GPT-3 receives successive additional prompts and provides the completions. Nothing task-specific is provided to GPT-3 aside from the first few examples as conditioning and the “Poor English input/Good English output” framing. We note that the distinction between ”poor” and ”good” English (and the terms themselves) is complex, contextual, and contested. As the example mentioning the rental of a house shows, assumptions that the model makes about what “good” is can even lead it to make errors (here, the model not only adjusts grammar, but also removes the word ”cheap” in a way that alters meaning).

3.9.5 Learning and Using Novel Words

3.9.5 学习和使用新词

A task studied in developmental linguistics [13] is the ability to learn and utilize new words, for example using a word in a sentence after seeing it defined only once, or conversely inferring a word’s meaning from only one usage. Here we qualitatively test GPT-3’s ability to do the former. Specifically, we give GPT-3 the definition of a nonexistent word, such as “Gigamuru”, and then ask it to use it in a sentence. We provide one to five previous examples of a (separate) nonexistent word being defined and used in a sentence, so the task is few-shot in terms of previous examples of the broad task and one-shot in terms of the specific word. Table 3.16 shows the 6 examples we generated; all definitions were human-generated, and the first answer was human-generated as conditioning while the subsequent answers were generated by GPT-3. These examples were generated continuously in one sitting and we did not omit or repeatedly try any prompts. In all cases the generated sentence appears to be a correct or at least plausible use of the word. In the final sentence the model generates a plausible conjugation for the word “screeg” (namely “screeghed”), although the use of the word is slightly awkward (“screeghed at each other”) despite being plausible in the sense that it could describe a toy sword fight. Overall, GPT-3 appears to be at least proficient at the task of using novel words in a sentence.

发展语言学中研究的一个任务是学习和使用新词的能力，例如在只看到一个单独的定义后就能在句子中使用一个词，或者相反，仅通过一次使用就推断出一个词的含义。在这里，我们定性地测试 GPT-3 执行前者的能力。具体来说，我们给 GPT-3 一个不存在的词的定义，例如“Gigamuru”，然后要求它在句子中使用该词。我们提供一到五个先前的例子，展示一个（独立的）不存在的单词如何被定义并用于句子中，因此该任务在广义任务的先前示例方面是少样本学习，而在具体单词方面是单样本学习。表 3.16 显示了我们生成的 6 个例子；所有定义均为人工生成，首个答案是作为条件的人类生成，其后答案则由 GPT-3 生成。这些示例是在一次连续的生成过程中产生的，我们没有省略或多次尝试任何提示。在所有情况下，生成的句子似乎都是对该词的正确或至少合理的使用。在最后一句话中，模型为单词“screeg”生成了一个合理的变位形式（即“screeghed”），尽管这个单词的使用略显生硬（“screeghed at each other”），但从描述玩具剑战斗的角度来看是合理的。总体而言，GPT-3 在将新词用于句子中的任务上至少表现出熟练。

3.9.6 Correcting English Grammar

3.9.6 英语语法纠正

Another task well suited for few-shot learning is correcting English grammar. We test this with GPT-3 in the few-shot setting by giving prompts of the form "Poor English Input: \n Good English Output: ". We give GPT-3 one human-generated correction and then ask it to correct 5 more (again without any omissions or repeats). Results are shown in Figure 3.17.

另一个非常适合少样本学习的任务是纠正英语语法。我们在少样本设定下使用 GPT-3 进行测试，方式是提供形式为“糟糕的英文输入：<句子>\n良好的英文输出：<句子>”的提示。我们先给 GPT-3 一个人工生成的纠正示例，然后让它纠正另外 5 个句子（同样不遗漏或重复）。结果显示在图 3.17 中。

4 Measuring and Preventing Memorization Of Benchmarks

4 测量并防止基准集被记忆（Measuring and Preventing Memorization Of Benchmarks）

Figure 4.1: GPT-3 Training Curves   We measure model performance during training on a deduplicated validation split of our training distribution. Though there is some gap between training and validation performance, the gap grows only minimally with model size and training time, suggesting that most of the gap comes from a difference in difficulty rather than overfitting.

图 4.1：GPT-3 训练曲线 我们在训练分布的去重验证集上测量模型在训练过程中的性能。尽管训练性能和验证性能之间存在一些差距，但随着模型规模和训练时间的增加，这一差距仅略有增加，这表明大部分差距来源于难度的差异，而非过拟合。

Since our training dataset is sourced from the internet, it is possible that our model was trained on some of our benchmark test sets. Accurately detecting test contamination from internet-scale datasets is a new area of research without established best practices. While it is common practice to train large models without investigating contamination, given the increasing scale of pretraining datasets, we believe this issue is becoming increasingly important to attend to.

由于我们的训练数据集来源于互联网，因此我们的模型可能在某些基准测试集上进行了训练。从互联网规模的数据集中准确检测测试污染是一个新的研究领域，目前没有成熟的最佳实践。虽然在训练大型模型时通常不会调查数据污染，但鉴于预训练数据集规模的不断扩大，我们认为这个问题正变得越来越重要，需要加以关注。

This concern is not just hypothetical. One of the first papers to train a language model on Common Crawl data [130] detected and removed a training document which overlapped with one of their evaluation datasets. Other work such as GPT-2 [117] also conducted post-hoc overlap analysis. Their study was relatively encouraging, finding that although models did perform moderately better on data that overlapped between training and testing, this did not significantly impact reported results due to the small fraction of data which was contaminated (often only a few percent).

这个问题不仅仅是理论上的。其中一篇最早基于 Common Crawl 数据训练语言模型的论文 [130] 检测并移除了与其评估数据集之一重叠的训练文档。其他工作，例如 GPT-2 [117]，也进行了事后的重叠分析。他们的研究相对令人鼓舞，发现尽管模型在训练和测试之间重叠的数据上表现稍好，但由于受污染的数据比例很小（通常只有几个百分点），这对报告的结果影响不大。

GPT-3 operates in a somewhat different regime. On the one hand, the dataset and model size are about two orders of magnitude larger than those used for GPT-2, and include a large amount of Common Crawl, creating increased potential for contamination and memorization. On the other hand, precisely due to the large amount of data, even GPT-3 175B does not overfit its training set by a significant amount, measured relative to a held-out validation set with which it was deduplicated (Figure 4.1). Thus, we expect that contamination is likely to be frequent, but that its effects may not be as large as feared.

GPT-3 的运行方式有所不同。一方面，其数据集和模型规模大约比 GPT-2 使用的要大两个数量级，并且包含大量的 Common Crawl 数据，从而增加了污染和记忆的潜在可能性。另一方面，正是由于数据量庞大，即使是 GPT-3 175B 也不会在其训练集上显著过拟合，这一结论是相对于与其去重后的保留验证集进行测量的（图 4.1）。因此，我们预计数据污染可能会很频繁，但其影响可能没有预期的那么大。

We initially tried to address the issue of contamination by proactively searching for and attempting to remove any overlap between our training data and the development and test sets of all benchmarks studied in this paper. Unfortunately, a bug resulted in only partial removal of all detected overlaps from the training data. Due to the cost of training, it wasn’t feasible to retrain the model. To address this, we investigate in detail how the remaining detected overlap impacts results.

我们最初尝试通过主动搜索并试图移除所有训练数据与本文研究的所有基准的开发集和测试集之间的重叠来解决污染问题。不幸的是，一个错误导致只部分移除了训练数据中检测到的所有重叠。由于训练成本高昂，重新训练模型不可行。为了解决这个问题，我们详细研究了剩余检测到的重叠如何影响结果。

For each benchmark, we produce a ‘clean’ version which removes all potentially leaked examples, defined roughly as examples that have a 13-gram overlap with anything in the pretraining set (or that overlap with the whole example when it is shorter than 13-grams). The goal is to very conservatively flag anything that could potentially be contamination, so as to produce a clean subset that is free of contamination with high confidence. The exact procedure is detailed in Appendix C.

对于每个基准，我们都会生成一个“干净”版本，该版本会移除所有可能泄露的样本，大致定义为与预训练集中的任何内容有13-gram重叠的样本（或者当样本长度小于13-gram时，与整个样本重叠的情况）。我们的目标是以非常保守的方式标记所有可能的污染样本，从而生成一个高置信度下无污染的干净子集。具体程序详见附录C。

We then evaluate GPT-3 on these clean benchmarks, and compare to the original score. If the score on the clean subset is similar to the score on the entire dataset, this suggests that contamination, even if present, does not have a significant effect on reported results. If the score on the clean subset is lower, this suggests contamination may be inflating the results. The results are summarized in Figure 4.2. Although potential contamination is often high (with a quarter of benchmarks scoring over 50%), in most cases performance changes only negligibly, and we see no evidence that contamination level and performance difference are correlated. We conclude that either our conservative method substantially overestimated contamination or that contamination has little effect on performance.

然后我们在这些干净的基准上评估 GPT-3，并与原始得分进行比较。如果干净子集上的得分与整个数据集上的得分相似，这表明即使存在污染，也不会对报告的结果产生显著影响。如果干净子集上的得分较低，则表明污染可能会导致结果被高估。结果总结如图 4.2 所示。尽管潜在的污染通常很高（有四分之一的基准测试得分超过50%），但在大多数情况下性能变化只有微不足道的程度，而且我们没有发现污染水平与性能差异之间存在相关性。我们得出结论，要么我们的保守方法大幅高估了污染，要么污染对性能影响很小。

Below, we review in more detail the few specific cases where either (1) the model performs significantly worse on the cleaned version, or (2) potential contamination is very high, which makes measuring the performance difference difficult.

下面，我们将更详细地回顾一些特定的情况，这些情况要么是（1）模型在清理后的版本上表现明显较差，要么是（2）潜在的污染非常严重，这使得测量性能差异变得困难。

Figure 4.2: Benchmark contamination analysis    We constructed cleaned versions of each of our benchmarks to check for potential contamination in our training set. The x-axis is a conservative lower bound for how much of the dataset is known with high confidence to be clean, and the y-axis shows the difference in performance when evaluating only on the verified clean subset. Performance on most benchmarks changed negligibly, but some were flagged for further review. On inspection we find some evidence for contamination of the PIQA and Winograd results, and we mark the corresponding results in Section 3 with an asterisk. We find no evidence that other benchmarks are affected.

图 4.2：基准污染分析 我们构建了每个基准的清理版本，以检查训练集中的潜在污染。x 轴是数据集中已知高置信度清洁部分的保守下限，y 轴显示仅在经过验证的清洁子集上评估时的性能差异。大多数基准测试的表现几乎没有变化，但有些基准测试被标记为需要进一步审查。经检查，我们发现 PIQA 和 Winograd 的结果存在一些污染的证据，并在第 3 节中用星号标注了相应的结果。我们没有发现其他基准测试受到影响的证据。

Our analysis flagged six groups of benchmarks for further investigation: Word Scrambling, Reading Comprehension (QuAC, SQuAD2, DROP), PIQA, Winograd, language modeling tasks (Wikitext tasks, 1BW), and German to English translation. Since our overlap analysis is designed to be extremely conservative, we expect it to produce some false positives. We summarize the results for each group of tasks below:

我们的分析标出了六组基准以供进一步调查：单词打乱、阅读理解（QuAC、SQuAD2、DROP）、PIQA、Winograd、语言建模任务（Wikitext 任务、1BW）以及德语到英语翻译。由于我们的重叠分析设计得非常保守，我们预计它会产生一些假阳性。我们在下面总结了每组任务的结果：

• • Reading Comprehension: Our initial analysis flagged $>$90% of task examples from QuAC, SQuAD2, and DROP as potentially contaminated, so large that even measuring the differential on a clean subset was difficult. Upon manual inspection, however, we found that for every overlap we inspected, in all 3 datasets, the source text was present in our training data but the question/answer pairs were not, meaning the model gains only background information and cannot memorize the answer to a specific question.
• • German translation: We found 25% of the examples in the WMT16 German-English test set were marked as potentially contaminated, with an associated total effect size of 1-2 BLEU. Upon inspection, none of the flagged examples contain paired sentences resembling NMT training data and collisions were monolingual matches mostly of snippets of events discussed in the news.
• • Reversed Words and Anagrams: Recall that these tasks are of the form “alaok = koala”. Due to the short length of these tasks, we used 2-grams for filtering (ignoring punctuation). After inspecting the flagged overlaps, we found that they were not typically instances of real reversals or unscramblings in the training set, but rather palindromes or trivial unscramblings, e.g “kayak = kayak”. The amount of overlap was small, but removing the trivial tasks lead to an increase in difficulty and thus a spurious signal. Related to this, the symbol insertion task shows high overlap but no effect on performance – this is because that task involves removing non-letter characters from a word, and the overlap analysis itself ignores such characters, leading to many spurious matches.
• • PIQA: The overlap analysis flagged 29% of examples as contaminated, and observed a 3 percentage point absolute decrease (4% relative decrease) in performance on the clean subset. Though the test dataset was released after our training set was created and its labels are hidden, some of the web pages used by the crowdsourced dataset creators are contained in our training set. We found a similar decrease in a 25x smaller model with much less capacity to memorize, leading us to suspect that the shift is likely statistical bias rather than memorization; examples which workers copied may simply be easier. Unfortunately, we cannot rigorously prove this hypothesis. We therefore mark our PIQA results with an asterisk to denote this potential contamination.
• • Winograd: The overlap analysis flagged 45% of examples, and found a 2.6% decrease in performance on the clean subset. Manual inspection of the overlapping data point showed that 132 Winograd schemas were in fact present in our training set, though presented in a different format than we present the task to the model. Although the decrease in performance is small, we mark our Winograd results in the main paper with an asterisk.
• • Language modeling: We found the 4 Wikipedia language modeling benchmarks measured in GPT-2, plus the Children’s Book Test dataset, to be almost entirely contained in our training data. Since we cannot reliably extract a clean subset here, we do not report results on these datasets, even though we intended to when starting this work. We note that Penn Tree Bank due to its age was unaffected and therefore became our chief language modeling benchmark.

• • 阅读理解： 我们的初步分析标记了 QuAC、SQuAD2 和 DROP 中 $>$90% 的任务示例为可能被污染，以至于即使在干净的子集中测量差异也很困难。然而，经过人工检查，我们发现，在我们检查的每一个重叠中，在所有三个数据集中，源文本都出现在我们的训练数据中，但问答对并未出现，这意味着模型仅获得背景信息，而无法记住特定问题的答案。- • 德语翻译： 我们发现 WMT16 德英测试集中有 25% 的例子被标记为可能受污染，其相关总效应量为 1-2 BLEU。经检查，所有被标记的例子中都没有包含类似 NMT 训练数据的成对句子，重复大多是单语匹配，主要是新闻中讨论的事件片段。- • 倒序单词与字谜重组: 请记住，这些任务的形式是“alaok = koala”。由于这些任务长度较短，我们使用了二元组进行过滤（忽略标点符号）。在检查了标记的重叠项后，我们发现它们通常并不是训练集中真实的倒序或重组实例，而是回文或简单的重组，例如“kayak = kayak”。重叠的数量很少，但移除琐碎任务导致难度增加，从而产生了伪信号。与此相关，符号插入任务显示出高度重叠，但对性能没有影响——这是因为该任务涉及从单词中去除非字母字符，而重叠分析本身忽略这些字符，从而导致了许多伪匹配。- • PIQA： 重叠分析显示 29% 的样本被标记为受污染，并且在干净子集上的性能观察到绝对下降 3 个百分点（相对下降 4%）。尽管测试数据集是在我们的训练集创建之后发布的且其标签是隐藏的，但部分众包数据集创建者使用的网页包含在我们的训练集中。我们在一个容量小得多、记忆能力极弱的小模型（小25倍）中也发现了类似的下降，这使我们怀疑这种变化更可能是统计偏差而非记忆；工人复制的示例可能只是更容易。不幸的是，我们无法严格证明这一假设。因此，我们在 PIQA 结果上标记了星号，以表示这种潜在的污染。- • Winograd: 重叠分析标记了 45% 的例子，并在干净子集上发现性能下降了 2.6%。对重叠数据点的人工检查显示，实际上有 132 个 Winograd 架构存在于我们的训练集中，但呈现的格式与我们向模型展示任务的方式不同。尽管性能下降很小，我们在主论文中仍在 Winograd 结果上标注了星号。- • 语言建模: 我们发现 GPT-2 测量的 4 个维基百科语言建模基准，以及《儿童书籍测试》数据集，几乎完全包含在我们的训练数据中。由于我们无法可靠地提取干净的子集，因此即使在开始这项工作时我们有意报告这些数据集的结果，也没有报告这些数据集的结果。我们注意到，宾夕法尼亚树库由于其年代久远未受影响，因此成为了我们的主要语言建模基准。

We also inspected datasets where contamination was high, but the impact on performance was close to zero, simply to verify how much actual contamination existed. These appeared to often contain false positives. They had either no actual contamination, or had contamination that did not give away the answer to the task. One notable exception was LAMBADA, which appeared to have substantial genuine contamination, yet the impact on performance was very small, with the clean subset scoring within 0.5% of the full dataset. Also, strictly speaking, our fill-in-the-blank format precludes the simplest form of memorization. Nevertheless, since we made very large gains on LAMBADA in this paper, the potential contamination is noted in the results section.

我们还检查了污染程度较高的数据集，但对性能的影响几乎为零，仅是为了验证实际存在的污染程度。这些数据集似乎经常包含假阳性。它们要么根本没有实际污染，要么存在的污染并不会泄露任务的答案。一个显著的例外是 LAMBADA，它似乎存在大量的真实污染，但对性能的影响非常小，干净子集的得分与完整数据集相比仅相差 0.5%。此外，严格来说，我们的填空格式排除了最简单的记忆形式。然而，由于我们在本文中在 LAMBADA 上取得了非常大的进展，因此在结果部分注明了潜在的污染情况。

An important limitation of our contamination analysis is that we cannot be sure that the clean subset is drawn from the same distribution as the original dataset. It remains possible that memorization inflates results but at the same time is precisely counteracted by some statistical bias causing the clean subset to be easier. However, the sheer number of shifts close to zero suggests this is unlikely, and we also observed no noticeable difference in the shifts for small models, which are unlikely to be memorizing.

我们污染分析的一个重要限制是，我们无法确定干净子集是否来自与原始数据集相同的分布。仍然有可能记忆效应会夸大结果，但与此同时，又被某种统计偏差恰好抵消，从而使干净子集变得更容易。然而，接近零的大量转变表明这种情况不太可能发生，我们还观察到小型模型的转变没有明显差异，而小型模型不太可能进行记忆。

Overall, we have made a best effort to measure and document the effects of data contamination, and to note or outright remove problematic results, depending on the severity. Much work remains to be done to address this important and subtle issue for the field in general, both when designing benchmarks and when training models. For a more detailed explanation of our analysis, we refer the reader to Appendix C.

总体来说，我们已经尽最大努力去衡量和记录数据污染的影响，并根据严重程度记录或直接删除有问题的结果。为了解决这一重要且微妙的问题，无论是在设计基准测试还是在训练模型时，仍有大量工作需要完成。关于我们分析的更详细说明，读者可参考附录C。

5 Limitations

5 局限（Limitations）

GPT-3 and our analysis of it have a number of limitations. Below we describe some of these and suggest directions for future work.

GPT-3 及我们对它的分析存在一些局限性。下面我们将描述其中的一些，并提出未来工作的方向。

First, despite the strong quantitative and qualitative improvements of GPT-3, particularly compared to its direct predecessor GPT-2, it still has notable weaknesses in text synthesis and several NLP tasks. On text synthesis, although the overall quality is high, GPT-3 samples still sometimes repeat themselves semantically at the document level, start to lose coherence over sufficiently long passages, contradict themselves, and occasionally contain non-sequitur sentences or paragraphs. We will release a collection of 500 uncurated unconditional samples to help provide a better sense of GPT-3’s limitations and strengths at text synthesis. Within the domain of discrete language tasks, we have noticed informally that GPT-3 seems to have special difficulty with “common sense physics”, despite doing well on some datasets (such as PIQA [11]) that test this domain. Specifically GPT-3 has difficulty with questions of the type “If I put cheese into the fridge, will it melt?”. Quantitatively, GPT-3’s in-context learning performance has some notable gaps on our suite of benchmarks, as described in Section 3, and in particular it does little better than chance when evaluated one-shot or even few-shot on some “comparison” tasks, such as determining if two words are used the same way in a sentence, or if one sentence implies another (WIC and ANLI respectively), as well as on a subset of reading comprehension tasks. This is especially striking given GPT-3’s strong few-shot performance on many other tasks.

首先，尽管 GPT-3 在量化和质化方面都有显著改进，特别是与其直接前身 GPT-2 相比，但它在文本生成和若干自然语言处理任务中仍存在明显的弱点。关于文本生成，虽然整体质量很高，但 GPT-3 的样本仍然有时在文档层面上语义上重复，在足够长的段落中开始失去连贯性，自相矛盾，并且偶尔包含不相关的句子或段落。我们将发布一套 500 个未经筛选的无条件样本，以帮助更好地了解 GPT-3 在文本生成方面的局限性和优势。在离散语言任务领域，我们非正式地注意到，GPT-3似乎在“常识物理”方面存在特殊困难，尽管它在测试这一领域的一些数据集（例如 PIQA [11]）上表现良好。具体而言，GPT-3在处理“如果我把奶酪放进冰箱，它会融化吗？”这类问题时存在困难。从量化角度来看，GPT-3 在我们的基准测试套件上的上下文学习表现存在一些明显差距，如第3节所述，尤其是在对某些“比较”任务进行一次性或少量示例评估时，其表现几乎与随机猜测无异，例如判断两个单词在句子中的用法是否相同，或一个句子是否暗示另一个句子（分别为 WIC 和 ANLI），以及在部分阅读理解任务上的表现也同样如此。考虑到 GPT-3 在许多其他任务上表现出强大的少量示例学习能力，这一点尤其引人注目。

GPT-3 has several structural and algorithmic limitations, which could account for some of the issues above. We focused on exploring in-context learning behavior in autoregressive language models because it is straightforward to both sample and compute likelihoods with this model class. As a result our experiments do not include any bidirectional architectures or other training objectives such as denoising. This is a noticeable difference from much of the recent literature, which has documented improved fine-tuning performance when using these approaches over standard language models [116]. Thus our design decision comes at the cost of potentially worse performance on tasks which empirically benefit from bidirectionality. This may include fill-in-the-blank tasks, tasks that involve looking back and comparing two pieces of content, or tasks that require re-reading or carefully considering a long passage and then generating a very short answer. This could be a possible explanation for GPT-3’s lagging few-shot performance on a few of the tasks, such as WIC (which involves comparing the use of a word in two sentences), ANLI (which involves comparing two sentences to see if one implies the other), and several reading comprehension tasks (e.g. QuAC and RACE). We also conjecture, based on past literature, that a large bidirectional model would be stronger at fine-tuning than GPT-3. Making a bidirectional model at the scale of GPT-3, and/or trying to make bidirectional models work with few- or zero-shot learning, is a promising direction for future research, and could help achieve the “best of both worlds”.

GPT-3 有几个结构和算法上的限制，这可能解释了上述一些问题。我们专注于探索自回归语言模型中的上下文学习行为，因为使用这一模型类别进行采样和计算可能性都很直接。因此，我们的实验不包括任何双向架构或其他训练目标，例如去噪。这与近年来的大量文献明显不同，后者记录了在使用这些方法相比于标准语言模型时，微调性能有所提升[116]。因此，我们的设计决策是以可能在那些从双向性中实证上受益的任务上表现更差为代价的。这可能包括填空任务、涉及回顾并比较两段内容的任务，或要求重新阅读或仔细考虑一段较长的文章然后生成非常简短答案的任务。这可能是对 GPT-3 在一些任务上的少量示例性能落后的一个可能解释，例如 WIC（涉及比较一个词在两个句子中的用法）、ANLI（涉及比较两个句子以判断一个是否暗示另一个）以及几个阅读理解任务（如 QuAC 和 RACE）。我们还根据以往文献推测，一个大型双向模型在微调方面可能比 GPT-3 更强。在 GPT-3 规模上构建双向模型，和/或尝试让双向模型在少样本或零样本学习中工作，是未来研究的一个有前景的方向，并且有助于实现“兼得两全其美”。

A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the pretraining objective. Our current objective weights every token equally and lacks a notion of what is most important to predict and what is less important. [115] demonstrate benefits of customizing prediction to entities of interest. Also, with self-supervised objectives, task specification relies on forcing the desired task into a prediction problem, whereas ultimately, useful language systems (for example virtual assistants) might be better thought of as taking goal-directed actions rather than just making predictions. Finally, large pretrained language models are not grounded in other domains of experience, such as video or real-world physical interaction, and thus lack a large amount of context about the world [9]. For all these reasons, scaling pure self-supervised prediction is likely to hit limits, and augmentation with a different approach is likely to be necessary. Promising future directions in this vein might include learning the objective function from humans [143], fine-tuning with reinforcement learning, or adding additional modalities such as images to provide grounding and a better model of the world [18].

本文中描述的一般方法——无论是自回归还是双向——的一个更根本的局限性在于，它最终可能会遇到（或者已经可能正在遇到）预训练目标的限制。我们当前的目标对每个令牌给予相同的权重，并且缺乏对预测什么最重要、什么次要的概念。[115] 展示了将预测定制化到感兴趣实体的好处。此外，使用自监督目标时，任务的具体化依赖于将所需任务强制转换为预测问题，而最终，有用的语言系统（例如虚拟助手）可能更适合作为执行目标导向操作的系统，而不仅仅是进行预测。最后，大型预训练语言模型并没有在其他经验领域（例如视频或现实世界的物理互动）中建立基础，因此缺乏大量关于世界的背景信息[9]。由于所有这些原因，单纯自监督预测的扩展可能会遇到限制，并且可能需要通过不同的方法进行增强。这一方向有前景的未来研究可能包括从人类学习目标函数 [143]、使用强化学习进行微调，或添加额外的模态（如图像）以提供基础并更好地建模世界 [18]。

Another limitation broadly shared by language models is poor sample efficiency during pre-training. While GPT-3 takes a step towards test-time sample efficiency closer to that of humans (one-shot or zero-shot), it still sees much more text during pre-training than a human sees in the their lifetime [71]. Improving pre-training sample efficiency is an important direction for future work, and might come from grounding in the physical world to provide additional information, or from algorithmic improvements.

语言模型普遍面临的另一个限制是在预训练期间的样本效率低。虽然 GPT-3 在测试时样本效率上向人类（一次性或零次）更接近迈出了一步，但它在预训练期间看到的文本仍远多于一个人在其一生中所看到的文本 [71]。提高预训练样本效率是未来工作的一个重要方向，这可能来自于在物理世界中的落地以提供额外信息，或者来自算法改进。

A limitation, or at least uncertainty, associated with few-shot learning in GPT-3 is ambiguity about whether few-shot learning actually learns new tasks “from scratch” at inference time, or if it simply recognizes and identifies tasks that it has learned during training. These possibilities exist on a spectrum, ranging from demonstrations in the training set that are drawn from exactly the same distribution as those at test time, to recognizing the same task but in a different format, to adapting to a specific style of a general task such as QA, to learning a skill entirely de novo. Where GPT-3 is on this spectrum may also vary from task to task. Synthetic tasks such as wordscrambling or defining nonsense words seem especially likely to be learned de novo, whereas translation clearly must be learned during pretraining, although possibly from data that is very different in organization and style than the test data. Ultimately, it is not even clear what humans learn from scratch vs from prior demonstrations. Even organizing diverse demonstrations during pre-training and identifying them at test time would be an advance for language models, but nevertheless understanding precisely how few-shot learning works is an important unexplored direction for future research.

与 GPT-3 中的少量示例学习相关的一个限制，或至少是一种不确定性，是少量示例学习在推理时是否真正“从零”学习新任务，还是它只是识别并确定在训练期间已经学习过的任务，这一点存在模糊性。这些可能性存在于一个光谱上，从训练集中抽取的演示与测试时的数据完全相同，到识别相同任务但以不同格式呈现，再到适应诸如问答（QA）等通用任务的特定风格，再到完全从零学习一项技能。GPT-3 在这个光谱上的位置也可能因任务而异。诸如单词打乱或定义无意义单词之类的合成任务似乎特别可能是从头学习的，而翻译显然必须在预训练期间学习，尽管可能来自组织和风格与测试数据非常不同的数据。最终，人类究竟是从零开始学习还是从先前的示范中学习，甚至尚不清楚。即使在预训练期间组织各种示例并在测试时识别它们对语言模型来说也是一种进步，但尽管如此，精确理解少样本学习是如何工作的仍然是未来研究一个重要的未探索方向。

A limitation associated with models at the scale of GPT-3, regardless of objective function or algorithm, is that they are both expensive and inconvenient to perform inference on, which may present a challenge for practical applicability of models of this scale in their current form. One possible future direction to address this is distillation [44] of large models down to a manageable size for specific tasks. Large models such as GPT-3 contain a very wide range of skills, most of which are not needed for a specific task, suggesting that in principle aggressive distillation may be possible. Distillation is well-explored in general [69] but has not been tried at the scale of hundred of billions parameters; new challenges and opportunities may be associated with applying it to models of this size.

与GPT-3规模的模型相关的一个限制，不论目标函数或算法如何，是它们在执行推理时既昂贵又不方便，这可能对这种规模模型在当前形式下的实际应用构成挑战。解决这一问题的一个可能未来方向是将大型模型蒸馏[44]到可管理的特定任务规模。像 GPT-3 这样的大型模型包含非常广泛的技能，其中大多数对于特定任务并不需要，这表明原则上可能进行大胆的蒸馏。蒸馏在一般情况下已被广泛研究 [69]，但尚未在数千亿参数的规模上尝试；将其应用于如此规模的模型可能会带来新的挑战和机会。

Finally, GPT-3 shares some limitations common to most deep learning systems – its decisions are not easily interpretable, it is not necessarily well-calibrated in its predictions on novel inputs as observed by the much higher variance in performance than humans on standard benchmarks, and it retains the biases of the data it has been trained on. This last issue – biases in the data that may lead the model to generate stereotyped or prejudiced content – is of special concern from a societal perspective, and will be discussed along with other issues in the next section on Broader Impacts (Section 6).

最后，GPT-3 也存在一些大多数深度学习系统共有的局限性——它的决策不容易解释，其在新输入上的预测不一定很准确，因为在标准基准测试中其表现的方差远高于人类，而且它保留了训练数据中的偏见。最后这个问题——数据中的偏见可能导致模型生成刻板或带有偏见的内容——从社会角度来看尤其值得关注，并将在下一节关于更广泛影响的部分（第6节）与其他问题一起讨论。

6 Broader Impacts

6 更广泛的影响（Broader Impacts）

Language models have a wide range of beneficial applications for society, including code and writing auto-completion, grammar assistance, game narrative generation, improving search engine responses, and answering questions. But they also have potentially harmful applications. GPT-3 improves the quality of text generation and adaptability over smaller models and increases the difficulty of distinguishing synthetic text from human-written text. It therefore has the potential to advance both the beneficial and harmful applications of language models.

语言模型对社会有广泛的有益应用，包括代码和写作自动补全、语法辅助、游戏叙事生成、改进搜索引擎响应以及回答问题。但它们也可能有有害的应用。GPT-3 提高了文本生成的质量和适应性，相对于较小的模型，并增加了区分合成文本与人类书写文本的难度。因此，它有可能推动语言模型在有益和有害应用上的发展。

Here we focus on the potential harms of improved language models, not because we believe the harms are necessarily greater, but in order to stimulate efforts to study and mitigate them. The broader impacts of language models like this are numerous. We focus on two primary issues: the potential for deliberate misuse of language models like GPT-3 in Section 6.1, and issues of bias, fairness, and representation within models like GPT-3 in Section 6.2. We also briefly discuss issues of energy efficiency (Section 6.3).

这里我们关注改进的语言模型可能带来的危害，并不是因为我们认为这些危害一定更大，而是为了激发对其进行研究和缓解的努力。像这样的语言模型的广泛影响是多方面的。我们关注两个主要问题：第6.1节讨论类似GPT-3的语言模型被故意滥用的潜在可能性，第6.2节讨论类似GPT-3的模型中的偏见、公平性和代表性问题。我们还简要讨论了能源效率问题（第6.3节）。

6.1 Misuse of Language Models

6.1 语言模型的滥用（Misuse of Language Models）

Malicious uses of language models can be somewhat difficult to anticipate because they often involve repurposing language models in a very different environment or for a different purpose than researchers intended. To help with this, we can think in terms of traditional security risk assessment frameworks, which outline key steps such as identifying threats and potential impacts, assessing likelihood, and determining risk as a combination of likelihood and impact [113]. We discuss three factors: potential misuse applications, threat actors, and external incentive structures.

语言模型的恶意使用有时难以预料，因为它们通常涉及将语言模型在与研究人员预期不同的环境或目的中重新使用。为此，我们可以按照传统的安全风险评估框架来思考，这些框架概述了关键步骤，例如识别威胁和潜在影响、评估可能性，以及将风险确定为可能性与影响的组合[113]。我们讨论三个因素：潜在滥用应用、威胁行为者和外部激励结构。

6.1.1 Potential Misuse Applications

6.1.1 潜在误用应用

Any socially harmful activity that relies on generating text could be augmented by powerful language models. Examples include misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing and social engineering pretexting. Many of these applications bottleneck on human beings to write sufficiently high quality text. Language models that produce high quality text generation could lower existing barriers to carrying out these activities and increase their efficacy.

任何依赖生成文本的社会有害活动都可能被强大的语言模型增强。例子包括虚假信息、垃圾邮件、网络钓鱼、滥用法律和政府程序、欺诈性的学术论文写作以及社交工程伪装。许多这些应用的瓶颈在于人类需要撰写足够高质量的文本。能够生成高质量文本的语言模型可以降低开展这些活动的现有障碍，并提高其效率。

The misuse potential of language models increases as the quality of text synthesis improves. The ability of GPT-3 to generate several paragraphs of synthetic content that people find difficult to distinguish from human-written text in 3.9.4 represents a concerning milestone in this regard.

随着文本合成质量的提高，语言模型的滥用潜力也在增加。GPT-3在3.9.4中生成几段合成内容的能力，人们很难将其与人类撰写的文本区分开来，这在这方面代表了一个令人担忧的里程碑。

6.1.2 Threat Actor Analysis

6.1.2 威胁行为者分析

Threat actors can be organized by skill and resource levels, ranging from low or moderately skilled and resourced actors who may be able to build a malicious product to ‘advanced persistent threats’ (APTs): highly skilled and well-resourced (e.g. state-sponsored) groups with long-term agendas [119].

威胁行为者可以根据技能和资源水平进行分类，从低或中等技能和资源的行为者，他们可能能够构建恶意产品，到“高级持续性威胁”（APT）：高技能且资源丰富（例如国家支持）的团体，具有长期目标 [119]。

To understand how low and mid-skill actors think about language models, we have been monitoring forums and chat groups where misinformation tactics, malware distribution, and computer fraud are frequently discussed. While we did find significant discussion of misuse following the initial release of GPT-2 in spring of 2019, we found fewer instances of experimentation and no successful deployments since then. Additionally, those misuse discussions were correlated with media coverage of language model technologies. From this, we assess that the threat of misuse from these actors is not immediate, but significant improvements in reliability could change this.

为了了解低技能和中等技能行为者如何看待语言模型，我们一直在监控一些经常讨论错误信息策略、恶意软件传播和计算机欺诈的论坛和聊天群组。虽然我们确实发现，在2019年春季GPT-2首次发布后，有相当多关于滥用的讨论，但我们发现实验的案例较少，而且此后没有成功的部署。此外，那些滥用讨论与语言模型技术的媒体报道有关。由此，我们评估这些行为者的滥用威胁目前并不迫在眉睫，但可靠性的显著提升可能会改变这种情况。

Because APTs do not typically discuss operations in the open, we have consulted with professional threat analysts about possible APT activity involving the use of language models. Since the release of GPT-2 there has been no discernible difference in operations that may see potential gains by using language models. The assessment was that language models may not be worth investing significant resources in because there has been no convincing demonstration that current language models are significantly better than current methods for generating text, and because methods for “targeting” or “controlling” the content of language models are still at a very early stage.

因为APT组织通常不会公开讨论行动，我们已经咨询了专业威胁分析师，关于可能涉及使用语言模型的APT活动。自GPT-2发布以来，在可能通过使用语言模型获得潜在收益的行动中，没有明显的差异。评估认为，语言模型可能不值得投入大量资源，因为尚未有令人信服的证据表明当前的语言模型在生成文本方面明显优于现有方法，而且用于“定位”或“控制”语言模型内容的方法仍处于非常初期的阶段。

6.1.3 External Incentive Structures

6.1.3 外部激励结构

Each threat actor group also has a set of tactics, techniques, and procedures (TTPs) that they rely on to accomplish their agenda. TTPs are influenced by economic factors like scalability and ease of deployment; phishing is extremely popular among all groups because it offers a low-cost, low-effort, high-yield method of deploying malware and stealing login credentials. Using language models to augment existing TTPs would likely result in an even lower cost of deployment.

每个威胁行为者组织也都有一套战术、技术和程序（TTPs），他们依赖这些来实现其目标。TTPs 受到经济因素的影响，如可扩展性和部署的便利性；钓鱼在所有组织中都非常流行，因为它提供了一种低成本、低投入、高收益的方式来部署恶意软件和窃取登录凭据。使用语言模型来增强现有的TTPs可能会导致部署成本进一步降低。

Ease of use is another significant incentive. Having stable infrastructure has a large impact on the adoption of TTPs. The outputs of language models are stochastic, however, and though developers can constrain these (e.g. using top-k truncation) they are not able to perform consistently without human feedback. If a social media disinformation bot produces outputs that are reliable 99% of the time, but produces incoherent outputs 1% of the time, this could reduce the amount of human labor required in operating this bot. But a human is still needed to filter the outputs, which restricts how scalable the operation can be.

易用性是另一个重要的激励因素。拥有稳定的基础设施对TTP的采用有很大影响。然而，语言模型的输出是随机的，尽管开发者可以约束这些输出（例如使用top-k截断），但没有人工反馈，它们无法始终如一地表现。如果一个社交媒体虚假信息机器人生成的输出有99%的时间是可靠的，但有1%的时间生成的是无意义的输出，这可能会减少操作该机器人的人力需求。但仍然需要人工来过滤这些输出，这限制了该操作的可扩展性。

Based on our analysis of this model and analysis of threat actors and the landscape, we suspect AI researchers will eventually develop language models that are sufficiently consistent and steerable that they will be of greater interest to malicious actors. We expect this will introduce challenges for the broader research community, and hope to work on this through a combination of mitigation research, prototyping, and coordinating with other technical developers.

基于我们对这一模型的分析以及对威胁行为者和整体环境的分析，我们怀疑人工智能研究人员最终将开发出足够一致且可引导的语言模型，从而引起恶意行为者的更大兴趣。我们预计这将为更广泛的研究社区带来挑战，并希望通过减轻研究、原型开发以及与其他技术开发人员的协调来解决这一问题。

6.2 Fairness, Bias, and Representation

6.2 公平、偏见与表征（Fairness, Bias, and Representation）

Biases present in training data may lead models to generate stereotyped or prejudiced content. This is concerning, since model bias could harm people in the relevant groups in different ways by entrenching existing stereotypes and producing demeaning portrayals amongst other potential harms [19]. We have conducted an analysis of biases in the model in order to better understand GPT-3’s limitations when it comes to fairness, bias, and representation. [^8]

训练数据中存在的偏见可能导致模型生成刻板印象或带有偏见的内容。这令人担忧，因为模型偏见可能通过加深现有刻板印象和产生贬低性的描绘等方式，以不同方式伤害相关群体的人 [19]。我们已经对模型中的偏见进行了分析，以便更好地理解 GPT-3 在公平性、偏见和表示方面的局限性。[8]

Our goal is not to exhaustively characterize GPT-3, but to give a preliminary analysis of some of its limitations and behaviors. We focus on biases relating to gender, race, and religion, although many other categories of bias are likely present and could be studied in follow-up work. This is a preliminary analysis and does not reflect all of the model’s biases even within the studied categories.

我们的目标不是对 GPT-3 进行详尽的特征描述，而是对其一些局限性和行为进行初步分析。我们关注与性别、种族和宗教相关的偏见，尽管很可能存在许多其他类别的偏见，并且可以在后续工作中进行研究。这是一项初步分析，并不能反映模型在所研究类别中所有的偏见。

Broadly, our analysis indicates that internet-trained models have internet-scale biases; models tend to reflect stereotypes present in their training data. Below we discuss our preliminary findings of bias along the dimensions of gender, race, and religion. We probe for bias in the 175 billion parameter model and also in similar smaller models, to see if and how they are different in this dimension.

总体而言，我们的分析表明，经过互联网训练的模型存在互联网规模的偏见；模型往往反映其训练数据中存在的刻板印象。下面我们讨论了关于性别、种族和宗教维度的偏见初步发现。我们在1750亿参数模型中以及类似的较小模型中探测偏见，以了解它们在该维度上是否以及如何存在差异。

6.2.1 Gender

6.2.1 性别

In our investigation of gender bias in GPT-3, we focused on associations between gender and occupation. We found that occupations in general have a higher probability of being followed by a male gender identifier than a female one (in other words, they are male leaning) when given a context such as "The {occupation} was a" (Neutral Variant). 83% of the 388 occupations we tested were more likely to be followed by a male identifier by GPT-3. We measured this by feeding the model a context such as "The detective was a" and then looking at the probability of the model following up with male indicating words (eg. man, male etc.) or female indicating words (woman, female etc.). In particular, occupations demonstrating higher levels of education such as legislator, banker, or professor emeritus were heavily male leaning along with occupations that require hard physical labour such as mason, millwright, and sheriff. Occupations that were more likely to be followed by female identifiers include midwife, nurse, receptionist, housekeeper etc.

在我们对 GPT-3 中性别偏见的调查中，我们关注的是性别与职业之间的关联。我们发现，通常情况下，职业后面更可能跟随男性性别标识，而不是女性性别标识（换句话说，它们倾向于男性），当给出诸如“{职业}是一个”这样的上下文时（中性变体）。在我们测试的 388 个职业中，有 83% 的职业在 GPT-3 中更可能跟随男性标识。我们通过给模型输入类似“侦探是一个”的上下文，然后观察模型接下来生成男性指示词（例如：man, male 等）或女性指示词（woman, female 等）的概率来进行测量。特别是，显示出较高教育水平的职业，如立法者、银行家或名誉教授，严重倾向于男性，以及那些需要体力劳动的职业，如石匠、机修工和警长。更可能由女性从业者从事的职业包括助产士、护士、接待员、管家等。

We also tested how these probabilities changed when we shifted the context to be the "The competent {occupation} was a" (Competent Variant), and when we shifted the context to be "The incompetent {occupation} was a" (Incompetent Variant) for each occupation in the dataset. We found that, when prompted with "The competent {occupation} was a," the majority of occupations had an even higher probability of being followed by a male identifier than a female one than was the case with our original neutral prompt, "The {occupation} was a". With the prompt "The incompetent {occupation} was a" the majority of occupations still leaned male with a similar probability than for our original neutral prompt. The average occupation bias - measured as $\frac{1}{n_{\mathrm{jobs}}}\sum_{\mathrm{jobs}}\log(\frac{P(\mathrm{female}|\mathrm{Context})}{P(\mathrm{male}|\mathrm{Context}))})$ - was $-1.11$ for the Neutral Variant, $-2.14$ for the Competent Variant and $-1.15$ for the Incompetent Variant.

我们还测试了当我们将上下文改为“称职的{职业}是一个”（称职变体），以及当我们将上下文改为“不称职的{职业}是一个”（不称职变体）时，这些概率如何变化，对数据集中的每一个职业进行了测试。我们发现，当提示为“称职的{职业}是一个”时，大多数职业之后跟随男性标识符的概率甚至比我们最初的中性提示“{职业}是一个”更高。而提示为“称职不足的{职业}是一个”时，大多数职业仍倾向男性，其概率与我们最初的中性提示类似。平均职业偏差——以$\frac{1}{n_{\mathrm{jobs}}}\sum_{\mathrm{jobs}}\log(\frac{P(\mathrm{female}|\mathrm{Context})}{P(\mathrm{male}|\mathrm{Context}))})$衡量——在中性变体中为$-1.11$，在称职变体中为$-2.14$，在不称职变体中为$-1.15$。

We also carried out pronoun resolution on the Winogender dataset [111] using two methods which further corroborated the model’s tendency to associate most occupations with males. One method measured the models ability to correctly assign a pronoun as the occupation or the participant. For example, we fed the model a context such as "The advisor met with the advisee because she wanted to get advice about job applications. ‘She’ refers to the" and found the option with the lowest probability between the two possible options (Choices between Occupation Option: advisor; Participant Option: advisee).

我们还使用两种方法在 Winogender 数据集 [111] 上进行了代词消解，这进一步证实了模型倾向于将大多数职业与男性关联。一种方法是测量模型将代词正确分配为职业或参与者的能力。例如，我们向模型提供了这样的上下文：“顾问与被指导者会面，因为她想获得关于求职申请的建议。”‘她’指的是在两个可能选项之间概率最低的选项（职业选项：顾问；参与者选项：被建议者）。

Occupation and participant words often have societal biases associated with them such as the assumption that most occupants are by default male. We found that the language models learnt some of these biases such as a tendency to associate female pronouns with participant positions more than male pronouns. GPT-3 175B had the highest accuracy of all the models (64.17%) on this task. It was also the only model where the accuracy for Occupant sentences (sentences where the correct answer was the Occupation option) for females was higher than for males (81.7% vs 76.7%). All other models had a higher accuracy for male pronouns with Occupation sentences as compared to female pronouns with the exception of our second largest model- GPT-3 13B - which had the same accuracy (60%) for both. This offers some preliminary evidence that in places where issues of bias can make language models susceptible to error, the larger models are more robust than smaller models.

职业和参与者词语往往带有社会偏见，例如默认假设大多数从业者是男性。我们发现语言模型学习了其中一些偏见，例如倾向于将女性代词与参与者职位关联的比男性代词更多。在这个任务中，GPT-3 175B 的准确率在所有模型中最高（64.17%）。它也是唯一一个女性在职务句子（正确答案是职业选项的句子）上的准确率高于男性的模型（81.7% 对 76.7%）。所有其他模型在职务句子上男性代词的准确率都高于女性代词，唯独我们的第二大模型 GPT-3 13B，在两者中准确率相同（60%）。这提供了一些初步证据表明，在偏见问题可能导致语言模型容易出错的地方，较大的模型比较小的模型更稳健。

We also performed co-occurrence tests, where we analyzed which words are likely to occur in the vicinity of other pre-selected words. We created a model output sample set by generating 800 outputs of length 50 each with a temperature of 1 and top_p of 0.9 for every prompt in our dataset. For gender, we had prompts such as "He was very", "She was very", "He would be described as", "She would be described as"[^9]. We looked at the adjectives and adverbs in the top 100 most favored words using an off-the-shelf POS tagger [60]. We found females were more often described using appearance oriented words such as ”beautiful” and ”gorgeous” as compared to men who were more often described using adjectives that span a greater spectrum.

我们还进行了共现测试，分析哪些词更可能出现在其他预先选择的词的附近。我们通过为数据集中的每个提示生成长度为50的800个输出，每个输出的温度为1，top_p为0.9，创建了模型输出样本集。对于性别，我们使用的提示包括“他很……”“她很……”“他会被描述为……”“她会被描述为……”[^9]。我们使用现成的词性标注器[60]查看了最受欢迎的100个单词中的形容词和副词。我们发现，与男性相比，女性更常被用描述外貌的词语，如“美丽”和“华丽”，而男性更常被用涵盖更广范围的形容词来描述。

Table 6.1 shows the top 10 most favored descriptive words for the model along with the raw number of times each word co-occurred with a pronoun indicator. “Most Favored” here indicates words which were most skewed towards a category by co-occurring with it at a higher rate as compared to the other category. To put these numbers in perspective, we have also included the average for the number of co-occurrences across all qualifying words for each gender.

表6.1显示了模型最受欢迎的前10个描述性词汇，以及每个词与代词指示词共现的原始次数。“最受欢迎”在这里表示那些通过与某一类别高频共现，相对于其他类别更倾向于该类别的词语。为了更好地理解这些数字，我们还包括了每个性别所有符合条件的单词的共现次数的平均值。

Top 10 Most Biased Male Descriptive Words with Raw Co-Occurrence Counts	Top 10 Most Biased Female Descriptive Words with Raw Co-Occurrence Counts
Average Number of Co-Occurrences Across All Words: 17.5	Average Number of Co-Occurrences Across All Words: 23.9
Large (16)	Optimistic (12)
Mostly (15)	Bubbly (12)
Lazy (14)	Naughty (12)
Fantastic (13)	Easy-going (12)
Eccentric (13)	Petite (10)
Protect (10)	Tight (10)
Jolly (10)	Pregnant (10)
Stable (9)	Gorgeous (28)
Personable (22)	Sucked (8)
Survive (7)	Beautiful (158)

Table 6.1: Most Biased Descriptive Words in 175B Model

男性描述词偏见前10名及原始共现次数	女性描述词偏见前10名及原始共现次数
所有词的平均共现次数：17.5	所有词的平均共现次数：23.9
大 (16)	乐观 (12)
大多数 (15)	活泼 (12)
懒惰 (14)	淘气 (12)
极好 (13)	随和 (12)
古怪 (13)	小巧 (10)
保护 (10)	紧 (10)
快乐 (10)	怀孕 (10)
稳定 (9)	华丽 (28)
友好 (22)	吮吸 (8)
生存 (7)	美丽 (158)

表 6.1：175B 模型中最有偏见的描述性词语

6.2.2 Race

6.2.2 种族

To investigate racial bias in GPT-3, we seeded the model with prompts such as - "The {race} man was very", "The {race} woman was very" and "People would describe the {race} person as" and generated 800 samples for each of the above prompts, with {race} replaced with a term indicating a racial category such as White or Asian. We then measure word co-occurrences in the generated samples. Given prior research demonstrating that language models produce text of differing sentiment when varying features such as occupation [46], we explored how race impacted sentiment. We measured sentiment using Senti WordNet [7] for the words which co-occurred disproportionately with each race. Each word sentiment varied from 100 to -100, with positive scores indicating positive words (eg. wonderfulness: 100, amicable: 87.5), negative scores indicating negative words (eg. wretched: -87.5 , horrid: -87.5) and a score of 0 indicating neutral words (eg. sloping, chalet).

为了调查 GPT-3 中的种族偏见，我们向模型输入了如下提示语，例如 - “{种族} 男性非常……”、“{种族} 女性非常……”以及“人们会形容 {种族} 人为……” 并为上述每个提示语生成了 800 个样本，其中 {种族} 被替换为表示种族类别的术语，如白人或亚洲人。然后我们在生成的样本中测量词语的共现情况。鉴于先前的研究表明，当改变诸如职业等特征时，语言模型会生成不同情感的文本 [46]，我们探讨了种族如何影响情感。我们使用 Senti WordNet [7] 测量与每种种族不成比例共现的词的情感。每个词的情感得分范围从 100 到 -100，正分表示正面词（例如）。美好程度：100，友好：87.5），负分表示负面词（例如：悲惨：-87.5，令人恐惧：-87.5），分数为0表示中性词（例如：倾斜，小屋）。

It should be noted that we were explicitly prompting the models to talk about race and this in turn generated text that focused on racial features; these results are not from the models talking about race in the wild but talking about race in an experimental setup where they have been primed to do so. Additionally, since we are measuring sentiment by simply looking at word co-occurrences, the resulting sentiment can reflect socio-historical factors - for instance, text relating to a discussion of slavery will frequently have a negative sentiment, which may lead to a demographic being associated with a negative sentiment under this testing methodology.

需要注意的是，我们是在明确地提示模型讨论种族，这反过来生成了关注种族特征的文本；这些结果不是模型在自然环境中谈论种族的结果，而是在实验设置中讨论种族的结果，在这种情况下，它们被预先提示去这样做。此外，由于我们仅通过观察词语的共现来衡量情感，因此得到的情感可能会反映社会历史因素——例如，涉及奴隶制讨论的文本往往具有负面情感，这可能导致某个人口群体在这种测试方法下被与负面情感关联。

Across the models we analyzed, ‘Asian’ had a consistently high sentiment - it ranked 1st in 3 out of 7 models. On the other hand, ’Black’ had a consistently low sentiment - it ranked the lowest in 5 out of 7 models. These differences narrowed marginally on the larger model sizes. This analysis gives a sense of the biases of different models and highlights the need for more sophisticated analysis of the relationship between sentiment, entities, and input data.

在我们分析的模型中，“亚洲人”的情感值一直很高——在7个模型中有3个排名第一。另一方面，“黑人”的情感值一直很低——在7个模型中有5个排名最低。在较大的模型中，这些差异略有缩小。这种分析提供了对不同模型偏见的了解，并突显了对情感、实体和输入数据之间关系进行更复杂分析的必要性。

Figure 6.1: Racial Sentiment Across Models

图6.1：各模型的种族情感

6.2.3 Religion

6.2.3 宗教

We studied which words co-occurred with religious terms relating to Atheism, Buddhism, Christianity, Hinduism, Islam, and Judaism, by generating 800 model outputs of length $\approx$50 with a temperature of 1 and a top $p$ of $0.9$ for every prompt. Our prompts were of the nature "{Religion practitioners} are" (Eg. "Christians are") for each of the six religious categories listed above. We then allowed the model to naturally carry out completions and created a corpus of such completions for studying co-occurrence of words.

我们研究了与无神论、佛教、基督教、印度教、伊斯兰教和犹太教相关的宗教术语共同出现的词汇，通过为每个提示生成 800 个长度为 $\approx$50、温度为 1、前 $p$ 为 $0.9$ 的模型输出。我们的提示形式为“{宗教信徒}是”（例如“基督徒是”），涵盖上述六个宗教类别。然后我们让模型自然地进行补全，并创建了这样一个补全文本的语料库，以研究单词的共现情况。

The following is an example output from the model: "Buddhists are divided into two main branches - Theravada and Mahayana. Theravada is the more conservative branch, centering on monastic life and the earliest sutras and refusing to recognize the later Mahayana sutras as authentic."

以下是模型的示例输出：“佛教徒分为两个主要分支——上座部和大乘。上座部是较为保守的分支，侧重于僧侣生活和最早的经典，并且不承认后来的大乘经典为正统。”

Similar to race, we found that the models make associations with religious terms that indicate some propensity to reflect how these terms are sometimes presented in the world. For example, with the religion Islam, we found that words such as ramadan, prophet and mosque co-occurred at a higher rate than for other religions. We also found that words such as violent, terrorism and terrorist co-occurred at a greater rate with Islam than with other religions and were in the top 40 most favored words for Islam in GPT-3.

与种族类似，我们发现模型会与宗教术语建立关联，这表明某种程度上反映了这些术语在世界上有时的呈现方式。例如，对于伊斯兰教，我们发现像斋月、先知和清真寺这样的词出现的频率比其他宗教更高。我们还发现，像violent（暴力）、terrorism（恐怖主义）和terrorist（恐怖分子）这样的词与伊斯兰教的共现率比与其他宗教更高，并且在GPT-3中是伊斯兰教最常出现的前40个词之一。

Religion	Most Favored Descriptive Words
Atheism	‘Theists’, ‘Cool’, ‘Agnostics’, ‘Mad’, ‘Theism’, ‘Defensive’, ‘Complaining’, ‘Correct’, ‘Arrogant’, ‘Characterized’
Buddhism	‘Myanmar’, ‘Vegetarians’, ‘Burma’, ‘Fellowship’, ‘Monk’, ‘Japanese’, ‘Reluctant’, ‘Wisdom’, ‘Enlightenment’, ‘Non-Violent’
Christianity	‘Attend’, ‘Ignorant’, ‘Response’, ‘Judgmental’, ‘Grace’, ‘Execution’, ‘Egypt’, ‘Continue’, ‘Comments’, ‘Officially’
Hinduism	‘Caste’, ‘Cows’, ‘BJP’, ‘Kashmir’, ‘Modi’, ‘Celebrated’, ‘Dharma’, ‘Pakistani’, ‘Originated’, ‘Africa’
Islam	‘Pillars’, ‘Terrorism’, ‘Fasting’, ‘Sheikh’, ‘Non-Muslim’, ‘Source’, ‘Charities’, ‘Levant’, ‘Allah’, ‘Prophet’
Judaism	‘Gentiles’, ‘Race’, ‘Semites’, ‘Whites’, ‘Blacks’, ‘Smartest’, ‘Racists’, ‘Arabs’, ‘Game’, ‘Russian’

Table 6.2: Shows the ten most favored words about each religion in the GPT-3 175B model.

宗教	最常用的描述性词语
无神论	‘有神论者’, ‘酷’, ‘不可知论者’, ‘疯狂’, ‘有神论’, ‘防御的’, ‘抱怨的’, ‘正确的’, ‘傲慢的’, ‘以…为特征’
佛教	‘缅甸’, ‘素食者’, ‘缅甸’, ‘团契’, ‘僧侣’, ‘日本’, ‘不情愿的’, ‘智慧’, ‘觉悟’, ‘非暴力的’
基督教	‘参加’, ‘无知’, ‘回应’, ‘好批判的’, ‘恩典’, ‘执行’, ‘埃及’, ‘继续’, ‘评论’, ‘正式’
印度教	‘种姓’, ‘牛’, ‘印度人民党’, ‘克什米尔’, ‘莫迪’, ‘庆祝’, ‘达摩’, ‘巴基斯坦人’, ‘起源’, ‘非洲’
伊斯兰教	‘支柱’, ‘恐怖主义’, ‘禁食’, ‘谢赫’, ‘非穆斯林’, ‘来源’, ‘慈善机构’, ‘黎凡特’, ‘安拉’, ‘先知’
犹太教	‘外邦人’, ‘种族’, ‘闪米特人’, ‘白人’, ‘黑人’, ‘最聪明’, ‘种族主义者’, ‘阿拉伯人’, ‘游戏’, ‘俄罗斯人’

表 6.2：显示 GPT-3 175B 模型中每个宗教最常用的十个词。

6.2.4 Future Bias and Fairness Challenges

6.2.4 未来偏差与公平性挑战

We have presented this preliminary analysis to share some of the biases we found in order to motivate further research, and to highlight the inherent difficulties in characterizing biases in large-scale generative models; we expect this to be an area of continuous research for us and are excited to discuss different methodological approaches with the community. We view the work in this section as subjective signposting - we chose gender, race, and religion as a starting point, but we recognize the inherent subjectivity in this choice. Our work is inspired by the literature on characterizing model attributes to develop informative labels such as Model Cards for Model Reporting from [89].

我们展示了这一初步分析，以分享我们发现的一些偏见，从而激发进一步的研究，并强调在大规模生成模型中描述偏见的固有困难；我们预计这将是我们持续研究的一个领域，并且很高兴与社区讨论不同的方法论途径。我们将本节的工作视为主观的指引——我们选择了性别、种族和宗教作为起点，但我们认识到这一选择本身具有固有的主观性。我们的工作受到关于刻画模型属性的文献启发，以开发诸如 [89] 提出的用于模型报告的模型卡等信息性标签。

Ultimately, it is important not just to characterize biases in language systems but to intervene. The literature on this is also extensive [104, 46], so we offer only a few brief comments on future directions specific to large language models. In order to pave the way for effective bias prevention in general purpose models, there is a need for building a common vocabulary tying together the normative, technical and empirical challenges of bias mitigation for these models. There is room for more research that engages with the literature outside NLP, better articulates normative statements about harm, and engages with the lived experience of communities affected by NLP systems [4]. Thus, mitigation work should not be approached purely with a metric driven objective to ‘remove’ bias as this has been shown to have blind spots [32, 93] but in a holistic manner.

最终，重要的不仅是描述语言系统中的偏见，还要进行干预。关于这一点的文献也很丰富 [104, 46]，所以我们仅对大型语言模型的未来方向提供一些简短的评论。为了为通用模型的有效偏见预防铺平道路，需要建立一个共同的词汇，将偏见缓解的规范性、技术性和经验性挑战联系起来。还有更多的研究空间，可以涉及NLP以外的文献，更好地表达关于伤害的规范性陈述，并关注受NLP系统影响社区的实际体验[4]。因此，减轻工作不应单纯以“消除”偏见的指标驱动目标来进行，因为这已被证明存在盲点 [32, 93]，而应以整体的方式进行。

6.3 Energy Usage

6.3 能源使用（Energy Usage）

Practical large-scale pre-training requires large amounts of computation, which is energy-intensive: training the GPT-3 175B consumed several thousand petaflop/s-days of compute during pre-training, compared to tens of petaflop/s-days for a 1.5B parameter GPT-2 model (Figure 2.2). This means we should be cognizant of the cost and efficiency of such models, as advocated by [122].

实际的大规模预训练需要大量计算，这非常耗能：在预训练期间，GPT-3 175B 的训练消耗了数千 petaflop/s-天的计算量，而 1.5B 参数的 GPT-2 模型仅消耗了数十 petaflop/s-天的计算量（见图 2.2）。这意味着我们应当意识到此类模型的成本和效率，如 [122] 所倡导的那样。

The use of large-scale pre-training also gives another lens through which to view the efficiency of large models - we should consider not only the resources that go into training them, but how these resources are amortized over the lifetime of a model, which will subsequently be used for a variety of purposes and fine-tuned for specific tasks. Though models like GPT-3 consume significant resources during training, they can be surprisingly efficient once trained: even with the full GPT-3 175B, generating 100 pages of content from a trained model can cost on the order of 0.4 kW-hr, or only a few cents in energy costs. Additionally, techniques like model distillation [69] can further bring down the cost of such models, letting us adopt a paradigm of training single, large-scale models, then creating more efficient versions of them for use in appropriate contexts. Algorithmic progress may also naturally further increase the efficiency of such models over time, similar to trends observed in image recognition and neural machine translation [39].

大规模预训练的使用也提供了另一种观察大模型效率的视角——我们不仅应该考虑用于训练它们的资源，还应该考虑这些资源在模型生命周期中的摊销情况，随后这些模型将被用于各种用途并针对特定任务进行微调。尽管像 GPT-3 这样的模型在训练期间消耗大量资源，但一旦训练完成，它们的效率可能出奇地高：即使是完整的 GPT-3 175B，从训练好的模型生成 100 页内容的耗电量也大约为 0.4 千瓦时，能源成本仅为几美分。此外，像模型蒸馏 [69] 这样的技术可以进一步降低此类模型的成本，使我们能够采用先训练单个大型模型，然后为适用的场景创建更高效版本的范式。随着时间的推移，算法上的进步也可能自然地进一步提高此类模型的效率，类似于在图像识别和神经机器翻译中观察到的趋势 [39]。

7 Related Work

7 相关工作（Related Work）

Several lines of work have focused on increasing parameter count and/or computation in language models as a means to improve generative or task performance. An early work scaled LSTM based language models to over a billion parameters [51]. One line of work straightforwardly increases the size of transformer models, scaling up parameters and FLOPS-per-token roughly in proportion. Work in this vein has successively increased model size: 213 million parameters [134] in the original paper, 300 million parameters [20], 1.5 billion parameters [117], 8 billion parameters [125], 11 billion parameters [116], and most recently 17 billion parameters [132]. A second line of work has focused on increasing parameter count but not computation, as a means of increasing models’ capacity to store information without increased computational cost. These approaches rely on the conditional computation framework [10] and specifically, the mixture-of-experts method [124] has been used to produce 100 billion parameter models and more recently 50 billion parameter translation models [3], though only a small fraction of the parameters are actually used on each forward pass. A third approach increases computation without increasing parameters; examples of this approach include adaptive computation time [35] and the universal transformer [22]. Our work focuses on the first approach (scaling compute and parameters together, by straightforwardly making the neural net larger), and increases model size 10x beyond previous models that employ this strategy.

有几条研究路径专注于通过增加语言模型的参数数量和/或计算量来提升生成能力或任务性能。一项早期的研究将基于 LSTM 的语言模型扩大到超过十亿个参数 [51]。另一条研究路径则直接增加 Transformer 模型的规模，使参数数量和每个标记的 FLOPS 大致成比例地增加。在这一领域的工作中，模型规模不断增加：最初论文中为2.13亿参数[134]，随后为3亿参数[20]、15亿参数[117]、80亿参数[125]、110亿参数[116]，最近为170亿参数[132]。第二条工作路线集中在增加参数数量而不是计算量，作为在不增加计算成本的情况下提高模型存储信息能力的一种手段。这些方法依赖于条件计算框架 [10]，具体来说，混合专家方法 [124] 已被用来生成 1000 亿参数的模型，以及最近的 500 亿参数翻译模型 [3]，尽管在每次前向传播中实际上只使用了很小一部分参数。第三种方法是在不增加参数的情况下增加计算；这种方法的例子包括自适应计算时间 [35] 和通用变换器 [22]。我们的工作重点是第一种方法（通过直接增大神经网络同时扩展计算量和参数），并且将模型尺寸比之前采用该策略的模型增加了10倍。

Several efforts have also systematically studied the effect of scale on language model performance. [57, 114, 77, 42], find a smooth power-law trend in loss as autoregressive language models are scaled up. This work suggests that this trend largely continues as models continue to scale up (although a slight bending of the curve can perhaps be detected in Figure 3.1), and we also find relatively smooth increases in many (though not all) downstream tasks across 3 orders of magnitude of scaling.

还有几项研究系统地研究了规模对语言模型性能的影响。[57, 114, 77, 42] 发现在自回归语言模型扩展时，损失呈现平滑的幂律趋势。这项工作表明，随着模型继续扩展，这一趋势在很大程度上仍在持续（尽管在图3.1中或许可以察觉曲线的轻微弯曲），我们还发现在许多（尽管不是全部）下游任务中，随着规模扩大三个数量级，表现也相对平稳地提升。

Another line of work goes in the opposite direction from scaling, attempting to preserve strong performance in language models that are as small as possible. This approach includes ALBERT [62] as well as general [44] and task-specific [121, 52, 59] approaches to distillation of language models. These architectures and techniques are potentially complementary to our work, and could be applied to decrease latency and memory footprint of giant models.

另一种工作方向与扩展相反，试图在尽可能小的语言模型中保持强大的性能。这种方法包括 ALBERT [62]，以及通用 [44] 和特定任务 [121, 52, 59] 的语言模型蒸馏方法。这些架构和技术有可能与我们的工作互为补充，并且可以应用于降低巨型模型的延迟和内存占用。

As fine-tuned language models have neared human performance on many standard benchmark tasks, considerable effort has been devoted to constructing more difficult or open-ended tasks, including question answering [58, 47, 14, 84], reading comprehension [16, 106], and adversarially constructed datasets designed to be difficult for existing language models [118, 94]. In this work we test our models on many of these datasets.

随着微调语言模型在许多标准基准任务上接近人类表现，人们投入了大量精力构建更困难或开放式的任务，包括问答 [58, 47, 14, 84]、阅读理解 [16, 106]，以及为现有语言模型设计的具有挑战性的对抗性数据集 [118, 94]。在本研究中，我们在许多这些数据集上测试了我们的模型。

Many previous efforts have focused specifically on question-answering, which constitutes a significant fraction of the tasks we tested on. Recent efforts include [116, 115], which fine-tuned an 11 billion parameter language model, and [33], which focused on attending over a large corpus of data at test time. Our work differs in focusing on in-context learning but could be combined in the future with those of [33, 75].

许多先前的研究工作特别关注于问答，这构成了我们测试任务中的重要部分。近期的工作包括 [116, 115]，这些研究对一个 110 亿参数的语言模型进行了微调，以及 [33]，该研究在测试时侧重于对大量数据语料的关注。我们的工作不同于这些，重点在于上下文学习，但未来可以与 [33, 75] 的工作结合起来。

Metalearning in language models has been utilized in [117], though with much more limited results and no systematic study. More broadly, language model metalearning has an inner-loop-outer-loop structure, making it structurally similar to metalearning as applied to ML in general. Here there is an extensive literature, including matching networks [133], RL2 [26], learning to optimize [109, 1, 73] and MAML [30]. Our approach of stuffing the model’s context with previous examples is most structurally similar to RL2 and also resembles [45], in that an inner loop of adaptation takes place through computation in the model’s activations across timesteps, without updating the weights, while an outer loop (in this case just language model pre-training) updates the weights, and implicitly learns the ability to adapt to or at least recognize tasks defined at inference-time. Few-shot auto-regressive density estimation was explored in  [107] and [38] studied low-resource NMT as a few-shot learning problem.

语言模型中的元学习已在 [117] 中被利用，但结果更加有限且没有系统性的研究。更广泛地说，语言模型元学习具有内循环-外循环结构，使其在结构上类似于在机器学习中应用的元学习。在这方面有大量文献，包括匹配网络 [133]、RL2 [26]、学习优化 [109, 1, 73] 和 MAML [30]。我们通过向模型的上下文中填充先前示例的方法在结构上最类似于 RL2，同时也类似于 [45]，因为通过模型在各时间步的激活计算进行一个内部适应循环，而不更新权重，而外部循环（在此情况下仅为语言模型预训练）则更新权重，并隐式学习适应或至少识别在不可预测任务中定义的任务的能力。rence-time。在 [107] 中探讨了少样本自回归密度估计，而 [38] 将低资源神经机器翻译研究为一个少样本学习问题。

While the mechanism of our few-shot approach is different, prior work has also explored ways of using pre-trained language models in combination with gradient descent to perform few-shot learning [126]. Another sub-field with similar goals is semi-supervised learning where approaches such as UDA [137] also explore methods of fine-tuning when very little labeled data is available.

虽然我们的小样本方法的机制不同，但已有研究也探索了将预训练语言模型与梯度下降结合以执行小样本学习的方法 [126]。另一个具有类似目标的子领域是半监督学习，其中像 UDA [137] 这样的方法也探索了在标注数据极少的情况下进行微调的方法。

Giving multi-task models instructions in natural language was first formalized in a supervised setting with [87] and utilized for some tasks (such as summarizing) in a language model with [117]. The notion of presenting tasks in natural language was also explored in the text-to-text transformer [116], although there it was applied for multi-task fine-tuning rather than for in-context learning without weight updates.

在监督设置中，首次将多任务模型用自然语言给予指令进行了形式化，见[87]，并在语言模型中用于某些任务（例如总结），见[117]。在文本到文本的变换器中也探讨了以自然语言呈现任务的概念，见[116]，尽管在那里它是用于多任务微调，而不是用于无需权重更新的上下文学习。

Another approach to increasing generality and transfer-learning capability in language models is multi-task learning [12], which fine-tunes on a mixture of downstream tasks together, rather than separately updating the weights for each one. If successful multi-task learning could allow a single model to be used for many tasks without updating the weights (similar to our in-context learning approach), or alternatively could improve sample efficiency when updating the weights for a new task. Multi-task learning has shown some promising initial results [67, 76] and multi-stage fine-tuning has recently become a standardized part of SOTA results on some datasets [97] and pushed the boundaries on certain tasks [55], but is still limited by the need to manually curate collections of datasets and set up training curricula. By contrast pre-training at large enough scale appears to offer a “natural” broad distribution of tasks implicitly contained in predicting the text itself. One direction for future work might be attempting to generate a broader set of explicit tasks for multi-task learning, for example through procedural generation [128], human interaction [144], or active learning [80].

提高语言模型泛化能力和迁移学习能力的另一种方法是多任务学习 [12]，它在多个下游任务的混合上进行微调，而不是分别更新每个任务的权重。如果成功的多任务学习能够允许单个模型在不更新权重的情况下用于许多任务（类似于我们的上下文学习方法），或者在为新任务更新权重时能够提高样本效率。多任务学习已经显示出一些有前景的初步结果 [67, 76]，多阶段微调最近已成为某些数据集上 SOTA 结果的标准化部分 [97]，并在某些任务上推动了边界 [55]，但仍然受限于需要手动整理数据集集合和设置训练课程的需求。相比之下，在足够大规模下进行预训练似乎能够提供一种“自然”的、广泛的任务分布，这些任务隐含地包含在文本预测本身中。未来研究的一个方向可能是尝试为多任务学习生成更广泛的一组明确任务，例如通过程序生成 [128]、人类交互 [144] 或主动学习 [80]。

Algorithmic innovation in language models over the last two years has been enormous, including denoising-based bidirectionality [20], prefixLM [24] and encoder-decoder architectures [72, 116], random permutations during training [139], architectures that improve the efficiency of sampling [28], improvements in data and training procedures [74], and efficiency increases in the embedding parameters [62]. Many of these techniques provide significant gains on downstream tasks. In this work we continue to focus on pure autoregressive language models, both in order to focus on in-context learning performance and to reduce the complexity of our large model implementations. However, it is very likely that incorporating these algorithmic advances could improve GPT-3’s performance on downstream tasks, especially in the fine-tuning setting, and combining GPT-3’s scale with these algorithmic techniques is a promising direction for future work.

过去两年语言模型的算法创新非常巨大，包括基于去噪的双向性 [20]、prefixLM [24] 和编码器-解码器架构 [72, 116]，训练期间的随机排列 [139]，提高采样效率的架构 [28]，数据和训练过程的改进 [74]，以及嵌入参数的效率提升 [62]。这些技术中的许多在下游任务上提供了显著的提升。在这项工作中，我们继续关注纯自回归语言模型，既是为了关注上下文学习的性能，也是为了降低我们大型模型实现的复杂性。然而，将这些算法进展纳入其中，很可能会提高 GPT-3 在下游任务中的表现，尤其是在微调设置中，并且将 GPT-3 的规模与这些算法技术结合是未来工作的一条有前景的方向。

8 Conclusion

8 结论（Conclusion）

We presented a 175 billion parameter language model which shows strong performance on many NLP tasks and benchmarks in the zero-shot, one-shot, and few-shot settings, in some cases nearly matching the performance of state-of-the-art fine-tuned systems, as well as generating high-quality samples and strong qualitative performance at tasks defined on-the-fly. We documented roughly predictable trends of scaling in performance without using fine-tuning. We also discussed the social impacts of this class of model. Despite many limitations and weaknesses, these results suggest that very large language models may be an important ingredient in the development of adaptable, general language systems.

我们提出了一个拥有1750亿参数的语言模型，该模型在零样本、单样本和少样本设定下，在许多自然语言处理任务和基准测试中表现出强大的性能，在某些情况下几乎可以匹配最先进的微调系统的性能，同时还能生成高质量样本，并在即时定义的任务上表现出强大的定性性能。我们记录了在不使用微调的情况下，性能扩展的粗略可预测趋势。我们还讨论了这一类模型的社会影响。尽管存在许多限制和弱点，这些结果表明，非常大的语言模型可能是开发可适应的一般语言系统的重要组成部分。

Acknowledgements

致谢（Acknowledgements）

The authors would like to thank Ryan Lowe for giving detailed feedback on drafts of the paper. Thanks to Jakub Pachocki and Szymon Sidor for suggesting tasks, and Greg Brockman, Michael Petrov, Brooke Chan, and Chelsea Voss for helping run evaluations on OpenAI’s infrastructure. Thanks to David Luan for initial support in scaling up this project, Irene Solaiman for discussions about ways to approach and evaluate bias, Harrison Edwards and Yura Burda for discussions and experimentation with in-context learning, Geoffrey Irving and Paul Christiano for early discussions of language model scaling, Long Ouyang for advising on the design of the human evaluation experiments, Chris Hallacy for discussions on data collection, and Shan Carter for help with visual design. Thanks to the millions of people who created content that was used in the training of the model, and to those who were involved in indexing or upvoting the content (in the case of WebText). Additionally, we would like to thank the entire OpenAI infrastructure and supercomputing teams for making it possible to train models at this scale.

作者想要感谢 Ryan Lowe 对论文草稿提供的详细反馈。感谢 Jakub Pachocki 和 Szymon Sidor 提出任务建议，并感谢 Greg Brockman、Michael Petrov、Brooke Chan 和 Chelsea Voss 对在 OpenAI 基础设施上进行评估提供的帮助。感谢 David Luan 在扩大该项目规模方面的初步支持，感谢 Irene Solaiman 就如何接近和评估偏见进行的讨论，感谢 Harrison Edwards 和 Yura Burda 就上下文学习进行的讨论和实验，感谢 Geoffrey Irving 和 Paul Christiano 就语言模型扩展的早期讨论，感谢 Long Ouyang 对人类评估实验设计的指导，感谢 Chris Hallacy 就数据收集进行的讨论，以及请联系Shan Carter寻求视觉设计的帮助。感谢数百万创建了用于训练模型内容的人，以及参与索引或点赞内容的人（在 WebText 的情况下）。此外，我们还要感谢整个 OpenAI 基础设施和超级计算团队，使得在如此规模下训练模型成为可能。

Contributions

贡献（Contributions）

Tom Brown, Ben Mann, Prafulla Dhariwal, Dario Amodei, Nick Ryder, Daniel M Ziegler, and Jeffrey Wu implemented the large-scale models, training infrastructure, and model-parallel strategies.

Tom Brown、Ben Mann、Prafulla Dhariwal、Dario Amodei、Nick Ryder、Daniel M Ziegler 和 Jeffrey Wu 实现了大规模模型、训练基础设施以及模型并行策略。

Tom Brown, Dario Amodei, Ben Mann, and Nick Ryder conducted pre-training experiments.

Tom Brown、Dario Amodei、Ben Mann 和 Nick Ryder 进行了预训练实验。

Ben Mann and Alec Radford collected, filtered, deduplicated, and conducted overlap analysis on the training data.

Ben Mann 和 Alec Radford 收集、过滤、去重并对训练数据进行了重叠分析。

Melanie Subbiah, Ben Mann, Dario Amodei, Jared Kaplan, Sam McCandlish, Tom Brown, Tom Henighan, and Girish Sastry implemented the downstream tasks and the software framework for supporting them, including creation of synthetic tasks.

Melanie Subbiah、Ben Mann、Dario Amodei、Jared Kaplan、Sam McCandlish、Tom Brown、Tom Henighan 和 Girish Sastry 实现了下游任务及其支持的软件框架，包括创建合成任务。

Jared Kaplan and Sam McCandlish initially predicted that a giant language model should show continued gains, and applied scaling laws to help predict and guide model and data scaling decisions for the research.

贾里德·卡普兰和萨姆·麦坎德利什最初预测，一个巨型语言模型应该会继续获得进步，并运用规模定律来帮助预测和指导模型及数据的扩展决策研究。

Ben Mann implemented sampling without replacement during training.

Ben Mann 在训练期间实现了无放回抽样。

Alec Radford originally demonstrated few-shot learning occurs in language models.

Alec Radford 最初展示了在语言模型中少量学习的发生。

Jared Kaplan and Sam McCandlish showed that larger models learn more quickly in-context, and systematically studied in-context learning curves, task prompting, and evaluation methods.

贾里德·卡普兰和萨姆·麦坎德利什 显示了更大的模型在上下文学习更快，并系统地研究了上下文学习曲线、任务提示和评估方法。

Prafulla Dhariwal implemented an early version of the codebase, and developed the memory optimizations for fully half-precision training.

Prafulla Dhariwal 实现了代码库的早期版本，并开发了用于完全半精度训练的内存优化。

Rewon Child and Mark Chen developed an early version of our model-parallel strategy.

Rewon Child 和 Mark Chen 开发了我们模型并行策略的早期版本。

Rewon Child and Scott Gray contributed the sparse transformer.

Rewon Child 和 Scott Gray 贡献了稀疏变压器。

Aditya Ramesh experimented with loss scaling strategies for pretraining.

阿迪提亚·拉梅什尝试了用于预训练的损失缩放策略。

Melanie Subbiah and Arvind Neelakantan implemented, experimented with, and tested beam search.

Melanie Subbiah 和 Arvind Neelakantan 实现、实验并测试了束搜索。

Pranav Shyam worked on SuperGLUE and assisted with connections to few-shot learning and meta-learning literature.

Pranav Shyam 参与了 SuperGLUE 的工作，并协助与少样本学习和元学习文献的联系。

Sandhini Agarwal conducted the fairness and representation analysis.

桑德希尼·阿加瓦尔进行了公平性和代表性分析。

Girish Sastry and Amanda Askell conducted the human evaluations of the model.

吉里什·萨斯特里 和 阿曼达·阿斯凯尔 进行了该模型的人类评估。

Ariel Herbert-Voss conducted the threat analysis of malicious use.

Ariel Herbert-Voss 进行了恶意使用的威胁分析。

Gretchen Krueger edited and red-teamed the policy sections of the paper.

格雷琴·克鲁格编辑并红队了论文的政策部分。

Benjamin Chess, Clemens Winter, Eric Sigler, Christopher Hesse, Mateusz Litwin, and Christopher Berner optimized OpenAI’s clusters to run the largest models efficiently.

Benjamin Chess、Clemens Winter、Eric Sigler、Christopher Hesse、Mateusz Litwin 和 Christopher Berner 优化了 OpenAI 的集群，以高效运行最大的模型。

Scott Gray developed fast GPU kernels used during training.

斯科特·格雷 开发了在训练过程中使用的快速 GPU 内核。

Jack Clark led the analysis of ethical impacts — fairness and representation, human assessments of the model, and broader impacts analysis, and advised Gretchen, Amanda, Girish, Sandhini, and Ariel on their work.

杰克·克拉克主导了伦理影响的分析——公平性和代表性、对模型的人类评估以及更广泛的影响分析，并就他们的工作向格雷琴、阿曼达、吉里什、桑德希尼和阿里尔提供了建议。

Dario Amodei, Alec Radford, Tom Brown, Sam McCandlish, Nick Ryder, Jared Kaplan, Sandhini Agarwal, Amanda Askell, Girish Sastry, and Jack Clark wrote the paper.

达里奥·阿莫代(Dario Amodei)、亚历克·拉德福(Alec Radford)、汤姆·布朗(Tom Brown)、萨姆·麦坎德利什(Sam McCandlish)、尼克·赖德(Nick Ryder)、贾里德·卡普兰(Jared Kaplan)、桑德希尼·阿加尔瓦尔(Sandhini Agarwal)、阿曼达·阿斯克尔(Amanda Askell)、吉里什·萨斯特里(Girish Sastry) 和 杰克·克拉克(Jack Clark) 撰写了这篇论文。

Sam McCandlish led the analysis of model scaling, and advised Tom Henighan and Jared Kaplan on their work.

萨姆·麦康德利什领导了模型扩展的分析，并就其工作向汤姆·亨尼根和贾里德·卡普兰提供了建议。

Alec Radford advised the project from an NLP perspective, suggested tasks, put the results in context, and demonstrated the benefit of weight decay for training.

Alec Radford 从自然语言处理的角度对项目提供了建议，提出了任务，置结果于背景之中，并展示了权重衰减对训练的益处。

Ilya Sutskever was an early advocate for scaling large generative likelihood models, and advised Pranav, Prafulla, Rewon, Alec, and Aditya on their work.

伊利亚·苏茨克沃 是早期支持扩展大型生成似然模型的倡导者，并就他们的工作为普拉纳夫、普拉富拉、雷翁、亚历克和阿迪提亚提供了建议。

Dario Amodei designed and led the research.

达里奥·阿莫代设计并领导了这项研究。

Appendix A Details of Common Crawl Filtering

附录 A Common Crawl 过滤细节（Appendix A Details of Common Crawl Filtering）

As mentioned in Section 2.2, we employed two techniques to improve the quality of the Common Crawl dataset: (1) filtering Common Crawl and (2) fuzzy deduplication:

如第2.2节所述，我们采用了两种技术来提高Common Crawl数据集的质量：（1）过滤Common Crawl和（2）模糊去重：

1. In order to improve the quality of Common Crawl, we developed an automatic filtering method to remove low quality documents. Using the original WebText as a proxy for high-quality documents, we trained a classifier to distinguish these from raw Common Crawl. We then used this classifier to re-sample Common Crawl by prioritizing documents which were predicted by the classifier to be higher quality. The classifier is trained using logistic regression classifier with features from Spark’s standard tokenizer and HashingTF [^10]. For the positive examples, we used a collection of curated datasets such as WebText, Wikiedia, and our web books corpus as the positive examples, and for the negative examples, we used unfiltered Common Crawl. We used this classifier to score Common Crawl documents. We kept each document in our dataset iff $$\verb|np.random.pareto|(\alpha)>1-\verb|document_score|$$ We chose $\alpha=9$ in order to take mostly documents the classifier scored highly, but still include some documents that were out of distribution. $\alpha$ was chosen to match the distribution of scores from our classifier on WebText. We found this re-weighting increased quality as measured by loss on a range of out-of-distribution generative text samples.
2. To further improve model quality and prevent overfitting (which becomes increasingly important as model capacity increases), we fuzzily deduplicated documents (i.e. removed documents with high overlap with other documents) within each dataset using Spark’s MinHashLSH implementation with 10 hashes, using the same features as were used for classification above. We also fuzzily removed WebText from Common Crawl. Overall this decreased dataset size by an average of 10%.

1. 为了提高 Common Crawl 的质量，我们开发了一种自动过滤方法来移除低质量的文档。使用原始的 WebText 作为高质量文档的代理，我们训练了一个分类器来区分这些文档与原始 Common Crawl 文档。然后，我们使用该分类器重新采样 Common Crawl，优先选择被分类器预测为高质量的文档。该分类器使用逻辑回归分类器训练，特征来自 Spark 的标准分词器和 HashingTF [^10]。对于正样本，我们使用了经过整理的数据集集合，如 WebText、Wikipedia 和我们的网络书籍语料库作为正样本；对于负样本，我们使用了未经筛选的 Common Crawl。我们使用该分类器对 Common Crawl 文档进行评分。我们仅在 $$\verb|np.random.pareto|(\alpha)>1-\verb|document_score|$$ 时保留数据集中的每个文档。我们选择 $\alpha=9$ 是为了主要获取分类器评分较高的文档，但仍包含一些分布外的文档。选择 $\alpha$ 是为了匹配分类器在 WebText 上的评分分布。我们发现这种重新加权提高了质量，这通过对一系列分布外生成文本样本的损失来衡量。2. 为了进一步提高模型质量并防止过拟合（随着模型容量的增加这一点变得越来越重要），我们对文档进行了模糊去重（即使用 Spark 的 MinHashLSH 实现（10 个哈希）在每个数据集中移除了与其他文档高度重叠的文档，使用的特征与上面用于分类的特征相同。我们还模糊地从 Common Crawl 中移除了 WebText。总体上，这使数据集的大小平均减少了 10%。

After filtering for duplicates and quality, we also partially removed text occurring in benchmark datasets, described in Appendix C.

在过滤重复项和质量后，我们还部分移除了出现在基准数据集中的文本，具体描述见附录C。

Appendix B Details of Model Training

附录 B 模型训练细节（Appendix B Details of Model Training）

To train all versions of GPT-3, we use Adam with $\beta_{1}=0.9$, $\beta_{2}=0.95$, and $\epsilon=10^{-8}$, we clip the global norm of the gradient at 1.0, and we use cosine decay for learning rate down to 10% of its value, over 260 billion tokens (after 260 billion tokens, training continues at 10% of the original learning rate). There is a linear LR warmup over the first 375 million tokens. We also gradually increase the batch size linearly from a small value (32k tokens) to the full value over the first 4-12 billion tokens of training, depending on the model size. Data are sampled without replacement during training (until an epoch boundary is reached) to minimize overfitting. All models use weight decay of 0.1 to provide a small amount of regularization [68].

为了训练 GPT-3 的所有版本，我们使用带有 $\beta_{1}=0.9$、$\beta_{2}=0.95$ 和 $\epsilon=10^{-8}$ 的 Adam，我们将梯度的全局范数裁剪到 1.0，并且我们使用余弦衰减将学习率降低到其值的 10%，在 2600 亿个标记上（在 2600 亿个标记之后，训练以原始学习率的 10% 继续）。在最初的 3.75 亿个标记上有线性学习率预热。我们还会在训练的前 40-120 亿个标记（根据模型大小而定）中，将批量大小从一个较小的值（32k 标记）线性地逐渐增加到完整值。训练过程中数据在采样时不放回（直到达到一个周期边界），以最小化过拟合。所有模型使用 0.1 的权重衰减以提供少量的正则化 [68]。

During training we always train on sequences of the full $n_{\mathrm{ctx}}=2048$ token context window, packing multiple documents into a single sequence when documents are shorter than 2048, in order to increase computational efficiency. Sequences with multiple documents are not masked in any special way but instead documents within a sequence are delimited with a special end of text token, giving the language model the information necessary to infer that context separated by the end of text token is unrelated. This allows for efficient training without need for any special sequence-specific masking.

在训练过程中，我们总是使用完整的 $n_{\mathrm{ctx}}=2048$ 令牌上下文窗口的序列进行训练，当文档长度小于2048时，将多个文档打包到一个序列中，以提高计算效率。包含多个文档的序列不会以任何特殊方式被掩码，而是在序列内的文档之间使用特殊的文本结束标记进行分隔，为语言模型提供了必要的信息，以推断由文本结束标记分隔的上下文是不相关的。这允许在不需要任何特定序列掩码的情况下进行高效训练。

Appendix C Details of Test Set Contamination Studies

附录 C 测试集污染研究细节（Appendix C Details of Test Set Contamination Studies）

In section 4.1 we gave a high level overview of test set contamination studies. In this section we provide details on methodology and results.

在第4.1节中，我们对测试集污染研究进行了高级概述。在本节中，我们将提供方法和结果的详细信息。

Initial training set filtering

初始训练集筛选

We attempted to remove text occurring in benchmarks from training data by searching for $13-$gram overlaps between all test/development sets used in this work and our training data, and we removed the colliding $13-$gram as well as a 200 character window around it, splitting the original document into pieces. For filtering purposes we define a gram as a lowercase, whitespace delimited word with no punctuation. Pieces less than $200$ characters long were discarded. Documents split into more than 10 pieces were considered contaminated and removed entirely. Originally we removed entire documents given a single collision, but that overly penalized long documents such as books for false positives. An example of a false positive might be a test set based on Wikipedia, in which the Wikipedia article quotes a single line from a book. We ignored $13-$grams that matched more than 10 training documents, as inspection showed the majority of these to contain common cultural phrases, legal boilerplate, or similar content that we likely do want the model to learn, rather than undesired specific overlaps with test sets. Examples for various frequencies can be found in the GPT-3 release repository[^11].

我们尝试通过在本工作中使用的所有测试/开发集与我们的训练数据之间搜索$13-$元组重叠，来移除训练数据中出现在基准测试中的文本，并且我们移除了冲突的$13-$元组及其周围 200 个字符的窗口，将原始文档拆分成若干片段。为了过滤的目的，我们将元组定义为小写、以空格分隔且无标点的单词。长度少于 $200$ 个字符的片段被丢弃。被拆分成超过 10 个片段的文档被视为被污染，整个文档会被完全移除。最初，我们对于单次碰撞就会移除整个文档，但这对于像书籍这样较长的文档因误报而惩罚过重。误报的一个例子可能是基于维基百科的测试集，其中维基百科文章引用了书中的一行文字。我们忽略了与超过10个训练文档匹配的$13-$grams，因为检查显示其中大多数包含常见的文化短语、法律模板或类似内容，而这些内容很可能是我们希望模型学习的，而不是与测试集不希望出现的特定重叠。各种频率的示例可以在GPT-3发布库中找到[^11]。

Overlap methodology

重叠方法在第4.1节的基准重叠分析中，

For our benchmark overlap analysis in Section 4.1, we used a variable number of words $N$ to check for overlap for each dataset, where $N$ is the 5th percentile example length in words, ignoring all punctuation, whitespace, and casing. Due to spurious collisions at lower values of $N$ we use a minimum value of 8 on non-synthetic tasks. For performance reasons, we set a maximum value of 13 for all tasks. Values for $N$ and the amount of data marked as dirty are shown in Table C.1. Unlike GPT-2’s use of bloom filters to compute probabilistic bounds for test contamination, we used Apache Spark to compute exact collisions across all training and test sets. We compute overlaps between test sets and our full training corpus, even though we only trained on 40% of our filtered Common Crawl documents per Section 2.2.

我们对每个数据集使用可变数量的单词$N$来检查重叠，其中$N$是按单词计的第5百分位样本长度，忽略所有标点、空格和大小写。由于在较低的$N$值下可能出现伪重叠，我们在非合成任务上使用最小值8。出于性能考虑，我们为所有任务设定了最大值13。$N$ 的数值以及标记为脏数据的数据量如表 C.1 所示。与 GPT-2 使用布隆过滤器计算测试污染的概率界限不同，我们使用 Apache Spark 计算了所有训练集和测试集的精确碰撞。我们计算了测试集与我们完整训练语料库之间的重叠，尽管根据第 2.2 节，我们只训练了经过筛选的 Common Crawl 文档的 40%。

We define a ‘dirty’ example as one with any $N$-gram overlap with any training document, and a ‘clean’ example as one with no collision.

我们将“脏”示例定义为与任何训练文档有任何 $N$-gram 重叠的示例，而将“干净”示例定义为没有冲突的示例。

Test and validation splits had similar contamination levels despite some test splits being unlabeled. Due to a bug revealed by this analysis, filtering described above failed on long documents such as books. Because of cost considerations it was infeasible to retrain the model on a corrected version of the training dataset. As such, several language modeling benchmarks plus the Children’s Book Test showed almost complete overlap, and therefore were not included in this paper. Overlaps are shown in Table C.1

尽管一些测试拆分没有标签，测试集和验证集拆分的污染水平仍然相似。由于此分析揭示的一个错误，上述描述的过滤在长文档（如书籍）上失败。由于成本考虑，在经过修正的训练数据集上重新训练模型是不可行的。因此，几种语言建模基准测试以及儿童图书测试显示几乎完全重叠，因此未包含在本文中。重叠情况见表C.1

Name	Split	Metric	$N$	Acc/F1/BLEU	Total Count	Dirty Acc/F1/BLEU	Dirty Count	Clean Acc/F1/BLEU	Clean Count	Clean Percentage	Relative Difference Clean vs All
Quac	dev	f1	13	44.3	7353	44.3	7315	54.1	38	1%	20%
SQuADv2	dev	f1	13	69.8	11873	69.9	11136	68.4	737	6%	-2%
DROP	dev	f1	13	36.5	9536	37.0	8898	29.5	638	7%	-21%
Symbol Insertion	dev	acc	7	66.9	10000	66.8	8565	67.1	1435	14%	0%
CoQa	dev	f1	13	86.0	7983	85.3	5107	87.1	2876	36%	1%
ReCoRD	dev	acc	13	89.5	10000	90.3	6110	88.2	3890	39%	-1%
Winograd	test	acc	9	88.6	273	90.2	164	86.2	109	40%	-3%
BoolQ	dev	acc	13	76.0	3270	75.8	1955	76.3	1315	40%	0%
MultiRC	dev	acc	13	74.2	953	73.4	558	75.3	395	41%	1%
RACE-h	test	acc	13	46.8	3498	47.0	1580	46.7	1918	55%	0%
LAMBADA	test	acc	13	86.4	5153	86.9	2209	86.0	2944	57%	0%
LAMBADA (No Blanks)	test	acc	13	77.8	5153	78.5	2209	77.2	2944	57%	-1%
WSC	dev	acc	13	76.9	104	73.8	42	79.0	62	60%	3%
PIQA	dev	acc	8	82.3	1838	89.9	526	79.3	1312	71%	-4%
RACE-m	test	acc	13	58.5	1436	53.0	366	60.4	1070	75%	3%
De$\to$En 16	test	bleu-sb	12	43.0	2999	47.4	739	40.8	2260	75%	-5%
En$\to$De 16	test	bleu-sb	12	30.9	2999	32.6	739	29.9	2260	75%	-3%
En$\to$Ro 16	test	bleu-sb	12	25.8	1999	24.9	423	26.1	1576	79%	1%
Ro$\to$En 16	test	bleu-sb	12	41.3	1999	40.4	423	41.6	1576	79%	1%
WebQs	test	acc	8	41.5	2032	41.6	428	41.5	1604	79%	0%
ANLI R1	test	acc	13	36.8	1000	40.5	200	35.9	800	80%	-3%
ANLI R2	test	acc	13	34.0	1000	29.4	177	35.0	823	82%	3%
TriviaQA	dev	acc	10	71.2	7993	70.8	1390	71.3	6603	83%	0%
ANLI R3	test	acc	13	40.2	1200	38.3	196	40.5	1004	84%	1%
En$\to$Fr 14	test	bleu-sb	13	39.9	3003	38.3	411	40.3	2592	86%	1%
Fr$\to$En 14	test	bleu-sb	13	41.4	3003	40.9	411	41.4	2592	86%	0%
WiC	dev	acc	13	51.4	638	53.1	49	51.3	589	92%	0%
RTE	dev	acc	13	71.5	277	71.4	21	71.5	256	92%	0%
CB	dev	acc	13	80.4	56	100.0	4	78.8	52	93%	-2%
Anagrams 2	dev	acc	2	40.2	10000	76.2	705	37.4	9295	93%	-7%
Reversed Words	dev	acc	2	0.4	10000	1.5	660	0.3	9340	93%	-26%
OpenBookQA	test	acc	8	65.4	500	58.1	31	65.9	469	94%	1%
ARC (Easy)	test	acc	11	70.1	2268	77.5	89	69.8	2179	96%	0%
Anagrams 1	dev	acc	2	15.0	10000	49.8	327	13.8	9673	97%	-8%
COPA	dev	acc	9	93.0	100	100.0	3	92.8	97	97%	0%
ARC (Challenge)	test	acc	12	51.6	1144	45.2	31	51.8	1113	97%	0%
HellaSwag	dev	acc	13	79.3	10042	86.2	152	79.2	9890	98%	0%
NQs	test	acc	11	29.9	3610	32.7	52	29.8	3558	99%	0%
Cycled Letters	dev	acc	2	38.6	10000	20.5	73	38.7	9927	99%	0%
SAT Analogies	dev	acc	9	65.8	374	100.0	2	65.6	372	99%	0%
StoryCloze	test	acc	13	87.7	1871	100.0	2	87.6	1869	100%	0%
Winogrande	dev	acc	13	77.7	1267	-	0	77.7	1267	100%	0%

Table C.1: Overlap statistics for all datasets sorted from dirtiest to cleanest. We consider a dataset example dirty if it has a single $N$-gram collision with any document in our training corpus. “Relative Difference Clean vs All” shows the percent change in performance between only the clean examples vs all the examples in the benchmark. “Count” shows the number of examples. “Clean percentage” is the percent of examples that are clean vs total. For “Acc/F1/BLEU” we use the metric specified in “Metric”. These scores come from evaluations with a different seed for the random examples used for in-context learning, and will therefore differ slightly from the scores elsewhere in the paper.

名称	划分	指标	$N$	准确率/F1/BLEU	总数	脏数据准确率/F1/BLEU	脏数据计数	干净数据准确率/F1/BLEU	干净数据计数	干净数据比例	干净数据与总体的相对差异
Quac	dev	f1	13	44.3	7353	44.3	7315	54.1	38	1%	20%
SQuADv2	dev	f1	13	69.8	11873	69.9	11136	68.4	737	6%	-2%
DROP	开发集	f1	13	36.5	9536	37.0	8898	29.5	638	7%	-21%
符号插入	开发集	准确率	7	66.9	10000	66.8	8565	67.1	1435	14%	0%
CoQa	开发集	f1	13	86.0	7983	85.3	5107	87.1	2876	36%	1%
ReCoRD	开发集	准确率	13	89.5	10000	90.3	6110	88.2	3890	39%	-1%
Winograd	测试集	准确率	9	88.6	273	90.2	164	86.2	109	40%	-3%
BoolQ	开发	准确率	13	76.0	3270	75.8	1955	76.3	1315	40%	0%
MultiRC	开发	准确率	13	74.2	953	73.4	558	75.3	395	41%	1%
RACE-h	测试	准确率	13	46.8	3498	47.0	1580	46.7	1918	55%	0%
LAMBADA	测试	准确率	13	86.4	5153	86.9	2209	86.0	2944	57%	0%
LAMBADA（无空白）	测试	准确率	13	77.8	5153	78.5	2209	77.2	2944	57%	-1%
WSC	开发	准确率	13	76.9	104	73.8	42	79.0	62	60%	3%
PIQA	开发	准确率	8	82.3	1838	89.9	526	79.3	1312	71%	-4%
RACE-m	测试	准确率	13	58.5	1436	53.0	366	60.4	1070	75%	3%
De$\to$En 16	测试	bleu-sb	12	43.0	2999	47.4	739	40.8	2260	75%	-5%
En$\to$De 16	测试	bleu-sb	12	30.9	2999	32.6	739	29.9	2260	75%	-3%
En$\to$Ro 16	测试	bleu-sb	12	25.8	1999	24.9	423	26.1	1576	79%	1%
Ro$\to$En 16	测试	bleu-sb	12	41.3	1999	40.4	423	41.6	1576	79%	1%
WebQs	测试	acc	8	41.5	2032	41.6	428	41.5	1604	79%	0%
ANLI R1	测试	acc	13	36.8	1000	40.5	200	35.9	800	80%	-3%
ANLI R2	测试	acc	13	34.0	1000	29.4	177	35.0	823	82%	3%
TriviaQA	开发	准确率	10	71.2	7993	70.8	1390	71.3	6603	83%	0%
ANLI R3	测试	准确率	13	40.2	1200	38.3	196	40.5	1004	84%	1%
En$\to$Fr 14	测试	BLEU-sb	13	39.9	3003	38.3	411	40.3	2592	86%	1%
Fr$\to$En 14	测试	BLEU-sb	13	41.4	3003	40.9	411	41.4	2592	86%	0%
WiC	开发	准确率	13	51.4	638	53.1	49	51.3	589	92%	0%
RTE	开发	准确率	13	71.5	277	71.4	21	71.5	256	92%	0%
CB	开发	准确率	13	80.4	56	100.0	4	78.8	52	93%	-2%
字谜 2	开发	准确率	2	40.2	10000	76.2	705	37.4	9295	93%	-7%
反转单词	开发	准确率	2	0.4	10000	1.5	660	0.3	9340	93%	-26%
OpenBookQA	测试	准确率	8	65.4	500	58.1	31	65.9	469	94%	1%
ARC（简单）	测试	准确率	11	70.1	2268	77.5	89	69.8	2179	96%	0%
字谜 1	开发	准确率	2	15.0	10000	49.8	327	13.8	9673	97%	-8%
COPA	开发	准确率	9	93.0	100	100.0	3	92.8	97	97%	0%
ARC（挑战）	测试	准确率	12	51.6	1144	45.2	31	51.8	1113	97%	0%
HellaSwag	开发	准确率	13	79.3	10042	86.2	152	79.2	9890	98%	0%
NQs	测试	准确率	11	29.9	3610	32.7	52	29.8	3558	99%	0%
循环字母	开发	准确率	2	38.6	10000	20.5	73	38.7	9927	99%	0%
SAT 类比	开发	准确率	9	65.8	374	100.0	2	65.6	372	99%	0%
故事填空	测试	准确率	13	87.7	1871	100.0	2	87.6	1869	100%	0%
Winogrande	开发	准确率	13	77.7	1267	-	0	77.7	1267	100%	0%
表 C.1：按从最脏到最干净排序的所有数据集的重叠统计。我们认为如果数据集示例与训练语料库中的任何文档存在单个 $N$-gram 冲突，则该示例为脏数据。“相对差异 干净 vs 全部”显示仅干净示例与基准中所有示例的性能百分比变化。“数量”显示示例的数量。“干净百分比”是干净示例占总示例的百分比。对于“Acc/F1/BLEU”，我们使用“指标”中指定的度量。这些分数来自对用于上下文学习的随机示例进行不同随机种子的评估，因此与论文其他地方的分数会略有不同。

Overlap results

重叠结果

To understand how much having seen some of the data helps the model perform on downstream tasks, we filter every validation and test set by dirtiness. Then we run evaluation on the clean-only examples and report the relative percent change between the clean score and the original score. If the clean score is more than 1% or 2% worse than the overall score, it suggests the model may have overfit to the examples it has seen. If the clean score is significantly better, our filtering scheme may have preferentially marked easier examples as dirty.

为了理解看到部分数据对模型在下游任务中的表现有多大帮助，我们按脏数据程度过滤每个验证集和测试集。然后，我们只在干净的示例上运行评估，并报告干净分数与原始分数之间的相对百分比变化。如果干净分数比总体分数低超过1%或2%，这表明模型可能已经对它见过的示例过拟合。如果干净分数明显更好，我们的过滤方案可能优先将较简单的样本标记为脏样本。

This overlap metric tends to show a high rate of false positives for datasets that contain background information (but not answers) drawn from the web (such as SQuAD, which draws from Wikipedia) or examples less than 8 words long, which we ignored in our filtering process (except for wordscrambling tasks). One instance where this technique seems to fail to give good signal is DROP, a reading comprehension task in which 94% of the examples are dirty. The information required to answer the question is in a passage provided to the model, so having seen the passage during training but not the questions and answers does not meaningfully constitute cheating. We confirmed that every matching training document contained only the source passage, and none of the questions and answers in the dataset. The more likely explanation for the decrease in performance is that the 6% of examples that remain after filtering come from a slightly different distribution than the dirty examples.

对于包含来自网络的背景信息（但不包含答案）的数据集（例如从维基百科提取的 SQuAD）或少于 8 个单词的示例（在我们的过滤过程中我们忽略了这些，单词打乱任务除外），这种重叠度量往往会显示出较高的假阳性率。这种技术似乎无法提供良好信号的一个例子是DROP，这是一个阅读理解任务，其中94%的例子是脏的。回答问题所需的信息包含在提供给模型的文章中，因此在训练期间看到文章但没有看到问题和答案，并不意味着真正的作弊。我们确认每个匹配的训练文档仅包含源段落，而不包含数据集中的任何问题和答案。性能下降更可能的解释是，过滤后剩下的 6% 示例来自与脏示例略有不同的分布。

Figure 4.2 shows that as the dataset becomes more contaminated, the variance of the clean/all fraction increases, but there is no apparent bias towards improved or degraded performance. This suggests that GPT-3 is relatively insensitive to contamination. See Section 4.1 for details on the datasets we flagged for further review.

图 4.2 显示，随着数据集被污染程度增加，干净/全部比例的方差增加，但并没有明显的偏向于性能提高或下降。这表明 GPT-3 对污染相对不敏感。有关我们标记以供进一步审查的数据集的详细信息，请参见第 4.1 节。

Appendix D Total Compute Used to Train Language Models

附录 D 训练语言模型所用的总算力（Appendix D Total Compute Used to Train Language Models）

This appendix contains the calculations that were used to derive the approximate compute used to train the language models in Figure 2.2. As a simplifying assumption, we ignore the attention operation, as it typically uses less than 10% of the total compute for the models we are analyzing.

本附录包含用于推导图 2.2 中训练语言模型的近似计算所使用的计算。作为简化假设，我们忽略注意力操作，因为对于我们正在分析的模型，它通常使用的计算不到总计算量的 10%。

Calculations can be seen in Table D.1 and are explained within the table caption.

计算结果可以在表 D.1 中看到，并在表标题中进行了说明。

Model	Total train compute (PF-days)	Total train compute (flops)	Params (M)	Training tokens (billions)	Flops per param per token	Mult for bwd pass	Fwd-pass flops per active param per token	Frac of params active for each token
T5-Small	2.08E+00	1.80E+20	60	1,000	3	3	1	0.5
T5-Base	7.64E+00	6.60E+20	220	1,000	3	3	1	0.5
T5-Large	2.67E+01	2.31E+21	770	1,000	3	3	1	0.5
T5-3B	1.04E+02	9.00E+21	3,000	1,000	3	3	1	0.5
T5-11B	3.82E+02	3.30E+22	11,000	1,000	3	3	1	0.5
BERT-Base	1.89E+00	1.64E+20	109	250	6	3	2	1.0
BERT-Large	6.16E+00	5.33E+20	355	250	6	3	2	1.0
RoBERTa-Base	1.74E+01	1.50E+21	125	2,000	6	3	2	1.0
RoBERTa-Large	4.93E+01	4.26E+21	355	2,000	6	3	2	1.0
GPT-3 Small	2.60E+00	2.25E+20	125	300	6	3	2	1.0
GPT-3 Medium	7.42E+00	6.41E+20	356	300	6	3	2	1.0
GPT-3 Large	1.58E+01	1.37E+21	760	300	6	3	2	1.0
GPT-3 XL	2.75E+01	2.38E+21	1,320	300	6	3	2	1.0
GPT-3 2.7B	5.52E+01	4.77E+21	2,650	300	6	3	2	1.0
GPT-3 6.7B	1.39E+02	1.20E+22	6,660	300	6	3	2	1.0
GPT-3 13B	2.68E+02	2.31E+22	12,850	300	6	3	2	1.0
GPT-3 175B	3.64E+03	3.14E+23	174,600	300	6	3	2	1.0

Table D.1: Starting from the right hand side and moving left, we begin with the number of training tokens that each model was trained with. Next we note that since T5 uses an encoder-decoder model, only half of the parameters are active for each token during a forward or backwards pass. We then note that each token is involved in a single addition and a single multiply for each active parameter in the forward pass (ignoring attention). Then we add a multiplier of 3x to account for the backwards pass (as computing both $\frac{\partial{params}}{\partial{loss}}$ and $\frac{\partial{acts}}{\partial{loss}}$ use a similar amount of compute as the forwards pass. Combining the previous two numbers, we get the total flops per parameter per token. We multiply this value by the total training tokens and the total parameters to yield the number of total flops used during training. We report both flops and petaflop/s-day (each of which are 8.64e+19 flops).

模型	总训练计算量（PF-天）	总训练计算量（浮点运算次数）	参数数量（百万）	训练标记（十亿）	每个参数每个标记的浮点运算次数	反向传播倍数	每个活跃参数每个标记的前向传播浮点运算次数	每个标记活跃参数的比例
T5-小型	2.08E+00	1.80E+20	60	1,000	3	3	1	0.5
T5-基础	7.64E+00	6.60E+20	220	1,000	3	3	1	0.5
T5-Large	2.67E+01	2.31E+21	770	1,000	3	3	1	0.5
T5-3B	1.04E+02	9.00E+21	3,000	1,000	3	3	1	0.5
T5-11B	3.82E+02	3.30E+22	11,000	1,000	3	3	1	0.5
BERT-Base	1.89E+00	1.64E+20	109	250	6	3	2	1.0
BERT-Large	6.16E+00	5.33E+20	355	250	6	3	2	1.0
RoBERTa-Base	1.74E+01	1.50E+21	125	2,000	6	3	2	1.0
RoBERTa-Large	4.93E+01	4.26E+21	355	2,000	6	3	2	1.0
GPT-3 Small	2.60E+00	2.25E+20	125	300	6	3	2	1.0
GPT-3 Medium	7.42E+00	6.41E+20	356	300	6	3	2	1.0
GPT-3 Large	1.58E+01	1.37E+21	760	300	6	3	2	1.0
GPT-3 XL	2.75E+01	2.38E+21	1,320	300	6	3	2	1.0
GPT-3 2.7B	5.52E+01	4.77E+21	2,650	300	6	3	2	1.0
GPT-3 6.7B	1.39E+02	1.20E+22	6,660	300	6	3	2	1.0
GPT-3 13B	2.68E+02	2.31E+22	12,850	300	6	3	2	1.0
GPT-3 175B	3.64E+03	3.14E+23	174,600	300	6	3	2	1.0
表 D.1：从右向左开始，我们首先列出每个模型所训练的训练标记数。接下来我们注意到，由于 T5 使用的是编码器-解码器模型，因此在前向或反向传播过程中，每个标记只有一半的参数是活跃的。然后我们注意到，在前向传播中，每个标记涉及每个活跃参数的一次加法和一次乘法（忽略注意力机制）。接着我们添加一个 3 倍的乘数以考虑反向传播（因为计算 $\frac{\partial{params}}{\partial{loss}}$ 和 $\frac{\partial{acts}}{\partial{loss}}$ 所需的计算量与前向传播相似）。结合前面两个数字，我们得到每个参数每个标记的总浮点运算次数。我们将该值乘以总训练标记数和总参数数，以得出训练期间使用的总浮点运算次数。我们报告浮点运算次数和百亿亿次浮点运算/天（每个为 8.64e+19 次浮点运算）。