SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

SuperGLUE：面向通用语言理解系统的更「黏」基准

Alex WangNew York University Thanks: Equal contribution. Correspondence: glue-benchmark-admin@googlegroups.com Yada PruksachatkunNew York University Nikita NangiaNew York University Amanpreet SinghFacebook AI ResearchJulian MichaelUniversity of WashingtonFelix HillDeepMindOmer LevyFacebook AI ResearchSamuel R. BowmanNew York University

Abstract

摘要

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

过去一年，预训练与迁移学习的新模型与方法在一系列语言理解任务上带来显著提升。一年多前推出的 GLUE 基准用单一数字汇总多样任务上的进展，但近期表现已超过非专家人类水平，意味着进一步研究所剩空间有限。本文提出 SuperGLUE：沿袭 GLUE 风格的新基准，包含一组更难的语言理解任务、软件工具包与公开排行榜。SuperGLUE 见 super.gluebenchmark.com。

1 Introduction

1 引言

Recently there has been notable progress across many natural language processing (NLP) tasks, led by methods such as ELMo (Peters et al. 2018), OpenAI GPT (Radford et al. 2018), and BERT (Devlin et al. 2019). The unifying theme of these methods is that they couple self-supervised learning from massive unlabelled text corpora with effective adapting of the resulting model to target tasks. The tasks that have proven amenable to this general approach include question answering, textual entailment, and parsing, among many others (Devlin et al. 2019; Kitaev et al. 2019, i.a.).

近来自然语言处理（NLP）多项任务取得显著进展，引领者包括 ELMo（Peters et al. 2018）、OpenAI GPT（Radford et al. 2018）与 BERT（Devlin et al. 2019）。这些方法的共同主题是：在海量无标注文本上做自监督学习，再把所得模型有效适配到目标任务。已被证明适合这一通用路线的任务包括问答、文本蕴含与句法分析等（Devlin et al. 2019；Kitaev et al. 2019 等）。

In this context, the GLUE benchmark (Wang et al. 2019a) has become a prominent evaluation framework for research towards general-purpose language understanding technologies. GLUE is a collection of nine language understanding tasks built on existing public datasets, together with private test data, an evaluation server, a single-number target metric, and an accompanying expert-constructed diagnostic set. GLUE was designed to provide a general-purpose evaluation of language understanding that covers a range of training data volumes, task genres, and task formulations. We believe it was these aspects that made GLUE particularly appropriate for exhibiting the transfer-learning potential of approaches like OpenAI GPT and BERT.

在此背景下，GLUE 基准（Wang et al. 2019a）已成为面向通用语言理解技术研究的重要评测框架。GLUE 汇集九项基于已有公开数据的语言理解任务，并配有私有测试数据、评测服务器、单一目标指标，以及专家构建的诊断集。GLUE 旨在提供覆盖不同训练数据量、任务体裁与任务形式的通用语言理解评测。我们认为，正是这些方面使 GLUE 特别适合展示 OpenAI GPT、BERT 一类方法的迁移学习潜力。

The progress of the last twelve months has eroded headroom on the GLUE benchmark dramatically. While some tasks (Figure 1) and some linguistic phenomena (Figure 2 in Appendix B) measured in GLUE remain difficult, the current state of the art GLUE Score as of early July 2019 (Yang et al. 2019, 88.4 from) surpasses human performance (Nangia and Bowman 2019, 87.1 from) by 1.3 points, and in fact exceeds this human performance estimate on four tasks. Consequently, while there remains substantial scope for improvement towards GLUE’s high-level goals, the original version of the benchmark is no longer a suitable metric for quantifying such progress.

过去十二个月的进展急剧压缩了 GLUE 上的提升空间。尽管 GLUE 中部分任务（图 1）与部分语言现象（附录 B 图 2）仍然困难，但截至 2019 年 7 月初的最优 GLUE 分数（Yang et al. 2019，88.4）已比人类表现（Nangia and Bowman 2019，87.1）高出 1.3 分，并在四项任务上超过该人类估计。因此，尽管朝向 GLUE 高层目标仍有大量改进空间，原版基准已不再适合量化这类进展。

Figure 1: GLUE benchmark performance for submitted systems, rescaled to set human performance to 1.0, shown as a single number score, and broken down into the nine constituent task performances. For tasks with multiple metrics, we use an average of the metrics. More information on the tasks included in GLUE can be found in Wang et al. 2019a and in Warstadt et al. 2019, Socher et al. 2013, Dolan and Brockett 2005, Cer et al. 2017, and Williams et al. 2018, and Rajpurkar et al. 2016.

图 1：提交系统在 GLUE 基准上的表现，按人类表现为 1.0 重新缩放；既给出总分，也拆成九项任务表现。对有多项指标的任务取平均。GLUE 任务详情见 Wang et al. 2019a，以及 Warstadt et al. 2019、Socher et al. 2013、Dolan and Brockett 2005、Cer et al. 2017、Williams et al. 2018、Rajpurkar et al. 2016。

In response, we introduce SuperGLUE, a new benchmark designed to pose a more rigorous test of language understanding. SuperGLUE has the same high-level motivation as GLUE: to provide a simple, hard-to-game measure of progress toward general-purpose language understanding technologies for English. We anticipate that significant progress on SuperGLUE should require substantive innovations in a number of core areas of machine learning, including sample-efficient, transfer, multitask, and unsupervised or self-supervised learning.

作为回应，我们推出 SuperGLUE：旨在对语言理解提出更严格考验的新基准。SuperGLUE 与 GLUE 高层动机相同：为英语通用语言理解技术提供简单、难以刷分的进展度量。我们预期，在 SuperGLUE 上取得显著进展需要机器学习若干核心方向的实质创新，包括样本高效学习、迁移学习、多任务学习，以及无监督或自监督学习。

SuperGLUE follows the basic design of GLUE: It consists of a public leaderboard built around eight language understanding tasks, drawing on existing data, accompanied by a single-number performance metric, and an analysis toolkit. However, it improves upon GLUE in several ways:

SuperGLUE 沿袭 GLUE 的基本设计：围绕八项语言理解任务搭建公开排行榜，任务基于已有数据，配有单一性能指标与分析工具包。同时在若干方面改进 GLUE：

More challenging tasks: SuperGLUE retains the two hardest tasks in GLUE. The remaining tasks were identified from those submitted to an open call for task proposals and were selected based on difficulty for current NLP approaches.

更具挑战的任务 ： SuperGLUE 保留 GLUE 中最难的两项任务。其余任务来自公开征集的任务提案，并按对当前 NLP 方法的难度筛选。

More diverse task formats: The task formats in GLUE are limited to sentence- and sentence-pair classification. We expand the set of task formats in SuperGLUE to include coreference resolution and question answering (QA).

更多样的任务形式 ： GLUE 的任务形式限于句子与句对分类。SuperGLUE 扩展到指代消解与问答（QA）。

Comprehensive human baselines: We include human performance estimates for all benchmark tasks, which verify that substantial headroom exists between a strong BERT-based baseline and human performance.

全面的人类基线 ： 我们为全部基准任务提供人类表现估计，以核实强 BERT 基线与人类表现之间仍有实质空间。

Improved code support: SuperGLUE is distributed with a new, modular toolkit for work on pretraining, multi-task learning, and transfer learning in NLP, built around standard tools including PyTorch (Paszke et al. 2017) and AllenNLP (Gardner et al. 2017).

更好的代码支持 ： SuperGLUE 随附面向 NLP 预训练、多任务学习与迁移学习的新模块化工具包，基于 PyTorch（Paszke et al. 2017）与 AllenNLP（Gardner et al. 2017）等标准工具。

Refined usage rules: The conditions for inclusion on the SuperGLUE leaderboard have been revamped to ensure fair competition, an informative leaderboard, and full credit assignment to data and task creators.

更精细的使用规则 ： SuperGLUE 排行榜收录条件经修订，以确保公平竞争、信息充分的排行榜，并向数据与任务创建者完整署名致谢。

The SuperGLUE leaderboard, data, and software tools are available at super.gluebenchmark.com.

SuperGLUE 排行榜、数据与软件工具见 super.gluebenchmark.com。

2 Related Work

2 相关工作

Much work prior to GLUE demonstrated that training neural models with large amounts of available supervision can produce representations that effectively transfer to a broad range of NLP tasks (Collobert and Weston 2008; Dai and Le 2015; Kiros et al. 2015; Hill et al. 2016; Conneau and Kiela 2018; McCann et al. 2017; Peters et al. 2018). GLUE was presented as a formal challenge affording straightforward comparison between such task-agnostic transfer learning techniques. Other similarly-motivated benchmarks include SentEval (Conneau and Kiela 2018), which specifically evaluates fixed-size sentence embeddings, and DecaNLP (McCann et al. 2018), which recasts a set of target tasks into a general question-answering format and prohibits task-specific parameters. In contrast, GLUE provides a lightweight classification API and no restrictions on model architecture or parameter sharing, which seems to have been well-suited to recent work in this area.

GLUE 之前已有大量工作表明：用大量可用监督训练神经模型，可得到有效迁移到广泛 NLP 任务的表示（Collobert and Weston 2008；Dai and Le 2015；Kiros et al. 2015；Hill et al. 2016；Conneau and Kiela 2018；McCann et al. 2017；Peters et al. 2018）。GLUE 被提出为正式挑战，便于直接比较这类任务无关的迁移学习技术。动机相近的其他基准包括专门评测固定维句向量的 SentEval（Conneau and Kiela 2018），以及把目标任务改写为通用问答格式并禁止任务特定参数的 DecaNLP（McCann et al. 2018）。相比之下，GLUE 提供轻量分类 API，且对模型架构与参数共享不加限制，这似乎很契合该领域近期工作。

Since its release, GLUE has been used as a testbed and showcase by the developers of several influential models, including GPT (Radford et al. 2018) and BERT (Devlin et al. 2019). As shown in Figure 1, progress on GLUE since its release has been striking. On GLUE, GPT and BERT achieved scores of 72.8 and 80.2 respectively, relative to 66.5 for an ELMo-based model (Peters et al. 2018) and 63.7 for the strongest baseline with no multitask learning or pretraining above the word level. Recent models (Liu et al. 2019d; Yang et al. 2019) have clearly surpassed estimates of non-expert human performance on GLUE (Nangia and Bowman 2019). The success of these models on GLUE has been driven by ever-increasing model capacity, compute power, and data quantity, as well as innovations in model expressivity (from recurrent to bidirectional recurrent to multi-headed transformer encoders) and degree of contextualization (from learning representation of words in isolation to using uni-directional contexts and ultimately to leveraging bidirectional contexts).

发布以来，GLUE 被若干有影响力模型的开发者用作试验床与展示平台，包括 GPT（Radford et al. 2018）与 BERT（Devlin et al. 2019）。如图 1 所示，发布后 GLUE 上的进展十分显著：GPT 与 BERT 分别达到 72.8 与 80.2，相对基于 ELMo 的模型（Peters et al. 2018）的 66.5，以及词级以上无多任务学习或预训练的最强基线 63.7。近期模型（Liu et al. 2019d；Yang et al. 2019）已明显超过 GLUE 上非专家人类表现估计（Nangia and Bowman 2019）。这些模型在 GLUE 上的成功，既来自模型容量、算力与数据量的持续扩大，也来自模型表达力（从循环到双向循环再到多头 Transformer 编码器）与上下文化程度（从孤立词表示，到单向上下文，再到双向上下文）的创新。

In parallel to work scaling up pretrained models, several studies have focused on complementary methods for augmenting performance of pretrained models. Phang et al. 2018 show that BERT can be improved using two-stage pretraining, i.e., fine-tuning the pretrained model on an intermediate data-rich supervised task before fine-tuning it again on a data-poor target task. Liu et al. 2019d; Liu et al. 2019c and Bach et al. 2018 get further improvements respectively via multi-task finetuning and using massive amounts of weak supervision. Clark et al. 2019b demonstrate that knowledge distillation (Hinton et al. 2015; Furlanello et al. 2018) can lead to student networks that outperform their teachers. Overall, the quantity and quality of research contributions aimed at the challenges posed by GLUE underline the utility of this style of benchmark for machine learning researchers looking to evaluate new application-agnostic methods on language understanding.

与扩大预训练模型并行，若干研究聚焦增强预训练模型表现的互补方法。Phang et al. 2018 表明 BERT 可用两阶段预训练改进：先在数据丰富的中间监督任务上微调预训练模型，再在数据稀缺的目标任务上微调。Liu et al. 2019d；Liu et al. 2019c 与 Bach et al. 2018 分别通过多任务微调与海量弱监督获得进一步提升。Clark et al. 2019b 表明知识蒸馏（Hinton et al. 2015；Furlanello et al. 2018）可得到优于教师的学生网络。总体而言，面向 GLUE 挑战的研究贡献的数量与质量，凸显了这类基准对希望在语言理解上评测新应用无关方法的机器学习研究者的价值。

R_{3}

R_{3}

3 SuperGLUE Overview

3 SuperGLUE 概览

|

Corpus	$\|$ Train $\|$	$\|$ Dev $\|$	$\|$ Test $\|$	Task	Metrics	Text Sources
BoolQ	9427	3270	3245	QA	acc.	Google queries, Wikipedia
CB	250	57	250	NLI	acc./F1	various
COPA	400	100	500	QA	acc.	blogs, photography encyclopedia
MultiRC	5100	953	1800	QA	F1a/EM	various
ReCoRD	101k	10k	10k	QA	F1/EM	news (CNN, Daily Mail)
RTE	2500	278	300	NLI	acc.	news, Wikipedia
WiC	6000	638	1400	WSD	acc.	WordNet, VerbNet, Wiktionary
WSC	554	104	146	coref.	acc.	fiction books

|

语料	$\|$ Train $\|$	$\|$ Dev $\|$	$\|$ Test $\|$	任务	指标	文本来源
BoolQ	9427	3270	3245	QA	acc.	Google 查询、维基百科
CB	250	57	250	NLI	acc./F1	多种
COPA	400	100	500	QA	acc.	博客、摄影百科
MultiRC	5100	953	1800	QA	F1a/EM	多种
ReCoRD	101k	10k	10k	QA	F1/EM	新闻（CNN、Daily Mail）
RTE	2500	278	300	NLI	acc.	新闻、维基百科
WiC	6000	638	1400	WSD	acc.	WordNet、VerbNet、Wiktionary
WSC	554	104	146	coref.	acc.	虚构书籍

BoolQ	Passage: Barq’s – Barq’s is an American soft drink. Its brand of root beer is notable for having caffeine. Barq’s, created by Edward Barq and bottled since the turn of the 20th century, is owned by the Barq family but bottled by the Coca-Cola Company. It was known as Barq’s Famous Olde Tyme Root Beer until 2012.
	Question: is barq’s root beer a pepsi product Answer: No
CB	Text: B: And yet, uh, I we-, I hope to see employer based, you know, helping out. You know, child, uh, care centers at the place of employment and things like that, that will help out. A: Uh-huh. B: What do you think, do you think we are, setting a trend?
	Hypothesis: they are setting a trend Entailment: Unknown
COPA	Premise: My body cast a shadow over the grass. Question: What’s the CAUSE for this?
	Alternative 1: The sun was rising. Alternative 2: The grass was cut.
	Correct Alternative: 1
MultiRC	Paragraph: Susan wanted to have a birthday party. She called all of her friends. She has five friends. Her mom said that Susan can invite them all to the party. Her first friend could not go to the party because she was sick. Her second friend was going out of town. Her third friend was not so sure if her parents would let her. The fourth friend said maybe. The fifth friend could go to the party for sure. Susan was a little sad. On the day of the party, all five friends showed up. Each friend had a present for Susan. Susan was happy and sent each friend a thank you card the next week
	Question: Did Susan’s sick friend recover? Candidate answers: Yes, she recovered (T), No (F), Yes (T), No, she didn’t recover (F), Yes, she was at Susan’s party (T)
ReCoRD	Paragraph: (CNN) Puerto Rico on Sunday overwhelmingly voted for statehood. But Congress, the only body that can approve new states, will ultimately decide whether the status of the US commonwealth changes. Ninety-seven percent of the votes in the nonbinding referendum favored statehood, an increase over the results of a 2012 referendum, official results from the State Electorcal Commission show. It was the fifth such vote on statehood. "Today, we the people of Puerto Rico are sending a strong and clear message to the US Congress … and to the world … claiming our equal rights as American citizens, Puerto Rico Gov. Ricardo Rossello said in a news release. @highlight Puerto Rico voted Sunday in favor of US statehood
	Query For one, they can truthfully say, “Don’t blame me, I didn’t vote for them, ” when discussing the presidency Correct Entities: US
RTE	Text: Dana Reeve, the widow of the actor Christopher Reeve, has died of lung cancer at age 44, according to the Christopher Reeve Foundation.
	Hypothesis: Christopher Reeve had an accident. Entailment: False
WiC	Context 1: Room and board. Context 2: He nailed boards across the windows.
	Sense match: False
WSC	Text: Mark told Pete many lies about himself, which Pete included in his book. He should have been more truthful. Coreference: False

BoolQ	Passage: Barq’s – Barq’s is an American soft drink. Its brand of root beer is notable for having caffeine. Barq’s, created by Edward Barq and bottled since the turn of the 20th century, is owned by the Barq family but bottled by the Coca-Cola Company. It was known as Barq’s Famous Olde Tyme Root Beer until 2012.
	Question: is barq’s root beer a pepsi product Answer: No
CB	Text: B: And yet, uh, I we-, I hope to see employer based, you know, helping out. You know, child, uh, care centers at the place of employment and things like that, that will help out. A: Uh-huh. B: What do you think, do you think we are, setting a trend?
	Hypothesis: they are setting a trend Entailment: Unknown
COPA	Premise: My body cast a shadow over the grass. Question: What’s the CAUSE for this?
	Alternative 1: The sun was rising. Alternative 2: The grass was cut.
	Correct Alternative: 1
MultiRC	Paragraph: Susan wanted to have a birthday party. She called all of her friends. She has five friends. Her mom said that Susan can invite them all to the party. Her first friend could not go to the party because she was sick. Her second friend was going out of town. Her third friend was not so sure if her parents would let her. The fourth friend said maybe. The fifth friend could go to the party for sure. Susan was a little sad. On the day of the party, all five friends showed up. Each friend had a present for Susan. Susan was happy and sent each friend a thank you card the next week
	Question: Did Susan’s sick friend recover? Candidate answers: Yes, she recovered (T), No (F), Yes (T), No, she didn’t recover (F), Yes, she was at Susan’s party (T)
ReCoRD	Paragraph: (CNN) Puerto Rico on Sunday overwhelmingly voted for statehood. But Congress, the only body that can approve new states, will ultimately decide whether the status of the US commonwealth changes. Ninety-seven percent of the votes in the nonbinding referendum favored statehood, an increase over the results of a 2012 referendum, official results from the State Electorcal Commission show. It was the fifth such vote on statehood. "Today, we the people of Puerto Rico are sending a strong and clear message to the US Congress … and to the world … claiming our equal rights as American citizens, Puerto Rico Gov. Ricardo Rossello said in a news release. @highlight Puerto Rico voted Sunday in favor of US statehood
	Query For one, they can truthfully say, “Don’t blame me, I didn’t vote for them, ” when discussing the presidency Correct Entities: US
RTE	Text: Dana Reeve, the widow of the actor Christopher Reeve, has died of lung cancer at age 44, according to the Christopher Reeve Foundation.
	Hypothesis: Christopher Reeve had an accident. Entailment: False
WiC	Context 1: Room and board. Context 2: He nailed boards across the windows.
	Sense match: False
WSC	Text: Mark told Pete many lies about himself, which Pete included in his book. He should have been more truthful. Coreference: False

3.1 Design Process

3.1 设计过程

The goal of SuperGLUE is to provide a simple, robust evaluation metric of any method capable of being applied to a broad range of language understanding tasks. To that end, in designing SuperGLUE, we identify the following desiderata of tasks in the benchmark:

SuperGLUE 的目标是：为任何可应用于广泛语言理解任务的方法，提供简单、稳健的评测指标。为此，在设计 SuperGLUE 时，我们确定基准任务应满足以下期望：

Task substance: Tasks should test a system’s ability to understand and reason about texts in English.

任务实质 ： 任务应检验系统理解并推理英语文本的能力。

Task difficulty: Tasks should be beyond the scope of current state-of-the-art systems, but solvable by most college-educated English speakers. We exclude tasks that require domain-specific knowledge, e.g. medical notes or scientific papers.

任务难度 ： 任务应超出当前最优系统能力，但多数受过大学教育的英语使用者可以解决。我们排除需要领域专业知识的任务，例如医嘱或科学论文。

Evaluability: Tasks must have an automatic performance metric that corresponds well to human judgments of output quality. Some text generation tasks fail to meet this criteria due to issues with automatic metrics like ROUGE and BLEU (Callison-Burch et al. 2006; Liu et al. 2016, i.a.).

可评测性 ： 任务须有与人类对输出质量判断良好对应的自动性能指标。部分文本生成任务因 ROUGE、BLEU 等自动指标问题而不满足该条件（Callison-Burch et al. 2006；Liu et al. 2016 等）。

Public data: We require that tasks have existing public training data in order to minimize the risks involved in newly-created datasets. We also prefer tasks for which we have access to (or could create) a test set with private labels.

公开数据 ： 我们要求任务已有 现成 公开训练数据，以降低新建数据集的风险。我们也偏好能获得（或能创建）带私有标签测试集的任务。

Task format: We prefer tasks that had relatively simple input and output formats, to avoid incentivizing the users of the benchmark to create complex task-specific model architectures. Still, while GLUE is restricted to tasks involving single sentence or sentence pair inputs, for SuperGLUE we expand the scope to consider tasks with longer inputs. This yields a set of tasks that requires understanding individual tokens in context, complete sentences, inter-sentence relations, and entire paragraphs.

任务形式 ： 我们偏好输入输出形式相对简单的任务，以免激励基准使用者构建复杂的任务特定模型架构。不过，GLUE 限于单句或句对输入；对 SuperGLUE，我们扩大范围以考虑更长输入的任务。由此得到的任务集合需要理解上下文中的单个词元、完整句子、句间关系以及整段文本。

License: Task data must be available under licences that allow use and redistribution for research purposes.

许可 ： 任务数据须在允许研究用途使用与再分发的许可下可用。

To identify possible tasks for SuperGLUE, we disseminated a public call for task proposals to the NLP community, and received approximately 30 proposals. We filtered these proposals according to our criteria. Many proposals were not suitable due to licensing issues, complex formats, and insufficient headroom; we provide examples of such tasks in Appendix D. For each of the remaining tasks, we ran a BERT-based baseline and a human baseline, and filtered out tasks which were either too challenging for humans without extensive training or too easy for our machine baselines.

为识别 SuperGLUE 的候选任务，我们向 NLP 社区公开征集任务提案，约收到 30 份。我们按上述标准筛选。许多提案因许可问题、形式复杂或提升空间不足而不合适；附录 D 给出此类任务示例。对其余任务，我们跑 BERT 基线与人类基线，并过滤对未受充分训练的人类过难、或对我们的机器基线过易的任务。

3.2 Selected Tasks

3.2 选定任务

Following this process, we arrived at eight tasks to use in SuperGLUE. See Tables 1 and 2 for details and specific examples of each task.

经此流程，我们确定 SuperGLUE 使用八项任务。各任务细节与具体示例见表 1、表 2。

BoolQ (Clark et al. 2019a, Boolean Questions,) is a QA task where each example consists of a short passage and a yes/no question about the passage. The questions are provided anonymously and unsolicited by users of the Google search engine, and afterwards paired with a paragraph from a Wikipedia article containing the answer. Following the original work, we evaluate with accuracy.

BoolQ （Clark et al. 2019a，Boolean Questions）是问答任务：每个示例含短段落与关于该段落的是/否问题。问题由 Google 搜索引擎用户匿名、自发提供，随后与含答案的维基百科段落配对。按原工作，我们用准确率评测。

80\%

80\%

COPA (Roemmele et al. 2011, Choice of Plausible Alternatives,) is a causal reasoning task in which a system is given a premise sentence and must determine either the cause or effect of the premise from two possible choices. All examples are handcrafted and focus on topics from blogs and a photography-related encyclopedia. Following the original work, we evaluate using accuracy.

COPA （Roemmele et al. 2011，Choice of Plausible Alternatives）是因果推理任务：系统给定前提句，须从两个选项中判定前提的原因或结果。全部示例手工编写，主题来自博客与摄影相关百科。按原工作，用准确率评测。

MultiRC (Khashabi et al. 2018, Multi-Sentence Reading Comprehension,) is a QA task where each example consists of a context paragraph, a question about that paragraph, and a list of possible answers. The system must predict which answers are true and which are false. While many QA tasks exist, we use MultiRC because of a number of desirable properties: (i) each question can have multiple possible correct answers, so each question-answer pair must be evaluated independent of other pairs, (ii) the questions are designed such that answering each question requires drawing facts from multiple context sentences, and (iii) the question-answer pair format more closely matches the API of other tasks in SuperGLUE than the more popular span-extractive QA format does. The paragraphs are drawn from seven domains including news, fiction, and historical text. The evaluation metrics are F1 over all answer-options (F1 a) and exact match of each question’s set of answers (EM).

MultiRC （Khashabi et al. 2018，Multi-Sentence Reading Comprehension）是问答任务：每个示例含上下文段落、关于该段落的问题，以及可能答案列表。系统须预测哪些答案为真、哪些为假。虽有许多 QA 任务，我们选用 MultiRC 因其若干可取性质：(i) 每个问题可有多个正确答案，因而每个问答对应独立于其他对应评测；(ii) 问题设计使作答需从多句上下文抽取事实；(iii) 问答对格式比更流行的跨度抽取 QA 格式更贴近 SuperGLUE 其他任务的 API。段落取自新闻、虚构与历史文本等七个领域。评测指标为全部答案选项上的 F1（F1 a ）以及每个问题答案集合的精确匹配（EM）。

ReCoRD (Zhang et al. 2018, Reading Comprehension with Commonsense Reasoning Dataset,) is a multiple-choice QA task. Each example consists of a news article and a Cloze-style question about the article in which one entity is masked out. The system must predict the masked out entity from a list of possible entities in the provided passage, where the same entity may be expressed with multiple different surface forms, which are all considered correct. Articles are from CNN and Daily Mail. We evaluate with max (over all mentions) token-level F1 and exact match (EM).

ReCoRD （Zhang et al. 2018，Reading Comprehension with Commonsense Reasoning Dataset）是多项选择 QA 任务。每个示例含新闻文章与关于该文章的完型填空式问题，其中一个实体被掩码。系统须从段落中可能实体列表预测被掩实体；同一实体可有多种表层形式，均视为正确。文章来自 CNN 与 Daily Mail。我们用（对所有提及取最大的）词元级 F1 与精确匹配（EM）评测。

\sim

RTE （Recognizing Textual Entailment）数据集来自文本蕴含年度竞赛系列。RTE 已纳入 GLUE，我们使用与 GLUE 相同的数据与格式：合并 RTE1（Dagan et al. 2006）、RTE2（Bar Haim et al. 2006）、RTE3（Giampiccolo et al. 2007）与 RTE5（Bentivogli et al. 2009），全部转为两类分类： entailment 与 not_entailment 。在全部 GLUE 任务中，RTE 是受益于迁移学习最多的任务之一：从 GLUE 发布时接近随机（约 56%）跃升到写作时的 86.3% 准确率（Liu et al. 2019d；Yang et al. 2019）。但相对人类表现仍有近八分差距，任务尚未被机器解决，我们预期剩余差距将难以弥合。

WiC (Pilehvar and Camacho-Collados 2019, Word-in-Context,) is a word sense disambiguation task cast as binary classification of sentence pairs. Given two text snippets and a polysemous word that appears in both sentences, the task is to determine whether the word is used with the same sense in both sentences. Sentences are drawn from WordNet (Miller 1995), VerbNet (Schuler 2005), and Wiktionary. We follow the original work and evaluate using accuracy.

WiC （Pilehvar and Camacho-Collados 2019，Word-in-Context）是改写为句对二分类的词义消歧任务。给定两个文本片段与同时出现在两句中的多义词，任务是判定该词在两句中是否同义。句子取自 WordNet（Miller 1995）、VerbNet（Schuler 2005）与 Wiktionary。我们按原工作用准确率评测。

WSC (Levesque et al. 2012, Winograd Schema Challenge,) is a coreference resolution task in which examples consist of a sentence with a pronoun and a list of noun phrases from the sentence. The system must determine the correct referrent of the pronoun from among the provided choices. Winograd schemas are designed to require everyday knowledge and commonsense reasoning to solve.

WSC （Levesque et al. 2012，Winograd Schema Challenge）是指代消解任务：示例含带代词的句子与句中名词短语列表。系统须从所给选项中判定代词的正确先行词。Winograd 模式旨在要求日常知识与常识推理才能解决。

\sim

\sim

3.3 Scoring

3.3 打分

As with GLUE, we seek to give a sense of aggregate system performance over all tasks by averaging scores of all tasks. Lacking a fair criterion with which to weight the contributions of each task to the overall score, we opt for the simple approach of weighing each task equally, and for tasks with multiple metrics, first averaging those metrics to get a task score.

与 GLUE 一样，我们通过对全部任务分数取平均，给出系统在全部任务上的总体表现。缺少公平加权各任务贡献的准则，我们采用简单做法：各任务等权；对有多项指标的任务，先对指标取平均得到任务分数。

3.4 Tools for Model Analysis

3.4 模型分析工具

R_{3}

R_{3}

Analyzing Gender Bias in Models Recent work has identified the presence and amplification of many social biases in data-driven machine learning models (Lu et al. 2018; Zhao et al. 2018, i.a.). To promote the detection of such biases, we include Winogender (Rudinger et al. 2018) as an additional diagnostic dataset. Winogender is designed to measure gender bias in coreference resolution systems. We use the Diverse Natural Language Inference Collection (Poliak et al. 2018) version that casts Winogender as a textual entailment task.Each example consists of a premise sentence with a male or female pronoun and a hypothesis giving a possible antecedent of the pronoun. Examples occur in minimal pairs, where the only difference between an example and its pair is the gender of the pronoun in the premise. Performance on Winogender is measured with accuracy and the gender parity score : the percentage of minimal pairs for which the predictions are the same. A system can trivially obtain a perfect gender parity score by guessing the same class for all examples, so a high gender parity score is meaningless unless accompanied by high accuracy. We collect non-expert annotations to estimate human performance, and observe an accuracy of 99.7% and a gender parity score of 0.99.

分析模型中的性别偏见 近期工作指出数据驱动机器学习模型中存在并放大多种社会偏见（Lu et al. 2018；Zhao et al. 2018 等）。为促进检测此类偏见，我们把 Winogender（Rudinger et al. 2018）作为额外诊断数据集纳入。Winogender 旨在度量指代消解系统中的性别偏见。我们使用 Diverse Natural Language Inference Collection（Poliak et al. 2018）将 Winogender 改写为文本蕴含任务的版本。每个示例含带男性或女性代词的前提句，以及给出代词可能先行词的假设。示例如 最小对 出现，示例与其对之间唯一差别是前提中代词的性别。Winogender 表现用准确率与 性别对等分数 度量：预测相同的最小对百分比。系统可通过猜测同一类对所有示例得到完美性别对等分数，因而高性别对等分数若无高准确率则无意义。我们收集非专家标注以估计人类表现，观察到准确率 99.7%、性别对等分数 0.99。

Like any diagnostic, Winogender has limitations. It offers only positive predictive value: A poor bias score is clear evidence that a model exhibits gender bias, but a good score does not mean that the model is unbiased. More specifically, in the DNC version of the task, a low gender parity score means that a model’s prediction of textual entailment can be changed with a change in pronouns, all else equal. It is plausible that there are forms of bias that are relevant to target tasks of interest, but that do not surface in this setting (Gonen and Goldberg 2019). Also, Winogender does not cover all forms of social bias, or even all forms of gender. For instance, the version of the data used here offers no coverage of gender-neutral they or non-binary pronouns. Despite these limitations, we believe that Winogender’s inclusion is worthwhile in providing a coarse sense of how social biases evolve with model performance and for keeping attention on the social ramifications of NLP models.

与任何诊断一样，Winogender 有局限。它只有正预测价值：偏见分数差是模型存在性别偏见的明确证据，但分数好并不意味着模型无偏。更具体地，在该任务的 DNC 版本中，低性别对等分数意味着仅改变代词即可改变模型的文本蕴含预测。可能存在与目标任务相关、却不在此设定中显现的偏见形式（Gonen and Goldberg 2019）。此外，Winogender 未覆盖全部社会偏见形式，甚至未覆盖全部性别形式。例如，此处所用数据版本未覆盖性别中立的 they 或非二元代词。尽管有这些局限，我们仍认为纳入 Winogender 有价值：它能粗略反映社会偏见如何随模型表现演化，并持续提醒人们注意 NLP 模型的社会影响。

4 Using SuperGLUE

4 使用 SuperGLUE

Software Tools To facilitate using SuperGLUE, we release jiant (Wang et al. 2019b),[^3] a modular software toolkit, built with PyTorch (Paszke et al. 2017), components from AllenNLP (Gardner et al. 2017), and the transformers package.[^4] jiant implements our baselines and supports the evaluation of custom models and training methods on the benchmark tasks. The toolkit includes support for existing popular pretrained models such as OpenAI GPT and BERT, as well as support for multistage and multitask learning of the kind seen in the strongest models on GLUE.

软件工具 为便于使用 SuperGLUE，我们发布 jiant（Wang et al. 2019b），[^3] 基于 PyTorch（Paszke et al. 2017）、AllenNLP（Gardner et al. 2017）组件与 transformers 包[^4] 的模块化软件工具包。jiant 实现我们的基线，并支持在基准任务上评测自定义模型与训练方法。工具包支持 OpenAI GPT、BERT 等流行预训练模型，以及 GLUE 最强模型中常见的多阶段与多任务学习。

Eligibility Any system or method that can produce predictions for the SuperGLUE tasks is eligible for submission to the leaderboard, subject to the data-use and submission frequency policies stated immediately below. There are no restrictions on the type of methods that may be used, and there is no requirement that any form of parameter sharing or shared initialization be used across the tasks in the benchmark. To limit overfitting to the private test data, users are limited to a maximum of two submissions per day and six submissions per month.

资格 任何能为 SuperGLUE 任务产出预测的系统或方法均可提交排行榜，但须遵守下文数据使用与提交频率政策。对方法类型无限制，也不要求跨基准任务使用任何形式的参数共享或共享初始化。为限制对私有测试数据过拟合，用户每天最多提交两次，每月最多六次。

Data Data for the tasks are available for download through the SuperGLUE site and through a download script included with the software toolkit. Each task comes with a standardized training set, development set, and unlabeled test set. Submitted systems may use any public or private data when developing their systems, with a few exceptions: Systems may only use the SuperGLUE-distributed versions of the task datasets, as these use different train/validation/test splits from other public versions in some cases. Systems also may not use the unlabeled test data for the tasks in system development in any way, may not use the structured source data that was used to collect the WiC labels (sense-annotated example sentences from WordNet, VerbNet, and Wiktionary) in any way, and may not build systems that share information across separate test examples in any way.

数据 任务数据可通过 SuperGLUE 站点及软件工具包中的下载脚本获取。每项任务配有标准化训练集、开发集与 无标签 测试集。提交系统在开发时可使用任何公开或私有数据，但有若干例外：系统只能使用 SuperGLUE 分发的任务数据集版本，因某些情形下划分与其他公开版本不同。系统也不得以任何方式将任务无标签测试数据用于系统开发，不得以任何方式使用用于收集 WiC 标签的结构化源数据（WordNet、VerbNet、Wiktionary 的带词义标注例句），且不得构建跨不同 测试 示例共享信息的系统。

To ensure reasonable credit assignment, because we build very directly on prior work, we ask the authors of submitted systems to directly name and cite the specific datasets that they use, including the benchmark datasets . We will enforce this as a requirement for papers to be listed on the leaderboard.

为保证合理署名，因我们直接建立在先前工作之上，我们要求提交系统的作者直接点名并引用所用具体数据集， 包括基准数据集 。这将作为论文列入排行榜的强制要求执行。

5 Experiments

5 实验

5.1 Baselines

5.1 基线

BERT Model Avg BoolQ CB COPA MultiRC ReCoRD RTE WiC WSC AX b AX g Metrics Acc. F1/Acc. Acc. F1 a /EM F1/EM Acc. Acc. Acc. MCC GPS Acc. Most Frequent 47.1 62.3 21.7 / 48.4 50.0 61.1 / 0.3 33.4 / 32.5 50.3 50.0 65.1 0.0 100.0 / 50.0 CBoW 44.3 62.1 49.0 / 71.2 51.6 0.0 / 0.4 14.0 / 13.6 49.7 53.0 65.1 -0.4 100.0 / 50.0 BERT 69.0 77.4 75.7 / 83.6 70.6 70.0 / 24.0 72.0 / 71.3 71.6 69.5 64.3 23.0 97.8 / 51.7 BERT++ 71.5 79.0 84.7 / 90.4 73.8 70.0 / 24.1 72.0 / 71.3 79.0 69.5 64.3 38.0 99.4 / 51.4 Outside Best - 80.4 - / - 84.4 70.4 / * 24.5 * 74.8 / 73.0 82.7 - - - - / - Human (est.) 89.8 89.0 95.8 / 98.9 100.0 81.8* / 51.9* 91.7 / 91.3 93.6 80.0 100.0 77.0 99.3 / 99.7 Table 3: Baseline performance on the SuperGLUE test sets and diagnostics. For CB we report accuracy and macro-average F1. For MultiRC we report F1 on all answer-options and exact match of each question’s set of correct answers. AX b is the broad-coverage diagnostic task, scored using Matthews’ correlation (MCC). AX g is the Winogender diagnostic, scored using accuracy and the gender parity score (GPS). All values are scaled by 100. The Avg column is the overall benchmark score on non-AX* tasks. The bolded numbers reflect the best machine performance on task. *MultiRC has multiple test sets released on a staggered schedule, and these results evaluate on an installation of the test set that is a subset of ours.

Model	Avg	BoolQ	CB	COPA	MultiRC	ReCoRD	RTE	WiC	WSC	AXb	AXg
Metrics		Acc.	F1/Acc.	Acc.	F1a/EM	F1/EM	Acc.	Acc.	Acc.	MCC	GPS	Acc.
Most Frequent	47.1	62.3	21.7 /	48.4	50.0	61.1 /	0.3	33.4 /	32.5	50.3	50.0	65.1	0.0	100.0 /	50.0
CBoW	44.3	62.1	49.0 /	71.2	51.6	0.0 /	0.4	14.0 /	13.6	49.7	53.0	65.1	-0.4	100.0 /	50.0
BERT	69.0	77.4	75.7 /	83.6	70.6	70.0 /	24.0	72.0 /	71.3	71.6	69.5	64.3	23.0	97.8 /	51.7
BERT++	71.5	79.0	84.7 /	90.4	73.8	70.0 /	24.1	72.0 /	71.3	79.0	69.5	64.3	38.0	99.4 /	51.4
Outside Best	-	80.4	- /	-	84.4	70.4 /*	24.5*	74.8 /	73.0	82.7	-	-	-	- /	-
Human (est.)	89.8	89.0	95.8 /	98.9	100.0	81.8* /	51.9*	91.7 /	91.3	93.6	80.0	100.0	77.0	99.3 /	99.7

BERT 模型 Avg BoolQ CB COPA MultiRC ReCoRD RTE WiC WSC AX b AX g 指标 Acc . F1/Acc . Acc . F1 a /EM F1/EM Acc 准确率 Acc . MCC GPS Acc . Most Frequent 47.1 62.3 21.7 / 48.4 50.0 61.1 / 0.3 33.4 / 32.5 50.3 50.0 65.1 0.0 100.0 / 50.0 CBoW 44.3 62.1 49.0 / 71.2 51.6 0.0 / 0.4 14.0 / 13.6 49.7 53.0 65.1 -0.4 100.0 / 50.0 BERT 69.0 77.4 75.7 / 83.6 70.6 70.0 / 24.0 72.0 / 71.3 71.6 69.5 64.3 23.0 97.8 / 51.7 BERT++ 71.5 79.0 84.7 / 90.4 73.8 70.0 / 24.1 72.0 / 71.3 79.0 69.5 64.3 38.0 99.4 / 51.4 Outside Best - 80.4 - / - 84.4 70.4 / * 24.5 * 74.8 / 73.0 82.7 - - - - / - Human (est.) 89.8 89.0 95.8 / 98.9 100.0 81.8* / 51.9* 91.7 / 91.3 93.6 80.0 100.0 77.0 99.3 / 99.7 表 3：基线在 SuperGLUE 测试集与诊断上的表现。对 CB 报告准确率与宏平均 F1。对 MultiRC 报告全部答案选项上的 F1 以及每个问题正确答案集合的精确匹配。AX b 为广覆盖诊断任务，用马修斯相关（MCC）打分。AX g 为 Winogender 诊断，用准确率与性别对等分数（GPS）打分。所有数值乘以 100。 Avg 列为非 AX* 任务上的总体基准分数。粗体数字为该任务最佳机器表现。*MultiRC 以交错时间表发布多个测试集，这些结果评测的是我们测试集的一个子集安装。

模型	Avg	BoolQ	CB	COPA	MultiRC	ReCoRD	RTE	WiC	WSC	AXb	AXg
指标		Acc.	F1/Acc.	Acc.	F1a/EM	F1/EM	Acc	准确率	Acc.	MCC	GPS	Acc.
Most Frequent	47.1	62.3	21.7 /	48.4	50.0	61.1 /	0.3	33.4 /	32.5	50.3	50.0	65.1	0.0	100.0 /	50.0
CBoW	44.3	62.1	49.0 /	71.2	51.6	0.0 /	0.4	14.0 /	13.6	49.7	53.0	65.1	-0.4	100.0 /	50.0
BERT	69.0	77.4	75.7 /	83.6	70.6	70.0 /	24.0	72.0 /	71.3	71.6	69.5	64.3	23.0	97.8 /	51.7
BERT++	71.5	79.0	84.7 /	90.4	73.8	70.0 /	24.1	72.0 /	71.3	79.0	69.5	64.3	38.0	99.4 /	51.4
Outside Best	-	80.4	- /	-	84.4	70.4 /*	24.5*	74.8 /	73.0	82.7	-	-	-	- /	-
Human (est.)	89.8	89.0	95.8 /	98.9	100.0	81.8* /	51.9*	91.7 /	91.3	93.6	80.0	100.0	77.0	99.3 /	99.7

10^{-5}

10^{-5}

For classification tasks with sentence-pair inputs (BoolQ, CB, RTE, WiC), we concatenate the sentences with a [sep] token, feed the fused input to BERT, and use a logistic regression classifier that sees the representation corresponding to [cls]. For WiC, we also concatenate the representation of the marked word. For COPA, MultiRC, and ReCoRD, for each answer choice, we similarly concatenate the context with that answer choice and feed the resulting sequence into BERT to produce an answer representation. For COPA, we project these representations into a scalar, and take as the answer the choice with the highest associated scalar. For MultiRC, because each question can have more than one correct answer, we feed each answer representation into a logistic regression classifier. For ReCoRD, we also evaluate the probability of each candidate independent of other candidates, and take the most likely candidate as the model’s prediction. For WSC, which is a span-based task, we use a model inspired by Tenney et al. 2019. Given the BERT representation for each word in the original sentence, we get span representations of the pronoun and noun phrase via a self-attention span-pooling operator (Lee et al. 2017), before feeding it into a logistic regression classifier.

对句对输入的分类任务（BoolQ、CB、RTE、WiC），我们用 [sep] 标记拼接句子，将融合输入送入 BERT，再用看到 [cls] 对应表示的逻辑回归分类器。对 WiC，还拼接被标记词的表示。对 COPA、MultiRC 与 ReCoRD，对每个答案选项，类似地把上下文与该答案选项拼接，送入 BERT 得到答案表示。对 COPA，把这些表示投影为标量，取关联标量最高的选项为答案。对 MultiRC，因每个问题可有多个正确答案，把每个答案表示送入逻辑回归分类器。对 ReCoRD，也独立于其他候选评估每个候选概率，取最可能候选为模型预测。对跨度式任务 WSC，我们使用受 Tenney et al. 2019 启发的模型：给定原句每个词的 BERT 表示，经自注意力跨度池化算子（Lee et al. 2017）得到代词与名词短语的跨度表示，再送入逻辑回归分类器。

BERT++ We also report results using BERT with additional training on related datasets before fine-tuning on the benchmark tasks, following the STILTs style of transfer learning (Phang et al. 2018). Given the productive use of MultiNLI in pretraining and intermediate fine-tuning of pretrained language models (Conneau et al. 2017; Phang et al. 2018, i.a.), for CB, RTE, and BoolQ, we use MultiNLI as a transfer task by first using the above procedure on MultiNLI. Similarly, given the similarity of COPA to SWAG (Zellers et al. 2018), we first fine-tune BERT on SWAG. These results are reported as BERT++. For all other tasks, we reuse the results of BERT fine-tuned on just that task.

BERT++ 我们还报告在基准任务微调前，先在相关数据集上额外训练 BERT 的结果，遵循 STILTs 式迁移学习（Phang et al. 2018）。鉴于 MultiNLI 在预训练语言模型的预训练与中间微调中被大量使用（Conneau et al. 2017；Phang et al. 2018 等），对 CB、RTE 与 BoolQ，我们先按上述程序在 MultiNLI 上训练作为迁移任务。类似地，鉴于 COPA 与 SWAG（Zellers et al. 2018）相似，我们先在 SWAG 上微调 BERT。这些结果记为 BERT++。对其余任务，复用仅在该任务上微调 BERT 的结果。

Other Baselines We include a baseline where for each task we simply predict the majority class,[^5] as well as a bag-of-words baseline where each input is represented as an average of its tokens’ GloVe word vectors (Pennington et al. 2014, the 300D/840B release from). Finally, we list the best known result on each task as of May 2019, except on tasks which we recast (WSC), resplit (CB), or achieve the best known result (WiC). The outside results for COPA, MultiRC, and RTE are from Sap et al. 2019, Trivedi et al. 2019, and Liu et al. 2019d respectively.

其他基线 我们纳入对每项任务简单预测多数类的基线，[^5] 以及把每个输入表示为词元 GloVe 词向量平均的词袋基线（Pennington et al. 2014，300D/840B 发布）。最后，列出截至 2019 年 5 月各任务已知最佳结果，但排除我们改写（WSC）、重新划分（CB）或取得已知最佳（WiC）的任务。COPA、MultiRC 与 RTE 的外部结果分别来自 Sap et al. 2019、Trivedi et al. 2019 与 Liu et al. 2019d。

Human Performance Pilehvar and Camacho-Collados 2019, Khashabi et al. 2018, Nangia and Bowman 2019, and Zhang et al. 2018 respectively provide estimates for human performance on WiC, MultiRC, RTE, and ReCoRD. For the remaining tasks, including the diagnostic set, we estimate human performance by hiring crowdworker annotators through Amazon’s Mechanical Turk platform to reannotate a sample of each test set. We follow a two step procedure where a crowd worker completes a short training phase before proceeding to the annotation phase, modeled after the method used by Nangia and Bowman 2019 for GLUE. See Appendix C for details.

人类表现 Pilehvar and Camacho-Collados 2019、Khashabi et al. 2018、Nangia and Bowman 2019 与 Zhang et al. 2018 分别提供 WiC、MultiRC、RTE 与 ReCoRD 上的人类表现估计。对其余任务（含诊断集），我们通过 Amazon Mechanical Turk 雇佣众包标注员对测试集样本重新标注来估计人类表现。我们遵循两步流程：众包工人先完成短训练阶段再进入标注阶段，仿照 Nangia and Bowman 2019 在 GLUE 上所用方法。细节见附录 C。

5.2 Results

5.2 结果

Table 3 shows results for all baselines. The most frequent class and CBOW baselines do not perform well overall, achieving near chance performance for several of the tasks. Using BERT increases the average SuperGLUE score by 25 points, attaining significant gains on all of the benchmark tasks, particularly MultiRC, ReCoRD, and RTE. On WSC, BERT actually performs worse than the simple baselines, likely due to the small size of the dataset and the lack of data augmentation. Using MultiNLI as an additional source of supervision for BoolQ, CB, and RTE leads to a 2-5 point improvement on all tasks. Using SWAG as a transfer task for COPA sees an 8 point improvement.

表 3 给出全部基线结果。多数类与 CBOW 基线总体不佳，在若干任务上接近随机。使用 BERT 使平均 SuperGLUE 分数提高约 25 分，在全部基准任务上取得显著收益，尤其是 MultiRC、ReCoRD 与 RTE。在 WSC 上 BERT 实际弱于简单基线，可能因数据集小且缺乏数据增强。用 MultiNLI 作为 BoolQ、CB 与 RTE 的额外监督源，三项任务均提升 2-5 分。用 SWAG 作为 COPA 的迁移任务带来 8 分提升。

Our best baselines still lag substantially behind human performance. On average, there is a nearly 20 point gap between BERT++ and human performance. The largest gap is on WSC, with a 35 point difference between the best model and human performance. The smallest margins are on BoolQ, CB, RTE, and WiC, with gaps of around 10 points on each of these. We believe these gaps will be challenging to close: On WSC and COPA, human performance is perfect. On three other tasks, it is in the mid-to-high 90s. On the diagnostics, all models continue to lag significantly behind humans. Though all models obtain near perfect gender parity scores on Winogender, this is due to the fact that they are obtaining accuracy near that of random guessing.

我们的最佳基线仍大幅落后于人类表现。平均而言，BERT++ 与人类表现相差近 20 分。最大差距在 WSC，最佳模型与人类相差 35 分。最小差距在 BoolQ、CB、RTE 与 WiC，各项约 10 分。我们相信这些差距将难以弥合：在 WSC 与 COPA 上人类表现完美；在另三项任务上处于中高 90 分。在诊断集上，全部模型继续显著落后于人类。尽管全部模型在 Winogender 上获得接近完美的性别对等分数，但这是因为准确率接近随机猜测。

6 Conclusion

6 结论

We present SuperGLUE, a new benchmark for evaluating general-purpose language understanding systems. SuperGLUE updates the GLUE benchmark by identifying a new set of challenging NLU tasks, as measured by the difference between human and machine baselines. The set of eight tasks in our benchmark emphasizes diverse task formats and low-data training data tasks, with nearly half the tasks having fewer than 1k examples and all but one of the tasks having fewer than 10k examples.

我们提出 SuperGLUE：评测通用语言理解系统的新基准。SuperGLUE 通过识别一组更具挑战的 NLU 任务（以人类与机器基线差距度量）更新 GLUE。基准中八项任务强调多样任务形式与低数据训练设定：近半数任务训练例少于 1k，除一项外全部少于 10k。

We evaluate BERT-based baselines and find that they still lag behind humans by nearly 20 points. Given the difficulty of SuperGLUE for BERT, we expect that further progress in multi-task, transfer, and unsupervised/self-supervised learning techniques will be necessary to approach human-level performance on the benchmark. Overall, we argue that SuperGLUE offers a rich and challenging testbed for work developing new general-purpose machine learning methods for language understanding.

我们评测基于 BERT 的基线，发现仍落后人类近 20 分。鉴于 SuperGLUE 对 BERT 的难度，我们预期需要多任务、迁移与无监督/自监督学习技术的进一步发展，才能接近基准上的人类水平。总体而言，我们认为 SuperGLUE 为开发面向语言理解的新通用机器学习方法提供了丰富且具挑战性的试验床。

7 Acknowledgments

7 致谢

We thank the original authors of the included datasets in SuperGLUE for their cooperation in the creation of the benchmark, as well as those who proposed tasks and datasets that we ultimately could not include. This work was made possible in part by a donation to NYU from Eric and Wendy Schmidt made by recommendation of the Schmidt Futures program. We gratefully acknowledge the support of the NVIDIA Corporation with the donation of a Titan V GPU used at NYU for this research, and funding from DeepMind for the hosting of the benchmark platform. AW is supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 1342536. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. This project is partly supported by Samsung Advanced Institute of Technology (Next Generation Deep Learning: from Pattern Recognition to AI) and Samsung Electronics (Improving Deep Learning using Latent Structure).

我们感谢 SuperGLUE 所含数据集的原作者在创建基准过程中的合作，以及提出我们最终未能纳入的任务与数据集的研究者。本工作部分得益于 Eric and Wendy Schmidt 经 Schmidt Futures 项目推荐向 NYU 的捐赠。我们感谢 NVIDIA 公司捐赠用于 NYU 本研究的 Titan V GPU，以及 DeepMind 资助基准平台托管。AW 获美国国家科学基金会研究生研究奖学金计划资助（Grant No. DGE 1342536）。本文材料中的任何意见、发现、结论或建议均为作者观点，不一定反映国家科学基金会的观点。本项目部分得到三星综合技术研究院（Next Generation Deep Learning: from Pattern Recognition to AI）与三星电子（Improving Deep Learning using Latent Structure）支持。

References

参考文献

Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810.12885 , 2018. Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , 2019. URL https://arxiv.org/abs/1904.01130 . Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Association for Computational Linguistics, 2018. doi: 10.18653/v1/N18-2003 . URL https://www.aclweb.org/anthology/N18-2003 .

Appendix A Development Set Results

附录 A 开发集结果

In Table 4, we present results of the baselines on the SuperGLUE tasks development sets.

表 4 给出基线在 SuperGLUE 任务开发集上的结果。

Model Avg BoolQ CB COPA MultiRC ReCoRD RTE WiC WSC Metrics Acc. Acc./F1 Acc. F1 a /EM F1/EM Acc. Acc. Acc. Most Frequent Class 47.7 62.2 50.0 / 22.2 55.0 59.9 / 0.8 32.4 / 31.5 52.7 50.0 63.5 CBOW 47.7 62.4 71.4 / 49.6 63.0 20.3 / 0.3 14.4 / 13.8 54.2 55.3 61.5 BERT 72.2 77.7 94.6 / 93.7 69.0 70.5 / 24.7 70.6 / 69.8 75.8 74.9 68.3 BERT++ 74.6 80.1 96.4 / 95.0 78.0 70.5 / 24.7 70.6 / 69.8 82.3 74.9 68.3 Table 4: Baseline performance on the SuperGLUE development.

Model	Avg	BoolQ	CB	COPA	MultiRC	ReCoRD	RTE	WiC	WSC
Metrics		Acc.	Acc./F1	Acc.	F1a/EM	F1/EM	Acc.	Acc.	Acc.
Most Frequent Class	47.7	62.2	50.0 /	22.2	55.0	59.9 /	0.8	32.4 /	31.5	52.7	50.0	63.5
CBOW	47.7	62.4	71.4 /	49.6	63.0	20.3 /	0.3	14.4 /	13.8	54.2	55.3	61.5
BERT	72.2	77.7	94.6 /	93.7	69.0	70.5 /	24.7	70.6 /	69.8	75.8	74.9	68.3
BERT++	74.6	80.1	96.4 /	95.0	78.0	70.5 /	24.7	70.6 /	69.8	82.3	74.9	68.3

模型 Avg BoolQ CB COPA MultiRC ReCoRD RTE WiC WSC 指标 Acc . Acc./F1 Acc . F1 a /EM F1/EM Acc 准确率 Acc . Most Frequent Class 47.7 62.2 50.0 / 22.2 55.0 59.9 / 0.8 32.4 / 31.5 52.7 50.0 63.5 CBOW 47.7 62.4 71.4 / 49.6 63.0 20.3 / 0.3 14.4 / 13.8 54.2 55.3 61.5 BERT 72.2 77.7 94.6 / 93.7 69.0 70.5 / 24.7 70.6 / 69.8 75.8 74.9 68.3 BERT++ 74.6 80.1 96.4 / 95.0 78.0 70.5 / 24.7 70.6 / 69.8 82.3 74.9 68.3 表 4：基线在 SuperGLUE 开发集上的表现。

模型	Avg	BoolQ	CB	COPA	MultiRC	ReCoRD	RTE	WiC	WSC
指标		Acc.	Acc./F1	Acc.	F1a/EM	F1/EM	Acc	准确率	Acc.
Most Frequent Class	47.7	62.2	50.0 /	22.2	55.0	59.9 /	0.8	32.4 /	31.5	52.7	50.0	63.5
CBOW	47.7	62.4	71.4 /	49.6	63.0	20.3 /	0.3	14.4 /	13.8	54.2	55.3	61.5
BERT	72.2	77.7	94.6 /	93.7	69.0	70.5 /	24.7	70.6 /	69.8	75.8	74.9	68.3
BERT++	74.6	80.1	96.4 /	95.0	78.0	70.5 /	24.7	70.6 /	69.8	82.3	74.9	68.3

Appendix B Performance on GLUE Diagnostics

附录 B GLUE 诊断上的表现

Figure 2 shows the performance on the GLUE diagnostics dataset for systems submitted to the public leaderboard.

图 2 展示公开排行榜提交系统在 GLUE 诊断数据集上的表现。

R_{3}

R_{3}

Appendix C Human Performance Estimation

附录 C 人类表现估计

For collecting data to establish human performance on the SuperGLUE tasks, we follow a two step procedure where we first provide some training to the crowd workers before they proceed to annotation. For both steps and all tasks, the average pay rate is $23.75/hr.[^6]

为收集确立 SuperGLUE 任务人类表现的数据，我们遵循两步流程：先为众包工人提供训练，再进入标注。两步与全部任务的平均时薪为 $23.75/hr。[^6]

In the training phase, workers are provided with instructions on the task, linked to an FAQ page, and are asked to annotate up to 30 examples from the development set. After answering each example, workers are also asked to check their work against the provided ground truth label. After the training phase is complete, we provide the qualification to work on the annotation phase to all workers who annotated a minimum of five examples, i.e. completed five HITs during training and achieved performance at, or above the median performance across all workers during training.

训练阶段，工人获得任务说明、FAQ 链接，并被要求标注开发集上最多 30 个示例。回答每个示例后，工人还需对照所给金标检查自己的答案。训练阶段结束后，我们向标注至少五个示例（即训练期间完成五个 HIT）且表现达到或超过训练期间全部工人中位表现的工人发放标注阶段资格。

In the annotation phase, workers are provided with the same instructions as the training phase, and are linked to the same FAQ page. The instructions for all tasks are provided in Appendix C. For the annotation phase we randomly sample 100 examples from the task’s test set, with the exception of WSC where we annotate the full test set. For each example, we collect annotations from five workers and take a majority vote to estimate human performance. For additional details, see Appendix C.3.

标注阶段，工人获得与训练阶段相同的说明，并链接到同一 FAQ 页。全部任务说明见附录 C。标注阶段我们从任务测试集随机抽样 100 个示例，WSC 除外（标注完整测试集）。对每个示例收集五名工人的标注，取多数票估计人类表现。更多细节见附录 C.3。

C.1 Training Phase Instructions

C.1 训练阶段说明

In the training step, we provide workers with brief instructions about the training phase. An example of these instructions is given Table 5. These training instructions are the same across tasks, only the task name in the instructions is changed.

在训练步骤中，我们向工人提供关于训练阶段的简要说明。表 5 给出示例。这些训练说明跨任务相同，仅说明中的任务名不同。

C.2 Task Instructions

C.2 任务说明

During training and annotation for each task, we provide workers with brief instructions tailored to the task. We also link workers to an FAQ page for the task. Tables 6, 7, 8, and 9, show the instructions we used for all four tasks: COPA, CommitmentBank, WSC, and BoolQ respectively. The instructions given to crowd workers for annotations on the diagnostic and bias diagnostic datasets are shown in Table 11.

各任务训练与标注期间，我们向工人提供针对该任务的简要说明，并链接到任务 FAQ 页。表 6、7、8、9 分别展示 COPA、CommitmentBank、WSC 与 BoolQ 所用说明。诊断与偏见诊断数据集上给众包工人的说明见表 11。

We collected data to produce conservative estimates for human performance on several tasks that we did not ultimately include in our benchmark, including GAP (Webster et al. 2018), PAWS (Zhang et al. 2019), Quora Insincere Questions,[^7] Ultrafine Entity Typing (Choi et al. 2018b), and Empathetic Reactions datasets (Buechel et al. 2018). The instructions we used for these tasks are shown in Tables 12, 13, 14, 15, and 16.

我们还为最终未纳入基准的若干任务收集数据以给出保守的人类表现估计，包括 GAP（Webster et al. 2018）、PAWS（Zhang et al. 2019）、Quora Insincere Questions、[^7] Ultrafine Entity Typing（Choi et al. 2018b）与 Empathetic Reactions 数据集（Buechel et al. 2018）。这些任务所用说明见表 12、13、14、15、16。

C.3 Task Specific Details

C.3 任务特定细节

For WSC and COPA we provide annotators with a two way classification problem. We then use majority vote across annotations to calculate human performance.

对 WSC 与 COPA，我们向标注员提供二分类问题，再跨标注取多数票计算人类表现。

CommitmentBank We follow the authors in providing annotators with a 7-way classification problem. We then collapse the annotations into 3 classes by using the same ranges for bucketing used by de Marneffe et al. 2019. We then use majority vote to get human performance numbers on the task.

CommitmentBank 我们按作者做法向标注员提供七分类问题，再用 de Marneffe et al. 2019 所用相同分桶区间把标注折叠为三类，然后取多数票得到任务上的人类表现数字。

Furthermore, for training on CommitmentBank we randomly sample examples from the low inter-annotator agreement portion of the CommitmentBank data that is not included in the benchmark version of the task. These low agreement examples are generally harder to classify since they are more ambiguous.

此外，CommitmentBank 训练阶段我们从 CommitmentBank 数据中标注者间一致率较低、未纳入基准版本的部分随机抽样示例。这些低一致率示例通常更难分类，因为更含糊。

Diagnostic Dataset Since the diagnostic dataset does not come with accompanying training data, we train our workers on examples from RTE’s development set. RTE is also a textual entailment task and is the most closely related task in the main benchmark. Providing the crowd workers with training on RTE enables them to learn label definitions which should generalize to the diagnostic dataset.

诊断数据集 因诊断数据集不附带训练数据，我们在 RTE 开发集示例上训练工人。RTE 也是文本蕴含任务，且是主基准中最密切相关的任务。在 RTE 上训练使众包工人能学习可泛化到诊断数据集的标签定义。

Ultrafine Entity Typing We cast the task into a binary classification problem to make it an easier task for non-expert crowd workers. We work in cooperation with the authors of the dataset (Choi et al. 2018b) to do this reformulation: We give workers one possible tag for a word or phrase and asked them to classify the tag as being applicable or not.

Ultrafine Entity Typing 我们把任务改写为二分类，以便非专家众包工人更易完成。我们与数据集作者（Choi et al. 2018b）合作完成改写：给工人一个词或短语的可能标签，要求判定该标签是否适用。

The authors used WordNet (Miller 1995) to expand the set of labels to include synonyms and hypernyms from WordNet. They then asked five annotators to validate these tags. The tags from this validation had high agreement, and were included in the publicly available Ultrafine Entity Typing dataset,[^8] This constitutes our set of positive examples. The rest of the tags from the validation procedure that are not in the public dataset constitute our negative examples.

作者用 WordNet（Miller 1995）扩展标签集以纳入同义词与上位词，再请五名标注员验证这些标签。验证中一致率高的标签被纳入公开 Ultrafine Entity Typing 数据集，[^8] 构成我们的正例集合。验证过程中未进入公开数据集的其余标签构成负例。

GAP For the Gendered Ambiguous Pronoun Coreference task (Webster et al. 2018, GAP,), we simplified the task by providing noun phrase spans as part of the input, thus reducing the original structure prediction task to a classification task. This task was presented to crowd workers as a three way classification problem: Choose span A, B, or neither.

GAP 对 Gendered Ambiguous Pronoun Coreference 任务（Webster et al. 2018，GAP），我们把名词短语跨度作为输入的一部分提供，从而把原结构预测任务简化为分类任务。该任务以三分类呈现给众包工人：选择跨度 A、B 或都不选。

Appendix D Excluded Tasks

附录 D 未纳入任务

In this section we provide some examples of tasks that we evaluated for inclusion but ultimately could not include. We report on these excluded tasks only with the permission of their authors. We turned down many medical text datasets because they are usually only accessible with explicit permission and credentials from the data owners.

本节给出我们评估纳入但最终未能纳入的若干任务示例。仅在作者许可下报告这些未纳入任务。我们拒绝了许多医学文本数据集，因为通常只有在数据所有者明确许可与凭证下才能访问。

Tasks like QuAC (Choi et al. 2018a) and STREUSLE (Schneider and Smith 2015) differed substantially from the format of other tasks in our benchmark, which we worried would incentivize users to spend significant effort on task-specific model designs, rather than focusing on general-purpose techniques. It was challenging to train annotators to do well on Quora Insincere Questions [^9], Empathetic Reactions (Buechel et al. 2018), and a recast version of Ultra-Fine Entity Typing (Choi et al. 2018b, see Appendix C.3 for details), leading to low human performance. BERT achieved very high or superhuman performance on Query Well-Formedness (Faruqui and Das 2018), PAWS (Zhang et al. 2019), Discovering Ongoing Conversations (Zanzotto and Ferrone 2017), and GAP (Webster et al. 2018).

QuAC（Choi et al. 2018a）与 STREUSLE（Schneider and Smith 2015）等任务与我们基准中其他任务形式差异很大，我们担心这会激励使用者把大量精力花在任务特定模型设计上，而非通用技术。在 Quora Insincere Questions、[^9] Empathetic Reactions（Buechel et al. 2018）以及改写版 Ultra-Fine Entity Typing（Choi et al. 2018b，细节见附录 C.3）上训练标注员表现良好很困难，导致人类表现偏低。BERT 在 Query Well-Formedness（Faruqui and Das 2018）、PAWS（Zhang et al. 2019）、Discovering Ongoing Conversations（Zanzotto and Ferrone 2017）与 GAP（Webster et al. 2018）上达到很高或超人类表现。

During the process of selecting tasks for our benchmark, we collected human performance baselines and run BERT-based machine baselines for some tasks that we ultimately excluded from our task list. We chose to exclude these tasks because our BERT baseline performs better than our human performance baseline or if the gap between human and machine performance is small.

在为基准筛选任务过程中，我们对最终从任务列表中排除的若干任务收集了人类表现基线并跑了基于 BERT 的机器基线。排除这些任务，是因为我们的 BERT 基线优于人类表现基线，或人机差距过小。

On Quora Insincere Questions our BERT baseline outperforms our human baseline by a small margin: an F1 score of 67.2 versus 66.7 for BERT and human baselines respectively. Similarly, on the Empathetic Reactions dataset, BERT outperforms our human baseline, where BERT’s predictions have a Pearson correlation of 0.45 on empathy and 0.55 on distress, compared to 0.45 and 0.35 for our human baseline. For PAWS-Wiki, we report that BERT achieves an accuracy of 91.9%, while our human baseline achieved 84% accuracy. These three tasks are excluded from the benchmark since our, admittedly conservative, human baselines are worse than machine performance. Our human performance baselines are subject to the clarity of our instructions (all instructions can be found in Appendix C), and crowd workers engagement and ability.

在 Quora Insincere Questions 上，我们的 BERT 基线以微小优势超过人类基线：F1 分别为 67.2 与 66.7。类似地，在 Empathetic Reactions 数据集上 BERT 也超过人类基线：BERT 预测与共情、痛苦的皮尔逊相关分别为 0.45 与 0.55，人类基线为 0.45 与 0.35。对 PAWS-Wiki，BERT 准确率 91.9%，我们的人类基线为 84%。这三项任务因我们（承认偏保守的）人类基线弱于机器表现而被排除。我们的人类表现基线受说明清晰度（全部说明见附录 C）以及众包工人投入与能力影响。

For the Query Well-Formedness task, the authors set an estimate human performance at 88.4% accuracy. Our BERT baseline model reaches an accuracy of 82.3%. While there is a positive gap on this task, the gap was smaller than we were were willing to tolerate. Similarly, on our recast version of the Ultrafine Entity Typing, we observe too small a gap between human (60.2 F1) and machine performance (55.0 F1). Our recasting for this task is described in Appendix C.2. On GAP, when taken as a classification problem without the related task of span selection (details in C.2), BERT performs (91.0 F1) comparably to our human baseline (94.9 F1). Given this small margin, we also exclude GAP.

对 Query Well-Formedness 任务，作者估计人类表现为 88.4% 准确率；我们的 BERT 基线达到 82.3%。虽有正差距，但小于我们愿意容忍的差距。类似地，在改写版 Ultrafine Entity Typing 上，人类（60.2 F1）与机器（55.0 F1）差距也过小。该任务改写见附录 C.2。对 GAP，在不做相关跨度选择任务、仅作分类时（细节见 C.2），BERT（91.0 F1）与人类基线（94.9 F1）相当；鉴于差距很小，我们也排除 GAP。

On Discovering Ongoing Conversations, our BERT baseline achieves an F1 of 51.9 on a version of the task cast as sentence pair classification (given two snippets of texts from plays, determine if the second snippet is a continuation of the first). This dataset is very class imbalanced (90% negative), so we also experimented with a class-balanced version on which our BERT baselines achieves 88.4 F1. Qualitatively, we also found the task challenging for humans as there was little context for the text snippets and the examples were drawn from plays using early English. Given this fairly high machine performance and challenging nature for humans, we exclude this task from our benchmark.

在 Discovering Ongoing Conversations 上，我们的 BERT 基线在改写为句对分类的版本上（给定剧本中两段文本，判定第二段是否为第一段的续写）达到 F1 51.9。该数据集类别极不平衡（90% 负例），因此我们也实验了类别平衡版本，BERT 基线达到 88.4 F1。定性上我们也发现任务对人类具有挑战性：文本片段上下文很少，且示例取自使用早期英语的剧本。鉴于机器表现相当高且对人类具有挑战性，我们从基准中排除该任务。

Instructions tables begin on the following page.

说明表格见下一页起。

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help! This project is a training task that needs to be completed before working on the main project on AMT named Human Performance: Plausible Answer. Once you are done with the training, please proceed to the main task! The qualification approval is not immediate but we will add you to our qualified workers list within a day. In this training, you must answer the question on the page and then, to see how you did, click the Check Work button at the bottom of the page before hitting Submit. The Check Work button will reveal the true label. Please use this training and the provided answers to build an understanding of what the answers to these questions look like (the main project, Human Performance: Plausible Answer, does not have the answers on the page). Table 5: The instructions given to crowd-sourced worker describing the training phase for the Choice of Plausible Answers (COPA) task.

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help!

This project is a training task that needs to be completed before working on the main project on AMT named Human Performance: Plausible Answer. Once you are done with the training, please proceed to the main task! The qualification approval is not immediate but we will add you to our qualified workers list within a day.

In this training, you must answer the question on the page and then, to see how you did, click the Check Work button at the bottom of the page before hitting Submit. The Check Work button will reveal the true label. Please use this training and the provided answers to build an understanding of what the answers to these questions look like (the main project, Human Performance: Plausible Answer, does not have the answers on the page).

纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！ 本项目是一项 训练任务 ，须在参与 AMT 主项目「Human Performance: Plausible Answer」之前完成。完成训练后，请继续主任务！资格审批不会立即生效，但我们会在一天内将您加入合格工人名单。 在本训练中，您必须回答页面上的问题，然后在点击提交前，先点击页面底部的 Check Work 按钮查看您的表现。Check Work 按钮会显示真实标签。请利用本次训练与所给答案，理解这类问题的答案应是什么样的（主项目 Human Performance: Plausible Answer 页面上不提供答案）。 表 5：向众包工人说明 Choice of Plausible Answers（COPA）任务训练阶段的指示。

纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！

本项目是一项训练任务，须在参与 AMT 主项目「Human Performance: Plausible Answer」之前完成。完成训练后，请继续主任务！资格审批不会立即生效，但我们会在一天内将您加入合格工人名单。

在本训练中，您必须回答页面上的问题，然后在点击提交前，先点击页面底部的 Check Work 按钮查看您的表现。Check Work 按钮会显示真实标签。请利用本次训练与所给答案，理解这类问题的答案应是什么样的（主项目 Human Performance: Plausible Answer 页面上不提供答案）。

Plausible Answer Instructions The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help! We will present you with a prompt sentence and a question. The question will either be about what caused the situation described in the prompt, or what a possible effect of that situation is. We will also give you two possible answers to this question. Your job is to decide, given the situation described in the prompt, which of the two options is a more plausible answer to the question: In the following example, option 1. is a more plausible answer to the question about what caused the situation described in the prompt, The girl received a trophy. What’s the CAUSE for this? 1. She won a spelling bee. 2. She made a new friend. In the following example, option 2. is a more plausible answer the question about what happened because of the situation described in the prompt, The police aimed their weapons at the fugitive. What happened as a RESULT? 1. The fugitive fell to the ground. 2. The fugitive dropped his gun. If you have any more questions, please refer to our FAQ page. Table 6: Task-specific instructions for Choice of Plausible Alternatives (COPA). These instructions were provided during both training and annotation phases.

Plausible Answer Instructions

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help!

We will present you with a prompt sentence and a question. The question will either be about what caused the situation described in the prompt, or what a possible effect of that situation is. We will also give you two possible answers to this question. Your job is to decide, given the situation described in the prompt, which of the two options is a more plausible answer to the question:

In the following example, option 1. is a more plausible answer to the question about what caused the situation described in the prompt,
The girl received a trophy. What’s the CAUSE for this? 1. She won a spelling bee. 2. She made a new friend. In the following example, option 2. is a more plausible answer the question about what happened because of the situation described in the prompt, The police aimed their weapons at the fugitive. What happened as a RESULT? 1. The fugitive fell to the ground. 2. The fugitive dropped his gun. If you have any more questions, please refer to our FAQ page.

合理答案说明 纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！ 我们会给您一个提示句和一个问题。问题要么关于提示所描述情境的原因，要么关于该情境的可能结果。我们还会给出该问题的两个可能答案。您的任务是：在提示所描述情境下，判断哪个选项是对该问题更合理的回答： 在下列例子中，选项 1 是关于提示情境原因的更合理答案， The girl received a trophy. What’s the CAUSE for this? 1. She won a spelling bee. 2. She made a new friend. 在下列例子中，选项 2 是关于提示情境结果的更合理答案， The police aimed their weapons at the fugitive. What happened as a RESULT? 1. The fugitive fell to the ground. 2. The fugitive dropped his gun. 如有更多问题，请参阅 FAQ 页。 表 6：Choice of Plausible Alternatives（COPA）的任务特定说明。训练与标注阶段均提供。

合理答案说明

纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！

我们会给您一个提示句和一个问题。问题要么关于提示所描述情境的原因，要么关于该情境的可能结果。我们还会给出该问题的两个可能答案。您的任务是：在提示所描述情境下，判断哪个选项是对该问题更合理的回答：

在下列例子中，选项 1 是关于提示情境原因的更合理答案，
The girl received a trophy. What’s the CAUSE for this? 1. She won a spelling bee. 2. She made a new friend. 在下列例子中，选项 2 是关于提示情境结果的更合理答案， The police aimed their weapons at the fugitive. What happened as a RESULT? 1. The fugitive fell to the ground. 2. The fugitive dropped his gun. 如有更多问题，请参阅 FAQ 页。

Speaker Commitment Instructions The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help! We will present you with a prompt taken from a piece of dialogue, this could be a single sentence, a few sentences, or a short exchange between people. Your job is to figure out, based on this first prompt (on top), how certain the speaker is about the truthfulness of the second prompt (on the bottom). You can choose from a 7 point scale ranging from (1) completely certain that the second prompt is true to (7) completely certain that the second prompt is false. Here are examples for a few of the labels: Choose 1 (certain that it is true) if the speaker from the first prompt definitely believes or knows that the second prompt is true. For example, "What fun to hear Artemis laugh. She’s such a serious child. I didn’t know she had a sense of humor." "Artemis had a sense of humor" Choose 4 (not certain if it is true or false) if the speaker from the first prompt is uncertain if the second prompt is true or false. For example, "Tess is committed to track. She’s always trained with all her heart and soul. One can only hope that she has recovered from the flu and will cross the finish line." "Tess crossed the finish line." Choose 7 (certain that it is false) if the speaker from the first prompt definitely believes or knows that the second prompt is false. For example, "Did you hear about Olivia’s chemistry test? She studied really hard. But even after putting in all that time and energy, she didn’t manage to pass the test". "Olivia passed the test." If you have any more questions, please refer to our FAQ page. Table 7: Task-specific instructions for Commitment Bank. These instructions were provided during both training and annotation phases.

Speaker Commitment Instructions

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help!

We will present you with a prompt taken from a piece of dialogue, this could be a single sentence, a few sentences, or a short exchange between people. Your job is to figure out, based on this first prompt (on top), how certain the speaker is about the truthfulness of the second prompt (on the bottom). You can choose from a 7 point scale ranging from (1) completely certain that the second prompt is true to (7) completely certain that the second prompt is false. Here are examples for a few of the labels:

Choose 1 (certain that it is true) if the speaker from the first prompt definitely believes or knows that the second prompt is true. For example, "What fun to hear Artemis laugh. She’s such a serious child. I didn’t know she had a sense of humor." "Artemis had a sense of humor" Choose 4 (not certain if it is true or false) if the speaker from the first prompt is uncertain if the second prompt is true or false. For example, "Tess is committed to track. She’s always trained with all her heart and soul. One can only hope that she has recovered from the flu and will cross the finish line." "Tess crossed the finish line." Choose 7 (certain that it is false) if the speaker from the first prompt definitely believes or knows that the second prompt is false. For example, "Did you hear about Olivia’s chemistry test? She studied really hard. But even after putting in all that time and energy, she didn’t manage to pass the test". "Olivia passed the test." If you have any more questions, please refer to our FAQ page.

说话人承诺说明 纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！ 我们会给您取自对话的一段提示，可能是单句、几句话，或人与人之间的简短交流。您的任务是：根据第一段提示（上方），判断说话人对第二段提示（下方）真实性的确定程度。您可在 7 点量表上选择，从 (1) 完全确定第二段提示为真，到 (7) 完全确定第二段提示为假。以下是若干标签示例： 若第一段提示中的说话人明确相信或知道第二段提示为真，选 1（确定为真）。例如， "What fun to hear Artemis laugh. She’s such a serious child. I didn’t know she had a sense of humor." "Artemis had a sense of humor" 若第一段提示中的说话人不确定第二段提示真假，选 4（不确定真假）。例如， "Tess is committed to track. She’s always trained with all her heart and soul. One can only hope that she has recovered from the flu and will cross the finish line." "Tess crossed the finish line." 若第一段提示中的说话人明确相信或知道第二段提示为假，选 7（确定为假）。例如， "Did you hear about Olivia’s chemistry test? She studied really hard. But even after putting in all that time and energy, she didn’t manage to pass the test". "Olivia passed the test." 如有更多问题，请参阅 FAQ 页。 表 7：Commitment Bank 的任务特定说明。训练与标注阶段均提供。

说话人承诺说明

纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！

我们会给您取自对话的一段提示，可能是单句、几句话，或人与人之间的简短交流。您的任务是：根据第一段提示（上方），判断说话人对第二段提示（下方）真实性的确定程度。您可在 7 点量表上选择，从 (1) 完全确定第二段提示为真，到 (7) 完全确定第二段提示为假。以下是若干标签示例：

若第一段提示中的说话人明确相信或知道第二段提示为真，选 1（确定为真）。例如，"What fun to hear Artemis laugh. She’s such a serious child. I didn’t know she had a sense of humor." "Artemis had a sense of humor" 若第一段提示中的说话人不确定第二段提示真假，选 4（不确定真假）。例如，"Tess is committed to track. She’s always trained with all her heart and soul. One can only hope that she has recovered from the flu and will cross the finish line." "Tess crossed the finish line." 若第一段提示中的说话人明确相信或知道第二段提示为假，选 7（确定为假）。例如，"Did you hear about Olivia’s chemistry test? She studied really hard. But even after putting in all that time and energy, she didn’t manage to pass the test". "Olivia passed the test." 如有更多问题，请参阅 FAQ 页。

Winograd Schema Instructions The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help! We will present you with a sentence that someone wrote, with one bolded pronoun. We will then ask if you if the pronoun refers to a specific word or phrase in the sentence. Your job is to figure out, based on the sentence, if the bolded pronoun refers to this selected word or phrase: Choose Yes if the pronoun refers to the selected word or phrase. For example, "I put the cake away in the refrigerator. It has a lot of butter in it." Does It in "It has a lot" refer to cake ? Choose No if the pronoun does not refer to the selected word or phrase. For example, "The large ball crashed right through the table because it was made of styrofoam." Does it in "it was made" refer to ball ? If you have any more questions, please refer to our FAQ page. Table 8: Task-specific instructions for Winograd Schema Challenge (WSC). These instructions were provided during both training and annotation phases.

Winograd Schema Instructions

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help!

We will present you with a sentence that someone wrote, with one bolded pronoun. We will then ask if you if the pronoun refers to a specific word or phrase in the sentence. Your job is to figure out, based on the sentence, if the bolded pronoun refers to this selected word or phrase:

Choose Yes if the pronoun refers to the selected word or phrase. For example, "I put the cake away in the refrigerator. It has a lot of butter in it." Does It in "It has a lot" refer to cake? Choose No if the pronoun does not refer to the selected word or phrase. For example, "The large ball crashed right through the table because it was made of styrofoam." Does it in "it was made" refer to ball? If you have any more questions, please refer to our FAQ page.

Winograd 模式说明 纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！ 我们会给您某人写的一句话，其中有一个加粗代词。然后我们会问该代词是否指代句中某个特定词或短语。您的任务是：根据句子判断，加粗代词是否指代这个被选中的词或短语： 若代词指代被选词或短语，选 Yes。例如， "I put the cake away in the refrigerator. It has a lot of butter in it." Does It in "It has a lot" refer to cake ? 若代词不指代被选词或短语，选 No。例如， "The large ball crashed right through the table because it was made of styrofoam." Does it in "it was made" refer to ball ? 如有更多问题，请参阅 FAQ 页。 表 8：Winograd Schema Challenge（WSC）的任务特定说明。训练与标注阶段均提供。

Winograd 模式说明

纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！

我们会给您某人写的一句话，其中有一个加粗代词。然后我们会问该代词是否指代句中某个特定词或短语。您的任务是：根据句子判断，加粗代词是否指代这个被选中的词或短语：

若代词指代被选词或短语，选 Yes。例如，"I put the cake away in the refrigerator. It has a lot of butter in it." Does It in "It has a lot" refer to cake? 若代词不指代被选词或短语，选 No。例如，"The large ball crashed right through the table because it was made of styrofoam." Does it in "it was made" refer to ball? 如有更多问题，请参阅 FAQ 页。

Question-Answering Instructions

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help!

We will present you with a passage taken from a Wikipedia article and a relevant question. Your job is to decide, given the information provided in the passage, if the answer to the question is Yes or No. For example,

In the following examples the correct answer is Yes, The thirteenth season of Criminal Minds was ordered on April 7, 2017, by CBS with an order of 22 episodes. The season premiered on September 27, 2017 in a new time slot at 10:00PM on Wednesday when it had previously been at 9:00PM on Wednesday since its inception. The season concluded on April 18, 2018 with a two-part season finale. will there be a 13th season of criminal minds? (In the above example, the first line of the passage says that the 13th season of the show was ordered.) As of 8 August 2016, the FDA extended its regulatory power to include e-cigarettes. Under this ruling the FDA will evaluate certain issues, including ingredients, product features and health risks, as well their appeal to minors and non-users. The FDA rule also bans access to minors. A photo ID is required to buy e-cigarettes, and their sale in all-ages vending machines is not permitted. The FDA in September 2016 has sent warning letters for unlawful underage sales to online retailers and retailers of e-cigarettes. is vaping illegal if you are under 18? (In the above example, the passage states that the "FDA rule also bans access to minors." The question uses the word "vaping," which is a synonym for e-cigrattes.) In the following examples the correct answer is No, Badgers are short-legged omnivores in the family Mustelidae, which also includes the otters, polecats, weasels, and wolverines. They belong to the caniform suborder of carnivoran mammals. The 11 species of badgers are grouped in three subfamilies: Melinae (Eurasian badgers), Mellivorinae (the honey badger or ratel), and Taxideinae (the American badger). The Asiatic stink badgers of the genus Mydaus were formerly included within Melinae (and thus Mustelidae), but recent genetic evidence indicates these are actually members of the skunk family, placing them in the taxonomic family Mephitidae. is a wolverine the same as a badger? (In the above example, the passage says that badgers and wolverines are in the same family, Mustelidae, which does not mean they are the same animal.)

问答说明

纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！

我们会给您取自维基百科文章的段落以及相关问题。您的任务是：根据段落所给信息，判断问题答案是 Yes 还是 No。例如，

下列例子正确答案为 Yes， The thirteenth season of Criminal Minds was ordered on April 7, 2017, by CBS with an order of 22 episodes. The season premiered on September 27, 2017 in a new time slot at 10:00PM on Wednesday when it had previously been at 9:00PM on Wednesday since its inception. The season concluded on April 18, 2018 with a two-part season finale. will there be a 13th season of criminal minds? （上例中，段落首行写明该剧第 13 季已获订购。） As of 8 August 2016, the FDA extended its regulatory power to include e-cigarettes. Under this ruling the FDA will evaluate certain issues, including ingredients, product features and health risks, as well their appeal to minors and non-users. The FDA rule also bans access to minors. A photo ID is required to buy e-cigarettes, and their sale in all-ages vending machines is not permitted. The FDA in September 2016 has sent warning letters for unlawful underage sales to online retailers and retailers of e-cigarettes. is vaping illegal if you are under 18? （上例中，段落写明 “FDA rule also bans access to minors。” 问题使用 “vaping”，与电子烟同义。）下列例子正确答案为 No， Badgers are short-legged omnivores in the family Mustelidae, which also includes the otters, polecats, weasels, and wolverines. They belong to the caniform suborder of carnivoran mammals. The 11 species of badgers are grouped in three subfamilies: Melinae (Eurasian badgers), Mellivorinae (the honey badger or ratel), and Taxideinae (the American badger). The Asiatic stink badgers of the genus Mydaus were formerly included within Melinae (and thus Mustelidae), but recent genetic evidence indicates these are actually members of the skunk family, placing them in the taxonomic family Mephitidae. is a wolverine the same as a badger? （上例中，段落说獾与狼獾同属 Mustelidae，并不意味着它们是同一种动物。）

More famously, Harley-Davidson attempted to register as a trademark the distinctive “chug” of a Harley-Davidson motorcycle engine. On February 1, 1994, the company filed its application with the following description: “The mark consists of the exhaust sound of applicant’s motorcycles, produced by V-twin, common crankpin motorcycle engines when the goods are in use.” Nine of Harley-Davidson’s competitors filed oppositions against the application, arguing that cruiser-style motorcycles of various brands use the same crankpin V-twin engine which produces the same sound. After six years of litigation, with no end in sight, in early 2000, Harley-Davidson withdrew their application. does harley davidson have a patent on their sound? (In the above example, the passage states that Harley-Davidson applied for a patent but then withdrew, so they do not have a patent on the sound.) If you have any more questions, please refer to our FAQ page. Table 10: Continuation from Table 9 of task-specific instructions for BoolQ. These instructions were provided during both training and annotation phases.

More famously, Harley-Davidson attempted to register as a trademark the distinctive “chug” of a Harley-Davidson motorcycle engine. On February 1, 1994, the company filed its application with the following description: “The mark consists of the exhaust sound of applicant’s motorcycles, produced by V-twin, common crankpin motorcycle engines when the goods are in use.” Nine of Harley-Davidson’s competitors filed oppositions against the application, arguing that cruiser-style motorcycles of various brands use the same crankpin V-twin engine which produces the same sound. After six years of litigation, with no end in sight, in early 2000, Harley-Davidson withdrew their application. does harley davidson have a patent on their sound? （上例中，段落写明 Harley-Davidson 曾申请专利后又撤回，因此他们对该声音没有专利。）如有更多问题，请参阅 FAQ 页。 表 10：表 9 中 BoolQ 任务特定说明的续篇。训练与标注阶段均提供。

Textual Entailment Instructions

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help!

We will present you with a prompt taken from an article someone wrote. Your job is to figure out, based on this correct prompt (the first prompt, on top), if another prompt (the second prompt, on bottom) is also necessarily true:

Choose True if the event or situation described by the first prompt definitely implies that the second prompt, on bottom, must also be true. For example, • "Murphy recently decided to move to London." "Murphy recently decided to move to England." (The above example is True because London is in England and therefore prompt 2 is clearly implied by prompt 1.) • "Russian cosmonaut Valery Polyakov set the record for the longest continuous amount of time spent in space, a staggering 438 days, between 1994 and 1995." "Russians hold record for longest stay in space." (The above example is True because the information in the second prompt is contained in the first prompt: Valery is Russian and she set the record for longest stay in space.) • "She does not disgree with her brother’s opinion, but she believes he’s too aggresive in his defense" "She agrees with her brother’s opinion, but she believes he’s too aggresive in his defense" (The above example is True because the second prompt is an exact paraphrase of the first prompt, with exactly the same meaning.) Choose False if the event or situation described with the first prompt on top does not necessarily imply that this second prompt must also be true. For example, • "This method was developed at Columbia and applied to data processing at CERN." "This method was developed at Columbia and applied to data processing at CERN with limited success." (The above example is False because the second prompt is introducing new information not implied in the first prompt: The first prompt does not give us any knowledge of how succesful the application of the method at CERN was.) • "This building is very tall." "This is the tallest building in New York." (The above example is False because a building being tall does not mean it must be the tallest building, nor that it is in New York.) • "Hours earlier, Yasser Arafat called for an end to attacks against Israeli civilians in the two weeks before Israeli elections." "Arafat condemned suicide bomb attacks inside Israel." (The above example is False because from the first prompt we only know that Arafat called for an end to attacks against Israeli citizens, we do not know what kind of attacks he may have been condemning.) You do not have to worry about whether the writing style is maintained between the two prompts.

If you have any more questions, please refer to our FAQ page.

文本蕴含说明

纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！

我们会给您取自某篇文章的提示。您的任务是：根据这段正确的提示（第一段，上方），判断另一段提示（第二段，下方）是否也必然为真：

若第一段提示所描述的事件或情境明确意味着下方第二段提示也必然为真，选 True。例如，• "Murphy recently decided to move to London." "Murphy recently decided to move to England." （上例为 True，因为伦敦在英格兰，因此提示 2 显然由提示 1 蕴含。） • "Russian cosmonaut Valery Polyakov set the record for the longest continuous amount of time spent in space, a staggering 438 days, between 1994 and 1995." "Russians hold record for longest stay in space." （上例为 True，因为第二段信息已包含在第一段中：Valery 是俄罗斯人，且她创下最长停留太空纪录。） • "She does not disgree with her brother’s opinion, but she believes he’s too aggresive in his defense" "She agrees with her brother’s opinion, but she believes he’s too aggresive in his defense" （上例为 True，因为第二段是第一段的精确改写，意义完全相同。）若上方第一段提示所描述的事件或情境并不必然意味着第二段也必然为真，选 False。例如，• "This method was developed at Columbia and applied to data processing at CERN." "This method was developed at Columbia and applied to data processing at CERN with limited success." （上例为 False，因为第二段引入了第一段未蕴含的新信息：第一段未告诉我们该方法在 CERN 的应用有多成功。） • "This building is very tall." "This is the tallest building in New York." （上例为 False，因为建筑物很高并不意味着它必须是最高的，也不意味着它在纽约。） • "Hours earlier, Yasser Arafat called for an end to attacks against Israeli civilians in the two weeks before Israeli elections." "Arafat condemned suicide bomb attacks inside Israel." （上例为 False，因为从第一段我们只知道 Arafat 呼吁停止针对以色列平民的袭击，并不知道他可能在谴责何种袭击。）您无需担心两段提示之间是否保持相同写作风格。

如有更多问题，请参阅 FAQ 页。

GAP Instructions

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help!

We will present you with an extract from a Wikipedia article, with one bolded pronoun. We will also give you two names from the text that this pronoun could refer to. Your job is to figure out, based on the extract, if the pronoun refers to option A, options B, or neither:

Choose A if the pronoun refers to option A. For example, "In 2010 Ella Kabambe was not the official Miss Malawi; this was Faith Chibale, but Kabambe represented the country in the Miss World pageant. At the 2012 Miss World, Susan Mtegha pushed Miss New Zealand, Collette Lochore, during the opening headshot of the pageant, claiming that Miss New Zealand was in her space." Does her* refer to option A or B below?* A Susan Mtegha B Collette Lochore C Neither Choose B if the pronoun refers to option B. For example, "In 1650 he started his career as advisor in the ministerium of finances in Den Haag. After he became a minister he went back to Amsterdam, and took place as a sort of chairing mayor of this city. After the death of his brother Cornelis, De Graeff became the strong leader of the republicans. He held this position until the rampjaar." Does He refer to option A or B below? A Cornelis B De Graeff C Neither Choose C if the pronoun refers to neither option. For example, "Reb Chaim Yaakov’s wife is the sister of Rabbi Moishe Sternbuch, as is the wife of Rabbi Meshulam Dovid Soloveitchik, making the two Rabbis his uncles. Reb Asher’s brother Rabbi Shlomo Arieli is the author of a critical edition of the novallae of Rabbi Akiva Eiger. Before his marriage, Rabbi Arieli studied in the Ponevezh Yeshiva headed by Rabbi Shmuel Rozovsky, and he later studied under his father-in-law in the Mirrer Yeshiva." Does his* refer to option A or B below?* A Reb Asher B Akiva Eiger C Neither If you have any more questions, please refer to our FAQ page.

GAP 说明 纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！ 我们会给您维基百科文章摘录，其中有一个加粗代词。我们还会给出文本中该代词可能指代的两个名字。您的任务是：根据摘录判断代词指代选项 A、选项 B，还是都不指代： 若代词指代选项 A，选 A。例如， "In 2010 Ella Kabambe was not the official Miss Malawi; this was Faith Chibale, but Kabambe represented the country in the Miss World pageant. At the 2012 Miss World, Susan Mtegha pushed Miss New Zealand, Collette Lochore, during the opening headshot of the pageant, claiming that Miss New Zealand was in her space." Does her refer to option A or B below? A Susan Mtegha B Collette Lochore C Neither 若代词指代选项 B，选 B。例如， "In 1650 he started his career as advisor in the ministerium of finances in Den Haag. After he became a minister he went back to Amsterdam, and took place as a sort of chairing mayor of this city. After the death of his brother Cornelis, De Graeff became the strong leader of the republicans. He held this position until the rampjaar." Does He refer to option A or B below? A Cornelis B De Graeff C Neither 若代词两者都不指代，选 C。例如， "Reb Chaim Yaakov’s wife is the sister of Rabbi Moishe Sternbuch, as is the wife of Rabbi Meshulam Dovid Soloveitchik, making the two Rabbis his uncles. Reb Asher’s brother Rabbi Shlomo Arieli is the author of a critical edition of the novallae of Rabbi Akiva Eiger. Before his marriage, Rabbi Arieli studied in the Ponevezh Yeshiva headed by Rabbi Shmuel Rozovsky, and he later studied under his father-in-law in the Mirrer Yeshiva." Does his refer to option A or B below? A Reb Asher B Akiva Eiger C Neither 如有更多问题，请参阅 FAQ 页。 表 12：Gendered Ambiguous Pronoun Coreference（GAP）任务的任务特定说明。训练与标注阶段均提供。

GAP 说明

纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！

我们会给您维基百科文章摘录，其中有一个加粗代词。我们还会给出文本中该代词可能指代的两个名字。您的任务是：根据摘录判断代词指代选项 A、选项 B，还是都不指代：

若代词指代选项 A，选 A。例如，"In 2010 Ella Kabambe was not the official Miss Malawi; this was Faith Chibale, but Kabambe represented the country in the Miss World pageant. At the 2012 Miss World, Susan Mtegha pushed Miss New Zealand, Collette Lochore, during the opening headshot of the pageant, claiming that Miss New Zealand was in her space." Does her* refer to option A or B below?* A Susan Mtegha B Collette Lochore C Neither 若代词指代选项 B，选 B。例如，"In 1650 he started his career as advisor in the ministerium of finances in Den Haag. After he became a minister he went back to Amsterdam, and took place as a sort of chairing mayor of this city. After the death of his brother Cornelis, De Graeff became the strong leader of the republicans. He held this position until the rampjaar." Does He refer to option A or B below? A Cornelis B De Graeff C Neither 若代词两者都不指代，选 C。例如，"Reb Chaim Yaakov’s wife is the sister of Rabbi Moishe Sternbuch, as is the wife of Rabbi Meshulam Dovid Soloveitchik, making the two Rabbis his uncles. Reb Asher’s brother Rabbi Shlomo Arieli is the author of a critical edition of the novallae of Rabbi Akiva Eiger. Before his marriage, Rabbi Arieli studied in the Ponevezh Yeshiva headed by Rabbi Shmuel Rozovsky, and he later studied under his father-in-law in the Mirrer Yeshiva." Does his* refer to option A or B below?* A Reb Asher B Akiva Eiger C Neither 如有更多问题，请参阅 FAQ 页。

Paraphrase Detection Instructions The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help! We will present you with two similar sentences taken from Wikipedia articles. Your job is to figure out if these two sentences are paraphrases of each other, and convey exactly the same meaning: Choose Yes if the sentences are paraphrases and have the exact same meaning. For example, "Hastings Ndlovu was buried with Hector Pieterson at Avalon Cemetery in Johannesburg." "Hastings Ndlovu , together with Hector Pieterson , was buried at the Avalon cemetery in Johannesburg ." "The complex of the Trabzon World Trade Center is close to Trabzon Airport ." "The complex of World Trade Center Trabzon is situated close to Trabzon Airport ." Choose No if the two sentences are not exact paraphrases and mean different things. For example, "She was only a few months in French service when she met some British frigates in 1809 ." "She was only in British service for a few months , when in 1809 , she encountered some French frigates ." "This work caused him to trigger important reflections on the practices of molecular genetics and genomics at a time when this was not considered ethical ." "This work led him to trigger ethical reflections on the practices of molecular genetics and genomics at a time when this was not considered important ." If you have any more questions, please refer to our FAQ page. Table 13: Task-specific instructions for the Paraphrase Adversaries from Word Scrambling (PAWS) task. These instructions were provided during both training and annotation phases.

Paraphrase Detection Instructions

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help!

We will present you with two similar sentences taken from Wikipedia articles. Your job is to figure out if these two sentences are paraphrases of each other, and convey exactly the same meaning:

Choose Yes if the sentences are paraphrases and have the exact same meaning. For example, "Hastings Ndlovu was buried with Hector Pieterson at Avalon Cemetery in Johannesburg." "Hastings Ndlovu , together with Hector Pieterson , was buried at the Avalon cemetery in Johannesburg ." "The complex of the Trabzon World Trade Center is close to Trabzon Airport ." "The complex of World Trade Center Trabzon is situated close to Trabzon Airport ." Choose No if the two sentences are not exact paraphrases and mean different things. For example, "She was only a few months in French service when she met some British frigates in 1809 ." "She was only in British service for a few months , when in 1809 , she encountered some French frigates ." "This work caused him to trigger important reflections on the practices of molecular genetics and genomics at a time when this was not considered ethical ." "This work led him to trigger ethical reflections on the practices of molecular genetics and genomics at a time when this was not considered important ." If you have any more questions, please refer to our FAQ page.

释义检测说明 纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！ 我们会给您取自维基百科文章的两个相似句子。您的任务是判断这两句是否互为释义，并传达完全相同的意义： 若两句是释义且意义完全相同，选 Yes。例如， "Hastings Ndlovu was buried with Hector Pieterson at Avalon Cemetery in Johannesburg." "Hastings Ndlovu , together with Hector Pieterson , was buried at the Avalon cemetery in Johannesburg ." "The complex of the Trabzon World Trade Center is close to Trabzon Airport ." "The complex of World Trade Center Trabzon is situated close to Trabzon Airport ." 若两句不是精确释义且意义不同，选 No。例如， "She was only a few months in French service when she met some British frigates in 1809 ." "She was only in British service for a few months , when in 1809 , she encountered some French frigates ." "This work caused him to trigger important reflections on the practices of molecular genetics and genomics at a time when this was not considered ethical ." "This work led him to trigger ethical reflections on the practices of molecular genetics and genomics at a time when this was not considered important ." 如有更多问题，请参阅 FAQ 页。 表 13：Paraphrase Adversaries from Word Scrambling（PAWS）任务的任务特定说明。训练与标注阶段均提供。

释义检测说明

纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！

我们会给您取自维基百科文章的两个相似句子。您的任务是判断这两句是否互为释义，并传达完全相同的意义：

若两句是释义且意义完全相同，选 Yes。例如，"Hastings Ndlovu was buried with Hector Pieterson at Avalon Cemetery in Johannesburg." "Hastings Ndlovu , together with Hector Pieterson , was buried at the Avalon cemetery in Johannesburg ." "The complex of the Trabzon World Trade Center is close to Trabzon Airport ." "The complex of World Trade Center Trabzon is situated close to Trabzon Airport ." 若两句不是精确释义且意义不同，选 No。例如，"She was only a few months in French service when she met some British frigates in 1809 ." "She was only in British service for a few months , when in 1809 , she encountered some French frigates ." "This work caused him to trigger important reflections on the practices of molecular genetics and genomics at a time when this was not considered ethical ." "This work led him to trigger ethical reflections on the practices of molecular genetics and genomics at a time when this was not considered important ." 如有更多问题，请参阅 FAQ 页。

Insincere Questions Instructions

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help!

We will present you with a question that someone posted on Quora. Your job is to figure out whether or not this is a sincere question. An insincere question is defined as a question intended to make a statement rather than look for helpful answers. Some characteristics that can signify that a question is insincere: • Has a non-neutral tone – Has an exaggerated tone to underscore a point about a group of people – Is rhetorical and meant to imply a statement about a group of people • Is disparaging or inflammatory – Suggests a discriminatory idea against a protected class of people, or seeks confirmation of a stereotype – Makes disparaging attacks/insults against a specific person or group of people – Based on an outlandish premise about a group of people – Disparages against a characteristic that is not fixable and not measurable • Isn’t grounded in reality – Based on false information, or contains absurd assumptions – Uses sexual content (incest, bestiality, pedophilia) for shock value, and not to seek genuine answers Please note that there are far fewer insincere questions than there are sincere questions! So you should expect to label most questions as sincere.

Examples,
Choose Sincere if you believe the person asking the question was genuinely seeking an answer from the forum. For example, "How do DNA and RNA compare and contrast?" "Are there any sports that you don’t like?" "What is the main purpose of penance?" Choose Insincere if you believe the person asking the question was not really seeking an answer but was being inflammatory, extremely rhetorical, or absurd. For example, "How do I sell Pakistan? I need lots of money so I decided to sell Pakistan any one wanna buy?" "If Hispanics are so proud of their countries, why do they move out?" "Why Chinese people are always not welcome in all countries?" If you have any more questions, please refer to our FAQ page.

不真诚问题说明 纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！ 我们会给您某人在 Quora 上发布的问题。您的任务是判断这是否为真诚问题。不真诚问题定义为意在表态而非寻求有用答案的问题。可标志问题不真诚的若干特征：• 语气非中立 - 语气夸张以强调关于某群体的观点 - 修辞性且意在暗示关于某群体的陈述 • 贬损或煽动 - 对受保护群体提出歧视性想法，或寻求刻板印象确认 - 对特定个人或群体进行贬损攻击/侮辱 - 基于关于某群体的荒诞前提 - 贬损不可改变且不可度量的特征 • 不立足现实 - 基于虚假信息，或含荒谬假设 - 使用性内容（乱伦、兽交、恋童）以求冲击效果，而非真诚求答 请注意不真诚问题远少于真诚问题！因此您应预期将多数问题标为真诚。 示例 ， 若您认为提问者真诚寻求论坛答案，选 Sincere。例如， "How do DNA and RNA compare and contrast?" "Are there any sports that you don’t like? " "What is the main purpose of penance?" 若您认为提问者并非真正求答，而是煽动、极度修辞或荒诞，选 Insincere。例如， "How do I sell Pakistan? I need lots of money so I decided to sell Pakistan any one wanna buy?" "If Hispanics are so proud of their countries, why do they move out?" "Why Chinese people are always not welcome in all countries?" 如有更多问题，请参阅 FAQ 页。 表 14：Quora Insincere Questions 任务的任务特定说明。训练与标注阶段均提供。

不真诚问题说明

纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！

我们会给您某人在 Quora 上发布的问题。您的任务是判断这是否为真诚问题。不真诚问题定义为意在表态而非寻求有用答案的问题。可标志问题不真诚的若干特征：• 语气非中立 – 语气夸张以强调关于某群体的观点 – 修辞性且意在暗示关于某群体的陈述 • 贬损或煽动 – 对受保护群体提出歧视性想法，或寻求刻板印象确认 – 对特定个人或群体进行贬损攻击/侮辱 – 基于关于某群体的荒诞前提 – 贬损不可改变且不可度量的特征 • 不立足现实 – 基于虚假信息，或含荒谬假设 – 使用性内容（乱伦、兽交、恋童）以求冲击效果，而非真诚求答请注意不真诚问题远少于真诚问题！因此您应预期将多数问题标为真诚。

示例，
若您认为提问者真诚寻求论坛答案，选 Sincere。例如，"How do DNA and RNA compare and contrast?" "Are there any sports that you don’t like?" "What is the main purpose of penance?" 若您认为提问者并非真正求答，而是煽动、极度修辞或荒诞，选 Insincere。例如，"How do I sell Pakistan? I need lots of money so I decided to sell Pakistan any one wanna buy?" "If Hispanics are so proud of their countries, why do they move out?" "Why Chinese people are always not welcome in all countries?" 如有更多问题，请参阅 FAQ 页。

Entity Typing Instructions The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help! We will provide you with a sentence with on bolded word or phrase. We will also give you a possible tag for this bolded word or phrase. Your job is to decide, in the context of the sentence, if this tag is correct and applicable to the bolded word or phrase: Choose Yes if the tag is applicable and accurately describes the selected word or phrase. For example, “Spain was the gold line." It started out with zero gold in 1937, and by 1945 it had 65.5 tons. Tag: nation Choose No if the tag is not applicable and does not describes the selected word or phrase. For example, Iraqi museum workers are starting to assess the damage to Iraq’s history. Tag: organism If you have any more questions, please refer to our FAQ page. Table 15: Task-specific instructions for the Ultrafine Entity Typing task. These instructions were provided during both training and annotation phases.

Entity Typing Instructions

The New York University Center for Data Science is collecting your answers for use in research on computer understanding of English. Thank you for your help!

We will provide you with a sentence with on bolded word or phrase. We will also give you a possible tag for this bolded word or phrase. Your job is to decide, in the context of the sentence, if this tag is correct and applicable to the bolded word or phrase:

Choose Yes if the tag is applicable and accurately describes the selected word or phrase. For example, “Spain was the gold line." It started out with zero gold in 1937, and by 1945 it had 65.5 tons. Tag: nation Choose No if the tag is not applicable and does not describes the selected word or phrase. For example, Iraqi museum workers are starting to assess the damage to Iraq’s history. Tag: organism If you have any more questions, please refer to our FAQ page.

实体类型说明 纽约大学数据科学中心正在收集您的回答，用于计算机理解英语的研究。感谢您的帮助！ 我们会给您一句带加粗词或短语的句子，以及该加粗词或短语的一个可能标签。您的任务是：在句子语境中，判断该标签是否正确且适用于加粗词或短语： 若标签适用且准确描述被选词或短语，选 Yes。例如， “Spain was the gold line." It started out with zero gold in 1937, and by 1945 it had 65.5 tons. Tag: nation 若标签不适用且不能描述被选词或短语，选 No。例如， Iraqi museum workers are starting to assess the damage to Iraq’s history. Tag: organism 如有更多问题，请参阅 FAQ 页。 表 15：Ultrafine Entity Typing 任务的任务特定说明。训练与标注阶段均提供。

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

SuperGLUE：面向通用语言理解系统的更「黏」基准

Abstract

摘要

1 Introduction

1 引言

2 Related Work

2 相关工作

3 SuperGLUE Overview

3 SuperGLUE 概览

3.1 Design Process

3.1 设计过程

3.2 Selected Tasks

3.2 选定任务

3.3 Scoring

3.3 打分

3.4 Tools for Model Analysis

3.4 模型分析工具

Analyzing Linguistic and World Knowledge in Models

分析模型中的语言与世界知识

Analyzing Gender Bias in Models

分析模型中的性别偏见

4 Using SuperGLUE

4 使用 SuperGLUE

Software Tools

软件工具

Eligibility

资格

Data

数据

5 Experiments

5 实验

5.1 Baselines

5.1 基线

BERT

BERT

BERT++

BERT++

Other Baselines

其他基线

Human Performance

人类表现

5.2 Results

5.2 结果

6 Conclusion

6 结论

7 Acknowledgments

7 致谢

References

参考文献

Appendix A Development Set Results

附录 A 开发集结果

Appendix B Performance on GLUE Diagnostics

附录 B GLUE 诊断上的表现

Appendix C Human Performance Estimation

附录 C 人类表现估计

C.1 Training Phase Instructions

C.1 训练阶段说明

C.2 Task Instructions

C.2 任务说明

C.3 Task Specific Details

C.3 任务特定细节

CommitmentBank

CommitmentBank

Diagnostic Dataset

诊断数据集

Ultrafine Entity Typing

Ultrafine Entity Typing

GAP

GAP

Appendix D Excluded Tasks

附录 D 未纳入任务