GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

GLUE：面向自然语言理解的多任务基准与分析平台

Alex Wang Affiliation: Courant Institute of Mathematical Sciences, New York University Amanpreet Singh Affiliation: Courant Institute of Mathematical Sciences, New York University Julian Michael Affiliation: Paul G. Allen School of Computer Science & Engineering, University of Washington Felix Hill Affiliation: DeepMind {alexwang,amanpreet,bowman}@nyu.edu{julianjm,omerlevy}@cs.washington.edufelixhill@google.com Omer Levy & Samuel R. Bowman Affiliation: Courant Institute of Mathematical Sciences, New York University Affiliation: Paul G. Allen School of Computer Science & Engineering, University of Washington

Abstract

摘要

For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task. However, the low absolute performance of our best model indicates the need for improved general NLU systems.

自然语言理解（NLU）若要真正有用，处理语言的方式就不能只绑在单一任务、体裁或数据集上。为此我们提出通用语言理解评估（GLUE）基准：一套工具，用于在多样既有 NLU 任务上评估模型表现。通过纳入训练数据有限的任务，GLUE 有意偏向并鼓励能跨任务共享一般语言知识的模型。GLUE 还包含手工构建的诊断测试套件，便于对模型做细粒度语言学分析。我们基于当前迁移与表示学习方法评估基线，发现在所有任务上做多任务训练优于每任务单独训一个模型。但最佳模型的绝对分数仍然偏低，说明仍需更好的通用 NLU 系统。

1 Introduction

1 引言

The human ability to understand language is general, flexible, and robust . In contrast, most NLU models above the word level are designed for a specific task and struggle with out-of-domain data. If we aspire to develop models with understanding beyond the detection of superficial correspondences between inputs and outputs, then it is critical to develop a more unified model that can learn to execute a range of different linguistic tasks in different domains.

人类理解语言的能力是 通用 、 灵活 且 稳健 的。相比之下，多数词级以上的 NLU 模型专为单一任务设计，遇到域外数据就吃力。若希望模型的理解超越输入输出之间的表面对应，就需要更统一的模型，能在不同领域学会执行一系列不同的语言任务。

To facilitate research in this direction, we present the General Language Understanding Evaluation (GLUE) benchmark: a collection of NLU tasks including question answering, sentiment analysis, and textual entailment, and an associated online platform for model evaluation, comparison, and analysis. GLUE does not place any constraints on model architecture beyond the ability to process single-sentence and sentence-pair inputs and to make corresponding predictions. For some GLUE tasks, training data is plentiful, but for others it is limited or fails to match the genre of the test set. GLUE therefore favors models that can learn to represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks. None of the datasets in GLUE were created from scratch for the benchmark; we rely on preexisting datasets because they have been implicitly agreed upon by the NLP community as challenging and interesting. Four of the datasets feature privately-held test data, which will be used to ensure that the benchmark is used fairly.[^1]

为推动这一方向，我们提出通用语言理解评估（GLUE）基准：一组含问答、情感分析、文本蕴含等的 NLU 任务，以及配套的在线平台，用于模型评估、比较与分析。除要求能处理单句与句对输入并给出相应预测外，GLUE 不对模型结构设限。部分任务训练数据充足，另一些则有限或与测试集体裁不匹配，因而更青睐能以样本高效方式表示语言知识、并有效跨任务迁移的模型。GLUE 中没有任何数据集是为本基准从零新建的；我们依赖既有数据集，因为 NLP 社区已隐含认定它们既难又有趣。其中四个数据集的测试标签不公开，用于保证评测公平。[^1]

To understand the types of knowledge learned by models and to encourage linguistic-meaningful solution strategies, GLUE also includes a set of hand-crafted analysis examples for probing trained models. This dataset is designed to highlight common challenges, such as the use of world knowledge and logical operators, that we expect models must handle to robustly solve the tasks.

为理解模型学到的知识类型、并鼓励有语言学意义的解题策略，GLUE 还包含一组手工分析样例，用于探测已训模型。该数据集意在凸显世界知识与逻辑算子等常见挑战——我们期望模型要稳健解题就必须处理这些现象。

To better understand the challenged posed by GLUE, we conduct experiments with simple baselines and state-of-the-art sentence representation models. We find that unified multi-task trained models slightly outperform comparable models trained on each task separately. Our best multi-task model makes use of ELMo (Peters et al. 2018), a recently proposed pre-training technique. However, this model still achieves a fairly low absolute score. Analysis with our diagnostic dataset reveals that our baseline models deal well with strong lexical signals but struggle with deeper logical structure.

为更好理解 GLUE 带来的挑战，我们用简单基线与当时最优句表示模型做实验。发现统一多任务训练的模型略优于各任务单独训练的可比模型。最佳多任务模型使用了当时提出的预训练技术 ELMo（Peters et al. 2018），但绝对分数仍偏低。诊断集分析显示：基线善于抓住强词汇信号，却在更深逻辑结构上挣扎。

In summary, we offer: (i) A suite of nine sentence or sentence-pair NLU tasks, built on established annotated datasets and selected to cover a diverse range of text genres, dataset sizes, and degrees of difficulty. (ii) An online evaluation platform and leaderboard, based primarily on privately-held test data. The platform is model-agnostic, and can evaluate any method capable of producing results on all nine tasks. (iii) An expert-constructed diagnostic evaluation dataset. (iv) Baseline results for several major existing approaches to sentence representation learning.

总结起来，我们提供：(i) 九个单句或句对 NLU 任务套件，建立在既有标注数据上，覆盖多样文本体裁、数据规模与难度；(ii) 主要基于不公开测试标签的在线评估平台与排行榜，与模型无关，凡能在九项任务上出结果的方法均可评测；(iii) 专家构建的诊断评估集；(iv) 若干主流句表示学习方法的基线结果。

|

Corpus	$\|$ Train $\|$	$\|$ Test $\|$	Task	Metrics	Domain
Single-Sentence Tasks
CoLA	8.5k	1k	acceptability	Matthews corr.	misc.
SST-2	67k	1.8k	sentiment	acc.	movie reviews
Similarity and Paraphrase Tasks
MRPC	3.7k	1.7k	paraphrase	acc./F1	news
STS-B	7k	1.4k	sentence similarity	Pearson/Spearman corr.	misc.
QQP	364k	391k	paraphrase	acc./F1	social QA questions
Inference Tasks
MNLI	393k	20k	NLI	matched acc./mismatched acc.	misc.
QNLI	105k	5.4k	QA/NLI	acc.	Wikipedia
RTE	2.5k	3k	NLI	acc.	news, Wikipedia
WNLI	634	146	coreference/NLI	acc.	fiction books

|

语料	$\|$ Train $\|$	$\|$ Test $\|$	任务	指标	领域
单句任务
CoLA	8.5k	1k	acceptability	Matthews corr.	misc.
SST-2	67k	1.8k	sentiment	acc.	movie reviews
相似度与复述任务
MRPC	3.7k	1.7k	paraphrase	acc./F1	news
STS-B	7k	1.4k	sentence similarity	Pearson/Spearman corr.	misc.
QQP	364k	391k	paraphrase	acc./F1	social QA questions
推理任务
MNLI	393k	20k	NLI	matched acc./mismatched acc.	misc.
QNLI	105k	5.4k	QA/NLI	acc.	Wikipedia
RTE	2.5k	3k	NLI	acc.	news, Wikipedia
WNLI	634	146	coreference/NLI	acc.	fiction books

2 Related Work

2 相关工作

Collobert et al. 2011 used a multi-task model with a shared sentence understanding component to jointly learn POS tagging, chunking, named entity recognition, and semantic role labeling. More recent work has explored using labels from core NLP tasks to supervise training of lower levels of deep neural networks (Søgaard & Goldberg 2016; Hashimoto et al. 2017) and automatically learning cross-task sharing mechanisms for multi-task learning (Ruder et al. 2017).

Collobert et al. 2011 用带共享句理解组件的多任务模型，联合学习词性标注、组块、命名实体识别与语义角色标注。后续工作探索用核心 NLP 任务标签监督深度网络较低层（Søgaard & Goldberg 2016；Hashimoto et al. 2017），以及自动学习多任务共享机制（Ruder et al. 2017）。

Beyond multi-task learning, much work in developing general NLU systems has focused on sentence-to-vector encoders (Le & Mikolov 2014; Kiros et al. 2015, i.a.), leveraging unlabeled data (Hill et al. 2016; Peters et al. 2018), labeled data (Conneau & Kiela 2018; McCann et al. 2017), and combinations of these (Collobert et al. 2011; Subramanian et al. 2018). In this line of work, a standard evaluation practice has emerged, recently codified as SentEval (Conneau et al. 2017; Conneau & Kiela 2018). Like GLUE, SentEval relies on a set of existing classification tasks involving either one or two sentences as inputs. Unlike GLUE, SentEval only evaluates sentence-to-vector encoders, making it well-suited for evaluating models on tasks involving sentences in isolation . However, cross-sentence contextualization and alignment are instrumental in achieving state-of-the-art performance on tasks such as machine translation (Bahdanau et al. 2015; Vaswani et al. 2017), question answering (Seo et al. 2017), and natural language inference (Rocktäschel et al. 2016). GLUE is designed to facilitate the development of these methods: It is model-agnostic, allowing for any kind of representation or contextualization, including models that use no explicit vector or symbolic representations for sentences whatsoever.

在多任务之外，通用 NLU 系统的大量工作聚焦句到向量编码器（Le & Mikolov 2014；Kiros et al. 2015 等），分别利用无标注数据（Hill et al. 2016；Peters et al. 2018）、有标注数据（Conneau & Kiela 2018；McCann et al. 2017）及其组合（Collobert et al. 2011；Subramanian et al. 2018）。该路线形成了标准评估实践，近期被整理为 SentEval（Conneau et al. 2017；Conneau & Kiela 2018）。与 GLUE 类似，SentEval 依赖一组既有分类任务，输入为一句或两句。不同的是，SentEval 只评估句到向量编码器，适合评估 孤立 句子上的模型。然而跨句上下文化与对齐对机器翻译（Bahdanau et al. 2015；Vaswani et al. 2017）、问答（Seo et al. 2017）与自然语言推理（Rocktäschel et al. 2016）等取得最优结果至关重要。GLUE 意在促进这类方法：它与模型无关，允许任意表示或上下文化，甚至包括完全不为句子使用显式向量或符号表示的模型。

GLUE also diverges from SentEval in the selection of evaluation tasks that are included in the suite. Many of the SentEval tasks are closely related to sentiment analysis, such as MR (Pang & Lee 2005), SST (Socher et al. 2013), CR (Hu & Liu 2004), and SUBJ (Pang & Lee 2004). Other tasks are so close to being solved that evaluation on them is relatively uninformative, such as MPQA (Wiebe et al. 2005) and TREC question classification (Voorhees et al. 1999). In GLUE, we attempt to construct a benchmark that is both diverse and difficult.

GLUE 与 SentEval 在所选评估任务上也不同。SentEval 许多任务紧贴情感分析，如 MR、SST、CR、SUBJ；另一些已接近被解决，评估信息量有限，如 MPQA 与 TREC 问题分类。在 GLUE 中，我们试图构造既多样又困难的基准。

McCann et al. 2018 introduce decaNLP, which also scores NLP systems based on their performance on multiple datasets. Their benchmark recasts the ten evaluation tasks as question answering, converting tasks like summarization and text-to-SQL semantic parsing into question answering using automatic transformations. That benchmark lacks the leaderboard and error analysis toolkit of GLUE, but more importantly, we see it as pursuing a more ambitious but less immediately practical goal: While GLUE rewards methods that yield good performance on a circumscribed set of tasks using methods like those that are currently used for those tasks, their benchmark rewards systems that make progress toward their goal of unifying all of NLU under the rubric of question answering.

McCann et al. 2018 提出 decaNLP，同样按多数据集表现给 NLP 系统打分。其基准把十项评估任务改写为问答，用自动变换把摘要、text-to-SQL 等转成问答。该基准缺少 GLUE 的排行榜与误差分析工具包；更重要的是，我们认为其目标更宏大但当下不那么落地：GLUE 奖励在既定任务集合上、用当前常用方法取得好结果的方法，而他们的基准奖励朝「把全部 NLU 统一到问答框架」迈进的系统。

3 Tasks

3 任务

GLUE is centered on nine English sentence understanding tasks, which cover a broad range of domains, data quantities, and difficulties. As the goal of GLUE is to spur development of generalizable NLU systems, we design the benchmark such that good performance should require a model to share substantial knowledge (e.g., trained parameters) across all tasks, while still maintaining some task-specific components. Though it is possible to train a single model for each task with no pretraining or other outside sources of knowledge and evaluate the resulting set of models on this benchmark, we expect that our inclusion of several data-scarce tasks will ultimately render this approach uncompetitive. We describe the tasks below and in Table 1. Appendix A includes additional details. Unless otherwise mentioned, tasks are evaluated on accuracy and are balanced across classes.

GLUE 围绕九项英语句子理解任务，覆盖广泛领域、数据量与难度。因目标是催生可泛化的 NLU 系统，我们把基准设计成：好成绩应要求模型在所有任务间共享大量知识（例如训练参数），同时仍保留部分任务专用组件。虽可为每任务各训一个无预训练、无外部知识的模型再汇总评估，但我们纳入若干数据稀缺任务，预期最终会使这种做法缺乏竞争力。任务说明见下及表 1。附录 A 有更多细节。除非另说明，任务以准确率为指标且类别大致均衡。

3.1 Single-Sentence Tasks

3.1 单句任务

CoLA The Corpus of Linguistic Acceptability (Warstadt et al. 2018) consists of English acceptability judgments drawn from books and journal articles on linguistic theory. Each example is a sequence of words annotated with whether it is a grammatical English sentence. Following the authors, we use Matthews correlation coefficient (Matthews 1975) as the evaluation metric, which evaluates performance on unbalanced binary classification and ranges from -1 to 1, with 0 being the performance of uninformed guessing. We use the standard test set, for which we obtained private labels from the authors. We report a single performance number on the combination of the in- and out-of-domain sections of the test set.

CoLA 语言可接受性语料库（Warstadt et al. 2018）收集来自语言学理论书籍与期刊的英语可接受性判断。每个例子是词序列，标注是否为合乎语法的英语句子。遵循作者做法，用 Matthews 相关系数（Matthews 1975）作指标：适用于不平衡二分类，取值 -1 到 1，0 相当于无信息猜测。我们使用标准测试集，并从作者处获得私有标签；在测试集域内与域外部分合并后报告单一分数。

SST-2 The Stanford Sentiment Treebank (Socher et al. 2013) consists of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence. We use the two-way (positive / negative) class split, and use only sentence-level labels.

SST-2 斯坦福情感树库（Socher et al. 2013）由电影评论句子及其人工情感标注构成。任务是预测给定句子的情感。我们使用二分类（ 正面 / 负面 ）划分，且只用句级标签。

3.2 Similarity and Paraphrase Tasks

3.2 相似度与复述任务

MRPC The Microsoft Research Paraphrase Corpus (Dolan & Brockett 2005) is a corpus of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent. Because the classes are imbalanced (68% positive), we follow common practice and report both accuracy and F1 score.

MRPC 微软研究院复述语料（Dolan & Brockett 2005）自动从在线新闻抽取句对，并由人工标注两句是否语义等价。因类别不平衡（68% 正例），我们遵循惯例同时报告准确率与 F1。

QQP The Quora Question Pairs[^2] dataset is a collection of question pairs from the community question-answering website Quora. The task is to determine whether a pair of questions are semantically equivalent. As in MRPC, the class distribution in QQP is unbalanced (63% negative), so we report both accuracy and F1 score. We use the standard test set, for which we obtained private labels from the authors. We observe that the test set has a different label distribution than the training set.

QQP Quora 问题对[^2]数据集收集社区问答网站 Quora 上的问题对。任务是判断一对问题是否语义等价。与 MRPC 类似，类别不平衡（63% 负例），故同时报告准确率与 F1。我们使用标准测试集并获得作者私有标签；观察到测试集标签分布与训练集不同。

STS-B The Semantic Textual Similarity Benchmark (Cer et al. 2017) is a collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data. Each pair is human-annotated with a similarity score from 1 to 5; the task is to predict these scores. Follow common practice, we evaluate using Pearson and Spearman correlation coefficients.

STS-B 语义文本相似度基准（Cer et al. 2017）收集来自新闻标题、视频与图像字幕以及自然语言推理数据的句对。每对由人工标注 1 到 5 的相似度分数；任务是预测这些分数。遵循惯例，用 Pearson 与 Spearman 相关系数评估。

3.3 Inference Tasks

3.3 推理任务

MNLI The Multi-Genre Natural Language Inference Corpus (Williams et al. 2018) is a crowd-sourced collection of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). The premise sentences are gathered from ten different sources, including transcribed speech, fiction, and government reports. We use the standard test set, for which we obtained private labels from the authors, and evaluate on both the matched (in-domain) and mismatched (cross-domain) sections. We also use and recommend the SNLI corpus (Bowman et al. 2015) as 550k examples of auxiliary training data.

MNLI 多体裁自然语言推理语料（Williams et al. 2018）是众包句对，带文本蕴含标注。给定前提与假设，任务是预测前提蕴含假设（ entailment ）、与假设矛盾（ contradiction ），或皆非（ neutral ）。前提句来自十个来源，含转写语音、小说与政府报告等。我们使用标准测试集并获得私有标签，在 匹配 （域内）与 不匹配 （跨域）两部分上评估。我们也使用并推荐 SNLI（Bowman et al. 2015）作为约 55 万例的辅助训练数据。

QNLI The Stanford Question Answering Dataset (Rajpurkar et al. 2016) is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator). We convert the task into sentence pair classification by forming a pair between each question and each sentence in the corresponding context, and filtering out pairs with low lexical overlap between the question and the context sentence. The task is to determine whether the context sentence contains the answer to the question. This modified version of the original task removes the requirement that the model select the exact answer, but also removes the simplifying assumptions that the answer is always present in the input and that lexical overlap is a reliable cue. This process of recasting existing datasets into NLI is similar to methods introduced in White et al. 2017 and expanded upon in Demszky et al. 2018. We call the converted dataset QNLI (Question-answering NLI).[^3]

QNLI 斯坦福问答数据集（Rajpurkar et al. 2016）由问题-段落对构成，段落中某句（取自维基）包含对应问题（由标注者撰写）的答案。我们把任务改成句对分类：对每个问题与上下文中每句组成对，并过滤与问题词汇重叠过低的对。任务是判断上下文句是否包含问题答案。该改写去掉了必须选出精确答案跨度的要求，也去掉了「答案必在输入中」以及「词汇重叠可靠」等简化假设。把既有数据改写成 NLI 的做法类似 White et al. 2017，并由 Demszky et al. 2018 扩展。我们称转换后的数据集为 QNLI（问答 NLI）。[^3]

RTE The Recognizing Textual Entailment (RTE) datasets come from a series of annual textual entailment challenges. We combine the data from RTE1 (Dagan et al. 2006), RTE2 (Bar Haim et al. 2006), RTE3 (Giampiccolo et al. 2007), and RTE5 (Bentivogli et al. 2009).[^4] Examples are constructed based on news and Wikipedia text. We convert all datasets to a two-class split, where for three-class datasets we collapse neutral and contradiction into not_entailment, for consistency.

RTE 识别文本蕴含（RTE）数据集来自系列年度文本蕴含挑战。我们合并 RTE1（Dagan et al. 2006）、RTE2（Bar Haim et al. 2006）、RTE3（Giampiccolo et al. 2007）与 RTE5（Bentivogli et al. 2009）。[^4] 例子基于新闻与维基文本。为保持一致，全部转为二分类：对三类数据把 neutral 与 contradiction 合并为 not_entailment 。

Coarse-Grained Categories Fine-Grained Categories Lexical Semantics Lexical Entailment, Morphological Negation, Factivity, Symmetry/Collectivity, Redundancy, Named Entities, Quantifiers Predicate-Argument Structure Core Arguments, Prepositional Phrases, Ellipsis/Implicits, Anaphora/Coreference Active/Passive, Nominalization, Genitives/Partitives, Datives, Relative Clauses, Coordination Scope, Intersectivity, Restrictivity Logic Negation, Double Negation, Intervals/Numbers, Conjunction, Disjunction, Conditionals, Universal, Existential, Temporal, Upward Monotone, Downward Monotone, Non-Monotone Knowledge Common Sense, World Knowledge Table 2: The types of linguistic phenomena annotated in the diagnostic dataset, organized under four major categories. For a description of each phenomenon, see Appendix E.

Coarse-Grained Categories	Fine-Grained Categories
Lexical Semantics	Lexical Entailment, Morphological Negation, Factivity,
Symmetry/Collectivity, Redundancy, Named Entities, Quantifiers
Predicate-Argument Structure	Core Arguments, Prepositional Phrases, Ellipsis/Implicits,
Anaphora/Coreference Active/Passive, Nominalization,
Genitives/Partitives, Datives, Relative Clauses,
	Coordination Scope, Intersectivity, Restrictivity
Logic	Negation, Double Negation, Intervals/Numbers, Conjunction, Disjunction,
Conditionals, Universal, Existential, Temporal, Upward Monotone,
	Downward Monotone, Non-Monotone
Knowledge	Common Sense, World Knowledge

粗粒度类别 细粒度类别 Lexical Semantics Lexical Entailment, Morphological Negation, Factivity, Symmetry/Collectivity, Redundancy, Named Entities, Quantifiers Predicate-Argument Structure Core Arguments, Prepositional Phrases, Ellipsis/Implicits, Anaphora/Coreference Active/Passive, Nominalization, Genitives/Partitives, Datives, Relative Clauses, Coordination Scope, Intersectivity, Restrictivity Logic Negation, Double Negation, Intervals/Numbers, Conjunction, Disjunction, Conditionals, Universal, Existential, Temporal, Upward Monotone, Downward Monotone, Non-Monotone Knowledge Common Sense, World Knowledge 表 2：诊断集标注的语言现象类型，归入四大类。各现象说明见附录 E。

粗粒度类别	细粒度类别
Lexical Semantics	Lexical Entailment, Morphological Negation, Factivity,
Symmetry/Collectivity, Redundancy, Named Entities, Quantifiers
Predicate-Argument Structure	Core Arguments, Prepositional Phrases, Ellipsis/Implicits,
Anaphora/Coreference Active/Passive, Nominalization,
Genitives/Partitives, Datives, Relative Clauses,
	Coordination Scope, Intersectivity, Restrictivity
Logic	Negation, Double Negation, Intervals/Numbers, Conjunction, Disjunction,
Conditionals, Universal, Existential, Temporal, Upward Monotone,
	Downward Monotone, Non-Monotone
Knowledge	Common Sense, World Knowledge

WNLI The Winograd Schema Challenge (Levesque et al. 2011) is a reading comprehension task in which a system must read a sentence with a pronoun and select the referent of that pronoun from a list of choices. The examples are manually constructed to foil simple statistical methods: Each one is contingent on contextual information provided by a single word or phrase in the sentence. To convert the problem into sentence pair classification, we construct sentence pairs by replacing the ambiguous pronoun with each possible referent. The task is to predict if the sentence with the pronoun substituted is entailed by the original sentence. We use a small evaluation set consisting of new examples derived from fiction books[^5] that was shared privately by the authors of the original corpus. While the included training set is balanced between two classes, the test set is imbalanced between them (65% not entailment). Also, due to a data quirk, the development set is adversarial : hypotheses are sometimes shared between training and development examples, so if a model memorizes the training examples, they will predict the wrong label on corresponding development set example. As with QNLI, each example is evaluated separately, so there is not a systematic correspondence between a model’s score on this task and its score on the unconverted original task. We call converted dataset WNLI (Winograd NLI).

WNLI Winograd 图式挑战（Levesque et al. 2011）是阅读理解任务：系统需读含代词的句子，并从候选中选出指代对象。例子手工构造以挫败简单统计方法：每个例子都取决于句中单个词或短语提供的语境信息。为改成句对分类，我们用每个可能指代替换歧义代词，得到句对；任务是预测替换后的句子是否被原句蕴含。评估集较小，由原语料作者私下共享、自小说书衍生的新例子构成。[^5] 训练集两类均衡，测试集不均衡（65% 非蕴含）。另因数据怪癖，开发集具有 对抗性 ：假设有时在训练与开发例之间共享，若模型死记训练例，会在对应开发例上预测错误标签。与 QNLI 一样，每例单独评估，因而本任务分数与未转换原任务分数无系统对应。我们称转换后数据集为 WNLI（Winograd NLI）。

Tags Sentence 1 Sentence 2 Fwd Bwd Lexical Entailment (Lexical Semantics), Downward Monotone (Logic) The timing of the meeting has not been set, according to a Starbucks spokesperson. The timing of the meeting has not been considered, according to a Starbucks spokesperson. N E Universal Quantifiers (Logic) Our deepest sympathies are with all those affected by this accident. Our deepest sympathies are with a victim who was affected by this accident. E N Quantifiers (Lexical Semantics), Double Negation (Logic) I have never seen a hummingbird not flying. I have never seen a hummingbird. N E Table 3: Examples from the diagnostic set. Fwd (resp. Bwd) denotes the label when sentence 1 (resp. sentence 2) is the premise. Labels are entailment (E), neutral (N), or contradiction (C). Examples are tagged with the phenomena they demonstrate, and each phenomenon belongs to one of four broad categories (in parentheses).

Tags	Sentence 1	Sentence 2	Fwd	Bwd
Lexical Entailment (Lexical Semantics), Downward Monotone (Logic)	The timing of the meeting has not been set, according to a Starbucks spokesperson.	The timing of the meeting has not been considered, according to a Starbucks spokesperson.	N	E
Universal Quantifiers (Logic)	Our deepest sympathies are with all those affected by this accident.	Our deepest sympathies are with a victim who was affected by this accident.	E	N
Quantifiers (Lexical Semantics), Double Negation (Logic)	I have never seen a hummingbird not flying.	I have never seen a hummingbird.	N	E

Tags Sentence 1 Sentence 2 Fwd Bwd Lexical Entailment (Lexical Semantics), Downward Monotone (Logic) The timing of the meeting has not been set, according to a Starbucks spokesperson. The timing of the meeting has not been considered, according to a Starbucks spokesperson. N E Universal Quantifiers (Logic) Our deepest sympathies are with all those affected by this accident. Our deepest sympathies are with a victim who was affected by this accident. E N Quantifiers (Lexical Semantics), Double Negation (Logic) I have never seen a hummingbird not flying. I have never seen a hummingbird. N E 表 3：诊断集样例。 Fwd （ Bwd ）表示以句子 1（句子 2）为前提时的标签。标签为 entailment （E）、 neutral （N）或 contradiction （C）。例子标有所展示的现象，每个现象属于四大类之一（括号内）。

Tags	Sentence 1	Sentence 2	Fwd	Bwd
Lexical Entailment (Lexical Semantics), Downward Monotone (Logic)	The timing of the meeting has not been set, according to a Starbucks spokesperson.	The timing of the meeting has not been considered, according to a Starbucks spokesperson.	N	E
Universal Quantifiers (Logic)	Our deepest sympathies are with all those affected by this accident.	Our deepest sympathies are with a victim who was affected by this accident.	E	N
Quantifiers (Lexical Semantics), Double Negation (Logic)	I have never seen a hummingbird not flying.	I have never seen a hummingbird.	N	E

3.4 Evaluation

3.4 评估

The GLUE benchmark follows the same evaluation model as SemEval and Kaggle. To evaluate a system on the benchmark, one must run the system on the provided test data for the tasks, then upload the results to the website gluebenchmark.com for scoring. The benchmark site shows per-task scores and a macro-average of those scores to determine a system’s position on the leaderboard. For tasks with multiple metrics (e.g., accuracy and F1), we use an unweighted average of the metrics as the score for the task when computing the overall macro-average. The website also provides fine- and coarse-grained results on the diagnostic dataset. See Appendix D for details.

GLUE 基准沿用与 SemEval、Kaggle 相同的评估模式。要在基准上评估系统，须在各任务提供的测试数据上运行系统，再把结果上传到 gluebenchmark.com 评分。网站展示每任务分数及其宏平均，以决定排行榜名次。对有多项指标的任务（如准确率与 F1），计算总体宏平均时用这些指标的未加权平均作为该任务分数。网站还提供诊断集上的细粒度与粗粒度结果。细节见附录 D。

4 Diagnostic Dataset

4 诊断数据集

Drawing inspiration from the FraCaS suite (Cooper et al. 1996) and the recent Build-It-Break-It competition (Ettinger et al. 2017), we include a small, manually-curated test set for the analysis of system performance. While the main benchmark mostly reflects an application-driven distribution of examples, our diagnostic dataset highlights a pre-defined set of phenomena that we believe are interesting and important for models to capture. We show the full set of phenomena in Table 2.

受 FraCaS 套件（Cooper et al. 1996）与近期 Build-It-Break-It 竞赛（Ettinger et al. 2017）启发，我们纳入一个小规模、人工精选的测试集，用于分析系统表现。主基准大多反映应用驱动的例子分布，诊断集则突出我们认为模型需要把握的一组预定现象。完整现象列表见表 2。

Each diagnostic example is an NLI sentence pair with tags for the phenomena demonstrated. The NLI task is well-suited to this kind of analysis, as it can easily evaluate the full set of skills involved in (ungrounded) sentence understanding, from resolution of syntactic ambiguity to pragmatic reasoning with world knowledge. We ensure the data is reasonably diverse by producing examples for a variety of linguistic phenomena and basing our examples on naturally-occurring sentences from several domains (news, Reddit, Wikipedia, academic papers). This approaches differs from that of FraCaS, which was designed to test linguistic theories with a minimal and uniform set of examples. A sample from our dataset is shown in Table 3.

每个诊断例是带现象标签的 NLI 句对。NLI 很适合这类分析，因它能轻松覆盖（非落地）句子理解的全套技能，从句法歧义消解到借助世界知识的语用推理。我们为多种语言现象编写例子，并以新闻、Reddit、维基、学术论文等多领域自然句子为底，以保证多样性。这与旨在用最小、均匀例子检验语言学理论的 FraCaS 不同。数据集样例见表 3。

Annotation Process We begin with a target set of phenomena, based roughly on those used in the FraCaS suite (Cooper et al. 1996). We construct each example by locating a sentence that can be easily made to demonstrate a target phenomenon, and editing it in two ways to produce an appropriate sentence pair. We make minimal modifications so as to maintain high lexical and structural overlap within each sentence pair and limit superficial cues. We then label the inference relationships between the sentences, considering each sentence alternatively as the premise, producing two labeled examples for each pair (1100 total). Where possible, we produce several pairs with different labels for a single source sentence, to have minimal sets of sentence pairs that are lexically and structurally very similar but correspond to different entailment relationships. The resulting labels are 42% entailment, 35% neutral, and 23% contradiction .

标注过程 我们先设定目标现象集，大致参照 FraCaS（Cooper et al. 1996）。构造每例时，找到易于改写成展示目标现象的句子，再做两处编辑得到合适句对。修改尽量少，以保持句对内高词汇与结构重叠、限制表面线索。然后标注两句之间的推理关系，并轮流把每句当前提，从而每对产生两个有标签例子（共 1100）。可能时，为同一源句生成若干标签不同的对，得到词汇结构极相近但蕴含关系不同的最小句对集。最终标签分布为 42% entailment 、35% neutral 、23% contradiction 。

R_{3}

R_{3}

In light of recent work showing that crowdsourced data often contains artifacts which can be exploited to perform well without solving the intended task (Schwartz et al. 2017; Poliak et al. 2018; Tsuchiya 2018, i.a.), we audit the data for such artifacts. We reproduce the methodology of Gururangan et al. 2018, training two fastText classifiers (Joulin et al. 2016) to predict entailment labels on SNLI and MNLI using only the hypothesis as input. The models respectively get near-chance accuracies of 32.7% and 36.4% on our diagnostic data, showing that the data does not suffer from such artifacts.

鉴于近期工作表明众包数据常含可被利用的伪影，使模型不真正解题也能得好分数（Schwartz et al. 2017；Poliak et al. 2018；Tsuchiya 2018 等），我们对数据做了审计。复现 Gururangan et al. 2018 的方法，训练两个仅以假设为输入预测蕴含标签的 fastText 分类器（Joulin et al. 2016）。模型在诊断数据上准确率接近随机，分别为 32.7% 与 36.4%，说明数据未受此类伪影困扰。

\kappa

\kappa

Intended Use The diagnostic examples are hand-picked to address certain phenomena, and NLI is a task with no natural input distribution, so we do not expect performance on the diagnostic set to reflect overall performance or generalization in downstream applications. Performance on the analysis set should be compared between models but not between categories. The set is provided not as a benchmark, but as an analysis tool for error analysis, qualitative model comparison, and development of adversarial examples.

预期用途 诊断例为针对特定现象精选，且 NLI 并无自然输入分布，因此我们不期望诊断集表现反映下游应用的总体表现或泛化。分析集表现应在模型之间比较，而非在类别之间比较。该集合并非作为基准提供，而是用作误差分析、定性模型比较与对抗例子开发的分析工具。

5 Baselines

5 基线

For baselines, we evaluate a multi-task learning model trained on the GLUE tasks, as well as several variants based on recent pre-training methods. We briefly describe them here. See Appendix B for details. We implement our models in the AllenNLP library (Gardner et al. 2017). Original code for the baselines is available at https://github.com/nyu-mll/GLUE-baselines and a newer version is available at https://github.com/jsalt18-sentence-repl/jiant .

作为基线，我们评估在 GLUE 任务上训练的多任务学习模型，以及基于近期预训练方法的若干变体。此处简述，细节见附录 B。模型在 AllenNLP 库（Gardner et al. 2017）中实现。基线原始代码见 https://github.com/nyu-mll/GLUE-baselines，更新版见 https://github.com/jsalt18-sentence-repl/jiant。

u,v

结构 最简单的基线结构基于句到向量编码器，暂未发挥 GLUE 评估更复杂结构的能力。受 Conneau et al. 2017 启发，模型使用两层、1500 维 BiLSTM，对 GloVe 词嵌入（Pennington et al. 2014）编码，再经最大池化得到句向量。对句对任务，把两句向量及其逐元素积、差拼接，送入分类器。分类器为多层感知机，隐层带 ReLU。

We also consider a variant of our model which for sentence pair tasks uses an attention mechanism inspired by Seo et al. 2017 between all pairs of words, followed by a second BiLSTM with max pooling. By explicitly modeling the interaction between sentences, these models fall outside the sentence-to-vector paradigm.

我们还考虑一种变体：对句对任务，在矩阵注意力匹配后，用第二套 BiLSTM 编码（类似 Seo et al. 2017 的矩阵注意力层，但更简单）。具体做法见附录 B。

Pre-Training We augment our base model with two recent methods for pre-training: ELMo and CoVe. We use existing trained models for both.

预训练 我们用两种近期句编码器预训练方法增强基座模型：

ELMo uses a pair of two-layer neural language models trained on the Billion Word Benchmark (Chelba et al. 2013). Each word is represented by a contextual embedding, produced by taking a linear combination of the corresponding hidden states of each layer of the two models. We follow the authors’ recommendations[^6] and use ELMo embeddings in place of any other embeddings.

ELMo 使用在 Billion Word Benchmark（Chelba et al. 2014）上训练的一对双层神经语言模型，把字符卷积与两层 BiLSTM 状态拼接为每个词的上下文表示。我们遵循 Peters et al. 2018，把 ELMo 向量与任务模型输入拼接，并对 ELMo 层做任务专用的加权平均与缩放。

CoVe (McCann et al. 2017) uses a two-layer BiLSTM encoder originally trained for English-to-German translation. The CoVe vector of a word is the corresponding hidden state of the top-layer LSTM. As in the original work, we concatenate the CoVe vectors to the GloVe word embeddings.

CoVe（McCann et al. 2017）使用原在机器翻译上训练的双层 BiLSTM 编码器。我们把 CoVe 向量与 GloVe 拼接作为输入。

10^{-4}

训练 我们用 BiLSTM 句编码器与注意力后编码器训练模型，先在各任务上单独训练，再在所有任务上多任务训练。多任务时共享编码器参数，每任务有专用分类器。按与 McCann et al. 2018 类似的启发式轮流从各任务采样批次。细节见附录 B。

We also train a set of single-task models, which are configured and trained identically, but share no parameters. To allow for fair comparisons with the multi-task analogs, we do not tune parameter or training settings for each task, so these single-task models do not generally represent the state of the art for each task.

我们也训练一组单任务模型，配置与训练方式同对应多任务模型，但只在单一任务上训练。

Sentence Representation Models Finally, we evaluate the following trained sentence-to-vector encoder models using our benchmark: average bag-of-words using GloVe embeddings (CBoW), Skip-Thought (Kiros et al. 2015), InferSent (Conneau et al. 2017), DisSent (Nie et al. 2017), and GenSen (Subramanian et al. 2018). For these models, we only train task-specific classifiers on the representations they produce.

句表示模型 最后评估下列预训练句到向量编码器：把其句向量输入与上相同的分类器结构，并在每任务上单独训练分类器（编码器冻结）。模型包括 CBoW、Skip-Thought、InferSent、DisSent 与 GenSen。细节见附录 B。

Single Sentence Similarity and Paraphrase Natural Language Inference Model Avg CoLA SST-2 MRPC QQP STS-B MNLI QNLI RTE WNLI Single-Task Training BiLSTM 63.9 15.7 85.9 69.3/79.4 81.7/61.4 66.0/62.8 70.3/70.8 75.7 52.8 65.1 +ELMo 66.4 35.0 90.2 69.0/80.8 85.7/65.6 64.0/60.2 72.9/73.4 71.7 50.1 65.1 +CoVe 64.0 14.5 88.5 73.4/81.4 83.3/59.4 67.2/64.1 64.5/64.8 75.4 53.5 65.1 +Attn 63.9 15.7 85.9 68.5/80.3 83.5/62.9 59.3/55.8 74.2/73.8 77.2 51.9 65.1 +Attn, ELMo 66.5 35.0 90.2 68.8/80.2 86.5 / 66.1 55.5/52.5 76.9 / 76.7 76.7 50.4 65.1 +Attn, CoVe 63.2 14.5 88.5 68.6/79.7 84.1/60.1 57.2/53.6 71.6/71.5 74.5 52.7 65.1 Multi-Task Training BiLSTM 64.2 11.6 82.8 74.3/81.8 84.2/62.5 70.3/67.8 65.4/66.1 74.6 57.4 65.1 +ELMo 67.7 32.1 89.3 78.0 / 84.7 82.6/61.1 67.2/67.9 70.3/67.8 75.5 57.4 65.1 +CoVe 62.9 18.5 81.9 71.5/78.7 84.9/60.6 64.4/62.7 65.4/65.7 70.8 52.7 65.1 +Attn 65.6 18.6 83.0 76.2/83.9 82.4/60.1 72.8/70.5 67.6/68.3 74.3 58.4 65.1 +Attn, ELMo 70.0 33.6 90.4 78.0 /84.4 84.3/63.1 74.2/72.3 74.1/74.5 79.8 58.9 65.1 +Attn, CoVe 63.1 8.3 80.7 71.8/80.0 83.4/60.5 69.8/68.4 68.1/68.6 72.9 56.0 65.1 Pre-Trained Sentence Representation Models CBoW 58.9 0.0 80.0 73.4/81.5 79.1/51.4 61.2/58.7 56.0/56.4 72.1 54.1 65.1 Skip-Thought 61.3 0.0 81.8 71.7/80.8 82.2/56.4 71.8/69.7 62.9/62.8 72.9 53.1 65.1 InferSent 63.9 4.5 85.1 74.1/81.2 81.7/59.1 75.9/75.3 66.1/65.7 72.7 58.0 65.1 DisSent 62.0 4.9 83.7 74.1/81.7 82.6/59.5 66.1/64.8 58.7/59.1 73.9 56.4 65.1 GenSen 66.2 7.7 83.1 76.6/83.0 82.9/59.8 79.3 / 79.2 71.4/71.3 78.6 59.2 65.1 Table 4: Baseline performance on the GLUE task test sets. For MNLI, we report accuracy on the matched and mismatched test sets. For MRPC and Quora, we report accuracy and F1. For STS-B, we report Pearson and Spearman correlation. For CoLA, we report Matthews correlation. For all other tasks we report accuracy. All values are scaled by 100. A similar table is presented on the online platform.

		Single Sentence	Similarity and Paraphrase	Natural Language Inference
Model	Avg	CoLA	SST-2	MRPC	QQP	STS-B	MNLI	QNLI	RTE	WNLI
Single-Task Training
BiLSTM	63.9	15.7	85.9	69.3/79.4	81.7/61.4	66.0/62.8	70.3/70.8	75.7	52.8	65.1
+ELMo	66.4	35.0	90.2	69.0/80.8	85.7/65.6	64.0/60.2	72.9/73.4	71.7	50.1	65.1
+CoVe	64.0	14.5	88.5	73.4/81.4	83.3/59.4	67.2/64.1	64.5/64.8	75.4	53.5	65.1
+Attn	63.9	15.7	85.9	68.5/80.3	83.5/62.9	59.3/55.8	74.2/73.8	77.2	51.9	65.1
+Attn, ELMo	66.5	35.0	90.2	68.8/80.2	86.5/66.1	55.5/52.5	76.9/76.7	76.7	50.4	65.1
+Attn, CoVe	63.2	14.5	88.5	68.6/79.7	84.1/60.1	57.2/53.6	71.6/71.5	74.5	52.7	65.1
Multi-Task Training
BiLSTM	64.2	11.6	82.8	74.3/81.8	84.2/62.5	70.3/67.8	65.4/66.1	74.6	57.4	65.1
+ELMo	67.7	32.1	89.3	78.0/84.7	82.6/61.1	67.2/67.9	70.3/67.8	75.5	57.4	65.1
+CoVe	62.9	18.5	81.9	71.5/78.7	84.9/60.6	64.4/62.7	65.4/65.7	70.8	52.7	65.1
+Attn	65.6	18.6	83.0	76.2/83.9	82.4/60.1	72.8/70.5	67.6/68.3	74.3	58.4	65.1
+Attn, ELMo	70.0	33.6	90.4	78.0/84.4	84.3/63.1	74.2/72.3	74.1/74.5	79.8	58.9	65.1
+Attn, CoVe	63.1	8.3	80.7	71.8/80.0	83.4/60.5	69.8/68.4	68.1/68.6	72.9	56.0	65.1
Pre-Trained Sentence Representation Models
CBoW	58.9	0.0	80.0	73.4/81.5	79.1/51.4	61.2/58.7	56.0/56.4	72.1	54.1	65.1
Skip-Thought	61.3	0.0	81.8	71.7/80.8	82.2/56.4	71.8/69.7	62.9/62.8	72.9	53.1	65.1
InferSent	63.9	4.5	85.1	74.1/81.2	81.7/59.1	75.9/75.3	66.1/65.7	72.7	58.0	65.1
DisSent	62.0	4.9	83.7	74.1/81.7	82.6/59.5	66.1/64.8	58.7/59.1	73.9	56.4	65.1
GenSen	66.2	7.7	83.1	76.6/83.0	82.9/59.8	79.3/79.2	71.4/71.3	78.6	59.2	65.1

单句 相似度与复述 Natural Language Inference Model Avg CoLA SST-2 MRPC QQP STS-B MNLI QNLI RTE WNLI 单任务训练 BiLSTM 63.9 15.7 85.9 69.3/79.4 81.7/61.4 66.0/62.8 70.3/70.8 75.7 52.8 65.1 +ELMo 66.4 35.0 90.2 69.0/80.8 85.7/65.6 64.0/60.2 72.9/73.4 71.7 50.1 65.1 +CoVe 64.0 14.5 88.5 73.4/81.4 83.3/59.4 67.2/64.1 64.5/64.8 75.4 53.5 65.1 +Attn 63.9 15.7 85.9 68.5/80.3 83.5/62.9 59.3/55.8 74.2/73.8 77.2 51.9 65.1 +Attn, ELMo 66.5 35.0 90.2 68.8/80.2 86.5 / 66.1 55.5/52.5 76.9 / 76.7 76.7 50.4 65.1 +Attn, CoVe 63.2 14.5 88.5 68.6/79.7 84.1/60.1 57.2/53.6 71.6/71.5 74.5 52.7 65.1 多任务训练 BiLSTM 64.2 11.6 82.8 74.3/81.8 84.2/62.5 70.3/67.8 65.4/66.1 74.6 57.4 65.1 +ELMo 67.7 32.1 89.3 78.0 / 84.7 82.6/61.1 67.2/67.9 70.3/67.8 75.5 57.4 65.1 +CoVe 62.9 18.5 81.9 71.5/78.7 84.9/60.6 64.4/62.7 65.4/65.7 70.8 52.7 65.1 +Attn 65.6 18.6 83.0 76.2/83.9 82.4/60.1 72.8/70.5 67.6/68.3 74.3 58.4 65.1 +Attn, ELMo 70.0 33.6 90.4 78.0 /84.4 84.3/63.1 74.2/72.3 74.1/74.5 79.8 58.9 65.1 +Attn, CoVe 63.1 8.3 80.7 71.8/80.0 83.4/60.5 69.8/68.4 68.1/68.6 72.9 56.0 65.1 预训练句表示模型 CBoW 58.9 0.0 80.0 73.4/81.5 79.1/51.4 61.2/58.7 56.0/56.4 72.1 54.1 65.1 Skip-Thought 61.3 0.0 81.8 71.7/80.8 82.2/56.4 71.8/69.7 62.9/62.8 72.9 53.1 65.1 InferSent 63.9 4.5 85.1 74.1/81.2 81.7/59.1 75.9/75.3 66.1/65.7 72.7 58.0 65.1 DisSent 62.0 4.9 83.7 74.1/81.7 82.6/59.5 66.1/64.8 58.7/59.1 73.9 56.4 65.1 GenSen 66.2 7.7 83.1 76.6/83.0 82.9/59.8 79.3 / 79.2 71.4/71.3 78.6 59.2 65.1 表 4：基线在 GLUE 任务测试集上的表现。MNLI 报告匹配与不匹配测试集准确率；MRPC 与 Quora 报告准确率与 F1；STS-B 报告 Pearson 与 Spearman 相关；CoLA 报告 Matthews 相关；其余任务报告准确率。数值均乘以 100。在线平台有类似表格。

		单句	相似度与复述	Natural Language Inference
Model	Avg	CoLA	SST-2	MRPC	QQP	STS-B	MNLI	QNLI	RTE	WNLI
单任务训练
BiLSTM	63.9	15.7	85.9	69.3/79.4	81.7/61.4	66.0/62.8	70.3/70.8	75.7	52.8	65.1
+ELMo	66.4	35.0	90.2	69.0/80.8	85.7/65.6	64.0/60.2	72.9/73.4	71.7	50.1	65.1
+CoVe	64.0	14.5	88.5	73.4/81.4	83.3/59.4	67.2/64.1	64.5/64.8	75.4	53.5	65.1
+Attn	63.9	15.7	85.9	68.5/80.3	83.5/62.9	59.3/55.8	74.2/73.8	77.2	51.9	65.1
+Attn, ELMo	66.5	35.0	90.2	68.8/80.2	86.5/66.1	55.5/52.5	76.9/76.7	76.7	50.4	65.1
+Attn, CoVe	63.2	14.5	88.5	68.6/79.7	84.1/60.1	57.2/53.6	71.6/71.5	74.5	52.7	65.1
多任务训练
BiLSTM	64.2	11.6	82.8	74.3/81.8	84.2/62.5	70.3/67.8	65.4/66.1	74.6	57.4	65.1
+ELMo	67.7	32.1	89.3	78.0/84.7	82.6/61.1	67.2/67.9	70.3/67.8	75.5	57.4	65.1
+CoVe	62.9	18.5	81.9	71.5/78.7	84.9/60.6	64.4/62.7	65.4/65.7	70.8	52.7	65.1
+Attn	65.6	18.6	83.0	76.2/83.9	82.4/60.1	72.8/70.5	67.6/68.3	74.3	58.4	65.1
+Attn, ELMo	70.0	33.6	90.4	78.0/84.4	84.3/63.1	74.2/72.3	74.1/74.5	79.8	58.9	65.1
+Attn, CoVe	63.1	8.3	80.7	71.8/80.0	83.4/60.5	69.8/68.4	68.1/68.6	72.9	56.0	65.1
预训练句表示模型
CBoW	58.9	0.0	80.0	73.4/81.5	79.1/51.4	61.2/58.7	56.0/56.4	72.1	54.1	65.1
Skip-Thought	61.3	0.0	81.8	71.7/80.8	82.2/56.4	71.8/69.7	62.9/62.8	72.9	53.1	65.1
InferSent	63.9	4.5	85.1	74.1/81.2	81.7/59.1	75.9/75.3	66.1/65.7	72.7	58.0	65.1
DisSent	62.0	4.9	83.7	74.1/81.7	82.6/59.5	66.1/64.8	58.7/59.1	73.9	56.4	65.1
GenSen	66.2	7.7	83.1	76.6/83.0	82.9/59.8	79.3/79.2	71.4/71.3	78.6	59.2	65.1

6 Benchmark Results

6 基准结果

We train three runs of each model and evaluate the run with the best macro-average development set performance (see Table 6 in Appendix C). For single-task and sentence representation models, we evaluate the best run for each individual task. We present performance on the main benchmark tasks in Table 4.

每个模型训练三次，取开发集宏平均最好的一次评估（见附录 C 表 6）。对单任务与句表示模型，每任务取各自最佳一次。主基准任务表现见表 4。

We find that multi-task training yields better overall scores over single-task training amongst models using attention or ELMo. Attention generally has negligible or negative aggregate effect in single task training, but helps in multi-task training. We see a consistent improvement in using ELMo embeddings in place of GloVe or CoVe embeddings, particularly for single-sentence tasks. Using CoVe has mixed effects over using only GloVe.

我们发现：在使用注意力或 ELMo 的模型中，多任务训练总体优于单任务训练。注意力在单任务训练中总体影响可忽略或为负，但在多任务训练中有帮助。用 ELMo 嵌入替代 GloVe 或 CoVe 带来稳定提升，单句任务尤甚。相对仅用 GloVe，使用 CoVe 效果参差。

Among the pre-trained sentence representation models, we observe fairly consistent gains moving from CBoW to Skip-Thought to Infersent and GenSen. Relative to the models trained directly on the GLUE tasks, InferSent is competitive and GenSen outperforms all but the two best.

在预训练句表示模型中，从 CBoW 到 Skip-Thought 再到 InferSent 与 GenSen，提升较一致。相对直接在 GLUE 任务上训练的模型，InferSent 具竞争力，GenSen 超过除最好两个以外的所有模型。

Looking at results per task, we find that the sentence representation models substantially underperform on CoLA compared to the models directly trained on the task. On the other hand, for STS-B, models trained directly on the task lag significantly behind the performance of the best sentence representation model. Finally, there are tasks for which no model does particularly well. On WNLI, no model exceeds most-frequent-class guessing (65.1%) and we substitute the model predictions for the most-frequent baseline. On RTE and in aggregate, even our best baselines leave room for improvement. These early results indicate that solving GLUE is beyond the capabilities of current models and methods.

分任务看，句表示模型在 CoLA 上明显弱于直接在该任务上训练的模型；而在 STS-B 上，直接训练的模型显著落后于最佳句表示模型。也有任务上谁都不好：WNLI 上无模型超过多数类猜测（65.1%），我们改用多数类基线预测。在 RTE 与总体分数上，即便最佳基线仍有改进空间。这些早期结果表明：解决 GLUE 超出了当时模型与方法的能力。

R_{3}

		Coarse-Grained	Fine-Grained
Model	All	LS	PAS	L	K	UQuant	MNeg	2Neg	Coref	Restr	Down
Single-Task Training
BiLSTM	21	25	24	16	16	70	53	4	21	-15	12
+ELMo	20	20	21	14	17	70	20	42	33	-26	-3
+CoVe	21	19	23	20	18	71	47	-1	33	-15	8
+Attn	25	24	30	20	14	50	47	21	38	-8	-3
+Attn, ELMo	28	30	35	23	14	85	20	42	33	-26	-3
+Attn, CoVe	24	29	29	18	12	77	50	1	18	-1	12
Multi-Task Training
BiLSTM	20	13	24	14	22	71	17	-8	31	-15	8
+ELMo	21	20	21	19	21	71	60	2	22	0	12
+CoVe	18	15	11	18	27	71	40	7	40	0	8
+Attn	18	13	24	11	16	71	1	-12	31	-15	8
+Attn, ELMo	22	18	26	13	19	70	27	5	31	-26	-3
+Attn, CoVe	18	16	25	16	13	71	26	-8	33	9	8
Pre-Trained Sentence Representation Models
CBoW	9	6	13	5	10	3	0	13	28	-15	-11
Skip-Thought	12	2	23	11	9	61	6	-2	30	-15	0
InferSent	18	20	20	15	14	77	50	-20	15	-15	-9
DisSent	16	16	19	13	15	70	43	-11	20	-36	-09
GenSen	20	28	26	14	12	78	57	2	21	-15	12

R_{3}

		Coarse-Grained	Fine-Grained
Model	All	LS	PAS	L	K	UQuant	MNeg	2Neg	Coref	Restr	Down
单任务训练
BiLSTM	21	25	24	16	16	70	53	4	21	-15	12
+ELMo	20	20	21	14	17	70	20	42	33	-26	-3
+CoVe	21	19	23	20	18	71	47	-1	33	-15	8
+Attn	25	24	30	20	14	50	47	21	38	-8	-3
+Attn, ELMo	28	30	35	23	14	85	20	42	33	-26	-3
+Attn, CoVe	24	29	29	18	12	77	50	1	18	-1	12
多任务训练
BiLSTM	20	13	24	14	22	71	17	-8	31	-15	8
+ELMo	21	20	21	19	21	71	60	2	22	0	12
+CoVe	18	15	11	18	27	71	40	7	40	0	8
+Attn	18	13	24	11	16	71	1	-12	31	-15	8
+Attn, ELMo	22	18	26	13	19	70	27	5	31	-26	-3
+Attn, CoVe	18	16	25	16	13	71	26	-8	33	9	8
预训练句表示模型
CBoW	9	6	13	5	10	3	0	13	28	-15	-11
Skip-Thought	12	2	23	11	9	61	6	-2	30	-15	0
InferSent	18	20	20	15	14	77	50	-20	15	-15	-9
DisSent	16	16	19	13	15	70	43	-11	20	-36	-09
GenSen	20	28	26	14	12	78	57	2	21	-15	12

7 Analysis

7 分析

We analyze the baselines by evaluating each model’s MNLI classifier on the diagnostic set to get a better sense of their linguistic capabilities. Results are presented in Table 5.

为更好把握基线的语言能力，我们把各模型的 MNLI 分类器放到诊断集上评估。结果见表 5。

Coarse Categories Overall performance is low for all models: The highest total score of 28 still denotes poor absolute performance. Performance tends to be higher on Predicate-Argument Structure and lower on Logic, though numbers are not closely comparable across categories. Unlike on the main benchmark, the multi-task models are almost always outperformed by their single-task counterparts. This is perhaps unsurprising, since with our simple multi-task training regime, there is likely some destructive interference between MNLI and the other tasks. The models trained on the GLUE tasks largely outperform the pretrained sentence representation models, with the exception of GenSen. Using attention has a greater influence on diagnostic scores than using ELMo or CoVe, which we take to indicate that attention is especially important for generalization in NLI.

粗粒度类别 所有模型总体表现偏低：最高总分 28 仍表示绝对水平很差。谓词论元结构往往较高、逻辑较低，但跨类数字不宜直接比较。与主基准不同，多任务模型几乎总被对应单任务模型超过。这或许不意外：在我们简单的多任务训练机制下，MNLI 与其他任务间可能存在破坏性干扰。在 GLUE 任务上训练的模型大体优于预训练句表示模型，GenSen 是例外。注意力对诊断分数的影响大于 ELMo 或 CoVe，我们视之为注意力对 NLI 泛化尤为重要的迹象。

Fine-Grained Subcategories Most models handle universal quantification relatively well. Looking at relevant examples, it seems that relying on lexical cues such as “all” often suffices for good performance. Similarly, lexical cues often provide good signal in morphological negation examples.

细粒度子类 多数模型对全称量化处理相对较好。查看相关例子可见，依赖「all」等词汇线索往往就够。类似地，形态否定例子中词汇线索也常提供好信号。

We observe varying weaknesses between models. Double negation is especially difficult for the GLUE-trained models that only use GloVe embeddings. This is ameliorated by ELMo, and to some degree CoVe. Also, attention has mixed effects on overall results, and models with attention tend to struggle with downward monotonicity. Examining their predictions, we found that the models are sensitive to hypernym/hyponym substitution and word deletion as a signal of entailment, but predict it in the wrong direction (as if the substituted/deleted word were in an upward monotone context). This is consistent with recent findings by McCoy & Linzen 2019 that these systems use the subsequence relation between premise and hypothesis as a heuristic shortcut. Restrictivity examples, which often depend on nuances of quantifier scope, are especially difficult for almost all models.

模型之间弱点各异。双重否定对仅用 GloVe 的 GLUE 训练模型尤其难；ELMo 能缓解，CoVe 也有一定帮助。注意力对总体结果影响参差，带注意力的模型往往在向下单调性上吃力。检查预测发现：模型对上下位替换与删词作为蕴含信号敏感，但预测方向错误（仿佛被替换/删除的词处于向上单调语境）。这与 McCoy & Linzen 2019 的发现一致——这些系统把前提与假设的子序列关系当作启发式捷径。限制性例子常依赖量词辖域细微差别，几乎对所有模型都特别难。

Overall, there is evidence that going beyond sentence-to-vector representations, e.g. with an attention mechanism, might aid performance on out-of-domain data, and that transfer methods like ELMo and CoVe encode linguistic information specific to their supervision signal. However, increased representational capacity may lead to overfitting, such as the failure of attention models in downward monotone contexts. We expect that our platform and diagnostic dataset will be useful for similar analyses in the future, so that model designers can better understand their models’ generalization behavior and implicit knowledge.

总体而言，有证据表明：超越句到向量表示（例如引入注意力机制）可能有助于域外数据表现；ELMo、CoVe 等迁移方法编码了与其监督信号相关的特定语言信息。但表征能力增强也可能导致过拟合，如注意力模型在向下单调语境中的失败。我们期望平台与诊断集将来可用于类似分析，使模型设计者更好理解泛化行为与隐含知识。

8 Conclusion

8 结论

We introduce GLUE, a platform and collection of resources for evaluating and analyzing natural language understanding systems. We find that, in aggregate, models trained jointly on our tasks see better performance than the combined performance of models trained for each task separately. We confirm the utility of attention mechanisms and transfer learning methods such as ELMo in NLU systems, which combine to outperform the best sentence representation models on the GLUE benchmark, but still leave room for improvement. When evaluating these models on our diagnostic dataset, we find that they fail (often spectacularly) on many linguistic phenomena, suggesting possible avenues for future work. In sum, the question of how to design general-purpose NLU models remains unanswered, and we believe that GLUE can provide fertile soil for addressing this challenge.

我们提出 GLUE：评估与分析自然语言理解系统的平台与资源集合。总体上看，在我们任务上联合训练的模型优于各任务单独训练模型表现的组合。我们确认注意力机制以及 ELMo 等迁移学习方法在 NLU 系统中的效用：二者结合可超过 GLUE 上最佳句表示模型，但仍有改进空间。在诊断集上评估时，模型在许多语言现象上失败（往往很惨），提示了可能的未来方向。总而言之，如何设计通用 NLU 模型仍未有答案；我们相信 GLUE 能为应对这一挑战提供肥沃土壤。

Acknowledgments

致谢

We thank Ellie Pavlick, Tal Linzen, Kyunghyun Cho, and Nikita Nangia for their comments on this work at its early stages, and we thank Ernie Davis, Alex Warstadt, and Quora’s Nikhil Dandekar and Kornel Csernai for providing access to private evaluation data. This project has benefited from financial support to SB by Google, Tencent Holdings, and Samsung Research, and to AW from AdeptMind and an NSF Graduate Research Fellowship.

感谢 Ellie Pavlick、Tal Linzen、Kyunghyun Cho 与 Nikita Nangia 的有益讨论与反馈，以及 Sam Bowman 实验室成员在诊断数据标注上的帮助。本工作部分得到 DARPA D18AP00042、Alfred P. Sloan 基金会与 NSF 1352440 支持。任何观点、发现、结论或建议均为作者个人，并不必然代表资助方观点。

References

参考文献

Single Sentence Similarity and Paraphrase Natural Language Inference Model Avg CoLA SST-2 MRPC QQP STS-B MNLI QNLI RTE WNLI Single-Task Training BiLSTM 66.7 17.6 87.5 77.9/85.1 85.3/82.0 71.6/72.0 66.7 77.0 58.5 56.3 +ELMo 68.7 44.1 91.5 70.8/82.3 88.0/84.3 70.3/70.5 68.6 71.2 53.4 56.3 +CoVe 66.8 25.1 89.2 76.5/83.4 86.2/81.8 70.7/70.8 62.4 74.4 59.6 54.9 +Attn 66.9 17.6 87.5 72.8/82.9 87.7/83.9 66.6/66.7 70.0 77.2 58.5 60.6 +Attn, ELMo 67.9 44.1 91.5 71.1/82.1 87.8/83.6 57.9/56.1 72.4 75.2 52.7 56.3 +Attn, CoVe 65.6 25.1 89.2 72.8/82.4 86.1/81.3 59.4/58.0 67.9 72.5 58.1 57.7 Multi-Task Training BiLSTM 60.0 18.6 82.3 75.0/82.7 84.4/79.3 69.0/66.9 65.6 74.9 59.9 9.9 +ELMo 63.1 26.4 90.9 80.2/86.7 84.2/79.7 72.9/71.5 67.4 76.0 55.6 14.1 +CoVe 59.3 9.8 82.0 73.8/81.0 83.4/76.6 64.5/61.9 65.5 70.4 52.7 32.4 +Attn 60.5 15.2 83.1 77.5/85.1 82.6/77.2 72.4/70.5 68.0 73.7 61.7 9.9 +Attn, ELMo 67.3 36.7 91.1 80.6/86.6 84.6/79.6 74.4/72.9 74.6 80.4 61.4 22.5 +Attn, CoVe 61.4 17.4 82.1 71.3/80.1 83.4/77.7 68.6/66.7 68.2 73.2 58.5 29.6 Pre-Trained Sentence Representation Models CBoW 61.4 4.6 79.5 75.0/83.7 75.0/65.5 70.6/71.1 57.1 62.5 71.9 56.3 Skip-Thought 61.8 0.0 82.0 76.2/84.3 78.9/70.7 74.8/74.8 63.4 58.5 73.4 49.3 InferSent 65.7 8.6 83.9 76.5/84.1 81.7/75.9 80.2/80.4 67.8 63.5 71.5 56.3 DisSent 63.8 11.7 82.5 77.0/84.4 81.8/75.6 68.9/69.0 61.2 59.9 73.9 56.3 GenSen 67.8 10.3 87.2 80.4/86.2 82.6/76.6 81.3/81.8 71.4 62.5 78.4 56.3 Table 6: Baseline performance on the GLUE tasks’ development sets. For MNLI, we report accuracy averaged over the matched and mismatched test sets. For MRPC and QQP, we report accuracy and F1. For STS-B, we report Pearson and Spearman correlation. For CoLA, we report Matthews correlation. For all other tasks we report accuracy. All values are scaled by 100.

		Single Sentence	Similarity and Paraphrase	Natural Language Inference
Model	Avg	CoLA	SST-2	MRPC	QQP	STS-B	MNLI	QNLI	RTE	WNLI
Single-Task Training
BiLSTM	66.7	17.6	87.5	77.9/85.1	85.3/82.0	71.6/72.0	66.7	77.0	58.5	56.3
+ELMo	68.7	44.1	91.5	70.8/82.3	88.0/84.3	70.3/70.5	68.6	71.2	53.4	56.3
+CoVe	66.8	25.1	89.2	76.5/83.4	86.2/81.8	70.7/70.8	62.4	74.4	59.6	54.9
+Attn	66.9	17.6	87.5	72.8/82.9	87.7/83.9	66.6/66.7	70.0	77.2	58.5	60.6
+Attn, ELMo	67.9	44.1	91.5	71.1/82.1	87.8/83.6	57.9/56.1	72.4	75.2	52.7	56.3
+Attn, CoVe	65.6	25.1	89.2	72.8/82.4	86.1/81.3	59.4/58.0	67.9	72.5	58.1	57.7
Multi-Task Training
BiLSTM	60.0	18.6	82.3	75.0/82.7	84.4/79.3	69.0/66.9	65.6	74.9	59.9	9.9
+ELMo	63.1	26.4	90.9	80.2/86.7	84.2/79.7	72.9/71.5	67.4	76.0	55.6	14.1
+CoVe	59.3	9.8	82.0	73.8/81.0	83.4/76.6	64.5/61.9	65.5	70.4	52.7	32.4
+Attn	60.5	15.2	83.1	77.5/85.1	82.6/77.2	72.4/70.5	68.0	73.7	61.7	9.9
+Attn, ELMo	67.3	36.7	91.1	80.6/86.6	84.6/79.6	74.4/72.9	74.6	80.4	61.4	22.5
+Attn, CoVe	61.4	17.4	82.1	71.3/80.1	83.4/77.7	68.6/66.7	68.2	73.2	58.5	29.6
Pre-Trained Sentence Representation Models
CBoW	61.4	4.6	79.5	75.0/83.7	75.0/65.5	70.6/71.1	57.1	62.5	71.9	56.3
Skip-Thought	61.8	0.0	82.0	76.2/84.3	78.9/70.7	74.8/74.8	63.4	58.5	73.4	49.3
InferSent	65.7	8.6	83.9	76.5/84.1	81.7/75.9	80.2/80.4	67.8	63.5	71.5	56.3
DisSent	63.8	11.7	82.5	77.0/84.4	81.8/75.6	68.9/69.0	61.2	59.9	73.9	56.3
GenSen	67.8	10.3	87.2	80.4/86.2	82.6/76.6	81.3/81.8	71.4	62.5	78.4	56.3

单句 相似度与复述 Natural Language Inference Model Avg CoLA SST-2 MRPC QQP STS-B MNLI QNLI RTE WNLI 单任务训练 BiLSTM 66.7 17.6 87.5 77.9/85.1 85.3/82.0 71.6/72.0 66.7 77.0 58.5 56.3 +ELMo 68.7 44.1 91.5 70.8/82.3 88.0/84.3 70.3/70.5 68.6 71.2 53.4 56.3 +CoVe 66.8 25.1 89.2 76.5/83.4 86.2/81.8 70.7/70.8 62.4 74.4 59.6 54.9 +Attn 66.9 17.6 87.5 72.8/82.9 87.7/83.9 66.6/66.7 70.0 77.2 58.5 60.6 +Attn, ELMo 67.9 44.1 91.5 71.1/82.1 87.8/83.6 57.9/56.1 72.4 75.2 52.7 56.3 +Attn, CoVe 65.6 25.1 89.2 72.8/82.4 86.1/81.3 59.4/58.0 67.9 72.5 58.1 57.7 多任务训练 BiLSTM 60.0 18.6 82.3 75.0/82.7 84.4/79.3 69.0/66.9 65.6 74.9 59.9 9.9 +ELMo 63.1 26.4 90.9 80.2/86.7 84.2/79.7 72.9/71.5 67.4 76.0 55.6 14.1 +CoVe 59.3 9.8 82.0 73.8/81.0 83.4/76.6 64.5/61.9 65.5 70.4 52.7 32.4 +Attn 60.5 15.2 83.1 77.5/85.1 82.6/77.2 72.4/70.5 68.0 73.7 61.7 9.9 +Attn, ELMo 67.3 36.7 91.1 80.6/86.6 84.6/79.6 74.4/72.9 74.6 80.4 61.4 22.5 +Attn, CoVe 61.4 17.4 82.1 71.3/80.1 83.4/77.7 68.6/66.7 68.2 73.2 58.5 29.6 预训练句表示模型 CBoW 61.4 4.6 79.5 75.0/83.7 75.0/65.5 70.6/71.1 57.1 62.5 71.9 56.3 Skip-Thought 61.8 0.0 82.0 76.2/84.3 78.9/70.7 74.8/74.8 63.4 58.5 73.4 49.3 InferSent 65.7 8.6 83.9 76.5/84.1 81.7/75.9 80.2/80.4 67.8 63.5 71.5 56.3 DisSent 63.8 11.7 82.5 77.0/84.4 81.8/75.6 68.9/69.0 61.2 59.9 73.9 56.3 GenSen 67.8 10.3 87.2 80.4/86.2 82.6/76.6 81.3/81.8 71.4 62.5 78.4 56.3 Table 6: Baseline performance on the GLUE tasks’ development sets. For MNLI, we report accuracy averaged over the matched and mismatched test sets. For MRPC and QQP, we report accuracy and F1. For STS-B, we report Pearson and Spearman correlation. For CoLA, we report Matthews correlation. For all other tasks we report accuracy. All values are scaled by 100.

		单句	相似度与复述	Natural Language Inference
Model	Avg	CoLA	SST-2	MRPC	QQP	STS-B	MNLI	QNLI	RTE	WNLI
单任务训练
BiLSTM	66.7	17.6	87.5	77.9/85.1	85.3/82.0	71.6/72.0	66.7	77.0	58.5	56.3
+ELMo	68.7	44.1	91.5	70.8/82.3	88.0/84.3	70.3/70.5	68.6	71.2	53.4	56.3
+CoVe	66.8	25.1	89.2	76.5/83.4	86.2/81.8	70.7/70.8	62.4	74.4	59.6	54.9
+Attn	66.9	17.6	87.5	72.8/82.9	87.7/83.9	66.6/66.7	70.0	77.2	58.5	60.6
+Attn, ELMo	67.9	44.1	91.5	71.1/82.1	87.8/83.6	57.9/56.1	72.4	75.2	52.7	56.3
+Attn, CoVe	65.6	25.1	89.2	72.8/82.4	86.1/81.3	59.4/58.0	67.9	72.5	58.1	57.7
多任务训练
BiLSTM	60.0	18.6	82.3	75.0/82.7	84.4/79.3	69.0/66.9	65.6	74.9	59.9	9.9
+ELMo	63.1	26.4	90.9	80.2/86.7	84.2/79.7	72.9/71.5	67.4	76.0	55.6	14.1
+CoVe	59.3	9.8	82.0	73.8/81.0	83.4/76.6	64.5/61.9	65.5	70.4	52.7	32.4
+Attn	60.5	15.2	83.1	77.5/85.1	82.6/77.2	72.4/70.5	68.0	73.7	61.7	9.9
+Attn, ELMo	67.3	36.7	91.1	80.6/86.6	84.6/79.6	74.4/72.9	74.6	80.4	61.4	22.5
+Attn, CoVe	61.4	17.4	82.1	71.3/80.1	83.4/77.7	68.6/66.7	68.2	73.2	58.5	29.6
预训练句表示模型
CBoW	61.4	4.6	79.5	75.0/83.7	75.0/65.5	70.6/71.1	57.1	62.5	71.9	56.3
Skip-Thought	61.8	0.0	82.0	76.2/84.3	78.9/70.7	74.8/74.8	63.4	58.5	73.4	49.3
InferSent	65.7	8.6	83.9	76.5/84.1	81.7/75.9	80.2/80.4	67.8	63.5	71.5	56.3
DisSent	63.8	11.7	82.5	77.0/84.4	81.8/75.6	68.9/69.0	61.2	59.9	73.9	56.3
GenSen	67.8	10.3	87.2	80.4/86.2	82.6/76.6	81.3/81.8	71.4	62.5	78.4	56.3

Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2018. Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the International Conference on Computer Vision , pp. 19-27, 2015.

Appendix A Additional Benchmark Details

附录 A 基准补充细节

QNLI To construct a balanced dataset, we select all pairs in which the most similar sentence to the question was not the answer sentence, as well as an equal amount of cases in which the correct sentence was the most similar to the question, but another distracting sentence was a close second. Our similarity metric is based on CBoW representations with pre-trained GloVe embeddings. This approach to converting pre-existing datasets into NLI format is closely related to recent work by White et al. 2017, as well as to the original motivation for textual entailment presented by Dagan et al. 2006. Both argue that many NLP tasks can be productively reduced to textual entailment.

QNLI 为构造均衡数据集，我们选取「与问题最相似的句子 不是 答案句」的全部对，以及等量的「正确句与问题最相似、但另一干扰句紧随其后」的情形。相似度基于带预训练 GloVe 的 CBoW 表示。这种把既有数据转成 NLI 格式的做法，与 White et al. 2017 的近期工作以及 Dagan et al. 2006 提出文本蕴含的原始动机密切相关。二者都主张许多 NLP 任务可富有成效地归约为文本蕴含。

Appendix B Additional Baseline Details

附录 B 基线补充细节

B.1 Attention Mechanism

B.1 注意力机制

u_{1},u_{2},\dots,u_{M}

u_{1},u_{2},\dots,u_{M}

B.2 Training

B.2 训练

10^{-3}

10^{-3}

B.3 Sentence Representation Models

B.3 句表示模型

We evaluate the following sentence representation models:

我们评估下列句表示模型：

CBoW, the average of the GloVe embeddings of the tokens in the sentence.

CBoW：句中各词 GloVe 嵌入的平均。

Skip-Thought (Kiros et al. 2015), a sequence-to-sequence(s) model trained to generate the previous and next sentences given the middle sentence. We use the original pre-trained model[^7] trained on sequences of sentences from the Toronto Book Corpus (Zhu et al. 2015, TBC).

Skip-Thought（Kiros et al. 2015）：给定中间句生成前后句的序列到序列模型。我们使用在 Toronto Book Corpus（Zhu et al. 2015，TBC）句序列上训练的原始预训练模型。[^7]

InferSent (Conneau et al. 2017), a BiLSTM with max-pooling trained on MNLI and SNLI.

InferSent（Conneau et al. 2017）：在 MNLI 与 SNLI 上训练的带最大池化 BiLSTM。

DisSent (Nie et al. 2017), a BiLSTM with max-pooling trained to predict the discourse marker (because, so, etc.) relating two sentences on data derived from TBC. We use the variant trained for eight-way classification.

DisSent（Nie et al. 2017）：在由 TBC 导出的数据上训练、预测连接两句的话语标记（ because 、 so 等）的带最大池化 BiLSTM。我们使用八分类变体。

GenSen (Subramanian et al. 2018), a sequence-to-sequence model trained on a variety of supervised and unsupervised objectives. We use the variant of the model trained on both MNLI and SNLI, the Skip-Thought objective on TBC, and a constituency parsing objective on the Billion Word Benchmark.

GenSen（Subramanian et al. 2018）：在多种有监督与无监督目标上训练的序列到序列模型。我们使用同时在 MNLI 与 SNLI、TBC 上的 Skip-Thought 目标、以及 Billion Word Benchmark 上的成分句法分析目标训练的变体。

We train task-specific classifiers on top of frozen sentence encoders, using the default parameters from SentEval. See https://github.com/nyu-mll/SentEval for details and code.

我们在冻结的句编码器之上训练任务专用分类器，使用 SentEval 默认参数。细节与代码见 https://github.com/nyu-mll/SentEval。

Appendix C Development Set Results

附录 C 开发集结果

The GLUE website limits users to two submissions per day in order to avoid overfitting to the private test data. To provide a reference for future work on GLUE, we present the best development set results achieved by our baselines in Table 6.

为避免在私有测试数据上过拟合，GLUE 网站限制用户每天提交两次。为给后续 GLUE 工作提供参考，我们在表 6 给出基线取得的最佳开发集结果。

Appendix D Benchmark Website Details

附录 D 基准网站细节

Figure 1: The benchmark website leaderboard. An expanded view shows additional details about each submission, including a brief prose description and parameter count.

图 1：基准网站排行榜。展开视图显示每条提交的更多细节，包括简短文字描述与参数量。

GLUE’s online platform is built using React, Redux and TypeScript. We use Google Firebase for data storage and Google Cloud Functions to host and run our grading script when a submission is made. Figure 1 shows the visual presentation of our baselines on the leaderboard.

GLUE 在线平台用 React、Redux 与 TypeScript 构建。数据存储用 Google Firebase，提交时用 Google Cloud Functions 托管并运行评分脚本。图 1 展示基线在排行榜上的视觉呈现。

Appendix E Additional Diagnostic Data Details

附录 E 诊断数据补充细节

The dataset is designed to allow for analyzing many levels of natural language understanding, from word meaning and sentence structure to high-level reasoning and application of world knowledge. To make this kind of analysis feasible, we first identify four broad categories of phenomena: Lexical Semantics, Predicate-Argument Structure, Logic, and Knowledge. However, since these categories are vague, we divide each into a larger set of fine-grained subcategories. Descriptions of all of the fine-grained categories are given in the remainder of this section. These categories are just one lens that can be used to understand linguistic phenomena and entailment, and there is certainly room to argue about how examples should be categorized, what the categories should be, etc. These categories are not based on any particular linguistic theory, but broadly based on issues that linguists have often identified and modeled in the study of syntax and semantics.

该数据集旨在支持从词义、句子结构到高层推理与世界知识应用等多层次 NLU 分析。为使分析可行，我们先划出四大现象类别：词汇语义、谓词论元结构、逻辑与知识。因这些类别较笼统，再各自拆成更多细粒度子类。本节余下给出全部细粒度类别说明。这些类别只是理解语言现象与蕴含的一种视角，例子如何归类、类别应如何设等尽可讨论。类别不绑定某一语言学理论，大致基于语法学家与语义学家常识别并建模的问题。

The dataset is provided not as a benchmark, but as an analysis tool to paint in broad strokes the kinds of phenomena a model may or may not capture, and to provide a set of examples that can serve for error analysis, qualitative model comparison, and development of adversarial examples that expose a model’s weaknesses. Because the distribution of language is somewhat arbitrary, it will not be helpful to compare performance of the same model on different categories. Rather, we recommend comparing performance that different models score on the same category, or using the reported scores as a guide for error analysis.

该数据集并非作为基准提供，而是分析工具：粗线条描绘模型可能/未能捕获的现象，并提供可用于误差分析、定性模型比较与开发暴露弱点的对抗例子的样例集。因语言分布有一定任意性，比较同一模型在不同类别上的表现帮助不大。我们建议比较不同模型在同一类别上的分数，或把报告分数当作误差分析的向导。

We show coarse-grain category counts and label distributions of the diagnostic set in Table 7.

诊断集按粗粒度类别的计数与标签分布见表 7。

Category Count % Neutral % Contradiction % Entailment Lexical Semantics 368 31.0 27.2 41.8 Predicate-Argument Structure 424 37.0 13.7 49.3 Logic 364 37.6 26.9 35.4 Knowledge 284 26.4 31.7 41.9 Table 7: Diagnostic dataset statistics by coarse-grained category. Note that some examples may be tagged with phenomena belonging to multiple categories.

Category	Count	% Neutral	% Contradiction	% Entailment
Lexical Semantics	368	31.0	27.2	41.8
Predicate-Argument Structure	424	37.0	13.7	49.3
Logic	364	37.6	26.9	35.4
Knowledge	284	26.4	31.7	41.9

类别 数量 % Neutral % Contradiction % Entailment Lexical Semantics 368 31.0 27.2 41.8 Predicate-Argument Structure 424 37.0 13.7 49.3 Logic 364 37.6 26.9 35.4 Knowledge 284 26.4 31.7 41.9 表 7：按粗粒度类别的诊断集统计。注意：有些例子可能标有属于多个类别的现象。

类别	数量	% Neutral	% Contradiction	% Entailment
Lexical Semantics	368	31.0	27.2	41.8
Predicate-Argument Structure	424	37.0	13.7	49.3
Logic	364	37.6	26.9	35.4
Knowledge	284	26.4	31.7	41.9

E.1 Lexical Semantics

E.1 词汇语义

These phenomena center on aspects of word meaning.

这些现象围绕词义的各个方面。

Lexical Entailment Entailment can be applied not only on the sentence level, but the word level. For example, we say “dog” lexically entails “animal” because anything that is a dog is also an animal, and “dog” lexically contradicts “cat” because it is impossible to be both at once. This relationship applies to many types of words (nouns, adjectives, verbs, many prepositions, etc.) and the relationship between lexical and sentential entailment has been deeply explored, e.g., in systems of natural logic. This connection often hinges on monotonicity in language, so many Lexical Entailment examples will also be tagged with one of the Monotone categories, though we do not do this in every case (see Monotonicity, under Logic).

词汇蕴含 蕴含不仅可作用在句子层面，也可作用在词层面。例如，我们说 “dog” 词汇蕴含 “animal”，因为凡是狗的也是动物；“dog” 词汇矛盾于 “cat”，因为不可能同时是两者。该关系适用于多种词类（名词、形容词、动词、许多介词等），词汇蕴含与句子蕴含之间的关系在自然逻辑等体系中被深入探讨。这种联系常依赖语言中的单调性，因而许多词汇蕴含例子也会标上某一单调类别，但我们并非每例都如此标注（见「逻辑」下的单调性）。

Morphological Negation This is a special case of lexical contradiction where one word is derived from the other: from “affordable” to “unaffordable”, “agree” to “disagree”, etc. We also include examples like “ever” and “never”. We also label these examples with Negation or Double Negation, since they can be viewed as involving a word-level logical negation.

形态否定 这是词汇矛盾的特例：一个词由另一个派生而来，如从 “affordable” 到 “unaffordable”、从 “agree” 到 “disagree” 等。也包括 “ever” 与 “never” 这类例子。我们还为其标注否定或双重否定，因为它们可视为涉及词级逻辑否定。

Factivity Propositions appearing in a sentence may be in any entailment relation with the sentence as a whole, depending on the context in which they appear. In many cases, this is determined by lexical triggers (usually verbs or adverbs) in the sentence. For example,

事实性 句中出现的命题相对整句可处于任意蕴含关系，取决于出现语境。很多时候由句中的词汇触发语（通常是动词或副词）决定。例如：

• “I recognize that X” entails “X”.

• “I recognize that X” 蕴含 “X”。

• “I did not recognize that X” entails “X”.

• “I did not recognize that X” 蕴含 “X”。

• “I believe that X” does not entail “X”.

• “I believe that X” 不蕴含 “X”。

• “I am refusing to do X” contradicts “I am doing X”.

• “I am refusing to do X” 与 “I am doing X” 矛盾。

• “I am not refusing to do X” does not contradict “I am doing X”.

• “I am not refusing to do X” 并不与 “I am doing X” 矛盾。

• “I almost finished X” contradicts “I finished X”.

• “I almost finished X” 与 “I finished X” 矛盾。

• “I barely finished X” entails “I finished X”.

• “I barely finished X” 蕴含 “I finished X”。

Constructions like “I recognize that X” are often called factive, since the entailment (of X above, regarded as a presupposition) persists even under negation. Constructions like “I am refusing to do X” above are often called implicative, and are sensitive to negation. There are also cases where a sentence (non-)entails the existence of an entity mentioned in it, for example “I have found a unicorn” entails “A unicorn exists” while “I am looking for a unicorn” doesn’t necessarily entail “A unicorn exists”. Readings where the entity does not necessarily exist are often called intensional readings, since they seem to deal with the properties denoted by a description (its intension) rather than being reducible to the set of entities that match the description (its extension, which in cases of non-existence will be empty).

“I recognize that X” 这类构造常称事实性构造，因为即便在否定下，（上例中 X 作为预设的）蕴含仍成立。“I am refusing to do X” 这类常称隐含性构造，对否定敏感。也有句子（不）蕴含其中所提实体存在的情形，例如 “I have found a unicorn” 蕴含 “A unicorn exists”，而 “I am looking for a unicorn” 并不必然蕴含。实体不一定存在的解读常称内涵解读，因其似乎关乎描述所表示的性质（内涵），而不能归约为匹配该描述的实体集合（外延；在不存在时为空）。

We place all examples involving these phenomena under the label of Factivity. While it often depends on context to determine whether a nested proposition or existence of an entity is entailed by the overall statement, very often it relies heavily on lexical triggers, so we place the category under Lexical Semantics.

凡涉及这些现象的例子，我们都标为事实性。嵌套命题或实体存在是否被总述蕴含虽常依赖语境，但往往严重依赖词汇触发语，故将该类放在词汇语义下。

Symmetry/Collectivity Some propositions denote symmetric relations, while others do not. For example, “John married Gary” entails “Gary married John” but “John likes Gary” does not entail “Gary likes John”. Symmetric relations can often be rephrased by collecting both arguments into the subject: “John met Gary” entails “John and Gary met”. Whether a relation is symmetric, or admits collecting its arguments into the subject, is often determined by its head word (e.g., “like”, “marry” or “meet”), so we classify it under Lexical Semantics.

对称性/集合性 有些命题表示对称关系，有些则否。例如 “John married Gary” 蕴含 “Gary married John”，但 “John likes Gary” 不蕴含 “Gary likes John”。对称关系常可通过把两个论元收进主语来改写：“John met Gary” 蕴含 “John and Gary met”。关系是否对称、或是否允许把论元收进主语，往往由中心词（如 “like”“marry”“meet”）决定，故归入词汇语义。

Redundancy If a word can be removed from a sentence without changing its meaning, that means the word’s meaning was more-or-less adequately expressed by the sentence; so, identifying these cases reflects an understanding of both lexical and sentential semantics.

冗余 若可从句中删去某词而不改变意义，说明该词意义大体已被句子表达；识别这些情形反映对词汇与句子语义的理解。

Named Entities Words often name entities that exist in the world. There are many different kinds of understanding we might wish to understand about these names, including their compositional structure (for example, the “Baltimore Police” is the same as the “Police of the City of Baltimore”) or their real-world referents and acronym expansions (for example, “SNL” is “Saturday Night Live”). This category is closely related to World Knowledge, but focuses on the semantics of names as lexical items rather than background knowledge about their denoted entities.

命名实体 词常为世界上存在的实体命名。我们可能希望理解这些名称的多方面，包括组合结构（例如 “Baltimore Police” 即 “Police of the City of Baltimore”）或其现实指称与缩写展开（例如 “SNL” 即 “Saturday Night Live”）。该类与世界知识密切相关，但聚焦名称作为词汇项的语义，而非关于其所指实体的背景知识。

Quantifiers Logical quantification in natural language is often expressed through lexical triggers such as “every”, “most”, “some”, and “no”. While we reserve the categories in Quantification and Monotonicity for entailments involving operations on these quantifiers and their arguments, we choose to regard the interchangeability of quantifiers (e.g., in many cases “most” entails “many”) as a question of lexical semantics.

量词 自然语言中的逻辑量化常通过 “every”“most”“some”“no” 等词汇触发语表达。对涉及这些量词及其论元上运算的蕴含，我们保留「量化」与「单调性」类别；量词之间的可互换性（例如许多情形下 “most” 蕴含 “many”）则视为词汇语义问题。

E.2 Predicate-Argument Structure

E.2 谓词论元结构

An important component of understanding the meaning of a sentence is understanding how its parts are composed together into a whole. In this category, we address issues across that spectrum, from syntactic ambiguity to semantic roles and coreference.

理解句子意义的重要组成部分，是理解其各部分如何组合成整体。本类覆盖从句法歧义到语义角色与共指的一系列问题。

Syntactic Ambiguity: Relative Clauses, Coordination Scope These two categories deal purely with resolving syntactic ambiguity. Relative clauses and coordination scope are both sources of a great amount of ambiguity in English.

句法歧义：关系从句、并列辖域 这两类纯粹关乎消解句法歧义。关系从句与并列辖域都是英语中大量歧义的来源。

Prepositional phrases Prepositional phrase attachment is a particularly difficult problem that syntactic parsers in NLP systems continue to struggle with. We view it as a problem both of syntax and semantics, since prepositional phrases can express a wide variety of semantic roles and often semantically apply beyond their direct syntactic attachment.

介词短语 介词短语附着仍是 NLP 句法分析器持续挣扎的难题。我们视之为句法与语义双重问题：介词短语可表达多种语义角色，且语义作用范围常超出其直接句法附着点。

Core Arguments Verbs select for particular arguments, particularly subjects and objects, which might be interchangeable depending on the context or the surface form. One example is the ergative alternation: “Jake broke the vase” entails “the vase broke” but “Jake broke the vase” does not entail “Jake broke”. Other rearrangements of core arguments, such as those seen in Symmetry/Collectivity, also fall under the Core Arguments label.

核心论元 动词选择特定论元，尤其是主语与宾语，它们可能随语境或表层形式互换。一例是作格交替：“Jake broke the vase” 蕴含 “the vase broke”，但不蕴含 “Jake broke”。对称性/集合性中见到的其他核心论元重排也归入本标签。

Alternations: Active/Passive, Genitives/Partitives, Nominalization, Datives All four of these categories correspond to syntactic alternations that are known to follow specific patterns in English:

交替：主动/被动、属格/部分格、名词化、与格 这四类对应英语中已知遵循特定模式的句法交替：

• Active/Passive: “I saw him” is equivalent to “He was seen by me” and entails “He was seen”.

• 主动/被动：“I saw him” 等价于 “He was seen by me”，并蕴含 “He was seen”。

• Genitives/Partitives: “the elephant’s foot” is the same thing as “the foot of the elephant”.

• 属格/部分格：“the elephant’s foot” 与 “the foot of the elephant” 是同一事物。

• Nominalization: “I caused him to submit his resignation” entails “I caused the submission of his resignation”.

• 名词化：“I caused him to submit his resignation” 蕴含 “I caused the submission of his resignation”。

• Datives: “I baked him a cake” entails “I baked a cake for him” and “I baked a cake” but not “I baked him”.

• 与格：“I baked him a cake” 蕴含 “I baked a cake for him” 与 “I baked a cake”，但不蕴含 “I baked him”。

Ellipsis/Implicits Often, the argument of a verb or other predicate is omitted (elided) in the text, with the reader filling in the gap. We can construct entailment examples by explicitly filling in the gap with the correct or incorrect referents. For example, the premise “Putin is so entrenched within Russia’s ruling system that many of its members can imagine no other leader” entails “Putin is so entrenched within Russia’s ruling system that many of its members can imagine no other leader than Putin” and contradicts “Putin is so entrenched within Russia’s ruling system that many of its members can imagine no other leader than themselves.”

省略/隐含 动词或其他谓词的论元常在文本中被省略，由读者填补。我们可通过用正确或不正确的指称显式填空来构造蕴含例子。例如，前提 “Putin is so entrenched within Russia’s ruling system that many of its members can imagine no other leader” 蕴含填入 “than Putin” 的版本，并与填入 “than themselves” 的版本矛盾。

This is often regarded as a special case of anaphora, but we decided to split out these cases from explicit anaphora, which is often also regarded as a case of coreference (and attempted to some degree in modern coreference resolution systems).

这常被视为回指的特例，但我们决定把这些情形与显式回指分开；显式回指也常被视为共指（现代共指消解系统会在一定程度上尝试）。

Anaphora/Coreference Coreference refers to when multiple expressions refer to the same entity or event. It is closely related to Anaphora, where the meaning of an expression depends on another (antecedent) expression in context. These two phenomena have significant overlap; for example, pronouns (“she”, “we”, “it”) are anaphors that are co-referent with their antecedents. However, they also may occur independently, such as coreference between two definite noun phrases (e.g., “Theresa May ”and the “British Prime Minister”) that refer to the same entity, or anaphora from a word like “other” which requires an antecedent to distinguish something from. In this category we only include cases where there is an explicit phrase (anaphoric or not) that is co-referent with an antecedent or other phrase. We construct examples for these in much the same way as for Ellipsis/Implicits.

回指/共指 共指指多个表达式指向同一实体或事件。它与回指密切相关：回指中表达式意义依赖语境中另一（先行）表达式。二者有显著重叠；例如代词（“she”“we”“it”）是与先行语共指的回指词。但也可独立出现，如两个有定名词短语共指（例如 “Theresa May” 与 “British Prime Minister”），或 “other” 这类需要先行语来区分对象的回指。本类只纳入存在显式短语（无论是否回指）与先行语或其他短语共指的情形。构造例子的方式与省略/隐含大致相同。

Intersectivity Many modifiers, especially adjectives, allow non-intersective uses, which affect their entailment behavior. For example:

相交性 许多修饰语（尤其形容词）允许非相交用法，影响蕴含行为。例如：

• Intersective: “He is a violinist and an old surgeon” entails “He is an old violinist” and “He is a surgeon”.

• 相交：“He is a violinist and an old surgeon” 蕴含 “He is an old violinist” 与 “He is a surgeon”。

• Non-intersective: “He is a violinist and a skilled surgeon” does not entail “He is a skilled violinist”.

• 非相交：“He is a violinist and a skilled surgeon” 不蕴含 “He is a skilled violinist”。

• Non-intersective: “He is a fake surgeon” does not entail “He is a surgeon”.

• 非相交：“He is a fake surgeon” 不蕴含 “He is a surgeon”。

Generally, an intersective use of a modifier, like “old” in “old men”, is one which may be interpreted as referring to the set of entities with both properties (they are old and they are men). Linguists often formalize this using set intersection, hence the name.

一般而言，修饰语的相交用法（如 “old men” 中的 “old”）可解释为指同时具有两种性质的实体集合（既老又是男人）。语言学家常用集合交形式化，故得名。

Intersectivity is related to Factivity. For example, “fake” may be regarded as a counter-implicative modifier, and these examples will be labeled as such. However, we choose to categorize intersectivity under predicate-argument structure rather than lexical semantics, because generally the same word will admit both intersective and non-intersective uses, so it may be regarded as an ambiguity of argument structure.

相交性与事实性相关。例如 “fake” 可视为反隐含修饰语，这些例子也会如此标注。但我们选择把相交性归入谓词论元结构而非词汇语义，因为同一词通常既可相交也可非相交使用，可视作论元结构歧义。

Restrictivity Restrictivity is most often used to refer to a property of uses of noun modifiers. In particular, a restrictive use of a modifier is one that serves to identify the entity or entities being described, whereas a non-restrictive use adds extra details to the identified entity. The distinction can often be highlighted by entailments:

限制性 限制性多用来指名词修饰语用法的一种性质。限制性用法用于识别被描述的实体；非限制性用法则给已识别实体补充细节。区别常可由蕴含凸显：

• Restrictive: “I finished all of my homework due today” does not entail “I finished all of my homework”.

• 限制性：“I finished all of my homework due today” 不蕴含 “I finished all of my homework”。

• Non-restrictive: “I got rid of all those pesky bedbugs” entails “I got rid of all those bedbugs”.

• 非限制性：“I got rid of all those pesky bedbugs” 蕴含 “I got rid of all those bedbugs”。

Modifiers that are commonly used non-restrictively are appositives, relative clauses starting with “which” or “who”, and expletives (e.g. “pesky”). Non-restrictive uses can appear in many forms.

常作非限制性使用的修饰语包括同位语、以 “which” 或 “who” 开头的关系从句，以及咒骂性形容词（如 “pesky”）。非限制性用法形式多样。

E.3 Logic

E.3 逻辑

With an understanding of the structure of a sentence, there is often a baseline set of shallow conclusions that can be drawn using logical operators and often modeled using the mathematical tools of logic. Indeed, the development of mathematical logic was initially guided by questions about natural language meaning, from Aristotelian syllogisms to Fregean symbols. The notion of entailment is also borrowed from mathematical logic.

在理解句子结构之后，常可用逻辑算子得出一组浅层结论，并往往用数理逻辑工具建模。事实上，数理逻辑的发展最初由自然语言意义问题引导，从亚里士多德三段论到弗雷格符号。蕴含概念也借自数理逻辑。

Propositional Structure: Negation, Double Negation, Conjunction, Disjunction, Conditionals All of the basic operations of propositional logic appear in natural language, and we tag them where they are relevant to our examples:

命题结构：否定、双重否定、合取、析取、条件 命题逻辑的基本运算都出现在自然语言中，我们在与例子相关处加以标注：

• Negation: “The cat sat on the mat” contradicts “The cat did not sit on the mat”.

• 否定：“The cat sat on the mat” 与 “The cat did not sit on the mat” 矛盾。

• Double negation: “The market is not impossible to navigate” entails “The market is possible to navigate”.

• 双重否定：“The market is not impossible to navigate” 蕴含 “The market is possible to navigate”。

• Conjunction: “Temperature and snow consistency must be just right” entails “Temperature must be just right”.

• 合取：“Temperature and snow consistency must be just right” 蕴含 “Temperature must be just right”。

• Disjunction: “Life is either a daring adventure or nothing at all” does not entail, but is entailed by, “Life is a daring adventure”.

• 析取：“Life is either a daring adventure or nothing at all” 不蕴含但被 “Life is a daring adventure” 蕴含。

• Conditionals: “If both apply, they are essentially impossible” does not entail “They are essentially impossible”.

• 条件：“If both apply, they are essentially impossible” 不蕴含 “They are essentially impossible”。

Conditionals are more complicated because their use in language does not always mirror their meaning in logic. For example, they may be used at a higher level than the at-issue assertion: “If you think about it, it’s the perfect reverse psychology tactic” entails “It’s the perfect reverse psychology tactic”.

条件句更复杂，因其在语言中的用法并不总镜像其在逻辑中的意义。例如，它们可能用在高于争议断言的层面：“If you think about it, it’s the perfect reverse psychology tactic” 蕴含 “It’s the perfect reverse psychology tactic”。

Quantification: Universal, Existential Quantifiers are often triggered by words such as “all”, “some”, “many”, and “no”. There is a rich body of work modeling their meaning in mathematical logic with generalized quantifiers. In these two categories, we focus on straightforward inferences from the natural language analogs of universal and existential quantification:

量化：全称、存在 量词常由 “all”“some”“many”“no” 等词触发。用广义量词在数理逻辑中建模其意义的工作很丰富。在这两类中，我们聚焦自然语言中对应全称与存在量化的直接推理：

• Universal: “All parakeets have two wings” entails, but is not entailed by, “My parakeet has two wings”.

• 全称：“All parakeets have two wings” 蕴含但不被 “My parakeet has two wings” 蕴含。

• Existential: “Some parakeets have two wings” does not entail, but is entailed by, “My parakeet has two wings”.

• 存在：“Some parakeets have two wings” 不蕴含但被 “My parakeet has two wings” 蕴含。

Monotonicity: Upward Monotone, Downward Monotone, Non-Monotone Monotonicity is a property of argument positions in certain logical systems. In general, it gives a way of deriving entailment relations between expressions that differ on only one subexpression. In language, it can explain how some entailments propagate through logical operators and quantifiers.

单调性：向上单调、向下单调、非单调 单调性是某些逻辑系统中论元位置的性质。一般而言，它给出仅在一个子表达式上不同的表达式之间推导蕴含关系的方式。在语言中，它可解释一些蕴含如何穿过逻辑算子与量词传播。

For example, “pet” entails “pet squirrel”, which further entails “happy pet squirrel”. We can demonstrate how the quantifiers “a”, “no” and “exactly one” differ with respect to monotonicity:

例如，“pet” 蕴含 “pet squirrel”，后者进一步蕴含 “happy pet squirrel”。我们可用量词 “a”“no” 与 “exactly one” 演示相对单调性的差异：

• “I have a pet squirrel” entails “I have a pet”, but not “I have a happy pet squirrel”.

• “I have a pet squirrel” 蕴含 “I have a pet”，但不蕴含 “I have a happy pet squirrel”。

• “I have no pet squirrels” does not entail “I have no pets”, but does entail “I have no happy pet squirrels”.

• “I have no pet squirrels” 不蕴含 “I have no pets”，但蕴含 “I have no happy pet squirrels”。

• “I have exactly one pet squirrel” entails neither “I have exactly one pet” nor “I have exactly one happy pet squirrel”.

• “I have exactly one pet squirrel” 既不蕴含 “I have exactly one pet”，也不蕴含 “I have exactly one happy pet squirrel”。

In all of these examples, “pet squirrel” appears in what we call the restrictor position of the quantifier. We say:

在所有这些例子中，“pet squirrel” 出现在我们所谓量词的限制语位置。我们说：

• “a” is upward monotone in its restrictor: an entailment in the restrictor yields an entailment of the whole statement.

• “a” 在其限制语上向上单调：限制语中的蕴含导出整句同向蕴含。

• “no” is downward monotone in its restrictor: an entailment in the restrictor yields an entailment of the whole statement in the opposite direction.

• “no” 在其限制语上向下单调：限制语中的蕴含导出整句反向蕴含。

• “exactly one” is non-monotone in its restrictor: entailments in the restrictor do not yield entailments of the whole statement.

• “exactly one” 在其限制语上非单调：限制语中的蕴含不导出整句蕴含。

In this way, entailments between sentences that are built off of entailments of sub-phrases almost always rely on monotonicity judgments; see, for example, Lexical Entailment. However, because this is such a general class of sentence pairs, to keep the Logic category meaningful we do not always tag these examples with monotonicity.

如此，建立在子短语蕴含之上的句间蕴含几乎总依赖单调性判断；例如见词汇蕴含。但因这是极大一类句对，为保持「逻辑」类别有意义，我们并非总给这些例子打上单调性标签。

Richer Logical Structure: Intervals/Numbers, Temporal Some higher-level facets of reasoning have been traditionally modeled using logic, such as actual mathematical reasoning (entailments based off of numbers) and temporal reasoning (which is often modeled as reasoning about a mathematical timeline).

更丰富的逻辑结构：区间/数字、时间 一些更高层推理侧面传统上用逻辑建模，如实际数学推理（基于数字的蕴含）与时间推理（常建模为关于数学时间线的推理）。

• Intervals/Numbers: “I have had more than 2 drinks tonight” entails “I have had more than 1 drink tonight”.

• 区间/数字：“I have had more than 2 drinks tonight” 蕴含 “I have had more than 1 drink tonight”。

• Temporal: “Mary left before John entered” entails “John entered after Mary left”.

• 时间：“Mary left before John entered” 蕴含 “John entered after Mary left”。

E.4 Knowledge

E.4 知识

Strictly speaking, world knowledge and common sense are required on every level of language understanding for disambiguating word senses, syntactic structures, anaphora, and more. So our entire suite (and any test of entailment) does test these features to some degree. However, in these categories, we gather examples where the entailment rests not only on correct disambiguation of the sentences, but also application of extra knowledge, whether concrete knowledge about world affairs or more common-sense knowledge about word meanings or social or physical dynamics.

严格说来，消歧词义、句法结构、回指等每一层语言理解都需要世界知识与常识，因此我们的整套套件（以及任何蕴含测试）在某种程度上都在测这些能力。但在这些类别中，我们收集的是：蕴含不仅依赖对句子的正确消歧，还依赖额外知识应用的例子——无论是关于世事的具体知识，还是关于词义或社会/物理动态的更常识性知识。

World Knowledge In this category we focus on knowledge that can clearly be expressed as facts, as well as broader and less common geographical, legal, political, technical, or cultural knowledge. Examples:

世界知识 本类聚焦可清楚表述为事实的知识，以及更广、较不常见的地理、法律、政治、技术或文化知识。例子：

• “This is the most oniony article I’ve seen on the entire internet” entails “This article reads like satire”.

• “This is the most oniony article I’ve seen on the entire internet” 蕴含 “This article reads like satire”。

• “The reaction was strongly exothermic” entails “The reaction media got very hot”.

• “The reaction was strongly exothermic” 蕴含 “The reaction media got very hot”。

• “There are amazing hikes around Mt. Fuji” entails “There are amazing hikes in Japan” but not “There are amazing hikes in Nepal”.

• “There are amazing hikes around Mt. Fuji” 蕴含 “There are amazing hikes in Japan”，但不蕴含 “There are amazing hikes in Nepal”。

Common Sense In this category we focus on knowledge that is more difficult to express as facts and that we expect to be possessed by most people independent of cultural or educational background. This includes a basic understanding of physical and social dynamics as well as lexical meaning (beyond simple lexical entailment or logical relations). Examples:

常识 本类聚焦更难表述为事实、且我们期望多数人不论文化或教育背景都具备的知识。包括对物理与社会动态的基本理解，以及词汇意义（超出简单词汇蕴含或逻辑关系）。例子：

• “The announcement of Tillerson’s departure sent shock waves across the globe” contradicts “People across the globe were prepared for Tillerson’s departure”.

• “The announcement of Tillerson’s departure sent shock waves across the globe” 与 “People across the globe were prepared for Tillerson’s departure” 矛盾。

• “Marc Sims has been seeing his barber once a week, for several years” entails “Marc Sims has been getting his hair cut once a week, for several years”.

• “Marc Sims has been seeing his barber once a week, for several years” 蕴含 “Marc Sims has been getting his hair cut once a week, for several years”。

• “Hummingbirds are really attracted to bright orange and red (hence why the feeders are usually these colours)” entails “The feeders are usually coloured so as to attract hummingbirds”.

• “Hummingbirds are really attracted to bright orange and red (hence why the feeders are usually these colours)” 蕴含 “The feeders are usually coloured so as to attract hummingbirds”。

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

GLUE：面向自然语言理解的多任务基准与分析平台

Abstract

摘要

1 Introduction

1 引言

2 Related Work

2 相关工作

3 Tasks

3 任务

3.1 Single-Sentence Tasks

3.1 单句任务

CoLA

CoLA

SST-2

SST-2

3.2 Similarity and Paraphrase Tasks

3.2 相似度与复述任务

MRPC

MRPC

QQP

QQP

STS-B

STS-B

3.3 Inference Tasks

3.3 推理任务

MNLI

MNLI

QNLI

QNLI

RTE

RTE

WNLI

WNLI

3.4 Evaluation

3.4 评估

4 Diagnostic Dataset

4 诊断数据集

Annotation Process

标注过程

Evaluation

评估

Intended Use

预期用途

5 Baselines

5 基线

Architecture

结构

Pre-Training

预训练

Training

训练

Sentence Representation Models

句表示模型

6 Benchmark Results

6 基准结果

7 Analysis

7 分析

Coarse Categories

粗粒度类别

Fine-Grained Subcategories

细粒度子类

8 Conclusion

8 结论

Acknowledgments

致谢

References

参考文献

Appendix A Additional Benchmark Details

附录 A 基准补充细节

QNLI

QNLI

Appendix B Additional Baseline Details

附录 B 基线补充细节

B.1 Attention Mechanism

B.1 注意力机制

B.2 Training

B.2 训练

B.3 Sentence Representation Models

B.3 句表示模型