A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表

暂无章节

返回论文列表
目录

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

收录于 NAACL HLT 2018 会议论文 / Oral;Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),论文集页码 1112–1122;DOI 10.18653/v1/N18-1101被引 5,847 次
打开论文来源

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Adina Williams^{1} Nikita Nangia^{2} Samuel R. Bowman^{1,2,3}

adinawilliams@nyu.edu nikitanangia@nyu.edu bowman@nyu.edu

^{1}Department of Linguistics, New York University
^{2}Center for Data Science, New York University
^{3}Department of Computer Science, New York University

Abstract

摘要

This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. At 433k examples, this resource is one of the largest corpora available for natural language inference (a.k.a. recognizing textual entailment), improving upon available resources in both its coverage and difficulty. MultiNLI accomplishes this by offering data from ten distinct genres of written and spoken English, making it possible to evaluate systems on nearly the full complexity of the language, while supplying an explicit setting for evaluating cross-genre domain adaptation. In addition, an evaluation using existing machine learning models designed for the Stanford NLI corpus shows that it represents a substantially more difficult task than does that corpus, despite the two showing similar levels of inter-annotator agreement.

本文介绍 Multi-Genre Natural Language Inference(MultiNLI)语料,这是一份用于开发和评估句子理解机器学习模型的数据集。该资源有 43.3 万条样本,是目前规模最大的自然语言推理(又称识别文本蕴含)语料之一,并在覆盖面和难度上都超过了已有资源。MultiNLI 的做法是提供来自十种不同书面和口语体裁的英语数据,从而可以在接近语言全部复杂度的条件下评估系统,同时也给出一个明确的跨体裁领域适应评测设置。此外,用为斯坦福自然语言推理语料设计的已有机器学习模型所做的评估表明,尽管两份语料的标注者一致性相近,MultiNLI 所代表的任务要明显更难。

1 Introduction

1 引言

Many of the most actively studied problems in NLP, including question answering, translation, and dialog, depend in large part on natural language understanding (NLU) for success. While there has been a great deal of work that uses representation learning techniques to pursue progress on these applied NLU problems directly, in order for a representation learning model to fully succeed at one of these problems, it must simultaneously succeed both at NLU, and at one or more additional hard machine learning problems like structured prediction or memory access. This makes it difficult to accurately judge the degree to which current models extract reasonable representations of language meaning in these settings.

自然语言处理里许多最活跃的问题,包括问答、翻译和对话,在很大程度上都要依赖自然语言理解(NLU)才能做好。虽然已有大量工作用表示学习直接在这些应用型 NLU 问题上推进,但表示学习模型若要在其中某一问题上真正成功,就必须同时做好 NLU,还要解决结构化预测或记忆访问这类额外的困难机器学习问题。这使得我们很难准确判断,当前模型在这些设置下究竟能在多大程度上抽出合理的语言意义表示。

The task of natural language inference (NLI) is well positioned to serve as a benchmark task for research on NLU. In this task, also known as recognizing textual entailment (Cooper et al., 1996; Fyodorov et al., 2000; Condoravdi et al., 2003; Bos and Markert, 2005; Dagan et al., 2006; MacCartney and Manning, 2009), a model is presented with a pair of sentences—like one of those in Figure 1—and asked to judge the relationship between their meanings by picking a label from a small set: typically ENTAILMENT, NEUTRAL, and CONTRADICTION. Succeeding at NLI does not require a system to solve any difficult machine learning problems except, crucially, that of extracting effective and thorough representations for the meanings of sentences (i.e., their lexical and compositional semantics). In particular, a model must handle phenomena like lexical entailment, quantification, coreference, tense, belief, modality, and lexical and syntactic ambiguity.

自然语言推理(NLI)很适合作为 NLU 研究的基准任务。这项任务也称识别文本蕴含(Cooper et al., 1996; Fyodorov et al., 2000; Condoravdi et al., 2003; Bos and Markert, 2005; Dagan et al., 2006; MacCartney and Manning, 2009):模型会看到一对句子——比如图 1 中的例子——并被要求从一小套标签里判断它们意义上的关系,通常是 ENTAILMENT(蕴含)、NEUTRAL(中立)和 CONTRADICTION(矛盾)。要在 NLI 上取得成功,系统不必再解决其他困难的机器学习问题,关键只在于为句子意义抽出有效而充分的表示(也就是词汇语义和组合语义)。具体来说,模型必须处理词汇蕴含、量化、指代、时态、信念、情态,以及词汇和句法歧义等现象。

As the only large human-annotated corpus for NLI currently available, the Stanford NLI Corpus (SNLI; Bowman et al., 2015) has enabled a good deal of progress on NLU, serving as a major benchmark for machine learning work on sentence understanding and spurring work on core representation learning techniques for NLU, such as attention (Wang and Jiang, 2016; Parikh et al., 2016), memory (Munkhdalai and Yu, 2017), and the use of parse structure (Mou et al., 2016b; Bowman et al., 2016; Chen et al., 2017). However, SNLI falls short of providing a sufficient testing ground for machine learning models in two ways.

作为当时唯一大规模人工标注的 NLI 语料,斯坦福自然语言推理语料(SNLI;Bowman et al., 2015)已经推动了不少 NLU 进展,既是句子理解机器学习工作的主要基准,也带动了注意力(Wang and Jiang, 2016; Parikh et al., 2016)、记忆(Munkhdalai and Yu, 2017)以及利用句法结构(Mou et al., 2016b; Bowman et al., 2016; Chen et al., 2017)等核心表示学习技术。然而,SNLI 作为机器学习模型的试验场,在两方面仍不够充分。

First, the sentences in SNLI are derived from only a single text genre—image captions—and are thus limited to descriptions of concrete visual scenes, rendering the hypothesis sentences used to describe these scenes short and simple, and rendering many important phenomena—like temporal reasoning (e.g., yesterday), belief (e.g., know), and modality (e.g., should)—rare enough to be irrelevant to task performance. Second, because of these issues, SNLI is not sufficiently demanding to serve as an effective benchmark for NLU, with the best current model performance falling within a few percentage points of human accuracy and limited room left for fine-grained comparisons between strong models.

第一,SNLI 中的句子只来自单一文本体裁——图像描述,因而局限于对具体视觉场景的描写,用来描述这些场景的假设句往往短而简单,许多重要现象——例如时间推理(如 yesterday)、信念(如 know)和情态(如 should)——出现得太少,几乎不影响任务表现。第二,正因如此,SNLI 已经不够苛刻,难以继续充当有效的 NLU 基准:当时最好的模型表现与人类准确率只差几个百分点,强模型之间也几乎没有细粒度比较的空间。

This paper introduces a new challenge dataset, the Multi-Genre NLI Corpus (MultiNLI), whose chief purpose is to remedy these limitations by making it possible to run large-scale NLI evaluations that capture more of the complexity of modern English. While its size (433k pairs) and mode of collection are modeled closely on SNLI, unlike that corpus, MultiNLI represents both written and spoken speech in a wide range of styles, degrees of formality, and topics.

本文介绍一份新的挑战数据集,即 Multi-Genre NLI Corpus(MultiNLI),其主要目的就是弥补上述局限,使大规模 NLI 评估能够覆盖更多现代英语的复杂度。它的规模(43.3 万对)和收集方式都紧密参照 SNLI,但与那份语料不同,MultiNLI 同时包含书面和口头语言,风格、正式程度和话题都很宽。

Our chief motivation in creating this corpus is to provide a benchmark for ambitious machine learning research on the core problems of NLU, but we are additionally interested in constructing a corpus that facilitates work on domain adaptation and cross-domain transfer learning. These techniques—which use labeled training data for a source domain, and aim to train a model that performs well on test data from a target domain with a different distribution—have resulted in gains across many tasks (Daume III and Marcu, 2006; Ben-David et al., 2007), including sequence and part-of-speech tagging (Blitzer et al., 2006; Peng and Dredze, 2017). Moreover, in application areas outside NLU, artificial neural network techniques have made it possible to train general-purpose feature extractors that, with no or minimal retraining, can extract useful features for a variety of styles of data (Krizhevsky et al., 2012; Zeiler and Fergus, 2014; Donahue et al., 2014). However, attempts to bring this kind of general purpose representation learning to NLU have seen only very limited success (see, for example, Mou et al., 2016a). Nearly all successful applications of representation learning to NLU have involved models that are trained on data closely resembling the target evaluation data in both task and style. This fact limits the usefulness of these tools for problems involving styles of language not represented in large annotated training sets.

我们创建这份语料的主要动机,是为 NLU 核心问题上雄心勃勃的机器学习研究提供基准;同时,我们也希望构造一份便于做领域适应和跨领域迁移学习的语料。这类方法利用源领域的带标注训练数据,目标是训练出能在分布不同的目标领域测试数据上表现良好的模型,已经在许多任务上带来提升(Daume III and Marcu, 2006; Ben-David et al., 2007),包括序列标注和词性标注(Blitzer et al., 2006; Peng and Dredze, 2017)。此外,在 NLU 以外的应用领域,人工神经网络已经可以训练出通用特征提取器,几乎不必再训练或只需极少再训练,就能为多种风格的数据抽出有用特征(Krizhevsky et al., 2012; Zeiler and Fergus, 2014; Donahue et al., 2014)。然而,把这种通用表示学习搬到 NLU 上的尝试,迄今只有非常有限的成功(例如见 Mou et al., 2016a)。几乎所有成功的 NLU 表示学习应用,所用模型都是在任务和风格上都与目标评测数据非常相近的数据上训练的。这一事实限制了这些工具在那些尚未出现在大规模标注训练集中的语言风格上的用处。

With this in mind, we construct MultiNLI so as to make it possible to explicitly evaluate models both on the quality of their sentence representations within the training domain and on their ability to derive reasonable representations in unfamiliar domains. The corpus is derived from ten different genres of written and spoken English, which are collectively meant to approximate the full diversity of ways in which modern standard American English is used. All of the genres appear in the test and development sets, but only five are included in the training set. Models thus can be evaluated on both the matched test examples, which are derived from the same sources as those in the training set, and on the mismatched examples, which do not closely resemble any of those seen at training time.

基于这一考虑,我们构造 MultiNLI,使模型既能在训练领域内被明确评估句子表示的质量,也能被评估在不熟悉领域中给出合理表示的能力。语料来自十种不同的书面和口语英语体裁,合在一起是为了近似现代标准美国英语的全部使用方式。所有体裁都出现在测试集和开发集中,但只有五种被纳入训练集。因此,模型既可以在 matched 测试样本上评估——这些样本与训练集来自同一来源——也可以在 mismatched 样本上评估,后者与训练时所见的任何内容都不太相近。

Table 1: Randomly chosen examples from the development set of our new corpus, shown with their genre labels, their selected gold labels, and the validation labels (abbreviated E, N, C) assigned by individual annotators.

Premise Genre / gold / votes Hypothesis
Met my first girlfriend that way. FACE-TO-FACE / contradiction / C C N C I didn’t meet my first girlfriend until later.
8 million in relief in the form of emergency housing. GOVERNMENT / neutral / N N N N The 8 million dollars for emergency housing was still not enough to solve the problem.
Now, as children tend their gardens, they have a new appreciation of their relationship to the land, their cultural heritage, and their community. LETTERS / neutral / N N N N All of the children love working in their gardens.
At 8:34, the Boston Center controller received a third transmission from American 11 9/11 / entailment / E E E E The Boston Center controller got a third transmission from American 11.
I am a lacto-vegetarian. SLATE / neutral / N N E N I enjoy eating cheese too much to abstain from dairy.
someone else noticed it and i said well i guess that’s true and it was somewhat melodious in other words it wasn’t just you know it was really funny TELEPHONE / contradiction / C C C C No one noticed and it wasn’t funny at all.

表 1:从我们新语料开发集中随机抽取的例子,并给出体裁标签、选定的黄金标签,以及各位标注者给出的验证标签(缩写为 E、N、C)。

前提 体裁 / 黄金标签 / 投票 假设
Met my first girlfriend that way. FACE-TO-FACE / contradiction / C C N C I didn’t meet my first girlfriend until later.
8 million in relief in the form of emergency housing. GOVERNMENT / neutral / N N N N The 8 million dollars for emergency housing was still not enough to solve the problem.
Now, as children tend their gardens, they have a new appreciation of their relationship to the land, their cultural heritage, and their community. LETTERS / neutral / N N N N All of the children love working in their gardens.
At 8:34, the Boston Center controller received a third transmission from American 11 9/11 / entailment / E E E E The Boston Center controller got a third transmission from American 11.
I am a lacto-vegetarian. SLATE / neutral / N N E N I enjoy eating cheese too much to abstain from dairy.
someone else noticed it and i said well i guess that’s true and it was somewhat melodious in other words it wasn’t just you know it was really funny TELEPHONE / contradiction / C C C C No one noticed and it wasn’t funny at all.

MultiNLI annotator prompt for written non-fiction genres, asking for one definitely correct, one possibly correct, and one definitely incorrect sentence

Figure 1: The main text of a prompt (truncated) that was presented to our annotators. This version is used for the written non-fiction genres.

图 1:提供给标注者的提示主干(有删节)。这一版本用于书面非虚构体裁。

2 The Corpus

2 语料

2.1 Data Collection

2.1 数据收集

The data collection methodology for MultiNLI is similar to that of SNLI: We create each sentence pair by selecting a premise sentence from a preexisting text source and asking a human annotator to compose a novel sentence to pair with it as a hypothesis. This section discusses the sources of our premise sentences, our collection method for hypotheses, and our validation (relabeling) strategy.

MultiNLI 的数据收集方法与 SNLI 相似:我们从已有文本来源中选出一句前提,再请人工标注者写出一句新的句子与之配对,作为假设。本节讨论前提句的来源、假设句的收集方法,以及验证(重新标注)策略。

Premise Text Sources The MultiNLI premise sentences are derived from ten sources of freely available text which are meant to be maximally diverse and roughly represent the full range of American English. We selected nine sources from the second release of the Open American National Corpus (OANC; Fillmore et al., 1998; Macleod et al., 2000; Ide and Macleod, 2001; Ide and Suderman, 2006, downloaded 12/2016), balancing the volume of source text roughly evenly across genres, and avoiding genres with content that would be too difficult for untrained annotators.

前提文本来源 MultiNLI 的前提句来自十个可免费获取的文本来源,这些来源力求尽量多样,并大致代表美国英语的全部范围。我们从开放美国国家语料库第二版(OANC;Fillmore et al., 1998; Macleod et al., 2000; Ide and Macleod, 2001; Ide and Suderman, 2006,下载于 2016 年 12 月)中选取了九个来源,使各体裁的源文本量大致均衡,并避开对未经训练的标注者来说内容过难的体裁。

1 http://www.anc.org/

OANC data constitutes the following nine genres: transcriptions from the Charlotte Narrative and Conversation Collection of two-sided, in-person conversations that took place in the early 2000s (FACE-TO-FACE); reports, speeches, letters, and press releases from public domain government websites (GOVERNMENT); letters from the Indiana Center for Intercultural Communication of Philanthropic Fundraising Discourse written in the late 1990s–early 2000s (LETTERS); the public report from the National Commission on Terrorist Attacks Upon the United States released on July 22, 2004 (9/11); five non-fiction works on the textile industry and child development published by the Oxford University Press (OUP); popular culture articles from the archives of Slate Magazine (SLATE) written between 1996–2000; transcriptions from University of Pennsylvania’s Linguistic Data Consortium Switchboard corpus of two-sided, telephone conversations that took place in 1990 or 1991 (TELEPHONE); travel guides published by Berlitz Publishing in the early 2000s (TRAVEL); and short posts about linguistics for non-specialists from the Verbatim archives written between 1990 and 1996 (VERBATIM).

OANC 数据构成了以下九种体裁:Charlotte Narrative and Conversation Collection 中对 21 世纪初面对面双边谈话的转写(FACE-TO-FACE);来自公有领域政府网站的报告、演讲、信件和新闻稿(GOVERNMENT);印第安纳跨文化交际中心慈善筹款话语库中写于 1990 年代末至 21 世纪初的信件(LETTERS);美国遭受恐怖袭击国家委员会于 2004 年 7 月 22 日发布的公开报告(9/11);牛津大学出版社出版的五部关于纺织业和儿童发展的非虚构作品(OUP);Slate Magazine 档案中 1996–2000 年的流行文化文章(SLATE);宾夕法尼亚大学语言数据联盟 Switchboard 语料中 1990 或 1991 年双边电话谈话的转写(TELEPHONE);Berlitz Publishing 在 21 世纪初出版的旅游指南(TRAVEL);以及 Verbatim 档案中 1990 至 1996 年面向非专业读者的语言学短文(VERBATIM)。

2 https://9-11commission.gov/

For our tenth genre, FICTION, we compile several freely available works of contemporary fiction written between 1912 and 2010, spanning various genres, including mystery (The Mysterious Affair at Styles, Christie, 1921; The Secret Adversary, Christie, 1922; Murder in the Gun Room, Piper, 1953), humor (Password Incorrect, Name, 2008), western (Rebel Spurs, Norton, 1962), science fiction (Seven Swords, Shea, 2008; Living History, Essex, 2016; The Sky Is Falling, Del Rey, 1973; Youth, Asimov, May 1952), and adventure (Captain Blood, Sabatini, 1922).

第十种体裁 FICTION,我们汇编了若干可免费获取、写于 1912 至 2010 年的当代小说,体裁多样,包括推理(《The Mysterious Affair at Styles》,Christie, 1921;《The Secret Adversary》,Christie, 1922;《Murder in the Gun Room》,Piper, 1953)、幽默(《Password Incorrect》,Name, 2008)、西部(《Rebel Spurs》,Norton, 1962)、科幻(《Seven Swords》,Shea, 2008;《Living History》,Essex, 2016;《The Sky Is Falling》,Del Rey, 1973;《Youth》,Asimov, May 1952)以及冒险(《Captain Blood》,Sabatini, 1922)。

3 gutenberg.org/files/863/863-0.txt
4 gutenberg.org/files/1155/1155-0.txt
5 gutenberg.org/files/17866/17866.txt
6 http://manybooks.net/pages/namenother09password_incorrect/0.html
7 gutenberg.org/files/20840/20840-0.txt
8 http://mikeshea.net/stories/seven_swords.html, shared with the author’s permission.
9 manybooks.net/pages/essexbother10living_history/0.html
10 gutenberg.org/cache/epub/18768/pg18768.txt
11 gutenberg.org/cache/epub/31547/pg31547.txt
12 gutenberg.org/files/1965/1965-0.txt

We construct premise sentences from these ten source texts with minimal preprocessing; we uniquify the sentences within genres, exclude very short sentences (under eight characters), and manually remove certain types of non-narrative writing, such as mathematical formulae, bibliographic references, and lists.

我们从这十种源文本中构造前提句,只做最少的预处理:在各体裁内对句子去重,去掉过短的句子(不足八个字符),并手工删除某些非叙事性文字,例如数学公式、参考文献和列表。

Although SNLI is collected in largely the same way as MultiNLI, and is also permissively licensed, we do not include SNLI in the MultiNLI corpus distribution. SNLI can be appended and treated as an unusually large additional CAPTIONS genre, built on image captions from the Flickr30k corpus (Young et al., 2014).

尽管 SNLI 的收集方式与 MultiNLI 大体相同,并且同样采用宽松许可,我们并没有把 SNLI 放进 MultiNLI 语料的发布包中。SNLI 可以另行追加,并当作一个规模特别大的额外 CAPTIONS 体裁来用,它建立在 Flickr30k 语料(Young et al., 2014)的图像描述之上。

Hypothesis Collection To collect a sentence pair, we present a crowdworker with a sentence from a source text and ask them to compose three novel sentences (the hypotheses): one which is necessarily true or appropriate whenever the premise is true (paired with the premise and labeled ENTAILMENT), one which is necessarily false or inappropriate whenever the premise is true (CONTRADICTION), and one where neither condition applies (NEUTRAL). This method of data collection ensures that the three classes will be represented equally in the raw corpus.

假设句收集 为收集一对句子,我们向众包工人出示源文本中的一句,并请他们写出三句新的句子(假设句):一句在前提为真时必然为真或恰当(与前提配对并标为 ENTAILMENT),一句在前提为真时必然为假或不恰当(CONTRADICTION),以及一句两种情形都不成立(NEUTRAL)。这种收集方法保证三个类别在原始语料中会均衡出现。

The prompts that surround each premise sentence during hypothesis collection are slightly tailored to fit the genre of that premise sentence. We pilot these prompts prior to data collection to ensure that the instructions are clear and that they yield hypothesis sentences that fit the intended meanings of the three classes. There are five unique prompts in total: one for written non-fiction genres (SLATE, OUP, GOVERNMENT, VERBATIM, TRAVEL; Figure 1), one for spoken genres (TELEPHONE, FACE-TO-FACE), one for each of the less formal written genres (FICTION, LETTERS), and a specialized one for 9/11, tailored to fit its potentially emotional content. Each prompt is accompanied by example premises and hypothesis that are specific to each genre.

收集假设句时,围绕每句前提的提示会按该前提所属体裁稍作调整。我们在正式收集前先试做这些提示,以确保说明清楚,并能得到符合三个类别预定含义的假设句。一共有五种不同的提示:一种用于书面非虚构体裁(SLATE、OUP、GOVERNMENT、VERBATIM、TRAVEL;见图 1),一种用于口语体裁(TELEPHONE、FACE-TO-FACE),非正式书面体裁(FICTION、LETTERS)各有一种,另外还有专为 9/11 设计的提示,以适配其可能带有的情绪内容。每条提示都附有针对该体裁的前提和假设例句。

Below the instructions, we present three text fields—one for each label—followed by a field for reporting issues, and a link to the frequently asked questions (FAQ) page. We provide one FAQ page per prompt. FAQs are modeled on their SNLI counterparts (supplied by the authors of that work) and include additional curated examples, answers to genre-specific questions arising from our pilot phase, and information about logistical concerns like payment.

在说明下方,我们提供三个文本框——每个标签一个——随后是一个用于报告问题的字段,以及指向常见问题(FAQ)页面的链接。每种提示都有单独的 FAQ 页面。FAQ 参照 SNLI 的对应页面(由该工作的作者提供),并补充了额外整理的例子、试做阶段出现的体裁特定问题的解答,以及关于报酬等后勤事项的说明。

For both hypothesis collection and validation, we present prompts to annotators using Hybrid (gethybrid.io), a crowdsoucring platform similar to the Amazon Mechanical Turk platform used for SNLI. We used this platform to hire an organized group of workers. 387 annotators contributed through this group, and at no point was any identifying information about them, including demographic information, available to the authors.

假设句收集和验证都通过 Hybrid(gethybrid.io)向标注者出示提示,这是一个与 SNLI 所用 Amazon Mechanical Turk 类似的众包平台(原文写作 crowdsoucring)。我们用该平台雇用了一个有组织的工人群体。共有 387 位标注者通过该群体参与,作者自始至终都无法获得他们的任何身份信息,包括人口统计信息。

Validation We perform an additional round of annotation on test and development examples to ensure accurate labelling. The validation phase follows the same procedure used for SICK (Marelli et al., 2014b) and SNLI: Workers are presented with pairs of sentences and asked to supply a single label (ENTAILMENT, CONTRADICTION, NEUTRAL) for the pair. Each pair is relabeled by four workers, yielding a total of five labels per example. Validation instructions are tailored by genre, based on the main data collection prompt (Figure 1); a single FAQ, modeled after the validation FAQ from SNLI, is provided for reference. In order to encourage thoughtful labeling, we manually label one percent of the validation examples and offer a $1 bonus each time a worker selects a label that matches ours.

验证 我们对测试和开发样本额外做一轮标注,以确保标签准确。验证阶段沿用 SICK(Marelli et al., 2014b)和 SNLI 的程序:工人看到一对句子,并为该对提供一个标签(ENTAILMENT、CONTRADICTION、NEUTRAL)。每对由四位工人重新标注,因此每条样本共有五个标签。验证说明按体裁调整,依据主要收集提示(图 1);另外提供一份参照 SNLI 验证 FAQ 写成的 FAQ 供查阅。为鼓励认真标注,我们手工标注了百分之一的验证样本,每当工人选出的标签与我们的一致,就给予 1 美元奖金。

For each validated sentence pair, we assign a gold label representing a majority vote between the initial label assigned to the pair by the original annotator, and the four additional labels assigned by validation annotators. A small number of examples did not receive a three-vote consensus on any one label. These examples are included in the distributed corpus, but are marked with ‘-’ in the gold label field, and should not be used in standard evaluations. Table 2 shows summary statistics capturing the results of validation, alongside corresponding figures for SNLI. These statistics indicate that the labels included in MultiNLI are about as reliable as those included in SNLI, despite MultiNLI’s more diverse text contents.

对每一对经过验证的句子,我们根据原标注者给出的初始标签与四位验证标注者给出的附加标签进行多数投票,得到黄金标签。少量样本没有在任何一个标签上达到三票一致。这些样本仍包含在发布的语料中,但黄金标签字段标为 “-”,不应用于标准评估。表 2 给出了验证结果的汇总统计,并附上 SNLI 的对应数字。这些统计表明,尽管 MultiNLI 的文本内容更加多样,其中的标签与 SNLI 中的标签大致同样可靠。

Table 2: Key validation statistics for SNLI (copied from Bowman et al., 2015) and MultiNLI.

Statistic SNLI MultiNLI
Pairs w/ unanimous gold label 58.3% 58.2%
Individual label = gold label 89.0% 88.7%
Individual label = author’s label 85.8% 85.2%
Gold label = author’s label 91.2% 92.6%
Gold label ≠ author’s label 6.8% 5.6%
No gold label (no 3 labels match) 2.0% 1.8%

表 2:SNLI(数字引自 Bowman et al., 2015)和 MultiNLI 的主要验证统计。

统计量 SNLI MultiNLI
黄金标签五票一致的样本对 58.3% 58.2%
单个标签 = 黄金标签 89.0% 88.7%
单个标签 = 作者标签 85.8% 85.2%
黄金标签 = 作者标签 91.2% 92.6%
黄金标签 ≠ 作者标签 6.8% 5.6%
无黄金标签(没有任何标签得到 3 票) 2.0% 1.8%

2.2 The Resulting Corpus

2.2 所得语料

Table 1 shows randomly chosen development set examples from the collected corpus. Hypotheses tend to be fluent and correctly spelled, though not all are complete sentences. Punctuation is often omitted. Hypotheses can rely heavily on knowledge about the world, and often don’t correspond closely with their premises in syntactic structure.

表 1 展示了从所收集语料的开发集中随机抽取的例子。假设句通常流畅、拼写正确,但并非都是完整句子。标点常常被省略。假设句可能在很大程度上依赖关于世界的知识,并且在句法结构上往往与前提并不接近。

Unlabeled test data is available on Kaggle for both matched and mismatched sets as competitions that will be open indefinitely; Evaluations on a subset of the test set have previously been conducted with different leaderboards through the RepEval 2017 Workshop (Nangia et al., 2017).

未标注的测试数据已在 Kaggle 上以比赛形式发布,matched 和 mismatched 两套都会长期开放;此前也通过 RepEval 2017 研讨会(Nangia et al., 2017)在测试集的一个子集上、用不同的排行榜做过评估。

The corpus is available in two formats—tab separated text and JSON Lines (jsonl), following SNLI. For each example, premise and hypothesis strings, unique identifiers for the pair and prompt, and the following additional fields are specified:

语料提供两种格式——制表符分隔文本和 JSON Lines(jsonl),与 SNLI 一致。对每条样本,都给出前提和假设字符串、该句对和提示的唯一标识,以及下列附加字段:

  • gold label: label used for classification. In examples rejected during the validation process, the value of this field will be ‘-’.
  • gold label:用于分类的标签。在验证过程中被剔除的样本,该字段的值为 “-”。
  • sentence{1,2} parse: Each sentence as parsed by the Stanford PCFG Parser 3.5.2 (Klein and Manning, 2003).
  • sentence{1,2} parse:由 Stanford PCFG Parser 3.5.2(Klein and Manning, 2003)解析后的各句。
  • sentence{1,2} binary parse: parses in unlabeled binary-branching format.
  • sentence{1,2} binary parse:无标签二叉形式的句法分析结果。
  • label[1]: The label assigned during the creation of the sentence pair. In rare cases this may be different from gold label, if a consensus of annotators chose a different label during the validation phase.
  • label[1]:创建句对时赋予的标签。在少数情况下,若验证阶段标注者达成的共识与之不同,它可能不同于黄金标签。
  • label[2...5]: The four labels assigned during validation by individual annotators to each development and test example. These fields will be empty for training examples.
  • label[2...5]:验证阶段由各位标注者为每条开发和测试样本赋予的四个标签。训练样本中这些字段为空。

The current version of the corpus is freely available at nyu.edu/projects/bowman/multinli/ for typical machine learning uses, and may be modified and redistributed. The majority of the corpus is released under the OANC’s license, which allows all content to be freely used, modified, and shared under permissive terms. The data in the FICTION section falls under several permissive licenses; Seven Swords is available under a Creative Commons Share-Alike 3.0 Unported License, and with the explicit permission of the author, Living History and Password Incorrect are available under Creative Commons Attribution 3.0 Unported Licenses; the remaining works of fiction are in the public domain in the United States (but may be licensed differently elsewhere).

当前版本的语料可在 nyu.edu/projects/bowman/multinli/ 免费获取,供典型的机器学习用途使用,并且可以修改和再分发。语料的大部分按 OANC 的许可发布,允许全部内容在宽松条款下自由使用、修改和共享。FICTION 部分的数据适用若干宽松许可:《Seven Swords》采用 Creative Commons Share-Alike 3.0 Unported License;在作者明确许可下,《Living History》和《Password Incorrect》采用 Creative Commons Attribution 3.0 Unported License;其余虚构作品在美国属于公有领域(但在其他地区许可可能不同)。

Partition The distributed corpus comes with an explicit train/test/development split. The test and development sets contain 2,000 randomly selected examples each from each of the genres, resulting in a total of 20,000 examples per set. No premise sentence occurs in more than one set.

划分 发布的语料带有明确的训练/测试/开发划分。测试集和开发集从每种体裁中各随机抽取 2,000 条样本,因此每套共 20,000 条。任何一句前提都不会出现在一个以上的集合中。

Table 3: Key statistics for the corpus by genre. The first five genres represent the matched section of the development and test sets, and the remaining five represent the mismatched section. The first three statistics provide the number of examples in each genre. #Wds. Prem. is the mean token count among premise sentences. ‘S’ parses is the percentage of sentences for which the Stanford Parser produced a parse rooted with an ‘S’ (sentence) node. Agrmt. is the percent of individual labels that match the gold label in validated examples. Model Acc. gives the test accuracy for ESIM and CBOW models (trained on either SNLI or MultiNLI), as described in Section 3.

Genre Train Dev. Test #Wds. Prem. ‘S’ parses Prem. ‘S’ parses Hyp. Agrmt. ESIM CBOW
SNLI 550,152 10,000 10,000 14.1 74% 88% 89.0% 86.7% 80.6 %
FICTION 77,348 2,000 2,000 14.4 94% 97% 89.4% 73.0% 67.5%
GOVERNMENT 77,350 2,000 2,000 24.4 90% 97% 87.4% 74.8% 67.5%
SLATE 77,306 2,000 2,000 21.4 94% 98% 87.1% 67.9% 60.6%
TELEPHONE 83,348 2,000 2,000 25.9 71% 97% 88.3% 72.2% 63.7%
TRAVEL 77,350 2,000 2,000 24.9 97% 98% 89.9% 73.7% 64.6%
9/11 0 2,000 2,000 20.6 98% 99% 90.1% 71.9% 63.2%
FACE-TO-FACE 0 2,000 2,000 18.1 91% 96% 89.5% 71.2% 66.3%
LETTERS 0 2,000 2,000 20.0 95% 98% 90.1% 74.7% 68.3%
OUP 0 2,000 2,000 25.7 96% 98% 88.1% 71.7% 62.8%
VERBATIM 0 2,000 2,000 28.3 93% 97% 87.3% 71.9% 62.7%
MultiNLI Overall 392,702 20,000 20,000 22.3 91% 98% 88.7% 72.2% 64.7%

表 3:按体裁给出的语料主要统计。前五种体裁构成开发和测试集中的 matched 部分,后五种构成 mismatched 部分。前三项统计给出各体裁的样本数。#Wds. Prem. 是前提句的平均 token 数。“S” parses 是斯坦福解析器给出以 “S”(句子)节点为根的句法树的句子百分比。Agrmt. 是在经过验证的样本中,单个标签与黄金标签相符的百分比。Model Acc. 给出 ESIM 和 CBOW 模型(在 SNLI 或 MultiNLI 上训练)的测试准确率,详见第 3 节。

体裁 训练 开发 测试 前提词数 前提 “S” 解析 假设 “S” 解析 一致率 ESIM CBOW
SNLI 550,152 10,000 10,000 14.1 74% 88% 89.0% 86.7% 80.6 %
FICTION 77,348 2,000 2,000 14.4 94% 97% 89.4% 73.0% 67.5%
GOVERNMENT 77,350 2,000 2,000 24.4 90% 97% 87.4% 74.8% 67.5%
SLATE 77,306 2,000 2,000 21.4 94% 98% 87.1% 67.9% 60.6%
TELEPHONE 83,348 2,000 2,000 25.9 71% 97% 88.3% 72.2% 63.7%
TRAVEL 77,350 2,000 2,000 24.9 97% 98% 89.9% 73.7% 64.6%
9/11 0 2,000 2,000 20.6 98% 99% 90.1% 71.9% 63.2%
FACE-TO-FACE 0 2,000 2,000 18.1 91% 96% 89.5% 71.2% 66.3%
LETTERS 0 2,000 2,000 20.0 95% 98% 90.1% 74.7% 68.3%
OUP 0 2,000 2,000 25.7 96% 98% 88.1% 71.7% 62.8%
VERBATIM 0 2,000 2,000 28.3 93% 97% 87.3% 71.9% 62.7%
MultiNLI 总体 392,702 20,000 20,000 22.3 91% 98% 88.7% 72.2% 64.7%

Statistics Table 3 shows some additional statistics. Premise sentences in MultiNLI tend to be longer (max 401 words, mean 22.3 words) than their hypotheses (max 70 words, mean 11.4 words), and much longer, on average, than premises in SNLI (mean 14.1 words); premises in MultiNLI also tend to be parsed as complete sentences at a much higher rate on average (91%) than their SNLI counterparts (74%). We observe that the two spoken genres differ in this—with FACE-TO-FACE showing more complete sentences (91%) than TELEPHONE (71%)—and speculate that the lack of visual feedback in a telephone setting may result in a high incidence of interrupted or otherwise incomplete sentences.

统计 表 3 给出了更多统计。MultiNLI 中的前提句往往比假设句更长(最长 401 词,平均 22.3 词;假设句最长 70 词,平均 11.4 词),并且平均也比 SNLI 中的前提(平均 14.1 词)长得多;MultiNLI 的前提被解析为完整句子的平均比例(91%)也远高于 SNLI 中的对应句子(74%)。我们还看到两种口语体裁在这一点上有差别——FACE-TO-FACE 的完整句子更多(91%),TELEPHONE 更少(71%)——并推测,电话场景缺少视觉反馈,可能导致打断或不完整的句子更多。

Hypothesis sentences in MultiNLI generally cannot be derived from their premise sentences using only trivial editing strategies. While 2.5% of the hypotheses in SNLI differ from their premises by deletion, only 0.9% of those in MultiNLI (170 examples total) are constructed in this way. Similarly, in SNLI, 1.6% of hypotheses differ from their premises by addition, substitution, or shuffling a single word, while in MultiNLI this only happens in 1.2% of examples. The percentage of hypothesis-premise pairs with high token overlap (>37%) was comparable between MultiNLI (30% of pairs) and SNLI (29%). These statistics suggest that MultiNLI’s annotations are comparable in quality to those of SNLI.

MultiNLI 中的假设句一般不能仅靠对前提句做琐碎编辑得到。SNLI 中有 2.5% 的假设与前提只差删除操作,而 MultiNLI 中这一比例只有 0.9%(共 170 条)。类似地,SNLI 中有 1.6% 的假设与前提只差增加、替换或打乱一个词,MultiNLI 中仅为 1.2%。假设与前提 token 重叠较高(>37%)的比例,在 MultiNLI(30% 的句对)和 SNLI(29%)之间相当。这些统计表明,MultiNLI 的标注质量与 SNLI 相当。

3 Baselines

3 基线

To test the difficulty of the corpus, we experiment with three neural network models. The first is a simple continuous bag of words (CBOW) model in which each sentence is represented as the sum of the embedding representations of its words. The second computes representations by averaging the states of a bidirectional LSTM RNN (BiLSTM; Hochreiter and Schmidhuber, 1997) over words. For the third, we implement and evaluate Chen et al.’s Enhanced Sequential Inference Model (ESIM), which is roughly tied for the state of the art on SNLI at the time of writing. We use the base ESIM without ensembling with a TreeLSTM (as in the ‘HIM’ runs in that work).

为检验这份语料的难度,我们用三种神经网络模型做实验。第一种是简单的连续词袋(CBOW)模型,每句表示为词嵌入之和。第二种通过对双向 LSTM RNN(BiLSTM;Hochreiter and Schmidhuber, 1997)在各词上的状态取平均来计算表示。第三种是我们实现并评估的 Chen et al. 的 Enhanced Sequential Inference Model(ESIM),在写作时它大致与当时 SNLI 上的最优结果持平。我们使用基础版 ESIM,不与 TreeLSTM 做集成(不同于该工作中的 “HIM” 实验)。

The first two models produce separate vector representations for each sentence and compute label predictions for pairs of representations. To do this, they concatenate the representations for premise and hypothesis, their difference, and their element-wise product, following Mou et al. (2016b), and pass the result to a single tanh layer followed by a three-way softmax classifier.

前两种模型为每句分别生成向量表示,再基于成对表示计算标签预测。具体做法是,按照 Mou et al. (2016b),把前提和假设的表示、它们的差以及逐元素积拼接起来,送入一个 tanh 层,再接三路 softmax 分类器。

All models are initialized with 300D reference GloVe vectors (840B token version; Pennington et al., 2014). Out-of-vocabulary (OOV) words are initialized randomly and word embeddings are fine-tuned during training. The models use 300D hidden states, as in most prior work on SNLI. We use Dropout (Srivastava et al., 2014) for regularization. For ESIM, we use a dropout rate of 0.5, following the paper. For CBOW and BiLSTM models, we tune Dropout on the SNLI development set and find that a drop rate of 0.1 works well. We use the Adam (Kingma and Ba, 2015) optimizer with default parameters. Code is available at github.com/nyu-mll/multiNLI/.

所有模型都用 300 维的参考 GloVe 向量初始化(8400 亿 token 版本;Pennington et al., 2014)。词表外(OOV)词随机初始化,词嵌入在训练中微调。模型使用 300 维隐状态,与大多数先前的 SNLI 工作一致。我们用 Dropout(Srivastava et al., 2014)做正则。ESIM 按原论文使用 0.5 的 dropout 率。对 CBOW 和 BiLSTM,我们在 SNLI 开发集上调节 Dropout,发现 0.1 的丢弃率效果较好。优化器采用默认参数的 Adam(Kingma and Ba, 2015)。代码见 github.com/nyu-mll/multiNLI/。

We train models on SNLI, MultiNLI, and a mixture; Table 4 shows the results. In the mixed setting, we use the full MultiNLI training set and randomly select 15% of the SNLI training set at each epoch, ensuring that each available genre is seen during training with roughly equal frequency.

我们分别在 SNLI、MultiNLI 以及二者的混合上训练模型;结果见表 4。在混合设置中,我们使用全部 MultiNLI 训练集,并在每个 epoch 随机抽取 15% 的 SNLI 训练集,以确保训练时各可用体裁被看到的频率大致相同。

Table 4: Test set accuracies (%) for all models; Match. represents test set performance on the MultiNLI genres that are also represented in the training set, Mis. represents test set performance on the remaining ones; Most freq. is a trivial ‘most frequent class’ baseline.

Train Model SNLI MNLI Match. MNLI Mis.
Most freq. 34.3 36.5 35.6
SNLI CBOW 80.6 - -
SNLI BiLSTM 81.5 - -
SNLI ESIM 86.7 - -
MNLI CBOW 51.5 64.8 64.5
MNLI BiLSTM 50.8 66.9 66.9
MNLI ESIM 60.7 72.3 72.1
MNLI+SNLI CBOW 74.7 65.2 64.6
MNLI+SNLI BiLSTM 74.0 67.5 67.1
MNLI+SNLI ESIM 79.7 72.4 71.9

表 4:全部模型的测试集准确率(%);Match. 表示在训练集中也出现过的 MultiNLI 体裁上的测试表现,Mis. 表示在其余体裁上的测试表现;Most freq. 是平凡的“最频繁类别”基线。

训练数据 模型 SNLI MNLI Match. MNLI Mis.
Most freq. 34.3 36.5 35.6
SNLI CBOW 80.6 - -
SNLI BiLSTM 81.5 - -
SNLI ESIM 86.7 - -
MNLI CBOW 51.5 64.8 64.5
MNLI BiLSTM 50.8 66.9 66.9
MNLI ESIM 60.7 72.3 72.1
MNLI+SNLI CBOW 74.7 65.2 64.6
MNLI+SNLI BiLSTM 74.0 67.5 67.1
MNLI+SNLI ESIM 79.7 72.4 71.9

We also train a separate CBOW model on each individual genre to establish the degree to which simple models already allow for effective transfer across genres, using a dropout rate of 0.2. When training on SNLI, a single random sample of 15% of the original training set is used. For each genre represented in the training set, the model that performs best on it was trained on that genre; a model trained only on SNLI performs worse on every genre than comparable models trained on any genre from MultiNLI.

我们还在每一种单独的体裁上各训练一个 CBOW 模型,dropout 率为 0.2,以衡量简单模型跨体裁迁移的效果已经有多好。在 SNLI 上训练时,使用原训练集的一个 15% 随机样本。对训练集中出现的每一种体裁,在其上表现最好的模型都是在该体裁上训练的;只在 SNLI 上训练的模型,在每一种体裁上的表现都差于在 MultiNLI 任一体裁上训练的可比模型。

Models trained on a single genre from MultiNLI perform well on similar genres; for example, the model trained on TELEPHONE attains the best accuracy (63%) on FACE-TO-FACE, which was nearly one point better than it received on itself. SLATE seems to be a difficult and relatively unusual genre and performance on it is relatively poor in this setting; when averaging over runs trained on SNLI and all genres in the matched section of the training set, average performance on SLATE was only 57.5%. Sentences in SLATE cover a wide range of topics and phenomena, making it hard to do well on, but also forcing models trained on it be broadly capable; the model trained on SLATE achieves the highest accuracy of any model on 9/11 (55.6%) and VERBATIM (57.2%), and relatively high accuracy on TRAVEL (57.4%) and GOVERNMENT (58.3%). We also observe that our models perform similarly on both the matched and mismatched test sets of MultiNLI. We expect genre mismatch issues to become more conspicuous as models are developed that can better fit MultiNLI’s training genres.

在 MultiNLI 单一体裁上训练的模型,在相近体裁上表现较好;例如,在 TELEPHONE 上训练的模型在 FACE-TO-FACE 上取得了最高准确率(63%),几乎比它在自身体裁上还高近 1 个百分点。SLATE 似乎是一种较难且相对特殊的体裁,在这一设置下表现较差;把在 SNLI 以及训练集 matched 部分所有体裁上训练的实验取平均,SLATE 上的平均表现只有 57.5%。SLATE 中的句子覆盖话题和现象很广,因而很难做好,但也会迫使在其上训练的模型具备更广的能力;在 SLATE 上训练的模型在 9/11(55.6%)和 VERBATIM(57.2%)上取得了所有模型中的最高准确率,在 TRAVEL(57.4%)和 GOVERNMENT(58.3%)上也相对较高。我们还观察到,模型在 MultiNLI 的 matched 和 mismatched 测试集上表现相近。我们预期,随着更能拟合 MultiNLI 训练体裁的模型被开发出来,体裁不匹配的问题会更加显眼。

To evaluate the contribution of sentence length to corpus difficulty, we binned premises and hypotheses by length in 25-word increments for premises and 10-word increments for hypotheses. Using the ESIM model, our strong baseline, we find a small effect (stronger for matched than mismatched) of premise length on model accuracy: accuracy decreases slightly as premise sentences increase in length. We find no effect of hypothesis length on accuracy.

为评估句子长度对语料难度的贡献,我们按长度把前提和假设分箱,前提以 25 词为间隔,假设以 10 词为间隔。使用我们的强基线 ESIM 模型,我们发现前提长度对模型准确率有较小影响(在 matched 上比 mismatched 更明显):前提句变长时,准确率略有下降。我们没有发现假设句长度对准确率有影响。

4 Discussion and Analysis

4 讨论与分析

4.1 Data Collection

4.1 数据收集

In data collection for NLI, different annotator decisions about the coreference between entities and events across the two sentences in a pair can lead to very different assignments of pairs to labels (de Marneffe et al., 2008; Marelli et al., 2014a; Bowman et al., 2015). Drawing an example from Bowman et al., the pair “a boat sank in the Pacific Ocean” and “a boat sank in the Atlantic Ocean” can be labeled either CONTRADICTION or NEUTRAL depending on (among other things) whether the two mentions of boats are assumed to refer to the same entity in the world. This uncertainty can present a serious problem for inter-annotator agreement, since it is not clear that it is possible to define an explicit set of rules around coreference that would be easily intelligible to an untrained annotator (or any non-expert).

在 NLI 的数据收集中,标注者对一对句子里实体和事件是否共指的不同判断,会导致这对句子被分到非常不同的标签(de Marneffe et al., 2008; Marelli et al., 2014a; Bowman et al., 2015)。沿用 Bowman et al. 的例子,“a boat sank in the Pacific Ocean” 和 “a boat sank in the Atlantic Ocean” 这一对,既可以标为 CONTRADICTION,也可以标为 NEUTRAL,这取决于(除其他因素外)两处 “boats” 是否被假定指世界上的同一实体。这种不确定性会给标注者一致性带来严重问题,因为并不清楚能否围绕共指定义出一套未经训练的标注者(或任何非专家)容易理解的明确规则。

Bowman et al. attempt to avoid this problem by using an annotation prompt that is highly dependent on the concreteness of image descriptions; but, as we engage with the much more abstract writing that is found in, for example, government documents, there is no reason to assume a priori that any similar prompt and annotation strategy can work. We are surprised to find that this is not a major issue. Through a relatively straightforward trial-and-error piloting phase, followed by discussion with our annotators, we manage to design prompts for abstract genres that yield high inter-annotator agreement scores nearly identical to those of SNLI (see Table 2). These high scores suggest that our annotators agreed on a single task definition, and were able to apply it consistently across genres.

Bowman et al. 试图通过高度依赖图像描述具体性的标注提示来回避这一问题;但当我们面对例如政府文件中那种抽象得多的写法时,事先并没有理由假定类似的提示和标注策略仍然可行。我们惊讶地发现,这并不是一个主要问题。经过相对直接的试做和试错,并与标注者讨论之后,我们为抽象体裁设计出的提示,得到了与 SNLI 几乎相同的高标注者一致性分数(见表 2)。这些高分表明,标注者就单一的任务定义达成了一致,并能在各体裁上稳定地应用它。

4.2 Overall Difficulty

4.2 总体难度

As expected, both the increase in the diversity of linguistic phenomena in MultiNLI and its longer average sentence length conspire to make MultiNLI dramatically more difficult than SNLI. Our three baseline models perform better on SNLI than MultiNLI by about 15% when trained on the respective datasets. All three models achieve accuracy above 80% on the SNLI test set when trained only on SNLI. However, when trained on MultiNLI, only ESIM surpasses 70% accuracy on MultiNLI’s test sets. When we train models on MultiNLI and downsampled SNLI, we see an expected significant improvement on SNLI, but no significant change in performance on the MultiNLI test sets, suggesting including SNLI in training doesn’t drive substantial improvement. These results attest to MultiNLI’s difficulty, and with its relatively high inter-annotator agreement, suggest that it presents a problem with substantial headroom for future work.

正如预期,MultiNLI 中语言现象多样性的增加,以及更长的平均句长,共同使它比 SNLI 难得多。三种基线模型在各自对应的数据集上训练时,在 SNLI 上的表现大约比在 MultiNLI 上高 15%。只在 SNLI 上训练时,三个模型在 SNLI 测试集上的准确率都超过 80%。然而,在 MultiNLI 上训练时,只有 ESIM 在 MultiNLI 测试集上超过 70%。当我们在 MultiNLI 和经过下采样的 SNLI 上训练时,SNLI 上出现了预期中的显著提升,但 MultiNLI 测试集上的表现没有显著变化,说明把 SNLI 加入训练并不能带来实质性改进。这些结果证明了 MultiNLI 的难度;再结合其相对较高的标注者一致性,说明它为后续工作留下了相当大的提升空间。

4.3 Analysis by Linguistic Phenomenon

4.3 按语言现象分析

To better understand the types of language understanding skills that MultiNLI tests, we analyze the collected corpus using a set of annotation tags chosen to reflect linguistic phenomena which are known to be potentially difficult. We use two methods to assign tags to sentences. First, we use the Penn Treebank (PTB; Marcus et al., 1993) part-of-speech tag set (via the included Stanford Parser parses) to automatically isolate sentences containing a range of easily-identified phenomena like comparatives. Second, we isolate sentences that contain hand-chosen key words indicative of additional interesting phenomena.

为更好地理解 MultiNLI 所考查的语言理解技能类型,我们用一套标注标签分析所收集的语料,这些标签用来反映已知可能较难的语言现象。我们用两种方法给句子打标签。第一,借助宾州树库(PTB;Marcus et al., 1993)的词性标签集(通过附带的斯坦福解析器结果),自动找出含有比较级等容易识别现象的句子。第二,找出包含手工选定、能指示其他有趣现象的关键词的句子。

The hand-chosen tag set covers the following phenomena: QUANTIFIERS contains single words with quantificational force (see, for example, Heim and Kratzer, 1998; Szabolcsi, 2010, e.g., many, all, few, some); BELIEF VERBS contains sentence-embedding verbs denoting mental states (e.g., know, believe, think), including irregular past tense forms; TIME TERMS contains single words with abstract temporal interpretation, (e.g., then, today) and month names and days of the week; DISCOURSE MARKERS contains words that facilitate discourse coherence (e.g., yet, however, but, thus, despite); PRESUPPOSITION TRIGGERS contains words with lexical presuppositions (Stalnaker, 1974; Schlenker, 2016, e.g., again, too, anymore); CONDITIONALS contains the word if. Table 5 presents the frequency of the tags in SNLI and MultiNLI, and model accuracy on MultiNLI (trained only on MultiNLI).

手工选定的标签集覆盖下列现象:QUANTIFIERS 包含具有量化力量的单词(例如见 Heim and Kratzer, 1998; Szabolcsi, 2010,如 many、all、few、some);BELIEF VERBS 包含表示心理状态的嵌句动词(如 know、believe、think),包括不规则过去式;TIME TERMS 包含具有抽象时间解释的单词(如 then、today)以及月份名和星期名;DISCOURSE MARKERS 包含促成话语连贯的词(如 yet、however、but、thus、despite);PRESUPPOSITION TRIGGERS 包含带词汇预设的词(Stalnaker, 1974; Schlenker, 2016,如 again、too、anymore);CONDITIONALS 包含单词 if。表 5 给出这些标签在 SNLI 和 MultiNLI 中的频率,以及模型在 MultiNLI 上的准确率(仅在 MultiNLI 上训练)。

13 Because their high frequency in the corpus, extremely common triggers like the were excluded from this tag.

13 因为它们在语料中频率很高,像 the 这样极为常见的触发词被排除在该标签之外。

Table 5: Dev. Freq. is the percentage of dev. set examples that include each phenomenon, ordered by greatest difference in frequency of occurrence (Diff.) between MultiNLI and SNLI. Most Frequent Label specifies which label is the most frequent for each tag in the MultiNLI dev. set, and % is its incidence. Model Acc. is the dev. set accuracy (%) by annotation tag for each baseline model (trained on MultiNLI only). (PTB) marks a tag as derived from Penn Treebank-style parser output tags (Marcus et al., 1993).

Tag SNLI MultiNLI Diff. Most Frequent Label % CBOW BiLSTM ESIM
Entire Corpus 100 100 0 entailment ∼35 ∼65 ∼67 ∼72
Pronouns (PTB) 34 68 34 entailment 34 66 68 73
Quantifiers 33 63 30 contradiction 36 66 68 73
Modals (PTB) <1 28 28 entailment 35 65 67 72
Negation (PTB) 5 31 26 contradiction 48 67 70 75
WH terms (PTB) 5 30 25 entailment 35 64 65 72
Belief Verbs <1 19 18 entailment 34 64 67 71
Time Terms 19 36 17 neutral 35 64 66 71
Discourse Mark. <1 14 14 neutral 34 62 64 70
Presup. Triggers 8 22 14 neutral 34 65 67 73
Compr./Supr.(PTB) 3 17 14 neutral 39 61 63 69
Conditionals 4 15 11 neutral 35 65 68 73
Tense Match (PTB) 62 69 7 entailment 37 67 68 73
Interjections (PTB) <1 5 5 entailment 36 67 70 75
>20 words <1 5 5 entailment 42 65 67 76

表 5:Dev. Freq. 是开发集中包含各现象的样本百分比,按 MultiNLI 与 SNLI 出现频率之差(Diff.)从大到小排列。Most Frequent Label 给出 MultiNLI 开发集中各标签最常见的类别,% 是其占比。Model Acc. 是各基线模型(仅在 MultiNLI 上训练)按标注标签统计的开发集准确率(%)。(PTB)表示该标签来自宾州树库风格的解析器输出标签(Marcus et al., 1993)。

标签 SNLI MultiNLI 差值 最常见标签 % CBOW BiLSTM ESIM
全部语料 100 100 0 entailment ∼35 ∼65 ∼67 ∼72
Pronouns (PTB) 34 68 34 entailment 34 66 68 73
Quantifiers 33 63 30 contradiction 36 66 68 73
Modals (PTB) <1 28 28 entailment 35 65 67 72
Negation (PTB) 5 31 26 contradiction 48 67 70 75
WH terms (PTB) 5 30 25 entailment 35 64 65 72
Belief Verbs <1 19 18 entailment 34 64 67 71
Time Terms 19 36 17 neutral 35 64 66 71
Discourse Mark. <1 14 14 neutral 34 62 64 70
Presup. Triggers 8 22 14 neutral 34 65 67 73
Compr./Supr.(PTB) 3 17 14 neutral 39 61 63 69
Conditionals 4 15 11 neutral 35 65 68 73
Tense Match (PTB) 62 69 7 entailment 37 67 68 73
Interjections (PTB) <1 5 5 entailment 36 67 70 75
>20 words <1 5 5 entailment 42 65 67 76

The incidence of tags varies by genre; the percentage of sentence pairs containing a particular annotation tag differs by a maximum over 30% across genres. Sentence pairs containing pronouns are predictably common for all genres, with 93% of Government and Face-to-face pairs including at least one. The Telephone genre has the highest percentage of sentence pairs containing one occurrence of negation, WH-words, belief-verbs and time terms, Verbatim has the highest percentage of pairs containing quantifiers and conversational pivots, and Letters has the highest percentage of pairs that contain one or more modals. Pairs containing comparatives and/or superlatives, which is the tag that our baseline models perform worst on, are most common in the Oxford University Press genre. Based on this, we conclude that the genres are sufficiently different, because they are not uniform with respect to the percentages of sentence pairs that contain each of the annotation tags.

各标签的出现率随体裁而变;含某一标注标签的句对百分比在体裁之间最多相差超过 30%。含代词的句对在所有体裁中都理所当然地常见,Government 和 Face-to-face 中有 93% 的句对至少包含一个代词。Telephone 体裁中,含否定、疑问词、信念动词和时间词语一次出现的句对比例最高;Verbatim 中含量化词和会话转折的句对比例最高;Letters 中含一个或多个情态词的句对比例最高。含比较级和/或最高级的句对——这是基线模型表现最差的标签——在牛津大学出版社体裁中最常见。据此我们得出结论:这些体裁足够不同,因为它们在含各标注标签的句对比例上并不均一。

The distributions of labels within each tagged subset of the corpus roughly mirrors the balanced overall distribution. The most frequent class overall (in this case, ENTAILMENT) occurs with a frequency of roughly one third (see Table 4) in most. Only two annotation tags differ from the baseline percentage of the most frequent class in the corpus by at least 5%: sentences containing negation, and sentences exceeding 20 words. Sentences that contain negation are slightly more likely than average to be labeled CONTRADICTION, reflecting a similar finding in SNLI, while long sentences are slightly more likely to be labeled ENTAILMENT.

语料中每个带标签子集的标签分布,大致镜像了总体上的均衡分布。总体最常见的类别(这里是 ENTAILMENT)在多数子集中的频率大约是三分之一(见表 4)。只有两个标注标签与语料中最常见类别的基线百分比相差至少 5%:含否定的句子,以及超过 20 词的句子。含否定的句子比平均更常被标为 CONTRADICTION,这与 SNLI 中的类似发现一致;而长句则稍更常被标为 ENTAILMENT。

None of the baseline models perform substantially better on any tagged set than they do on the corpus overall, with average model accuracies on sentences containing specific tags falling within about 3 points of overall averages. Using baseline model test accuracy overall as a metric (see Table 4), our baseline models had the most trouble on sentences containing comparatives or superlatives (losing 3-4 points each). Despite the fact that 17% of sentence pairs in the corpus contained at least one instance of comparative or superlative, our baseline models don’t utilize the information present in these sentences to predict the correct label for the pair, although presence of a comparative or superlative is slightly more predictive of a NEUTRAL label.

没有任何一种基线模型在任一带标签集合上的表现显著好于在全部语料上的表现,含特定标签的句子上的平均模型准确率与总体平均只相差大约 3 个百分点。以基线模型的总体测试准确率为尺度(见表 4),我们的基线模型在含比较级或最高级的句子上最吃力(各掉 3–4 个百分点)。尽管语料中 17% 的句对至少含有一处比较级或最高级,基线模型并没有利用这些句子中的信息来预测正确标签,虽然出现比较级或最高级对 NEUTRAL 标签略有预示作用。

Moreover, the baseline models perform below average on discourse markers, such as despite and however, losing roughly 2 to 3 points each. Unsurprisingly, the attention-based ESIM model performs better than the other two on sentences with greater than 20 words. Additionally, our baseline models do show slight improvements in accuracy on negation, suggesting that they may be tracking it as a predictor of CONTRADICTION.

此外,基线模型在话语标记(如 despite 和 however)上的表现低于平均,大约各掉 2 到 3 个百分点。不出意外,基于注意力的 ESIM 模型在超过 20 词的句子上优于另外两个模型。另外,我们的基线模型在否定上确实显示出轻微的准确率提升,表明它们可能把否定当作 CONTRADICTION 的一个预测线索在跟踪。

5 Conclusion

5 结论

Natural language inference makes it easy to judge the degree to which neural network models for sentence understanding capture the full meanings for natural language sentences. Existing NLI datasets like SNLI have facilitated substantial advances in modeling, but have limited headroom and coverage of the full diversity of meanings expressed in English. This paper presents a new dataset that offers dramatically greater linguistic difficulty and diversity, and also serves as a benchmark for cross-genre domain adaptation.

自然语言推理使我们容易判断,用于句子理解的神经网络模型在多大程度上把握了自然语言句子的完整意义。像 SNLI 这样的已有 NLI 数据集已经推动了建模上的显著进展,但提升空间有限,对英语所表达意义的全部多样性覆盖也不够。本文给出一份新的数据集,语言难度和多样性都大幅提高,同时也作为跨体裁领域适应的基准。

Our new corpus, MultiNLI, improves upon SNLI in its empirical coverage—because it includes a representative sample of text and speech from ten different genres, as opposed to just simple image captions—and its difficulty, containing a much higher percentage of sentences tagged with one or more elements from our tag set of thirteen difficult linguistic phenomena. This greater diversity is reflected in the dramatically lower baseline model performance on MultiNLI than on SNLI (see Table 5) and comparable inter-annotator agreement, suggesting that MultiNLI has a lot of headroom remaining for future work.

我们的新语料 MultiNLI 在经验覆盖上改进了 SNLI——因为它包含来自十种不同体裁的文本和口语的代表性样本,而不仅仅是简单的图像描述——也在难度上有所提高,被标上我们十三种困难语言现象标签集中一个或多个成分的句子比例高得多。这种更大的多样性反映在基线模型在 MultiNLI 上的表现远低于 SNLI(见表 5),而标注者一致性却相当,说明 MultiNLI 仍为后续工作留下了很大空间。

The MultiNLI corpus was first released in draft form in the first half of 2017, and in the time since its initial release, work by others (Conneau et al., 2017) has shown that NLI can also be an effective source task for pre-training and transfer learning in the context of sentence-to-vector models, with models trained on SNLI and MultiNLI substantially outperforming all prior models on a suite of established transfer learning benchmarks. We hope that this corpus will continue to serve for many years as a resource for the development and evaluation of methods for sentence understanding.

MultiNLI 语料在 2017 年上半年以草稿形式首次发布;自初次发布以来,他人的工作(Conneau et al., 2017)已经表明,NLI 也可以作为句子到向量模型中预训练和迁移学习的有效源任务,在 SNLI 和 MultiNLI 上训练的模型,在一套既有迁移学习基准上大幅超过了此前所有模型。我们希望这份语料在今后许多年里,能够继续作为开发和评估句子理解方法的资源。

Acknowledgments

致谢

This work was made possible by a Google Faculty Research Award. SB also gratefully acknowledges support from Tencent Holdings and Samsung Research. We also thank George Dahl, the organizers of the RepEval 2016 and RepEval 2017 workshops, Andrew Drozdov, Angeliki Lazaridou, and our other NYU colleagues for help and advice.

本工作得益于一项 Google Faculty Research Award。SB 也感谢腾讯控股和三星研究院的支持。我们还感谢 George Dahl、RepEval 2016 和 RepEval 2017 研讨会的组织者、Andrew Drozdov、Angeliki Lazaridou,以及其他纽约大学同事的帮助和建议。

References

参考文献

Isaac Asimov. May 1952. Youth. Space Science Fiction Magazine, Republic Features Syndicate, Inc.

Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2007. Analysis of representations for domain adaptation. In Advances in neural information processing systems. pages 137–144.

John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 120–128. http://www.aclweb.org/anthology/W06-1615.

Johan Bos and Katja Markert. 2005. Recognising textual entailment with logical inference. In Proceedings of the 2005 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 628–635. http://www.aclweb.org/ anthology/H05-1079.

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 632–642. https://doi.org/10.18653/v1/D15-1075.

Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 1466–1477. https://doi.org/10.18653/v1/P16-1139.

Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 1657–1668. https://doi.org/10.18653/v1/P17-1152.

Agatha Christie. 1921. Mysterious Affair at Styles. The Bodley Head.

Agatha Christie. 1922. The Secret Adversary. Dodd, Mead.

Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G. Bobrow. 2003. Entailment, intensionality and text understanding. In Proceedings of the Human Language Technology-North American Association for Computational Linguistics 2003 Workshop on Text Meaning.

Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364 .

Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Steve Pulman Ted Briscoe Holger Maier Poesio, and Karsten Konrad. 1996. Using the framework. Technical report, Technical Report LRE 62-051 D-16, The FraCaS Consortium.

Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. Evaluating predictive uncertainty, visual object classification, and recognising textual entailment, Springer, pages 177–190.

Hal Daume III and Daniel Marcu. 2006. Domain adaptation for statistical classifiers. Journal of Artificial Intelligence Research 26:101–126. https://doi.org/doi:10.1613/jair.1872.

Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In Proceedings of Association for Computational Linguistics-08: Human Language Technology. Association for Computational Linguistics, pages 1039–1047. http://www.aclweb.org/ anthology/P08-1118.

Lester Del Rey. 1973. The Sky Is Falling. Ace Books.

Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. DeCAF: A deep convolutional activation feature for generic visual recognition. In Proceedings of the International Conference on Machine Learning (ICML).

Ben Essex. 2016. Living History. CreateSpace Independent Publishing Platform.

Charles Fillmore, Nancy Ide, Daniel Jurafsky, and Catherine Macleod. 1998. An American National Corpus: A proposal. In Proceedings of the First Annual Conference on Language Resources and Evaluation. pages 965–969.

Yaroslav Fyodorov, Yoad Winter, and Nissim Francez. 2000. A natural logic inference system. In Proceedings of the 2nd Workshop on Inference in Computational Semantics.

Irene Heim and Angelika Kratzer. 1998. Semantics in generative grammar. Blackwell Publishers.

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780.

Nancy Ide and Catherine Macleod. 2001. The American National Corpus: A standardized resource of American English. In Proceedings of Corpus Linguistics. Lancaster University Centre for Computer Corpus Research on Language, volume 3, pages 1– 7.

Nancy Ide and Keith Suderman. 2006. Integrating linguistic resources: The national corpus model. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC). European Language Resources Association (ELRA). http://www.aclweb.org/ anthology/L06-1138.

Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).

Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proc. ACL. https://doi.org/10.3115/1075096.1075150.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pages 1097–1105.

Bill MacCartney and Christopher D Manning. 2009. An extended model of natural logic. In Proceedings of the of the Eighth International Conference on Computational Semantics. pages 140–156.

Catherine Macleod, Nancy Ide, and Ralph Grishman. 2000. The American National Corpus: A standardized resource for American English. In Conference on Language Resources and Evaluation (LREC).

Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn treebank. Computational linguistics 19(2):313–330.

Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014a. Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). Association for Computational Linguistics, pages 1–8. https://doi.org/10.3115/v1/S14-2001.

Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014b. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC).

Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2016a. How transferable are neural networks in NLP applications? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pages 479–489. https://doi.org/10.18653/v1/D16-1046.

Lili Mou, Men Rui, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016b. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, pages 130–136. https://doi.org/10.18653/v1/P16-2022.

Tsendsuren Munkhdalai and Hong Yu. 2017. Neural semantic encoders. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Association for Computational Linguistics, pages 397–407. http://www.aclweb.org/ anthology/E17-1038.

Nick Name. 2008. Password Incorrect. Published online. Author also known as Piotr Kowalczyk.

Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel R Bowman. 2017. The repeval 2017 shared task: Multi-genre natural language inference with sentence representations. In Proceedings of RepEval 2017: The Second Workshop on Evaluating Vector Space Representations for NLP.

Andre Norton. 1962. The Rebel Spurs. The World Publishing Company, Cleveland & New York.

Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 2249– 2255. https://doi.org/10.18653/v1/D16-1244.

Nanyun Peng and Mark Dredze. 2017. Multi-task domain adaptation for sequence tagging. In Proceedings of the 2nd Workshop on Representation Learning for NLP. pages 91–100.

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pages 1532–1543. https://doi.org/10.3115/v1/D14-1162.

Henry Beam Piper. 1953. Murder In the Gun Room. H.B. Piper, New York.

Rafael Sabatini. 1922. Captain Blood. Houghton Mifflin Company.

Philippe Schlenker. 2016. The Cambridge Handbook of Formal Semantics, Cambridge University Press, chapter The Semantics/Pragmatics Interface, pages 664–727. https://doi.org/10.1017/CBO9781139236157.023.

Michael Shea. 2008. Seven Swords. Published online.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR) 15:1929–1958.

Robert Stalnaker. 1974. Semantics and Philosophy, New York University Press, chapter Pragmatic Presupposition, pages 329–355.

Anna Szabolcsi. 2010. Quantification. Cambridge University Press.

Shuohang Wang and Jing Jiang. 2016. Learning natural language inference with LSTM. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 1442–1451. https://doi.org/10.18653/v1/N16-1170.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association of Computational Linguistics 2:67–78. http://www.aclweb.org/ anthology/Q14-1006.

Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV). pages 818–833.