A large annotated corpus for learning natural language inference
A large annotated corpus for learning natural language inference
Abstract
Understanding entailment and contradiction is fundamental to understanding natural language, and inference about entailment and contradiction is a valuable testing ground for the development of semantic representations. However, machine learning research in this area has been dramatically limited by the lack of large-scale resources. To address this, we introduce the Stanford Natural Language Inference corpus, a new, freely available collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning. At 570K pairs, it is two orders of magnitude larger than all other resources of its type. This increase in scale allows lexicalized classifiers to outperform some sophisticated existing entailment models, and it allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time.
理解蕴涵和矛盾,是理解自然语言的基础;围绕蕴涵和矛盾做推理,也是发展语义表示的重要试验场。然而,这一方向的机器学习研究一直严重受限于缺乏大规模资源。为此,我们推出斯坦福自然语言推理语料(Stanford Natural Language Inference corpus):一套新的、可免费获取的带标注句子对集合,由人在一项基于图像描述的新颖具身任务中写就。该语料有 57 万对样本,比以往所有同类资源大约大两个数量级。规模的提升,使带词法特征的分类器能够超过某些已有的复杂蕴涵模型,也使基于神经网络的模型第一次在自然语言推理基准上具备了竞争力。
Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642,
Lisbon, Portugal, 17-21 September 2015. ©2015 Association for Computational Linguistics.
1 Introduction
The semantic concepts of entailment and contradiction are central to all aspects of natural language meaning (Katz, 1972; van Benthem, 2008), from the lexicon to the content of entire texts. Thus, natural language inference (NLI) — characterizing and using these relations in computational systems (Fyodorov et al., 2000; Condoravdi et al., 2003; Bos and Markert, 2005; Dagan et al., 2006; MacCartney and Manning, 2009) — is essential in tasks ranging from information retrieval to semantic parsing to commonsense reasoning.
蕴涵和矛盾这些语义概念,贯穿自然语言意义的各个层面(Katz, 1972; van Benthem, 2008),从词汇一直到整篇文本的内容。因此,自然语言推理(NLI)——在计算系统中刻画并使用这些关系(Fyodorov et al., 2000; Condoravdi et al., 2003; Bos and Markert, 2005; Dagan et al., 2006; MacCartney and Manning, 2009)——对从信息检索到语义解析、再到常识推理的诸多任务都至关重要。
NLI has been addressed using a variety of techniques, including those based on symbolic logic, knowledge bases, and neural networks. In recent years, it has become an important testing ground for approaches employing distributed word and phrase representations. Distributed representations excel at capturing relations based in similarity, and have proven effective at modeling simple dimensions of meaning like evaluative sentiment (e.g., Socher et al. 2013), but it is less clear that they can be trained to support the full range of logical and commonsense inferences required for NLI (Bowman et al., 2015; Weston et al., 2015b; Weston et al., 2015a). In a SemEval 2014 task aimed at evaluating distributed representations for NLI, the best-performing systems relied heavily on additional features and reasoning capabilities (Marelli et al., 2014a).
NLI 已用多种技术加以研究,包括基于符号逻辑、知识库和神经网络的方法。近年来,它也成为检验分布式词和短语表示的重要试验场。分布式表示擅长捕捉基于相似性的关系,在评价性情感这类较简单的意义维度上已被证明有效(例如 Socher et al. 2013),但它们能否被训练来支持 NLI 所需的全部逻辑和常识推理,则尚不清楚(Bowman et al., 2015; Weston et al., 2015b; Weston et al., 2015a)。在一项旨在评估分布式表示用于 NLI 的 SemEval 2014 任务中,表现最好的系统大量依赖额外特征和推理能力(Marelli et al., 2014a)。
Our ultimate objective is to provide an empirical evaluation of learning-centered approaches to NLI, advancing the case for NLI as a tool for the evaluation of domain-general approaches to semantic representation. However, in our view, existing NLI corpora do not permit such an assessment. They are generally too small for training modern data-intensive, wide-coverage models, many contain sentences that were algorithmically generated, and they are often beset with indeterminacies of event and entity coreference that significantly impact annotation quality.
我们的最终目标,是对以学习为中心的 NLI 方法做经验评估,并推动把 NLI 作为检验通用语义表示方法的工具。但在我们看来,现有的 NLI 语料还不足以支撑这样的评估。它们通常太小,无法训练现代的、依赖数据的宽覆盖模型;其中不少句子是算法生成的;而且往往受困于事件和实体共指的不确定性,这会明显影响标注质量。
To address this, this paper introduces the Stanford Natural Language Inference (SNLI) corpus, a collection of sentence pairs labeled for entailment, contradiction, and semantic independence. At 570,152 sentence pairs, SNLI is two orders of magnitude larger than all other resources of its type. And, in contrast to many such resources, all of its sentences and labels were written by humans in a grounded, naturalistic context. In a separate validation phase, we collected four additional judgments for each label for 56,941 of the examples. Of these, 98% of cases emerge with a three-annotator consensus, and 58% see a unanimous consensus from all five annotators.
为此,本文推出斯坦福自然语言推理(SNLI)语料:一套标有蕴涵、矛盾和语义独立的句子对集合。SNLI 共有 570,152 对句子,比以往所有同类资源大约大两个数量级。而且与许多此类资源不同,其中所有句子和标签都是由人在具身、自然的情境中写出的。在单独的验证阶段,我们对其中 56,941 个例子的每个标签又收集了四个额外判断。在这些例子中,98% 出现了三名标注者的一致意见,58% 得到全部五名标注者的一致同意。
In this paper, we use this corpus to evaluate a variety of models for natural language inference, including rule-based systems, simple linear classifiers, and neural network-based models. We find that two models achieve comparable performance: a feature-rich classifier model and a neural network model centered around a Long Short-Term Memory network (LSTM; Hochreiter and Schmidhuber 1997). We further evaluate the LSTM model by taking advantage of its ready support for transfer learning, and show that it can be adapted to an existing NLI challenge task, yielding the best reported performance by a neural network model and approaching the overall state of the art.
本文用这一语料评估多种自然语言推理模型,包括基于规则的系统、简单的线性分类器,以及基于神经网络的模型。我们发现有两个模型表现相当:一个特征丰富的分类器模型,以及一个以长短期记忆网络(LSTM;Hochreiter and Schmidhuber 1997)为核心的神经网络模型。我们还利用 LSTM 便于做迁移学习这一点对它做了进一步评估,表明它可以适配到已有的 NLI 挑战任务上,取得神经网络模型当时最好的报告成绩,并接近总体最好水平。
2 A new corpus for NLI
To date, the primary sources of annotated NLI corpora have been the Recognizing Textual Entailment (RTE) challenge tasks.[^1] These are generally high-quality, hand-labeled data sets, and they have stimulated innovative logical and statistical models of natural language reasoning, but their small size (fewer than a thousand examples each) limits their utility as a testbed for learned distributed representations. The data for the SemEval 2014 task called Sentences Involving Compositional Knowledge (SICK) is a step up in terms of size, but only to 4,500 training examples, and its partly automatic construction introduced some spurious patterns into the data (Marelli et al. 2014a, §6). The Denotation Graph entailment set (Young et al., 2014) contains millions of examples of entailments between sentences and artificially constructed short phrases, but it was labeled using fully automatic methods, and is noisy enough that it is probably suitable only as a source of supplementary training data. Outside the domain of sentence-level entailment, Levy et al. (2014) introduce a large corpus of semi-automatically annotated entailment examples between subject–verb–object relation triples, and the second release of the Paraphrase Database (Pavlick et al., 2015) includes automatically generated entailment annotations over a large corpus of pairs of words and short phrases.
迄今为止,带标注的 NLI 语料主要来自识别文本蕴涵(RTE)挑战任务。[^1] 这些数据集通常质量高、由人工标注,也推动了自然语言推理方面有创新性的逻辑和统计模型,但其规模很小(每套往往不到一千例),作为学习到的分布式表示的试验场,用处有限。名为 Sentences Involving Compositional Knowledge(SICK)的 SemEval 2014 任务数据在规模上进了一步,但也只有 4,500 条训练样本,而且其部分自动构造的方式给数据引入了一些虚假规律(Marelli et al. 2014a, §6)。Denotation Graph 蕴涵集(Young et al., 2014)包含数百万条句子与人工构造短短语之间的蕴涵例子,但完全由自动方法标注,噪声较大,大概只适合作为补充训练数据。在句子级蕴涵之外,Levy et al.(2014)给出了一套规模较大、半自动标注的主–谓–宾关系三元组蕴涵语料;Paraphrase Database 的第二版(Pavlick et al., 2015)则包含在大量词对和短短语上自动生成的蕴涵标注。
Existing resources suffer from a subtler issue that impacts even projects using only human-provided annotations: indeterminacies of event and entity coreference lead to insurmountable indeterminacy concerning the correct semantic label (de Marneffe et al. 2008 §4.3; Marelli et al. 2014b). For an example of the pitfalls surrounding entity coreference, consider the sentence pair A boat sank in the Pacific Ocean and A boat sank in the Atlantic Ocean. The pair could be labeled as a contradiction if one assumes that the two sentences refer to the same single event, but could also be reasonably labeled as neutral if that assumption is not made. In order to ensure that our labeling scheme assigns a single correct label to every pair, we must select one of these approaches across the board, but both choices present problems. If we opt not to assume that events are coreferent, then we will only ever find contradictions between sentences that make broad universal assertions, but if we opt to assume coreference, new counterintuitive predictions emerge. For example, Ruth Bader Ginsburg was appointed to the US Supreme Court and I had a sandwich for lunch today would unintuitively be labeled as a contradiction, rather than neutral, under this assumption.
现有资源还有一个更隐蔽的问题,即便只使用人工标注的项目也会受到影响:事件和实体共指的不确定性,会带来关于正确语义标签无法消解的不确定(de Marneffe et al. 2008 §4.3; Marelli et al. 2014b)。以实体共指的陷阱为例,考虑句子对 A boat sank in the Pacific Ocean 和 A boat sank in the Atlantic Ocean。若假定两句说的是同一件事,这对句子可以被标为矛盾;若不做这一假定,标为中立也合理。为了保证我们的标注方案对每一对都给出唯一正确标签,必须在这两种做法中统一选择一种,但两种选择都有问题。如果选择不假定事件共指,那么只有在句子做出宽泛的全称断言时才会出现矛盾;如果选择假定共指,又会出现不合直觉的新预测。例如,在这一假定下,Ruth Bader Ginsburg was appointed to the US Supreme Court 和 I had a sandwich for lunch today 会被不合直觉地标成矛盾,而不是中立。
Entity coreference presents a similar kind of indeterminacy, as in the pair A tourist visited New York and A tourist visited the city. Assuming coreference between New York and the city justifies labeling the pair as an entailment, but without that assumption the city could be taken to refer to a specific unknown city, leaving the pair neutral. This kind of indeterminacy of label can be resolved only once the questions of coreference are resolved.
实体共指也会带来类似的不确定,例如 A tourist visited New York 和 A tourist visited the city 这一对。若假定 New York 与 the city 共指,就可以把这对标为蕴涵;若不做这一假定,the city 也可以被理解成某个特定的未知城市,这对就应标为中立。这类标签上的不确定,只有在共指问题得到解决之后才能消除。
With SNLI, we sought to address the issues of size, quality, and indeterminacy. To do this, we employed a crowdsourcing framework with the following crucial innovations. First, the examples were grounded in specific scenarios, and the premise and hypothesis sentences in each example were constrained to describe that scenario from the same perspective, which helps greatly in controlling event and entity coreference.[^2] Second, the prompt gave participants the freedom to produce entirely novel sentences within the task setting, which led to richer examples than we see with the more proscribed string-editing techniques of earlier approaches, without sacrificing consistency. Third, a subset of the resulting sentences were sent to a validation task aimed at providing a highly reliable set of annotations over the same data, and at identifying areas of inferential uncertainty.
在 SNLI 中,我们试图同时解决规模、质量和不确定性这些问题。为此,我们采用了一套众包框架,并做了如下关键改进。第一,例子被锚定在具体场景中,每个例子里的前提句和假设句都被约束为从同一视角描述该场景,这在控制事件和实体共指上帮助很大。[^2] 第二,提示语让参与者可以在任务设定内写出全新的句子,因此得到的例子比早期方法中限制更多的改写技巧更加丰富,同时并不牺牲一致性。第三,一部分得到的句子被送入验证任务,目的是在同一批数据上提供一套高度可靠的标注,并找出推理上存在不确定的地方。
[^2]: Issues of coreference are not completely solved, but greatly mitigated. For example, with the premise sentence A dog is lying in the grass, a worker could safely assume that the dog is the most prominent thing in the photo, and very likely the only dog, and build contradicting sentences assuming reference to the same dog.
[^2]: 共指问题并没有被完全解决,但得到了大幅缓解。例如,当前提句是 A dog is lying in the grass 时,工人可以有把握地假定这只狗是照片中最突出的物体,而且很可能是唯一的一只狗,并在假定指称同一只狗的前提下写出矛盾句。
2.1 Data collection
We used Amazon Mechanical Turk for data collection. In each individual task (each HIT), a worker was presented with premise scene descriptions from a pre-existing corpus, and asked to supply hypotheses for each of our three labels—entailment, neutral, and contradiction—forcing the data to be balanced among these classes.
我们用 Amazon Mechanical Turk 收集数据。在每一项单独任务(每个 HIT)中,工人会看到来自已有语料的前提场景描述,并被要求为三种标签——蕴涵、中立和矛盾——各提供一个假设,从而强制数据在这些类别之间保持均衡。
The instructions that we provided to the workers are shown in Figure 1. Below the instructions were three fields for each of three requested sentences, corresponding to our entailment, neutral, and contradiction labels, a fourth field (marked optional) for reporting problems, and a link to an FAQ page.
我们提供给工人的说明见图 1。说明下方有三个输入栏,分别对应所要求的三个句子,也即蕴涵、中立和矛盾标签;另有第四个栏(标为可选)用于报告问题,以及一个指向常见问题页面的链接。
图 1:在 Mechanical Turk 上用于数据收集的说明。
That FAQ grew over the course of data collection. It warned about disallowed techniques (e.g., reusing the same sentence for many different prompts, which we saw in a few cases), provided guidance concerning sentence length and complexity (we did not enforce a minimum length, and we allowed bare NPs as well as full sentences), and reviewed logistical issues around payment timing. About 2,500 workers contributed.
这份常见问题说明在收集过程中不断扩充。它提醒了不允许使用的做法(例如把同一句话反复用于许多不同提示,我们在少数情况下看到过),就句子长度和复杂度给出了指导(我们没有强制最短长度,既允许光杆名词短语,也允许完整句子),并说明了与付款时间有关的事务安排。大约有 2,500 名工人参与。
For the premises, we used captions from the Flickr30k corpus (Young et al., 2014), a collection of approximately 160k captions (corresponding to about 30k images) collected in an earlier crowdsourced effort.[^3] The captions were not authored by the photographers who took the source images, and they tend to contain relatively literal scene descriptions that are suited to our approach, rather than those typically associated with personal photographs (as in their example: Our trip to the Olympic Peninsula). In order to ensure that the label for each sentence pair can be recovered solely based on the available text, we did not use the images at all during corpus collection.
前提句来自 Flickr30k 语料(Young et al., 2014)中的描述,该语料约有 16 万条描述(对应约 3 万张图像),由更早的一次众包工作收集。[^3] 这些描述并非由拍摄原图的摄影师撰写,其中往往是比较写实的场景描写,正好适合我们的做法,而不是个人照片里常见的那种写法(他们举的例子是:Our trip to the Olympic Peninsula)。为了保证每个句子对的标签可以仅凭现有文本恢复,我们在语料收集过程中完全没有使用图像。
[^3]: We additionally include about 4k sentence pairs from a pilot study in which the premise sentences were instead drawn from the VisualGenome corpus (under construction; visualgenome.org). These examples appear only in the training set, and have pair identifiers prefixed with vg in our corpus.
[^3]: 我们另外收入了约 4 千对来自预试验的句子,其中前提句取自当时仍在建设中的 VisualGenome 语料(visualgenome.org)。这些例子只出现在训练集中,语料里的配对标识以 vg 为前缀。
Table 2 reports some key statistics about the collected corpus, and Figure 2 shows the distributions of sentence lengths for both our source hypotheses and our newly collected premises. We observed that while premise sentences varied considerably in length, hypothesis sentences tended to be as short as possible while still providing enough information to yield a clear judgment, clustering at around seven words. We also observed that the bulk of the sentences from both sources were syntactically complete rather than fragments, and the frequency with which the parser produces a parse rooted with an ‘S’ (sentence) node attests to this.
表 2 报告了所收集语料的一些关键统计,图 2 给出了来源假设句和我们新收集的前提句的长度分布。我们观察到,前提句长度变化较大,而假设句往往在仍能提供足够信息、从而做出明确判断的前提下尽量短,聚集在大约七个词附近。我们还观察到,来自两个来源的句子大多在句法上是完整的,而不是片段;解析器经常给出以 ‘S’(句子)节点为根的句法分析,也说明了这一点。
Table 2: Key statistics for the raw sentence pairs in SNLI. Since the two halves of each pair were collected separately, we report some statistics for both.
| Data set sizes: | |
| Training pairs | 550,152 |
| Development pairs | 10,000 |
| Test pairs | 10,000 |
| Sentence length: | |
| Premise mean token count | 14.1 |
| Hypothesis mean token count | 8.3 |
| Parser output: | |
| Premise ‘S’-rooted parses | 74.0% |
| Hypothesis ‘S’-rooted parses | 88.9% |
| Distinct words (ignoring case) | 37,026 |
表 2:SNLI 中原始句子对的关键统计。由于每一对的两半是分开收集的,有些统计我们对两边分别报告。
| 数据集规模: | |
| 训练对 | 550,152 |
| 开发对 | 10,000 |
| 测试对 | 10,000 |
| 句子长度: | |
| 前提平均词元数 | 14.1 |
| 假设平均词元数 | 8.3 |
| 解析器输出: | |
| 前提以 ‘S’ 为根的分析 | 74.0% |
| 假设以 ‘S’ 为根的分析 | 88.9% |
| 不同词形(忽略大小写) | 37,026 |
图 2:句子长度的分布。
2.2 Data validation
In order to measure the quality of our corpus, and in order to construct maximally useful testing and development sets, we performed an additional round of validation for about 10% of our data. This validation phase followed the same basic form as the Mechanical Turk labeling task used to label the SICK entailment data: we presented workers with pairs of sentences in batches of five, and asked them to choose a single label for each pair. We supplied each pair to four annotators, yielding five labels per pair including the label used by the original author. The instructions were similar to the instructions for initial data collection shown in Figure 1, and linked to a similar FAQ. Though we initially used a very restrictive qualification (based on past approval rate) to select workers for the validation task, we nonetheless discovered (and deleted) some instances of random guessing in an early batch of work, and subsequently instituted a fully closed qualification restricted to about 30 trusted workers.
为了衡量语料质量,并构建尽量有用的测试集和开发集,我们对大约 10% 的数据做了额外一轮验证。这一验证阶段的基本形式,与用于标注 SICK 蕴涵数据的 Mechanical Turk 标注任务相同:我们把句子对按每批五个呈现给工人,并请他们为每一对选择一个标签。每对交给四名标注者,加上原作者使用的标签,每对共有五个标签。说明与图 1 中最初数据收集的说明相似,并链向类似的常见问题页。尽管我们起初用非常严格的资格条件(基于过往通过率)来挑选验证任务的工人,仍在一批早期工作中发现(并删除)了若干随机猜测的情形,随后改为完全封闭的资格,只允许大约 30 名受信任的工人参加。
For each pair that we validated, we assigned a gold label. If any one of the three labels was chosen by at least three of the five annotators, it was chosen as the gold label. If there was no such consensus, which occurred in about 2% of cases, we assigned the placeholder label ‘-’. While these unlabeled examples are included in the corpus distribution, they are unlikely to be helpful for the standard NLI classification task, and we do not include them in either training or evaluation in the experiments that we discuss in this paper.
对每一个经过验证的句子对,我们指定一个金标。若三个标签中的某一个被五名标注者中的至少三人选中,就把它定为金标。若没有这样的一致意见(约 2% 的情形),我们就标上占位标签 ‘-’。这些未标注的例子仍包含在语料发布中,但对标准的 NLI 分类任务大概没有帮助,本文讨论的实验在训练和评估中都不使用它们。
The results of this validation process are summarized in Table 3. Nearly all of the examples received a majority label, indicating broad consensus about the nature of the data and categories. The gold-labeled examples are very nearly evenly distributed across the three labels. The Fleiss κ scores (computed over every example with a full five annotations) are likely to be conservative given our large and unevenly distributed pool of annotators, but they still provide insights about the levels of disagreement across the three semantic classes. This disagreement likely reflects not just the limitations of large crowdsourcing efforts but also the uncertainty inherent in naturalistic NLI. Regardless, the overall rate of agreement is extremely high, suggesting that the corpus is sufficiently high quality to pose a challenging but realistic machine learning task.
这一验证过程的结果汇总于表 3。几乎所有例子都得到了多数标签,说明对数据性质和类别存在广泛共识。带金标的例子在三个标签上分布几乎均匀。Fleiss κ 分数(在所有有完整五个标注的例子上计算)鉴于我们标注者人数多、分布不均,很可能偏保守,但仍然能说明三个语义类别上的分歧程度。这种分歧既可能反映大规模众包的局限,也可能反映自然情境下 NLI 本身固有的不确定。尽管如此,总体一致率非常高,表明该语料质量足以构成一项有挑战、但现实的机器学习任务。
Table 3: Statistics for the validated pairs. The author’s label is the label used by the worker who wrote the premise to create the sentence pair. A gold label reflects a consensus of three votes from among the author and the four annotators.
| General: | |
| Validated pairs | 56,951 |
| Pairs w/ unanimous gold label | 58.3% |
| Individual annotator label agreement: | |
| Individual label = gold label | 89.0% |
| Individual label = author’s label | 85.8% |
| Gold label/author’s label agreement: | |
| Gold label = author’s label | 91.2% |
| Gold label ≠ author’s label | 6.8% |
| No gold label (no 3 labels match) | 2.0% |
| Fleiss κ: | |
| contradiction | 0.77 |
| entailment | 0.72 |
| neutral | 0.60 |
| Overall | 0.70 |
表 3:经过验证的句子对的统计。作者标签是写出前提、从而构成该句子对的那位工人所使用的标签。金标反映作者与四名标注者中三票一致的结果。
| 总体: | |
| 经过验证的句子对 | 56,951 |
| 金标五人一致的句子对 | 58.3% |
| 个体标注者标签一致率: | |
| 个体标签 = 金标 | 89.0% |
| 个体标签 = 作者标签 | 85.8% |
| 金标与作者标签的一致: | |
| 金标 = 作者标签 | 91.2% |
| 金标 ≠ 作者标签 | 6.8% |
| 无金标(没有三个标签一致) | 2.0% |
| Fleiss κ: | |
| 矛盾 | 0.77 |
| 蕴涵 | 0.72 |
| 中立 | 0.60 |
| 总体 | 0.70 |
2.3 The distributed corpus
Table 1 shows a set of randomly chosen validated examples from the development set with their labels. Qualitatively, we find the data that we collected draws fairly extensively on commonsense knowledge, and that hypothesis and premise sentences often differ structurally in significant ways, suggesting that there is room for improvement beyond superficial word alignment models. We also find the sentences that we collected to be largely fluent, correctly spelled English, with a mix of full sentences and caption-style noun phrase fragments, though punctuation and capitalization are often omitted.
表 1 给出一组从开发集中随机抽取、并带有标签的已验证例子。从性质上看,我们收集到的数据相当广泛地依赖常识,而且假设句和前提句常常在结构上有明显差异,说明在浅层词对齐模型之外仍有改进空间。我们还发现,收集到的句子大体上是流畅、拼写正确的英语,既有完整句子,也有描述风格的名词短语片段,不过标点和大小写常常被省略。
Table 1: Randomly chosen examples from the development section of our new corpus, shown with both the selected gold labels and the full set of labels (abbreviated) from the individual annotators, including (in the first position) the label used by the initial author of the pair.
| Premise | Gold / labels | Hypothesis |
|---|---|---|
| A man inspects the uniform of a figure in some East Asian country. | contradiction / C C C C C | The man is sleeping |
| An older and younger man smiling. | neutral / N N E N N | Two men are smiling and laughing at the cats playing on the floor. |
| A black race car starts up in front of a crowd of people. | contradiction / C C C C C | A man is driving down a lonely road. |
| A soccer game with multiple males playing. | entailment / E E E E E | Some men are playing a sport. |
| A smiling costumed woman is holding an umbrella. | neutral / N N E C N | A happy woman in a fairy costume holds an umbrella. |
表 1:从我们新语料开发集中随机选取的例子,同时给出选定的金标,以及各位标注者的全部标签(缩写),其中第一个位置是该对最初作者使用的标签。
| 前提 | 金标 / 各位标签 | 假设 |
|---|---|---|
| A man inspects the uniform of a figure in some East Asian country. | contradiction / C C C C C | The man is sleeping |
| An older and younger man smiling. | neutral / N N E N N | Two men are smiling and laughing at the cats playing on the floor. |
| A black race car starts up in front of a crowd of people. | contradiction / C C C C C | A man is driving down a lonely road. |
| A soccer game with multiple males playing. | entailment / E E E E E | Some men are playing a sport. |
| A smiling costumed woman is holding an umbrella. | neutral / N N E C N | A happy woman in a fairy costume holds an umbrella. |
The corpus is available under a CreativeCommons Attribution-ShareAlike license, the same license used for the Flickr30k source captions. It can be downloaded at:
nlp.stanford.edu/projects/snli/
该语料以 Creative Commons Attribution-ShareAlike 许可发布,与 Flickr30k 来源描述所用许可相同。下载地址为:
nlp.stanford.edu/projects/snli/
Partition We distribute the corpus with a pre-specified train/test/development split. The test and development sets contain 10k examples each. Each original ImageFlickr caption occurs in only one of the three sets, and all of the examples in the test and development sets have been validated.
划分 我们发布语料时附带预先指定的训练/测试/开发划分。测试集和开发集各含 1 万个例子。每条原始 ImageFlickr 描述只出现在三个集合中的一个里,测试集和开发集中的全部例子都经过了验证。
Parses The distributed corpus includes parses produced by the Stanford PCFG Parser 3.5.2 (Klein and Manning, 2003), trained on the standard training set as well as on the Brown Corpus (Francis and Kucera 1979), which we found to improve the parse quality of the descriptive sentences and noun phrases found in the descriptions.
句法分析 发布的语料包含由 Stanford PCFG Parser 3.5.2(Klein and Manning, 2003)给出的分析,该解析器在标准训练集以及布朗语料库(Francis and Kucera 1979)上训练;我们发现,这能提高描述句和描述中名词短语的分析质量。
3 Our data as a platform for evaluation
The most immediate application for our corpus is in developing models for the task of NLI. In particular, since it is dramatically larger than any existing corpus of comparable quality, we expect it to be suitable for training parameter-rich models like neural networks, which have not previously been competitive at this task. Our ability to evaluate standard classifier-base NLI models, however, was limited to those which were designed to scale to SNLI’s size without modification, so a more complete comparison of approaches will have to wait for future work. In this section, we explore the performance of three classes of models which could scale readily: (i) models from a well-known NLI system, the Excitement Open Platform; (ii) variants of a strong but simple feature-based classifier model, which makes use of both unlexicalized and lexicalized features, and (iii) distributed representation models, including a baseline model and neural network sequence models.
我们这套语料最直接的用途,是用来发展 NLI 任务的模型。尤其是,由于它比任何现有的、质量相当的语料都大得多,我们预期它适合训练像神经网络这样参数量大的模型,而这类模型此前在这一任务上并不具备竞争力。不过,我们能够评估的标准分类器式 NLI 模型,只限于那些无需修改就能扩展到 SNLI 这一规模的模型,因此更完整的方法比较只能留待后续工作。本节考察三类能够较容易扩展的模型:(i)来自著名 NLI 系统 Excitement Open Platform 的模型;(ii)一个虽简单但很强的基于特征的分类器模型的多种变体,同时使用非词法和词法特征;(iii)分布式表示模型,包括一个基线模型以及神经网络序列模型。
3.1 Excitement Open Platform models
The first class of models is from the Excitement Open Platform (EOP, Padó et al. 2014; Magnini et al. 2014)—an open source platform for RTE research. EOP is a tool for quickly developing NLI systems while sharing components such as common lexical resources and evaluation sets. We evaluate on two algorithms included in the distribution: a simple edit-distance based algorithm and a classifier-based algorithm, the latter both in a bare form and augmented with EOP’s full suite of lexical resources.
第一类模型来自 Excitement Open Platform(EOP,Padó et al. 2014; Magnini et al. 2014)——一个用于 RTE 研究的开源平台。EOP 的作用是在共享常见词汇资源和评估集等组件的同时,快速开发 NLI 系统。我们评估发行版中包含的两种算法:一种简单的基于编辑距离的算法,以及一种基于分类器的算法;后者既评估其基本形式,也评估加入 EOP 全部词汇资源后的形式。
Our initial goal was to better understand the difficulty of the task of classifying SNLI corpus inferences, rather than necessarily the performance of a state-of-the-art RTE system. We approached this by running the same system on several data sets: our own test set, the SICK test data, and the standard RTE-3 test set (Giampiccolo et al., 2007). We report results in Table 4. Each of the models was separately trained on the training set of each corpus. All models are evaluated only on 2-class entailment. To convert 3-class problems like SICK and SNLI to this setting, all instances of contradiction and unknown are converted to nonentailment. This yields a most-frequent-class baseline accuracy of 66% on SNLI, and 71% on SICK. This is intended primarily to demonstrate the difficulty of the task, rather than necessarily the performance of a state-of-the-art RTE system. The edit distance algorithm tunes the weight of the three case-insensitive edit distance operations on the training set, after removing stop words. In addition to the base classifier-based system distributed with the platform, we train a variant which includes information from WordNet (Miller, 1995) and VerbOcean (Chklovski and Pantel, 2004), and makes use of features based on tree patterns and dependency tree skeletons (Wang and Neumann, 2007).
我们最初的目标,是更好地理解对 SNLI 语料中的推理做分类这一任务有多难,而不一定是要看最先进的 RTE 系统能达到怎样的成绩。为此,我们在多个数据集上运行同一系统:我们自己的测试集、SICK 测试数据,以及标准的 RTE-3 测试集(Giampiccolo et al., 2007)。结果见表 4。每个模型都在各语料的训练集上单独训练。所有模型都只在二分类蕴涵设定下评估。要把 SICK 和 SNLI 这类三分类问题转换到这一设定,所有矛盾和未知的例子都被转为非蕴涵。这样得到的最大类基线准确率在 SNLI 上是 66%,在 SICK 上是 71%。这样做主要是为了说明任务难度,而不一定是为了衡量最先进 RTE 系统的表现。编辑距离算法在去掉停用词后,在训练集上调节三种不区分大小写的编辑距离操作的权重。除了平台发行的基本分类器系统外,我们还训练了一个变体,它加入了来自 WordNet(Miller, 1995)和 VerbOcean(Chklovski and Pantel, 2004)的信息,并使用基于树模式和依存树骨架的特征(Wang and Neumann, 2007)。
Table 4: 2-class test accuracy for two simple baseline systems included in the Excitement Open Platform, as well as SICK and RTE results for a model making use of more sophisticated lexical resources.
| System | SNLI | SICK | RTE-3 |
|---|---|---|---|
| Edit Distance Based | 71.9 | 65.4 | 61.9 |
| Classifier Based | 72.2 | 71.4 | 61.5 |
| + Lexical Resources | 75.0 | 78.8 | 63.6 |
表 4:Excitement Open Platform 中两个简单基线系统的二分类测试准确率,以及一个使用更复杂词汇资源的模型在 SICK 和 RTE 上的结果。
| 系统 | SNLI | SICK | RTE-3 |
|---|---|---|---|
| Edit Distance Based | 71.9 | 65.4 | 61.9 |
| Classifier Based | 72.2 | 71.4 | 61.5 |
| + Lexical Resources | 75.0 | 78.8 | 63.6 |
3.2 Lexicalized Classifier
Unlike the RTE datasets, SNLI’s size supports approaches which make use of rich lexicalized features. We evaluate a simple lexicalized classifier to explore the ability of non-specialized models to exploit these features in lieu of more involved language understanding. Our classifier implements 6 feature types; 3 unlexicalized and 3 lexicalized:
与 RTE 数据集不同,SNLI 的规模足以支持使用丰富词法特征的方法。我们评估一个简单的词法分类器,以考察非专门化的模型能否利用这些特征,从而替代更复杂的语言理解。我们的分类器实现了 6 类特征;其中 3 类是非词法的,3 类是词法的:
- The BLEU score of the hypothesis with respect to the premise, using an n-gram length between 1 and 4.
- 假设句相对于前提句的 BLEU 分数,所用 n-gram 长度介于 1 到 4 之间。
- The length difference between the hypothesis and the premise, as a real-valued feature.
- 假设句与前提句的长度差,作为实值特征。
- The overlap between words in the premise and hypothesis, both as an absolute count and a percentage of possible overlap, and both over all words and over just nouns, verbs, adjectives, and adverbs.
- 前提和假设中词的重叠,既作为绝对计数,也作为相对于可能重叠的百分比;既在全部词上计算,也只在名词、动词、形容词和副词上计算。
- An indicator for every unigram and bigram in the hypothesis.
- 假设句中每一个一元语法和二元语法的指示特征。
- Cross-unigrams: for every pair of words across the premise and hypothesis which share a POS tag, an indicator feature over the two words.
- 跨句一元语法:对前提和假设中每一对具有相同词性标记的词,用这两个词构成一个指示特征。
- Cross-bigrams: for every pair of bigrams across the premise and hypothesis which share a POS tag on the second word, an indicator feature over the two bigrams.
- 跨句二元语法:对前提和假设中每一对在第二个词上具有相同词性标记的二元语法,用这两个二元语法构成一个指示特征。
We report results in Table 5, along with ablation studies for removing the cross-bigram features (leaving only the cross-unigram feature) and for removing all lexicalized features. On our large corpus in particular, there is a substantial jump in accuracy from using lexicalized features, and another from using the very sparse cross-bigram features. The latter result suggests that there is value in letting the classifier automatically learn to recognize structures like explicit negations and adjective modification. A similar result was shown in Wang and Manning (2012) for bigram features in sentiment analysis.
结果见表 5,其中还包括去掉跨句二元语法特征(只留下跨句一元语法特征)以及去掉全部词法特征的消融实验。尤其是在我们这个大规模语料上,使用词法特征会带来准确率的大幅提升,使用非常稀疏的跨句二元语法特征还会再提升一截。后一结果表明,让分类器自动学会识别显式否定和形容词修饰这类结构是有价值的。Wang and Manning(2012)在情感分析的二元语法特征上也给出了类似结果。
Table 5: 3-class accuracy, training on either our data or SICK, including models lacking cross-bigram features (Feature 6), and lacking all lexical features (Features 4–6). We report results both on the test set and the training set to judge overfitting.
| System | SNLI Train | SNLI Test | SICK Train | SICK Test |
|---|---|---|---|---|
| Lexicalized | 99.7 | 78.2 | 90.4 | 77.8 |
| Unigrams Only | 93.1 | 71.6 | 88.1 | 77.0 |
| Unlexicalized | 49.4 | 50.4 | 69.9 | 69.6 |
表 5:三分类准确率,分别在我们的数据或 SICK 上训练,包括去掉跨句二元语法特征(特征 6)以及去掉全部词法特征(特征 4–6)的模型。我们同时报告测试集和训练集上的结果,以判断过拟合。
| 系统 | SNLI 训练 | SNLI 测试 | SICK 训练 | SICK 测试 |
|---|---|---|---|---|
| Lexicalized | 99.7 | 78.2 | 90.4 | 77.8 |
| Unigrams Only | 93.1 | 71.6 | 88.1 | 77.0 |
| Unlexicalized | 49.4 | 50.4 | 69.9 | 69.6 |
It is surprising that the classifier performs as well as it does without any notion of alignment or tree transformations. Although we expect that richer models would perform better, the results suggest that given enough data, cross bigrams with the noisy part-of-speech overlap constraint can produce an effective model.
令人惊讶的是,这个分类器在没有任何对齐或树变换概念的情况下,表现已经如此之好。尽管我们预期更丰富的模型还会更好,但这些结果表明,只要数据足够,带有带噪声的词性重叠约束的跨句二元语法,也可以形成一个有效的模型。
3.3 Sentence embeddings and NLI
SNLI is suitably large and diverse to make it possible to train neural network models that produce distributed representations of sentence meaning. In this section, we compare the performance of three such models on the corpus. To focus specifically on the strengths of these models at producing informative sentence representations, we use sentence embedding as an intermediate step in the NLI classification task: each model must produce a vector representation of each of the two sentences without using any context from the other sentence, and the two resulting vectors are then passed to a neural network classifier which predicts the label for the pair. This choice allows us to focus on existing models for sentence embedding, and it allows us to evaluate the ability of those models to learn useful representations of meaning (which may be independently useful for subsequent tasks), at the cost of excluding from consideration possible strong neural models for NLI that directly compare the two inputs at the word or phrase level.
SNLI 的规模和多样性已经足以训练能够产生句子意义分布式表示的神经网络模型。本节比较三种此类模型在该语料上的表现。为了专门考察这些模型在生成有信息量的句子表示上的能力,我们把句嵌入作为 NLI 分类任务中的中间步骤:每个模型必须在不使用另一句上下文的情况下,为两句各产生一个向量表示,再把这两个向量送入一个神经网络分类器,由后者预测该对的标签。这一选择让我们可以聚焦于已有的句嵌入模型,并评估这些模型学习有用意义表示的能力(这些表示也可能独立用于后续任务),代价是不考虑那些在词或短语层面直接比较两个输入、可能很强的 NLI 神经网络模型。
图 3:神经网络分类结构:对表 6 和表 7 中评估的每一种句嵌入模型,都运行两个相同的副本,分别以两句为输入,其输出即这里所示的两个 100 维输入。
Our neural network classifier, depicted in Figure 3 (and based on a one-layer model in Bowman et al. 2015), is simply a stack of three 200d tanh layers, with the bottom layer taking the concatenated sentence representations as input and the top layer feeding a softmax classifier, all trained jointly with the sentence embedding model itself.
我们的神经网络分类器见图 3(基于 Bowman et al. 2015 中的单层模型),它只是三层 200 维 tanh 层的堆叠:最底层以拼接后的句子表示作为输入,最顶层送入 softmax 分类器,并与句嵌入模型本身联合训练。
We test three sentence embedding models, each set to use 100d phrase and sentence embeddings. Our baseline sentence embedding model simply sums the embeddings of the words in each sentence. In addition, we experiment with two simple sequence embedding models: a plain RNN and an LSTM RNN (Hochreiter and Schmidhuber, 1997).
我们测试三种句嵌入模型,均设定使用 100 维的短语和句子嵌入。基线句嵌入模型只是把各句中词的嵌入求和。此外,我们还实验了两种简单的序列嵌入模型:普通 RNN,以及 LSTM RNN(Hochreiter and Schmidhuber, 1997)。
The word embeddings for all of the models are initialized with the 300d reference GloVe vectors (840B token version, Pennington et al. 2014) and fine-tuned as part of training. In addition, all of the models use an additional tanh neural network layer to map these 300d embeddings into the lower-dimensional phrase and sentence embedding space. All of the models are randomly initialized using standard techniques and trained using AdaDelta (Zeiler, 2012) minibatch SGD until performance on the development set stops improving. We applied L2 regularization to all models, manually tuning the strength coefficient λ for each, and additionally applied dropout (Srivastava et al., 2014) to the inputs and outputs of the sentence embedding models (though not to its internal connections) with a fixed dropout rate. All models were implemented in a common framework for this paper, and the implementations will be made available at publication time.
所有模型的词嵌入都用 300 维的参考 GloVe 向量初始化(840B 词元版本,Pennington et al. 2014),并在训练中微调。此外,所有模型都再使用一个 tanh 神经网络层,把这些 300 维嵌入映射到更低维的短语和句子嵌入空间。所有模型都用标准技术随机初始化,并用 AdaDelta(Zeiler, 2012)小批量 SGD 训练,直到开发集上的表现不再提高。我们对所有模型都使用了 L2 正则,并分别手工调节强度系数 λ;还对句嵌入模型的输入和输出(但不对其内部连接)使用了固定比率的 dropout(Srivastava et al., 2014)。本文中的所有模型都在同一框架中实现,实现将在发表时公开。
Table 6: Accuracy in 3-class classification on our training and test sets for each model.
| Sentence model | Train | Test |
|---|---|---|
| 100d Sum of words | 79.3 | 75.3 |
| 100d RNN | 73.1 | 72.2 |
| 100d LSTM RNN | 84.8 | 77.6 |
表 6:各模型在我们的训练集和测试集上的三分类准确率。
| 句子模型 | 训练 | 测试 |
|---|---|---|
| 100d Sum of words | 79.3 | 75.3 |
| 100d RNN | 73.1 | 72.2 |
| 100d LSTM RNN | 84.8 | 77.6 |
The results are shown in Table 6. The sum of words model performed slightly worse than the fundamentally similar lexicalized classifier—while the sum of words model can use pretrained word embeddings to better handle rare words, it lacks even the rudimentary sensitivity to word order that the lexicalized model’s bigram features provide. Of the two RNN models, the LSTM’s more robust ability to learn long-term dependencies serves it well, giving it a substantial advantage over the plain RNN, and resulting in performance that is essentially equivalent to the lexicalized classifier on the test set (LSTM performance near the stopping iteration varies by up to 0.5% between evaluation steps). While the lexicalized model fits the training set almost perfectly, the gap between train and test set accuracy is relatively small for all three neural network models, suggesting that research into significantly higher capacity versions of these models would be productive.
结果见表 6。词向量求和模型的表现略差于本质上相近的词法分类器——虽然词向量求和模型可以用预训练词嵌入更好地处理稀有词,但它甚至不具备词法模型的二元语法特征所提供的那种初步的词序敏感性。在两种 RNN 模型中,LSTM 更稳健地学习长期依赖的能力对它很有帮助,使它明显优于普通 RNN,并在测试集上达到与词法分类器基本相当的表现(接近停止迭代时,LSTM 的成绩在各评估步之间最多会有 0.5% 的波动)。词法模型几乎完美拟合了训练集,而三种神经网络模型的训练与测试准确率差距都相对较小,这表明研究容量明显更大的这类模型版本会是有成效的。
3.4 Analysis and discussion
Figure 4 shows a learning curve for the LSTM and the lexicalized and unlexicalized feature-based models. It shows that the large size of the corpus is crucial to both the LSTM and the lexicalized model, and suggests that additional data would yield still better performance for both. In addition, though the LSTM and the lexicalized model show similar performance when trained on the current full corpus, the somewhat steeper slope for the LSTM hints that its ability to learn arbitrarily structured representations of sentence meaning may give it an advantage over the more constrained lexicalized model on still larger datasets.
图 4 给出了 LSTM 以及带词法和不带词法的特征模型的学习曲线。它表明,语料的大规模对 LSTM 和词法模型都很关键,并暗示再增加数据仍会进一步提高二者的表现。此外,虽然 LSTM 和词法模型在当前完整语料上训练时表现相近,但 LSTM 的斜率略陡,暗示它学习任意结构的句子意义表示的能力,可能在更大的数据集上给它带来相对于更受约束的词法模型的优势。
Figure 4: A learning curve showing how the baseline classifiers and the LSTM perform when trained to convergence on varied amounts of training data. The y-axis starts near a random-chance accuracy of 33%. The minibatch size of 64 that we used to tune the LSTM sets a lower bound on data for that model.
图 4:一条学习曲线,显示基线分类器和 LSTM 在不同规模的训练数据上训练至收敛时的表现。纵轴起点接近随机猜测的 33% 准确率。我们用来调节 LSTM 的批量大小 64,为该模型设定了数据量的下限。
We were struck by the speed with which the lexicalized classifier outperforms its unlexicalized counterpart. With only 100 training examples, the cross-bigram classifier is already performing better. Empirically, we find that the top weighted features for the classifier trained on 100 examples tend to be high precision entailments; e.g., playing → outside (most scenes are outdoors), a banana → person eating. If relatively few spurious entailments get high weight—as it appears is the case—then it makes sense that, when these do fire, they boost accuracy in identifying entailments.
词法分类器超过非词法对照模型的速度,给我们留下了深刻印象。只需 100 个训练例子,跨句二元语法分类器就已经表现更好。经验上看,在这 100 个例子上训练的分类器中,权重最高的特征往往是高精度的蕴涵,例如 playing → outside(多数场景在户外),a banana → person eating。如果只有相对较少的虚假蕴涵获得高权重——看起来正是如此——那么当这些特征被触发时,它们就会提高识别蕴涵的准确率,这是说得通的。
There are revealing patterns in the errors common to all the models considered here. Despite the large size of the training corpus and the distributional information captured by GloVe initialization, many lexical relationships are still misanalyzed, leading to incorrect predictions of independent, even for pairs that are common in the training corpus like beach/surf and sprinter/runner. Semantic mistakes at the phrasal level (e.g., predicting contradiction for A male is placing an order in a deli/A man buying a sandwich at a deli) indicate that additional attention to compositional semantics would pay off. However, many of the persistent problems run deeper, to inferences that depend on world knowledge and context-specific inferences, as in the entailment pair A race car driver leaps from a burning car/A race car driver escaping danger, for which both the lexicalized classifier and the LSTM predict neutral. In other cases, the models’ attempts to shortcut this kind of inference through lexical cues can lead them astray. Some of these examples have qualities reminiscent of Winograd schemas (Winograd, 1972; Levesque, 2013). For example, all the models wrongly predict entailment for A young girl throws sand toward the ocean/A girl can’t stand the ocean, presumably because of distributional associations between throws and can’t stand.
这里考察的所有模型都有一些共同的、很能说明问题的错误模式。尽管训练语料规模很大,GloVe 初始化也捕捉了分布信息,许多词汇关系仍会被分析错,从而错误地预测为独立,哪怕是训练集中常见的配对,如 beach/surf 和 sprinter/runner。短语层面的语义错误(例如把 A male is placing an order in a deli/A man buying a sandwich at a deli 预测为矛盾)表明,对组合语义给予更多关注会有回报。不过,许多顽固问题要更深,涉及依赖世界知识和情境特定推理的推断,例如蕴涵对 A race car driver leaps from a burning car/A race car driver escaping danger,词法分类器和 LSTM 都预测为中立。在另一些情形中,模型试图借助词汇线索走捷径来完成这类推理,反而会走偏。有些例子带有类似 Winograd schema 的性质(Winograd, 1972; Levesque, 2013)。例如,对 A young girl throws sand toward the ocean/A girl can’t stand the ocean,所有模型都错误地预测为蕴涵,这大概是因为 throws 与 can’t stand 之间的分布关联。
Analysis of the models’ predictions also yields insights into the extent to which they grapple with event and entity coreference. For the most part, the original image prompts contained a focal element that the caption writer identified with a syntactic subject, following information structuring conventions associating subjects and topics in English (Ward and Birner, 2004). Our annotators generally followed suit, writing sentences that, while structurally diverse, share topic/focus (theme/rheme) structure with their premises. This promotes a coherent, situation-specific construal of each sentence pair. This is information that our models can easily take advantage of, but it can lead them astray. For instance, all of them stumble with the amusingly simple case A woman prepares ingredients for a bowl of soup/A soup bowl prepares a woman, in which prior expectations about parallelism are not met. Another headline example of this type is A man wearing padded arm protection is being bitten by a German shepherd dog/A man bit a dog, which all the models wrongly diagnose as entailment, though the sentences report two very different stories. A model with access to explicit information about syntactic or semantic structure should perform better on cases like these.
对模型预测的分析,也让我们看到它们在处理事件和实体共指上走到了哪一步。在大多数情况下,原来的图像提示中有一个焦点成分,描述作者按英语中主语与话题相关联的信息结构惯例(Ward and Birner, 2004),把它写成了句法主语。我们的标注者大体上也照此办理,写出的句子虽然结构多样,但与前提共享话题/焦点(主位/述位)结构。这有助于对每个句子对形成连贯的、针对具体情境的理解。这类信息很容易被我们的模型利用,但也可能把它们带偏。例如,它们都会在这个简单得有些好笑的例子上跌倒:A woman prepares ingredients for a bowl of soup/A soup bowl prepares a woman,其中关于平行结构的先验预期并未成立。这类问题的另一个典型例子是 A man wearing padded arm protection is being bitten by a German shepherd dog/A man bit a dog,所有模型都错误地判断为蕴涵,尽管这两句讲的是完全不同的故事。若模型能够获得关于句法或语义结构的显式信息,在这类情形上应当会表现更好。
4 Transfer learning with SICK
To the extent that successfully training a neural network model like our LSTM on SNLI forces that model to encode broadly accurate representations of English scene descriptions and to build an entailment classifier over those relations, we should expect it to be readily possible to adapt the trained model for use on other NLI tasks. In this section, we evaluate on the SICK entailment task using a simple transfer learning method (Pratt et al., 1991) and achieve competitive results.
如果在 SNLI 上成功训练像我们的 LSTM 这样的神经网络,会迫使模型编码出大体准确的英语场景描述表示,并在这些关系上建立起蕴涵分类器,那么我们就应预期,把训练好的模型适配到其他 NLI 任务上是完全可行的。本节用一种简单的迁移学习方法(Pratt et al., 1991)在 SICK 蕴涵任务上做评估,并取得了有竞争力的结果。
To perform transfer, we take the parameters of the LSTM RNN model trained on SNLI and use them to initialize a new model, which is trained from that point only on the training portion of SICK. The only newly initialized parameters are softmax layer parameters and the embeddings for words that appear in SICK, but not in SNLI (which are populated with GloVe embeddings as above). We use the same model hyperparameters that were used to train the original model, with the exception of the L2 regularization strength, which is re-tuned. We additionally transfer the accumulators that are used by AdaDelta to set the learning rates. This lowers the starting learning rates, and is intended to ensure that the model does not learn too quickly in its first few epochs after transfer and destroy the knowledge accumulated in the pre-transfer phase of training.
做迁移时,我们取出在 SNLI 上训练好的 LSTM RNN 模型参数,用它们初始化一个新模型,此后只在 SICK 的训练部分上继续训练。唯一重新初始化的参数,是 softmax 层参数,以及出现在 SICK 中但不在 SNLI 中的词的嵌入(仍按上文用 GloVe 嵌入填充)。除了重新调节的 L2 正则强度外,我们使用与训练原模型相同的超参数。我们还迁移 AdaDelta 用来设定学习率的累加器。这会降低起始学习率,目的是确保模型在迁移后的前几个 epoch 里不会学得太快,从而破坏迁移前阶段积累的知识。
Table 7: LSTM 3-class accuracy on the SICK train and test sets under three training regimes.
| Training sets | Train | Test |
|---|---|---|
| Our data only | 42.0 | 46.7 |
| SICK only | 100.0 | 71.3 |
| Our data and SICK (transfer) | 99.9 | 80.8 |
表 7:在三种训练设定下,LSTM 在 SICK 训练集和测试集上的三分类准确率。
| 训练集 | 训练 | 测试 |
|---|---|---|
| 仅用我们的数据 | 42.0 | 46.7 |
| 仅用 SICK | 100.0 | 71.3 |
| 我们的数据与 SICK(迁移) | 99.9 | 80.8 |
The results are shown in Table 7. Training on SICK alone yields poor performance, and the model trained on SNLI fails when tested on SICK data, labeling more neutral examples as contradictions than correctly, possibly as a result of subtle differences in how the labeling task was presented. In contrast, transferring SNLI representations to SICK yields the best performance yet reported for an unaugmented neural network model, surpasses the available EOP models, and approaches both the overall state of the art at 84.6% (Lai and Hockenmaier, 2014) and the 84% level of interannotator agreement, which likely represents an approximate performance ceiling. This suggests that the introduction of a large high-quality corpus makes it possible to train representation-learning models for sentence meaning that are competitive with the best hand-engineered models on inference tasks.
结果见表 7。只在 SICK 上训练,表现较差;只在 SNLI 上训练的模型在 SICK 数据上测试也会失败,把更多中立例子标成了矛盾而不是标对,这可能是因为标注任务的呈现方式存在细微差别。相比之下,把 SNLI 表示迁移到 SICK,得到了未经增强的神经网络模型当时最好的报告成绩,超过了可用的 EOP 模型,并接近总体最好水平 84.6%(Lai and Hockenmaier, 2014)以及 84% 的标注者间一致率,后者很可能代表一个大致的表现上限。这表明,引入大规模、高质量的语料,使人们可以训练出在推理任务上能与最好的手工设计模型相竞争的句子意义表示学习模型。
We attempted to apply this same transfer evaluation technique to the RTE-3 challenge, but found that the small training set (800 examples) did not allow the model to adapt to the unfamiliar genre of text used in that corpus, such that no training configuration yielded competitive performance. Further research on effective transfer learning on small data sets with neural models might facilitate improvements here.
我们曾试图把同一套迁移评估方法用到 RTE-3 挑战上,但发现其很小的训练集(800 个例子)不足以让模型适应该语料中不熟悉的文本体裁,因此没有任何训练配置能得到有竞争力的表现。若能进一步研究如何在小数据集上用神经模型做有效的迁移学习,或许会带来改进。
5 Conclusion
Natural languages are powerful vehicles for reasoning, and nearly all questions about meaningfulness in language can be reduced to questions of entailment and contradiction in context. This suggests that NLI is an ideal testing ground for theories of semantic representation, and that training for NLI tasks can provide rich domain-general semantic representations. To date, however, it has not been possible to fully realize this potential due to the limited nature of existing NLI resources. This paper sought to remedy this with a new, large-scale, naturalistic corpus of sentence pairs labeled for entailment, contradiction, and independence. We used this corpus to evaluate a range of models, and found that both simple lexicalized models and neural network models perform well, and that the representations learned by a neural network model on our corpus can be used to dramatically improve performance on a standard challenge dataset. We hope that SNLI presents valuable training data and a challenging testbed for the continued application of machine learning to semantic representation.
自然语言是强大的推理载体,几乎所有关于语言是否有意义的问题,都可以还原为情境中的蕴涵和矛盾问题。这表明,NLI 是检验语义表示理论的理想试验场,而为 NLI 任务做训练,也可以提供丰富的、领域通用的语义表示。然而迄今为止,由于现有 NLI 资源的局限,这一潜力还无法充分实现。本文试图用一套新的、大规模的、自然的句子对语料来弥补这一点,这些句子对被标为蕴涵、矛盾或独立。我们用这一语料评估了一系列模型,发现简单的词法模型和神经网络模型都表现很好,而且神经网络在我们语料上学到的表示,可以显著提高在标准挑战数据集上的表现。我们希望 SNLI 能为机器学习在语义表示上的继续应用,提供有价值的训练数据和一项有挑战的试验平台。
Acknowledgments
We gratefully acknowledge support from a Google Faculty Research Award, a gift from Bloomberg L.P., the Defense Advanced Research Projects Agency (DARPA) Deep Exploration and Filtering of Text (DEFT) Program under Air Force Research Laboratory (AFRL) contract no. FA8750-13-2-0040, the National Science Foundation under grant no. IIS 1159679, and the Department of the Navy, Office of Naval Research, under grant no. N00014-10-1-0109. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Google, Bloomberg L.P., DARPA, AFRL NSF, ONR, or the US government. We also thank our many excellent Mechanical Turk contributors.
我们衷心感谢 Google Faculty Research Award、Bloomberg L.P. 的赠款、美国国防高级研究计划局(DARPA)Deep Exploration and Filtering of Text(DEFT)项目在空军研究实验室(AFRL)合同号 FA8750-13-2-0040 下的支持、美国国家科学基金会资助号 IIS 1159679,以及海军部海军研究办公室资助号 N00014-10-1-0109。本文中的任何意见、发现、结论或建议都只代表作者本人,不一定反映 Google、Bloomberg L.P.、DARPA、AFRL NSF、ONR 或美国政府的观点。我们也感谢众多出色的 Mechanical Turk 参与者。
References
Johan Bos and Katja Markert. 2005. Recognising textual entailment with logical inference. In Proc. EMNLP.
Samuel R. Bowman, Christopher Potts, and Christopher D. Manning. 2015. Recursive neural networks can learn logical semantics. In Proc. of the 3rd Workshop on Continuous Vector Space Models and their Compositionality.
Timothy Chklovski and Patrick Pantel. 2004. VerbOcean: Mining the web for fine-grained semantic verb relations. In Proc. EMNLP.
Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G. Bobrow. 2003. Entailment, intensionality and text understanding. In Proc. of the HLT-NAACL 2003 Workshop on Text Meaning.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. Evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177–190. Springer.
Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In Proc. ACL.
W. Nelson Francis and Henry Kucera. 1979. Brown corpus manual. Brown University.
Yaroslav Fyodorov, Yoad Winter, and Nissim Francez. 2000. A natural logic inference system. In Proc. of the 2nd Workshop on Inference in Computational Semantics.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proc. of the ACL-PASCAL workshop on textual entailment and paraphrasing.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
Jerrold J. Katz. 1972. Semantic Theory. Harper & Row, New York.
Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proc. ACL.
Alice Lai and Julia Hockenmaier. 2014. Illinois-LH: A denotational and distributional approach to semantics. In Proc. SemEval.
Hector J. Levesque. 2013. On our best behaviour. In Proc. AAAI.
Omer Levy, Ido Dagan, and Jacob Goldberger. 2014. Focused entailment graphs for open IE propositions. In Proc. CoNLL.
Bill MacCartney and Christopher D Manning. 2009. An extended model of natural logic. In Proc. of the Eighth International Conference on Computational Semantics.
Bernardo Magnini, Roberto Zanoli, Ido Dagan, Kathrin Eichler, Günter Neumann, Tae-Gil Noh, Sebastian Pado, Asher Stern, and Omer Levy. 2014. The Excitement Open Platform for textual inferences. Proc. ACL.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014a. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proc. SemEval.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014b. A SICK cure for the evaluation of compositional distributional semantic models. In Proc. LREC.
George A Miller. 1995. WordNet: a lexical database for english. Communications of the ACM, 38(11):39–41.
Sebastian Padó, Tae-Gil Noh, Asher Stern, Rui Wang, and Roberto Zanoli. 2014. Design and realization of a modular architecture for textual entailment. Journal of Natural Language Engineering.
Ellie Pavlick, Johan Bos, Malvina Nissim, Charley Beller, Ben Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proc. ACL.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proc. EMNLP.
Lorien Y Pratt, Jack Mostow, Candace A Kamm, and Ace A Kamm. 1991. Direct transfer of learned information among neural networks. In Proc. AAAI.
Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. EMNLP.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. JMLR.
Johan van Benthem. 2008. A brief history of natural logic. In M. Chakraborty, B. Löwe, M. Nath Mitra, and S. Sarukki, editors, Logic, Navya-Nyaya and Applications: Homage to Bimal Matilal. College Publications.
Sida I. Wang and Christopher D. Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proc. ACL.
Rui Wang and Günter Neumann. 2007. Recognizing textual entailment using sentence similarity based on dependency tree skeletons. In ACL-PASCAL Workshop on Textual Entailment and Paraphrasing.
Gregory Ward and Betty Birner. 2004. Information structure and non-canonical syntax. In Laurence R. Horn and Gregory Ward, editors, Handbook of Pragmatics, pages 153–174. Blackwell, Oxford.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015a. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv:1502.05698.
Jason Weston, Sumit Chopra, and Antoine Bordes. 2015b. Memory networks. In Proc. ICLR.
Terry Winograd. 1972. Understanding natural language. Cognitive Psychology, 3(1):1–191.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67–78.
Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701.



