SQuAD: 100,000+ Questions for Machine Comprehension of Text

SQuAD：面向文本机器理解的十万余道问题

Pranav Rajpurkar and Jian Zhang and Konstantin Lopyrev and Percy Liang
{pranavsr,zjian,klopyrev,pliang}@cs.stanford.edu
Computer Science Department
Stanford University

Abstract

摘要

We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com.

我们发布斯坦福问答数据集（Stanford Question Answering Dataset，SQuAD）。这是一套新的阅读理解数据集，由众包工人针对一组维基百科文章提出十万余道问题，每道题的答案都是对应阅读段落中的一段原文。我们分析了回答这些问题所需的推理类型，分析时主要依靠依存树和成分树。我们还训练了一个较强的逻辑回归模型，F1 达到 51.0%，明显高于简单基线的 20%。不过人类表现是 86.8%，高出不少，说明这套数据对后续研究仍是一道有分量的挑战题。数据集可在 https://stanford-qa.com 免费获取。

1 Introduction

1 引言

In meteorology, precipitation is any product of the condensation of atmospheric water vapor that falls under gravity. The main forms of precipitation include drizzle, rain, sleet, snow, graupel and hail... Precipitation forms as smaller droplets coalesce via collision with other rain drops or ice crystals within a cloud. Short, intense periods of rain in scattered locations are called “showers”.

What causes precipitation to fall?
gravity
What is another main form of precipitation besides drizzle, rain, snow, sleet and hail?
graupel
Where do water droplets collide with ice crystals to form precipitation?
within a cloud

Figure 1: Question-answer pairs for a sample passage in the SQuAD dataset. Each of the answers is a segment of text from the passage.

图 1：SQuAD 数据集中一段样例段落及其问答对。每个答案都是段落里的一段原文。

Reading Comprehension (RC), or the ability to read text and then answer questions about it, is a challenging task for machines, requiring both understanding of natural language and knowledge about the world. Consider the question “what causes precipitation to fall?” posed on the passage in Figure 1. In order to answer the question, one might first locate the relevant part of the passage “precipitation ... falls under gravity”, then reason that “under” refers to a cause (not location), and thus determine the correct answer: “gravity”.

阅读理解（Reading Comprehension，RC）是指读完一段文字后再回答与之相关的问题。这对机器来说并不容易，既要理解自然语言，也要具备世界知识。以图 1 段落上的问题“what causes precipitation to fall?”为例：要答题，也许会先定位到段落里“precipitation ... falls under gravity”这一处，再判断这里的“under”说的是原因而不是位置，从而得出正确答案“gravity”。

How can we get a machine to make progress on the challenging task of reading comprehension? Historically, large, realistic datasets have played a critical role for driving fields forward—famous examples include ImageNet for object recognition (Deng et al., 2009) and the Penn Treebank for syntactic parsing (Marcus et al., 1993). Existing datasets for RC have one of two shortcomings: (i) those that are high in quality (Richardson et al., 2013; Berant et al., 2014) are too small for training modern data-intensive models, while (ii) those that are large (Hermann et al., 2015; Hill et al., 2015) are semi-synthetic and do not share the same characteristics as explicit reading comprehension questions.

怎样才能让机器在这项困难的阅读理解任务上往前走？历史上，大规模、贴近真实的数据集往往是推动一个领域前进的关键——物体识别里的 ImageNet（Deng et al., 2009）和句法分析里的 Penn Treebank（Marcus et al., 1993）就是常被提起的例子。已有的阅读理解数据集通常会落进两个短板之一：（i）质量高的（Richardson et al., 2013; Berant et al., 2014）规模太小，养不动如今这种吃数据的模型；（ii）规模大的（Hermann et al., 2015; Hill et al., 2015）又是半合成的，和真正的阅读理解提问并不同一类。

To address the need for a large and high-quality reading comprehension dataset, we present the Stanford Question Answering Dataset v1.0 (SQuAD), freely available at https://stanford-qa.com, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage. SQuAD contains 107,785 question-answer pairs on 536 articles, and is almost two orders of magnitude larger than previous manually labeled RC datasets such as MCTest (Richardson et al., 2013).

为了补上“既要大、又要质量高”这块缺口，我们发布斯坦福问答数据集 1.0 版（SQuAD），可在 https://stanford-qa.com 免费获取。数据由众包工人针对一组维基百科文章提问构成，每道题的答案都是对应阅读段落中的一段原文，也就是一个跨度（span）。SQuAD 覆盖 536 篇文章、共 107,785 个问答对，规模几乎比此前人工标注的阅读理解数据集（如 MCTest，Richardson et al., 2013）高出近两个数量级。

In contrast to prior datasets, SQuAD does not provide a list of answer choices for each question. Rather, systems must select the answer from all possible spans in the passage, thus needing to cope with a fairly large number of candidates. While questions with span-based answers are more constrained than the more interpretative questions found in more advanced standardized tests, we still find a rich diversity of questions and answer types in SQuAD. We develop automatic techniques based on distances in dependency trees to quantify this diversity and stratify the questions by difficulty. The span constraint also comes with the important benefit that span-based answers are easier to evaluate than free-form answers.

和以往数据不同，SQuAD 不为每道题提供备选项。系统必须从段落里所有可能的跨度中选出答案，候选数量相当大。以跨度为答案的问题，确实比更高级标准化考试里那些偏解释性的问题更受约束，但我们仍然在 SQuAD 里看到了丰富的问题类型和答案类型。我们基于依存树上的距离，设计了自动方法来量化这种多样性，并按难度把问题分层。跨度约束还有一个实际好处：比起自由作答，基于跨度的答案更容易评测。

To assess the difficulty of SQuAD, we implemented a logistic regression model with a range of features. We find that lexicalized and dependency tree path features are important to the performance of the model. We also find that the model performance worsens with increasing complexity of (i) answer types and (ii) syntactic divergence between the question and the sentence containing the answer; interestingly, there is no such degradation for humans. Our best model achieves an F1 score of 51.0%, [All experimental results in this paper are on SQuAD v1.0.] which is much better than the sliding window baseline (20%). Over the last four months (since June 2016), we have witnessed significant improvements from more sophisticated neural network-based models. For example, Wang and Jiang (2016) obtained 70.3% F1 on SQuAD v1.1 (results on v1.0 are similar). These results are still well behind human performance, which is 86.8% F1 based on inter-annotator agreement. This suggests that there is plenty of room for advancement in modeling and learning on the SQuAD dataset.

为了衡量 SQuAD 的难度，我们实现了一个带多种特征的逻辑回归模型。结果表明，词汇化特征和依存树路径特征对模型表现很重要。随着（i）答案类型变复杂、以及（ii）问题与含答案句子之间的句法分歧变大，模型成绩会下降；有意思的是，人类并没有出现同样的下滑。我们最好的模型 F1 为 51.0%［本文全部实验结果均在 SQuAD v1.0 上报告］，明显高于滑动窗口基线的 20%。在过去四个月里（自 2016 年 6 月以来），更复杂的神经网络模型已经带来明显提升。例如，Wang and Jiang (2016) 在 SQuAD v1.1 上得到 70.3% 的 F1（在 v1.0 上的结果相近）。这些成绩距离人类表现仍有不小差距——按标注者一致性计算，人类 F1 为 86.8%。这说明，在 SQuAD 上，建模和学习都还有很大的前进空间。

2 Existing Datasets

2 已有数据集

Dataset	Question source	Formulation	Size
SQuAD	crowdsourced	RC, spans in passage	100K
MCTest (Richardson et al., 2013)	crowdsourced	RC, multiple choice	2640
Algebra (Kushman et al., 2014)	standardized tests	computation	514
Science (Clark and Etzioni, 2016)	standardized tests	reasoning, multiple choice	855
WikiQA (Yang et al., 2015)	query logs	IR, sentence selection	3047
TREC-QA (Voorhees and Tice, 2000)	query logs + human editor	IR, free form	1479
CNN/Daily Mail (Hermann et al., 2015)	summary + cloze	RC, fill in single entity	1.4M
CBT (Hill et al., 2015)	cloze	RC, fill in single word	688K

Table 1: A survey of several reading comprehension and question answering datasets. SQuAD is much larger than all datasets except the semi-synthetic cloze-style datasets, and it is similar to TREC-QA in the open-endedness of the answers.

数据集	问题来源	任务形式	规模
SQuAD	众包	阅读理解，段落中的跨度	100K
MCTest（Richardson et al., 2013）	众包	阅读理解，多项选择	2640
Algebra（Kushman et al., 2014）	标准化考试	计算	514
Science（Clark and Etzioni, 2016）	标准化考试	推理，多项选择	855
WikiQA（Yang et al., 2015）	查询日志	信息检索，句子选择	3047
TREC-QA（Voorhees and Tice, 2000）	查询日志 + 人工编辑	信息检索，自由作答	1479
CNN/Daily Mail（Hermann et al., 2015）	摘要 + 完形填空	阅读理解，填入单个实体	1.4M
CBT（Hill et al., 2015）	完形填空	阅读理解，填入单个词	688K

表 1：若干阅读理解和问答数据集的对照。除半合成的完形填空数据外，SQuAD 比表中其他数据集都大得多；在答案的开放程度上，它更接近 TREC-QA。

We begin with a survey of existing reading comprehension and question answering (QA) datasets, highlighting a variety of task formulation and creation strategies (see Table 1 for an overview).

我们先回顾已有的阅读理解和问答（QA）数据集，着重比较不同的任务设定和数据构建方式（总览见表 1）。

Reading comprehension. A data-driven approach to reading comprehension goes back to Hirschman et al. (1999), who curated a dataset of 600 real 3rd–6th grade reading comprehension questions. Their pattern matching baseline was subsequently improved by a rule-based system (Riloff and Thelen, 2000) and a logistic regression model (Ng et al., 2000). More recently, Richardson et al. (2013) curated MCTest, which contains 660 stories created by crowdworkers, with 4 questions per story and 4 answer choices per question. Because many of the questions require commonsense reasoning and reasoning across multiple sentences, the dataset remains quite challenging, though there has been noticeable progress (Narasimhan and Barzilay, 2015; Sachan et al., 2015; Wang et al., 2015). Both curated datasets, although real and difficult, are too small to support very expressive statistical models.

阅读理解。 用数据驱动方法做阅读理解，可以追溯到 Hirschman et al. (1999)：他们整理了 600 道真实的三到六年级阅读理解题。随后，基于规则的系统（Riloff and Thelen, 2000）和逻辑回归模型（Ng et al., 2000）改进了他们的模式匹配基线。更近一些，Richardson et al. (2013) 发布了 MCTest，包含众包工人创作的 660 个故事，每个故事 4 道题、每题 4 个选项。许多问题需要常识推理，还要跨句思考，因此这套数据至今仍然很难，不过已经看到明显进展（Narasimhan and Barzilay, 2015; Sachan et al., 2015; Wang et al., 2015）。这两套精心整理的数据虽然真实、也有难度，但规模太小，撑不起表达能力很强的统计模型。

Some datasets focus on deeper reasoning abilities. Algebra word problems require understanding a story well enough to turn it into a system of equations, which can be easily solved to produce the answer (Kushman et al., 2014; Hosseini et al., 2014). BAbI (Weston et al., 2015), a fully synthetic RC dataset, is stratified by different types of reasoning required to solve each task. Clark and Etzioni (2016) describe the task of solving 4th grade science exams, and stress the need to reason with world knowledge.

还有一些数据集更看重更深的推理能力。代数应用题要求把故事理解到足以列成方程组，再求解得到答案（Kushman et al., 2014; Hosseini et al., 2014）。BAbI（Weston et al., 2015）是完全合成的阅读理解数据，按完成各任务所需的推理类型分层。Clark and Etzioni (2016) 讨论了求解四年级科学考试的任务，并强调必须借助世界知识来推理。

Open-domain question answering. The goal of open-domain QA is to answer a question from a large collection of documents. The annual evaluations at the Text REtreival Conference (TREC) (Voorhees and Tice, 2000) led to many advances in open-domain QA, many of which were used in IBM Watson for Jeopardy! (Ferrucci et al., 2013).

开放域问答。 开放域问答的目标，是从大规模文档集合中回答问题。文本检索会议（Text REtreival Conference，TREC）的年度评测（Voorhees and Tice, 2000）推动了开放域问答的许多进展，其中不少后来用进了 IBM Watson 的 Jeopardy! 系统（Ferrucci et al., 2013）。

Recently, Yang et al. (2015) created the WikiQA dataset, which, like SQuAD, use Wikipedia passages as a source of answers, but their task is sentence selection, while ours requires selecting a specific span in the sentence.

最近，Yang et al. (2015) 发布了 WikiQA。和 SQuAD 一样，它也用维基百科段落作为答案来源，但他们的任务是句子选择，而我们要求在句子内部选出具体跨度。

Selecting the span of text that answers a question is similar to answer extraction, the final step in the open-domain QA pipeline, methods for which include bootstrapping surface patterns (Ravichandran and Hovy, 2002), using dependency trees (Shen and Klakow, 2006), and using a factor graph over multiple sentences (Sun et al., 2013). One key difference between our RC setting and answer extraction is that answer extraction typically exploits the fact that the answer occurs in multiple documents (Brill et al., 2002), which is more lenient than in our setting, where a system only has access to a single reading passage.

选出回答问题的那段文字，接近开放域问答流水线的最后一步，也就是答案抽取。已有方法包括自举表层模式（Ravichandran and Hovy, 2002）、利用依存树（Shen and Klakow, 2006），以及在多句上建立因子图（Sun et al., 2013）。我们的阅读理解设定和答案抽取有一个关键差别：答案抽取通常可以借助答案出现在多篇文档中这一事实（Brill et al., 2002），条件更宽松；而在我们的设定里，系统只能看到单独一段阅读材料。

Cloze datasets. Recently, researchers have constructed cloze datasets, in which the goal is to predict the missing word (often a named entity) in a passage. Since these datasets can be automatically generated from naturally occurring data, they can be extremely large. The Children’s Book Test (CBT) (Hill et al., 2015), for example, involves predicting a blanked-out word of a sentence given the 20 previous sentences. Hermann et al. (2015) constructed a corpus of cloze style questions by blanking out entities in abstractive summaries of CNN / Daily News articles; the goal is to fill in the entity based on the original article. While the size of this dataset is impressive, Chen et al. (2016) showed that the dataset requires less reasoning than previously thought, and concluded that performance is almost saturated.

完形填空数据集。 近来也有人构建完形填空（cloze）数据集，目标是预测段落中被挖去的词（常常是命名实体）。这类数据可以从自然文本自动生成，因此可以做得非常大。例如，儿童读物测试（Children’s Book Test，CBT）（Hill et al., 2015）要求根据前 20 句来预测某句中被挖空的词。Hermann et al. (2015) 则在 CNN / Daily News 文章的抽象摘要里挖掉实体，做成完形填空题，目标是依据原文把实体填回去。这套数据的规模令人印象深刻，但 Chen et al. (2016) 表明，它所需要的推理比原先以为的要少，并认为性能已接近饱和。

One difference between SQuAD questions and cloze-style queries is that answers to cloze queries are single words or entities, while answers in SQuAD often include non-entities and can be much longer phrases. Another difference is that SQuAD focuses on questions whose answers are entailed by the passage, whereas the answers to cloze-style queries are merely suggested by the passage.

SQuAD 的问题与完形填空式查询有两点不同。第一，完形填空的答案通常是单个词或实体，而 SQuAD 的答案常常包含非实体，也可以是更长的短语。第二，SQuAD 关注答案可由段落蕴含的问题；完形填空的答案往往只是被段落所暗示。

3 Dataset Collection

3 数据收集

Figure 2: The crowd-facing web interface used to collect the dataset encourages crowdworkers to use their own words while asking questions.

图 2：用于收集本数据集的面向众包工人的网页界面，鼓励他们用自己的话来提问。

We collect our dataset in three stages: curating passages, crowdsourcing question-answers on those passages, and obtaining additional answers.

我们分三个阶段收集数据：整理段落、在这些段落上众包问答，以及补充更多答案。

Passage curation. To retrieve high-quality articles, we used Project Nayuki’s Wikipedia’s internal PageRanks to obtain the top 10000 articles of English Wikipedia, from which we sampled 536 articles uniformly at random. From each of these articles, we extracted individual paragraphs, stripping away images, figures, tables, and discarding paragraphs shorter than 500 characters. The result was 23,215 paragraphs for the 536 articles covering a wide range of topics, from musical celebrities to abstract concepts. We partitioned the articles randomly into a training set (80%), a development set (10%), and a test set (10%).

段落整理。 为了找到高质量文章，我们借助 Project Nayuki 提供的维基百科内部 PageRank，取出英文维基百科排名前 10000 的条目，再从中均匀随机抽样 536 篇。对每篇文章，我们抽出各个段落，去掉图片、插图和表格，并丢弃短于 500 个字符的段落。最终得到 536 篇文章的 23,215 个段落，主题覆盖很广，既有音乐明星，也有抽象概念。我们把文章随机划分为训练集（80%）、开发集（10%）和测试集（10%）。

Question-answer collection. Next, we employed crowdworkers to create questions. We used the Daemo platform (Gaikwad et al., 2015), with Amazon Mechanical Turk as its backend. Crowdworkers were required to have a 97% HIT acceptance rate, a minimum of 1000 HITs, and be located in the United States or Canada. Workers were asked to spend 4 minutes on every paragraph, and paid $9 per hour for the number of hours required to complete the article. The task was reviewed favorably by crowdworkers, receiving positive comments on Turkopticon.

问答收集。 接下来，我们请来众包工人出题。平台用的是 Daemo（Gaikwad et al., 2015），后端是 Amazon Mechanical Turk。工人需要达到 97% 的 HIT 通过率、至少完成过 1000 个 HIT，并且位于美国或加拿大。我们要求每段大约花 4 分钟，并按读完整篇文章所需时间以每小时 9 美元付酬。这项任务在众包工人中评价不错，Turkopticon 上也有正面评论。

On each paragraph, crowdworkers were tasked with asking and answering up to 5 questions on the content of that paragraph. The questions had to be entered in a text field, and the answers had to be highlighted in the paragraph. To guide the workers, tasks contained a sample paragraph, and examples of good and bad questions and answers on that paragraph along with the reasons they were categorized as such. Additionally, crowdworkers were encouraged to ask questions in their own words, without copying word phrases from the paragraph. On the interface, this was reinforced by a reminder prompt at the beginning of every paragraph, and by disabling copy-paste functionality on the paragraph text.

对每个段落，众包工人最多要就该段内容提出并回答 5 道题。问题写在文本框里，答案则必须在段落中高亮标出。为了给工人提供指引，任务里配有样例段落，以及该段上好问题和坏问题、好答案和坏答案的例子，并说明这样分类的原因。我们还鼓励工人用自己的话提问，不要照抄段落中的词组。界面上，每个段落开头都会弹出提醒，并且禁用了段落文本的复制粘贴。

Additional answers collection. To get an indication of human performance on SQuAD and to make our evaluation more robust, we obtained at least 2 additional answers for each question in the development and test sets. In the secondary answer generation task, each crowdworker was shown only the questions along with the paragraphs of an article, and asked to select the shortest span in the paragraph that answered the question. If a question was not answerable by a span in the paragraph, workers were asked to submit the question without marking an answer. Workers were recommended a speed of 5 questions for 2 minutes, and paid at the same rate of $9 per hour for the number of hours required for the entire article. Over the development and test sets, 2.6% of questions were marked unanswerable by at least one of the additional crowdworkers.

补充答案收集。 为了估计人类在 SQuAD 上的表现，并使评测更稳健，我们为开发集和测试集中的每道题另外收集了至少 2 个答案。在这项二次作答任务里，工人只能看到某篇文章的问题和段落，并被要求在段落中标出能回答该问题的最短跨度。如果问题无法用段落中的跨度作答，工人应提交该题但不标记答案。建议速度是 2 分钟完成 5 道题，报酬同样按整篇文章所需时间以每小时 9 美元计算。在开发集和测试集上，有 2.6% 的问题被至少一名补充标注的工人标为无法回答。

4 Dataset Analysis

4 数据分析

Answer type	Percentage	Example
Date	8.9%	19 October 1512
Other Numeric	10.9%	12
Person	12.9%	Thomas Coke
Location	4.4%	Germany
Other Entity	15.3%	ABC Sports
Common Noun Phrase	31.8%	property damage
Adjective Phrase	3.9%	second-largest
Verb Phrase	5.5%	returned to Earth
Clause	3.7%	to avoid trivialization
Other	2.7%	quietly

Table 2: We automatically partition our answers into the following categories. Our dataset consists of large number of answers beyond proper noun entities.

答案类型	比例	例子
日期	8.9%	19 October 1512
其他数值	10.9%	12
人物	12.9%	Thomas Coke
地点	4.4%	Germany
其他实体	15.3%	ABC Sports
普通名词短语	31.8%	property damage
形容词短语	3.9%	second-largest
动词短语	5.5%	returned to Earth
从句	3.7%	to avoid trivialization
其他	2.7%	quietly

表 2：我们把答案自动划分到下列类别。本数据集中有大量答案并不属于专有名词实体。

To understand the properties of SQuAD, we analyze the questions and answers in the development set. Specifically, we explore the (i) diversity of answer types, (ii) the difficulty of questions in terms of type of reasoning required to answer them, and (iii) the degree of syntactic divergence between the question and answer sentences.

为了把握 SQuAD 的性质，我们分析了开发集中的问题和答案，具体考察：（i）答案类型的多样性；（ii）按所需推理类型衡量的问题难度；（iii）问句与答案句之间的句法分歧程度。

Diversity in answers. We automatically categorize the answers as follows: We first separate the numerical and non-numerical answers. The non-numerical answers are categorized using constituency parses and POS tags generated by Stanford CoreNLP. The proper noun phrases are further split into person, location and other entities using NER tags. In Table 2, we can see dates and other numbers make up 19.8% of the data; 32.6% of the answers are proper nouns of three different types; 31.8% are common noun phrases answers; and the remaining 15.8% are made up of adjective phrases, verb phrases, clauses and other types.

答案的多样性。 我们按如下方式自动给答案分类：先把数值答案和非数值答案分开；非数值答案再用斯坦福 CoreNLP 生成的成分分析和词性标注来归类；专有名词短语则借助命名实体识别标签，进一步分成人物、地点和其他实体。从表 2 可以看到，日期和其他数字合计占 19.8%；三类专有名词共占 32.6%；普通名词短语答案占 31.8%；其余 15.8% 由形容词短语、动词短语、从句和其他类型构成。

Reasoning required to answer questions. To get a better understanding of the reasoning required to answer the questions, we sampled 4 questions from each of the 48 articles in the development set, and then manually labeled the examples with the categories shown in Table 3. The results show that all examples have some sort of lexical or syntactic divergence between the question and the answer in the passage. Note that some examples fall into more than one category.

回答问题所需的推理。 为了更清楚地了解答题需要怎样的推理，我们从开发集的 48 篇文章中每篇抽样 4 道题，再按表 3 中的类别人工标注。结果显示，所有例子在问题与段落中的答案之间，都存在某种词汇或句法上的分歧。需要说明的是，有些例子会同时落入多个类别。

Reasoning	Description	Example	Percentage
Lexical variation (synonymy)	Major correspondences between the question and the answer sentence are synonyms.	Q: What is the Rankine cycle sometimes called? Sentence: The Rankine cycle is sometimes referred to as a practical Carnot cycle.	33.3%
Lexical variation (world knowledge)	Major correspondences between the question and the answer sentence require world knowledge to resolve.	Q: Which governing bodies have veto power? Sen.: The European Parliament and the Council of the European Union have powers of amendment and veto during the legislative process.	9.1%
Syntactic variation	After the question is paraphrased into declarative form, its syntactic dependency structure does not match that of the answer sentence even after local modifications.	Q: What Shakespeare scholar is currently on the faculty? Sen.: Current faculty include the anthropologist Marshall Sahlins, ..., Shakespeare scholar David Bevington.	64.1%
Multiple sentence reasoning	There is anaphora, or higher-level fusion of multiple sentences is required.	Q: What collection does the V&A Theatre & Performance galleries hold? Sen.: The V&A Theatre & Performance galleries opened in March 2009. ... They hold the UK’s biggest national collection of material about live performance.	13.6%
Ambiguous	We don’t agree with the crowdworkers’ answer, or the question does not have a unique answer.	Q: What is the main goal of criminal punishment? Sen.: Achieving crime control via incapacitation and deterrence is a major goal of criminal punishment.	6.1%

Table 3: We manually labeled 192 examples into one or more of the above categories. Words relevant to the corresponding reasoning type are bolded, and the crowdsourced answer is underlined.

推理类型	说明	例子	比例
词汇变化（同义）	问题与答案句之间的主要对应关系是同义词。	Q: What is the Rankine cycle sometimes called? Sentence: The Rankine cycle is sometimes referred to as a practical Carnot cycle.	33.3%
词汇变化（世界知识）	问题与答案句之间的主要对应关系需要世界知识才能对齐。	Q: Which governing bodies have veto power? Sen.: The European Parliament and the Council of the European Union have powers of amendment and veto during the legislative process.	9.1%
句法变化	把问题改写成陈述句后，即使做局部修改，其句法依存结构仍与答案句不一致。	Q: What Shakespeare scholar is currently on the faculty? Sen.: Current faculty include the anthropologist Marshall Sahlins, ..., Shakespeare scholar David Bevington.	64.1%
多句推理	存在指代，或需要在更高层面融合多个句子。	Q: What collection does the V&A Theatre & Performance galleries hold? Sen.: The V&A Theatre & Performance galleries opened in March 2009. ... They hold the UK’s biggest national collection of material about live performance.	13.6%
歧义	我们不认同众包工人给出的答案，或问题本身没有唯一答案。	Q: What is the main goal of criminal punishment? Sen.: Achieving crime control via incapacitation and deterrence is a major goal of criminal punishment.	6.1%

表 3：我们把 192 个例子人工标进上述一个或多个类别。与相应推理类型有关的词加粗，众包得到的答案加下划线。

Figure 3: An example walking through the computation of the syntactic divergence between the question Q and answer sentence S.

图 3：一个逐步演示如何计算问题 Q 与答案句 S 之间句法分歧的例子。

Figure 4: We use the edit distance between the unlexicalized dependency paths in the question and the sentence containing the answer to measure syntactic divergence.

图 4：我们用问题与含答案句子中、未词汇化依存路径之间的编辑距离，来衡量句法分歧。

Stratification by syntactic divergence. We also develop an automatic method to quantify the syntactic divergence between a question and the sentence containing the answer. This provides another way to measure the difficulty of a question and to stratify the dataset, which we return to in Section 6.3.

按句法分歧分层。 我们还设计了一种自动方法，用于量化问题与含答案句子之间的句法分歧。这提供了另一种衡量问题难度、并对数据集分层的途径，我们会在第 6.3 节再回到这一点。

We illustrate how we measure the divergence with the example in Figure 3. We first detect anchors (word-lemma pairs common to both the question and answer sentences); in the example, the anchor is “first”. The two unlexicalized paths, one from the anchor “first” in the question to the wh-word “what”, and the other from the anchor in the answer sentence and to the answer span “Bainbridge’s”, are then extracted from the dependency parse trees.

我们用图 3 中的例子说明如何度量这种分歧。首先找出锚点（问题和答案句中都出现的词–词元对）；在这个例子里，锚点是“first”。接着从依存分析树中抽出两条未词汇化路径：一条从问题中的锚点“first”连到疑问词“what”，另一条从答案句中的锚点连到答案跨度“Bainbridge’s”。

We measure the edit distance between these two paths, which we define as the minimum number of deletions or insertions to transform one path into the other. The syntactic divergence is then defined as the minimum edit distance over all possible anchors. The histogram in Figure 4 shows that there is a wide range of syntactic divergence in our dataset. We also show a concrete example where the edit distance is 0 and another where it is 6. Note that our syntactic divergence ignores lexical variation. Also, small divergence does not mean that a question is easy since there could be other candidates with similarly small divergence.

我们计算这两条路径的编辑距离，定义为把一条路径变成另一条所需的最少删除或插入次数。句法分歧则定义为在所有可能锚点上的最小编辑距离。图 4 的直方图表明，本数据集中的句法分歧分布很广。我们还给出了一个编辑距离为 0 的具体例子，以及一个编辑距离为 6 的例子。需要注意，我们的句法分歧并不计入词汇变化。另外，分歧小并不意味着问题就容易，因为可能还有其他候选也具有同样小的分歧。

5 Methods

5 方法

We developed a logistic regression model and compare its accuracy with that of three baseline methods.

我们设计了一个逻辑回归模型，并把它的准确率与三种基线方法进行比较。

Candidate answer generation. For all four methods, rather than considering all $O(L^2)$ spans as candidate answers, where $L$ is the number of words in the sentence, we only use spans which are constituents in the constituency parse generated by Stanford CoreNLP. Ignoring punctuation and articles, we find that 77.3% of the correct answers in the development set are constituents. This places an effective ceiling on the accuracy of our methods. During training, when the correct answer of an example is not a constituent, we use the shortest constituent containing the correct answer as the target.

候选答案生成。 对全部四种方法，我们都不把句子中所有 $O(L^2)$ 个跨度当作候选答案（$L$ 为句中词数），而只使用斯坦福 CoreNLP 成分分析树中的成分作为跨度。忽略标点和冠词后，开发集里 77.3% 的正确答案本身就是成分。这给我们方法的准确率加上了一个实际上限。训练时，如果某道题的正确答案不是成分，我们就用包含该正确答案的最短成分作为目标。

Feature Groups	Description	Examples
Matching Word Frequencies	Sum of the TF-IDF of the words that occur in both the question and the sentence containing the candidate answer.

Separate features are used for the words to the left, to the right, inside the span, and in the whole sentence. | Span: $[0 ≤$ sum $< 0.01]$  Left: $[7.9 ≤$ sum $< 10.7]$ |

| Matching Bigram Frequencies | Same as above, but using bigrams. We use the generalization of the TF-IDF described in Shirakawa et al. (2015). | Span: $[0 ≤$ sum $< 2.4]$ Left: $[0 ≤$ sum $< 2.7]$ |
| Root Match | Whether the dependency parse tree roots of the question and sentence match,
whether the sentence contains the root of the dependency parse tree of the question,
and whether the question contains the root of the dependency parse tree of the sentence. | Root Match = False |
| Lengths | Number of words to the left, to the right, inside the span, and in the whole sentence. | Span: $[1 <=$ num $< 2]$ Left: $[15 ≤$ num $< 19]$ |
| Span Word Frequencies | Sum of the TF-IDF of the words in the span, regardless of whether they appear in the question. | Span: $[5.2 ≤$ sum $< 6.9]$ |
| Constituent Label | Constituency parse tree label of the span, optionally combined with the wh-word in the question. | Span: NP Span: NP, wh-word: “what” |
| Span POS Tags | Sequence of the part-of-speech tags in the span, optionally combined with the wh-word in the question. | Span: [NN] Span: [NN], wh-word: “what” |
| Lexicalized | Lemmas of question words combined with the lemmas of words within distance 2 to the span in the sentence based on the dependency parse trees.
Separately, question word lemmas combined with answer word lemmas. | Q: “cause”, S: “under” $case$ Q: “fall”, A: “gravity” |
| Dependency Tree Paths | For each word that occurs in both the question and sentence, the path in the dependency parse tree from that word in the sentence to the span, optionally combined with the path from the wh-word to the word in the question. POS tags are included in the paths. | VBZ $nmod$ NN what $nsubj$ VBZ $advcl$ 0.2cm + VBZ$nmod$NN |

Table 4: Features used in the logistic regression model with examples for the question “What causes precipitation to fall?”, sentence “In meteorology, precipitation is any product of the condensation of atmospheric water vapor that falls under gravity.” and answer “gravity”. Q denotes question, A denotes candidate answer, and S denotes sentence containing the candidate answer.

特征组	说明	例子
匹配词频率	同时出现在问题和含候选答案的句子中的那些词的 TF-IDF 之和。对跨度左侧、右侧、跨度内部以及整句，分别使用单独的特征。	Span: $[0 ≤$ sum $< 0.01]$ Left: $[7.9 ≤$ sum $< 10.7]$
匹配二元语法频率	同上，但改用二元语法。我们采用 Shirakawa et al. (2015) 描述的 TF-IDF 推广形式。	Span: $[0 ≤$ sum $< 2.4]$ Left: $[0 ≤$ sum $< 2.7]$
根匹配	问题与句子的依存分析树根是否相同；句子是否包含问题依存树的根；问题是否包含句子依存树的根。	Root Match = False
长度	跨度左侧、右侧、跨度内部以及整句的词数。	Span: $[1 <=$ num $< 2]$ Left: $[15 ≤$ num $< 19]$
跨度词频率	跨度中词的 TF-IDF 之和，无论这些词是否出现在问题中。	Span: $[5.2 ≤$ sum $< 6.9]$
成分标签	跨度的成分分析树标签，也可选择与问题中的疑问词组合。	Span: NP Span: NP, wh-word: “what”
跨度词性序列	跨度中的词性标注序列，也可选择与问题中的疑问词组合。	Span: [NN] Span: [NN], wh-word: “what”
词汇化特征	根据依存分析树，把问题词的词元与句子中距离跨度不超过 2 的词的词元组合起来；另外，再把问题词词元与答案词词元组合。	Q: “cause”, S: “under” $case$ Q: “fall”, A: “gravity”
依存树路径	对同时出现在问题和句子中的每个词，取出句子依存树中从该词到跨度的路径，也可选择与问题中从疑问词到该词的路径组合。路径中包含词性标签。	VBZ $nmod$ NN what $nsubj$ VBZ $advcl$ VBZ$nmod$NN

表 4：逻辑回归模型所用特征，以及针对问题“What causes precipitation to fall?”、句子“In meteorology, precipitation is any product of the condensation of atmospheric water vapor that falls under gravity.”和答案“gravity”的例子。Q 表示问题，A 表示候选答案，S 表示包含该候选答案的句子。

5.1 Sliding Window Baseline

5.1 滑动窗口基线

For each candidate answer, we compute the unigram/bigram overlap between the sentence containing it (excluding the candidate itself) and the question. We keep all the candidates that have the maximal overlap. Among these, we select the best one using the sliding-window approach proposed in Richardson et al. (2013).

对每个候选答案，我们计算包含它的句子（不含候选本身）与问题之间的一元／二元语法重叠，并保留重叠最大的全部候选。再在这些候选中，用 Richardson et al. (2013) 提出的滑动窗口方法选出最好的一个。

In addition to the basic sliding window approach, we also implemented the distance-based extension (Richardson et al., 2013). Whereas Richardson et al. (2013) used the entire passage as the context of an answer, we used only the sentence containing the candidate answer for efficiency.

除了基本的滑动窗口方法，我们还实现了基于距离的扩展（Richardson et al., 2013）。Richardson et al. (2013) 把整段阅读材料当作答案的上下文；为了提高效率，我们只使用包含候选答案的那一句。

5.2 Logistic Regression

5.2 逻辑回归

In our logistic regression model, we extract several types of features for each candidate answer. We discretize each continuous feature into 10 equally-sized buckets, building a total of 180 million features, most of which are lexicalized features or dependency tree path features. The descriptions and examples of the features are summarized in Table 4.

在逻辑回归模型中，我们为每个候选答案提取多类特征。每个连续特征被离散成 10 个大小相等的桶，总共构造了 1.8 亿个特征，其中大多数是词汇化特征或依存树路径特征。各特征的说明和例子汇总在表 4。

The matching word and bigram frequencies as well as the root match features help the model pick the correct sentences. Length features bias the model towards picking common lengths and positions for answer spans, while span word frequencies bias the model against uninformative words. Constituent label and span POS tag features guide the model towards the correct answer types. In addition to these basic features, we resolve lexical variation using lexicalized features, and syntactic variation using dependency tree path features.

匹配词频率、匹配二元语法频率以及根匹配特征，帮助模型选出正确的句子。长度特征会让模型偏向常见的答案跨度长度和位置，跨度词频率则抑制无信息的词。成分标签和跨度词性特征引导模型走向正确的答案类型。在这些基本特征之外，我们用词汇化特征应对词汇变化，用依存树路径特征应对句法变化。

The multiclass log-likelihood loss is optimized using AdaGrad with an initial learning rate of 0.1. Each update is performed on the batch of all questions in a paragraph for efficiency, since they share the same candidates. $L_2$ regularization is used, with a coefficient of 0.1 divided by the number of batches. The model is trained with three passes over the training data.

多类对数似然损失用 AdaGrad 优化，初始学习率为 0.1。为了提高效率，每次更新都在同一段落的全部问题上进行，因为它们共享候选。模型使用 $L_2$ 正则，系数为 0.1 除以批次数。训练时对训练数据扫描三遍。

6 Experiments

6 实验

6.1 Model Evaluation

6.1 模型评测

We use two different metrics to evaluate model accuracy. Both metrics ignore punctuations and articles (a, an, the).

我们用两种指标评估模型准确率。两种指标都忽略标点和冠词（a、an、the）。

Exact match. This metric measures the percentage of predictions that match any one of the ground truth answers exactly.

精确匹配。 该指标衡量预测结果与任意一个标准答案完全一致的比例。

(Macro-averaged) F1 score. This metric measures the average overlap between the prediction and ground truth answer. We treat the prediction and ground truth as bags of tokens, and compute their F1. We take the maximum F1 over all of the ground truth answers for a given question, and then average over all of the questions.

（宏平均）F1 分数。 该指标衡量预测与标准答案之间的平均重叠。我们把预测和标准答案看作词袋，计算它们的 F1；对一道题，取它与所有标准答案之间的最大 F1，再对所有问题求平均。

6.2 Human Performance

6.2 人类表现

We assess human performance on SQuAD’s development and test sets. Recall that each of the questions in these sets has at least three answers. To evaluate human performance, we treat the second answer to each question as the human prediction, and keep the other answers as ground truth answers. The resulting human performance score on the test set is 77.0% for the exact match metric, and 86.8% for F1. Mismatch occurs mostly due to inclusion/exclusion of non-essential phrases (e.g., monsoon trough versus movement of the monsoon trough) rather than fundamental disagreements about the answer.

我们在 SQuAD 的开发集和测试集上评估人类表现。这两个集合中的每道题至少有三个答案。评估时，我们把每道题的第二个答案当作人类预测，其余答案留作标准答案。由此得到的测试集人类成绩是：精确匹配 77.0%，F1 为 86.8%。不一致主要来自是否纳入非必要短语（例如 monsoon trough 对 movement of the monsoon trough），而不是对答案本身有根本分歧。

6.3 Model Performance

6.3 模型表现

Table 5 shows the performance of our models alongside human performance on the v1.0 of development and test sets. The logistic regression model significantly outperforms the baselines, but underperforms humans. We note that the model is able to select the sentence containing the answer correctly with 79.3% accuracy; hence, the bulk of the difficulty lies in finding the exact span within the sentence.

表 5 给出了各模型与人类在 SQuAD v1.0 开发集和测试集上的表现。逻辑回归明显超过基线，但仍低于人类。我们注意到，模型以 79.3% 的准确率就能选对含答案的句子；因此，主要难点在于在句子内部找出精确跨度。

	Exact Match			F1
	Dev	Test		Dev	Test
Random Guess	1.1%	1.3%		4.1%	4.3%
Sliding Window	13.2%	12.5%		20.2%	19.7%
Sliding Win. + Dist.	13.3%	13.0%		20.2%	20.0%
Logistic Regression	40.0%	40.4%		51.0%	51.0%
Human	80.3%	77.0%		90.5%	86.8%

Table 5: Performance of various methods and humans. Logistic regression outperforms the baselines, while there is still a significant gap between humans.

	精确匹配			F1
	开发集	测试集		开发集	测试集
随机猜测	1.1%	1.3%		4.1%	4.3%
滑动窗口	13.2%	12.5%		20.2%	19.7%
滑动窗口 + 距离	13.3%	13.0%		20.2%	20.0%
逻辑回归	40.0%	40.4%		51.0%	51.0%
人类	80.3%	77.0%		90.5%	86.8%

表 5：各种方法和人类的表现。逻辑回归超过了基线，但与人类仍有明显差距。

	F$_1$
	Train	Dev
Logistic Regression	91.7%	51.0%
– Lex., – Dep. Paths	33.9%	35.8%
– Lexicalized	53.5%	45.4%
– Dep. Paths	91.4%	46.4%
– Match. Word Freq.	91.7%	48.1%
– Span POS Tags	91.7%	49.7%
– Match. Bigram Freq.	91.7%	50.3%
– Constituent Label	91.7%	50.4%
– Lengths	91.8%	50.5%
– Span Word Freq.	91.7%	50.5%
– Root Match	91.7%	50.6%

Table 6: Performance with feature ablations. We find that lexicalized and dependency tree path features are most important.

	F$_1$
	训练集	开发集
逻辑回归	91.7%	51.0%
– 词汇化，– 依存路径	33.9%	35.8%
– 词汇化	53.5%	45.4%
– 依存路径	91.4%	46.4%
– 匹配词频率	91.7%	48.1%
– 跨度词性	91.7%	49.7%
– 匹配二元语法频率	91.7%	50.3%
– 成分标签	91.7%	50.4%
– 长度	91.8%	50.5%
– 跨度词频率	91.7%	50.5%
– 根匹配	91.7%	50.6%

表 6：特征消融后的表现。我们看到，词汇化特征和依存树路径特征最为重要。

Feature ablations. In order to understand the features that are responsible for the performance of the logistic regression model, we perform a feature ablation where we remove one group of features from our model at a time. The results, shown in Table 6, indicate that lexicalized and dependency tree path features are most important. Comparing our analysis to the one in Chen et al. (2016), we note that the dependency tree path features play a much bigger role in our dataset. Additionally, we note that with lexicalized features, the model significantly overfits the training set; however, we found that increasing $L_2$ regularization hurts performance on the development set.

特征消融。 为了弄清逻辑回归模型的表现主要靠哪些特征，我们每次从模型中去掉一组特征，做特征消融。表 6 的结果表明，词汇化特征和依存树路径特征最为重要。与 Chen et al. (2016) 的分析相比，依存树路径特征在我们的数据里作用要大得多。另外，加上词汇化特征后，模型在训练集上明显过拟合；不过我们发现，加大 $L_2$ 正则会损害开发集上的成绩。

	Logistic Regression	Human
	Dev F1	Dev F1
Date	72.1%	93.9%
Other Numeric	62.5%	92.9%
Person	56.2%	95.4%
Location	55.4%	94.1%
Other Entity	52.2%	92.6%
Common Noun Phrase	46.5%	88.3%
Adjective Phrase	37.9%	86.8%
Verb Phrase	31.2%	82.4%
Clause	34.3%	84.5%
Other	34.8%	86.1%

Table 7: Performance stratified by answer types. Logistic regression performs better on certain types of answers, namely numbers and entities. On the other hand, human performance is more uniform.

	逻辑回归	人类
	开发集 F1	开发集 F1
日期	72.1%	93.9%
其他数值	62.5%	92.9%
人物	56.2%	95.4%
地点	55.4%	94.1%
其他实体	52.2%	92.6%
普通名词短语	46.5%	88.3%
形容词短语	37.9%	86.8%
动词短语	31.2%	82.4%
从句	34.3%	84.5%
其他	34.8%	86.1%

表 7：按答案类型分层后的表现。逻辑回归在某些答案类型上更好，尤其是数字和实体；相比之下，人类表现更加均匀。

Performance stratified by answer type. To gain more insight into the performance of our logistic regression model, we report its performance across the answer types explored in Table 2. The results (shown in Table 7) show that the model performs best on dates and other numbers, categories for which there are usually only a few plausible candidates, and most answers are single tokens. The model is challenged more on other named entities (i.e., location, person and other entities) because there are many more plausible candidates. However, named entities are still relatively easy to identify by their POS tag features. The model performs worst on other answer types, which together form 47.6% of the dataset. Humans have exceptional performance on dates, numbers and all named entities. Their performance on other answer types degrades only slightly.

按答案类型分层后的表现。 为了更细致地观察逻辑回归模型，我们按表 2 中的答案类型分别报告成绩。表 7 显示，模型在日期和其他数字上最好：这类答案通常只有少数几个说得通的候选，而且多数是单标记。其他命名实体（即地点、人物和其他实体）对模型更有挑战，因为说得通的候选更多。不过，凭借词性特征，命名实体相对仍较容易识别。模型在其余答案类型上最差，而这些类型合计占数据集的 47.6%。人类在日期、数字和各类命名实体上都有非常高的成绩，在其他答案类型上只是略有下降。

Figure 5: Performance stratified by syntactic divergence of questions and sentences. The performance of logistic regression degrades with increasing divergence. In contrast, human performance is stable across the full range of divergence.

图 5：按问题与句子的句法分歧分层后的表现。逻辑回归的成绩随分歧增大而下降；相比之下，人类表现在整个分歧范围内都保持稳定。

Performance stratified by syntactic divergence. As discussed in Section 4, another challenging aspect of the dataset is the syntactic divergence between the question and answer sentence. Figure 5 shows that the more divergence there is, the lower the performance of the logistic regression model. Interestingly, humans do not seem to be sensitive to syntactic divergence, suggesting that deep understanding is not distracted by superficial differences. Measuring the degree of degradation could therefore be useful in determining the extent to which a model is generalizing in the right way.

按句法分歧分层后的表现。 如第 4 节所述，这套数据的另一个难点，是问题与答案句之间的句法分歧。图 5 表明，分歧越大，逻辑回归的表现越差。有意思的是，人类似乎并不受句法分歧影响，这说明真正的深入理解不会被表面上的差异带走。因此，衡量这种下降的幅度，也许有助于判断一个模型是否在以正确的方式泛化。

7 Conclusion

7 结论

Towards the end goal of natural language understanding, we introduce the Stanford Question Answering Dataset, a large reading comprehension dataset on Wikipedia articles with crowdsourced question-answer pairs. SQuAD features a diverse range of question and answer types. The performance of our logistic regression model, with 51.0% F1, against the human F1 of 86.8% suggests ample opportunity for improvement. We have made our dataset freely available to encourage exploration of more expressive models. Since the release of our dataset, we have already seen considerable interest in building models on this dataset, and the gap between our logistic regression model and human performance has more than halved (Wang and Jiang, 2016). We expect that the remaining gap will be harder to close, but that such efforts will result in significant advances in reading comprehension.

面向自然语言理解这一长远目标，我们发布了斯坦福问答数据集：这是一套基于维基百科文章、由众包问答对构成的大规模阅读理解数据。SQuAD 的问题和答案类型都相当多样。我们的逻辑回归模型 F1 为 51.0%，人类 F1 为 86.8%，两者之间的差距说明仍有很大的改进空间。我们已把数据集公开，以便大家探索表达能力更强的模型。数据集发布后，已经有不少工作在这套数据上建模，我们的逻辑回归与人类之间的差距已经缩小了一半以上（Wang and Jiang, 2016）。我们预计，剩下的差距会更难填平，但这类努力将推动阅读理解取得实质性进展。

Reproducibility

可复现性

All code, data, and experiments for this paper are available on the CodaLab platform: https://worksheets.codalab.org/worksheets/0xd53d03a48ef64b329c16b9baf0f99b0c/.

本文的全部代码、数据与实验均可在 CodaLab 平台获取：https://worksheets.codalab.org/worksheets/0xd53d03a48ef64b329c16b9baf0f99b0c/。

Acknowledgments

致谢

We would like to thank Durim Morina and Professor Michael Bernstein for their help in crowdsourcing the collection of our dataset, both in terms of funding and technical support of the Daemo platform.

我们感谢 Durim Morina 与 Michael Bernstein 教授在众包收集本数据集过程中给予的帮助，包括资助以及对 Daemo 平台的技术支持。

References

参考文献

[1] J. Berant, V. Srikumar, P. Chen, A. V. Linden, B. Harding, B. Huang, P. Clark, and C. D. Manning. 2014. Modeling biological processes for reading comprehension. In Empirical Methods in Natural Language Processing (EMNLP).

[2] E. Brill, S. Dumais, and M. Banko. 2002. An analysis of the AskMSR question-answering system. In Association for Computational Linguistics (ACL), pages 257–264.

[3] D. Chen, J. Bolton, and C. D. Manning. 2016. A thorough examination of the CNN / Daily Mail reading comprehension task. In Association for Computational Linguistics (ACL).

[4] P. Clark and O. Etzioni. 2016. My computer is an honor student but how intelligent is it? standardized tests as a measure of AI. AI Magazine, 37(1):5–12.

[5] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), pages 248–255.

[6] D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, N. Schlaefer, and C. Welty. 2013. Building Watson: An overview of the DeepQA project. AI Magazine, 31(3):59–79.

[7] S. N. Gaikwad, D. Morina, R. Nistala, M. Agarwal, A. Cossette, R. Bhanu, S. Savage, V. Narwal, K. Rajpal, J. Regino, et al. 2015. Daemo: A self-governed crowdsourcing marketplace. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, pages 101–102.

[8] K. M. Hermann, T. Kočiský, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS).

[9] F. Hill, A. Bordes, S. Chopra, and J. Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. In International Conference on Learning Representations (ICLR).

[10] L. Hirschman, M. Light, E. Breck, and J. D. Burger. 1999. Deep read: A reading comprehension system. In Association for Computational Linguistics (ACL), pages 325–332.

[11] M. J. Hosseini, H. Hajishirzi, O. Etzioni, and N. Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Empirical Methods in Natural Language Processing (EMNLP), pages 523–533.

[12] N. Kushman, Y. Artzi, L. Zettlemoyer, and R. Barzilay. 2014. Learning to automatically solve algebra word problems. In Association for Computational Linguistics (ACL).

[13] M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19:313–330.

[14] K. Narasimhan and R. Barzilay. 2015. Machine comprehension with discourse relations. In Association for Computational Linguistics (ACL).

[15] H. T. Ng, L. H. Teo, and J. L. P. Kwan. 2000. A machine learning approach to answering questions for reading comprehension tests. In Joint SIGDAT conference on empirical methods in natural language processing and very large corpora - Volume 13, pages 124–132.

[16] D. Ravichandran and E. Hovy. 2002. Learning surface text patterns for a question answering system. In Association for Computational Linguistics (ACL), pages 41–47.

[17] M. Richardson, C. J. Burges, and E. Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 193–203.

[18] E. Riloff and M. Thelen. 2000. A rule-based question answering system for reading comprehension tests. In ANLP/NAACL Workshop on reading comprehension tests as evaluation for computer-based language understanding sytems - Volume 6, pages 13–19.

[19] M. Sachan, A. Dubey, E. P. Xing, and M. Richardson. 2015. Learning answer-entailing structures for machine comprehension. In Association for Computational Linguistics (ACL).

[20] D. Shen and D. Klakow. 2006. Exploring correlation of dependency relation paths for answer extraction. In International Conference on Computational Linguistics and Association for Computational Linguistics (COLING/ACL), pages 889–896.

[21] M. Shirakawa, T. Hara, and S. Nishio. 2015. N-gram idf: A global term weighting scheme based on information distance. In World Wide Web (WWW), pages 960–970.

[22] H. Sun, N. Duan, Y. Duan, and M. Zhou. 2013. Answer extraction from passage graph for question answering. In International Joint Conference on Artificial Intelligence (IJCAI).

[23] E. M. Voorhees and D. M. Tice. 2000. Building a question answering test collection. In ACM Special Interest Group on Information Retreival (SIGIR), pages 200–207.

[24] Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. CoRR, abs/1608.07905.

[25] H. Wang, M. Bansal, K. Gimpel, and D. McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In Association for Computational Linguistics (ACL).

[26] J. Weston, A. Bordes, S. Chopra, and T. Mikolov. 2015. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv.

[27] Y. Yang, W. Yih, and C. Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Empirical Methods in Natural Language Processing (EMNLP), pages 2013–2018.