HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表
1 引言2 数据收集3 处理与基准设置4 数据集分析5 实验5.1 模型架构与训练5.2 结果5.3 确定人的表现6 相关工作7 结论致谢参考文献A 数据收集细节A.1 数据预处理A.2 更多数据收集细节A.3 众包工人界面B 进一步的数据分析C 全文维基设置的细节C.1 倒排索引过滤策略C.2 比较 train-medium 划分与困难样本
返回论文列表
目录

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

收录于 EMNLP 2018 会议论文 / long paper被引 5,401 次
打开论文来源

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Zhilin Yang*♠ Peng Qi*♥ Saizheng Zhang*♣
Yoshua Bengio♣♦ William W. Cohen†
Ruslan Salakhutdinov♠ Christopher D. Manning♥
♠Carnegie Mellon University ♥Stanford University ♣Mila, Université de Montréal
♦CIFAR Senior Fellow † Google AI
{zhiliny, rsalakhu}@cs.cmu.edu, {pengqi, manning}@cs.stanford.edu
saizheng.zhang@umontreal.ca, yoshua.bengio@gmail.com, wcohen@google.com

∗These authors contributed equally. The order of authorship is decided through dice rolling.

†Work done when WWC was at CMU.

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380
Brussels, Belgium, October 31 - November 4, 2018. ©2018 Association for Computational Linguistics

《2018 年自然语言处理经验方法会议论文集》,第 2369–2380 页
比利时布鲁塞尔,2018 年 10 月 31 日–11 月 4 日。©2018 计算语言学协会

Abstract

摘要

Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HOTPOTQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. We show that HOTPOTQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.

已有的问答(QA)数据集,不足以训练问答系统去做复杂推理,并给答案提供解释。我们提出 HOTPOTQA,这是一套新的数据集,包含 11.3 万条基于维基百科的问答对,并具有四个要点:(1)问题需要找到多篇支持文档并在其上推理才能作答;(2)问题多样,不受既有知识库或知识模式约束;(3)我们提供推理所需的句子级支持事实,使问答系统能够在强监督下推理,并解释预测;(4)我们还提供一种新的事实比较题,用来检验问答系统抽取相关事实并完成必要比较的能力。我们表明,HOTPOTQA 对当时最新的问答系统仍有难度,而支持事实能够提升模型表现,并帮助做出可解释的预测。

1 Introduction

1 引言

The ability to perform reasoning and inference over natural language is an important aspect of intelligence. The task of question answering (QA) provides a quantifiable and objective way to test the reasoning ability of intelligent systems. To this end, a few large-scale QA datasets have been proposed, which sparked significant progress in this direction. However, existing datasets have limitations that hinder further advancements of machine reasoning over natural language, especially in testing QA systems’ ability to perform multi-hop reasoning, where the system has to reason with information taken from more than one document to arrive at the answer.

在自然语言上做推理,是智能的一个重要方面。问答(QA)任务为检验智能系统的推理能力提供了一种可量化、客观的方式。为此,已有若干大规模问答数据集被提出,并推动了这一方向的显著进展。然而,已有数据集仍有局限,妨碍了机器在自然语言推理上的进一步推进,尤其是在检验问答系统做多跳推理(multi-hop reasoning)的能力时:系统必须利用来自不止一篇文档的信息,才能得出答案。

HotpotQA multi-hop question example with two Wikipedia paragraphs, answer, and supporting facts

Figure 1: An example of the multi-hop questions in HOTPOTQA. We also highlight the supporting facts in blue italics, which are also part of the dataset.

图 1:HOTPOTQA 中多跳问题的一个例子。我们也用蓝色斜体标出支持事实,这些支持事实同样是数据集的一部分。

First, some datasets mainly focus on testing the ability of reasoning within a single paragraph or document, or single-hop reasoning. For example, in SQuAD (Rajpurkar et al., 2016) questions are designed to be answered given a single paragraph as the context, and most of the questions can in fact be answered by matching the question with a single sentence in that paragraph. As a result, it has fallen short at testing systems’ ability to reason over a larger context. TriviaQA (Joshi et al., 2017) and SearchQA (Dunn et al., 2017) create a more challenging setting by using information retrieval to collect multiple documents to form the context given existing question-answer pairs. Nevertheless, most of the questions can be answered by matching the question with a few nearby sentences in one single paragraph, which is limited as it does not require more complex reasoning (e.g., over multiple paragraphs).

第一,有些数据集主要考察在单一段落或文档内推理的能力,也就是单跳推理。例如,在 SQuAD(Rajpurkar et al., 2016)中,问题被设计成在给定单一段落作为上下文时即可作答,而且多数问题实际上只需把问题与该段中的一句话匹配起来。因此,它不足以检验系统在更大上下文中推理的能力。TriviaQA(Joshi et al., 2017)和 SearchQA(Dunn et al., 2017)设定了更难的场景:在已有问答对的基础上,用信息检索收集多篇文档作为上下文。尽管如此,多数问题仍可通过把问题与单一段落中邻近的几句话匹配来作答,这种设置是有限的,因为它并不要求更复杂的推理(例如跨越多段)。

Second, existing datasets that target multi-hop reasoning, such as QAngaroo (Welbl et al., 2018) and COMPLEXWEBQUESTIONS (Talmor and Berant, 2018), are constructed using existing knowledge bases (KBs). As a result, these datasets are constrained by the schema of the KBs they use, and therefore the diversity of questions and answers is inherently limited.

第二,面向多跳推理的已有数据集,如 QAngaroo(Welbl et al., 2018)和 COMPLEXWEBQUESTIONS(Talmor and Berant, 2018),是用已有知识库(KB)构建的。因此,这些数据集受所用知识库模式的约束,问题和答案的多样性也就受到先天限制。

Third, all of the above datasets only provide distant supervision; i.e., the systems only know what the answer is, but do not know what supporting facts lead to it. This makes it difficult for models to learn about the underlying reasoning process, as well as to make explainable predictions.

第三,上述数据集都只提供远程监督;也就是说,系统只知道答案是什么,却不知道通向答案的支持事实是哪些。这使模型难以学习背后的推理过程,也难以做出可解释的预测。

To address the above challenges, we aim at creating a QA dataset that requires reasoning over multiple documents, and does so in natural language, without constraining itself to an existing knowledge base or knowledge schema. We also want it to provide the system with strong supervision about what text the answer is actually derived from, to help guide systems to perform meaningful and explainable reasoning.

为应对上述挑战,我们希望构建一套需要在多篇文档上推理的问答数据集,并且问题以自然语言提出,不受既有知识库或知识模式的约束。我们还希望它能向系统提供强监督,说明答案实际来自哪些文本,以引导系统做有意义、可解释的推理。

We present HOTPOTQA1^11, a large-scale dataset that satisfies these desiderata. HOTPOTQA is collected by crowdsourcing based on Wikipedia articles, where crowd workers are shown multiple supporting context documents and asked explicitly to come up with questions requiring reasoning about all of the documents. This ensures it covers multi-hop questions that are more natural, and are not designed with any pre-existing knowledge base schema in mind. Moreover, we also ask the crowd workers to provide the supporting facts they use to answer the question, which we also provide as part of the dataset (see Figure 1 for an example). We have carefully designed a data collection pipeline for HOTPOTQA, since the collection of high-quality multi-hop questions is non-trivial. We hope that this pipeline also sheds light on future work in this direction. Finally, we also collected a novel type of questions—comparison questions—as part of HOTPOTQA, in which we require systems to compare two entities on some shared properties to test their understanding of both language and common concepts such as numerical magnitude. We make HOTPOTQA publicly available at https://HotpotQA.github.io.

我们提出 HOTPOTQA1^11,这是一套满足上述要求的大规模数据集。HOTPOTQA 基于维基百科文章众包收集:众包工人会看到多篇支持性上下文文档,并被明确要求提出需要对所有文档进行推理的问题。这样可以覆盖更自然的多跳问题,而不是按既有知识库模式来设计。此外,我们还要求众包工人提供他们用来回答问题的支持事实,这些内容也作为数据集的一部分发布(例子见图 1)。由于高质量多跳问题的收集并不容易,我们为 HOTPOTQA 仔细设计了一套数据收集流程,也希望这套流程能为后续工作提供参考。最后,我们还在 HOTPOTQA 中收集了一类新问题——比较题:系统需要就某些共有属性比较两个实体,以检验它们对语言以及数值大小等常见概念的理解。我们将 HOTPOTQA 公开发布在 https://HotpotQA.github.io。

1^11The name comes from the first three authors’ arriving at the main idea during a discussion at a hot pot restaurant.

1^11这个名称来自前三位作者在一家火锅店讨论时形成的主要想法。

2 Data Collection

2 数据收集

The main goal of our work is to collect a diverse and explainable question answering dataset that requires multi-hop reasoning. One way to do so is to define reasoning chains based on a knowledge base (Welbl et al., 2018; Talmor and Berant, 2018). However, the resulting datasets are limited by the incompleteness of entity relations and the lack of diversity in the question types. Instead, in this work, we focus on text-based question answering in order to diversify the questions and answers. The overall setting is that given some context paragraphs (e.g., a few paragraphs, or the entire Web) and a question, a QA system answers the question by extracting a span of text from the context, similar to Rajpurkar et al. (2016). We additionally ensure that it is necessary to perform multi-hop reasoning to correctly answer the question.

我们这项工作的主要目标,是收集一套多样、可解释、并且需要多跳推理的问答数据集。一种做法是基于知识库来定义推理链(Welbl et al., 2018; Talmor and Berant, 2018)。但这样得到的数据集会受实体关系不完整、问题类型不够多样的限制。因此,我们转而关注基于文本的问答,以使问题和答案更加多样。总体设置是:给定若干上下文段落(例如几段文字,或整个网络)和一个问题,问答系统从上下文中抽取一个文本跨度作为答案,这与 Rajpurkar et al. (2016) 类似。我们还额外保证,要正确回答问题,必须进行多跳推理。

It is non-trivial to collect text-based multi-hop questions. In our pilot studies, we found that simply giving an arbitrary set of paragraphs to crowd workers is counterproductive, because for most paragraph sets, it is difficult to ask a meaningful multi-hop question. To address this challenge, we carefully design a pipeline to collect text-based multi-hop questions. Below, we will highlight the key design choices in our pipeline.

收集基于文本的多跳问题并不容易。在预实验中,我们发现,如果只是把任意一组段落交给众包工人,往往适得其反,因为对大多数段落组合来说,很难提出有意义的多跳问题。为应对这一挑战,我们仔细设计了一套收集基于文本的多跳问题的流程。下面将说明流程中的主要设计选择。

Building a Wikipedia Hyperlink Graph. We use the entire English Wikipedia dump as our corpus.2^22 In this corpus, we make two observations: (1) hyper-links in the Wikipedia articles often naturally entail a relation between two (already disambiguated) entities in the context, which could potentially be used to facilitate multi-hop reasoning; (2) the first paragraph of each article often contains much information that could be queried in a meaningful way. Based on these observations, we extract all the hyperlinks from the first paragraphs of all Wikipedia articles. With these hyperlinks, we build a directed graph GGG, where each edge (a,b)(a, b)(a,b) indicates there is a hyperlink from the first paragraph of article aaa to article bbb.

构建维基百科超链接图。 我们以整个英文维基百科转储作为语料库。2^22 在该语料中,我们有两点观察:(1)维基百科文章中的超链接,常常自然蕴含上下文里两个(已经消歧的)实体之间的关系,这有可能用来促进多跳推理;(2)每篇文章的第一段往往包含许多可以有意义地提问的信息。基于这些观察,我们从所有维基百科文章的第一段中抽出全部超链接,并据此构建有向图 GGG,其中每条边 (a,b)(a, b)(a,b) 表示文章 aaa 的第一段中有指向文章 bbb 的超链接。

2^22https://dumps.wikimedia.org/

Generating Candidate Paragraph Pairs. To generate meaningful pairs of paragraphs for multi-hop question answering with GGG, we start by considering an example question “when was the singer and songwriter of Radiohead born?” To answer this question, one would need to first reason that the “singer and songwriter of Radiohead” is “Thom Yorke”, and then figure out his birthday in the text. We call “Thom Yorke” a bridge entity in this example. Given an edge (a,b)(a, b)(a,b) in the hyperlink graph GGG, the entity of bbb can usually be viewed as a bridge entity that connects aaa and bbb. As we observe articles bbb usually determine the theme of the shared context between aaa and bbb, but not all articles bbb are suitable for collecting multi-hop questions. For example, entities like countries are frequently referred to in Wikipedia, but don’t necessarily have much in common with all incoming links. It is also difficult, for instance, for the crowd workers to ask meaningful multi-hop questions about highly technical entities like the IPv4 protocol. To alleviate this issue, we constrain the bridge entities to a set of manually curated pages in Wikipedia (see Appendix A). After curating a set of pages BBB, we create candidate paragraph pairs by sampling edges (a,b)(a, b)(a,b) from the hyperlink graph such that b∈Bb \in Bb∈B.

生成候选段落对。 为了用图 GGG 生成适合多跳问答的有意义段落对,我们先看一个例子:“Radiohead 的歌手兼词曲作者是什么时候出生的?”要回答这个问题,需要先推理出 “Radiohead 的歌手兼词曲作者” 是 “Thom Yorke”,再在文本中找出他的出生日期。在这个例子里,我们把 “Thom Yorke” 称为桥接实体(bridge entity)。给定超链接图 GGG 中的一条边 (a,b)(a, b)(a,b),bbb 所对应的实体通常可以看作连接 aaa 与 bbb 的桥接实体。我们观察到,文章 bbb 通常决定了 aaa 与 bbb 共享上下文的主题,但并非所有文章 bbb 都适合用来收集多跳问题。例如,国家这类实体在维基百科中经常被引用,却不一定与所有入边页面有很多共同点。众包工人也很难对 IPv4 协议这类高度技术性的实体提出有意义的多跳问题。为缓解这一问题,我们把桥接实体限制在一组人工整理过的维基百科页面上(见附录 A)。在整理得到页面集合 BBB 之后,我们从超链接图中抽样满足 b∈Bb \in Bb∈B 的边 (a,b)(a, b)(a,b),作为候选段落对。

Comparison Questions. In addition to questions collected using bridge entities, we also collect another type of multi-hop questions—comparison questions. The main idea is that comparing two entities from the same category usually results in interesting multi-hop questions, e.g., “Who has played for more NBA teams, Michael Jordan or Kobe Bryant?” To facilitate collecting this type of question, we manually curate 42 lists of similar entities (denoted as LLL) from Wikipedia.3^33

比较题。 除了用桥接实体收集的问题,我们还收集另一类多跳问题——比较题。主要想法是:比较同一类别中的两个实体,通常能得到有意思的多跳问题,例如:“Michael Jordan 和 Kobe Bryant,谁效力过更多 NBA 球队?”为便于收集这类问题,我们从维基百科中人工整理了 42 份相似实体清单(记为 LLL)。3^33

3^33This is achieved by manually curating lists from the Wikipedia “List of lists of lists” (https://wiki.sh/y8qv). One example is “Highest Mountains on Earth”.

3^33具体做法是从维基百科的 “List of lists of lists”(https://wiki.sh/y8qv)中人工整理清单。其中一个例子是 “Highest Mountains on Earth”。

To generate candidate paragraph pairs, we randomly sample two paragraphs from the same list and present them to the crowd worker.

为生成候选段落对,我们从同一份清单中随机抽取两段,交给众包工人。

To increase the diversity of multi-hop questions, we also introduce a subset of yes/no questions in comparison questions. This complements the original scope of comparison questions by offering new ways to require systems to reason over both paragraphs. For example, consider the entities Iron Maiden (from the UK) and AC/DC (from Australia). Questions like “Is Iron Maiden or AC/DC from the UK?” are not ideal, because one would deduce the answer is “Iron Maiden” even if one only had access to that article. With yes/no questions, one may ask “Are Iron Maiden and AC/DC from the same country?”, which requires reasoning over both paragraphs.

为增加多跳问题的多样性,我们还在比较题中引入了一部分是非题。这补充了比较题原有的范围,提供了新的方式,要求系统同时对两段进行推理。例如,考虑实体 Iron Maiden(来自英国)和 AC/DC(来自澳大利亚)。“Is Iron Maiden or AC/DC from the UK?” 这类问题并不理想,因为即使只看到 Iron Maiden 这一篇,也能推出答案是 “Iron Maiden”。若改用是非题,可以问 “Are Iron Maiden and AC/DC from the same country?”,这就必须同时对两段推理。

Algorithm 1 Overall data collection procedure

Input: question type ratio r1=0.75r_1 = 0.75r1​=0.75, yes/no ratio r2=0.5r_2 = 0.5r2​=0.5

while not finished do

  if random()<r1\mathrm{random}() < r_1random()<r1​ then

    Uniformly sample an entity b∈Bb \in Bb∈B

    Uniformly sample an edge (a,b)(a, b)(a,b)

    Workers ask a question about paragraphs aaa and bbb

  else

    Sample a list from LLL, with probabilities weighted by list sizes

    Uniformly sample two entities (a,b)(a, b)(a,b) from the list

    if random()<r2\mathrm{random}() < r_2random()<r2​ then

      Workers ask a yes/no question to compare aaa and bbb

    else

      Workers ask a question with a span answer to compare aaa and bbb

    end if

  end if

  Workers provide the supporting facts

end while

算法 1 总体数据收集流程

输入: 问题类型比例 r1=0.75r_1 = 0.75r1​=0.75,是非题比例 r2=0.5r_2 = 0.5r2​=0.5

while 尚未结束 do

  if random()<r1\mathrm{random}() < r_1random()<r1​ then

    从 BBB 中均匀抽样一个实体 b∈Bb \in Bb∈B

    均匀抽样一条边 (a,b)(a, b)(a,b)

    工人针对段落 aaa 和 bbb 提出一个问题

  else

    按清单规模加权的概率,从 LLL 中抽样一份清单

    从该清单中均匀抽取两个实体 (a,b)(a, b)(a,b)

    if random()<r2\mathrm{random}() < r_2random()<r2​ then

      工人提出一个比较 aaa 和 bbb 的是非题

    else

      工人提出一个以文本跨度为答案、比较 aaa 和 bbb 的问题

    end if

  end if

  工人提供支持事实

end while

To the best of our knowledge, text-based comparison questions are a novel type of questions that have not been considered by previous datasets. More importantly, answering these questions usually requires arithmetic comparison, such as comparing ages given birth dates, which presents a new challenge for future model development.

据我们所知,基于文本的比较题是以往数据集尚未考虑过的一类新问题。更重要的是,回答这类问题通常需要进行算术比较,例如根据出生日期比较年龄,这为后续模型开发提出了新的挑战。

Collecting Supporting Facts. To enhance the explainability of question answering systems, we want them to output a set of supporting facts necessary to arrive at the answer, when the answer is generated. To this end, we also collect the sentences that determine the answers from crowd workers. These supporting facts can serve as strong supervision for what sentences to pay attention to. Moreover, we can now test the explainability of a model by comparing the predicted supporting facts to the ground truth ones. The overall procedure of data collection is illustrated in Algorithm 1.

收集支持事实。 为增强问答系统的可解释性,我们希望系统在生成答案时,同时输出得出该答案所必需的一组支持事实。为此,我们还从众包工人那里收集决定答案的那些句子。这些支持事实可以作为强监督,说明应当关注哪些句子。此外,我们现在可以通过把预测的支持事实与金标进行比较,来检验模型的可解释性。数据收集的总体流程如算法 1 所示。

3 Processing and Benchmark Settings

3 处理与基准设置

We collected 112,779 valid examples in total on Amazon Mechanical Turk4^44 using the ParlAI interface (Miller et al., 2017) (see Appendix A). To isolate potential single-hop questions from the desired multi-hop ones, we first split out a subset of data called train-easy. Specifically, we randomly sampled questions (∼\sim∼3–10 per Turker) from top-contributing turkers, and categorized all their questions into the train-easy set if an overwhelming percentage in the sample only required reasoning over one of the paragraphs. We sampled these turkers because they contributed more than 70% of our data. This train-easy set contains 18,089 mostly single-hop examples.

我们共在 Amazon Mechanical Turk4^44 上用 ParlAI 界面(Miller et al., 2017)收集了 112,779 条有效样本(见附录 A)。为了把可能的单跳问题从所需的多跳问题中分离出来,我们先拆出称为 train-easy 的子集。具体做法是:从贡献最多的众包工人中随机抽样问题(每人约 3–10 条),若样本里绝大多数只需对其中一段推理即可作答,就把该工人的全部问题归入 train-easy。我们抽样这些工人,是因为他们贡献了超过 70% 的数据。这个 train-easy 子集包含 18,089 条大多为单跳的样本。

4^44https://www.mturk.com/

We implemented a question answering model based on the current state-of-the-art architectures, which we discuss in detail in Section 5.1. Based on this model, we performed a three-fold cross validation on the remaining multi-hop examples. Among these examples, the models were able to correctly answer 60% of the questions with high confidence (determined by thresholding the model loss). These correctly-answered questions (56,814 in total, 60% of the multi-hop examples) are split out and marked as the train-medium subset, which will also be used as part of our training set.

我们基于当时最先进的架构实现了一个问答模型,细节在第 5.1 节讨论。基于该模型,我们在剩余的多跳样本上做了三折交叉验证。在这些样本中,模型能够以高置信度正确回答 60% 的问题(置信度由模型损失阈值确定)。这些被正确回答的问题(共 56,814 条,占多跳样本的 60%)被拆出并标记为 train-medium 子集,也将作为训练集的一部分使用。

After splitting out train-easy and train-medium, we are left with hard examples. As our ultimate goal is to solve multi-hop question answering, we focus on questions that the latest modeling techniques are not able to answer. Thus we constrain our dev and test sets to be hard examples. Specifically, we randomly divide the hard examples into four subsets, train-hard, dev, test-distractor, and test-fullwiki. Statistics about the data split can be found in Table 1. In Section 5, we will show that combining train-easy, train-medium, and train-hard to train models yields the best performance, so we use the combined set as our default training set. The two test sets test-distractor and test-fullwiki are used in two different benchmark settings, which we introduce next.

拆出 train-easy 和 train-medium 之后,剩下的是困难样本。由于我们的最终目标是解决多跳问答,我们关注当时最新建模技术仍无法回答的问题,因此把开发集和测试集限制为困难样本。具体而言,我们把困难样本随机分成四个子集:train-hard、dev、test-distractor 和 test-fullwiki。数据划分的统计见表 1。在第 5 节中我们将表明,把 train-easy、train-medium 和 train-hard 合并起来训练模型效果最好,因此把合并后的集合作为默认训练集。两套测试集 test-distractor 和 test-fullwiki 分别用于接下来介绍的两种基准设置。

Table 1: Data split. The splits train-easy, train-medium, and train-hard are combined for training. The distractor and full wiki settings use different test sets so that the gold paragraphs in the full wiki test set remain unknown to any models.

Name Desc. Usage # Examples
train-easy single-hop training 18,089
train-medium multi-hop training 56,814
train-hard hard multi-hop training 15,661
dev hard multi-hop dev 7,405
test-distractor hard multi-hop test 7,405
test-fullwiki hard multi-hop test 7,405
Total 112,779

表 1: 数据划分。train-easy、train-medium 和 train-hard 合并用于训练。干扰项设置和全文维基设置使用不同的测试集,以使全文维基测试集中的金标段落对任何模型都保持未知。

名称 说明 用途 样本数
train-easy 单跳 训练 18,089
train-medium 多跳 训练 56,814
train-hard 困难多跳 训练 15,661
dev 困难多跳 开发 7,405
test-distractor 困难多跳 测试 7,405
test-fullwiki 困难多跳 测试 7,405
合计 112,779

We create two benchmark settings. In the first setting, to challenge the model to find the true supporting facts in the presence of noise, for each example we employ bigram tf-idf (Chen et al., 2017) to retrieve 8 paragraphs from Wikipedia as distractors, using the question as the query. We mix them with the 2 gold paragraphs (the ones used to collect the question and answer) to construct the distractor setting. The 2 gold paragraphs and the 8 distractors are shuffled before they are fed to the model. In the second setting, we fully test the model’s ability to locate relevant facts as well as reasoning about them by requiring it to answer the question given the first paragraphs of all Wikipedia articles without the gold paragraphs specified. This full wiki setting truly tests the performance of the systems’ ability at multi-hop reasoning in the wild.5^55 The two settings present different levels of difficulty, and would require techniques ranging from reading comprehension to information retrieval. As shown in Table 1, we use separate test sets for the two settings to avoid leaking information, because the gold paragraphs are available to a model in the distractor setting, but should not be accessible in the full wiki setting.

我们设置了两种基准。第一种设置是为了在有噪声的情况下,考验模型能否找到真正的支持事实:对每个样本,我们以问题为查询,用二元语法 tf-idf(Chen et al., 2017)从维基百科检索 8 段作为干扰项(distractors),再与 2 段金标段落(用来收集该问答对的段落)混合,构成干扰项(distractor)设置。这 2 段金标段落和 8 段干扰项在输入模型前会被打乱。第二种设置则全面考察模型定位相关事实并加以推理的能力:在不指定金标段落的情况下,把全部维基百科文章的第一段交给模型来回答问题。这一全文维基(full wiki)设置,真正检验系统在开放环境中做多跳推理的能力。5^55 两种设置难度不同,所需技术从阅读理解到信息检索不等。如表 1 所示,两种设置使用不同的测试集以避免信息泄露,因为金标段落在干扰项设置中对模型是可见的,但在全文维基设置中不应被获取。

5^55As we required the crowd workers to use complete entity names in the question, the majority of the questions are unambiguous in the full wiki setting.

5^55由于我们要求众包工人在问题中使用完整的实体名称,大多数问题在全文维基设置中是无歧义的。

We also try to understand the model’s good performance on the train-medium split. Manual analysis shows that the ratio of multi-hop questions in train-medium is similar to that of the hard examples (93.3% in train-medium vs. 92.0% in dev), but one of the question types appears more frequently in train-medium compared to the hard splits (Type II: 32.0% in train-medium vs. 15.0% in dev, see Section 4 for the definition of Type II questions). These observations demonstrate that given enough training data, existing neural architectures can be trained to answer certain types and certain subsets of the multi-hop questions. However, train-medium remains challenging when not just the gold paragraphs are present—we show in Appendix C that the retrieval problem on these examples are as difficult as that on their hard cousins.

我们还试图理解模型在 train-medium 划分上表现较好的原因。人工分析表明,train-medium 中多跳问题的比例与困难样本相近(train-medium 中为 93.3%,dev 中为 92.0%),但有一类问题在 train-medium 中比在困难划分中出现得更频繁(类型 II:train-medium 中为 32.0%,dev 中为 15.0%,类型 II 的定义见第 4 节)。这些观察表明,在训练数据足够时,已有的神经网络架构可以被训练来回答某些类型、某些子集的多跳问题。不过,当不仅仅是金标段落在场时,train-medium 仍然有难度——我们在附录 C 中表明,这些样本上的检索问题与其困难对应样本同样困难。

4 Dataset Analysis

4 数据集分析

In this section, we analyze the types of questions, types of answers, and types of multi-hop reasoning covered in the dataset.

在这一节中,我们分析数据集所覆盖的问题类型、答案类型以及多跳推理类型。

Question Types. We heuristically identified question types for each collected question. To identify the question type, we first locate the central question word (CQW) in the question. Since HOTPOTQA contains comparison questions and yes/no questions, we consider as question words WH-words, copulas (“is”, “are”), and auxiliary verbs (“does”, “did”). Because questions often involve relative clauses beginning with WH-words, we define the CQW as the first question word in the question if it can be found in the first three tokens, or the last question word otherwise. Then, we determine question type by extracting words up to 2 tokens away to the right of the CQW, along with the token to the left if it is one of a few common prepositions (e.g., in the cases of “in which” and “by whom”).

问题类型。 我们用启发式方法为每条收集到的问题确定类型。为识别问题类型,我们首先在问题中定位中心疑问词(central question word,CQW)。由于 HOTPOTQA 包含比较题和是非题,我们把 WH 词、系动词(“is”、“are”)以及助动词(“does”、“did”)都视为疑问词。因为问题中常常包含以 WH 词引导的关系从句,我们把 CQW 定义为:若疑问词出现在前三个词之内,则取第一个疑问词,否则取最后一个疑问词。随后,我们提取 CQW 右侧最多 2 个词,若左侧是少数常见介词之一(例如 “in which”、“by whom” 的情形),则一并取出左侧那个词,以此来确定问题类型。

Sunburst chart of question types covered in HotpotQA

Figure 2: Types of questions covered in HOTPOTQA. Question types are extracted heuristically, starting at question words or prepositions preceding them. Empty colored blocks indicate suffixes that are too rare to show individually. See main text for more details.

图 2:HOTPOTQA 所覆盖的问题类型。问题类型用启发式方法抽取,从疑问词或其前的介词开始。空白的彩色块表示过于罕见、不便单独展示的后缀。更多细节见正文。

We visualize the distribution of question types in Figure 2, and label the ones shared among more than 250 questions. As is shown, our dataset covers a diverse variety of questions centered around entities, locations, events, dates, and numbers, as well as yes/no questions directed at comparing two entities (“Are both A and B ...?”), to name a few.

我们在图 2 中可视化了问题类型的分布,并标出了超过 250 条问题所共有的类型。如图所示,我们的数据集覆盖了围绕实体、地点、事件、日期和数字的多种问题,以及用于比较两个实体的是非题(“Are both A and B ...?”)等。

Answer Types. We further sample 100 examples from the dataset, and present the types of answers in Table 2. As can be seen, HOTPOTQA covers a broad range of answer types, which matches our initial analysis of question types. We find that a majority of the questions are about entities in the articles (68%), and a non-negligible amount of questions also ask about various properties like date (9%) and other descriptive properties such as numbers (8%) and adjectives (4%).

答案类型。 我们进一步从数据集中抽样 100 条,并在表 2 中展示答案类型。可以看到,HOTPOTQA 覆盖了广泛的答案类型,这与我们起初对问题类型的分析相符。我们发现,多数问题是关于文章中的实体(68%),也有不可忽视的一部分问题询问各种属性,如日期(9%)以及其他描述性属性,例如数字(8%)和形容词(4%)。

Table 2: Types of answers in HOTPOTQA.

Answer Type % Example(s)
Person 30 King Edward II, Rihanna
Group / Org 13 Cartoonito, Apalachee
Location 10 Fort Richardson, California
Date 9 10th or even 13th century
Number 8 79.92 million, 17
Artwork 8 Die schweigsame Frau
Yes/No 6 –
Adjective 4 conservative
Event 1 Prix Benois de la Danse
Other proper noun 6 Cold War, Laban Movement Analysis
Common noun 5 comedy, both men and women

表 2: HOTPOTQA 中的答案类型。

答案类型 % 例子
人物 30 King Edward II, Rihanna
团体 / 组织 13 Cartoonito, Apalachee
地点 10 Fort Richardson, California
日期 9 10th or even 13th century
数字 8 79.92 million, 17
作品 8 Die schweigsame Frau
是/否 6 –
形容词 4 conservative
事件 1 Prix Benois de la Danse
其他专有名词 6 Cold War, Laban Movement Analysis
普通名词 5 comedy, both men and women

Multi-hop Reasoning Types. We also sampled 100 examples from the dev and test sets and manually classified the types of reasoning required to answer each question. Besides comparing two entities, there are three main types of multi-hop reasoning required to answer these questions, which we show in Table 3 accompanied with examples.

多跳推理类型。 我们还从开发集和测试集中抽样 100 条,并人工标注了回答每个问题所需的推理类型。除比较两个实体之外,回答这些问题主要需要三类多跳推理,我们在表 3 中连同例子一并展示。

Table 3: Types of multi-hop reasoning required to answer questions in the HOTPOTQA dev and test sets. We show in orange bold italics bridge entities if applicable, blue italics supporting facts from the paragraphs that connect directly to the question, and green bold the answer in the paragraph or following the question. The remaining 8% are single-hop (6%) or unanswerable questions (2%) by our judgement.

Reasoning Type % Example(s)
Inferring the bridge entity to complete the 2nd-hop question (Type I) 42 Paragraph A: The 2015 Diamond Head Classic was a college basketball tournament ... Buddy Hield was named the tournament’s MVP. Paragraph B: Chavano Rainier “Buddy” Hield is a Bahamian professional basketball player for the Sacramento Kings of the NBA... Q: Which team does the player named 2015 Diamond Head Classic’s MVP play for?
Comparing two entities (Comparison) 27 Paragraph A: LostAlone were a British rock band ... consisted of Steven Battelle, Alan Williamson, and Mark Gibson... Paragraph B: Guster is an American alternative rock band ... Founding members Adam Gardner, Ryan Miller, and Brian Rosenworcel began... Q: Did LostAlone and Guster have the same number of members? (yes)
Locating the answer entity by checking multiple properties (Type II) 15 Paragraph A: Several current and former members of the Pittsburgh Pirates – ... John Milner, Dave Parker, and Rod Scurry... Paragraph B: David Gene Parker, nicknamed “The Cobra”, is an American former player in Major League Baseball... Q: Which former member of the Pittsburgh Pirates was nicknamed “The Cobra”?
Inferring about the property of an entity in question through a bridge entity (Type III) 6 Paragraph A: Marine Tactical Air Command Squadron 28 is a United States Marine Corps aviation command and control unit based at Marine Corps Air Station Cherry Point... Paragraph B: Marine Corps Air Station Cherry Point ... is a United States Marine Corps airfield located in Havelock, North Carolina, USA ... Q: What city is the Marine Air Control Group 28 located in?
Other types of reasoning that require more than two supporting facts (Other) 2 Paragraph A: ... the towns of Yodobashi, Okubo, Totsuka, and Ochiai town were merged into Yodobashi ward. ... Yodobashi Camera is a store with its name taken from the town and ward. Paragraph B: Yodobashi Camera Co., Ltd. is a major Japanese retail chain specializing in electronics, PCs, cameras and photographic equipment. Q: Aside from Yodobashi, what other towns were merged into the ward which gave the major Japanese retail chain specializing in electronics, PCs, cameras, and photographic equipment it’s name?

表 3: 回答 HOTPOTQA 开发集和测试集问题所需的多跳推理类型。若适用,我们用橙色粗斜体标出桥接实体,用蓝色斜体标出段落中直接连接问题的支持事实,并用绿色粗体标出段落中或问题后的答案。按我们的判断,其余 8% 是单跳问题(6%)或不可回答的问题(2%)。

推理类型 % 例子
先推断出桥接实体,再完成第二跳问题(类型 I) 42 段落 A: The 2015 Diamond Head Classic was a college basketball tournament ... Buddy Hield was named the tournament’s MVP. 段落 B: Chavano Rainier “Buddy” Hield is a Bahamian professional basketball player for the Sacramento Kings of the NBA... 问: Which team does the player named 2015 Diamond Head Classic’s MVP play for?
比较两个实体(比较) 27 段落 A: LostAlone were a British rock band ... consisted of Steven Battelle, Alan Williamson, and Mark Gibson... 段落 B: Guster is an American alternative rock band ... Founding members Adam Gardner, Ryan Miller, and Brian Rosenworcel began... 问: Did LostAlone and Guster have the same number of members?(yes)
通过同时检查多个属性来定位答案实体(类型 II) 15 段落 A: Several current and former members of the Pittsburgh Pirates – ... John Milner, Dave Parker, and Rod Scurry... 段落 B: David Gene Parker, nicknamed “The Cobra”, is an American former player in Major League Baseball... 问: Which former member of the Pittsburgh Pirates was nicknamed “The Cobra”?
借助桥接实体推断所问实体的属性(类型 III) 6 段落 A: Marine Tactical Air Command Squadron 28 is a United States Marine Corps aviation command and control unit based at Marine Corps Air Station Cherry Point... 段落 B: Marine Corps Air Station Cherry Point ... is a United States Marine Corps airfield located in Havelock, North Carolina, USA ... 问: What city is the Marine Air Control Group 28 located in?
需要两条以上支持事实的其他推理类型(其他) 2 段落 A: ... the towns of Yodobashi, Okubo, Totsuka, and Ochiai town were merged into Yodobashi ward. ... Yodobashi Camera is a store with its name taken from the town and ward. 段落 B: Yodobashi Camera Co., Ltd. is a major Japanese retail chain specializing in electronics, PCs, cameras and photographic equipment. 问: Aside from Yodobashi, what other towns were merged into the ward which gave the major Japanese retail chain specializing in electronics, PCs, cameras, and photographic equipment it’s name?

Most of the questions require at least one supporting fact from each paragraph to answer. A majority of sampled questions (42%) require chain reasoning (Type I in the table), where the reader must first identify a bridge entity before the second hop can be answered by filling in the bridge. One strategy to answer these questions would be to decompose them into consecutive single-hop questions. The bridge entity could also be used implicitly to help infer properties of other entities related to it. In some questions (Type III), the entity in question shares certain properties with a bridge entity (e.g., they are collocated), and we can infer its properties through the bridge entity. Another type of question involves locating the answer entity by satisfying multiple properties simultaneously (Type II). Here, to answer the question, one could find the set of all entities that satisfy each of the properties mentioned, and take an intersection to arrive at the final answer. Questions comparing two entities (Comparison) also require the system to understand the properties in question about the two entities (e.g., nationality), and sometimes require arithmetic such as counting (as seen in the table) or comparing numerical values (“Who is older, A or B?”). Finally, we find that sometimes the questions require more than two supporting facts to answer (Other). In our analysis, we also find that for all of the examples shown in the table, the supporting facts provided by the Turkers match exactly with the limited context shown here, showing that the supporting facts collected are of high quality.

大多数问题至少需要从每段中取一条支持事实才能作答。抽样问题中的多数(42%)需要链式推理(表中的类型 I):读者必须先找出桥接实体,才能把桥补上并完成第二跳。一种答题策略是把这些问题分解成连续的单跳问题。桥接实体也可以被隐式地用来帮助推断与之相关的其他实体的属性。在有些问题(类型 III)中,所问实体与某个桥接实体共享某些属性(例如它们位于同一地点),我们便可通过桥接实体来推断其属性。另一类问题是通过同时满足多个属性来定位答案实体(类型 II)。这时,可以先找出满足所提各项属性的实体集合,再取交集得到最终答案。比较两个实体的问题(比较)也要求系统理解这两个实体在所问属性上的情况(例如国籍),有时还需要计数(如表中所示)或比较数值(“Who is older, A or B?”)这类算术。最后,我们发现有些问题需要两条以上支持事实才能作答(其他)。在分析中我们还发现,表中列出的全部例子里,众包工人提供的支持事实与这里展示的有限上下文完全吻合,说明所收集的支持事实质量很高。

Aside from the reasoning types mentioned above, we also estimate that about 6% of the sampled questions can be answered with one of the two paragraphs, and 2% of them unanswerable. We also randomly sampled 100 examples from train-medium and train-hard combined, and the proportions of reasoning types are: Type I 38%, Type II 29%, Comparison 20%, Other 7%, Type III 2%, single-hop 2%, and unanswerable 2%.

除上述推理类型外,我们还估计约 6% 的抽样问题只需两段中的一段即可作答,另有 2% 不可回答。我们还从 train-medium 和 train-hard 的合集中随机抽样 100 条,推理类型比例为:类型 I 38%,类型 II 29%,比较 20%,其他 7%,类型 III 2%,单跳 2%,不可回答 2%。

5 Experiments

5 实验

5.1 Model Architecture and Training

5.1 模型架构与训练

To test the performance of leading QA systems on our data, we reimplemented the architecture described in Clark and Gardner (2017) as our baseline model. We note that our implementation without weight averaging achieves performance very close to what the authors reported on SQuAD (about 1 point worse in F1_11​). Our implemented model subsumes the latest technical advances on question answering, including character-level models, self-attention (Wang et al., 2017), and bi-attention (Seo et al., 2017). Combining these three key components is becoming standard practice, and various state-of-the-art or competitive architectures (Liu et al., 2018; Clark and Gardner, 2017; Wang et al., 2017; Seo et al., 2017; Pan et al., 2017; Salant and Berant, 2018; Xiong et al., 2018) on SQuAD can be viewed as similar to our implemented model. To accommodate yes/no questions, we also add a 3-way classifier after the last recurrent layer to produce the probabilities of “yes”, “no”, and span-based answers. During decoding, we first use the 3-way output to determine whether the answer is “yes”, “no”, or a text span. If it is a text span, we further search for the most probable span.

为了在我们的数据上测试领先问答系统的表现,我们重实现了 Clark and Gardner (2017) 所描述的架构,作为基线模型。需要说明的是,我们的实现未使用权重平均,在 SQuAD 上的表现与原作者报告的结果非常接近(F1_11​ 大约低 1 个点)。我们实现的模型涵盖了当时问答领域的最新技术进展,包括字符级模型、自注意力(Wang et al., 2017)以及双向注意力(Seo et al., 2017)。把这三项关键组件结合起来正在成为标准做法,SQuAD 上各种最先进或有竞争力的架构(Liu et al., 2018; Clark and Gardner, 2017; Wang et al., 2017; Seo et al., 2017; Pan et al., 2017; Salant and Berant, 2018; Xiong et al., 2018)都可以看成与我们实现的模型类似。为适应是非题,我们还在最后一个循环层之后加入一个三路分类器,输出 “yes”、“no” 以及基于跨度的答案的概率。解码时,我们先用这三路输出判断答案是 “yes”、“no” 还是文本跨度;若是文本跨度,再进一步搜索最可能的跨度。

Supporting Facts as Strong Supervision. To evaluate the baseline model’s performance in predicting explainable supporting facts, as well as how much they improve QA performance, we additionally design a component to incorporate such strong supervision into our model. For each sentence, we concatenate the output of the self-attention layer at the first and last positions, and use a binary linear classifier to predict the probability that the current sentence is a supporting fact. We minimize a binary cross entropy loss for this classifier. This objective is jointly optimized with the normal question answering objective in a multi-task learning setting, and they share the same low-level representations. With this classifier, the model can also be evaluated on the task of supporting fact prediction to gauge its explainability. Our overall architecture is illustrated in Figure 3. Though it is possible to build a pipeline system, in this work we focus on an end-to-end one, which is easier to tune and faster to train.

把支持事实作为强监督。 为评估基线模型在预测可解释支持事实方面的表现,以及它们能在多大程度上提升问答表现,我们额外设计了一个组件,把这种强监督纳入模型。对每个句子,我们把自注意力层在该句首、末位置的输出拼接起来,再用一个二分类线性分类器预测当前句子是支持事实的概率,并最小化该分类器的二元交叉熵损失。这一目标与常规问答目标在多任务学习设置中联合优化,并共享同样的底层表示。有了这个分类器,模型还可以在支持事实预测任务上评估,以衡量其可解释性。总体架构见图 3。尽管也可以搭建流水线系统,本文关注的是端到端系统,它更易于调参,训练也更快。

HotpotQA baseline model architecture with supporting-fact supervision

Figure 3: Our model architecture. Strong supervision over supporting facts is used in a multi-task setting.

图 3:我们的模型架构。对支持事实的强监督在多任务设置中使用。

5.2 Results

5.2 结果

We evaluate our model in the two benchmark settings. In the full wiki setting, to enable efficient tf-idf retrieval among 5,000,000+ wiki paragraphs, given a question we first return a candidate pool of at most 5,000 paragraphs using an inverted-index-based filtering strategy6^66 and then select the top 10 paragraphs in the pool as the final candidates using bigram tf-idf.7^77 Retrieval performance is shown in Table 5. After retrieving these 10 paragraphs, we then use the model trained in the distractor setting to evaluate its performance on these final candidate paragraphs.

我们在两种基准设置上评估模型。在全文维基设置中,为了在 5,000,000 余段维基段落中高效进行 tf-idf 检索,给定一个问题,我们先用基于倒排索引的过滤策略返回至多 5,000 段的候选池,6^66 再用二元语法 tf-idf 从池中选出前 10 段作为最终候选。7^77 检索表现见表 5。检索出这 10 段之后,我们再用在干扰项设置中训练的模型,在这些最终候选段落上评估其表现。

6^66See Appendix C for details.

6^66细节见附录 C。

7^77We choose the number of final candidates as 10 to stay consistent with the distractor setting where candidates are 2 gold paragraphs plus 8 distractors.

7^77我们将最终候选数定为 10,以与干扰项设置保持一致,那里的候选是 2 段金标段落加上 8 段干扰项。

Following previous work (Rajpurkar et al., 2016), we use exact match (EM) and F1_11​ as two evaluation metrics. To assess the explainability of the models, we further introduce two sets of metrics involving the supporting facts. The first set focuses on evaluating the supporting facts directly, namely EM and F1_11​ on the set of supporting fact sentences as compared to the gold set. The second set features joint metrics that combine the evaluation of answer spans and supporting facts as follows. For each example, given its precision and recall on the answer span (P(ans)P(\mathrm{ans})P(ans), R(ans)R(\mathrm{ans})R(ans)) and the supporting facts (P(sup)P(\mathrm{sup})P(sup), R(sup)R(\mathrm{sup})R(sup)), respectively, we calculate joint F1_11​ as

沿用以往工作(Rajpurkar et al., 2016),我们使用精确匹配(EM)和 F1_11​ 作为两项评估指标。为评估模型的可解释性,我们还引入两套涉及支持事实的指标。第一套直接评估支持事实,也就是支持事实句子集合相对于金标集合的 EM 和 F1_11​。第二套是把答案跨度与支持事实的评估结合起来的联合指标,计算如下。对每个样本,分别给定其在答案跨度上的精确率与召回率(P(ans)P(\mathrm{ans})P(ans),R(ans)R(\mathrm{ans})R(ans))以及在支持事实上的精确率与召回率(P(sup)P(\mathrm{sup})P(sup),R(sup)R(\mathrm{sup})R(sup)),联合 F1_11​ 计算为

P(joint)=P(ans)P(sup),R(joint)=R(ans)R(sup),P(\mathrm{joint}) = P(\mathrm{ans})P(\mathrm{sup}),\quad R(\mathrm{joint}) = R(\mathrm{ans})R(\mathrm{sup}),P(joint)=P(ans)P(sup),R(joint)=R(ans)R(sup),

Joint F1=2P(joint)R(joint)P(joint)+R(joint).\mathrm{Joint\ F}_1 = \frac{2P(\mathrm{joint})R(\mathrm{joint})}{P(\mathrm{joint}) + R(\mathrm{joint})}.Joint F1​=P(joint)+R(joint)2P(joint)R(joint)​.

Joint EM is 1 only if both tasks achieve an exact match and otherwise 0. Intuitively, these metrics penalize systems that perform poorly on either task. All metrics are evaluated example-by-example, and then averaged over examples in the evaluation set.

联合 EM 仅当两项任务都取得精确匹配时为 1,否则为 0。直观上,这些指标会惩罚在任一任务上表现不佳的系统。所有指标都按样本逐条计算,再在评估集上取平均。

The performance of our model on the benchmark settings is reported in Table 4, where all numbers are obtained with strong supervision over supporting facts. From the distractor setting to the full wiki setting, expanding the scope of the context increases the difficulty of question answering. The performance in the full wiki setting is substantially lower, which poses a challenge to existing techniques on retrieval-based question answering. Overall, model performance in all settings is significantly lower than human performance as shown in Section 5.3, which indicates that more technical advancements are needed in future work.

模型在两种基准设置上的表现见表 4,其中所有数字都是在对支持事实使用强监督的情况下得到的。从干扰项设置到全文维基设置,上下文范围扩大,问答难度也随之增加。全文维基设置上的表现明显更低,这对已有的检索式问答技术提出了挑战。总体而言,模型在所有设置上的表现都显著低于第 5.3 节所示的人的表现,说明后续工作还需要更多技术进步。

We also investigate the explainability of our model by measuring supporting fact prediction performance. Our model achieves 60+ supporting fact prediction F1_11​ and ∼\sim∼40 joint F1_11​, which indicates there is room for further improvement in terms of explainability.

我们还通过衡量支持事实预测的表现,考察模型的可解释性。模型的支持事实预测 F1_11​ 达到 60 以上,联合 F1_11​ 约为 40,说明在可解释性方面仍有改进空间。

In Table 6, we break down the performance on different question types. In the distractor setting, comparison questions have lower F1_11​ scores than questions involving bridge entities (as defined in Section 2), which indicates that better modeling this novel question type might need neural architectures. In the full wiki setting, the performance of bridge entity questions drops significantly while that of comparison questions decreases only marginally. This is because both entities usually appear in the comparison questions, and thus reduces the difficulty of retrieval. Combined with the retrieval performance in Table 5, we believe that the deterioration in the full wiki setting in Table 4 is largely due to the difficulty of retrieving both entities.

在表 6 中,我们按不同问题类型拆开看表现。在干扰项设置中,比较题的 F1_11​ 低于涉及桥接实体的问题(定义见第 2 节),这说明要更好地建模这一新问题类型,可能需要相应的神经网络架构。在全文维基设置中,桥接实体问题的表现大幅下降,而比较题只略有下降。这是因为比较题中两个实体通常都会出现,从而降低了检索难度。结合表 5 中的检索表现,我们认为表 4 中全文维基设置上的下降,很大程度上是由于难以同时检索到两个实体。

We perform an ablation study in the distractor setting, and report the results in Table 7. Both self-attention and character-level models contribute notably to the final performance, which is consistent with prior work. This means that techniques targeted at single-hop QA are still somewhat effective in our setting. Moreover, removing strong supervision over supporting facts decreases performance, which demonstrates the effectiveness of our approach and the usefulness of the supporting facts. We establish an estimate of the upper bound of strong supervision by only considering the supporting facts as the oracle context input to our model, which achieves a 10+ F1_11​ improvement over not using the supporting facts. Compared with the gain of strong supervision in our model (∼\sim∼2 points in F1_11​), our proposed method of incorporating supporting facts supervision is most likely suboptimal, and we leave the challenge of better modeling to future work. At last, we show that combining all data splits (train-easy, train-medium, and train-hard) yields the best performance, which is adopted as the default setting.

我们在干扰项设置中做了消融实验,结果见表 7。自注意力和字符级模型都对最终表现有明显贡献,这与以往工作一致。这意味着面向单跳问答的技术在我们的设置中仍然有一定效果。此外,去掉对支持事实的强监督会降低表现,说明我们的方法是有效的,支持事实也是有用的。我们还给出了强监督上限的一个估计:只把支持事实作为oracle上下文输入模型,相比不使用支持事实,F1_11​ 提高了 10 以上。与我们模型中强监督带来的增益(F1_11​ 约 2 个点)相比,我们把支持事实监督纳入模型的方法很可能并非最优,更好的建模方式留给后续工作。最后,我们表明,合并所有数据划分(train-easy、train-medium 和 train-hard)能得到最好的表现,这也被采用为默认设置。

Table 4: Main results: the performance of question answering and supporting fact prediction in the two benchmark settings. We encourage researchers to report these metrics when evaluating their methods.

Setting Split Answer EM Answer F1_11​ Sup Fact EM Sup Fact F1_11​ Joint EM Joint F1_11​
distractor dev 44.44 58.28 21.95 66.66 11.56 40.86
distractor test 45.46 58.99 22.24 66.62 12.04 41.37
full wiki dev 24.68 34.36 5.28 40.98 2.54 17.73
full wiki test 25.23 34.40 5.07 40.69 2.63 17.85

表 4: 主要结果:问答与支持事实预测在两种基准设置上的表现。我们鼓励研究者在评估自己的方法时报告这些指标。

设置 划分 答案 EM 答案 F1_11​ 支持事实 EM 支持事实 F1_11​ 联合 EM 联合 F1_11​
干扰项 开发 44.44 58.28 21.95 66.66 11.56 40.86
干扰项 测试 45.46 58.99 22.24 66.62 12.04 41.37
全文维基 开发 24.68 34.36 5.28 40.98 2.54 17.73
全文维基 测试 25.23 34.40 5.07 40.69 2.63 17.85

Table 5: Retrieval performance in the full wiki setting. Mean Rank is averaged over the ranks of two gold paragraphs.

Set MAP Mean Rank Hits@2 Hits@10
dev 43.93 314.71 39.43 56.06
test 43.21 314.05 38.67 55.88

表 5: 全文维基设置中的检索表现。Mean Rank 是对两个金标段落的排名取平均。

集合 MAP Mean Rank Hits@2 Hits@10
开发 43.93 314.71 39.43 56.06
测试 43.21 314.05 38.67 55.88

Table 6: Performance breakdown over different question types on the dev set in the distractor setting. “Br” denotes questions collected using bridge entities, and “Cp” denotes comparison questions.

Setting Br EM Br F1_11​ Cp EM Cp F1_11​
distractor 43.41 59.09 48.55 55.05
full wiki 19.76 30.42 43.87 50.70

表 6: 在干扰项设置的开发集上,按不同问题类型拆分的表现。“Br” 表示用桥接实体收集的问题,“Cp” 表示比较题。

设置 Br EM Br F1_11​ Cp EM Cp F1_11​
干扰项 43.41 59.09 48.55 55.05
全文维基 19.76 30.42 43.87 50.70

Table 7: Ablation study of question answering performance on the dev set in the distractor setting. “– sup fact” means removing strong supervision over supporting facts from our model. “– train-easy” and “– train-medium” means discarding the according data splits from training. “gold only” and “sup fact only” refer to using the gold paragraphs or the supporting facts as the only context input to the model.

Setting EM F1_11​
our model 44.44 58.28
– sup fact 42.79 56.19
– sup fact, self attention 41.59 55.19
– sup fact, char model 41.66 55.25
– sup fact, train-easy 41.61 55.12
– sup fact, train-easy, train-medium 31.07 43.61
gold only 48.38 63.58
sup fact only 51.95 66.98

表 7: 干扰项设置开发集上的问答表现消融实验。“– sup fact” 表示从我们的模型中去掉对支持事实的强监督。“– train-easy” 和 “– train-medium” 表示从训练中去掉相应的数据划分。“gold only” 和 “sup fact only” 指只把金标段落或只把支持事实作为模型的上下文输入。

设置 EM F1_11​
我们的模型 44.44 58.28
– sup fact 42.79 56.19
– sup fact, self attention 41.59 55.19
– sup fact, char model 41.66 55.25
– sup fact, train-easy 41.61 55.12
– sup fact, train-easy, train-medium 31.07 43.61
gold only 48.38 63.58
sup fact only 51.95 66.98

5.3 Establishing Human Performance

5.3 确定人的表现

To establish human performance on our dataset, we randomly sampled 1,000 examples from the dev and test sets, and had at least three additional Turkers provide answers and supporting facts for these examples. As a baseline, we treat the original Turker during data collection as the prediction, and the newly collected answers and supporting facts as references, to evaluate human performance. For each example, we choose the answer and supporting fact reference that maximize the F1_11​ score to report the final metrics to reduce the effect of ambiguity (Rajpurkar et al., 2016).

为确定人在我们数据集上的表现,我们从开发集和测试集中随机抽样 1,000 条,并让至少另外三名众包工人为这些样本提供答案和支持事实。作为基线,我们把数据收集时的原始标注者当作预测,把新收集的答案和支持事实当作参照,以此来评估人的表现。对每个样本,我们选择使 F1_11​ 最高的那条答案和支持事实参照来报告最终指标,以降低歧义的影响(Rajpurkar et al., 2016)。

Table 8: Comparing baseline model performance with human performance on 1,000 random samples. “Human UB” stands for the upper bound on annotator performance on HOTPOTQA. For details please refer to the main body.

Setting Answer EM Answer F1_11​ Sp Fact EM Sp Fact F1_11​ Joint EM Joint F1_11​
gold only 65.87 74.67 59.76 90.41 41.54 68.15
distractor 60.88 68.99 30.99 74.67 20.06 52.37
Human 83.60 91.40 61.50 90.04 52.30 82.55
Human UB 96.80 98.77 87.40 97.56 84.60 96.37

表 8: 在 1,000 条随机样本上比较基线模型表现与人的表现。“Human UB” 表示标注者在 HOTPOTQA 上表现的上界。细节请见正文。

设置 答案 EM 答案 F1_11​ 支持事实 EM 支持事实 F1_11​ 联合 EM 联合 F1_11​
gold only 65.87 74.67 59.76 90.41 41.54 68.15
干扰项 60.88 68.99 30.99 74.67 20.06 52.37
人 83.60 91.40 61.50 90.04 52.30 82.55
人的上界 96.80 98.77 87.40 97.56 84.60 96.37

As can be seen in Table 8, the original crowd worker achieves very high performance in both finding supporting facts, and answering the question correctly. If the baseline model were provided with the correct supporting paragraphs to begin with, it achieves parity with the crowd worker in finding supporting facts, but still falls short at finding the actual answer. When distractor paragraphs are present, the performance gap between the baseline model and the crowd worker on both tasks is enlarged to ∼\sim∼30% for both EM and F1_11​.

如表 8 所示,原始众包工人在寻找支持事实和正确回答问题上都达到了很高的表现。如果一开始就把正确的支持段落提供给基线模型,它在寻找支持事实上可以与众包工人持平,但在找出实际答案上仍然不足。当存在干扰段落时,基线模型与众包工人在两项任务上的差距都会扩大到约 30% 的 EM 和 F1_11​。

We further establish the upper bound of human performance in HOTPOTQA, by taking the maximum EM and F1_11​ for each example. Here, we use each Turker’s answer in turn as the prediction, and evaluate it against all other workers’ answers. As can be seen in Table 8, most of the metrics are close to 100%, illustrating that on most examples, at least a subset of Turkers agree with each other, showing high inter-annotator agreement. We also note that crowd workers agree less on supporting facts, which could reflect that this task is inherently more subjective than answering the question.

我们还进一步给出了人在 HOTPOTQA 上表现的上界,方法是对每个样本取最大的 EM 和 F1_11​。这里,我们轮流把每位众包工人的答案当作预测,用所有其他工人的答案来评估。如表 8 所示,多数指标接近 100%,说明在大多数样本上,至少有一部分众包工人彼此一致,标注者间一致性很高。我们也注意到,众包工人在支持事实上的一致性较低,这可能反映出这一任务本质上比回答问题本身更主观。

6 Related Work

6 相关工作

Various recently-proposed large-scale QA datasets can be categorized in four categories.

近来提出的各种大规模问答数据集,可以分成四类。

Single-document datasets. SQuAD (Rajpurkar et al., 2016, 2018) questions that are relatively simple because they usually require no more than one sentence in the paragraph to answer.

单文档数据集。 SQuAD(Rajpurkar et al., 2016, 2018)中的问题相对简单,因为它们通常最多只需要段落中的一句话即可作答。

Multi-document datasets. TriviaQA (Joshi et al., 2017) and SearchQA (Dunn et al., 2017) contain question answer pairs that are accompanied with more than one document as the context. This further challenges QA systems’ ability to accommodate longer contexts. However, since the supporting documents are collected after the question answer pairs with information retrieval, the questions are not guaranteed to involve interesting reasoning between multiple documents.

多文档数据集。 TriviaQA(Joshi et al., 2017)和 SearchQA(Dunn et al., 2017)包含配有不止一篇文档作为上下文的问答对。这进一步考验了问答系统处理更长上下文的能力。不过,由于支持文档是在问答对之后通过信息检索收集的,这些问题并不保证会涉及多篇文档之间有意思的推理。

KB-based multi-hop datasets. Recent datasets like QAngaroo (Welbl et al., 2018) and COMPLEXWEBQUESTIONS (Talmor and Berant, 2018) explore different approaches of using pre-existing knowledge bases (KB) with pre-defined logic rules to generate valid QA pairs, to test QA models’ capability of performing multi-hop reasoning. The diversity of questions and answers is largely limited by the fixed KB schemas or logical forms. Furthermore, some of the questions might be answerable by one text sentence due to the incompleteness of KBs.

基于知识库的多跳数据集。 近来的数据集,如 QAngaroo(Welbl et al., 2018)和 COMPLEXWEBQUESTIONS(Talmor and Berant, 2018),探索了用带有预定义逻辑规则的既有知识库(KB)来生成有效问答对的不同方法,以检验问答模型做多跳推理的能力。问题和答案的多样性在很大程度上受固定知识库模式或逻辑形式的限制。此外,由于知识库不完整,有些问题可能凭一句文本就能回答。

Free-form answer-generation datasets. MS MARCO (Nguyen et al., 2016) contains 100k user queries from Bing Search with human generated answers. Systems generate free-form answers and are evaluated by automatic metrics such as ROUGE-L and BLEU-1. However, the reliability of these metrics is questionable because they have been shown to correlate poorly with human judgement (Novikova et al., 2017).

自由形式答案生成数据集。 MS MARCO(Nguyen et al., 2016)包含 10 万条来自必应搜索的用户查询,并配有人工撰写的答案。系统生成自由形式的答案,并用 ROUGE-L、BLEU-1 等自动指标评估。不过,这些指标的可靠性存疑,因为已有研究表明它们与人的判断相关性较差(Novikova et al., 2017)。

7 Conclusions

7 结论

We present HOTPOTQA, a large-scale question answering dataset aimed at facilitating the development of QA systems capable of performing explainable, multi-hop reasoning over diverse natural language. We also offer a new type of factoid comparison questions to test systems’ ability to extract and compare various entity properties in text.

我们提出 HOTPOTQA,这是一套大规模问答数据集,旨在促进能够在多样的自然语言上做可解释、多跳推理的问答系统的发展。我们还提供一类新的事实比较题,用来检验系统从文本中抽取并比较各种实体属性的能力。

Acknowledgements

致谢

This work is partly funded by the Facebook ParlAI Research Award. ZY, WWC, and RS are supported by a Google grant, the DARPA grant D17AP00001, the ONR grants N000141512791, N000141812861, and the Nvidia NVAIL Award. SZ and YB are supported by Mila, Université de Montréal. PQ and CDM are supported by the National Science Foundation under Grant No. IIS-1514268. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

本工作部分由 Facebook ParlAI Research Award 资助。ZY、WWC 和 RS 得到一项 Google 资助、DARPA 资助 D17AP00001、ONR 资助 N000141512791、N000141812861 以及 Nvidia NVAIL Award 的支持。SZ 和 YB 得到 Mila、Université de Montréal 的支持。PQ 和 CDM 得到美国国家科学基金会资助编号 IIS-1514268 的支持。本文所表达的任何观点、发现、结论或建议均属于作者,并不一定反映美国国家科学基金会的观点。

References

参考文献

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Association for Computational Linguistics (ACL).

Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. In Proceedings of the 55th Annual Meeting of the Association of Computational Linguistics.

Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179.

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.

Xiaodong Liu, Yelong Shen, Kevin Duh, and Jianfeng Gao. 2018. Stochastic answer networks for machine reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.

Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60.

Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. arXiv preprint arXiv:1705.06476.

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS).

Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.

Boyuan Pan, Hao Li, Zhou Zhao, Bin Cao, Deng Cai, and Xiaofei He. 2017. Memen: Multi-layer embedding with memory networks for machine comprehension. arXiv preprint arXiv:1707.09098.

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Shimi Salant and Jonathan Berant. 2018. Contextualized word representations for reading comprehension. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics.

Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of the International Conference on Learning Representations.

Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics.

Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 189–198.

Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association of Computational Linguistics.

Caiming Xiong, Victor Zhong, and Richard Socher. 2018. DCN+: Mixed objective and deep residual coattention for question answering. In Proceedings of the International Conference on Learning Representations.

Zhilin Yang, Saizheng Zhang, Jack Urbanek, Will Feng, Alexander H Miller, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Mastering the dungeon: Grounded language learning by mechanical turker descent. In Proceedings of the International Conference on Learning Representations.

A Data Collection Details

A 数据收集细节

A.1 Data Preprocessing

A.1 数据预处理

We downloaded the dump of English Wikipedia of October 1, 2017, and extracted text and hyperlinks with WikiExtractor.8^88 We use Stanford CoreNLP 3.8.0 (Manning et al., 2014) for word and sentence tokenization. We use the resulting sentence boundaries for collection of supporting facts, and use token boundaries to check whether Turkers are providing answers that cover spans of entire tokens to avoid nonsensical partial-word answers.

我们下载了 2017 年 10 月 1 日的英文维基百科转储,并用 WikiExtractor 抽取文本和超链接。8^88 我们使用 Stanford CoreNLP 3.8.0(Manning et al., 2014)做词和句子切分。我们用得到的句子边界来收集支持事实,并用词边界检查众包工人提供的答案是否覆盖完整的词跨度,以避免没有意义的部分词答案。

8^88https://github.com/attardi/wikiextractor

A.2 Further Data Collection Details

A.2 更多数据收集细节

Details on Curating Wikipedia Pages. To make sure the sampled candidate paragraph pairs are intuitive for crowd workers to ask high-quality multi-hop questions about, we manually curate 591 categories from the lists of popular pages by WikiProject.9^99 For each category, we sample (a,b)(a, b)(a,b) pairs from the graph GGG where bbb is in the considered category, and manually check whether a multi-hop question can be asked given the pair (a,b)(a, b)(a,b). Those categories with a high probability of permitting multi-hop questions are selected.

整理维基百科页面的细节。 为确保抽样得到的候选段落对便于众包工人提出高质量的多跳问题,我们从 WikiProject 的热门页面清单中人工整理了 591 个类别。9^99 对每个类别,我们从图 GGG 中抽样 bbb 属于该类别的 (a,b)(a, b)(a,b) 对,并人工检查给定这对 (a,b)(a, b)(a,b) 是否能提出多跳问题。那些有较高概率允许提出多跳问题的类别被选中。

9^99https://wiki.sh/y8qu

Bonus Structures. To incentivize crowd workers to produce higher-quality data more efficiently, we follow Yang et al. (2018), and employ bonus structures. We mix two settings in our data collection process. In the first setting, we reward the top (in terms of numbers of examples) workers every 200 examples. In the second setting, the workers get bonuses based on their productivity (measured as the number of examples per hour).

奖励结构。 为激励众包工人更高效地产出更高质量的数据,我们沿用 Yang et al. (2018) 的做法,采用奖励结构。我们在数据收集过程中混合使用两种设置。第一种设置中,我们每 200 条样本奖励(按样本数量计的)排名靠前的工人。第二种设置中,工人根据生产率(以每小时完成的样本数衡量)获得奖金。

A.3 Crowd Worker Interface

A.3 众包工人界面

Our crowd worker interface is based on ParlAI (Miller et al., 2017), an open-source project that facilitates the development of dialog systems and data collection with a dialog interface. We adapt ParlAI for collecting question answer pairs by converting the collection workflow into a system-oriented dialog. This allows us to have more control over the turkers input, as well as provide turkers with in-the-loop feedbacks or helpful hints to help Turkers finish the task, and therefore speed up the collection process.

我们的众包工人界面基于 ParlAI(Miller et al., 2017),这是一个开源项目,便于用对话界面开发对话系统和收集数据。我们把收集流程改成面向系统的对话,从而把 ParlAI 改造成用于收集问答对。这样我们可以对众包工人的输入有更多控制,也能在流程中提供反馈或有用提示,帮助工人完成任务,从而加快收集速度。

Please see Figure 4 for an example of the worker interface during data collection.

数据收集过程中工人界面的例子请见图 4。

Screenshot of the HotpotQA Mechanical Turk worker interface

Figure 4: Screenshot of our worker interface on Amazon Mechanical Turk.

图 4:我们在 Amazon Mechanical Turk 上的工人界面截图。

Histogram of question lengths in HotpotQA

Figure 5: Distribution of lengths of questions in HOTPOTQA.

图 5:HOTPOTQA 中问题长度的分布。

B Further Data Analysis

B 进一步的数据分析

To further look into the diversity of the data in HOTPOTQA, we further visualized the distribution of question lengths in the dataset in Figure 5. Besides being diverse in terms of types as is show in the main text, questions also vary greatly in length, indicating different levels of complexity and details covered.

为进一步观察 HOTPOTQA 数据的多样性,我们还在图 5 中可视化了数据集中问题长度的分布。除了正文所示在类型上多样之外,问题的长度也差异很大,表明它们覆盖了不同程度的复杂性和细节。

C Full Wiki Setting Details

C 全文维基设置的细节

C.1 The Inverted Index Filtering Strategy

C.1 倒排索引过滤策略

In the full wiki setting, we adopt an efficient inverted-index-based filtering strategy for preliminary candidate paragraph retrieval. We provide details in Algorithm 2, where we set the control threshold N=5000N = 5000N=5000 in our experiments. For some of the question qqq, its corresponding gold paragraphs may not be included in the output candidate pool ScandS_{\mathrm{cand}}Scand​, we set such missing gold paragraph’s rank as ∣Scand∣+1|S_{\mathrm{cand}}| + 1∣Scand​∣+1 during the evaluation, so MAP and Mean Rank reported in this paper are upper bounds of their true values.

在全文维基设置中,我们采用一种基于倒排索引的高效过滤策略,做初步的候选段落检索。细节见算法 2,实验中控制阈值设为 N=5000N = 5000N=5000。对某些问题 qqq,其对应的金标段落可能不会被包含在输出候选池 ScandS_{\mathrm{cand}}Scand​ 中,评估时我们把这类缺失金标段落的排名设为 ∣Scand∣+1|S_{\mathrm{cand}}| + 1∣Scand​∣+1,因此本文报告的 MAP 和 Mean Rank 是其真实值的上界。

Algorithm 2 Inverted Index Filtering Strategy

Input: question text qqq, control threshold NNN, ngram-to-Wikidoc inverted index DDD

Inintialize:
Extract unigram + bigram set rqr_qrq​ from qqq
Ncand=+∞N_{\mathrm{cand}} = +\inftyNcand​=+∞
Cgram=0C_{\mathrm{gram}} = 0Cgram​=0

while Ncands>NN_{\mathrm{cands}} > NNcands​>N do

  Cgram=Cgram+1C_{\mathrm{gram}} = C_{\mathrm{gram}} + 1Cgram​=Cgram​+1

  Set SoverlapS_{\mathrm{overlap}}Soverlap​ to be an empty dictionary

  for w∈rqw \in r_qw∈rq​ do

    for d∈D[w]d \in D[w]d∈D[w] do

      if ddd not in SoverlapS_{\mathrm{overlap}}Soverlap​ then

        Soverlap[d]=1S_{\mathrm{overlap}}[d] = 1Soverlap​[d]=1

      else

        Soverlap[d]=Soverlap[d]+1S_{\mathrm{overlap}}[d] = S_{\mathrm{overlap}}[d] + 1Soverlap​[d]=Soverlap​[d]+1

      end if

    end for

  end for

  Scand=∅S_{\mathrm{cand}} = \emptysetScand​=∅

  for ddd in SoverlapS_{\mathrm{overlap}}Soverlap​ do

    if Soverlap[d]≥CgramS_{\mathrm{overlap}}[d] \ge C_{\mathrm{gram}}Soverlap​[d]≥Cgram​ then

      Scand=Scand∪{d}S_{\mathrm{cand}} = S_{\mathrm{cand}} \cup \{d\}Scand​=Scand​∪{d}

    end if

  end for

  Ncands=∣Scand∣N_{\mathrm{cands}} = |S_{\mathrm{cand}}|Ncands​=∣Scand​∣

end while

return ScandS_{\mathrm{cand}}Scand​

算法 2 倒排索引过滤策略

输入: 问题文本 qqq,控制阈值 NNN,n 元语法到维基文档的倒排索引 DDD

Inintialize:
从 qqq 中抽取 unigram + bigram 集合 rqr_qrq​
Ncand=+∞N_{\mathrm{cand}} = +\inftyNcand​=+∞
Cgram=0C_{\mathrm{gram}} = 0Cgram​=0

while Ncands>NN_{\mathrm{cands}} > NNcands​>N do

  Cgram=Cgram+1C_{\mathrm{gram}} = C_{\mathrm{gram}} + 1Cgram​=Cgram​+1

  把 SoverlapS_{\mathrm{overlap}}Soverlap​ 设为空字典

  for w∈rqw \in r_qw∈rq​ do

    for d∈D[w]d \in D[w]d∈D[w] do

      if ddd 不在 SoverlapS_{\mathrm{overlap}}Soverlap​ 中 then

        Soverlap[d]=1S_{\mathrm{overlap}}[d] = 1Soverlap​[d]=1

      else

        Soverlap[d]=Soverlap[d]+1S_{\mathrm{overlap}}[d] = S_{\mathrm{overlap}}[d] + 1Soverlap​[d]=Soverlap​[d]+1

      end if

    end for

  end for

  Scand=∅S_{\mathrm{cand}} = \emptysetScand​=∅

  for ddd in SoverlapS_{\mathrm{overlap}}Soverlap​ do

    if Soverlap[d]≥CgramS_{\mathrm{overlap}}[d] \ge C_{\mathrm{gram}}Soverlap​[d]≥Cgram​ then

      Scand=Scand∪{d}S_{\mathrm{cand}} = S_{\mathrm{cand}} \cup \{d\}Scand​=Scand​∪{d}

    end if

  end for

  Ncands=∣Scand∣N_{\mathrm{cands}} = |S_{\mathrm{cand}}|Ncands​=∣Scand​∣

end while

return ScandS_{\mathrm{cand}}Scand​

C.2 Compare train-medium Split to Hard Ones

C.2 比较 train-medium 划分与困难样本

Table 9 shows the comparison between train-medium split and hard examples like dev and test under retrieval metrics in full wiki setting. As we can see, the performance gap between train-medium split and its dev/test is close, which implies that train-medium split has a similar level of difficulty as hard examples under the full wiki setting in which a retrieval model is necessary as the first processing step.

表 9 给出了 train-medium 划分与开发集、测试集这类困难样本,在全文维基设置下检索指标上的比较。可以看到,train-medium 划分与其开发/测试集之间的表现差距很小,这意味着在必须先用检索模型作为第一步的全文维基设置中,train-medium 划分与困难样本具有相近的难度。

Table 9: Retrieval performance comparison on full wiki setting for train-medium, dev and test with 1,000 random samples each. MAP and are in %. Mean Rank averages over retrieval ranks of two gold paragraphs. CorAns Rank refers to the rank of the gold paragraph containing the answer.

Set MAP Mean Rank CorAns Rank
train-medium 41.89 288.19 82.76
dev 42.79 304.30 97.93
test 45.92 286.20 74.85

表 9: 在全文维基设置上,对 train-medium、开发集和测试集各 1,000 条随机样本的检索表现比较。MAP 以百分比计。Mean Rank 是对两个金标段落检索排名的平均。CorAns Rank 指包含答案的那一段金标段落的排名。

集合 MAP Mean Rank CorAns Rank
train-medium 41.89 288.19 82.76
开发 42.79 304.30 97.93
测试 45.92 286.20 74.85