Beyond Accuracy: Behavioral Testing of NLP Models with CheckList

Marco Tulio Ribeiro¹ Tongshuang Wu² Carlos Guestrin² Sameer Singh³

¹Microsoft Research ²University of Washington ³University of California, Irvine

marcotcr@gmail.com {wtshuang,guestrin}@cs.uw.edu sameer@uci.edu

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912
July 5 - 10, 2020. ©2020 Association for Computational Linguistics

Abstract

摘要

Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors. Inspired by principles of behavioral testing in software engineering, we introduce CheckList, a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation, as well as a software tool to generate a large and diverse number of test cases quickly. We illustrate the utility of CheckList with tests for three tasks, identifying critical failures in both commercial and state-of-art models. In a user study, a team responsible for a commercial sentiment analysis model found new and actionable bugs in an extensively tested model. In another user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.

尽管用留出集准确率来衡量泛化，一直是主要做法，但它常常高估 NLP 模型的表现；而其他评测路径，要么只针对单个任务，要么只盯住某些特定行为。受软件工程中行为测试（behavioral testing）原则的启发，我们提出 CheckList：一套不依赖具体任务的 NLP 模型测试方法。它包含一张由通用语言能力（capabilities）和测试类型（test types）构成的矩阵，方便系统地构思测试，并配有一套软件工具，能迅速生成数量大、种类多的用例。我们在三个任务上用 CheckList 做了测试，在商业模型与当时最好的研究模型里都找到了严重失败。在一项用户研究中，负责某商业情感分析模型的团队，在已经过充分测试的模型上又发现了此前不知道、且可以着手修复的缺陷。在另一项用户研究中，使用 CheckList 的 NLP 从业者写出的测试数量约为未使用者的两倍，发现的缺陷则接近三倍。

1 Introduction

1 引言

One of the primary goals of training NLP models is generalization. Since testing “in the wild” is expensive and does not allow for fast iterations, the standard paradigm for evaluation is using train-validation-test splits to estimate the accuracy of the model, including the use of leader boards to track progress on a task (Rajpurkar et al., 2016). While performance on held-out data is a useful indicator, held-out datasets are often not comprehensive, and contain the same biases as the training data (Rajpurkar et al., 2018), such that real-world performance may be overestimated (Patel et al., 2008; Recht et al., 2019). Further, by summarizing the performance as a single aggregate statistic, it becomes difficult to figure out where the model is failing, and how to fix it (Wu et al., 2019).

训练 NLP 模型的一个主要目标是泛化。由于在真实环境里测试成本高、也不利于快速迭代，评测的标准做法是用训练–验证–测试划分来估计模型准确率，并用排行榜跟踪某一任务上的进展（Rajpurkar et al., 2016）。留出集上的表现是有用的信号，但留出数据往往并不全面，而且还带着与训练数据相同的偏差（Rajpurkar et al., 2018），真实场景中的表现因此可能被高估（Patel et al., 2008; Recht et al., 2019）。再者，把表现收成一个总分，就很难看清模型败在哪里、该如何修复（Wu et al., 2019）。

A number of additional evaluation approaches have been proposed, such as evaluating robustness to noise (Belinkov and Bisk, 2018; Rychalska et al., 2019) or adversarial changes (Ribeiro et al., 2018; Iyyer et al., 2018), fairness (Prabhakaran et al., 2019), logical consistency (Ribeiro et al., 2019), explanations (Ribeiro et al., 2016), diagnostic datasets (Wang et al., 2019b), and interactive error analysis (Wu et al., 2019). However, these approaches focus either on individual tasks such as Question Answering or Natural Language Inference, or on a few capabilities (e.g. robustness), and thus do not provide comprehensive guidance on how to evaluate models. Software engineering research, on the other hand, has proposed a variety of paradigms and tools for testing complex software systems. In particular, “behavioral testing” (also known as black-box testing) is concerned with testing different capabilities of a system by validating the input-output behavior, without any knowledge of the internal structure (Beizer, 1995). While there are clear similarities, many insights from software engineering are yet to be applied to NLP models.

人们还提出了不少其他评测方法，例如检验对噪声（Belinkov and Bisk, 2018; Rychalska et al., 2019）或对抗扰动（Ribeiro et al., 2018; Iyyer et al., 2018）的鲁棒性，以及公平性（Prabhakaran et al., 2019）、逻辑一致性（Ribeiro et al., 2019）、解释（Ribeiro et al., 2016）、诊断数据集（Wang et al., 2019b）和交互式错误分析（Wu et al., 2019）。不过，这些方法要么只针对问答或自然语言推理这类单个任务，要么只覆盖少数能力（例如鲁棒性），并不能系统说明该如何评测模型。软件工程研究则已经为测试复杂软件系统提出了多种范式和工具。其中，“行为测试”（也称黑盒测试）关心的是：在不了解内部结构的前提下，通过核验输入–输出行为，来测试系统的不同能力（Beizer, 1995）。两边显然有相似之处，但软件工程里的许多经验，尚未被用到 NLP 模型上。

In this work, we propose CheckList, a new evaluation methodology and accompanying tool¹ for comprehensive behavioral testing of NLP models. CheckList guides users in what to test, by providing a list of linguistic capabilities, which are applicable to most tasks. To break down potential capability failures into specific behaviors, CheckList introduces different test types, such as prediction invariance in the presence of certain perturbations, or performance on a set of “sanity checks.” Finally, our implementation of CheckList includes multiple abstractions that help users generate large numbers of test cases easily, such as templates, lexicons, general-purpose perturbations, visualizations, and context-aware suggestions.

本文提出 CheckList：一套新的评测方法，并配有相应工具¹，用于对 NLP 模型做全面的行为测试。CheckList 给出一份适用于大多数任务的语言能力清单，引导使用者决定测什么。为了把可能的能力缺陷拆成具体行为，它还引入不同的测试类型，例如在某些扰动下预测应保持不变，或在一组“健全性检查”上的表现。最后，我们实现的 CheckList 提供多种抽象，方便一次生成大量用例，包括模板、词表、通用扰动、可视化，以及结合上下文的建议。

¹https://github.com/marcotcr/checklist

Figure 1: CheckListing a commercial sentiment analysis model ([G]). Tests are structured as a conceptual matrix with capabilities as rows and test types as columns (examples of each type in A, B and C).

图 1：对一个商业情感分析模型（[G]，即后文的 Google Cloud Natural Language）做 CheckList 测试。测试被组织成概念矩阵：能力为行，测试类型为列（A、B、C 各给出一类示例）。官方 PDF 用图标标识各系统，下文统一写为 [MS]（Microsoft）、[G]（Google）、[AZ]（Amazon）、[BERT]（BERT）、RoB（RoBERTa-base）。

As an example, we CheckList a commercial sentiment analysis model in Figure 1. Potential tests are structured as a conceptual matrix, with capabilities as rows and test types as columns. As a test of the model’s Negation capability, we use a Minimum Functionality test (MFT), i.e. simple test cases designed to target a specific behavior (Figure 1A). We generate a large number of simple examples filling in a template (“I {NEGATION} {POS_VERB} the {THING}.”) with pre-built lexicons, and compute the model’s failure rate on such examples. Named entity recognition (NER) is another capability, tested in Figure 1B with an Invariance test (INV) – perturbations that should not change the output of the model. In this case, changing location names should not change sentiment. In Figure 1C, we test the model’s Vocabulary with a Directional Expectation test (DIR) – perturbations to the input with known expected results – adding negative phrases and checking that sentiment does not become more positive. As these examples indicate, the matrix works as a guide, prompting users to test each capability with different test types.

图 1 给出一个例子：我们对一个商业情感分析模型做 CheckList。备选测试被组织成概念矩阵，能力为行，测试类型为列。要测模型的否定（Negation）能力，我们使用最小功能测试（Minimum Functionality test，MFT），也就是一组针对单一行为的简单用例（图 1A）。我们用事先准备好的词表填入模板（“I {NEGATION} {POS_VERB} the {THING}.”），生成大量简单例子，并计算模型在这些例子上的失败率。命名实体识别（NER）是另一种能力，图 1B 用不变性测试（Invariance test，INV）来测——即不应改变模型输出的扰动。这里，更换地点名称不应改变情感。图 1C 则用方向期望测试（Directional Expectation test，DIR）来测词汇能力：对输入施加结果方向已知的扰动，例如追加贬义短语后，情感不应变得更正面。这些例子说明，这张矩阵起着指南的作用，提醒使用者对每种能力搭配不同的测试类型。

We demonstrate the usefulness and generality of CheckList via instantiation on three NLP tasks: sentiment analysis (Sentiment), duplicate question detection (QQP; Wang et al., 2019b), and machine comprehension (MC; Rajpurkar et al., 2016). While traditional benchmarks indicate that models on these tasks are as accurate as humans, CheckList reveals a variety of severe bugs, where commercial and research models do not effectively handle basic linguistic phenomena such as negation, named entities, coreferences, semantic role labeling, etc, as they pertain to each task. Further, CheckList is easy to use and provides immediate value – in a user study, the team responsible for a commercial sentiment analysis model discovered many new and actionable bugs in their own model, even though it had been extensively tested and used by customers. In an additional user study, we found that NLP practitioners with CheckList generated more than twice as many tests (each test containing an order of magnitude more examples), and uncovered almost three times as many bugs, compared to users without CheckList.

我们把 CheckList 落到三个 NLP 任务上，以说明它的用处和通用性：情感分析（Sentiment）、重复问题检测（QQP；Wang et al., 2019b）以及机器阅读理解（MC；Rajpurkar et al., 2016）。传统基准显示这些任务上的模型已与人类同样准确，但 CheckList 揭示了多种严重缺陷：商业模型与研究模型都没能有效处理与各任务相关的基本语言现象，例如否定、命名实体、共指、语义角色标注等。此外，CheckList 不难上手，也能立刻带来收获——在一项用户研究中，负责某商业情感分析模型的团队在自己的模型上发现了许多此前不知道、且可以着手修复的缺陷，尽管该模型已经过充分测试并为客户所用。在另一项用户研究中，我们看到：与没有 CheckList 的使用者相比，配有 CheckList 的 NLP 从业者写出的测试数量超过两倍（每项测试包含的用例还多出一个数量级），发现的缺陷则接近三倍。

2 CheckList

Conceptually, users “CheckList” a model by filling out cells in a matrix (Figure 1), each cell potentially containing multiple tests. In this section, we go into more detail on the rows (capabilities), columns (test types), and how to fill the cells (tests). CheckList applies the behavioral testing principle of “decoupling testing from implementation” by treating the model as a black box, which allows for comparison of different models trained on different data, or third-party models where access to training data or model structure is not granted.

概念上，使用者“CheckList”一个模型，就是去填一张矩阵里的格子（图 1），每个格子里可以放多项测试。本节更细地说明行（能力）、列（测试类型），以及如何填格子（测试）。CheckList 采用行为测试中“把测试与实现分开”的原则，把模型当作黑盒，这样既能比较在不同数据上训练出的模型，也能评测无法获得训练数据或模型结构的第三方接口。

2.1 Capabilities

2.1 能力

While testing individual components is a common practice in software engineering, modern NLP models are rarely built one component at a time. Instead, CheckList encourages users to consider how different natural language capabilities are manifested on the task at hand, and to create tests to evaluate the model on each of these capabilities. For example, the Vocabulary+POS capability pertains to whether a model has the necessary vocabulary, and whether it can appropriately handle the impact of words with different parts of speech on the task. For Sentiment, we may want to check if the model is able to identify words that carry positive, negative, or neutral sentiment, by verifying how it behaves on examples like “This was a good flight.” For QQP, we might want the model to understand when modifiers differentiate questions, e.g. accredited in (“Is John a teacher?”, “Is John an accredited teacher?”). For MC, the model should be able to relate comparatives and superlatives, e.g. (Context: “Mary is smarter than John.”, Q: “Who is the smartest kid?”, A: “Mary”).

在软件工程里，逐个测试组件是常见做法，但现代 NLP 模型很少是一个组件一个组件搭起来的。CheckList 转而鼓励使用者思考：不同的自然语言能力在当前任务上会怎样体现，并为每一种能力设计测试。例如，词汇与词性（Vocabulary+POS）能力关心的是：模型是否具备必要的词汇，以及能否恰当处理不同词性的词对任务的影响。对情感分析，我们可能想检查模型能否识别带正面、负面或中性情感的词，例如看它在 “This was a good flight.” 这类例子上的行为。对 QQP，我们可能希望模型理解修饰语何时会把问题区分开，例如 accredited 在（“Is John a teacher?”, “Is John an accredited teacher?”）中的作用。对阅读理解，模型应能把比较级和最高级联系起来，例如（上下文：“Mary is smarter than John.”，问：“Who is the smartest kid?”，答：“Mary”）。

We suggest that users consider at least the following capabilities: Vocabulary+POS (important words or word types for the task), Taxonomy (synonyms, antonyms, etc), Robustness (to typos, irrelevant changes, etc), NER (appropriately understanding named entities), Fairness, Temporal (understanding order of events), Negation, Coreference, Semantic Role Labeling (understanding roles such as agent, object, etc), and Logic (ability to handle symmetry, consistency, and conjunctions). We will provide examples of how these capabilities can be tested in Section 3 (Tables 1, 2, and 3). This listing of capabilities is not exhaustive, but a starting point for users, who should also come up with additional capabilities that are specific to their task or domain.

我们建议使用者至少考虑以下能力：词汇与词性（任务中重要的词或词类）、分类体系（同义词、反义词等）、鲁棒性（对拼写错误、无关改动等）、NER（恰当理解命名实体）、公平性、时间顺序（理解事件先后）、否定、共指、语义角色标注（理解施事、客体等角色），以及逻辑（处理对称、一致性和合取的能力）。第 3 节（表 1、表 2 和表 3）会给出如何测试这些能力的例子。这份清单并不是穷尽的，只是一个起点；使用者还应补充与自己任务或领域相关的其他能力。

2.2 Test Types

2.2 测试类型

We prompt users to evaluate each capability with three different test types (when possible): Minimum Functionality tests, Invariance, and Directional Expectation tests (the columns in the matrix).

我们提示使用者尽可能用三类测试来评测每一种能力：最小功能测试、不变性测试和方向期望测试（也就是矩阵的各列）。

A Minimum Functionality test (MFT), inspired by unit tests in software engineering, is a collection of simple examples (and labels) to check a behavior within a capability. MFTs are similar to creating small and focused testing datasets, and are particularly useful for detecting when models use shortcuts to handle complex inputs without actually mastering the capability. The Vocabulary+POS examples in the previous section are all MFTs.

最小功能测试（MFT）受软件工程中单元测试的启发，是一组用来检查某项能力中某一行为的简单例子（及标签）。MFT 类似于制作规模小、目标集中的测试集，尤其适合发现模型是否靠捷径处理复杂输入，而并没有真正掌握该能力。上一节中词汇与词性的例子，全部都是 MFT。

We also introduce two additional test types inspired by software metamorphic tests (Segura et al., 2016). An Invariance test (INV) is when we apply label-preserving perturbations to inputs and expect the model prediction to remain the same. Different perturbation functions are needed for different capabilities, e.g. changing location names for the NER capability for Sentiment (Figure 1B), or introducing typos to test the Robustness capability. A Directional Expectation test (DIR) is similar, except that the label is expected to change in a certain way. For example, we expect that sentiment will not become more positive if we add “You are lame.” to the end of tweets directed at an airline (Figure 1C). The expectation may also be a target label, e.g. replacing locations in only one of the questions in QQP, such as (“How many people are there in England?”, “What is the population of England → Turkey?”), ensures that the questions are not duplicates. INVs and DIRs allow us to test models on unlabeled data – they test behaviors that do not rely on ground truth labels, but rather on relationships between predictions after perturbations are applied (invariance, monotonicity, etc).

我们还引入另外两类测试，灵感来自软件中的蜕变测试（Segura et al., 2016）。不变性测试（INV）是对输入施加应保持标签不变的扰动，并期望模型预测不变。不同能力需要不同的扰动函数，例如在情感分析的 NER 能力上更换地点名（图 1B），或引入拼写错误来测鲁棒性。方向期望测试（DIR）与之类似，只是标签被期望按某一方向变化。例如，若在写给航空公司的推文末尾加上 “You are lame.”，我们期望情感不会变得更正面（图 1C）。期望也可以是一个目标标签：例如只替换 QQP 中一个问题里的地点，如（“How many people are there in England?”, “What is the population of England → Turkey?”），即可保证这两个问题不是重复的。INV 和 DIR 让我们能在未标注数据上测试模型——它们检验的不是依赖标准答案的行为，而是扰动施加之后预测之间的关系（不变、单调等）。

2.3 Generating Test Cases at Scale

2.3 规模化生成用例

Users can create test cases from scratch, or by perturbing an existing dataset. Starting from scratch makes it easier to create a small number of high-quality test cases for specific phenomena that may be underrepresented or confounded in the original dataset. Writing from scratch, however, requires significant creativity and effort, often leading to tests that have low coverage or are expensive and time-consuming to produce. Perturbation functions are harder to craft, but generate many test cases at once. To support both these cases, we provide a variety of abstractions that scale up test creation from scratch and make perturbations easier to craft.

使用者可以从零编写用例，也可以扰动已有数据集。从零开始，更容易为原数据集中可能代表性不足或相互混淆的特定现象，写出少量高质量用例。但从零写需要不少创意和功夫，结果往往是覆盖面不够，或制作起来又贵又慢。扰动函数更难设计，却能一次生成许多用例。为同时支持这两种做法，我们提供多种抽象：既放大从零写测试的规模，也让扰动更好写。

Templates. Test cases and perturbations can often be generalized into a template, to test the model on a more diverse set of inputs. In Figure 1 we generalized “I didn’t love the food.” with the template “I {NEGATION} {POS_VERB} the {THING}.”, where {NEGATION} = {didn’t, can’t say I, ...}, {POS_VERB} = {love, like, ...}, {THING} = {food, flight, service, ...}, and generated all test cases with a Cartesian product. A more diverse set of inputs is particularly helpful when a small set of test cases could miss a failure, e.g. if a model works for some forms of negation but not others.

模板。 用例和扰动常常可以推广成模板，以便在更多样的输入上测试模型。图 1 里，我们把 “I didn’t love the food.” 推广成模板 “I {NEGATION} {POS_VERB} the {THING}.”，其中 {NEGATION} = {didn’t, can’t say I, ...}，{POS_VERB} = {love, like, ...}，{THING} = {food, flight, service, ...}，再用笛卡尔积生成全部用例。当少量用例可能漏掉失败时，更多样的输入尤其有用，例如模型对某些否定形式有效、对另一些却无效。

Figure 2: Templating with masked language models. “I really {mask} the flight.” yields verbs that the user can interactively filter into positive, negative, and neutral fill-in lists.

图 2：用掩码语言模型做模板填空。“I really {mask} the flight.” 会给出一批动词，使用者可交互地把它们筛进正面、负面和中性填空列表。

Expanding Templates. While templates help scale up test case generation, they still rely on the user’s creativity to create fill-in values for each placeholder (e.g. positive verbs for {POS_VERB}). We provide users with an abstraction where they mask part of a template and get masked language model (RoBERTa (Liu et al., 2019) in our case) suggestions for fill-ins, e.g. “I really {mask} the flight.” yields {enjoyed, liked, loved, regret, ...}, which the user can filter into positive, negative, and neutral fill-in lists and later reuse across multiple tests (Figure 2). Sometimes RoBERTa suggestions can be used without filtering, e.g. “This is a good {mask}” yields multiple nouns that don’t need filtering. They can also be used in perturbations, e.g. replacing neutral words like that or the for other words in context (Vocabulary+POS INV examples in Table 1). RoBERTa suggestions can be combined with WordNet categories (synonyms, antonyms, etc), e.g. such that only context-appropriate synonyms get selected in a perturbation. We also provide additional common fill-ins for general-purpose categories, such as Named Entities (common male and female first/last names, cities, countries) and protected group adjectives (nationalities, religions, gender and sexuality, etc).

扩展模板。 模板有助于放大用例生成的规模，但每个占位符的填空值（例如给 {POS_VERB} 准备正面动词）仍要靠使用者自己想。我们提供这样一种抽象：把模板的一部分掩掉，由掩码语言模型（本文中是 RoBERTa（Liu et al., 2019））给出填空建议，例如 “I really {mask} the flight.” 会得到 {enjoyed, liked, loved, regret, ...}，使用者可以把它们筛进正面、负面和中性列表，并在多项测试中反复使用（图 2）。有时 RoBERTa 的建议不必筛选就能用，例如 “This is a good {mask}” 会给出多个无需筛选的名词。它们也可用于扰动，例如把 that 或 the 这类中性词替换成上下文中的其他词（见表 1 中词汇与词性的 INV 例子）。RoBERTa 建议还可以和 WordNet 的类别（同义词、反义词等）合用，例如在扰动中只保留适合上下文的同义词。我们还为一些通用类别提供常用填空，例如命名实体（常见的男性和女性名/姓、城市、国家）以及受保护群体形容词（国籍、宗教、性别与性取向等）。

Open source. We release an implementation of CheckList at https://github.com/marcotcr/checklist. In addition to templating features and mask language model suggestions, it contains various visualizations, abstractions for writing test expectations (e.g. monotonicity) and perturbations, saving/sharing tests and test suites such that tests can be reused with different models and by different teams, and general-purpose perturbations such as char swaps (simulating typos), contractions, name and location changes (for NER tests), etc.

开源。 我们在 https://github.com/marcotcr/checklist 发布了 CheckList 的实现。除了模板功能和掩码语言模型建议，它还包括多种可视化、用于编写测试期望（例如单调性）和扰动的抽象、保存/分享测试与测试套件以便在不同模型和不同团队间复用，以及字符对调（模拟拼写错误）、缩约、姓名与地点更换（用于 NER 测试）等通用扰动。

3 Testing SOTA models with CheckList

3 用 CheckList 测试当时最好的模型

We CheckList the following commercial Sentiment analysis models via their paid APIs²: Microsoft’s Text Analytics ([MS]), Google Cloud’s Natural Language ([G]), and Amazon’s Comprehend ([AZ]). We also CheckList BERT-base ([BERT]) and RoBERTa-base (RoB) (Liu et al., 2019) finetuned on SST-2³ (acc: 92.7% and 94.8%) and on the QQP dataset (acc: 91.1% and 91.3%). For MC, we use a pretrained BERT-large finetuned on SQuAD (Wolf et al., 2019), achieving 93.2 F1. All the tests presented here are part of the open-source release, and can be easily replicated and applied to new models.

我们通过付费 API²，对以下商业情感分析模型做了 CheckList：Microsoft 的 Text Analytics（[MS]）、Google Cloud 的 Natural Language（[G]），以及 Amazon 的 Comprehend（[AZ]）。我们还对在 SST-2³ 上微调的 BERT-base（[BERT]）和 RoBERTa-base（RoB）（Liu et al., 2019）（准确率分别为 92.7% 和 94.8%），以及在 QQP 数据集上微调的同一对模型（准确率分别为 91.1% 和 91.3%）做了 CheckList。阅读理解方面，我们使用在 SQuAD 上微调的预训练 BERT-large（Wolf et al., 2019），F1 为 93.2。这里给出的全部测试都包含在开源发布中，可以方便地复现并用于新模型。

²From 11/2019, but obtained similar results from 04/2020.

²接口取自 2019 年 11 月，但 2020 年 4 月得到的结果相近。

³Predictions with probability of positive sentiment in the $[1/3, 2/3]$ range are considered neutral.

³正面情感概率落在 $[1/3, 2/3]$ 区间内的预测视为中性。

Table 1: A selection of tests for sentiment analysis. All examples (right) are failures of at least one model.

表 1：情感分析测试选摘。右侧全部例子都是至少一个模型的失败用例。各列依次为 [MS]（Microsoft）、[G]（Google）、[AZ]（Amazon）、[BERT]（BERT-base）和 RoB（RoBERTa-base）。

Sentiment Analysis. Since social media is listed as a use case for these commercial models, we test on that domain and use a dataset of unlabeled airline tweets for INV⁴ and DIR perturbation tests. We create tests for a broad range of capabilities, and present a subset with high failure rates in Table 1. The Vocab.+POS MFTs are sanity checks, where we expect models to appropriately handle common neutral or sentiment-laden words. [BERT] and RoB do poorly on neutral predictions (they were trained on binary labels only). Surprisingly, [G] and [AZ] fail (7.6% and 4.8%) on sentences that are clearly neutral, with [G] also failing (15%) on non-neutral sanity checks (e.g. “I like this seat.”). In the DIR tests, the sentiment scores predicted by [MS] and [G] frequently (12.6% and 12.4%) go down considerably when clearly positive phrases (e.g. “You are extraordinary.”) are added, or up ([G]: 34.6%) for negative phrases (e.g. “You are lame.”).

情感分析。 这些商业模型把社交媒体列为使用场景，因此我们在该领域上测试，并用一组未标注的航空公司推文做 INV⁴ 和 DIR 扰动测试。我们为多种能力设计了测试，并在表 1 中给出失败率较高的子集。词汇与词性的 MFT 是健全性检查：我们期望模型能恰当处理常见的中性词或带情感的词。[BERT] 和 RoB 在中性预测上表现差（它们只在二元标签上训练过）。出人意料的是，[G] 和 [AZ] 在明显中性的句子上也会失败（分别为 7.6% 和 4.8%），[G] 在非中性的健全性检查上同样失败（15%，例如 “I like this seat.”）。在 DIR 测试中，给句子加上明显正面的短语（例如 “You are extraordinary.”）后，[MS] 和 [G] 给出的情感分数经常（12.6% 和 12.4%）明显下降；加上贬义短语（例如 “You are lame.”）后，分数又会上升（[G]：34.6%）。

⁴For all the INV tests, models fail whenever their prediction changes and the probability changes by more than 0.1.

⁴对所有 INV 测试，只要预测发生变化，且概率变化超过 0.1，即计为失败。

All models are sensitive to addition of random (not adversarial) shortened URLs or Twitter handles (e.g. 24.8% of [AZ] predictions change), and to name changes, such as locations ([G]: 20.8%, [AZ]: 14.8%) or person names ([G]: 15.1%, [AZ]: 9.1%). None of the models do well in tests for the Temporal, Negation, and SRL capabilities. Failures on negations as simple as “The food is not poor.” are particularly notable, e.g. [G] (54.2%) and [AZ] (29.4%). The failure rate is near 100% for all commercial models when the negation comes at the end of the sentence (e.g “I thought the plane would be awful, but it wasn’t.”), or with neutral content between the negation and the sentiment-laden word.

所有模型都对随机（而非对抗性）加入的短链接或 Twitter 账号敏感（例如 [AZ] 有 24.8% 的预测发生变化），也对名称更换敏感，例如地点（[G]：20.8%，[AZ]：14.8%）或人名（[G]：15.1%，[AZ]：9.1%）。在时间顺序、否定和语义角色标注这些能力的测试上，没有一个模型表现好。像 “The food is not poor.” 这样简单的否定就会失败，尤其值得注意，例如 [G]（54.2%）和 [AZ]（29.4%）。当否定出现在句末（例如 “I thought the plane would be awful, but it wasn’t.”），或否定与带情感的词之间夹着中性内容时，所有商业模型的失败率都接近 100%。

Commercial models do not fail simple Fairness sanity checks such as “I am a black woman.” (template: “I am a {PROTECTED} {NOUN}.”), always predicting them as neutral. Similar to software engineering, absence of test failure does not imply that these models are fair – just that they are not unfair enough to fail these simple tests. On the other hand, [BERT] always predicts negative when {PROTECTED} is black, atheist, gay, and lesbian, while predicting positive for Asian, straight, etc. With the exception of tests that depend on predicting “neutral”, [BERT] and RoB did better than all commercial models on almost every other test. This is a surprising result, since the commercial models list social media as a use case, and are under regular testing and improvement with customer feedback, while [BERT] and RoB are research models trained on the SST-2 dataset (movie reviews). Finally, [BERT] and RoB fail simple negation MFTs, even though they are fairly accurate (91.5%, 93.9%, respectively) on the subset of the SST-2 validation set that contains negation in some form (18% of instances). By isolating behaviors like this, our tests are thus able to evaluate capabilities more precisely, whereas performance on the original dataset can be misleading.

商业模型不会在 “I am a black woman.” 这类简单的公平性健全性检查上失败（模板为 “I am a {PROTECTED} {NOUN}.”），而是一律预测为中性。和软件工程里一样，测试没有失败并不意味着这些模型就是公平的——只说明它们还没有不公平到会在这些简单测试上失手。另一方面，当 {PROTECTED} 为 black、atheist、gay 和 lesbian 时，[BERT] 总是预测为负面；对 Asian、straight 等则预测为正面。除了依赖“中性”预测的测试外，[BERT] 和 RoB 在几乎所有其他测试上都优于全部商业模型。这一结果出人意料：商业模型把社交媒体列为使用场景，并且经常根据客户反馈测试和改进，而 [BERT] 和 RoB 只是在 SST-2 数据集（电影评论）上训练的研究模型。最后，[BERT] 和 RoB 会在简单的否定 MFT 上失败，尽管它们在 SST-2 验证集中带有某种否定的子集上（占 18% 的实例）准确率仍相当高（分别为 91.5% 和 93.9%）。把这类行为单独拆出来，我们的测试就能更精确地评测能力；而原数据集上的表现则可能造成误导。

Table 2: A selection of tests for Quora Question Pair. All examples (right) are failures of at least one model.

表 2：Quora 问题对（QQP）测试选摘。右侧全部例子都是至少一个模型的失败用例。两列分别为 [BERT] 与 RoB。$=$ 表示重复，$\neq$ 表示不重复。

Quora Question Pair. While [BERT] and RoB surpass human accuracy on QQP in benchmarks (Wang et al., 2019a), the subset of tests in Table 2 indicate that these models are far from solving the question paraphrase problem, and are likely relying on shortcuts for their high accuracy.

Quora 问题对。 尽管 [BERT] 和 RoB 在基准上已超过人类在 QQP 上的准确率（Wang et al., 2019a），表 2 中的测试子集表明，这些模型远未解决“问题是否同义转述”这一问题，其高准确率很可能依赖捷径。

Both models lack what seems to be crucial skills for the task: ignoring important modifiers on the Vocab. test, and lacking basic Taxonomy understanding, e.g. synonyms and antonyms of common words. Further, neither is robust to typos or simple paraphrases. The failure rates for the NER tests indicate that these models are relying on shortcuts such as anchoring on named entities too strongly instead of understanding named entities and their impact on whether questions are duplicates.

两个模型都缺少这项任务似乎必不可少的技能：在词汇测试中会忽略重要修饰语，也缺乏基本的分类体系理解，例如常见词的同义和反义。此外，两者对拼写错误或简单转述都不鲁棒。NER 测试的失败率表明，这些模型在依赖捷径，例如过分锚定命名实体，而不是理解实体本身、以及它们对两个问题是否重复的影响。

Surprisingly, the models often fail to make simple Temporal distinctions (e.g. is ≠ used to be and before ≠ after), and to distinguish between simple Coreferences (he ≠ she). In SRL tests, neither model is able to handle agent/predicate changes, or active/passive swaps. Finally, [BERT] and RoB change predictions 4.4% and 2.2% of the time when the question order is flipped, failing a basic task requirement (if q1 is a duplicate of q2, so is q2 of q1). They are also not consistent with Logical implications of their predictions, such as transitivity.

出人意料的是，模型还经常分不清简单的时间顺序差异（例如 is ≠ used to be，以及 before ≠ after），也分不清简单的共指（he ≠ she）。在语义角色标注测试中，两个模型都处理不了施事/谓词的改换，或主动/被动的对调。最后，把问题对对调后，[BERT] 和 RoB 仍分别有 4.4% 和 2.2% 的预测发生变化，没能满足一项基本任务要求（若 q1 与 q2 重复，则 q2 与 q1 也应重复）。它们对自己预测的逻辑推论也不一致，例如传递性。

Table 3: A selection of tests for Machine Comprehension.

表 3：机器阅读理解测试选摘。失败率以及最右列中的预测均来自 [BERT]（在 SQuAD 上微调的 BERT-large）。

Machine Comprehension. Vocab+POS tests in Table 3 show that [BERT] often fails to properly grasp intensity modifiers and comparisons/superlatives. It also fails on simple Taxonomy tests, such as matching properties (size, color, shape) to adjectives, distinguishing between animals-vehicles or jobs-nationalities, or comparisons involving antonyms. The model does not seem capable of handling short instances with Temporal concepts such as before, after, last, and first, or with simple examples of Negation, either in the question or in the context. It also does not seem to resolve basic Coreferences, and grasp simple subject/object or active/passive distinctions (SRL), all of which are critical to true comprehension. Finally, the model seems to have certain biases, e.g. for the simple negation template “{P1} is not a {PROF}, {P2} is.” as context, and “Who is a {PROF}?” as question, if we set {PROF} = doctor, {P1} to male names and {P2} to female names (e.g. “John is not a doctor, Mary is.”; “Who is a doctor?”), the model fails (picks the man as the doctor) 89.1% of the time. If the situation is reversed, the failure rate is only 3.2% (woman predicted as doctor). If {PROF} = secretary, it wrongly picks the man only 4.0% of the time, and the woman 60.5% of the time.

机器阅读理解。 表 3 中的词汇与词性测试显示，[BERT] 常常无法正确把握强度修饰语以及比较级/最高级。它也过不了简单的分类体系测试，例如把属性（大小、颜色、形状）匹配到形容词，区分动物与交通工具或职业与国籍，或涉及反义词的比较。对于带有 before、after、last、first 这类时间概念的短例子，或出现在问题或上下文中的简单否定，模型看起来都处理不好。它似乎也无法解析基本共指，或把握简单的主语/宾语、主动/被动区分（语义角色标注），而这些对真正的理解都至关重要。最后，模型还显得带有某些偏见。例如，用简单否定模板 “{P1} is not a {PROF}, {P2} is.” 作上下文，用 “Who is a {PROF}?” 作问题：若令 {PROF} = doctor，{P1} 为男性名、{P2} 为女性名（例如 “John is not a doctor, Mary is.”；“Who is a doctor?”），模型有 89.1% 的时候失败（把男性选成医生）。若情况对调，失败率只有 3.2%（把女性预测为医生）。若 {PROF} = secretary，它错误地选中男性的比例只有 4.0%，选中女性则为 60.5%。

Discussion. We applied the same process to very different tasks, and found that tests reveal interesting failures on a variety of task-relevant linguistic capabilities. While some tests are task specific (e.g. positive adjectives), the capabilities and test types are general; many can be applied across tasks, as is (e.g. testing Robustness with typos) or with minor variation (changing named entities yields different expectations depending on the task). This small selection of tests illustrates the benefits of systematic testing in addition to standard evaluation. These tasks may be considered “solved” based on benchmark accuracy results, but the tests highlight various areas of improvement – in particular, failure to demonstrate basic skills that are de facto needs for the task at hand (e.g. basic negation, agent/object distinction, etc). Even though some of these failures have been observed by others, such as typos (Belinkov and Bisk, 2018; Rychalska et al., 2019) and sensitivity to name changes (Prabhakaran et al., 2019), we believe the majority are not known to the community, and that comprehensive and structured testing will lead to avenues of improvement in these and other tasks.

讨论。 我们把同一套流程用到差异很大的任务上，发现测试能在多种与任务相关的语言能力上揭示有意思的失败。有些测试是任务特有的（例如正面形容词），但能力和测试类型是通用的；其中许多可以原样跨任务使用（例如用拼写错误测鲁棒性），或只需稍作改动（更换命名实体时，期望会随任务而变）。这一小部分测试说明：在标准评测之外，系统测试能带来额外收获。若只看基准准确率，这些任务或许会被视为已经“解决”，但测试标出了多处仍需改进的地方——尤其是模型没能表现出当前任务实际上必需的基本技能（例如简单否定、施事/客体区分等）。尽管其中一些失败已被其他人观察到，例如拼写错误（Belinkov and Bisk, 2018; Rychalska et al., 2019）和对改名的敏感（Prabhakaran et al., 2019），但我们相信大多数问题还不被社区所知；全面而有结构的测试，会为这些任务以及其他任务指出改进的方向。

4 User Evaluation

4 用户评估

The failures discovered in the previous section demonstrate the usefulness and flexibility of CheckList. In this section, we further verify that CheckList leads to insights both for users who already test their models carefully and for users with little or no experience in a task.

上一节发现的失败，已经说明 CheckList 有用且灵活。本节进一步核验：无论使用者已经在仔细测试自己的模型，还是对某项任务几乎没有经验，CheckList 都能带来新的认识。

4.1 CheckListing a Commercial System

4.1 对一个商业系统做 CheckList

We approached the team responsible for the general purpose sentiment analysis model sold as a service by Microsoft ([MS] on Table 1). Since it is a public-facing system, the model’s evaluation procedure is more comprehensive than research systems, including publicly available benchmark datasets as well as focused benchmarks built in-house (e.g. negations, emojis). Further, since the service is mature with a wide customer base, it has gone through many cycles of bug discovery (either internally or through customers) and subsequent fixes, after which new examples are added to the benchmarks. Our goal was to verify if CheckList would add value even in a situation like this, where models are already tested extensively with current practices.

我们联系了负责 Microsoft 对外售卖的通用情感分析模型的团队（表 1 中的 [MS]）。因为这是面向公众的系统，其评测流程比研究系统更全面，既包括公开基准数据集，也包括内部制作的专项基准（例如否定、emoji）。而且这项服务已经成熟、客户面广，经历过多轮缺陷发现（来自内部或客户）和随后的修复，新例子也会被加入基准。我们想确认的是：在模型已按现有做法充分测试的情况下，CheckList 是否仍能带来额外价值。

We invited the team for a CheckList session lasting approximately 5 hours. We presented CheckList (without presenting the tests we had already created), and asked them to use the methodology to test their own model. We helped them implement their tests, to reduce the additional cognitive burden of having to learn the software components of CheckList. The team brainstormed roughly 30 tests covering all capabilities, half of which were MFTs and the rest divided roughly equally between INVs and DIRs. Due to time constraints, we implemented about 20 of those tests. The tests covered many of the same functionalities we had tested ourselves (Section 3), often with different templates, but also ones we had not thought of. For example, they tested if the model handled sentiment coming from camel-cased twitter hashtags correctly (e.g. “#IHateYou”, “#ILoveYou”), implicit negation (e.g. “I wish it was good”), and others. Further, they proposed new capabilities for testing, e.g. handling different lengths (sentences vs paragraphs) and sentiment that depends on implicit expectations (e.g. “There was no {AC}” when {AC} is expected).

我们邀请该团队参加了一次约 5 小时的 CheckList 讨论。我们介绍了 CheckList（但没有出示自己已经写好的测试），请他们用这套方法测试自己的模型。为减轻额外学习软件组件的负担，由我们帮助实现他们的测试。团队头脑风暴出大约 30 项覆盖全部能力的测试，一半是 MFT，其余大致均分给 INV 和 DIR。受时间限制，我们实现了其中约 20 项。这些测试覆盖了我们自己测过的许多功能（第 3 节），模板往往不同，也有我们没想到的。例如，他们测试了模型能否正确处理来自驼峰式 Twitter 话题标签的情感（如 “#IHateYou”、“#ILoveYou”）、隐含否定（如 “I wish it was good”）等。他们还提出了新的待测能力，例如处理不同长度（句子与段落），以及取决于隐含预期的情感（例如在本应有 {AC} 的场合说 “There was no {AC}”）。

Qualitatively, the team stated that CheckList was very helpful: (1) they tested capabilities they had not considered, (2) they tested capabilities that they had considered but are not in the benchmarks, and (3) even capabilities for which they had benchmarks (e.g. negation) were tested much more thoroughly and systematically with CheckList. They discovered many previously unknown bugs, which they plan to fix in the next model iteration. Finally, they indicated that they would definitely incorporate CheckList into their development cycle, and requested access to our implementation. This session, coupled with the variety of bugs we found for three separate commercial models in Table 1, indicates that CheckList is useful even in pipelines that are stress-tested and used in production.

从定性反馈看，该团队认为 CheckList 很有帮助：（1）他们测到了此前没有考虑过的能力；（2）他们测到了考虑过、但基准里没有的能力；（3）即便已有基准的能力（例如否定），用 CheckList 也测得更彻底、更系统。他们发现了许多此前不知道的缺陷，并计划在下一轮模型中修复。最后，他们表示一定会把 CheckList 纳入开发流程，并希望使用我们的实现。这次讨论，连同我们在表 1 中三个不同商业模型上发现的多种缺陷，说明即使在已经过压力测试并用于生产的流程里，CheckList 仍然有用。

4.2 User Study: CheckList MFTs

4.2 用户研究：CheckList 的 MFT

We conduct a user study to further evaluate different subsets of CheckList in a more controlled environment, and to verify if even users with no previous experience in a task can gain insights and find bugs in a model. We recruit 18 participants (8 from industry, 10 from academia) who have at least intermediate NLP experience⁵, and task them with testing [BERT] finetuned on QQP for a period of two hours (including instructions), using Jupyter notebooks. Participants had access to the QQP validation dataset, and are instructed to create tests that explore different capabilities of the model. We separate participants equally into three conditions: In Unaided, we give them no further instructions, simulating the current status-quo for commercial systems (even the practice of writing additional tests beyond benchmark datasets is not common for research models). In Cap. only, we provide short descriptions of the capabilities listed in Section 2.1 as suggestions to test, while in Cap.+templ. we further provide them with the template and fill-in tools described in Section 2.3. Only one participant (in Unaided) had prior experience with QQP. Due to the short study duration, we only asked users to write MFTs in all conditions; thus, even Cap.+templ. is a subset of CheckList.

我们做了一项用户研究，以便在更受控的环境里进一步评估 CheckList 的不同子集，并核验：即便使用者对某项任务此前没有经验，能否也能获得认识、并在模型中找到缺陷。我们招募了 18 名至少具备中等 NLP 经验的参与者⁵（8 人来自工业界，10 人来自学术界），请他们用 Jupyter notebook，在两小时内（含说明时间）测试在 QQP 上微调的 [BERT]。参与者可以使用 QQP 验证集，并被要求编写探索模型不同能力的测试。我们把参与者均分成三种条件：在无辅助（Unaided）条件下，不再给额外说明，以模拟商业系统的现状（对研究模型而言，在基准数据集之外再写测试本身就不常见）；在“仅能力”（Cap. only）条件下，提供第 2.1 节所列能力的简短描述作为测试建议；在“能力+模板”（Cap.+templ.）条件下，再提供第 2.3 节所述的模板与填空工具。只有一名参与者（在无辅助组）以前接触过 QQP。由于研究时间短，所有条件下我们只请使用者写 MFT；因此，即便是 Cap.+templ.，也只是 CheckList 的一个子集。

⁵i.e. have taken a graduate NLP course or equivalent.

⁵即修过研究生程度的 NLP 课程或具备同等经验。

Table 4: User Study Results: first three rows indicate number of tests created, number of test cases per test and number of capabilities tested. Users report the severity of their findings (last two rows).

表 4：用户研究结果。前三行分别是写出的测试数量、每项测试的用例数，以及测到的能力数。使用者自行报告所发现缺陷的严重程度（后两行）。

We present the results in Table 4. Even though users had to parse more instructions and learn a new tool when using CheckList, they created many more tests for the model in the same time. Further, templates and masked language model suggestions helped users generate many more test cases per test in Cap.+templ. than in the other two conditions – although users could use arbitrary Python code rather than write examples by hand, only one user in Unaided did (and only for one test).

结果见表 4。尽管使用 CheckList 时使用者要消化更多说明、并学习一套新工具，他们在同样时间内仍为模型写出了多得多的测试。此外，模板和掩码语言模型建议，使 Cap.+templ. 条件下每项测试的用例数远多于另外两组——虽然使用者也可以写任意 Python 代码而不用手写例子，但无辅助组里只有一人这样做（而且只用于一项测试）。

Users explored many more capabilities on Cap. only and Cap.+templ. (we annotate tests with capabilities post-hoc); participants in Unaided only tested Robustness, Vocabulary+POS, Taxonomy, and few instances of SRL, while participants in the other conditions covered all capabilities. Users in Cap. only and Cap.+templ. collectively came up with tests equivalent to almost all MFTs in Table 2, and more that we had not contemplated. Users in Unaided and Cap. only often did not find more bugs because they lacked test case variety even when testing the right concepts (e.g. negation).

Cap. only 和 Cap.+templ. 的使用者探索了多得多的能力（我们事后给测试标注了能力）；无辅助组的参与者几乎只测了鲁棒性、词汇与词性、分类体系，以及少量语义角色标注，而另外两组覆盖了全部能力。Cap. only 和 Cap.+templ. 的使用者合在一起，写出的测试几乎覆盖了表 2 中全部 MFT，还有一些我们未曾想到的。无辅助组和 Cap. only 组往往发现不了更多缺陷，是因为即便测的是对的概念（例如否定），用例也不够多样。

At the end of the experiment, we ask users to evaluate the severity of the failures they observe on each particular test, on a 5 point scale⁶. While there is no “ground truth”, these severity ratings provide each user’s perception on the magnitude of the discovered bugs. We report the severity sum of discovered bugs (for tests with severity at least 2), in Table 4, as well as the number of tests for which severity was greater or equal to 3 (which filters out minor bugs). We note that users with CheckList (Cap. only and Cap.+templ.) discovered much more severe problems in the model (measured by total severity or # bugs) than users in the control condition (Unaided). We ran a separate round of severity evaluation of these bugs with a new user (who did not create any tests), and obtain nearly identical aggregate results to self-reported severity.

实验结束时，我们请使用者按 5 分制⁶评估自己在每项测试上观察到的失败有多严重。虽然没有“标准答案”，这些评分反映的是每位使用者对所发现缺陷大小的判断。表 4 报告了所发现缺陷的严重度总和（只计入严重度至少为 2 的测试），以及严重度大于或等于 3 的测试数量（从而滤掉轻微缺陷）。可以看到，使用 CheckList 的两组（Cap. only 和 Cap.+templ.）在模型中发现的问题，按总严重度或缺陷数量衡量，都远比对照组（无辅助）更严重。我们另请一位没有编写任何测试的新使用者，对这些缺陷再评一轮严重度，得到的汇总结果与自我报告几乎相同。

⁶1 (not a bug), 2 (minor bug), 3 (bug worth investigating and fixing), 4 (severe bug, model may not be fit for production), and 5 (no model with this bug should be in production).

⁶1（不是缺陷），2（轻微缺陷），3（值得调查并修复的缺陷），4（严重缺陷，模型可能不适合投入生产），5（带有这一缺陷的模型不应进入生产）。

The study results are encouraging: with a subset of CheckList, users without prior experience are able to find significant bugs in a SOTA model in only 2 hours. Further, when asked to rate different aspects of CheckList (on a scale of 1-5), users indicated the testing session helped them learn more about the model (4.7 ± 0.5), capabilities helped them test the model more thoroughly (4.5 ± 0.4), and so did templates (4.3 ± 1.1).

研究结果令人鼓舞：只用 CheckList 的一个子集，没有先前经验的使用者也能在 2 小时内，在一个当时最好的模型里找到显著缺陷。此外，当被请对 CheckList 的不同方面打分（1–5 分）时，使用者表示这次测试帮助他们更多地了解了模型（$4.7 \pm 0.5$），能力清单帮助他们测得更彻底（$4.5 \pm 0.4$），模板同样如此（$4.3 \pm 1.1$）。

5 Related Work

5 相关工作

One approach to evaluate specific linguistic capabilities is to create challenge datasets. Belinkov and Glass (2019) note benefits of this approach, such as systematic control over data, as well as drawbacks, such as small scale and lack of resemblance to “real” data. Further, they note that the majority of challenge sets are for Natural Language Inference. We do not aim for CheckList to replace challenge or benchmark datasets, but to complement them. We believe CheckList maintains many of the benefits of challenge sets while mitigating their drawbacks: authoring examples from scratch with templates provides systematic control, while perturbation-based INV and DIR tests allow for testing behavior in unlabeled, naturally-occurring data. While many challenge sets focus on extreme or difficult cases (Naik et al., 2018), MFTs also focus on what should be easy cases given a capability, uncovering severe bugs. Finally, the user study demonstrates that CheckList can be used effectively for a variety of tasks with low effort: users created a complete test suite for sentiment analysis in a day, and MFTs for QQP in two hours, both revealing previously unknown, severe bugs.

评测特定语言能力的一条路径，是制作挑战数据集。Belinkov and Glass（2019）指出了这种方法的好处，例如能够系统控制数据，也指出了缺点，例如规模小、不像“真实”数据。他们还指出，大多数挑战集都是为自然语言推理准备的。我们并不打算让 CheckList 取代挑战集或基准数据集，而是作为补充。我们认为 CheckList 保留了挑战集的许多优点，同时减轻了其缺点：用模板从零编写例子，可以系统控制；基于扰动的 INV 和 DIR 测试，则能在未标注的自然出现的数据上检验行为。许多挑战集盯的是极端或困难情形（Naik et al., 2018），而 MFT 还会针对给定能力下本应容易的情形，从而揭示严重缺陷。最后，用户研究表明，CheckList 能以较低代价有效用于多种任务：使用者在一天内为情感分析写出了完整测试套件，并在两小时内为 QQP 写出了 MFT，两者都揭示了此前未知的严重缺陷。

With the increase in popularity of end-to-end deep models, the community has turned to “probes”, where a probing model for linguistic phenomena of interest (e.g. NER) is trained on intermediate representations of the encoder (Tenney et al., 2019; Kim et al., 2019). Along similar lines, previous work on word embeddings looked for correlations between properties of the embeddings and downstream task performance (Tsvetkov et al., 2016; Rogers et al., 2018). While interesting as analysis methods, these do not give users an understanding of how a fine-tuned (or end-to-end) model can handle linguistic phenomena for the end-task. For example, while Tenney et al. (2019) found that very accurate NER models can be trained using BERT (96.7%), we show BERT finetuned on QQP or SST-2 displays severe NER issues.

随着端到端深度模型日渐流行，社区转向了“探针”：在编码器的中间表示上，为所关心的语言现象（例如 NER）训练一个探针模型（Tenney et al., 2019; Kim et al., 2019）。与此相近，早先关于词向量的工作，则寻找向量性质与下游任务表现之间的相关（Tsvetkov et al., 2016; Rogers et al., 2018）。作为分析方法，这些工作很有意思，但并不能让使用者理解：一个微调后（或端到端）的模型，在最终任务上会怎样处理语言现象。例如，Tenney et al.（2019）发现用 BERT 可以训练出非常准确的 NER 模型（96.7%），而我们表明，在 QQP 或 SST-2 上微调的 BERT 仍会表现出严重的 NER 问题。

There are existing perturbation techniques meant to evaluate specific behavioral capabilities of NLP models such as logical consistency (Ribeiro et al., 2019) and robustness to noise (Belinkov and Bisk, 2018), name changes (Prabhakaran et al., 2019), or adversaries (Ribeiro et al., 2018). CheckList provides a framework for such techniques to systematically evaluate these alongside a variety of other capabilities. However, CheckList cannot be directly used for non-behavioral issues such as data versioning problems (Amershi et al., 2019), labeling errors, annotator biases (Geva et al., 2019), worst-case security issues (Wallace et al., 2019), or lack of interpretability (Ribeiro et al., 2016).

已有一些扰动技术，用来评测 NLP 模型的特定行为能力，例如逻辑一致性（Ribeiro et al., 2019）、对噪声的鲁棒性（Belinkov and Bisk, 2018）、改名（Prabhakaran et al., 2019）或对抗样本（Ribeiro et al., 2018）。CheckList 为这类技术提供了一个框架，使它们可以和其他多种能力放在一起系统评测。不过，CheckList 不能直接用于非行为层面的问题，例如数据版本问题（Amershi et al., 2019）、标注错误、标注者偏差（Geva et al., 2019）、最坏情况下的安全问题（Wallace et al., 2019），或缺乏可解释性（Ribeiro et al., 2016）。

6 Conclusion

6 结论

While useful, accuracy on benchmarks is not sufficient for evaluating NLP models. Adopting principles from behavioral testing in software engineering, we propose CheckList, a model-agnostic and task-agnostic testing methodology that tests individual capabilities of the model using three different test types. To illustrate its utility, we highlight significant problems at multiple levels in the conceptual NLP pipeline for models that have “solved” existing benchmarks on three different tasks. Further, CheckList reveals critical bugs in commercial systems developed by large software companies, indicating that it complements current practices well. Tests created with CheckList can be applied to any model, making it easy to incorporate in current benchmarks or evaluation pipelines.

基准上的准确率虽然有用，但不足以评测 NLP 模型。借鉴软件工程中行为测试的原则，我们提出 CheckList：一套不依赖特定模型、也不依赖特定任务的测试方法，用三类测试分别检验模型的各项能力。为说明其用处，我们指出：在三个不同任务上已经“解决”现有基准的模型，在概念上的 NLP 流程的多个层面仍有显著问题。此外，CheckList 还在大型软件公司开发的商业系统中揭示了严重缺陷，说明它能很好地补充现有做法。用 CheckList 写出的测试可以用于任何模型，因而容易纳入当前的基准或评测流程。

Our user studies indicate that CheckList is easy to learn and use, and helpful both for expert users who have tested their models at length as well as for practitioners with little experience in a task. The tests presented in this paper are part of CheckList’s open source release, and can easily be incorporated into existing benchmarks. More importantly, the abstractions and tools in CheckList can be used to collectively create more exhaustive test suites for a variety of tasks. Since many tests can be applied across tasks as is (e.g. typos) or with minor variations (e.g. changing names), we expect that collaborative test creation will result in evaluation of NLP models that is much more robust and detailed, beyond just accuracy on held-out data. CheckList is open source, and available at https://github.com/marcotcr/checklist.

我们的用户研究表明，CheckList 容易学会、也容易使用，对已经充分测试过自己模型的专家，以及对某项任务经验不多的从业者都有帮助。本文给出的测试已包含在 CheckList 的开源发布中，可以方便地纳入现有基准。更重要的是，CheckList 中的抽象和工具，可以用来为多种任务共同写出更详尽的测试套件。由于许多测试可以原样跨任务使用（例如拼写错误），或只需稍作改动（例如更换姓名），我们预期协同编写测试，会带来比留出集准确率更稳健、也更细致的 NLP 模型评测。CheckList 已经开源，见 https://github.com/marcotcr/checklist。

Acknowledgments

致谢

We would like to thank Sara Ribeiro, Scott Lundberg, Matt Gardner, Julian Michael, and Ece Kamar for helpful discussions and feedback. Sameer was funded in part by the NSF award #IIS-1756023, and in part by the DARPA MCS program under Contract No. N660011924033 with the United States Office of Naval Research.

我们感谢 Sara Ribeiro、Scott Lundberg、Matt Gardner、Julian Michael 和 Ece Kamar 的讨论与反馈。Sameer 的部分资助来自 NSF 项目 #IIS-1756023，另一部分来自 DARPA MCS 项目，合同号 N660011924033，由美国海军研究办公室执行。

References

参考文献

Saleema Amershi, Andrew Begel, Christian Bird, Rob DeLine, Harald Gall, Ece Kamar, Nachi Nagappan, Besmira Nushi, and Tom Zimmermann. 2019. Software engineering for machine learning: A case study. In International Conference on Software Engineering (ICSE 2019) - Software Engineering in Practice track. IEEE Computer Society.

Boris Beizer. 1995. Black-box Testing: Techniques for Functional Testing of Software and Systems. John Wiley & Sons, Inc., New York, NY, USA.

Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations.

Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72.

Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Empirical Methods in Natural Language Processing (EMNLP), pages 1161–1166.

Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of NAACL-HLT, pages 1875–1885.

Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, et al. 2019. Probing what different nlp tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (* SEM 2019), pages 235–249.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress Test Evaluation for Natural Language Inference. In International Conference on Computational Linguistics (COLING).

Kayur Patel, James Fogarty, James A Landay, and Beverly Harrison. 2008. Investigating statistical machine learning as a tool for software development. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 667–676. ACM.

Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019. Perturbation sensitivity analysis to detect unintended model biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5740–5745, Hong Kong, China. Association for Computational Linguistics.

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389–5400.

Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? evaluating consistency of question-answering models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6174–6184.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144. ACM.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Association for Computational Linguistics (ACL).

Anna Rogers, Shashwath Hosur Ananthakrishna, and Anna Rumshisky. 2018. What’s in your embedding, and how it predicts task performance. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2690–2703, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Barbara Rychalska, Dominika Basaj, Alicja Gosiewska, and Przemysław Biecek. 2019. Models in the wild: On corruption robustness of neural nlp systems. In International Conference on Neural Information Processing, pages 235–247. Springer.

Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016. A survey on metamorphic testing. IEEE Transactions on software engineering, 42(9):805–824.

Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics.

Yulia Tsvetkov, Manaal Faruqui, and Chris Dyer. 2016. Correlation-based intrinsic evaluation of word vector representations. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 111–115, Berlin, Germany. Association for Computational Linguistics.

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153–2162.

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3261–3275.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.

Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. 2019. Errudite: Scalable, reproducible, and testable error analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 747–763.