CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Abstract
摘要
When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.
要建造能够推理、并能回答视觉问题的人工智能系统,就需要有诊断测试,用来分析进展、找出短板。已有的视觉问答基准可以帮忙,但偏差往往很重,模型不必真正推理也能答对;多种错误来源又缠在一起,很难准确指出模型弱在哪里。我们提出一套诊断数据集,用来检验多种视觉推理能力。它尽量压低偏差,并为每道题标注了所需的推理类型。我们用这套数据去分析一批当代视觉推理系统,从而对它们的能力与局限给出新的观察。
Q: Are there an equal number of large things and metal spheres?
Q: What size is the cylinder that is left of the brown metal thing that is left of the big sphere?
Q: There is a sphere with the same size as the metal cube; is it made of the same material as the small red sphere?
Q: How many objects are either small cylinders or metal things?
Figure 1. A sample image and questions from CLEVR. Questions test aspects of visual reasoning such as attribute identification, counting, comparison, multiple attention, and logical operations.
问:大物体和金属球体的数量是否相等?
问:位于棕色金属物体左侧、而该棕色金属物体又位于大球体左侧的那个圆柱,尺寸是多少?
问:有一个球体与金属立方体尺寸相同;它的材质是否和小红色球体一样?
问:有多少物体要么是小圆柱,要么是金属物体?
图 1. CLEVR 中的一张样例图像及问题。这些问题考查属性识别、计数、比较、多重注意和逻辑运算等视觉推理方面。
- Introduction
- 引言
A long-standing goal of artificial intelligence research is to develop systems that can reason and answer questions about visual information. Recently, several datasets have been introduced to study this problem [4, 10, 21, 26, 32, 46, 49]. Each of these Visual Question Answering (VQA) datasets contains challenging natural language questions about images. Correctly answering these questions requires perceptual abilities such as recognizing objects, attributes, and spatial relationships as well as higher-level skills such as counting, performing logical inference, making comparisons, or leveraging commonsense world knowledge [31]. Numerous methods have attacked these problems [2, 3, 9, 24, 44], but many show only marginal improvements over strong baselines [4, 16, 48]. Unfortunately, our ability to understand the limitations of these methods is impeded by the inherent complexity of the VQA task. Are methods hampered by failures in recognition, poor reasoning, lack of commonsense knowledge, or something else?
人工智能研究的一个长期目标,是做出能够对视觉信息加以推理并回答问题的系统。近来已有若干数据集被提出来研究这一问题 [4, 10, 21, 26, 32, 46, 49]。这些视觉问答(Visual Question Answering,VQA)数据集都包含针对图像的、有难度的自然语言问题。要正确作答,既需要识别物体、属性和空间关系等感知能力,也需要计数、逻辑推断、比较,或动用常识性世界知识等更高层的技能 [31]。已有许多方法试图解决这些问题 [2, 3, 9, 24, 44],但不少方法相对强基线只有很小的提升 [4, 16, 48]。可惜的是,VQA 任务本身过于复杂,使我们很难看清这些方法的局限究竟在哪里:是识别失败,推理不行,缺少常识,还是别的原因?
The difficulty of understanding a system's competences is exemplified by Clever Hans, a 1900s era horse who appeared to be able to answer arithmetic questions. Careful observation revealed that Hans was correctly “answering” questions by reacting to cues read off his human observers [30]. Statistical learning systems, like those used for VQA, may develop similar “cheating” approaches to superficially “solve” tasks without learning the underlying reasoning processes [35, 36]. For instance, a statistical learner may correctly answer the question “What covers the ground?” not because it understands the scene but because biased datasets often ask questions about the ground when it is snow-covered [1, 47]. How can we determine whether a system is capable of sophisticated reasoning and not just exploiting biases of the world, similar to Clever Hans?
要理解一个系统究竟具备哪些能力,有一个常被提起的例子:二十世纪初那匹名叫“聪明的汉斯”(Clever Hans)的马,看起来会做算术题。仔细观察后发现,汉斯其实是在读取人类观察者给出的暗示,才“答对”了问题 [30]。用于 VQA 的统计学习系统,也可能发展出类似的“取巧”办法,只在表面上“解决”任务,并没有学会底层的推理过程 [35, 36]。例如,一个统计学习者也许能正确回答“地面上覆盖着什么?”,并不是因为它看懂了场景,而是因为有偏的数据集常常只在地面被雪覆盖时才问到地面 [1, 47]。我们怎样才能判断一个系统是真的具备复杂推理能力,而不是像聪明的汉斯那样,只是在利用世界里的偏差?
In this paper we propose a diagnostic dataset for studying the ability of VQA systems to perform visual reasoning. We refer to this dataset as the Compositional Language and Elementary Visual Reasoning diagnostics dataset (CLEVR; pronounced as clever in homage to Hans). CLEVR contains 100k rendered images and about one million automatically generated questions, of which 853k are unique. It has challenging images and questions that test visual reasoning abilities such as counting, comparing, logical reasoning, and storing information in memory, as illustrated in Figure 1.
本文提出一套诊断数据集,用来研究 VQA 系统做视觉推理的能力。我们把它称作 Compositional Language and Elementary Visual Reasoning 诊断数据集(CLEVR;读音接近 clever,以向汉斯致敬)。CLEVR 包含 10 万张渲染图像和约 100 万道自动生成的问题,其中 85.3 万道是独特的。如图 1 所示,图像和问题都有相当难度,用来考查计数、比较、逻辑推理以及在记忆中保存信息等视觉推理能力。
We designed CLEVR with the explicit goal of enabling detailed analysis of visual reasoning. Our images depict simple 3D shapes; this simplifies recognition and allows us to focus on reasoning skills. We ensure that the information in each image is complete and exclusive so that external information sources, such as commonsense knowledge, cannot increase the chance of correctly answering questions. We control question-conditional bias via rejection sampling within families of related questions, and avoid degenerate questions that are seemingly complex but contain simple shortcuts to the correct answer. Finally, we use structured ground-truth representations for both images and questions: images are annotated with ground-truth object positions and attributes, and questions are represented as functional programs that can be executed to answer the question (see Section 3). These representations facilitate in-depth analyses not possible with traditional VQA datasets.
我们设计 CLEVR 的明确目标,就是让视觉推理可以得到细致的分析。图像刻画的是简单的三维形状,这样可以降低识别难度,把重点放到推理技能上。我们保证每张图里的信息既完整又封闭,外部信息来源——例如常识——并不能提高答对的机会。我们在相关问题族内部用拒绝采样来控制问题条件偏差,并避开那些看起来复杂、实际上却有简单捷径通向正确答案的退化问题。最后,图像和问题都采用结构化的真值表示:图像标注了物体的真实位置和属性,问题则表示为可以执行并得到答案的功能程序(见第 3 节)。这些表示使得传统 VQA 数据集做不到的深入分析成为可能。
These design choices also mean that while images in CLEVR may be visually simple, its questions are complex and require a range of reasoning skills. For instance, factorized representations may be required to generalize to unseen combinations of objects and attributes. Tasks such as counting or comparing may require short-term memory [15] or attending to specific objects [24, 44]. Answering questions that combine multiple subtasks in diverse ways may require compositional systems [2, 3].
这些设计选择也意味着:CLEVR 的图像在视觉上也许简单,问题却很复杂,需要多种推理技能。例如,要泛化到未见过的物体与属性组合,可能需要因子化的表示。计数或比较一类任务,可能需要短时记忆 [15],或把注意放到特定物体上 [24, 44]。而以多样方式组合多个子任务的问题,则可能需要组合式的系统 [2, 3]。
We use CLEVR to analyze a suite of VQA models and discover weaknesses that are not widely known. For example, we find that current state-of-the-art VQA models struggle on tasks requiring short-term memory, such as comparing the attributes of objects, or compositional reasoning, such as recognizing novel attribute combinations. These observations point to novel avenues for further research.
我们用 CLEVR 分析了一批 VQA 模型,并发现了一些尚未被广泛认识到的弱点。例如,当前最先进的 VQA 模型在需要短时记忆的任务上会感到吃力,比如比较物体属性;在组合推理上同样如此,比如识别未见过的属性组合。这些观察为后续研究指出了新的方向。
Finally, we stress that accuracy on CLEVR is not an end goal in itself: a hand-crafted system with explicit knowledge of the CLEVR universe might work well, but will not generalize to real-world settings. Therefore CLEVR should be used in conjunction with other VQA datasets in order to study the reasoning abilities of general VQA systems.
最后我们要强调,CLEVR 上的准确率本身并不是终点:一个写死了 CLEVR 世界规则的手写系统也许能表现很好,却推广不到真实场景。因此,CLEVR 应当与其他 VQA 数据集一起使用,用来研究通用 VQA 系统的推理能力。
The CLEVR dataset is publicly available at http://cs.stanford.edu/people/jcjohns/clevr/.
CLEVR 数据集已公开发布,地址为 http://cs.stanford.edu/people/jcjohns/clevr/。
- Related Work
- 相关工作
In recent years, a range of benchmarks for visual understanding have been proposed, including datasets for image captioning [7, 8, 23, 45], referring to objects [19], relational graph prediction [21], and visual Turing tests [12, 27]. CLEVR, our diagnostic dataset, is most closely related to benchmarks for visual question answering [4, 10, 21, 26, 32, 37, 46, 49], as it involves answering natural-language questions about images. The two main differences between CLEVR and other VQA datasets are that: (1) CLEVR controls biases found in prior VQA datasets that can be used by learning systems to answer questions correctly without visual reasoning and (2) CLEVR’s synthetic nature and detailed annotations facilitate in-depth analyses of reasoning abilities that are impossible with existing datasets.
近年来已有一批视觉理解基准被提出,包括图像描述 [7, 8, 23, 45]、指代物体 [19]、关系图预测 [21] 以及视觉图灵测试 [12, 27]。我们这套诊断数据集 CLEVR,与视觉问答基准最为接近 [4, 10, 21, 26, 32, 37, 46, 49],因为它同样是要回答关于图像的自然语言问题。CLEVR 与其他 VQA 数据集的两个主要差别是:(1)CLEVR 控制了先前 VQA 数据集里常见的偏差,那些偏差可被学习系统利用,从而不必做视觉推理也能答对;(2)CLEVR 是合成的,并且带有细致标注,便于对推理能力做现有数据集做不到的深入分析。
Prior work has attempted to mitigate biases in VQA datasets in simple cases such as yes/no questions [12, 47], but it is difficult to apply such bias-reduction approaches to more complex questions without a high-quality semantic representation of both questions and answers. In CLEVR, this semantic representation is provided by the functional program underlying each image-question pair, and biases are largely eliminated via sampling. Winograd schemas [22] are another approach for controlling bias in question answering: these questions are carefully designed to be ambiguous based on syntax alone and require commonsense knowledge. Unfortunately this approach does not scale gracefully: the first phase of the 2016 Winograd Schema Challenge consists of just 60 hand-designed questions. CLEVR is also related to the bAbI question answering tasks [38] in that it aims to diagnose a set of clearly defined competences of a system, but CLEVR focuses on visual reasoning whereas bAbI is purely textual.
先前工作曾试图在是否题这类较简单的情形里减轻 VQA 数据集的偏差 [12, 47],但若没有问题和答案的高质量语义表示,这类降偏方法很难用到更复杂的问题上。在 CLEVR 里,这种语义表示由每对图像–问题背后的功能程序提供,偏差则主要通过采样来消除。Winograd 图式 [22] 是控制问答偏差的另一条路:这些问题被仔细设计成仅凭句法是含糊的,必须依靠常识。可惜这种方法很难放大规模:2016 年 Winograd Schema Challenge 的第一阶段只有 60 道手工题目。CLEVR 也与 bAbI 问答任务 [38] 有亲缘关系,因为它同样旨在诊断系统一组定义清楚的能力;不同的是,CLEVR 聚焦视觉推理,而 bAbI 完全是文本的。
We are also not the first to consider synthetic data for studying (visual) reasoning. SHRDLU performed simple, interactive visual reasoning with the goal of moving specific objects in the visual scene [40]; this study was one of the first to demonstrate the brittleness of manually programmed semantic understanding. The pioneering DAQUAR dataset [28] contains both synthetic and human-written questions, but they only generate 420 synthetic questions using eight text templates. VQA [4] contains 150,000 natural-language questions about abstract scenes [50], but these questions do not control for question-conditional bias and are not equipped with functional program representations. CLEVR is similar in spirit to the SHAPES dataset [3], but is more complex and varied both in terms of visual content and question variety and complexity: SHAPES contains 15,616 total questions with just 244 unique questions while CLEVR contains nearly a million questions of which 853,554 are unique.
我们也不是最早用合成数据来研究(视觉)推理的。SHRDLU 做过简单的交互式视觉推理,目标是在场景里移动指定物体 [40];这项研究是最早表明手工编写的语义理解很脆弱的工作之一。开创性的 DAQUAR 数据集 [28] 同时包含合成问题和人工撰写的问题,但他们只用 8 个文本模板生成了 420 道合成题。VQA [4] 包含 15 万道关于抽象场景的自然语言问题 [50],但这些题没有控制问题条件偏差,也没有功能程序表示。CLEVR 在精神上与 SHAPES 数据集 [3] 相近,但无论视觉内容还是问题的多样性和复杂度都更高:SHAPES 共有 15,616 道题,其中独特问题只有 244 道,而 CLEVR 有近 100 万道题,其中 853,554 道是独特的。
- The CLEVR Diagnostic Dataset
- CLEVR 诊断数据集
CLEVR provides a dataset that requires complex reasoning to solve and that can be used to conduct rich diagnostics to better understand the visual reasoning capabilities of VQA systems. This requires complete control over the dataset, which we achieve by using synthetic images and automatically generated questions. The images have associated ground-truth object locations and attributes, and the questions have an associated machine-readable form.
CLEVR 提供的数据集既需要复杂推理才能解答,也可用来做丰富的诊断,以便更好地理解 VQA 系统的视觉推理能力。这要求对数据集有完全的控制,我们通过合成图像和自动生成问题来做到这一点。图像带有物体位置和属性的真值,问题则有对应的机器可读形式。
图 2. CLEVR 世界指南。左:形状、属性和空间关系。中:问题示例及其对应的功能程序。右:用于构造问题的基本函数目录。细节见第 3 节。
Objects and relationships. The CLEVR universe contains three object shapes (cube, sphere, and cylinder) that come in two absolute sizes (small and large), two materials (shiny “metal” and matte “rubber”), and eight colors. Objects are spatially related via four relationships: “left”, “right”, “behind”, and “in front”. The semantics of these prepositions are complex and depend not only on relative object positions but also on camera viewpoint and context. We found that generating questions that invoke spatial relationships with semantic accord was difficult. Instead we rely on a simple and unambiguous definition: projecting the camera viewpoint vector onto the ground plane defines the “behind” vector, and one object is behind another if its ground-plane position is further along the “behind” vector. The other relationships are similarly defined. Figure 2 (left) illustrates the objects, attributes, and spatial relationships in CLEVR. The CLEVR universe also includes one non-spatial relationship type that we refer to as the same-attribute relation. Two objects are in this relationship if they have equal attribute values for a specified attribute.
物体与关系。CLEVR 世界里有三种物体形状(立方体、球体和圆柱),各有两种绝对尺寸(小和大)、两种材质(有光泽的“金属”和哑光的“橡胶”)以及八种颜色。物体之间有四种空间关系:“左”“右”“后”和“前”。这些介词的语义很复杂,不仅取决于相对位置,还取决于相机视点和上下文。我们发现,要生成在语义上完全吻合这些空间关系的问题并不容易。于是我们改用一个简单而明确的定义:把相机视线向量投影到地面,得到“后”方向向量;若一个物体在地面上的位置沿该“后”向量更远,就说它在另一个物体的后面。其余关系也按同样方式定义。图 2(左)展示了 CLEVR 中的物体、属性和空间关系。CLEVR 世界里还有一类非空间关系,我们称作相同属性关系。若两个物体在指定属性上取值相同,它们就处于这种关系中。
Scene representation. Scenes are represented as collections of objects annotated with shape, size, color, material, and position on the ground-plane. A scene can also be represented by a scene graph [17, 21], where nodes are objects annotated with attributes and edges connect spatially related objects. A scene graph contains all ground-truth information for an image and could be used to replace the vision component of a VQA system with perfect sight.
场景表示。场景被表示为一组物体,并标注形状、尺寸、颜色、材质以及在地面上的位置。场景也可以用场景图来表示 [17, 21]:节点是带属性的物体,边连接空间上相关的物体。场景图包含一张图像的全部真值信息,可用来把 VQA 系统的视觉部分替换成完美的视觉。
Image generation. CLEVR images are generated by randomly sampling a scene graph and rendering it using Blender [6]. Every scene contains between three and ten objects with random shapes, sizes, materials, colors, and positions. When placing objects we ensure that no objects intersect, that all objects are at least partially visible, and that there are small horizontal and vertical margins between the image-plane centers of each pair of objects; this helps reduce ambiguity in spatial relationships. In each image the positions of the lights and camera are randomly jittered.
图像生成。CLEVR 图像的生成方式是:随机采样一张场景图,再用 Blender 渲染 [6]。每个场景有 3 到 10 个物体,形状、尺寸、材质、颜色和位置都是随机的。放置物体时我们保证物体不相交、全部至少部分可见,并且每对物体在图像平面上的中心之间留有较小的水平和垂直间距;这有助于减少空间关系上的歧义。每张图里,灯光和相机的位置都会被随机扰动。
Question representation. Each question in CLEVR is associated with a functional program that can be executed on an image’s scene graph, yielding the answer to the question. Functional programs are built from simple basic functions that correspond to elementary operations of visual reasoning such as querying object attributes, counting sets of objects, or comparing values. As shown in Figure 2, complex questions can be represented by compositions of these simple building blocks. Full details about each basic function can be found in the supplementary material.
问题表示。CLEVR 中的每道题都对应一段功能程序,可以在图像的场景图上执行,从而得到答案。功能程序由简单的基本函数构成,这些函数对应视觉推理中的基本操作,例如查询物体属性、对物体集合计数,或比较取值。如图 2 所示,复杂问题可以由这些简单构件组合而成。各基本函数的完整说明见补充材料。
As we will see in Section 4, representing questions as functional programs enables rich analysis that would be impossible with natural-language questions. A question’s functional program tells us exactly which reasoning abilities are required to solve it, allowing us to compare performance on questions requiring different types of reasoning.
正如我们将在第 4 节看到的,把问题表示成功能程序,可以做自然语言问题做不到的丰富分析。一道题的功能程序准确地告诉我们解答它需要哪些推理能力,从而可以比较模型在需要不同类型推理的问题上的表现。
We categorize questions by question type, defined by the outermost function in the question’s program; for example the questions in Figure 2 have types query-color and exist. Figure 3 shows the number of questions of each type.
我们按问题类型给题目分类,类型由程序最外层的函数定义;例如图 2 中的问题类型是 query-color 和 exist。图 3 给出了各类问题的数量。
Figure 3. Top: Statistics for CLEVR; the majority of questions are unique and few questions from the val and test sets appear in the training set. Bottom left: Comparison of question lengths for different VQA datasets; CLEVR questions are generally much longer. Bottom right: Distribution of question types in CLEVR.
图 3. 上:CLEVR 的统计;大多数问题是独特的,验证集和测试集中很少有问题出现在训练集里。下左:不同 VQA 数据集的问题长度比较;CLEVR 的问题通常要长得多。下右:CLEVR 中的问题类型分布。
Question families. We must overcome several key challenges to generate a VQA dataset using functional programs. Functional building blocks can be used to construct an infinite number of possible functional programs, and we must decide which program structures to consider. We also need a method for converting functional programs to natural language in a way that minimizes question-conditional bias. We solve these problems using question families.
问题族。要用功能程序生成 VQA 数据集,必须克服几个关键困难。功能构件可以构造出无穷多种可能的功能程序,我们必须决定考虑哪些程序结构。我们还需要一种把功能程序转成自然语言的方法,并尽量减小问题条件偏差。我们用问题族来解决这些问题。
A question family contains a template for constructing functional programs and several text templates providing multiple ways of expressing these programs in natural language. For example, the question “How many red things are there?” can be formed by instantiating the text template “How many things are there?”, binding the parameters and (with types “color” and “material”) to the values red and nil. The functional program count(filter color(red, scene())) for this question can be formed by instantiating the associated program template count(filter color(, filter material(, scene()))) with the same values, using the convention that functions taking a nil input are removed after instantiation.
一个问题族包含一个用于构造功能程序的模板,以及若干文本模板,用来把这些程序用多种自然语言方式表达出来。例如,问题“How many red things are there?”可以由文本模板“How many things are there?”实例化得到,把类型为“color”和“material”的参数 和 分别绑定为 red 和 nil。该题对应的功能程序 count(filter color(red, scene())),则可由相关的程序模板 count(filter color(, filter material(, scene()))) 用同一组取值实例化得到;约定是:以 nil 为输入的函数在实例化后会被去掉。
CLEVR contains a total of 90 question families, each with a single program template and an average of four text templates. Text templates were generated by manually writing one or two templates per family and then crowdsourcing question rewrites. To further increase language diversity we use a set of synonyms for each shape, color, and material. With up to 19 parameters per template, a small number of families can generate a huge number of unique questions; Figure 3 shows that of the nearly one million questions in CLEVR, more than 853k are unique. CLEVR can easily be extended by adding new question families.
CLEVR 一共有 90 个问题族,每个族有一个程序模板,平均约四个文本模板。文本模板的做法是:先为每个族手工写一到两个模板,再通过众包改写题目。为了进一步增加语言多样性,我们为每种形状、颜色和材质准备了一组同义词。每个模板最多可有 19 个参数,因此少量问题族就能生成极多独特问题;图 3 显示,CLEVR 近 100 万道题中,超过 85.3 万道是独特的。只要加入新的问题族,CLEVR 就可以方便地扩展。
Question generation. Generating a question for an image is conceptually simple: we choose a question family, select values for each of its template parameters, execute the resulting program on the image’s scene graph to find the answer, and use one of the text templates from the question family to generate the final natural-language question.
问题生成。为图像生成一道题,概念上很简单:先选定一个问题族,为模板中的每个参数取值,把得到的程序在该图像的场景图上执行以得到答案,再用该问题族中的一个文本模板生成最终的自然语言问题。
However, many combinations of values give rise to questions which are either ill-posed or degenerate. The question “What color is the cube to the right of the sphere?” would be ill-posed if there were many cubes right of the sphere, or degenerate if there were only one cube in the scene since the reference to the sphere would then be unnecessary. Avoiding such ill-posed and degenerate questions is critical to ensure the correctness and complexity of our questions.
然而,许多取值组合会得到不适定或退化的问题。例如“球体右侧的立方体是什么颜色?”:若球体右侧有许多立方体,这道题就是不适定的;若场景里只有一个立方体,提到球体就没有必要,这道题就是退化的。避开这类不适定和退化问题,对于保证题目的正确性和复杂度至关重要。
A naïve solution is to randomly sample combinations of values and reject those which lead to ill-posed or degenerate questions. However, the number of possible configurations for a question family is exponential in its number of parameters, and most of them are undesirable. This makes brute-force search intractable for our complex question families.
一种朴素的办法是随机采样取值组合,再拒绝那些会得到不适定或退化问题的组合。可是,一个问题族可能的配置数目随参数个数指数增长,而且其中大多数都不可取。对我们这些复杂的问题族来说,暴力搜索因此不可行。
Instead, we employ a depth-first search to find valid values for instantiating question families. At each step of the search, we use ground-truth scene information to prune large swaths of the search space which are guaranteed to produce undesirable questions; for example we need not entertain questions of the form “What color is the to the of the sphere” for scenes that do not contain spheres.
我们改用深度优先搜索,为问题族的实例化寻找合法取值。搜索的每一步都利用场景真值,剪掉大片注定会得到不良问题的搜索空间;例如,不含球体的场景就不必再考虑“球体 侧的 是什么颜色”这类问题。
Finally, we use rejection sampling to produce an approximately uniform answer distribution for each question family; this helps minimize question-conditional bias since all questions from the same family share linguistic structure.
最后,我们用拒绝采样,使每个问题族的答案分布近似均匀;由于同一族的问题共享语言结构,这有助于尽量减小问题条件偏差。
- VQA Systems on CLEVR
- CLEVR 上的 VQA 系统
4.1. Models
4.1. 模型
VQA models typically represent images with features from pretrained CNNs and use word embeddings or recurrent networks to represent questions and/or answers. Models may train recurrent networks for answer generation [10, 28, 41], multiclass classifiers over common answers [4, 24, 25, 32, 48, 49], or binary classifiers on image-question-answer triples [9, 16, 33]. Many methods incorporate attention over the image [9, 33, 44, 49, 43] or question [24]. Some methods incorporate memory [42] or dynamic network architectures [2, 3].
VQA 模型通常用预训练 CNN 的特征来表示图像,并用词嵌入或循环网络来表示问题和/或答案。模型可以训练循环网络来生成答案 [10, 28, 41],也可以在常见答案上训练多类分类器 [4, 24, 25, 32, 48, 49],或在图像–问题–答案三元组上训练二分类器 [9, 16, 33]。许多方法对图像 [9, 33, 44, 49, 43] 或问题 [24] 引入注意力。还有些方法引入记忆 [42] 或动态网络结构 [2, 3]。
Experimenting with all methods is logistically challenging, so we reproduced a representative subset of methods: baselines that do not look at the image (Q-type mode, LSTM), a simple baseline (CNN+BoW) that performs near state-of-the-art [16, 48], and more sophisticated methods using recurrent networks (CNN+LSTM), sophisticated feature pooling (CNN+LSTM+MCB), and spatial attention (CNN+LSTM+SA).^{1} These are described in detail below.
要对所有方法做实验,在操作上很困难,因此我们复现了一组有代表性的方法:不看图像的基线(问题类型众数,LSTM)、表现接近当时最先进水平的简单基线(CNN+BoW)[16, 48],以及更复杂的方法,分别使用循环网络(CNN+LSTM)、更精细的特征汇合(CNN+LSTM+MCB)和空间注意力(CNN+LSTM+SA)。^{1} 这些方法下面再详细说明。
^{1}We performed initial experiments with dynamic module networks [2] but its parsing heuristics did not generalize to the complex questions in CLEVR so it did not work out-of-the-box; see supplementary material.
^{1}我们曾对动态模块网络 [2] 做过初步实验,但它的解析启发式无法推广到 CLEVR 中的复杂问题,因而不能开箱即用;见补充材料。
Q-type mode: Similar to the “per Q-type prior” method in [4], this baseline predicts the most frequent training-set answer for each question’s type.
问题类型众数:与 [4] 中的“按问题类型先验”方法类似,这一基线对每种问题类型预测训练集中最常见的答案。
LSTM: Similar to “LSTM Q” in [4], the question is processed with learned word embeddings followed by a word-level LSTM [15]. The final LSTM hidden state is passed to a multi-layer perceptron (MLP) that predicts a distribution over answers. This method uses no image information so it can only model question-conditional bias.
LSTM:与 [4] 中的“LSTM Q”类似,问题先经过学习得到的词嵌入,再送入词级 LSTM [15]。LSTM 的最终隐状态被送入多层感知机(MLP),以预测答案上的分布。该方法不使用图像信息,因而只能建模问题条件偏差。
CNN+BoW: Following [48], the question is encoded by averaging word vectors for each word in the question and the image is encoded using features from a convolutional network (CNN). The question and image features are concatenated and passed to a MLP which predicts a distribution over answers. We use word vectors trained on the Google News corpus [29]; these are not fine-tuned during training.
CNN+BoW:沿用 [48],问题通过对其每个词的词向量取平均来编码,图像则用卷积网络(CNN)特征来编码。问题和图像特征被拼接后送入 MLP,以预测答案分布。我们使用在 Google News 语料上训练的词向量 [29];训练过程中不对这些词向量做微调。
CNN+LSTM: Images and questions are encoded using CNN features and final LSTM hidden states, respectively. These features are concatenated and passed to an MLP that predicts an answer distribution.
CNN+LSTM:图像和问题分别用 CNN 特征和 LSTM 的最终隐状态来编码。这些特征被拼接后送入 MLP,以预测答案分布。
CNN+LSTM+MCB: Images and questions are encoded as above, but instead of concatenation, their features are pooled using compact multimodal pooling (MCB) [9, 11].
CNN+LSTM+MCB:图像和问题的编码方式同上,但不再做拼接,而是用紧凑多模态汇合(compact multimodal pooling,MCB)来融合特征 [9, 11]。
CNN+LSTM+SA: Again, the question and image are encoded using a CNN and LSTM, respectively. Following [44], these representations are combined using one or more rounds of soft spatial attention and the final answer distribution is predicted with an MLP.
CNN+LSTM+SA:问题和图像同样分别用 LSTM 和 CNN 编码。按照 [44],这些表示通过一轮或多轮软空间注意力加以结合,最终的答案分布由 MLP 预测。
Human: We used Mechanical Turk to collect human responses for 5500 random questions from the test set, taking a majority vote among three workers for each question.
人类:我们通过 Mechanical Turk,从测试集中随机抽取 5500 道题收集人类作答,每道题取三名工作人员的多数票。
Implementation details. Our CNNs are ResNet-101 models pretrained on ImageNet [14] that are not finetuned; images are resized to 224×224 prior to feature extraction. CNN+LSTM+SA extracts features from the last layer of the conv4 stage, giving 14 × 14 × 1024-dimensional features. All other methods extract features from the final average pooling layer, giving 2048-dimensional features. LSTMs use one or two layers with 512 or 1024 units per layer. MLPs use ReLU functions and dropout [34]; they have one or two hidden layers with between 1024 and 8192 units per layer. All models are trained using Adam [20].
实现细节。我们的 CNN 是在 ImageNet 上预训练、并且不再微调的 ResNet-101 [14];提取特征前把图像缩放到 224×224。CNN+LSTM+SA 从 conv4 阶段的最后一层提取特征,得到 14 × 14 × 1024 维特征。其余方法从最后的平均池化层提取特征,得到 2048 维特征。LSTM 使用一层或两层,每层 512 或 1024 个单元。MLP 使用 ReLU 和 dropout [34];它们有一层或两层隐层,每层单元数在 1024 到 8192 之间。所有模型都用 Adam 训练 [20]。
Experimental protocol. CLEVR is split into train, validation, and test sets (see Figure 3). We tuned hyperparameters (learning rate, dropout, word vector size, number and size of LSTM and MLP layers) independently per model based on the validation error. All experiments were designed on the validation set; after finalizing the design we ran each model once on the test set. All experimental findings generalized from the validation set to the test set.
实验协议。CLEVR 划分为训练、验证和测试集(见图 3)。我们根据验证误差,为每个模型独立调节超参数(学习率、dropout、词向量大小,以及 LSTM 和 MLP 的层数与宽度)。所有实验都在验证集上设计;方案确定之后,每个模型只在测试集上跑一次。全部实验发现都从验证集推广到了测试集。
4.2. Analysis by Question Type
4.2. 按问题类型分析
We can use the program representation of questions to analyze model performance on different forms of reasoning. We first evaluate performance on each question type, defined as the outermost function in the program. Figure 4 shows results and detailed findings are discussed below.
我们可以利用问题的程序表示,分析模型在不同形式推理上的表现。我们首先按问题类型评估性能,类型定义为程序最外层的函数。图 4 给出了结果,详细发现在下面讨论。
图 4. 六种 VQA 方法在 CLEVR 数据集上按问题类型的准确率(越高越好)。本图宜对照彩色版本观看。
Querying attributes: Query questions ask about an attribute of a particular object (e.g. “What color is the thing right of the red sphere?”). The CLEVR world has two sizes, eight colors, two materials, and three shapes. On questions asking about these different attributes, Q-type mode and LSTM obtain accuracies close to 50%, 12.5%, 50%, and 33.3% respectively, showing that the dataset has minimal question-conditional bias for these questions. CNN+LSTM+SA substantially outperforms all other models on these questions; its attention mechanism may help it focus on the target object and identify its attributes.
查询属性:查询类问题询问某一特定物体的某个属性(例如“红色球体右侧的那个东西是什么颜色?”)。CLEVR 世界有两种尺寸、八种颜色、两种材质和三种形状。在分别询问这些属性的问题上,问题类型众数和 LSTM 的准确率分别接近 50%、12.5%、50% 和 33.3%,说明这些题的问题条件偏差很小。CNN+LSTM+SA 在这类问题上明显优于其他所有模型;它的注意力机制可能有助于把焦点放到目标物体上并识别其属性。
Comparing attributes: Attribute comparison questions ask whether two objects have the same value for some attribute (e.g. “Is the cube the same size as the sphere?”). The only valid answers are “yes” and “no”. Q-type mode and LSTM achieve accuracies close to 50%, confirming there is no dataset bias for these questions. Unlike attribute-query questions, attribute-comparison questions require a limited form of memory: models must identify the attributes of two objects and keep them in memory to compare them. Interestingly, none of the models are able to do so: all models have an accuracy of approximately 50%. This is also true for the CNN+LSTM+SA model, suggesting that its attention mechanism is not capable of attending to two objects at once to compare them. This illustrates how CLEVR can reveal limitations of models and motivate follow-up research, e.g., augmenting attention models with explicit memory.
比较属性:属性比较问题询问两个物体在某个属性上的取值是否相同(例如“立方体是否和球体一样大?”)。有效答案只有“是”和“否”。问题类型众数和 LSTM 的准确率接近 50%,说明这些题没有数据集偏差。与属性查询不同,属性比较需要一种有限形式的记忆:模型必须识别两个物体的属性,并在记忆中保存它们以便比较。有意思的是,没有一个模型能做到这一点:所有模型的准确率都大约是 50%。CNN+LSTM+SA 也是如此,说明它的注意力机制无法同时关注两个物体并加以比较。这正说明 CLEVR 能够揭示模型的局限,并推动后续研究,例如给注意力模型加上显式记忆。
Existence: Existence questions ask whether a certain type of object is present (e.g., “Are there any cubes to the right of the red thing?”). The 50% accuracy of Q-type mode shows that both answers are a priori equally likely, but the LSTM result of 60% does suggest a question-conditional bias. There may be correlations between question length and answer: questions with more filtering operations (e.g., “large red cube” vs. “red cube”) may be more likely to have “no” as the answer. Such biases may be present even with uniform answer distributions per question family, since questions from the same family may have different numbers of filtering functions. CNN+LSTM(+SA) outperforms LSTM, but its performance is still quite low.
存在性:存在性问题询问某类物体是否出现(例如“红色物体右侧有没有立方体?”)。问题类型众数 50% 的准确率说明两种答案先验上同样可能,但 LSTM 得到 60%,说明仍存在问题条件偏差。问题长度和答案之间可能有相关:过滤操作更多的问题(例如“大红立方体”相对“红立方体”)答案更可能是“否”。即使每个问题族的答案分布是均匀的,这类偏差仍可能存在,因为同一族的问题可以有不同数量的过滤函数。CNN+LSTM(+SA)优于 LSTM,但绝对表现仍然很低。
Counting: Counting questions ask for the number of objects fulfilling some conditions (e.g. “How many red cubes are there?”); valid answers range from zero to ten. Images have three and ten objects and counting questions refer to subsets of objects, so ensuring a uniform answer distribution is very challenging; our rejection sampler therefore pushes towards a uniform distribution for these questions rather than enforcing it as a hard constraint. This results in a question-conditional bias, reflected in the 35% and 42% accuracies achieved by Q-type mode and LSTM. CNN+LSTM(+MCB) performs on par with LSTM, suggesting that CNN features contain little information relevant to counting. CNN+LSTM+SA performs slightly better, but at 52% its absolute performance is low.
计数:计数问题询问满足某些条件的物体数量(例如“有多少个红色立方体?”);有效答案从 0 到 10。图像有 3 到 10 个物体,计数问题又只涉及其中的子集,因此很难保证答案分布完全均匀;我们的拒绝采样器于是只是把这些题的答案分布往均匀方向推,而不是把它当作硬约束来执行。这会带来问题条件偏差,表现为问题类型众数和 LSTM 分别只有 35% 和 42% 的准确率。CNN+LSTM(+MCB)与 LSTM 相当,说明 CNN 特征里几乎没有对计数有用的信息。CNN+LSTM+SA 略好一些,但 52% 的绝对表现仍然偏低。
Integer comparison: Integer comparison questions ask which of two object sets is larger (e.g. “Are there fewer cubes than red things?”); this requires counting, memory, and comparing integer quantities. The answer distribution is unbiased (see Q-type mode) but a set’s size may correlate with the length of its description, explaining the gap between LSTM and Q-type mode. CNN+BoW performs no better than chance: BoW mixes the words describing each set, making it impossible for the learner to discriminate between them. CNN+LSTM+SA outperforms LSTM on “less” and “more” questions, but no model outperforms LSTM on “equal” questions. Most models perform better on “less” than “more” due to asymmetric question families.
整数比较:整数比较问题询问两个物体集合哪个更大(例如“立方体是否比红色物体更少?”);这需要计数、记忆,并比较整数数量。答案分布是无偏的(见问题类型众数),但一个集合的大小可能与其描述长度相关,这可以解释 LSTM 与问题类型众数之间的差距。CNN+BoW 并不比随机更好:词袋会把描述两个集合的词混在一起,学习者无法把它们区分开。CNN+LSTM+SA 在“更少”和“更多”问题上优于 LSTM,但在“相等”问题上没有模型超过 LSTM。由于问题族不对称,多数模型在“更少”上比在“更多”上表现更好。
4.3. Analysis by Relationship Type
4.3. 按关系类型分析
图 5. 只含一个空间关系的问题,与只含一个相同属性关系的问题,二者准确率的对比。对查询和计数问题,模型在相同属性关系的问题上一般更差。存在性问题上的结果则比较混杂。
CLEVR questions contain two types of relationships: spatial and same-attribute (see Section 3). We can compare the relative difficulty of these two types by comparing model performance on questions with a single spatial relationship and questions with a single same-attribute relationship; results are shown in Figure 5. On query-attribute and counting questions we see that same-attribute questions are generally more difficult; the gap between CNN+LSTM+SA on spatial and same-relate query questions is particularly large (93% vs. 78%). Same-attribute relationships may require a model to keep attributes of one object “in memory” for comparison, suggesting again that models augmented with explicit memory may perform better on these questions.
CLEVR 的问题包含两类关系:空间关系和相同属性关系(见第 3 节)。我们可以通过比较模型在只含一个空间关系的问题和只含一个相同属性关系的问题上的表现,来看这两类关系的相对难度;结果见图 5。在查询属性和计数问题上,相同属性的问题一般更难;CNN+LSTM+SA 在空间查询和相同属性查询上的差距尤其大(93% 对 78%)。相同属性关系可能要求模型把一个物体的属性“记在脑子里”再去做比较,这再次提示:配上显式记忆的模型在这些问题上可能会更好。
4.4. Analysis by Question Topology
4.4. 按问题拓扑分析
图 6. 含有两个空间关系的问题的准确率,按问题拓扑分解:链状结构的问题,与用逻辑与运算连接起来的树状结构问题。
We next evaluate model performance on different question topologies: chain-structured questions vs. tree-structured questions with two branches joined by a logical AND (see Figure 2). In Figure 6, we compare performance on chain-structured questions with two spatial relationships vs. tree-structured questions with one relationship along each branch. On query questions, CNN+LSTM+SA shows a large gap between chain and tree questions (92% vs. 74%); on count questions, CNN+LSTM+SA slightly outperforms LSTM on chain questions (55% vs. 49%) but no method outperforms LSTM on tree questions. Tree questions may be more difficult since they require models to perform two subtasks in parallel before fusing their results.
接下来我们评估模型在不同问题拓扑上的表现:链状结构的问题,以及两支用逻辑与连接起来的树状结构问题(见图 2)。在图 6 中,我们比较含有两个空间关系的链状问题,与每支上各有一个关系的树状问题。在查询问题上,CNN+LSTM+SA 在链状和树状之间差距很大(92% 对 74%);在计数问题上,CNN+LSTM+SA 在链状题上略优于 LSTM(55% 对 49%),但在树状题上没有方法超过 LSTM。树状问题可能更难,因为模型需要并行完成两个子任务,再把结果融合起来。
4.5. Effect of Question Size
4.5. 问题规模的影响
Figure 7. Top: Many questions can be answered correctly without correctly solving all subtasks. For a given question and scene we can prune functions from the question’s program to generate an effective question which is shorter but gives the same answer. Bottom: Accuracy on query questions vs. actual and effective question size. Accuracy decreases with effective question size but not with actual size. Shaded area shows a 95% confidence interval.
图 7. 上:许多问题即使没有正确完成全部子任务,也仍能答对。对给定的问题和场景,我们可以从问题的程序里剪掉一些函数,得到一道更短、但答案相同的有效问题。下:查询问题的准确率随实际问题规模和有效问题规模的变化。准确率随有效规模下降,并不随实际规模下降。阴影区域表示 95% 置信区间。
Intuitively, longer questions should be harder since they involve more reasoning steps. We define a question’s size to be the number of functions in its program, and in Figure 7 (bottom left) we show accuracy on query-attribute questions as a function of question size.^{2} Surprisingly accuracy appears unrelated to question size.
直观上,更长的问题应当更难,因为推理步骤更多。我们把一道题的规模定义为程序中函数的个数,并在图 7(左下)中给出查询属性问题的准确率随问题规模的变化。^{2} 出乎意料的是,准确率看起来与问题规模无关。
^{2}We exclude questions with same-attribute relations since their max size is 10, introducing unwanted correlations between size and difficulty. Excluded questions show the same trends (see supplementary material).
^{2}我们排除了带相同属性关系的问题,因为它们的最大规模是 10,会在规模和难度之间引入不希望有的相关。被排除的问题显示出相同的趋势(见补充材料)。
However, many questions can be correctly answered even when some subtasks are not solved correctly. For example, the question in Figure 7 (top) can be answered correctly without identifying the correct large blue cylinder, because all large objects left of a cylinder are cylinders.
不过,许多问题即使某些子任务没有被正确完成,也仍能答对。例如,图 7(上)中的问题即使没有正确找出那个大蓝色圆柱,也能答对,因为所有位于某个圆柱左侧的大物体都是圆柱。
To quantify this effect, we define the effective question of an image-question pair: we prune functions from the question’s program to find the smallest program that, when executed on the scene graph for the question’s image, gives the same answer as the original question.^{3} A question’s effective size is the size of its effective question. Questions whose effective size is smaller than their actual size need not be degenerate. The question in Figure 7 is not degenerate because the entire question is needed to resolve its object references (there are two blue cylinders and two rubber cylinders), but it has a small effective size since it can be correctly answered without resolving those references.
为了量化这一效应,我们为每一对图像–问题定义有效问题:从原问题的程序里剪掉函数,找出在该题图像的场景图上执行后仍给出同一答案的最小程序。^{3} 一道题的有效规模就是其有效问题的规模。有效规模小于实际规模的问题不一定是退化的。图 7 中的问题并不是退化的,因为要消解其中的物体指代,整道题都用得上(那里有两个蓝色圆柱和两个橡胶圆柱);但它的有效规模很小,因为不必消解这些指代也能答对。
^{3}Pruned questions may be ill-posed (Section 3) so they are executed with modified semantics; see supplementary material for details.
^{3}剪枝后的问题可能是不适定的(第 3 节),因此执行时使用了修改后的语义;细节见补充材料。
In Figure 7 (bottom), we show accuracy on query questions as a function of effective question size. The error rate of all models increases with effective question size, suggesting that models struggle with long reasoning chains.
在图 7(下)中,我们给出查询问题的准确率随有效问题规模的变化。所有模型的错误率都随有效规模上升,说明模型在面对长推理链时会感到吃力。
4.6. Spatial Reasoning
4.6. 空间推理
Figure 8. Top: Some questions can be correctly answered using absolute definitions for spatial relationships; for example in this image there is only one purple cube in the bottom half of the image. Bottom: Accuracy of each model on chain-structured questions as a function of the number of spatial relationships in the question, separated by question type. Top row shows all chain-structured questions; bottom row excludes questions that can be correctly answered using absolute spatial reasoning.
图 8. 上:有些问题可以用空间关系的绝对定义来正确作答;例如在这张图里,图像下半部分只有一个紫色立方体。下:各模型在链状问题上的准确率,随问题中空间关系数目变化,并按问题类型分开。上一行是全部链状问题;下一行排除了只用绝对空间推理就能答对的问题。
We expect that questions with more spatial relationships should be more challenging since they require longer chains of reasoning. The top set of plots in Figure 8 shows accuracy on chain-structured questions with different numbers of relationships.^{4} Across all three question types, CNN+LSTM+SA shows a significant drop in accuracy for questions with one or more spatial relationship; other models are largely unaffected by spatial relationships.
我们预期,空间关系更多的问题应当更难,因为它们需要更长的推理链。图 8 上面一组图给出了含有不同数目关系的链状问题的准确率。^{4} 在全部三类问题上,CNN+LSTM+SA 在含有一个或多个空间关系的问题上准确率都明显下降;其他模型则基本上不受空间关系影响。
^{4}We restrict to chain-structured questions to avoid unwanted correlations between question topology and number of relationships.
^{4}我们只考虑链状问题,以避免问题拓扑与关系数目之间出现不希望有的相关。
Spatial relationships force models to reason about objects’ relative positions. However, as shown in Figure 8, some questions can be answered using absolute spatial reasoning. In this question the purple cube can be found by simply looking in the bottom half of the image; reasoning about its position relative to the metal sphere is unnecessary.
空间关系迫使模型去推理物体的相对位置。不过,如图 8 所示,有些问题可以用绝对空间推理来作答。在这道题里,只需看图像的下半部分就能找到那个紫色立方体,不必再推理它相对于金属球体的位置。
Questions only requiring absolute spatial reasoning can be identified by modifying the semantics of spatial relationship functions in their programs: instead of returning sets of objects related to the input object, they ignore their input object and return the set of objects in the half of the image corresponding to the relationship. A question only requires absolute spatial reasoning if executing its program with these modified semantics does not change its answer.
只需绝对空间推理就能解答的问题,可以通过修改程序里空间关系函数的语义来识别:这些函数不再返回与输入物体相关的物体集合,而是忽略输入物体,返回图像中与该关系相对应的那一半里的物体集合。若用这种修改后的语义执行程序,答案并不改变,这道题就只需要绝对空间推理。
The bottommost plots of Figure 8 show accuracy on chain-structured questions with different number of relationships, excluding questions that can be answered with absolute spatial reasoning. On query questions, CNN+LSTM+SA performs significantly worse when absolute spatial reasoning is excluded; on count questions no model outperforms LSTM, and on exist questions no model outperforms Q-type mode. These results suggest that models have not learned the semantics of spatial relationships.
图 8 最下面一组图给出了含有不同数目关系的链状问题的准确率,其中已排除可用绝对空间推理作答的问题。在查询问题上,排除绝对空间推理后,CNN+LSTM+SA 的表现明显变差;在计数问题上没有模型超过 LSTM,在存在性问题上没有模型超过问题类型众数。这些结果说明,模型并没有学会空间关系的语义。
4.7. Compositional Generalization
4.7. 组合泛化
Figure 9. In Condition A all cubes are gray, blue, brown, or yellow and all cylinders are red, green, purple, or cyan; in Condition B color palettes are swapped. We train models in Condition A and test in both conditions to assess their generalization performance. We show accuracy on “query color” and “query material” questions, separating questions by shape of the object being queried.
图 9. 在条件 A 中,所有立方体为灰、蓝、棕或黄,所有圆柱为红、绿、紫或青;在条件 B 中,两种形状的颜色调色板对调。我们在条件 A 上训练模型,并在两种条件下测试,以评估其泛化表现。我们给出“查询颜色”和“查询材质”问题的准确率,并按被查询物体的形状分开统计。
Practical VQA systems should perform well on images and questions that contain novel combinations of attributes not seen during training. To do so models might need to learn disentangled representations for attributes, for example learning separate representations for color and shape instead of memorizing all possible color/shape combinations.
实用的 VQA 系统应当能在包含训练时未见过的新属性组合的图像和问题上表现良好。要做到这一点,模型可能需要学习解开纠缠的属性表示,例如分别学习颜色和形状的表示,而不是把所有可能的颜色/形状组合都记下来。
We created the CLEVR Compositional Generalization Test (CoGenT) to test models for this ability. CLEVR-CoGenT contains two conditions: in Condition A all cubes are gray, blue, brown, or yellow and all cylinders are red, green, purple, or cyan; in Condition B these shapes swap color palettes. Both conditions have spheres of all colors.
我们为此制作了 CLEVR 组合泛化测试(CLEVR Compositional Generalization Test,CoGenT)。CLEVR-CoGenT 包含两个条件:在条件 A 中,所有立方体为灰、蓝、棕或黄,所有圆柱为红、绿、紫或青;在条件 B 中,这两种形状对调颜色调色板。两种条件下球体都具有全部颜色。
We retrain models on Condition A and compare their performance when testing on Condition A (A → A) and testing on Condition B (A → B). In Figure 9 we show accuracy on query-color and query-material questions, separating questions asking about spheres (which are the same in A and B) and cubes/cylinders (which change from A to B).
我们在条件 A 上重新训练模型,并比较它们在条件 A 上测试(A → A)和在条件 B 上测试(A → B)时的表现。图 9 给出查询颜色和查询材质问题的准确率,并把问球体的题(在 A 和 B 中相同)与问立方体/圆柱的题(从 A 到 B 会变化)分开。
Between A → A and A → B, all models perform about the same when asked about the color of spheres, but perform much worse when asked about the color of cubes or cylinders; CNN+LSTM+SA drops from 85% to 51%. Models seem to learn strong biases about the colors of objects and cannot overcome these biases when conditions change.
在 A → A 与 A → B 之间,所有模型在被问到球体颜色时表现大致相当,但在被问到立方体或圆柱颜色时会差很多;CNN+LSTM+SA 从 85% 降到 51%。模型似乎学到了关于物体颜色的很强偏差,条件一变就无法克服这些偏差。
When asked about the material of cubes and cylinders, CNN+LSTM+SA shows a smaller gap between A → A and A → B (90% vs 81%); other models show no gap. Having seen metal cubes and red metal objects during training, models can understand the material of red metal cubes.
当被问到立方体和圆柱的材质时,CNN+LSTM+SA 在 A → A 与 A → B 之间的差距较小(90% 对 81%);其他模型则没有差距。由于训练时见过金属立方体和红色金属物体,模型能够理解红色金属立方体的材质。
- Discussion and Future Work
- 讨论与未来工作
This paper has introduced CLEVR, a dataset designed to aid in diagnostic evaluation of visual question answering (VQA) systems by minimizing dataset bias and providing rich ground-truth representations for both images and questions. Our experiments demonstrate that CLEVR facilitates in-depth analysis not possible with other VQA datasets: our question representations allow us to slice the dataset along different axes (question type, relationship type, question topology, etc.), and comparing performance along these different axes allows us to better understand the reasoning capabilities of VQA systems. Our analysis has revealed several key shortcomings of current VQA systems:
本文介绍了 CLEVR。这套数据集的设计目的,是通过尽量减小数据集偏差,并为图像和问题提供丰富的真值表示,来帮助对视觉问答(VQA)系统做诊断性评估。我们的实验表明,CLEVR 能够支持其他 VQA 数据集做不到的深入分析:问题表示让我们可以沿不同轴线切分数据(问题类型、关系类型、问题拓扑等),再比较这些轴线上的表现,就能更好地理解 VQA 系统的推理能力。分析揭示了当前 VQA 系统的几项关键短板:
• Short-term memory: All systems we tested performed poorly in situations requiring short-term memory, including attribute comparison and integer equality questions (Section 4.2), same-attribute relationships (Section 4.3), and tree-structured questions (Section 4.4). Attribute comparison questions are of particular interest, since models can successfully identity attributes of objects but struggle to compare attributes.
• 短时记忆:我们测试的所有系统,在需要短时记忆的情形里都表现不佳,包括属性比较和整数相等问题(第 4.2 节)、相同属性关系(第 4.3 节)以及树状结构问题(第 4.4 节)。属性比较尤其值得注意:模型能够成功识别物体的属性,却难以比较这些属性。
• Long reasoning chains: Systems struggle to answer questions requiring long chains of nontrivial reasoning, including questions with large effective sizes (Section 4.5) and count and existence questions with many spatial relationships (Section 4.6).
• 长推理链:系统很难回答需要一长串非平凡推理的问题,包括有效规模很大的问题(第 4.5 节),以及含有许多空间关系的计数和存在性问题(第 4.6 节)。
• Spatial Relationships: Models fail to learn the true semantics of spatial relationships, instead relying on absolute image position (Section 4.6).
• 空间关系:模型没有学会空间关系的真正语义,而是依赖图像中的绝对位置(第 4.6 节)。
• Disentangled Representations: By training and testing models on different data distributions (Section 4.7) we argue that models do not learn representations that properly disentangle object attributes; they seem to learn strong biases from the training data and cannot overcome these biases when conditions change.
• 解开纠缠的表示:通过在不同数据分布上训练和测试模型(第 4.7 节),我们认为模型并没有学到能够妥善解开物体属性的表示;它们似乎从训练数据里学到了很强的偏差,条件一变就无法克服。
Our study also shows cases where current VQA systems are successful. In particular, spatial attention [44] allows models to focus on objects and identify their attributes even on questions requiring multiple steps of reasoning.
我们的研究也显示了当前 VQA 系统取得成功的情形。特别是,空间注意力 [44] 能够让模型把焦点放到物体上并识别其属性,即使问题需要多步推理也是如此。
These observations present clear avenues for future work on VQA. We plan to use CLEVR to study models with explicit short-term memory, facilitating comparisons between values [13, 18, 39, 42]; explore approaches that encourage learning disentangled representations [5]; and investigate methods that compile custom network architectures for different patterns of reasoning [2, 3]. We hope that diagnostic datasets like CLEVR will help guide future research in VQA and enable rapid progress on this important task.
这些观察为 VQA 的后续工作指出了清楚的方向。我们计划用 CLEVR 去研究带有显式短时记忆、便于比较取值的模型 [13, 18, 39, 42];探索鼓励学习解开纠缠表示的方法 [5];并考察为不同推理模式编译定制网络结构的方法 [2, 3]。我们希望,像 CLEVR 这样的诊断数据集能够帮助引导 VQA 的后续研究,并推动这一重要任务尽快取得进展。
Acknowledgments
致谢
We thank Deepak Pathak, Piotr Dollár, Ranjay Krishna, Animesh Garg, and Danfei Xu for helpful comments and discussion.
我们感谢 Deepak Pathak、Piotr Dollár、Ranjay Krishna、Animesh Garg 和 Danfei Xu 提出的有益意见与讨论。
References
参考文献
[1] A. Agrawal, D. Batra, and D. Parikh. Analyzing the behavior of visual question answering models. In EMNLP, 2016. 1
[2] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Learning to compose neural networks for question answering. In NAACL, 2016. 1, 2, 4, 5, 8
[3] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In CVPR, 2016. 1, 2, 4, 8
[4] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Zitnick, and D. Parikh. VQA: Visual question answering. In ICCV, 2015. 1, 2, 4, 5
[5] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. TPAMI, 35(8):1798–1828, 2014. 8
[6] Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam, 2016. 3
[7] J. Chen, P. Kuznetsova, D. Warren, and Y. Choi. Deja imagecaptions: A corpus of expressive image descriptions in repe- tition. In NAACL, 2015. 2
[8] A. Farhadi, M. Hejrati, A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences for images. In ECCV, 2010. 2
[9] A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In arXiv:1606.01847, 2016. 1, 4, 5
[10] H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? Dataset and methods for multilingual image question answering. In NIPS, 2015. 1, 2, 4
[11] Y. Gao, O. Beijbom, N. Zhang, and T. Darrell. Compact bilinear pooling. In CVPR, 2016. 5
[12] D. Geman, S. Geman, N. Hallonquist, and L. Younes. Visual Turing test for computer vision systems. Proceedings of the National Academy of Sciences, 112(12):3618–3623, 2015. 2
[13] A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwinska, S. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, A. Badia, K. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. . Kavukcuoglu, and D. Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature, 2016. 8
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 5
[15] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. 2, 5
[16] A. Jabri, A. Joulin, and L. van der Maaten. Revisiting visual question answering baselines. In ECCV, 2016. 1, 4
[17] J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015. 3
[18] A. Joulin and T. Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In NIPS, 2015. 8
[19] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014. 2
[20] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 5
[21] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Jia-Li, D. Shamma, M. Bernstein, and L. Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2016. 1, 2, 3
[22] H. J. Levesque, E. Davis, and L. Morgenstern. The Winograd schema challenge. In AAAI Spring Symposium: Logi- cal Formalizations of Commonsense Reasoning, volume 46, page 47, 2011. 2
[23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. Zitnick. Microsoft COCO: Com- mon objects in context. In ECCV, 2014. 2
[24] J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, 2016. 1, 2, 4
[25] L. Ma, Z. Lu, and H. Li. Learning to answer questions from image using convolutional neural network. In AAAI, 2016. 4
[26] M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014. 1, 2
[27] M. Malinowski and M. Fritz. Towards a visual Turing challenge. In NIPS 2014 Workshop on Learning Semantics, 2014. 2
[28] M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. In ICCV, 2015. 2, 4
[29] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. In arXiv 1301.3781, 2013. 5
[30] O. Pfungst. Clever Hans (The horse of Mr. von Osten): A contribution to experimental animal and human psychology. Henry Holt, New York, 1911. 1
[31] A. Ray, G. Christie, M. Bansal, D. Batra, and D. Parikh. Question relevance in vqa: Identifying non-visual and falsepremise questions. In EMNLP, 2016. 1
[32] M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. In NIPS, 2015. 1, 2, 4
[33] K. Shih, S. Singh, and D. Hoiem. Where to look: Focus regions for visual question answering. In CVPR, 2016. 4
[34] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929–1958, 2014. 5
[35] B. Sturm. A simple method to determine if a music information retrieval system is a horse. IEEE Transactions on Multimedia, 16(6):1636–1644, 2014. 1
[36] B. Sturm. Horse taxonomy and taxidermy. HORSE2016, 2016. 1
[37] M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. Movieqa: Understanding stories in movies through question-answering. In CVPR, 2016. 2
[38] J. Weston, A. Bordes, S. Chopra, A. Rush, B. van Merri¨enboer, A. Joulin, and T. Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks. In ICLR, 2016. 2
[39] J. Weston, S. Chopra, and A. Bordes. Memory networks. In ICLR, 2015. 8
[40] T. Winograd. Understanding Natural Language. Academic Press, 1972. 2
[41] Q. Wu, C. Shen, A. van den Hengel, P. Wang, and A. Dick. Image captioning and visual question answering based on attributes and their related external knowledge. In arXiv 1603.02814, 2016. 4
[42] C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. ICML, 2016. 4, 8
[43] H. Xu and K. Saenko. Ask, attend, and answer: Exploring question-guided spatial attention for visual question answering. In ECCV, 2016. 4
[44] Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. In CVPR, 2016. 1, 2, 4, 5, 8
[45] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. In TACL, pages 67–78, 2014. 2
[46] L. Yu, E. Park, A. Berg, and T. Berg. Visual madlibs: Fill in the blank image generation and question answering. In ICCV, 2015. 1, 2
[47] P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and yang: Balancing and answering binary visual questions. In CVPR, 2016. 1, 2
[48] B. Zhou, Y. Tian, S. Sukhbataar, A. Szlam, and R. Fergus. Simple baseline for visual question answering. In arXiv:1512.02167, 2015. 1, 4, 5
[49] Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7w: Grounded question answering in images. In CVPR, 2016. 1, 2, 4
[50] C. Zitnick and D. Parikh. Bringing semantics into focus using visual abstraction. In CVPR, 2013. 2








