Visual Programming: Compositional visual reasoning without training
Visual Programming: Compositional visual reasoning without training
视觉编程：无需训练的组合式视觉推理
Tanmay Gupta Affiliation: PRIOR @ Allen Institute for AI Aniruddha Kembhavi Affiliation: https://prior.allenai.org/projects/visprog
Abstract
摘要
We present VisProg , a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. VisProg avoids the need for any task-specific training. Instead, it uses the in-context learning ability of large language models to generate python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing subroutines, or python functions to produce intermediate outputs that may be consumed by subsequent parts of the program. We demonstrate the flexibility of VisProg on 4 diverse tasks - compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing. We believe neuro-symbolic approaches like VisProg are an exciting avenue to easily and effectively expand the scope of AI systems to serve the long tail of complex tasks that people may wish to perform.
我们提出 VisProg：一种神经符号方法，在给定自然语言指令时求解复杂、组合式的视觉任务。VisProg 不需要任何面向具体任务的训练，而是利用大语言模型的上下文学习能力，生成类 Python 的模块化程序，再执行该程序同时得到答案与可解释的推理过程。程序中每一行可调用若干现成的计算机视觉模型、图像处理子程序或 Python 函数，产生供后续步骤消费的中间结果。我们在四类任务上展示了 VisProg 的灵活性：组合式视觉问答、图像对上的零样本推理、基于事实知识的目标标注，以及语言引导的图像编辑。我们认为，像 VisProg 这样的神经符号路径，能把 AI 系统的覆盖面轻松而有效地扩展到人们希望完成的那些长尾复杂任务。
Figure 1: VisProg****is a modular and interpretable neuro-symbolic system for compositional visual reasoning. Given a few examples of natural language instructions and the desired high-level programs, VisProg generates a program for any new instruction using in-context learning in GPT-3 and then executes the program on the input image(s) to obtain the prediction. VisProg also summarizes the intermediate outputs into an interpretable visual rationale (Fig. 4). We demonstrate VisProg on tasks that require composing a diverse set of modules for image understanding and manipulation, knowledge retrieval, and arithmetic and logical operations.
图 1：VisProg 是面向组合式视觉推理的模块化、可解释神经符号系统。给定少量自然语言指令与对应高层程序的示例后，VisProg 借助 GPT-3 的上下文学习为任意新指令生成程序，再在输入图像上执行以得到预测。VisProg 还会把中间输出汇总成可解释的视觉理据（图 4）。我们在需要组合多样模块来做图像理解与操控、知识检索以及算术与逻辑运算的任务上验证了 VisProg。
1 Introduction
1 引言
The pursuit of general purpose AI systems has lead to the development of capable end-to-end trainable models[1, 19, 8, 13, 25, 5, 27], many of which aspire to provide a simple natural language interface for a user to interact with the model. The predominant approach to building these systems has been massive-scale unsupervised pretraining followed by supervised multitask training. However, this approach requires a well curated dataset for each task that makes it challenging to scale to the infinitely long tail of complex tasks we would eventually like these systems to perform. In this work, we explore the use of large language models to tackle the long tail of complex tasks by decomposing these tasks described in natural language into simpler steps that may be handled by specialized end-to-end trained models or other programs.
追求通用 AI 系统，催生了一批能力很强的端到端可训练模型[1, 19, 8, 13, 25, 5, 27]，其中许多希望用简单的自然语言接口与用户交互。主流做法是大规模无监督预训练，再做有监督多任务训练。但这要求每个任务都有精心整理的数据集，难以覆盖我们最终希望系统能做的那条无限长的复杂任务长尾。本文探索用大语言模型来啃这条长尾：把自然语言描述的复杂任务拆成更简单的步骤，再交给专门的端到端模型或其他程序去处理。
Imagine instructing a vision system to “Tag the 7 main characters on the TV show Big Bang Theory in this image.” To perform this task, the system first needs to understand the intent of the instruction and then perform a sequence of steps - detect the faces, retrieve list of main characters on Big Bang Theory from a knowledge base, classify faces using the list of characters, and tag the image with recognized character’s faces and names. While different vision and language systems exist to perform each of these steps, executing this task described in natural language is beyond the scope of end-to-end trained systems.
设想你对视觉系统说：「给这张图里《生活大爆炸》的七位主要角色打上标签。」系统首先要理解指令意图，再依次：检测人脸、从知识库取出该剧主要角色名单、用名单做人脸分类、再把识别到的角色脸与名字标回图像。上述每一步都已有视觉或语言系统可用，但用自然语言直接完成整条任务，仍超出端到端训练系统的能力范围。
We introduce VisProg which inputs visual data (a single image or a set of images) along with a natural language instruction, generates a sequence of steps, a visual program if you will, and then executes these steps to produce the desired output. Each line in a visual program invokes one among a wide range of modules currently supported by the system. Modules may be off-the-shelf computer vision models, language models, image processing subroutines in OpenCV [4], or arithmetic and logical operators. Modules consume inputs that are produced by executing previous lines of code and output intermediate results that can be consumed downstream. In the example above, the visual program generated by VisProg invokes a face detector [18], GPT-3 [5] as a knowledge retrieval system, and CLIP [23] as an open-vocabulary image classifier to produce the desired output (see Fig. 1).
我们提出 VisProg：它接收视觉数据（单张图或一组图）与自然语言指令，生成一串步骤——不妨称作视觉程序——再执行这些步骤得到目标输出。程序每一行调用系统当前支持的某个模块；模块可以是现成的计算机视觉模型、语言模型、OpenCV [4] 中的图像处理子程序，或算术与逻辑算子。模块消费执行前序语句得到的输入，并输出可供下游使用的中间结果。在上面的例子里，VisProg 生成的视觉程序会调用人脸检测器 [18]、把 GPT-3 [5] 当作知识检索系统，以及把 CLIP [23] 当作开放词汇图像分类器，从而得到期望输出（见图 1）。
Figure 2: Modules currently supported in VisProg . Red modules use neural models (OWL-ViT [21], DSFD [18], MaskFormer [6], CLIP [23], ViLT [16], and Stable Diffusion [28]). Blue modules use image processing and other python subroutines. These modules are invoked in programs generated from natural language instructions. Adding new modules to extend VisProg ’s capabilities is straightforward (Code. ).
图 2：VisProg 当前支持的模块。 红色模块使用神经模型（OWL-ViT [21]、DSFD [18]、MaskFormer [6]、CLIP [23]、ViLT [16] 与 Stable Diffusion [28]）。蓝色模块使用图像处理及其他 Python 子程序。这些模块由自然语言指令生成的程序调用。扩展 VisProg 能力时，新增模块很直接（代码 1）。
VisProg improves upon previous methods for generating and executing programs for vision applications. For the visual question answering (VQA) task, Neural Module Networks (NMN) [2, 10, 12, 9] compose a question-specific, end-to-end trainable network from specialized, differentiable neural modules. These approaches either use brittle, off-the-shelf semantic parsers to deterministically compute the layout of modules, or learn a layout generator through weak answer supervision via Reinforce [33]. In contrast, VisProg uses a powerful language model (GPT-3 ) and a small number of in-context examples to create complex programs without requiring any training[^1]. Programs created by VisProg also use a higher-level of abstraction than NMNs and invoke trained state-of-the-art models and non-neural python subroutines (Fig. 2). These advantages make VisProg an easy-to-use, performant, and modular neuro-symbolic system.
VisProg 改进了以往为视觉应用生成并执行程序的方法。在视觉问答（VQA）任务上，神经模块网络（NMN）[2, 10, 12, 9] 会从可微的专用神经模块拼出一道题专用的端到端网络。这类方法要么用脆弱的现成语义解析器确定性计算模块布局，要么借助 Reinforce [33] 与弱答案监督去学布局生成器。相比之下，VisProg 使用强大的语言模型（GPT-3）与少量上下文示例即可创建复杂程序，无需任何训练[^1]。VisProg 生成的程序抽象层级也高于 NMN，会调用已训练好的先进模型与非神经的 Python 子程序（图 2）。这些优势使 VisProg 成为易用、有效且模块化的神经符号系统。
VisProg is also highly interpretable. First, VisProg produces easy-to-understand programs which a user can verify for logical correctness. Second, by breaking down the prediction into simple steps, VisProg allows a user to inspect the outputs of intermediate steps to diagnose errors and if required, intervene in the reasoning process. Altogether, an executed program with intermediate step results (e.g. text, bounding boxes, segmentation masks, generated images, etc.) linked together to depict the flow of information serves as a visual rationale for the prediction.
VisProg 的可解释性也很强。第一，它产出易于理解的程序，用户可核对逻辑是否正确。第二，把预测拆成简单步骤后，用户可检查中间输出以诊断错误，必要时还能介入推理过程。整体上，把执行过的程序与中间步骤结果（如文本、框、分割掩码、生成图像等）串起来，就构成对预测的视觉理据。
To demonstrate its flexibility, we use VisProg for 4 different tasks that share some common skills (e.g. for image parsing) while also requiring some degree of specialized reasoning and visual manipulation capabilities. These tasks are - (i) compositional visual question answering; (ii) zero-shot natural language visual reasoning (NLVR) on image pairs; (iii) factual knowledge object tagging from natural language instructions; and (iv) language-guided image editing. We emphasize that neither the language model nor any of the modules are finetuned in any way. Adapting VisProg to any task is as simple as providing a few in-context examples consisting of natural language instructions and the corresponding programs. While easy to use, VisProg shows an impressive gain of points over a base VQA model on the compositional VQA task, strong zero-shot accuracy of on NLVR without ever training on image pairs, and delightful qualitative and quantitative results on knowledge tagging and image editing tasks.
为展示灵活性，我们在四项任务上使用 VisProg；它们共享部分技能（如图像解析），又各自需要专门的推理与视觉操控能力。这四项是：(i) 组合式视觉问答；(ii) 图像对上的零样本自然语言视觉推理（NLVR）；(iii) 由自然语言指令驱动的事实知识目标标注；以及 (iv) 语言引导的图像编辑。我们强调：语言模型与各个模块都没有做任何微调。把 VisProg 适配到任意任务，只需提供若干由自然语言指令与对应程序组成的上下文示例。尽管用法简单，VisProg 在组合式 VQA 上相对基座 VQA 模型提升 个点，在从未用图像对训练的情况下于 NLVR 上达到 的零样本准确率，并在知识标注与图像编辑上给出令人满意的定性与定量结果。
Our key contributions include - (i) VisProg - a system that uses the in-context learning ability of a language model to generate visual programs from natural language instructions for compositional visual tasks (Sec. 3); (ii) demonstrating the flexibility of VisProg on complex visual tasks such as factual knowledge object tagging and language guided image editing (Secs. 4.3 and 4.4) that have eluded or seen limited success with a single end-to-end model; and (iii) producing visual rationales for these tasks and showing their utility for error analysis and user-driven instruction tuning to improve VisProg ’s performance significantly (Sec. 5.3).
主要贡献包括：(i) VisProg——利用语言模型的上下文学习能力，从自然语言指令为组合式视觉任务生成视觉程序的系统（第 3 节）；(ii) 在事实知识目标标注与语言引导图像编辑等复杂视觉任务上展示灵活性（第 4.3、4.4 节），这些任务单靠一个端到端模型往往难以或很少成功；(iii) 为这些任务产出视觉理据，并说明它们对误差分析与用户驱动的指令调优有用，从而显著提升 VisProg 表现（第 5.3 节）。
2 Related Work
2 相关工作
Neuro-symbolic approaches have seen renewed momentum owing to the incredible understanding, generation, and in-context learning capabilities of large language models (LLMs). We now discuss previous program generation and execution approaches for visual tasks, recent work in using LLMs for vision, and advances in reasoning methods for language tasks.
大语言模型在理解、生成与上下文学习上的能力，让神经符号方法重新受到关注。下面讨论面向视觉任务的程序生成与执行、把 LLM 用于视觉的近期工作，以及语言任务上推理方法的进展。
Program generation and execution for visual tasks. Neural module networks (NMN) [2] pioneered modular and compositional approaches for the visual question answering (VQA) task. NMNs compose neural modules into an end-to-end differentiable network. While early attempts use off-the-shelf parsers [2], recent methods [10, 12, 9] learn the layout generation model jointly with the neural modules using Reinforce [33] and weak answer supervision. While similar in spirit to NMNs, VisProg has several advantages over NMNs. First, VisProg generates high-level programs that invoke trained state-of-the-art neural models and other python functions at intermediate steps as opposed to generating end-to-end neural networks. This makes it easy to incorporate symbolic, non-differentiable modules. Second, VisProg leverages the in-context learning ability of LLMs [5] to generate programs by prompting the LLM (GPT-3 ) with a natural language instruction (or a visual question or a statement to be verified) along with a few examples of similar instructions and their corresponding programs thereby removing the need to train specialized program generators for each task.
面向视觉任务的程序生成与执行。 神经模块网络（NMN） [2] 开创了 VQA 上的模块化与组合式路径：把神经模块拼成端到端可微网络。早期工作用现成解析器 [2]，较新方法 [10, 12, 9] 则借助 Reinforce [33] 与弱答案监督，与神经模块联合学习布局生成。VisProg 精神相近，但有若干优势。第一，它生成高层程序，在中间步骤调用已训练好的先进神经模型及其他 Python 函数，而不是生成整网可微的端到端网络，因而易于纳入符号式、不可微模块。第二，它借助 LLM [5] 的上下文学习：用自然语言指令（或视觉问题、待验证陈述）加上少量同类指令与程序示例提示 LLM（GPT-3）即可生成程序，无需为每个任务训练专用程序生成器。
LLMs for visual tasks. LLMs and in-context learning have been applied to visual tasks. PICa [34] uses LLMs for a knowledge-based VQA [20] task. PICa represents the visual information in images as text via captions, objects, and attributes and feeds this textual representation to GPT-3 along with the question and in-context examples to directly generate the answer. Socratic models (SMs) [36], compose pretrained models from different modalities such as language (BERT [7], GPT-2 [24]), vision-language (CLIP [23]), and audio-language (mSLAM [3]), to perform a number of zero-shot tasks, including image captioning, video-to-text retrieval, and robot planning. However, in SMs the composition is pre-determined and fixed for each task. In contrast, VisProg determines how to compose models for each instance by generating programs based on the instruction, question, or statement. We demonstrate VisProg ’s ability to handle complex instructions that involve diverse capabilities (20 modules) and varied input (text, image, and image pairs), intermediate (text, image, bounding boxes, segmentation masks), and output modalities (text and images). Similar to VisProg , ProgPrompt [29] is a concurrent work that demonstrates the ability of LLMs to generate python-like situated robot action plans from natural language instructions. While ProgPrompt modules (such as “find” or “grab”) take strings (typically object names) as input, VisProg programs are more general. In each step in a VisProg program, a module could accept multiple arguments including strings, numbers, arithmetic and logical expressions, or arbitrary python objects (such as list() or dict() instances containing bounding boxes or segmentation masks) produced by previous steps.
面向视觉任务的 LLM。 LLM 与上下文学习已被用于视觉任务。PICa [34] 把图像信息转成标题、物体与属性等文本，连同问题与上下文示例送入 GPT-3，直接生成知识型 VQA [20] 的答案。苏格拉底模型（SMs） [36] 组合来自语言（BERT [7]、GPT-2 [24]）、视觉–语言（CLIP [23]）与音频–语言（mSLAM [3]）等模态的预训练模型，完成图像描述、视频–文本检索与机器人规划等零样本任务；但在 SMs 里，每个任务的组合方式是预先固定的。相反，VisProg 根据指令、问题或陈述为每个实例决定如何组合模型。我们展示它能处理涉及多样能力（20 个模块）、多种输入（文本、单图、图像对）、中间结果（文本、图像、框、分割掩码）与输出模态（文本与图像）的复杂指令。与 VisProg 同期的 ProgPrompt [29] 表明 LLM 能从自然语言指令生成类 Python 的机器人动作计划；ProgPrompt 的模块（如 “find”“grab”）通常以字符串（多为物体名）为输入，而 VisProg 的程序更一般：每一步可接受多个参数，包括字符串、数字、算术与逻辑表达式，或由前序步骤产生的任意 Python 对象（如含框或分割掩码的 list()/dict()）。
Reasoning via Prompting in NLP. There is a growing body of literature [14, 17] on using LLMs for language reasoning tasks via prompting. Chain-of-Thought (CoT) prompting [32], where a language model is prompted with in-context examples of inputs, chain-of-thought rationales (a series of intermediate reasoning steps), and outputs, has shown impressive abilities for solving math reasoning problems. While CoT relies on the ability of LLMs to both generate a reasoning path and execute it, approaches similar to VisProg have been applied to language tasks, where a decomposer pompt [15] is used first to generate a sequence of sub-tasks which are then handled by sub-task handlers.
NLP 中的提示式推理。 用 LLM 通过提示做语言推理的文献日益增多 [14, 17]。思维链（CoT）提示 [32] 在上下文中给出输入、思维链理据（一系列中间推理步骤）与输出，在数学推理上表现突出。CoT 依赖 LLM 既生成推理路径又执行它；与 VisProg 类似的做法也已用于语言任务：先用分解提示 [15] 生成子任务序列，再交给子任务处理器。
3 Visual Programming
3 视觉编程
Figure 3: Program generation in VisProg .
图 3：VisProg 中的程序生成。
Over the last few years, the AI community has produced high-performance, task-specific models for many vision and language tasks such as object detection, segmentation, VQA, captioning, and text-to-image generation. While each of these models solves a well-defined but narrow problem, the tasks we usually want to solve in the real world are often broader and loosely defined.
过去几年，AI 社区为检测、分割、VQA、描述、文生图等许多视觉与语言任务产出了高性能、任务专用的模型。这些模型各自解决定义清晰但狭窄的问题，而现实中我们想解决的任务通常更宽、定义也更松。
To solve such practical tasks, one has to either collect a new task-specific dataset, which can be expensive, or meticulously compose a program that invokes multiple neural models, image processing subroutines (e.g. image resizing, cropping, filtering, and colorspace conversions), and other computation (e.g. database lookup, or arithmetic and logical operations). Manually creating these programs for the infinitely long tail of complex tasks we encounter daily not only requires programming expertise but is also slow, labor intensive, and ultimately insufficient to cover the space of all tasks. What if, we could describe the task in natural language and have an AI system generate and execute the corresponding visual program without any training?
要解决这类实用任务，要么为新任务再采一套昂贵的数据集，要么仔细手写一段程序，调用多个神经模型、图像处理子程序（如缩放、裁剪、滤波、色彩空间转换）以及其他计算（如查库、算术与逻辑运算）。为日常遇到的无限长尾复杂任务手工写程序，既需要编程专长，又慢又费力，终究也盖不全任务空间。若能用自然语言描述任务，让 AI 系统无需训练就生成并执行对应的视觉程序，会怎样？
Large language models for visual programming. Large language models such as GPT-3 have shown a remarkable ability to generalize to new samples for a task having seen a handful of input and output demonstrations in-context. For example, prompting GPT-3 with two English-to-French translation examples and a new English phrase
面向视觉编程的大语言模型。 像 GPT-3 这样的大语言模型，在上下文中只看过少量输入–输出示范后，就能泛化到同一任务的新样本。例如，用两条英译法示例加上一句新的英文短语提示 GPT-3：
produces the French translation “bonsoir”. Note that we did not have to finetune GPT-3 to perform the task of translation on the thrid phrase. VisProg uses this in-context learning ability of GPT-3 to output visual programs for natural language instructions.
会得到法文翻译 “bonsoir”。注意我们不必为了翻译第三句而去微调 GPT-3。VisProg 正是利用 GPT-3 的这种上下文学习能力，为自然语言指令输出视觉程序。
Similar to English and French translation pairs in the example above, we prompt GPT-3 with pairs of instructions and the desired high-level program. Fig. 3 shows such a prompt for an image editing task. The programs in the in-context examples are manually written and can typically be constructed without an accompanying image. Each line of a VisProg program, or a program step, consists of the name of a module, module’s input argument names and their values, and an output variable name. VisProg programs often use output variables from past steps as inputs to future steps. We use descriptive module names (e.g. “Select”, “ColorPop”, “Replace”), argument names (e.g. “image”, “object”, “query”), and variable names (e.g. “IMAGE”, “OBJ”) to allow GPT-3 to understand the input and output type, and function of each module. During execution the output variables may be used to store arbitrary data types. For instance “OBJ”s are list of objects in the image, with mask, bounding box, and text (e.g. category name) associated with each object.
与上面的英法对照类似，我们用指令与期望高层程序的成对示例提示 GPT-3。图 3 给出图像编辑任务的此类提示。上下文示例中的程序为手工编写，通常不必对着真实图像也能写出来。VisProg 程序的每一行（或称程序步骤）包含模块名、模块输入参数名与取值，以及输出变量名。程序常把过去步骤的输出变量用作后续输入。我们使用描述性的模块名（如 “Select”“ColorPop”“Replace”）、参数名（如 “image”“object”“query”）与变量名（如 “IMAGE”“OBJ”），以便 GPT-3 理解每个模块的输入输出类型与功能。执行时，输出变量可存任意数据类型：例如 “OBJ” 是图像中物体的列表，每个物体带有掩码、边界框与文本（如类别名）。
These in-context examples are fed into GPT-3 along with a new natural language instruction. Without observing the image or its content, VisProg generates a program (bottom of Fig. 3) that can be executed on the input image(s) to perform the described task.
这些上下文示例与新的自然语言指令一并送入 GPT-3。在不观察图像或其内容的情况下，VisProg 生成一段可在输入图像上执行以完成所述任务的程序（图 3 底部）。
class VisProgModule():
def init(self):
load a trained model; move to GPU
def html(self,inputs: List,output: Any):
return an html string visualizing step I/O
def parse(self,step: str):
parse step and return list of input values
and variables, and output variable name
def execute(self,step: str,state: Dict):
inputs, input_var_names, output_var_name =
self.parse(step)
get values of input variables from state
for var_name in input_var_names:
inputs.append(state[var_name])
perform computation using the loaded model
output = some_computation(inputs)
update state
state[output_var_name] = output
visual summary of the step computation
step_html = self.html(inputs,output)
return output, step_html
Code 1: Implementation of a VisProg module.
class VisProgModule():
def init(self):
加载训练好的模型；移到 GPU
def html(self,inputs: List,output: Any):
返回可视化该步输入/输出的 html 字符串
def parse(self,step: str):
解析步骤，返回输入取值与变量列表，以及输出变量名
def execute(self,step: str,state: Dict):
inputs, input_var_names, output_var_name =
self.parse(step)
从 state 取输入变量的值
for var_name in input_var_names:
inputs.append(state[var_name])
用已加载模型做计算
output = some_computation(inputs)
更新 state
state[output_var_name] = output
该步计算的可视化摘要
step_html = self.html(inputs,output)
return output, step_html
代码 1：VisProg 模块的实现。
Modules. VisProg currently supports 20 modules (Fig. 2) for enabling capabilities such as image understanding, image manipulation (including generation), knowledge retrieval, and performing arithmetic and logical operations. In VisProg , each module is implemented as a Python class (Code. ) that has methods to: (i) parse the line to extract the input argument names and values, and the output variable name; (ii) execute the necessary computation that may involve trained neural models and update the program state with the output variable name and value; and (iii) summarize the step’s computation visually using html (used later to create a visual rationale). Adding new modules to VisProg simply requires implementing and registering a module class, while the execution of the programs using this module is handled automatically by the VisProg interpreter, which is described next.
模块。 VisProg 当前支持 20 个模块（图 2），覆盖图像理解、图像操控（含生成）、知识检索以及算术与逻辑运算等能力。每个模块实现为一个 Python 类（代码 1），提供：(i) parse——解析该行，抽出输入参数名与取值以及输出变量名；(ii) execute——执行可能涉及已训练神经模型的计算，并用输出变量名与值更新程序状态；(iii) 用 html 可视化该步计算（随后用于构建视觉理据）。向 VisProg 添加新模块只需实现并注册模块类；使用该模块的程序执行由下文所述的 VisProg 解释器自动处理。
Figure 4: Visual rationales generated by VisProg . These rationales visually summarize the input and output of each computational step in the generated program during inference for an image editing (top) and NLVR task (bottom).
图 4：VisProg 生成的视觉理据。 这些理据在推理时可视化地汇总生成程序中每一步计算的输入与输出；上图为图像编辑任务，下图为 NLVR 任务。
Program Execution. The program execution is handled by an interpreter. The interpreter initializes the program state (a dictionary mapping variables names to their values) with the inputs, and steps through the program line-by-line while invoking the correct module with the inputs specified in that line. After executing each step, the program state is updated with the name and value of the step’s output.
程序执行。 程序由解释器执行。解释器用输入初始化程序状态（变量名到值的字典），再逐行步进，按该行指定的输入调用正确模块。每步执行后，用该步输出的名称与值更新程序状态。
Visual Rationale. In addition to performing the necessary computation, each module class also implements a method called html() to visually summarize the inputs and outputs of the module in an HTML snippet. The interpreter simply stitches the HTML summary of all program steps into a visual rationale (Fig. 4) that can be used to analyze the logical correctness of the program as well as inspect the intermediate outputs. The visual rationales also enable users to understand reasons for failure and tweak the natural language instructions minimally to improve performance. See Sec. 5.3 for more details.
视觉理据。 除必要计算外，每个模块类还实现 html()，用 HTML 片段可视化该模块的输入与输出。解释器把各程序步骤的 HTML 摘要拼成视觉理据（图 4），既可分析程序逻辑是否正确，也可检查中间输出。视觉理据还能帮助用户理解失败原因，并对自然语言指令做最小改动以提升表现。详见第 5.3 节。
4 Tasks
4 任务
VisProg provides a flexible framework that can be applied to a diverse range of complex visual tasks. We evaluate VisProg on 4 tasks that require capabilities ranging from spatial reasoning, reasoning about multiple images, knowledge retrieval, and image generation and manipulation. Fig. 5 summarizes the inputs, outputs, and modules used for these tasks. We now describe these tasks, their evaluation settings, and the choice of in-context examples.
VisProg 提供可应用于多种复杂视觉任务的灵活框架。我们在四项任务上评估它，这些任务分别需要空间推理、多图推理、知识检索以及图像生成与操控等能力。图 5 汇总了各任务的输入、输出与所用模块。下面描述任务、评估设定与上下文示例的选择。
Figure 5: We evaluate VisProg on a diverse set of tasks. The tasks span a variety of inputs and outputs and reuse modules (Loc, FaceDet, VQA) whenever possible.
图 5：我们在一组多样任务上评估 VisProg。 任务覆盖多种输入与输出，并在可能时复用模块（Loc、FaceDet、VQA）。
4.1 Compositional Visual Question Answering
4.1 组合式视觉问答
VisProg is compositional by construction which makes it suitable for the compositional, multi-step visual question answering task: GQA [11]. Modules for the GQA task include those for open vocabulary localization, a VQA module, functions for cropping image regions given bounding box co-ordinates or spatial prepositions (such as above, left, etc.), module to count boxes, and a module to evaluate Python expressions. For example, consider the question: “Is the small truck to the left or to the right of the people that are wearing helmets?”. VisProg first localizes “people wearing helmets”, crops the region to the left (or right) of these people, checks if there is a “small truck” on that side, and return “left” if so and “right” otherwise. VisProg uses the question answering module based on ViLT [16], but instead of simply passing the complex original question to ViLT , VisProg invokes it for simpler tasks like identifying the contents within an image patch. As a result, our resulting VisProg for GQA is not only more interpretable than ViLT but also more accurate (Tab. 1). Alternatively, one could completely eliminate the need for a QA model like ViLT and use other systems like CLIP and object detectors, but we leave that for future investigation.
VisProg 在构造上就是组合式的，因而适合 GQA [11] 这类组合式、多步视觉问答。GQA 所用模块包括开放词汇定位、VQA 模块、按边界框坐标或空间介词（如 above、left 等）裁剪图像区域的函数、数框模块，以及求值 Python 表达式的模块。例如问题：「戴头盔的那些人左边还是右边有小卡车？」VisProg 先定位「戴头盔的人」，裁出这些人左侧（或右侧）区域，检查该侧是否有「小卡车」，有则返回 “left”，否则 “right”。VisProg 使用基于 ViLT [16] 的问答模块，但并不把复杂原问题直接丢给 ViLT，而是在更简单的子任务上调用它，例如识别某个图像块里的内容。因此，面向 GQA 的 VisProg 不仅比单独的 ViLT 更可解释，也更准确（表 1）。也可以完全不用像 ViLT 这样的 QA 模型，改用 CLIP 与目标检测器等，我们留给后续研究。
Evaluation. In order to limit the money spent on generating programs with GPT-3 , we create a subset of GQA for evaluation. Each question in GQA is annotated with a question type. To evaluate on a diverse set of question types ( detailed types), we randomly sample up to samples per question type from the balanced val () and testdev () sets.
评估。 为控制用 GPT-3 生成程序的费用，我们构造了 GQA 的评估子集。GQA 每道题标有问题类型。为覆盖多样题型（约 种细类），我们从平衡 val（）与 testdev（）集中按题型最多随机抽 条。
Figure 6: Qualitative results for image editing (top) and knowledge tagging tasks (bottom).
图 6：图像编辑（上）与知识标注（下）任务的定性结果。
Prompts. We manually annotate random questions from the balanced train set with desired VisProg programs. Annotating questions with programs is easy and requires writing down the chain of reasoning required to answer that particular question. We provide a smaller subset of in context examples to GPT-3 , randomly sampled from this list to reduce the cost of answering each GQA question.
提示。 我们从平衡 train 集随机抽取 道题，手工标注期望的 VisProg 程序。为问题写程序并不难，只需写下回答该题所需的推理链。为降低每道 GQA 题的费用，送给 GPT-3 的是从该列表随机抽样得到的更小上下文集。
4.2 Zero-Shot Reasoning on Image Pairs
4.2 图像对上的零样本推理
VQA models are trained to answer questions about a single image. In practice, one might require a system to answer questions about a collection of images. For example, a user may ask a system to parse their vacation photo album and answer the question: “Which landmark did we visit, the day after we saw the Eiffel Tower?”. Instead of assembling an expensive dataset and training a multi-image model, we demonstrate the ability of VisProg to use a single-image VQA system to solve a task involving multiple images without training on multi-image examples.
VQA 模型通常针对单张图像问答来训练。实践中，系统可能需要对一组图像作答。例如用户让系统解析假期相册并回答：「我们看完埃菲尔铁塔的第二天去了哪座地标？」与其花大钱采数据再训多图模型，我们展示 VisProg 能在从未用多图样本训练的情况下，借助单图 VQA 系统解决涉及多图的任务。
We showcase this ability on the NLVRv2 [30] benchmark, which involves verifying statements about image pairs. Typically, tackling the NLVRv2 challenge requires training custom architectures that take image pairs as input on NLVRv2 ’s train set. Instead, VisProg achieves this by decomposing a complex statement into simpler questions about individual images and a python expression involving arithmetic and logical operators and answers to the image-level questions. The VQA model ViLT-vqa is used to get image-level answers, and the python expression is evaluated to verify the statement.
我们在 NLVRv2 [30] 基准上展示这一能力：该基准要求验证关于图像对的陈述。通常需要在 NLVRv2 训练集上训练接受图像对输入的定制架构。VisProg 则把复杂陈述分解为关于单张图像的简单问题，以及涉及算术与逻辑算子、并以图像级答案为操作数的 Python 表达式；用 ViLT-vqa 得到图像级答案，再求值该表达式以验证陈述。
Evaluation. We create a small validation set by sampling random samples from the NLVRv2 dev set to guide prompt selection, and test generalization on NLVRv2 ’s full public test set.
评估。 我们从 NLVRv2 dev 集随机抽 条构成小验证集以指导提示选择，并在完整公开 test 集上测泛化。
Prompts. We sample and annotate VisProg programs for random statements in the NLVRv2 train set. Since some of these examples are redundant (similar program structure) we also create a curated subset of examples by removing redundant ones.
提示。 我们从 NLVRv2 train 集随机抽 条陈述并标注 VisProg 程序。由于部分示例冗余（程序结构相似），我们再去掉 条冗余，得到 条的精选集。
4.3 Factual Knowledge Object Tagging
4.3 事实知识目标标注
We often want to identify people and objects in images whose names are unknown to us. For instance, we might want to identify celebrities, politicians, characters in TV shows, flags of countries, logos of corporations, popular cars and their manufacturers, species of organisms, and so on. Solving this task requires not only localizing people, faces, and objects but also looking up factual knowledge in an external knowledge base to construct a set of categories for classification, such as names of the characters on a TV show. We refer to this task as Factual Knowledge Object Tagging or Knowledge Tagging for short.
我们常想辨认图中不知其名的人或物：明星、政客、剧中角色、国旗、公司 logo、车型与厂商、生物物种等。完成该任务不仅要定位人、脸与物体，还要在外部知识库中查找事实，以构造分类用的类别集合，例如某部剧的角色名。我们称之为事实知识目标标注，简称知识标注。
For solving Knowledge Tagging, VisProg uses GPT-3 as an implicit knowledge base that can be queried with natural language prompts such as “List the main characters on the TV show Big Bang Theory separated by commas.” This generated category list can then be used by a CLIP image classification module that classifies image regions produced by localization and face detection modules. VisProg ’s program generator automatically determines whether to use a face detector or an open-vocabulary localizer depending on the context in the natural language instruction. VisProg also estimates the maximum size of the category list retrieved. For instance, “Tag the logos of the top 5 german car companies” generates a list of 5 categories, while “Tag the logos of german car companies” produces a list of arbitrary length determined by GPT-3 with a cut-off at 20. This allows users to easily control the noise in the classification process by tweaking their instructions.
做知识标注时，VisProg 把 GPT-3 当作可用自然语言提示查询的隐式知识库，例如：「列出《生活大爆炸》主要角色，用逗号分隔。」生成的类别列表再交给 CLIP 图像分类模块，对由定位与人脸检测模块得到的图像区域分类。程序生成器会根据自然语言指令的语境自动决定用人脸检测器还是开放词汇定位器，并估计检索类别列表的最大长度。例如「标注前 5 家德国车企的 logo」生成 5 类，而「标注德国车企的 logo」则生成由 GPT-3 决定、上限为 20 的任意长度列表。用户可通过微调指令方便地控制分类过程中的噪声。
Evaluation. To evaluate VisProg on this task, we annotate 100 tagging instructions across 46 images that require external knowledge to tag 253 object instances including personalities across pop culture, politics, sports, and art, as well as a varieties of objects (e.g. cars, flags, fruits, appliances, furniture etc.). For each instruction, we measure both localization and tagging performance via precision (fraction of predicted boxes that are correct) and recall (fraction of ground truth objects that are correctly predicted). Tagging metrics require both the predicted bounding box and the associated tag or class label to be correct, while localization ignores the tag. To determine localization correctness, we use an IoU threshold of 0.5. We summarize localization and tagging performance by F1 scores (harmonic mean of the average precision and recall across instructions).
评估。 我们在 46 张图上标注 100 条标注指令，需外部知识才能标注 253 个物体实例，覆盖流行文化、政治、体育与艺术中的人物，以及汽车、旗帜、水果、家电、家具等各类物体。对每条指令，我们用精确率（预测框中正确的比例）与召回（真值物体被正确预测的比例）衡量定位与标注表现。标注指标要求预测框与关联标签/类别都正确；定位则忽略标签。定位正确性用 IoU 阈值 0.5。我们用各指令平均精确率与召回的调和平均（F1）汇总定位与标注表现。
Prompts. We create 14 in-context examples for this task. Note that the instructions for these examples were hallucinated i.e. no images were associated with these examples.
提示。 我们为该任务创建 14 条上下文示例。注意这些示例的指令是虚构的，即没有关联图像。
4.4 Image Editing with Natural Language
4.4 自然语言图像编辑
Text to image generation has made impressive strides over the last few years with models like DALL-E [26], Parti [35], and Stable Diffusion [28]. However, it is still beyond the capability of these models to handle prompts like ”Hide the face of Daniel Craig with :p” (de-identification or privacy preservation), or ”Create a color pop of Daniel Craig and blur the background” (object highlighting) even though these are relatively simple to achieve programmatically using a combination of face detection, segmentation and image processing modules. Achieving a sophisticated edit such as ”Replace Barack Obama with Barack Obama wearing sunglasses” (object replacement), first requires identifying the object of interest, generating a mask of the object to be replaced and then invoking an image inpainting model (we use Stable Diffusion) with the original image, mask specifying the pixels to replace, and a description of the new pixels to generate at that location. VisProg , when equipped with the necessary modules and example programs, can handle very complex instructions with ease.
文生图近年来进步显著，如 DALL-E [26]、Parti [35] 与 Stable Diffusion [28]。但像「用 :p 挡住 Daniel Craig 的脸」（去标识或隐私保护），或「对 Daniel Craig 做 color pop 并模糊背景」（物体高亮）这类提示，这些模型仍做不到，尽管用脸检测、分割与图像处理模块组合起来相对容易。更复杂的编辑如「把 Barack Obama 换成戴墨镜的 Barack Obama」（物体替换），需先识别目标、生成待替换掩码，再把原图、指定替换像素的掩码与新像素描述交给图像修复模型（我们用 Stable Diffusion）。VisProg 在配备必要模块与示例程序后，能轻松处理非常复杂的指令。
Evaluation. To test VisProg on the image editing instructions for de-identification, object highlighting, and object replacement, we collect 107 instructions across 65 images. We manually score the predictions for correctness and report accuracy. Note that we do not penalize visual artifacts for the object replacement sub-task which uses Stable Diffusion as long as the generated image is semantically correct.
评估。 为测试去标识、物体高亮与物体替换三类图像编辑指令，我们在 65 张图上收集 107 条指令。人工对预测是否语义正确打分并报告准确率。物体替换子任务使用 Stable Diffusion：只要生成图语义正确，就不因视觉伪影扣分。
Prompts. Similar to knowledge tagging, we create 10 in-context examples for this task with no associated images.
提示。 与知识标注类似，我们为该任务创建 10 条无关联图像的上下文示例。
5 Experiments and Analysis
5 实验与分析
Our experiments evaluate the effect of number of prompts on GQA and NLVR performance (Sec. 5.1), generalization of VisProg on the four tasks comparing various prompting strategies (Sec. 5.2), analyze the sources of error for each task (Fig. 8), and study the utility of visual rationales for diagnosing errors and improving VisProg ’s performance through instruction tuning (Sec. 5.3).
实验评估提示数量对 GQA 与 NLVR 表现的影响（第 5.1 节）、VisProg 在四项任务上的泛化及多种提示策略的比较（第 5.2 节）、各任务的误差来源（图 8），以及视觉理据对诊断错误、并通过指令调优提升表现的作用（第 5.3 节）。
5.1 Effect of prompt size
5.1 提示规模的影响
Figure 7: Performance improves with number of in-context examples on GQA and NLVRv2 validation sets. The error bars represent confidence interval across runs. Predictions from the same runs are used for majority voting. (Sec. 5.1)
图 7：GQA 与 NLVRv2 验证集上，表现随上下文示例数量增加而提升。 误差条为 次运行的 置信区间。多数投票使用同批运行的预测。（第 5.1 节）
Fig. 7 shows that validation performance increases progressively with the number of in-context examples used in the prompts for both GQA and NLVR. Each run randomly selects a subset of the annotated in-context examples based on a random seed. We also find that majority voting across the random seeds leads to consistently better performance than the average performance across runs. This is consistent with findings in Chain-of-Thought [32] reasoning literature for math reasoning problems [31]. On NLVR, the performance of VisProg saturates with fewer prompts than GQA . We believe this is because NLVRv2 programs require fewer modules and hence fewer demonstrations for using those modules than GQA .
图 7 显示，GQA 与 NLVR 上验证表现都随提示中上下文示例数逐步上升。每次运行按随机种子从已标注上下文示例中抽子集。我们还发现，跨随机种子做多数投票，稳定地优于各次运行的平均表现，这与数学推理上思维链 [32] 文献 [31] 的发现一致。NLVR 上 VisProg 用比 GQA 更少的提示就趋于饱和；我们认为是因为 NLVRv2 程序所需模块更少，因而演示这些模块用法所需的示例也更少。
5.2 Generalization
5.2 泛化
| Method | Prompting strategy | Runs | Context examples per run | Accuracy |
|---|---|---|---|---|
| ViLT-vqa | - | 1 | - | 47.8 |
| VisProg | curated | 1 | 20 | 50.0 |
| VisProg | random | 1 | 24 | 48.2 |
| VisProg | voting | 5 | 24 | 50.5 |
Table 1: GQAtestdev results. We report performance on a subset of the original GQA testdev set as described in Sec. 4.1.
| 方法 | 提示策略 | 运行次数 | 每次上下文示例数 | 准确率 |
|---|---|---|---|---|
| ViLT-vqa | - | 1 | - | 47.8 |
| VisProg | curated | 1 | 20 | 50.0 |
| VisProg | random | 1 | 24 | 48.2 |
| VisProg | voting | 5 | 24 | 50.5 |
表 1：GQA testdev 结果。 我们按第 4.1 节所述，在原始 GQA testdev 的一个子集上报告表现。
| Method | Prompting strategy | Finetuned | Runs | Context examples per run | Accuracy |
|---|---|---|---|---|---|
| ViLT-nlvr | - | ✓ | 1 | - | 76.3 |
| VisProg | curated | ✗ | 1 | 12 | 61.8 |
| VisProg | random | ✗ | 1 | 16 | 61.3 |
| VisProg | voting | ✗ | 5 | 16 | 62.4 |
Table 2: NLVRv2test results. VisProg performs NLVR zero-shot i.e. without training any module on image pairs. ViLT-nlvr , a ViLT model finetuned on NLVRv2 , serves as an upper bound.
| 方法 | 提示策略 | 是否微调 | 运行次数 | 每次上下文示例数 | 准确率 |
|---|---|---|---|---|---|
| ViLT-nlvr | - | ✓ | 1 | - | 76.3 |
| VisProg | curated | ✗ | 1 | 12 | 61.8 |
| VisProg | random | ✗ | 1 | 16 | 61.3 |
| VisProg | voting | ✗ | 5 | 16 | 62.4 |
表 2：NLVRv2 test 结果。 VisProg 以零样本方式做 NLVR，即未在任何图像对上训练模块。在 NLVRv2 上微调过的 ViLT-nlvr 作为上界。
| Instructions | Tagging | Localization | |||||
|---|---|---|---|---|---|---|---|
| precision | recall | F1 | precision | recall | F1 | ||
| Original | 69.0 | 59.1 | 63.7 | 87.2 | 74.9 | 80.6 | |
| Modified | 77.6 | 73.9 | 75.7 | 87.4 | 82.5 | 84.9 |
Table 3: Knowledge tagging results. The table shows performance on original instructions as well as modified instructions created after inspecting visual rationales to understand instance-specific sources of errors.
| 指令 | 标注 | 定位 | |||||
|---|---|---|---|---|---|---|---|
| 精确率 | 召回 | F1 | 精确率 | 召回 | F1 | ||
| 原始 | 69.0 | 59.1 | 63.7 | 87.2 | 74.9 | 80.6 | |
| 修改后 | 77.6 | 73.9 | 75.7 | 87.4 | 82.5 | 84.9 |
表 3：知识标注结果。 表中给出原始指令以及在检查视觉理据、理解实例级误差来源后修改的指令上的表现。
| Original | Modified | |
|---|---|---|
| Accuracy | 59.8 | 66.4 |
Table 4: Image editing results. We manually evaluate each prediction for semantic correctness.
| 原始 | 修改后 | |
|---|---|---|
| 准确率 | 59.8 | 66.4 |
表 4：图像编辑结果。 我们人工评估每条预测的语义正确性。
GQA. In Tab. 1 we evaluate different prompting strategies on the GQA testdev set. For the largest prompt size evaluated on the val set ( in-context examples), we compare the random strategy consisting of the VisProg ’s best prompt chosen amongst 5 runs on the validation set (each run randomly samples in-context examples from 31 annotated examples) and the majority voting strategy which takes maximum consensus predictions for each question across 5 runs. While “random” prompts only slightly outperform ViLT-vqa , voting leads to a significant gain of points. This is because voting across multiple runs, each with a different set of in-context examples, effectively increases the total number of in-context examples seen for each prediction. We also evaluate a manually curated prompt consisting of examples - from the annotated examples, and additional hallucinated examples meant to provide a better coverage for failure cases observed in the validation set. The curated prompt performs just as well as the voting strategy while using less compute, highlighting the promise of prompt engineering.
GQA。 表 1 在 GQA testdev 集上评估不同提示策略。对验证集上评估过的最大提示规模（ 条上下文示例），我们比较：从验证集 5 次运行（每次从 31 条已标注示例中随机抽样）中选出的最佳 “random” 提示，以及跨 5 次运行取最大共识预测的多数投票策略。“random” 提示仅略优于 ViLT-vqa，而投票带来 个点的显著提升——因为跨多组上下文示例投票，等效增大了每次预测所见的上下文总量。我们还评估了手工精选的 条提示：其中 条来自 条已标注示例，另加 条虚构示例以更好覆盖验证集上观察到的失败情形。精选提示表现与投票相当，而计算量约为其 ，凸显了提示工程的潜力。
NLVR. Tab. 2 shows performance of VisProg on the NLVRv2 test set and compares random, voting, and curated prompting strategies as done with GQA. While VisProg performs the NLVR task zero-shot without ever training on image pairs, we report ViLT-nlvr , a ViLT model finetuned on NLVRv2 as an upper bound on performance. While several points behind the upper bound, VisProg shows strong zero-shot performance using only a single-image VQA model for image understanding, and an LLM for reasoning. Note that, VisProg uses ViLT-vqa for its VQA module which is trained on VQAv2 a single image question answer task, but not NLVRv2 .
NLVR。 表 2 给出 VisProg 在 NLVRv2 test 上的表现，并像 GQA 一样比较 random、voting 与 curated 策略。VisProg 以零样本方式完成 NLVR，从未在图像对上训练；我们报告在 NLVRv2 上微调过的 ViLT-nlvr 作为表现上界。虽落后上界若干点，VisProg 仅用单图 VQA 模型做图像理解、用 LLM 做推理，仍展现出较强的零样本能力。注意其 VQA 模块使用的是在 VQAv2（单图问答）上训练、而非在 NLVRv2 上训练的 ViLT-vqa。
Knowledge Tagging. Tab. 3 shows localization and tagging performance for the Knowledge Tagging task. All instructions for this task not only require open vocabulary localization but also querying a knowledge base to fetch the categories to tag localized objects with. This makes it an impossible task for object detectors alone. With the original instructions, VisProg achieves an impressive F1 score for tagging, which involves both correctly localizing and naming the objects, and F1 score for localization alone. Visual rationales in VisProg allow further performance gains by modifying the instructions. See Fig. 6 for qualitative examples and Sec. 5.3 for more details on instruction tuning.
知识标注。 表 3 给出该任务的定位与标注表现。所有指令既需要开放词汇定位，也需要查询知识库以获取给定位物体打标签的类别，因此单靠目标检测器无法完成。在原始指令下，VisProg 的标注 F1 达到可观的 （需同时正确定位并命名），单独定位 F1 为 。视觉理据还允许通过修改指令进一步提升表现。定性例子见图 6；关于指令调优详见第 5.3 节。
Image Editing. Tab. 4 shows the performance on the language-guided image editing task. Fig. 6 shows the wide range of manipulations possible with the current set of modules in VisProg including face manipulations, highlighting one or more objects in the image via stylistic effects like color popping and background blur, and changing scene context by replacing key elements in the scene (e.g. desert).
图像编辑。 表 4 给出语言引导图像编辑任务的表现。图 6 展示了 VisProg 当前模块集所能完成的广泛操控：人脸处理、用 color pop 与背景模糊等风格效果高亮一个或多个物体，以及通过替换场景关键要素（如沙漠）改变场景语境。
Figure 8: Sources of error in VisProg .
图 8：VisProg 的误差来源。
Figure 9: Instruction tuning using visual rationales. By revealing the reason for failure, VisProg allows a user to modify the original instruction to improve performance.
图 9：利用视觉理据做指令调优。 通过揭示失败原因，VisProg 允许用户修改原始指令以提升表现。
5.3 Utility of Visual Rationales
5.3 视觉理据的用处
Error Analysis. Rationales generated by VisProg allow a thorough analysis of failure modes as shown in Fig. 8. For each task, we manually inspect rationales for samples to break down the sources of errors. Such analysis provides a clear path towards improving performance of VisProg on various tasks. For instance, since incorrect programs are the leading source of errors on GQA affecting of samples, performance on GQA may be improved by providing more in-context examples similar to the instructions that VisProg currently fails on. Performance may also be improved by upgrading models used to implement the high-error modules to more performant ones. For example, replacing the ViLT-vqa model with a better VQA model for NLVR could improve performance by up to . Similarly, improving models used to implement “List” and “Select” modules, the major sources of error for knowledge tagging and image editing tasks, could significantly reduce errors.
误差分析。 VisProg 生成的理据便于彻底分析失败模式，如图 8。对每项任务，我们人工检查约 个样本的理据以拆解误差来源。这类分析为提升各任务表现指明路径。例如，GQA 上错误程序是主因，影响 样本，可通过提供与当前失败指令相似的更多上下文示例来改进；也可把高误差模块背后的模型换成更强的模型——例如用更好的 VQA 模型替换 ViLT-vqa，NLVR 表现或可最多提升 。类似地，改进实现 “List” 与 “Select” 模块的模型——知识标注与图像编辑的主要误差源——可显著减少错误。
Instruction tuning. To be useful, a visual rationale must ultimately allow users to improve the performance of the system on their task. For knowledge tagging and image editing tasks, we study if visual rationales can help a user modify or tune the instructions to achieve better performance. Fig. 9 shows that modified instructions: (i) result in a better query for the localization module (e.g. “kitchen appliance” instead of “item”); (ii) provide a more informative query for knowledge retrieval (e.g. “most recent CEO of IBM” instead of “CEO of IBM”); (iii) provide a category name (e.g. “table-merged”) for the Select module to restrict search to segmented regions belonging to the specified category; or (iv) control the number of classification categories for knowledge tagging through the max argument in the List module. Tables 3 and 4 show that instruction tuning results in significant gains for knowledge tagging and image editing tasks.
指令调优。 要真正有用，视觉理据最终须帮助用户提升系统在其任务上的表现。对知识标注与图像编辑，我们研究视觉理据能否帮助用户修改或调优指令以取得更好结果。图 9 显示，修改后的指令可以：(i) 给定位模块更好的查询（如用 “kitchen appliance” 代替 “item”）；(ii) 给知识检索更信息量的查询（如 “most recent CEO of IBM” 而非 “CEO of IBM”）；(iii) 为 Select 模块提供类别名（如 “table-merged”），把搜索限制在属于该类别的分割区域；(iv) 通过 List 模块的 max 参数控制知识标注的分类类别数。表 3 与表 4 表明，指令调优为知识标注与图像编辑带来显著增益。
6 Conclusion
6 结论
VisProg proposes visual programming as a simple and effective way of bringing the reasoning capabilities of LLMs to bear on complex visual tasks. VisProg demonstrates strong performance while generating highly interpretable visual rationales. Investigating better prompting strategies and exploring new ways of incorporating user feedback to improve the performance of neuro-symbolic systems such as VisProg is an exciting direction for building the next generation of general-purpose vision systems.
VisProg 提出视觉编程：一种把 LLM 的推理能力带到复杂视觉任务上的简单而有效的方式。它在取得较强表现的同时生成高度可解释的视觉理据。研究更好的提示策略，并探索纳入用户反馈以提升 VisProg 这类神经符号系统表现，是构建下一代通用视觉系统的激动方向。
7 Acknowledgement
7 致谢
We thank Kanchan Aggarwal for helping with the annotation process for the image editing and knowledge tagging tasks. We are also grateful to the amazing Hugging Face ecosystem for simplifying the use of state-of-the-art neural models for implementing VisProg modules.
感谢 Kanchan Aggarwal 协助图像编辑与知识标注任务的标注。也感谢出色的 Hugging Face 生态，它极大简化了用先进神经模型实现 VisProg 模块的过程。
References
参考文献
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv , abs/2204.14198, 2022.
[2] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 39–48, 2016.
[3] Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. mslam: Massively multilingual joint pre-training for speech and text. ArXiv , abs/2202.01374, 2022.
[4] Gary Bradski. The opencv library. Dr. Dobb’s Journal: Software Tools for the Professional Programmer , 25(11):120–123, 2000.
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv , abs/2005.14165, 2020.
[6] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS , 2021.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv , abs/1810.04805, 2019.
[8] Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems. ArXiv , abs/2104.00743, 2021.
[9] Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In ECCV , 2018.
[10] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. 2017 IEEE International Conference on Computer Vision (ICCV) , pages 804–813, 2017.
[11] Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6693–6702, 2019.
[12] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. Inferring and executing programs for visual reasoning. 2017 IEEE International Conference on Computer Vision (ICCV) , pages 3008–3017, 2017.
[13] Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. Webly supervised concept expansion for general purpose vision models. In ECCV , 2022.
[14] Tushar Khot, Kyle Richardson, Daniel Khashabi, and Ashish Sabharwal. Learning to solve complex tasks by talking to agents. ArXiv , abs/2110.08542, 2021.
[15] Tushar Khot, H. Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. ArXiv , abs/2210.02406, 2022.
[16] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In ICML , 2021.
[17] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXiv , abs/2205.11916, 2022.
[18] Jian Li, Yabiao Wang, Changan Wang, Ying Tai, Jianjun Qian, Jian Yang, Chengjie Wang, Jilin Li, and Feiyue Huang. Dsfd: Dual shot face detector. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5055–5064, 2019.
[19] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. ArXiv , abs/2206.08916, 2022.
[20] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3190–3199, 2019.
[21] Matthias Minderer, Alexey A. Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection with vision transformers. ArXiv , abs/2205.06230, 2022.
[22] Zoe Papakipos and Joanna Bitton. Augly: Data augmentations for robustness. ArXiv , abs/2201.06494, 2022.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748–8763. PMLR, 2021.
[24] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[25] Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv , abs/1910.10683, 2020.
[26] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ArXiv , abs/2102.12092, 2021.
[27] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. ArXiv , abs/2205.06175, 2022.
[28] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10674–10685, 2022.
[29] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. ArXiv , abs/2209.11302, 2022.
[30] Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. ArXiv , abs/1811.00491, 2019.
[31] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ArXiv , abs/2203.11171, 2022.
[32] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv , abs/2201.11903, 2022.
[33] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229–256, 1992.
[34] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In AAAI , 2022.
[35] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Benton C. Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. ArXiv , abs/2206.10789, 2022.
[36] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv , 2022.
Appendix A Appendix
附录 A
This appendix includes
本附录包括
- • Task prompts for VisProg (Sec. A.1)
- • Module implementation details (Sec. A.2)
- • Many more qualitative results with visual rationales for both successful and failure cases can be found at https://prior.allenai.org/projects/visprog.
- • VisProg 的任务提示（第 A.1 节）
- • 模块实现细节（第 A.2 节）
- • 更多成功与失败案例的定性结果与视觉理据见 https://prior.allenai.org/projects/visprog。
A.1 Task Prompts
A.1 任务提示
We show the prompt structures for GQA (Figure 10), NLVR (Figure 11), knowledge tagging (Figure 13), and language-guided image editing (Figure 12) tasks with 3 in-context examples each.
我们分别用 3 条上下文示例展示 GQA（图 10）、NLVR（图 11）、知识标注（图 13）与语言引导图像编辑（图 12）任务的提示结构。
Figure 10: GQA prompt
图 10：GQA 提示
Figure 11: NLVR prompt
图 11：NLVR 提示
Figure 12: Image editing prompt. Note that the prompt includes a mapping of emojis to their names in the AugLy [22] library that is used to implement Emoji module. The third example shows how to provide the category value for the Select module.
图 12：图像编辑提示。 注意提示包含 emoji 到 AugLy [22] 库中名称的映射，该库用于实现 Emoji 模块。第三个示例展示如何为 Select 模块提供 category 取值。
Figure 13: Knowledge tagging prompt. Note that the prompt has an additional placeholder to configure the default max value for List module. While the first example infers max from a natural instruction, the third example demonstrates how a user might minimally augment a natural instruction to provide argument values.
图 13：知识标注提示。 注意提示另有占位符，用于配置 List 模块的默认 max。第一个示例从自然语言指令推断 max，第三个示例展示用户可如何对自然语言指令做最小增补以提供参数值。
A.2 Module Details
A.2 模块细节
To help understand the generated programs better, we now provide a few implementation details about some of the modules.
为更好理解生成的程序，下面补充若干模块的实现细节。
Select. The module takes a query and a category argument. When the category is provided, the selection is only performed over the regions that have been identified as belonging to that category by a previous module in the program (typically the Seg module). If category is None, the selection is performed over all regions. The query is the text to be used for region-text scoring to perform the selection. We use CLIP-ViT [23] to select the region with the maximum score for the query. When the query contains multiple phrases separated by commas, the highest-scoring region is selected for each phrase.
Select。 该模块接受 query 与 category 参数。若提供了 category，则只在程序前序模块（通常是 Seg）已识别为该类别的区域上做选择；若 category 为 None，则在所有区域上选择。query 是用于区域–文本打分以完成选择的文本。我们用 CLIP-ViT [23] 选取对该 query 得分最高的区域。若 query 含多个由逗号分隔的短语，则为每个短语各选得分最高的区域。
Classify. The Classify module takes lists of object regions and categories and tries to assign one of the categories to each region. For simplicity, we assume the images in the tagging task has at most 1 instance of each category. The Classify module operates differently based on whether the category list has 1 or more elements. If the category list has only 1 element, the category is assigned to the region with the highest CLIP score, similar to the Select module. When more than one category is provided, first, each region is assigned the category with the best score. Due to classification errors, this can lead to multiple regions being assigned the same category. Therefore, for each of the assigned categories (excluding the ones that were not assigned to any region), we perform a de-duplication step that retains only the maximum scoring region for each category.
Classify。 Classify 模块接受物体区域列表与类别列表，试图为每个区域分配一个类别。为简单起见，我们假设标注任务图像中每个类别最多 1 个实例。类别列表长度为 1 或更多时，行为不同：若只有 1 个类别，则类似 Select，把该类别赋给 CLIP 得分最高的区域；若有多个类别，则先为每个区域赋得分最高的类别。因分类错误，可能多个区域落到同一类别，因此对每个已分配类别（排除未被任何区域选中的类别）做去重，每个类别只保留得分最高的区域。
List. The List module uses GPT3 to create a flexible and powerful knowledge retriever. Fig. 14 shows the prompt provided to GPT3 to retrieve factual knowledge.
List。 List 模块用 GPT-3 构成灵活而强大的知识检索器。图 14 给出向 GPT-3 检索事实知识时的提示。
Figure 14: Prompt for the List module. list_max denotes the default maximum list length and new_query is the placeholder for the new retrieval query
图 14：List 模块的提示。 list_max 表示默认最大列表长度，new_query 是新检索查询的占位符。