CLIPort: What and Where Pathways for Robotic Manipulation

CLIPort：机器人操作中的 What 与 Where 通路

Mohit Shridhar † † thanks: Work done partly while the author was a part-time intern at NVIDIA. Affiliation:  University of Washington Email:  mshr@cs.washington.edu    Lucas Manuelli Affiliation:  NVIDIA Email:  lmanuelli@nvidia.com    Dieter Fox Affiliation:  University of Washington Affiliation:  NVIDIA Email:  fox@cs.washington.educliport.github.io

Abstract

摘要

How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.

如何让机器人既能精确操作物体，又能用抽象概念去理解它们？近期操作领域的工作表明，端到端网络可以学会需要精细空间推理的灵巧技能，但这些方法往往难以泛化到新目标，也难以及时在任务间迁移可复用概念。与此同时，借助大规模互联网数据训练，视觉与语言的可泛化语义表征进展显著，但这些表征缺少精细操作所需的空间理解。为此，我们提出一种融合两者优势的框架：面向视觉操作的语义与空间双通路架构。具体地，我们提出 CLIPort——一种语言条件的模仿学习智能体，把 CLIP [1] 的宽泛语义理解（what）与 Transporter [2] 的空间精度（where）结合起来。该端到端框架能解决从装箱未见物体到折布等各类语言指定的桌面任务，全程不依赖物体位姿、实例分割、记忆、符号状态或句法结构的显式表示。仿真与真实实验表明，方法在少样本设定下数据效率高，并能有效泛化到已见与未见语义概念。我们甚至为 10 个仿真任务与 9 个真实任务学到一个多任务策略，表现优于或可比于单任务策略。

1 Introduction

1 引言

Ask a person to “get a scoop of coffee beans” or “fold the cloth in half” and they can naturally take concepts like scoop or fold and ground them in concrete physical actions within an accuracy of a few centimeters. We humans do this intuitively, without explicit geometric or kinematic models of coffee beans or cloths. Moreover, we can generalize to a broad range of tasks and concepts from a minimal set of examples on what needs to be achieved. How can we imbue robots with this ability to efficiently ground abstract semantic concepts in precise spatial reasoning?

让人「舀一勺咖啡豆」或「把布对折」，他们自然会把 舀、折 这类概念落到厘米级精度的具体动作上。我们凭直觉做到这一点，并不需要咖啡豆或布料的显式几何与运动学模型。而且，只需很少示例说明要达成什么，就能泛化到广泛的任务与概念。怎样才能让机器人同样高效地把抽象语义概念锚定到精确的空间推理上？

Recently, a number of end-to-end frameworks have been proposed for vision-based manipulation  [2, 3, 4, 5]. While these methods do not use any explicit representations of object poses, instance segmentations, or symbolic states, they can only replicate demonstrations with a narrow range of variability and have no notion of the semantics underlying the tasks. Switching from packing red pens to blue pens involves collecting a new training set [2], or if using goal-conditioned policies, involves the user providing a goal-image from the scene [5, 6]. In realistic human-robot interaction settings, collecting additional demonstrations or providing goal-images is often infeasible and unscalable. A natural solution to both these problems is to condition policies with natural language. Language provides an intuitive interface for specifying goals and also for implicitly transferring concepts across tasks. While language-grounding for manipulation has been explored in the past [7, 8, 9, 10], these pipelines are limited by object-centric representations that cannot handle granular or deformable objects and often do not reason about perception and action in an integrated manner. In parallel, there has been great progress in learning models for visual representations [11, 12] and aligning representations of vision and language  [13, 14, 15] by training on large-scale internet data. However, these models lack a fine-grained understanding on how to manipulate objects, i.e. physical affordances.

近来，不少面向视觉操作的端到端框架相继提出 [2, 3, 4, 5]。它们虽不使用物体位姿、实例分割或符号状态的显式表示，却只能复现变体范围很窄的演示，也缺乏对任务语义的理解。从装红笔换成装蓝笔，往往意味着重新采集训练集 [2]；若用目标条件策略，用户还需从场景中提供目标图像 [5, 6]。在真实人机交互中，再采演示或再给目标图往往既不可行也难扩展。一个自然的解法是用自然语言条件化策略：语言既便于指定目标，也便于在任务间隐式迁移概念。面向操作的语言锚定虽有探索 [7, 8, 9, 10]，但这些管线受限于以物体为中心的表示，难以处理细粒度或可变形物体，也常无法把感知与动作一体推理。并行地，视觉表征学习 [11, 12] 以及视觉—语言对齐 [13, 14, 15] 借助大规模互联网数据进展很大，但这些模型缺少对「如何操作物体」——即物理可供性——的细粒度理解。

Figure 1: Language-Conditioned Manipulation Tasks: CLIPort is a broad framework applicable to a wide range of language-conditioned manipulation tasks in tabletop settings. We conduct large-scale experiments in Ravens [2] on 10 simulated tasks (a-j) with 1000s of unique instances per task. See Appendix A for challenges pertaining to each task. CLIPort can even learn one multi-task model for all 10 tasks that achieves better or comparable performance to single-task models. Similarly, we demonstrate our approach on a Franka Panda manipulator with one multi-task model for 9 real-world tasks (k-o; only 5 shown) trained with just 179 image-action pairs.

图 1：语言条件操作任务： CLIPort 是一个适用范围广的框架，可用于多种桌面设定下的语言条件操作任务。我们在 Ravens [2] 上对 10 个仿真任务（a–j）做大规模实验，每任务有上千个独特实例。各任务难点见附录 A。CLIPort 甚至能为全部 10 个任务学到一个多任务模型，表现优于或可比于单任务模型。类似地，我们在 Franka Panda 上用仅 179 对图像—动作样本训练的一个多任务模型，演示了 9 个真实任务（k–o；图中仅示 5 个）。

To this end, we propose the first framework that combines the best of both worlds: end-to-end learning for fine-grained manipulation with the multi-goal and multi-task generalization capabilities of vision-language grounding systems. We introduce a two-stream architecture for manipulation with semantic and spatial pathways broadly inspired by (or vaguely analogous to) the two-stream hypothesis in cognitive psychology [16, 17, 18]. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that integrates the semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Transporter has been applied to a wide range of rearragement tasks from industrial packing [2] to manipulating deformable objects [6]. The key insight of the approach is formulating tabletop manipulation as a series of pick-and-place affordance predictions, where the objective is to detect actions rather than detect objects and then learn a policy. This action-centric approach to perception [19] is data efficient and effective at circumventing the need for explicit “objectness” in learnt representations. However, Transporter is a tabula rasa system that learns all visual representations from scratch and so every new goal or task requires collecting a new set of demonstrations. To address this problem, we bake in a strong semantic prior while learning policies. We condition our semantic stream with visual and language-goal features from a pre-trained CLIP model [1]. Since CLIP is pre-trained to align image and language features from millions of image-caption pairs from the internet, it provides a powerful prior for grounding semantic concepts that are common across tasks like categories, parts, shapes, colors, texts, and other visual attributes, all without a top-down pipeline that requires bounding boxes or instance segmentations [13, 14, 15, 20]. This allows us to formulate tabletop rearrangement as a series of language-conditioned affordance predictions, a predominantly vision-based inference problem, and thus benefit from the strengths of data-driven paradigms like scale and generalization.

为此，我们提出首个把两者优势结合起来的框架：面向精细操作的端到端学习，加上视觉—语言锚定系统的多目标、多任务泛化能力。我们引入带 语义 与 空间 通路的操作双流架构，灵感（或宽泛类比）来自认知心理学中的双流通假说 [16, 17, 18]。具体地，我们提出 CLIPort——一种语言条件的模仿学习智能体，把 CLIP [1] 的语义理解（what）与 Transporter [2] 的空间精度（where）整合起来。Transporter 已用于从工业装箱 [2] 到可变形物体操作 [6] 等多种重排任务。其关键洞察是把桌面操作表述为一系列抓取—放置可供性预测，目标是 检测动作 而非 检测物体 再学策略。这种以动作为中心的感知 [19] 数据效率高，也能绕开学习表征里对显式「物体性」的依赖。但 Transporter 是白板系统，视觉表征从零学起，每个新目标或新任务都要重新采演示。为解决这一问题，我们在学策略时注入强语义先验：用预训练 CLIP 模型 [1] 的视觉与语言目标特征条件化 语义 流。CLIP 在数百万图像—标题对上对齐图像与语言特征，为类别、部件、形状、颜色、文字及其他视觉属性等跨任务共通的语义概念提供强先验，且无需依赖边界框或实例分割的自上而下管线 [13, 14, 15, 20]。于是桌面重排可写成一系列语言条件的可供性预测——本质是视觉推断问题——从而受益于数据驱动范式的规模与泛化优势。

To study these benefits, we conduct large-scale experiments in the Ravens [2] framework with a simulated suction-gripper robot. We propose 10 language-conditioned tasks with 1000s of unique instances per task that require both semantic and spatial reasoning (see Figure 1 a-j). CLIPort  is not only effective at solving these tasks, but surprisingly, it can even learn a multi-task model for all 10 tasks that achieves better or comparable performance to single-task models. Further, our evaluations indicate that our multi-task model can effectively transfer attributes like “pink block” across tasks, having never seen pink blocks or the word ‘pink’ in the context of the evaluation task. We also demonstrate our approach on a Franka Panda manipulator with one multi-task model for 9 real-world tasks trained with just 179 image-action pairs (see Figure 1 k-o).

为检验这些收益，我们在 Ravens [2] 框架中用仿真吸盘机器人做大规模实验，提出 10 个同时需要语义与空间推理的语言条件任务，每任务上千独特实例（见图 1 a–j）。CLIPort 不仅能有效完成这些任务，更意外的是，还能为全部 10 个任务学到一个多任务模型，表现优于或可比于单任务模型。进一步评估表明，多任务模型能把诸如「粉色方块」这类属性跨任务迁移：即使在评测任务语境中从未见过粉色方块或「pink」一词。我们还在 Franka Panda 上用仅 179 对图像—动作样本训练的一个多任务模型，演示了 9 个真实任务（见图 1 k–o）。

In summary, our contributions are as follows:

综上，本文贡献如下：

• • An extended benchmark of language-grounding tasks for manipulation in Ravens [2].
• • Two-stream architecture for using internet pre-trained vision-language models for conditioning precise manipulation policies with language goals.
• • Empirical results on a broad range of manipulation tasks, including multi-task models, validated with real-robot experiments.

• • 在 Ravens [2] 上扩展的操作 语言锚定任务基准。
• • 用互联网预训练视觉—语言模型，以语言目标条件化精确操作策略的 双流架构。
• • 覆盖多类操作任务（含多任务模型）的 实证结果，并以真实机器人实验验证。

The benchmark, code, and pre-trained models are available at: cliport.github.io.

基准、代码与预训练模型见：cliport.github.io。

2 Related Work

2 相关工作

Vision-based Manipulation. Traditionally, perception for manipulation has centered around object detectors, segmentors, and pose estimators [21, 22, 23, 24, 25, 26]. These methods cannot handle deformable objects, granular media, or generalize to unseen objects without object-specific training data. Alternatively, dense descriptors [27, 28, 29] and keypoint representations [30, 31, 32] forgo segmentation and pose representations, but do not reason about sequential actions and struggle to represent scenes with variable numbers of objects. On the other hand, end-to-end perception-to-action models can learn precise sequential policies [2, 4, 6, 33, 34, 35], but these methods have limited understanding of semantic concepts and rely on goal-images to condition policies. In contrast, Yen-Chen et. al [36] showed that pre-training on semantic tasks like classification and segmentation helps in improving efficiency and generalization of grasping predictions.

基于视觉的操作。 传统上，操作感知围绕物体检测器、分割器与位姿估计器 [21, 22, 23, 24, 25, 26]。这些方法难以处理可变形物体、颗粒介质，也难在缺少物体专用训练数据时泛化到未见物体。稠密描述子 [27, 28, 29] 与关键点表示 [30, 31, 32] 放弃了分割与位姿，但不推理序列动作，也难表示物体数量可变的场景。另一方面，端到端感知到动作模型能学到精确的序列策略 [2, 4, 6, 33, 34, 35]，但对语义概念理解有限，并依赖目标图像来条件化策略。相较之下，Yen-Chen 等 [36] 表明，在分类与分割等语义任务上预训练有助于提升抓取预测的效率与泛化。

Semantic Models. With the advent of large-scale models [37, 38, 39], a number of methods for learning joint vision and language representations have been proposed  [13, 14, 15, 20, 40]. However, these methods are restricted to bounding boxes or instance segmentations, which make them inapplicable for detecting things like piles of coffee beans or squares on a chessboard. Alternatively, works in contrastive learning forgo top-down object-detection and learn continuous representations by pre-training on unlabeled data [11, 12]. Recently, CLIP [1] applied a similar approach to align vision and language representations by training on millions of image-caption pairs from the internet.

语义模型。 随着大规模模型兴起 [37, 38, 39]，涌现出一批学习视觉—语言联合表征的方法 [13, 14, 15, 20, 40]。但它们受限于边界框或实例分割，难以检测一堆咖啡豆或棋盘上的方格这类目标。对比学习工作则放弃自上而下的物体检测，通过对未标注数据预训练学习连续表征 [11, 12]。近来，CLIP [1] 用类似思路，在数百万互联网图像—标题对上对齐视觉与语言表征。

Language Grounding for Robotics. Several works have proposed systems for instructing robots with natural language [7, 8, 9, 10, 41, 42, 43, 44, 45, 46, 47]. However, these methods use disentangled pipelines for perception and action with the language primarily being used to guide the perception. As such, these pipelines lack the spatial precision necessary for tasks like folding cloths. Recently, Lynch et. al [48] proposed an end-to-end system for grounding language in continuous control, but it requires several hours of human teleoperation data for a single simulated desk setting.

机器人语言锚定。 已有多项工作用自然语言指挥机器人 [7, 8, 9, 10, 41, 42, 43, 44, 45, 46, 47]。但这些方法把感知与动作拆成解耦管线，语言主要用来引导感知，因而缺少折布等任务所需的空间精度。近来 Lynch 等 [48] 提出把语言锚定到连续控制的端到端系统，但单个仿真桌面设定就需要数小时人工遥操作数据。

Two-Stream Architectures are prevalent in action-recognition networks [49, 50, 51] and audio-recognition systems [52, 53]. In robotics, Zeng et. al [54] and Jang et. al [55] have proposed two-stream pipelines for affordance predictions of novel objects. The former requires goal-images and the latter is restricted to one-step grasps with single-category goals. In contrast, our framework provides a rich and intuitive interface with composable language commands for sequential tasks.

双流架构 在动作识别网络 [49, 50, 51] 与音频识别系统 [52, 53] 中很常见。在机器人领域，Zeng 等 [54] 与 Jang 等 [55] 提出了面向新物体可供性预测的双流管线。前者需要目标图像，后者限于单步抓取与单类别目标。相较之下，我们的框架为序列任务提供了可组合语言指令这一丰富而直观的接口。

3 CLIPort

CLIPort is an imitation-learning agent based on four key principles: (1) Manipulation through a two-step primitive where each action involves a start and final end-effector pose. (2) Visual representations of actions that are equivariant to translations and rotations [56, 57]. (3) Two separate pathways for semantic and spatial information. (4) Language-conditioned policies for specifying goals and also transferring concepts across tasks. Combining (1) and (2) from Transporter with (3) and (4) allows us to achieve generalizable policies that go beyond just imitating demonstrations.

CLIPort 是基于四条原则的模仿学习智能体：（1）用两步原语操作，每步动作包含末端执行器的起始与终止位姿；（2）对平移与旋转等变的动作视觉表征 [56, 57]；（3）语义与空间信息分走两条通路；（4）用语言条件策略指定目标并在任务间迁移概念。把 Transporter 的（1）（2）与（3）（4）结合，可得到超越单纯模仿演示的可泛化策略。

Section 3.1 describes the problem formulation, gives an overview of Transporter [2], and presents our language-conditioned model. Section 3.2 provides details on the training approach.

第 3.1 节给出问题形式化、概述 Transporter [2]，并提出我们的语言条件模型。第 3.2 节给出训练细节。

Figure 2: CLIPort Two-Stream Architecture. An overview of the semantic and spatial streams. The semantic stream uses a frozen CLIP ResNet50 [1] to encode RGB input, and its decoder layers are conditioned with tiled language features from the CLIP sentence encoder. The spatial stream encodes RGB-D input, and its decoder layers are laterally fused with the semantic stream. The final output is a map of dense pixelwise features that is used for pick or place affordance predictions. This same two-stream architecture is used in all 3 Fully-Convolutional-Networks $f_{\text{pick}},\Phi_{\text{query}}$, and $\Phi_{\text{key}}$ with $f_{\text{pick}}$ is used to predict pick actions, and $\Phi_{\text{query}}$ and $\Phi_{\text{key}}$ are used to predict place actions. See Appendix C for the exact architecture.

图 2：CLIPort 双流架构。 语义 流与 空间 流概览。语义 流用冻结的 CLIP ResNet50 [1] 编码 RGB 输入，其解码层用 CLIP 句子编码器得到的平铺语言特征做条件化。空间 流编码 RGB-D 输入，其解码层与 语义 流做侧向融合。最终输出稠密逐像素特征图，用于抓取或放置可供性预测。同一双流架构用于全部 3 个全卷积网络 $f_{\text{pick}}$、$\Phi_{\text{query}}$ 与 $\Phi_{\text{key}}$：$f_{\text{pick}}$ 预测抓取动作，$\Phi_{\text{query}}$ 与 $\Phi_{\text{key}}$ 预测放置动作。精确结构见附录 C。

3.1 Language-Conditioned Manipulation

3.1 语言条件操作

We consider the problem of learning a goal-conditioned policy $\pi$ that outputs actions $\mathbf{a}_{t}$ given input $\gamma_{t}=(\mathbf{o}_{t},\mathbf{l}_{t})$ consisting of a visual observation $\mathbf{o}_{t}$ and an English language instruction $\mathbf{l}_{t}$:

我们考虑学习目标条件策略 $\pi$：给定由视觉观测 $\mathbf{o}_{t}$ 与英语语言指令 $\mathbf{l}_{t}$ 组成的输入$\gamma_{t}=(\mathbf{o}_{t},\mathbf{l}_{t})$，输出动作 $\mathbf{a}_{t}$：

$$\pi(\bm{\gamma}_{t})=\pi(\mathbf{o}_{t},\mathbf{l}_{t})\to\mathbf{a}_{t}=(\mathcal{T}_{\text{pick}},\mathcal{T}_{\text{place}})\in\mathcal{A} \tag{1}$$

The actions $\mathbf{a}=(\mathcal{T}_{\text{pick}},\mathcal{T}_{\text{place}})$ specify the end-effector pose for picking and placing, respectively. We consider tabletop tasks where $\mathcal{T}_{\text{pick}},\mathcal{T}_{\text{place}}\in\mathbf{SE}(2)$. The visual observation $\mathbf{o}_{t}$ is a top-down orthographic RGB-D reconstruction of the scene where each pixel corresponds to a point in 3D space. The language instruction $\mathbf{l}_{t}$ either specifies step-by-step instructions e.g. “pack the scissors” $\to$ “pack the purple tape” $\to$ etc., or a single goal description for the whole task e.g “pack all the blue and yellow boxes in the brown box”. See Figure 4 for specific examples.

动作 $\mathbf{a}=(\mathcal{T}_{\text{pick}},\mathcal{T}_{\text{place}})$ 分别指定抓取与放置的末端执行器位姿。我们考虑桌面任务，其中 $\mathcal{T}_{\text{pick}},\mathcal{T}_{\text{place}}\in\mathbf{SE}(2)$。视觉观测 $\mathbf{o}_{t}$ 是场景的俯视正交 RGB-D 重建，每个像素对应三维空间中一点。语言指令 $\mathbf{l}_{t}$ 可以逐步给出，例如 「把剪刀装进去」 $\to$ 「把紫色胶带装进去」 $\to$ 等；也可以用一句目标描述覆盖整项任务，例如 「把所有蓝色和黄色盒子装进棕色箱子」。具体例子见图 4。

We assume access to a dataset $\mathcal{D}=\{\zeta_{1},\zeta_{2},\ldots,\zeta_{n}\}$ of $n$ expert demonstrations with associated discrete-time input-action pairs $\zeta_{i}=\{(\mathbf{o}_{1},\mathbf{l}_{1},\mathbf{a}_{1}),(\mathbf{o}_{2},\mathbf{l}_{2},\mathbf{a}_{2}),\ldots\}$ where $\mathbf{a}_{t}=(\mathcal{T}_{\text{pick}},\mathcal{T}_{\text{place}})$ corresponds to expert pick-and-place coordinates at timestep $t$. These expert demonstrations are used to supervise the policy $\pi$.

我们假设可访问专家演示数据集 $\mathcal{D}=\{\zeta_{1},\zeta_{2},\ldots,\zeta_{n}\}$，共 $n$ 条，每条含离散时间的输入—动作对 $\zeta_{i}=\{(\mathbf{o}_{1},\mathbf{l}_{1},\mathbf{a}_{1}),(\mathbf{o}_{2},\mathbf{l}_{2},\mathbf{a}_{2}),\ldots\}$，其中 $\mathbf{a}_{t}=(\mathcal{T}_{\text{pick}},\mathcal{T}_{\text{place}})$ 为时刻 $t$ 的专家抓取—放置坐标。这些专家演示用于监督策略 $\pi$。

Transporter for Pick-and-Place. The policy $\pi$ is trained with Transporter [2] to perform spatial manipulation. The model first (i) attends to a local region to decide where to pick, then (ii) computes a placement location by finding the best match through cross-correlation of deep visual features.

Transporter 用于抓取—放置。 策略 $\pi$ 用 Transporter [2] 训练以完成空间操作。模型先（i）关注局部区域决定抓哪里，再（ii）通过对深度视觉特征做互相关寻找最佳匹配，从而计算放置位置。

Following Transporter [2, 6], the policy $\pi$ is composed of two action-value modules (Q-functions): The pick module $\mathcal{Q}_{\text{pick}}$ decides where to pick, and conditioned on this pick action the place module $\mathcal{Q}_{\text{place}}$ decides where to place. These modules are implemented as Fully-Convolutional-Networks (FCNs) that are translationally equivariant by design. As we will describe in more detail below, we extend these networks to two-stream architectures that can handle language input. The pick FCN $f_{\text{pick}}$ takes input $\bm{\gamma}_{t}=(\mathbf{o}_{t},\mathbf{l}_{t})$ and outputs a dense pixelwise prediction $\mathcal{Q}_{\text{pick}}\in\mathbb{R}^{H\times W}$ of action-values, where are used to predict the pick action $\mathcal{T}_{\text{pick}}$:

沿用 Transporter [2, 6]，策略 $\pi$ 由两个动作价值模块（Q 函数）组成：抓取模块 $\mathcal{Q}_{\text{pick}}$ 决定抓哪里；条件于该抓取动作，放置模块 $\mathcal{Q}_{\text{place}}$ 决定放到哪里。这些模块实现为设计上对平移等变的全卷积网络（FCN）。下文将详述如何把它们扩展为可处理语言输入的双流架构。抓取 FCN $f_{\text{pick}}$ 以 $\bm{\gamma}_{t}=(\mathbf{o}_{t},\mathbf{l}_{t})$ 为输入，输出稠密逐像素动作价值预测 $\mathcal{Q}_{\text{pick}}\in\mathbb{R}^{H\times W}$，据此预测抓取动作 $\mathcal{T}_{\text{pick}}$：

$$\mathcal{T}_{\text{pick}}=\argmax_{(u,v)}\mathcal{Q}_{\text{pick}}((u,v)|\bm{\gamma}_{t}) \tag{2}$$

Since $\mathbf{o}_{t}$ is an orthographic heightmap, each pixel location $(u,v)$ can be mapped to a 3D picking location using the known camera calibration. $f_{\text{pick}}$ is trained in a supervised manner to predict the pick action $\mathcal{T}_{\text{pick}}$ that imitates the expert demonstration with the specified language instruction at timestep $t$.

由于 $\mathbf{o}_{t}$ 是正交高度图，每个像素位置 $(u,v)$ 可用已知相机标定映射到三维抓取位置。$f_{\text{pick}}$ 以监督方式训练，预测模仿专家演示、并符合时刻 $t$ 指定语言指令的抓取动作 $\mathcal{T}_{\text{pick}}$。

The second FCN $\Phi_{\text{query}}$ takes in $\bm{\gamma}_{t}[\mathcal{T}_{\text{pick}}]$, which is a $c\times c$ crop of $\mathbf{o}_{t}$ centered at $\mathcal{T}_{\text{pick}}$ along with the language instruction $\mathbf{l}_{t}$, and outputs a query feature embedding of shape $\mathbb{R}^{c\times c\times d}$. The third FCN $\Phi_{\text{key}}$ consumes the full input $\bm{\gamma}_{t}$ and outputs a key feature embedding of shape $\mathbb{R}^{H\times W\times d}$. The place action-values $\mathcal{Q}_{\text{place}}$ are then computed by cross-correlating the query and key features:

第二个 FCN $\Phi_{\text{query}}$ 接收 $\bm{\gamma}_{t}[\mathcal{T}_{\text{pick}}]$——即以 $\mathcal{T}_{\text{pick}}$ 为中心、对 $\mathbf{o}_{t}$ 裁出的 $c\times c$ 块，连同语言指令 $\mathbf{l}_{t}$——输出形状为 $\mathbb{R}^{c\times c\times d}$ 的查询特征嵌入。第三个 FCN $\Phi_{\text{key}}$ 消费完整输入 $\bm{\gamma}_{t}$，输出形状为 $\mathbb{R}^{H\times W\times d}$ 的键特征嵌入。放置动作价值 $\mathcal{Q}_{\text{place}}$ 再通过对查询与键特征做互相关得到：

$$\mathcal{Q}_{\text{place}}(\Delta\tau|\gamma_{t},\mathcal{T}_{\text{pick}})=\big(\Phi_{\text{query}}(\bm{\gamma}_{t}[\mathcal{T}_{\text{pick}}])\ast\Phi_{\text{key}}(\bm{\gamma}_{t})\big)[\Delta\tau] \tag{3}$$

where $\Delta\tau\in\bm{SE}(2)$ represents a potential placement pose. Since $\mathbf{o}_{t}$ is an orthographic heightmap, rotations in the placement pose $\Delta\tau$ can be captured by stacking $k$ discrete angle rotations of the crop before passing it through the query network $\Phi_{\text{query}}$. Then $\mathcal{T}_{\text{place}}=\argmax_{\Delta\tau}\mathcal{Q}_{\text{place}}(\Delta\tau|\gamma_{t},\mathcal{T}_{\text{pick}})$, where the place module is trained to imitate the placements in the expert demonstrations. For all models, we use $c=64$, $k=36$ and $d=3$. As in Transporter [2, 6], our framework can be extended to handle any motion primitive like pushing, sliding, etc. that can be parameterized by two end-effector poses at each timestep. For more details, we refer the reader to the original paper [2].

其中 $\Delta\tau\in\bm{SE}(2)$ 表示候选放置位姿。由于 $\mathbf{o}_{t}$ 是正交高度图，放置位姿 $\Delta\tau$ 中的旋转可通过在送入查询网络 $\Phi_{\text{query}}$ 前，把裁块堆叠为 $k$ 个离散角度旋转来刻画。随后 $\mathcal{T}_{\text{place}}=\argmax_{\Delta\tau}\mathcal{Q}_{\text{place}}(\Delta\tau|\gamma_{t},\mathcal{T}_{\text{pick}})$，放置模块训练为模仿专家演示中的放置。所有模型取 $c=64$、$k=36$、$d=3$。与 Transporter [2, 6] 一样，本框架可扩展到推、滑等任意可用两个末端执行器位姿在每时刻参数化的运动原语。更多细节见原文 [2]。

Two-Stream Architecture. In CLIPort, we extend the network architecture of all three FCNs $f_{\text{pick}},\Phi_{\text{query}}$ and $\Phi_{\text{key}}$ from Transporter [2] to allow for language input and reasoning about high-level semantic concepts. We extend the FCNs to two-pathways: semantic (ventral) and spatial (dorsal). The semantic stream is conditioned with language features at the bottleneck and fused with intermediate features from the spatial stream. See Figure 2 for an overview of the architecture.

双流架构。 在 CLIPort 中，我们把 Transporter [2] 中三个 FCN $f_{\text{pick}}$、$\Phi_{\text{query}}$ 与 $\Phi_{\text{key}}$ 的网络结构加以扩展，以接受语言输入并推理高层语义概念。FCN 扩展为双通路：语义（腹侧）与 空间（背侧）。语义 流在瓶颈处用语言特征条件化，并与 空间 流的中间特征融合。架构概览见图 2。

The spatial stream is identical to the ResNet architecture in Transporter – a tabula rasa network that takes in RGB-D input $\mathbf{o}_{t}$ and outputs dense features through an hourglass encoder-decoder model. The semantic stream uses a frozen pre-trained CLIP ResNet50 [1] to encode the RGB input[^1] $\tilde{\mathbf{o}}_{t}$ up until the penultimate layer $\tilde{\mathbf{o}}_{t}\to\mathbf{v}^{(0)}_{t}:\mathbb{R}^{7\times 7\times 2048}$, and then introduces decoding layers that upsample the feature tensors to mimic the spatial stream $\mathbf{v}^{(l-1)}_{t}\to\mathbf{v}^{(l)}_{t}:\mathbb{R}^{h\times w\times C}$ at each layer $l$.

空间 流与 Transporter 中的 ResNet 架构相同——白板网络，以 RGB-D 输入 $\mathbf{o}_{t}$，经沙漏式编码器—解码器输出稠密特征。语义 流用冻结的预训练 CLIP ResNet50 [1] 编码 RGB 输入[^1] $\tilde{\mathbf{o}}_{t}$，直至倒数第二层 $\tilde{\mathbf{o}}_{t}\to\mathbf{v}^{(0)}_{t}:\mathbb{R}^{7\times 7\times 2048}$，再引入解码层对特征张量上采样，使每层 $l$ 上 $\mathbf{v}^{(l-1)}_{t}\to\mathbf{v}^{(l)}_{t}:\mathbb{R}^{h\times w\times C}$ 与 空间 流对齐。

The language instruction $\mathbf{l}_{t}$ is encoded with CLIP’s Transformer-based sentence encoder to produce a goal encoding $\mathbf{l}_{t}\to\mathbf{g}_{t}:\mathbb{R}^{1024}$. This goal encoding $\mathbf{g}_{t}$ is downsampled with fully-connected layers to match the channel dimension $C$ and tiled to match the spatial dimensions of the decoder features such that $\mathbf{g}_{t}\to\mathbf{g}^{(l)}_{t}:\mathbb{R}^{h\times w\times C}$. The decoder features are then conditioned with the tiled goal features through an element-wise product $\mathbf{v}^{(l)}_{t}\odot\mathbf{g}^{(l)}_{t}$ (Hadamard product). Since CLIP was trained with contrastive loss on the dot-product alignment between pooled image features and language encodings, the element-wise product allows us to use this alignment while the tiling preserves the spatial dimensions of the visual features. This language conditioning is repeated for three subsequent layers after the bottleneck inspired by LingUNet [58]. We also add skip connections to these layers from the CLIP ResNet50 encoder to utilize different levels of semantic information from shapes to parts to object-level concepts [59]. Finally, following existing two-stream architectures in video-action recognition [51], we add lateral connections from the spatial stream to the semantic stream. These connections involve concatenating two feature tensors and applying $1\times 1\ \texttt{conv}$ to reduce the channel dimension $[\mathbf{v}^{(l)}_{t}\odot~\mathbf{g}^{(l)}_{t};\mathbf{d}^{(l)}_{t}]:\mathbb{R}^{h\times w\times C_{\mathbf{v}}+C_{\mathbf{d}}}\to\mathbb{R}^{h\times w\times C_{\mathbf{v}}}$, where $\mathbf{v}^{(l)}_{t}$ and $\mathbf{d}^{(l)}_{t}$ are the semantic and spatial tensors at layer $l$, respectively. For the final fusion of dense features, addition for $f_{\text{pick}}$ and $1\times 1\ \texttt{conv}$ fusion for $\Phi_{\text{query}}$ and $\Phi_{\text{key}}$ worked the best empirically. See Appendix C for details on the exact architecture.

语言指令 $\mathbf{l}_{t}$ 经 CLIP 基于 Transformer 的句子编码器得到目标编码 $\mathbf{l}_{t}\to\mathbf{g}_{t}:\mathbb{R}^{1024}$。该目标编码 $\mathbf{g}_{t}$ 经全连接层下采样以匹配通道维 $C$，再平铺以匹配解码特征的空间维，即 $\mathbf{g}_{t}\to\mathbf{g}^{(l)}_{t}:\mathbb{R}^{h\times w\times C}$。解码特征再通过逐元素乘积 $\mathbf{v}^{(l)}_{t}\odot\mathbf{g}^{(l)}_{t}$（Hadamard 积）与平铺目标特征条件化。由于 CLIP 用对比损失在池化图像特征与语言编码的点积对齐上训练，逐元素乘积可利用这一对齐，而平铺则保留视觉特征的空间维。受 LingUNet [58] 启发，这一语言条件化在瓶颈之后的连续三层重复进行。我们还从 CLIP ResNet50 编码器向这些层加入跳跃连接，以利用从形状到部件再到物体级概念的不同层次语义信息 [59]。最后，沿用视频动作识别中的双流做法 [51]，从 空间 流向 语义 流加入侧向连接：拼接两个特征张量并用 $1\times 1\ \texttt{conv}$ 降通道维 $[\mathbf{v}^{(l)}_{t}\odot~\mathbf{g}^{(l)}_{t};\mathbf{d}^{(l)}_{t}]:\mathbb{R}^{h\times w\times C_{\mathbf{v}}+C_{\mathbf{d}}}\to\mathbb{R}^{h\times w\times C_{\mathbf{v}}}$，其中 $\mathbf{v}^{(l)}_{t}$ 与 $\mathbf{d}^{(l)}_{t}$ 分别为第 $l$ 层的 语义 与 空间 张量。稠密特征的最终融合上，$f_{\text{pick}}$ 用加法、$\Phi_{\text{query}}$ 与 $\Phi_{\text{key}}$ 用 $1\times 1\ \texttt{conv}$ 融合在实验中效果最好。精确架构见附录 C。

3.2 Implementation Details

3.2 实现细节

Training from demonstrations. Similar to Transporter [2] we train CLIPort through imitation learning from a set of expert demonstrations $\mathcal{D}=\{\zeta_{1},\zeta_{2},\ldots,\zeta_{n}\}$ consisting of discrete-time input-action pairs $\zeta_{i}=\{(\mathbf{o}_{1},\mathbf{l}_{1},\mathbf{a}_{1}),(\mathbf{o}_{2},\mathbf{l}_{2},\mathbf{a}_{2}),\ldots\}$. During training, we randomly sample an input-action pair from the dataset and supervise the model end-to-end with one-hot pixel encodings of demonstration actions $Y_{\textrm{pick}}:\mathbb{R}^{H\times W\times k}$ and $Y_{\textrm{place}}:\mathbb{R}^{H\times W\times k}$ with $k$ discrete rotations. In simulated experiments with the suction-gripper, we use $k=1$ for pick actions and $k=36$ for place actions. The model is trained with cross-entropy loss: $\mathcal{L}=-\mathbb{E}_{Y_{\textrm{pick}}}[\textrm{log}\,\mathcal{V}_{\textrm{pick}}]-\mathbb{E}_{Y_{\textrm{place}}}[\textrm{log}\mathcal{V}_{\textrm{place}}]$ where $\mathcal{V}_{\textrm{pick}}=\text{softmax}(\mathcal{Q}_{\text{pick}}((u,v)|\bm{\gamma}_{t}))$ and $\mathcal{V}_{\textrm{place}}=\text{softmax}(\mathcal{Q}_{\text{place}}((u^{\prime},v^{\prime},\omega^{\prime})|\bm{\gamma}_{t},\mathcal{T}_{\text{pick}}))$. Compared to the original Transporter models that were trained for 40K iterations, we train our models for 200K iterations (with data augmentation; see Appendix E) to account for additional semantic variation in tasks – randomized colors, shapes, objects. All models are trained on a single commodity GPU for 2 days with a batch size of 1.

从演示训练。 与 Transporter [2] 类似，我们在专家演示集 $\mathcal{D}=\{\zeta_{1},\zeta_{2},\ldots,\zeta_{n}\}$ 上以模仿学习训练 CLIPort，每条含离散时间输入—动作对 $\zeta_{i}=\{(\mathbf{o}_{1},\mathbf{l}_{1},\mathbf{a}_{1}),(\mathbf{o}_{2},\mathbf{l}_{2},\mathbf{a}_{2}),\ldots\}$。训练时从数据集随机采样一对输入—动作，用演示动作的 one-hot 像素编码 $Y_{\textrm{pick}}:\mathbb{R}^{H\times W\times k}$ 与 $Y_{\textrm{place}}:\mathbb{R}^{H\times W\times k}$（$k$ 个离散旋转）端到端监督模型。吸盘仿真实验中，抓取动作取 $k=1$，放置动作取 $k=36$。模型用交叉熵损失训练：$\mathcal{L}=-\mathbb{E}_{Y_{\textrm{pick}}}[\textrm{log}\,\mathcal{V}_{\textrm{pick}}]-\mathbb{E}_{Y_{\textrm{place}}}[\textrm{log}\mathcal{V}_{\textrm{place}}]$，其中 $\mathcal{V}_{\textrm{pick}}=\text{softmax}(\mathcal{Q}_{\text{pick}}((u,v)|\bm{\gamma}_{t}))$，$\mathcal{V}_{\textrm{place}}=\text{softmax}(\mathcal{Q}_{\text{place}}((u^{\prime},v^{\prime},\omega^{\prime})|\bm{\gamma}_{t},\mathcal{T}_{\text{pick}}))$。相对原版 Transporter 训练 40K 迭代，我们训练 200K 迭代（含数据增强；见附录 E），以覆盖任务中额外的语义变化——随机颜色、形状、物体。所有模型在单块商用 GPU 上训练 2 天，批大小为 1。

Training multi-task models. Multi-task training is nearly identical to single-task training except for the sampling of training data. First, we randomly sample a task, and then select a random input-action pair from that task in the dataset. Using this strategy, all tasks are equally likely to be sampled but longer horizon tasks are less likely to reach full coverage of input-action pairs available in the dataset. To compensate for this, we train all multi-task models $3\times$ longer for 600K iterations or 6 GPU days.

训练多任务模型。 多任务训练与单任务几乎相同，差别在于训练数据的采样：先随机选一个任务，再从该任务数据集中选一对随机输入—动作。按此策略，各任务被采样的概率相等，但更长视界任务更难覆盖数据集中全部输入—动作对。为补偿这一点，所有多任务模型训练时长 $\times 3$，即 600K 迭代或 6 GPU 天。

4 Results

4 结果

We perform experiments both in simulation and hardware aimed at answering the following questions: 1) How effective is the language-conditioned two-stream architecture for fine-grained manipulation compared to one-stream alternatives and other simpler baselines? 2) Is it possible to train a multi-task model for all tasks, and how well does it perform and generalize? 3) How well do these models generalize to seen and unseen semantic attributes like colors, shapes, and object categories?

我们在仿真与真机上开展实验，旨在回答：1）相对单流替代方案与其他更简单基线，语言条件双流架构对精细操作有多有效？2）能否为全部任务训练一个多任务模型，其表现与泛化如何？3）这些模型对已见与未见的颜色、形状、物体类别等语义属性泛化得怎样？

4.1 Simulation Setup

4.1 仿真设定

Environment. All simulated experiments are based on a Universal Robot UR5e with a suction gripper. The setup provides a systematic and reproducible environment for evaluation, especially for benchmarking the ability to ground semantic concepts like colors and object categories. The input observation is a top-down RGB-D reconstruction from 3 cameras positioned around a rectangular table: one in the front, one on the left shoulder, and one on the right shoulder, all pointing towards the center. Each camera has a resolution of $640\times 480$ and is noiseless.

环境。 全部仿真实验基于带吸盘的 Universal Robot UR5e。该设定提供系统、可复现的评测环境，尤其便于基准测试颜色与物体类别等语义概念的锚定能力。输入观测是由环绕矩形桌面的 3 台相机重建的俯视 RGB-D：一台在前方，一台在左肩，一台在右肩，均指向中心。每台相机分辨率 $640\times 480$，且无噪声。

Language-Conditioned Manipulation Tasks. We extend the Ravens benchmark [2] set in PyBullet [60] with 10 language-conditioned manipulation tasks. See Figure 1 for examples and Table 3 for challenges associated with each task. Each task instance is constructed by sampling a set of objects and attributes: poses, colors, sizes, and object categories. 8 of the 10 tasks have two variants, denoted by seen and unseen, depending on whether the task has unseen attributes (e.g. color) at test time. For colors: $\mathbb{T_{\text{seen colors}}}=\{\texttt{yellow, brown, gray, cyan}\}$ and $\mathbb{T_{\text{unseen colors}}}=\{\texttt{orange, purple, pink, white}\}$ with 3 overlapping colors $\mathbb{T_{\text{all}}}=\{\texttt{red, green, blue}\}$ used in both the seen and unseen spilts. For packing objects, we use 56 tabletop objects from the Google Scanned Objects dataset [61] and split them into 37 seen and 19 unseen objects. The language instructions are constructed from templates for simulated experiments, and human-annotated for real-world experiments. For more details about individual tasks, see Appendix A.

语言条件操作任务。 我们在 PyBullet [60] 中的 Ravens 基准 [2] 上扩展出 10 个语言条件操作任务。示例见图 1，各任务挑战见表 3。每个任务实例通过采样一组物体与属性构造：位姿、颜色、尺寸与物体类别。10 个任务中有 8 个具有两个变体，按测试时是否含未见属性（如颜色）记为 seen 与 unseen。颜色方面：$\mathbb{T_{\text{seen colors}}}=\{\texttt{yellow, brown, gray, cyan}\}$，$\mathbb{T_{\text{unseen colors}}}=\{\texttt{orange, purple, pink, white}\}$，另有 3 种重叠颜色 $\mathbb{T_{\text{all}}}=\{\texttt{red, green, blue}\}$ 同时用于 seen 与 unseen 划分。装箱物体方面，我们使用 Google Scanned Objects 数据集 [61] 中的 56 个桌面物体，划分为 37 个 seen 与 19 个 unseen。仿真实验的语言指令由模板构造，真实实验则由人工标注。各任务更多细节见附录 A。

Evaluation Metric. We adopt the 0 (fail) to 100 (success) scores proposed in the Ravens benchmark [2]. The score assigns partial credit based on the task, e.g. $3/5\Rightarrow 60.0$ for packing 3 out of 5 objects specified in the instructions, or $30/56\Rightarrow 53.6$ for pushing 30 out of 56 particles into the correct zone. See Appendix A for the specific evaluation metric used in each task. During an evaluation episode, an agent keeps interacting with the scene until an oracle indicates task-completion. We report scores on 100 evaluation runs for agents trained with $n=1,10,100,1000$ demonstrations.

评测指标。 我们采用 Ravens 基准 [2] 提出的 0（失败）到 100（成功）分数。分数按任务给予部分分，例如按指令装对 3/5 个物体记为 $3/5\Rightarrow 60.0$，或把 30/56 个粒子推进正确区域记为 $30/56\Rightarrow 53.6$。各任务具体评测见附录 A。在评测回合中，智能体持续与场景交互，直至预言机指示任务完成。我们报告在 $n=1,10,100,1000$ 条演示上训练的智能体在 100 次评测上的分数。

4.2 Simulation Results

4.2 仿真结果

Figure 3: Average scores across seen and unseen splits for all tasks in Table 1.

图 3：表 1 全部任务在 seen 与 unseen 划分上的平均分数。

Table 1 presents results from our large-scale experiments in Ravens [2] and Figure 3 summarizes these results with average scores across seen and unseen splits.

表 1 给出我们在 Ravens [2] 上大规模实验的结果，图 3 用 seen 与 unseen 划分上的平均分加以概括。

Baseline Methods. To study the effectiveness of our two-stream architecture, we broadly compare against two baselines: Transporter-only and CLIP-only. Transporter-only is the original Transporter [2], or equivalently, the spatial stream of CLIPort with RGB-D input. Although Transporter-only does not receive any language goals, it shows what can be achieved through chance by exploiting the most likely actions seen during training. On the other hand, CLIP-only is just the semantic stream of CLIPort with RGB and language input. CLIP-only shows what can be achieved by fine-tuning a pre-trained semantic model for manipulation without spatial information, particularly depth.

基线方法。 为研究双流架构的有效性，我们主要对比两类基线：Transporter-only 与 CLIP-only。Transporter-only 即原版 Transporter [2]，或等价地，CLIPort 的 空间 流加 RGB-D 输入。尽管 Transporter-only 不接收语言目标，它仍展示了仅靠利用训练中最常见动作所能达到的「偶然」水平。另一方面，CLIP-only 只是 CLIPort 的 语义 流，输入为 RGB 与语言。CLIP-only 展示了在缺少空间信息（尤其是深度）时，微调预训练语义模型做操作能达到什么程度。

Two-Stream Performance. Figure 3 (seen) captures the essence of our main claims. The performance of Transporter-only saturates at $50\%$ since it doesn’t use the language instruction to ground the desired goal. CLIP-only does have a goal, but lacks the spatial precision to go the last mile and thus saturates at $76\%$. Only CLIPort (single) achieves more than $90\%$, which indicates that both the semantic and spatial streams are crucial for fine-grained manipulation. Further, CLIPort (single) achieves $86\%$ on most tasks with just 100 demonstrations, showcasing its efficiency.

双流表现。 图 3（seen）抓住了我们主要主张的要点。Transporter-only 的表现饱和在约 $50\%$，因为它不用语言指令锚定期望目标。CLIP-only 有目标，但缺少走完「最后一公里」的空间精度，因而饱和在约 $76\%$。只有 CLIPort (single) 超过 $90\%$，说明精细操作同时需要语义流与空间流。进一步地，CLIPort (single) 在多数任务上仅用 100 条演示即可达到约 $86\%$，显示其效率。

In addition to these baselines, we present various ablations and alternative one-stream and two-stream models in Appendix F. To briefly summarize these results, CLIP is essential for few-shot learning (i.e. $n\geq 10$) in lieu of semantic stream alternatives like ImageNet-trained ResNet50 [62] with BERT [38]. Image-goal models outperform CLIPort (single) in packing Google objects, but this is only because they do not have to solve the language-grounding problem.

除这些基线外，附录 F 还给出多种消融以及替代的单流与双流模型。简要概括：相对 ImageNet 训练的 ResNet50 [62] 配 BERT [38] 等 语义 流替代，CLIP 对少样本学习（即 $n\geq 10$）至关重要。图像目标模型在装箱 Google 物体上优于 CLIPort (single)，但这只是因为它们不必解决语言锚定问题。

Multi-Task Performance. In realistic scenarios, we want the robot to be capable of any task, not just one task. We investigate this through CLIPort (multi) in Table 1 with one multi-task model trained on all 10 tasks. CLIPort (multi) models are trained only on seen-splits of tasks, so an unseen attribute like ‘pink’ is consistent throughout single and multi-task settings. Surprisingly, CLIPort (multi) outperforms single-task CLIPort (single) models in $41/72=57\%$ of the evaluations in Table 1. This trend is also evident in Figure 3 (seen), especially in instances with 100 demonstrations or less. Although CLIPort (multi) is trained on more diverse data from other tasks, both CLIPort (multi) and CLIPort (single) have access to the same amount of data per task. This supports our premise that language is a strong conditioning mechanism for reusing concepts from other tasks without learning them from scratch. It also validates a trait of data-driven approaches where training on lots of diverse data leads to more robust and generalizable representations [1, 63]. However, CLIPort (multi) performs worse on longer-horizon tasks like align-rope. We hypothesize that this is because longer-horizon tasks get less coverage of input-action pairs in the dataset. Future works could use better sampling methods that balance tasks according to their average time horizon.

多任务表现。 在真实场景中，我们希望机器人能做任意任务，而非单一任务。我们通过表 1 中的 CLIPort (multi) 加以考察：一个在全部 10 个任务上训练的多任务模型。CLIPort (multi) 仅在各任务的 seen 划分上训练，因此像「pink」这样的未见属性在单任务与多任务设定中保持一致。出人意料的是，在表 1 的 $41/72=57\%$ 评测中，CLIPort (multi) 优于单任务 CLIPort (single)。这一趋势在图 3（seen）中同样明显，尤其在 100 条及以下演示时。尽管 CLIPort (multi) 训练数据来自其他任务更丰富，但每个任务上 CLIPort (multi) 与 CLIPort (single) 可用的数据量相同。这支持我们的前提：语言是强条件化机制，可复用其他任务中的概念而无需从零学习。它也验证了数据驱动方法的一个特点——在大量多样数据上训练会得到更稳健、更可泛化的表征 [1, 63]。不过，CLIPort (multi) 在 align-rope 等长视界任务上更差。我们猜想这是因为长视界任务在数据集中对输入—动作对的覆盖更少。未来工作可采用更好的采样方法，按平均时间视界平衡各任务。

	packing-box-pairs seen-colors	packing-box-pairs	seen-colors	packing-box-pairs unseen-colors	packing-box-pairs	unseen-colors	packing-seen-google objects-seq	packing-seen-google	objects-seq	packing-unseen-google objects-seq	packing-unseen-google	objects-seq	packing-seen-google objects-group	packing-seen-google	objects-group	packing-unseen-google objects-group	packing-unseen-google	objects-group
packing-box-pairs
seen-colors
packing-box-pairs
unseen-colors
packing-seen-google
objects-seq
packing-unseen-google
objects-seq
packing-seen-google
objects-group
packing-unseen-google
objects-group
Method	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000
Transporter-only [2]	44.2	55.2	54.2	52.4	34.6	48.7	47.2	54.1	26.2	39.7	45.4	46.3	19.9	29.8	28.7	37.3	60.0	54.3	61.5	59.9	46.2	54.7	49.8	52.0
CLIP-only	38.6	69.7	88.5	87.1	33.0	65.5	68.8	61.2	29.1	67.9	89.3	95.8	37.1	49.4	60.4	57.8	52.5	62.0	89.6	92.7	43.4	65.9	73.1	70.0
RN50-BERT	36.2	64.0	94.7	90.3	31.4	52.7	65.6	72.1	32.9	48.4	87.9	94.0	29.3	48.5	48.3	56.1	46.4	52.9	76.5	86.4	43.2	52.0	66.3	73.7
CLIPort (single)	51.6	82.9	92.7	98.2	45.6	65.3	68.6	71.5	14.8	59.5	86.8	96.2	27.2	50.0	65.5	71.9	52.7	67.0	84.1	94.0	61.5	66.2	78.4	81.5
CLIPort (multi)	66.8	88.6	94.1	96.6	59.0	69.7	76.2	71.4	41.6	78.4	85.0	84.4	40.7	51.1	65.8	70.3	71.3	84.6	89.6	88.3	68.4	69.6	78.4	80.3
CLIPort (multi-attr)	–	–	–	–	46.2	72.0	86.2	80.3	–	–	–	–	35.4	45.1	78.9	87.4	–	–	–	–	48.6	69.3	84.8	89.1
	stack-block-pyramid seq-seen-colors	stack-block-pyramid	seq-seen-colors	stack-block-pyramid seq-unseen-colors	stack-block-pyramid	seq-unseen-colors	separating-piles seen-colors	separating-piles	seen-colors	separating-piles unseen-colors	separating-piles	unseen-colors	towers-of-hanoi seq-seen-colors	towers-of-hanoi	seq-seen-colors	towers-of-hanoi seq-unseen-colors	towers-of-hanoi	seq-unseen-colors
stack-block-pyramid
seq-seen-colors
stack-block-pyramid
seq-unseen-colors
separating-piles
seen-colors
separating-piles
unseen-colors
towers-of-hanoi
seq-seen-colors
towers-of-hanoi
seq-unseen-colors
	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000
Transporter-only [2]	4.5	2.3	5.2	4.5	3.0	4.0	2.3	5.8	42.7	52.3	42.0	48.4	41.2	49.2	44.7	52.3	25.4	67.9	98.0	99.9	24.3	44.6	71.7	80.7
CLIP-only	6.3	28.7	55.7	54.8	2.0	12.2	18.3	19.5	43.5	55.0	84.9	90.2	59.9	49.6	73.0	71.0	9.4	52.6	88.6	45.3	24.7	47.0	67.0	58.0
RN50-BERT	5.3	35.0	89.0	97.5	6.2	12.2	21.5	30.7	31.8	47.8	46.5	46.5	33.4	44.4	41.3	44.9	28.0	66.1	91.3	92.1	17.4	75.1	85.3	89.3
CLIPort (single)	28.3	64.7	93.3	98.8	13.7	24.3	31.2	41.3	54.5	59.5	93.1	98.0	47.2	51.0	76.6	75.2	59.4	92.9	97.4	100	56.1	89.7	95.9	99.4
CLIPort (multi)	33.5	75.3	96.8	96.5	23.3	26.8	31.7	22.2	48.9	72.4	90.3	89.0	56.6	62.6	64.9	62.8	61.6	96.3	98.7	98.1	60.1	65.6	76.7	68.7
CLIPort (multi-attr)	–	–	–	–	15.5	51.5	59.3	79.8	–	–	–	–	49.9	51.8	48.2	59.8	–	–	–	–	56.7	78.0	88.3	96.9
	align-rope	packing-unseen-shapes	assembling-kits-seq seen-colors	assembling-kits-seq	seen-colors	assembling-kits-seq unseen-colors	assembling-kits-seq	unseen-colors	put-blocks-in-bowls seen-colors	put-blocks-in-bowls	seen-colors	put-blocks-in-bowls unseen-colors	put-blocks-in-bowls	unseen-colors
assembling-kits-seq
seen-colors
assembling-kits-seq
unseen-colors
put-blocks-in-bowls
seen-colors
put-blocks-in-bowls
unseen-colors
	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000
Transporter-only [2]	6.9	30.6	33.1	51.5	16.0	20.0	22.0	22.0	5.8	11.6	28.6	29.6	7.8	17.6	25.6	28.4	16.8	33.3	62.7	64.7	11.7	17.2	14.8	18.7
CLIP-only	13.4	48.7	70.4	70.7	13.0	28.0	44.0	50.0	0.8	9.2	19.8	23.0	2.0	4.6	10.8	19.8	23.5	60.2	93.5	97.7	11.2	34.2	33.2	44.5
RN50-BERT	3.1	25.0	63.8	57.1	19.0	25.0	32.0	44.0	2.2	5.6	11.6	21.8	1.6	6.4	10.4	18.4	13.8	44.5	81.2	91.8	16.2	23.0	30.3	23.8
CLIPort (single)	20.1	77.4	85.6	95.4	21.0	26.0	40.0	37.0	12.2	17.8	47.0	66.6	16.2	18.0	35.4	34.8	23.5	68.3	92.5	100	18.0	35.3	37.3	25.0
CLIPort (multi)	19.6	49.3	82.4	74.9	25.0	35.0	37.0	31.0	11.4	34.8	46.2	52.4	7.8	21.6	29.0	25.4	54.0	90.2	99.5	100	32.0	48.8	55.3	45.8
CLIPort (multi-attr)	–	–	–	–	–	–	–	–	–	–	–	–	7.6	10.4	43.8	34.6	–	–	–	–	23.0	41.8	66.5	75.7

packing-box-pairs
seen-colors

packing-box-pairs
unseen-colors

packing-seen-google
objects-seq

packing-unseen-google
objects-seq

packing-seen-google
objects-group

packing-unseen-google
objects-group

stack-block-pyramid
seq-seen-colors

stack-block-pyramid
seq-unseen-colors

separating-piles
seen-colors

separating-piles
unseen-colors

towers-of-hanoi
seq-seen-colors

towers-of-hanoi
seq-unseen-colors

assembling-kits-seq
seen-colors

assembling-kits-seq
unseen-colors

put-blocks-in-bowls
seen-colors

put-blocks-in-bowls
unseen-colors

Table 1: Language-Conditioned Test Results. Task success scores (mean %) from 100 evaluation instances vs. # of training demonstrations (1, 10, 100, or 1000). The challenges pertaining to each task are described in Appendix A. CLIPort (single) models are trained on seen splits, and evaluated on both seen and unseen splits. CLIPort (multi) models are trained on seen splits of all 10 tasks with $1\mathbb{T}$, $10\mathbb{T}$, $100\mathbb{T}$, and $1000\mathbb{T}$ demonstrations where $\mathbb{T}=10$. CLIPort (multi-attr) indicate CLIPort (multi) models trained on seen-and-unseen splits from all tasks except for that one particular heldout task, for which it is trained only the seen split. See Figure 3 for an overview with average scores.

	packing-box-pairs seen-colors	packing-box-pairs	seen-colors	packing-box-pairs unseen-colors	packing-box-pairs	unseen-colors	packing-seen-google objects-seq	packing-seen-google	objects-seq	packing-unseen-google objects-seq	packing-unseen-google	objects-seq	packing-seen-google objects-group	packing-seen-google	objects-group	packing-unseen-google objects-group	packing-unseen-google	objects-group
packing-box-pairs
seen-colors
packing-box-pairs
unseen-colors
packing-seen-google
objects-seq
packing-unseen-google
objects-seq
packing-seen-google
objects-group
packing-unseen-google
objects-group
方法	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000
Transporter-only [2]	44.2	55.2	54.2	52.4	34.6	48.7	47.2	54.1	26.2	39.7	45.4	46.3	19.9	29.8	28.7	37.3	60.0	54.3	61.5	59.9	46.2	54.7	49.8	52.0
CLIP-only	38.6	69.7	88.5	87.1	33.0	65.5	68.8	61.2	29.1	67.9	89.3	95.8	37.1	49.4	60.4	57.8	52.5	62.0	89.6	92.7	43.4	65.9	73.1	70.0
RN50-BERT	36.2	64.0	94.7	90.3	31.4	52.7	65.6	72.1	32.9	48.4	87.9	94.0	29.3	48.5	48.3	56.1	46.4	52.9	76.5	86.4	43.2	52.0	66.3	73.7
CLIPort (single)	51.6	82.9	92.7	98.2	45.6	65.3	68.6	71.5	14.8	59.5	86.8	96.2	27.2	50.0	65.5	71.9	52.7	67.0	84.1	94.0	61.5	66.2	78.4	81.5
CLIPort (multi)	66.8	88.6	94.1	96.6	59.0	69.7	76.2	71.4	41.6	78.4	85.0	84.4	40.7	51.1	65.8	70.3	71.3	84.6	89.6	88.3	68.4	69.6	78.4	80.3
CLIPort (multi-attr)	–	–	–	–	46.2	72.0	86.2	80.3	–	–	–	–	35.4	45.1	78.9	87.4	–	–	–	–	48.6	69.3	84.8	89.1
	stack-block-pyramid seq-seen-colors	stack-block-pyramid	seq-seen-colors	stack-block-pyramid seq-unseen-colors	stack-block-pyramid	seq-unseen-colors	separating-piles seen-colors	separating-piles	seen-colors	separating-piles unseen-colors	separating-piles	unseen-colors	towers-of-hanoi seq-seen-colors	towers-of-hanoi	seq-seen-colors	towers-of-hanoi seq-unseen-colors	towers-of-hanoi	seq-unseen-colors
stack-block-pyramid
seq-seen-colors
stack-block-pyramid
seq-unseen-colors
separating-piles
seen-colors
separating-piles
unseen-colors
towers-of-hanoi
seq-seen-colors
towers-of-hanoi
seq-unseen-colors
	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000
Transporter-only [2]	4.5	2.3	5.2	4.5	3.0	4.0	2.3	5.8	42.7	52.3	42.0	48.4	41.2	49.2	44.7	52.3	25.4	67.9	98.0	99.9	24.3	44.6	71.7	80.7
CLIP-only	6.3	28.7	55.7	54.8	2.0	12.2	18.3	19.5	43.5	55.0	84.9	90.2	59.9	49.6	73.0	71.0	9.4	52.6	88.6	45.3	24.7	47.0	67.0	58.0
RN50-BERT	5.3	35.0	89.0	97.5	6.2	12.2	21.5	30.7	31.8	47.8	46.5	46.5	33.4	44.4	41.3	44.9	28.0	66.1	91.3	92.1	17.4	75.1	85.3	89.3
CLIPort (single)	28.3	64.7	93.3	98.8	13.7	24.3	31.2	41.3	54.5	59.5	93.1	98.0	47.2	51.0	76.6	75.2	59.4	92.9	97.4	100	56.1	89.7	95.9	99.4
CLIPort (multi)	33.5	75.3	96.8	96.5	23.3	26.8	31.7	22.2	48.9	72.4	90.3	89.0	56.6	62.6	64.9	62.8	61.6	96.3	98.7	98.1	60.1	65.6	76.7	68.7
CLIPort (multi-attr)	–	–	–	–	15.5	51.5	59.3	79.8	–	–	–	–	49.9	51.8	48.2	59.8	–	–	–	–	56.7	78.0	88.3	96.9
	align-rope	packing-unseen-shapes	assembling-kits-seq seen-colors	assembling-kits-seq	seen-colors	assembling-kits-seq unseen-colors	assembling-kits-seq	unseen-colors	put-blocks-in-bowls seen-colors	put-blocks-in-bowls	seen-colors	put-blocks-in-bowls unseen-colors	put-blocks-in-bowls	unseen-colors
assembling-kits-seq
seen-colors
assembling-kits-seq
unseen-colors
put-blocks-in-bowls
seen-colors
put-blocks-in-bowls
unseen-colors
	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000	1	10	100	1000
Transporter-only [2]	6.9	30.6	33.1	51.5	16.0	20.0	22.0	22.0	5.8	11.6	28.6	29.6	7.8	17.6	25.6	28.4	16.8	33.3	62.7	64.7	11.7	17.2	14.8	18.7
CLIP-only	13.4	48.7	70.4	70.7	13.0	28.0	44.0	50.0	0.8	9.2	19.8	23.0	2.0	4.6	10.8	19.8	23.5	60.2	93.5	97.7	11.2	34.2	33.2	44.5
RN50-BERT	3.1	25.0	63.8	57.1	19.0	25.0	32.0	44.0	2.2	5.6	11.6	21.8	1.6	6.4	10.4	18.4	13.8	44.5	81.2	91.8	16.2	23.0	30.3	23.8
CLIPort (single)	20.1	77.4	85.6	95.4	21.0	26.0	40.0	37.0	12.2	17.8	47.0	66.6	16.2	18.0	35.4	34.8	23.5	68.3	92.5	100	18.0	35.3	37.3	25.0
CLIPort (multi)	19.6	49.3	82.4	74.9	25.0	35.0	37.0	31.0	11.4	34.8	46.2	52.4	7.8	21.6	29.0	25.4	54.0	90.2	99.5	100	32.0	48.8	55.3	45.8
CLIPort (multi-attr)	–	–	–	–	–	–	–	–	–	–	–	–	7.6	10.4	43.8	34.6	–	–	–	–	23.0	41.8	66.5	75.7

packing-box-pairs
seen-colors

packing-box-pairs
unseen-colors

packing-seen-google
objects-seq

packing-unseen-google
objects-seq

packing-seen-google
objects-group

packing-unseen-google
objects-group

stack-block-pyramid
seq-seen-colors

stack-block-pyramid
seq-unseen-colors

separating-piles
seen-colors

separating-piles
unseen-colors

towers-of-hanoi
seq-seen-colors

towers-of-hanoi
seq-unseen-colors

assembling-kits-seq
seen-colors

assembling-kits-seq
unseen-colors

put-blocks-in-bowls
seen-colors

put-blocks-in-bowls
unseen-colors

表 1：语言条件测试结果。 在 100 次评测实例上的任务成功分数（均值 %），对比训练演示条数（1、10、100 或 1000）。各任务挑战见附录 A。CLIPort (single) 在 seen 划分上训练，并在 seen 与 unseen 划分上评测。CLIPort (multi) 在全部 10 个任务的 seen 划分上训练，演示规模为 $1\mathbb{T}$、$10\mathbb{T}$、$100\mathbb{T}$、$1000\mathbb{T}$，其中 $\mathbb{T}=10$。CLIPort (multi-attr) 表示：除被留出评测的那一个任务仅在 seen 划分上训练外，其余任务的 seen 与 unseen 划分均参与训练的 CLIPort (multi)。平均分概览见图 3。

Generalizing to Unseen Attributes. Tasks that require generalizing to novel colors, shapes, and objects are more difficult and all our agents achieve relatively lower performance on these tasks, as shown in Figure 3 (unseen). However, CLIPort (single) models do substantially better than chance, i.e., Transporter-only. The lower performances are due to the difficulty of grounding unseen attributes such as ‘pink’ and ‘orange’ in the language instruction “put the pink block on the orange bowl”, when the agent has never encountered words ‘orange’, ‘pink’ or their corresponding visual characteristics in the context of the physical environment. Although pre-trained CLIP has been exposed to the attribute ‘pink’, it could correspond to different concepts in the physical setting depending on factors like lighting condition, and thus requires at least few examples to condition the trainable semantic decoder layers. Additionally, we notice that CLIPort (single) is also less prone to overfitting compared to Transporter-only. As evidenced in towers-of-hanoi-seq-unseen-colors task in Table 1, Transporter-only suffers from a performance drop because of rings with unseen colors despite the fact that Tower of Hanoi can be solved without attending to the colors and simply focusing on the ring size. We hypothesize that since CLIP was trained on diverse internet data, it enables our agent to focus on task-relevant concepts while ignoring irrelevant aspects of the task.

泛化到未见属性。 需要泛化到新颜色、形状与物体的任务更难，所有智能体在这些任务上的表现相对较低，如图 3（unseen）所示。不过，CLIPort (single) 仍明显好于偶然水平，即 Transporter-only。分数偏低，是因为要在诸如「把粉色方块放到橙色碗上」这类语言指令中锚定「pink」「orange」等未见属性——而智能体从未在物理环境语境中见过这些词或其对应视觉特征。尽管预训练 CLIP 接触过「pink」属性，但在物理设定中它可能因光照等因素对应不同概念，因而至少需要少量示例来条件化可训练的 语义 解码层。此外，我们注意到相对 Transporter-only，CLIPort (single) 也更不易过拟合。如表 1 中 towers-of-hanoi-seq-unseen-colors 所示：尽管汉诺塔只需关注环尺寸、不必看颜色即可求解，Transporter-only 仍因未见颜色的环而掉分。我们猜想，由于 CLIP 在多样互联网数据上训练，它帮助智能体聚焦任务相关概念并忽略无关方面。

Transferring Attributes across Tasks. One solution for dealing with unseen attributes is to explicitly learn these attributes from other tasks. We study this with CLIPort (multi-attr) in Table 1 and Figure 3 (unseen). For these models, CLIPort (multi) is trained on both seen-and-unseen splits from all tasks except for the task being evaluated on, for which it was only trained on the seen split. As such, this evaluation measures whether having seen pink blocks in put-blocks-in-bowl-unseen-colors helps solve “pack all the pink and cyan boxes” in packing-box-pairs-unseen-colors. Results indicate that such explicit transfers result in significant improvements. For instance, on the put-blocks-in-bowls-unseen-colors task for $n=1000$, CLIPort (multi)’s performance increases from $45.8$ to $75.7$.

跨任务迁移属性。 处理未见属性的一个办法，是显式从其他任务学习这些属性。我们用表 1 与图 3（unseen）中的 CLIPort (multi-attr) 加以研究。对这些模型，CLIPort (multi) 在除评测任务外的全部任务的 seen 与 unseen 划分上训练；对被评测任务，则仅在 seen 划分上训练。因此，该评测衡量的是：在 put-blocks-in-bowl-unseen-colors 中见过粉色方块，是否有助于在 packing-box-pairs-unseen-colors 中完成「装好所有粉色与青色盒子」。结果表明，这种显式迁移带来显著提升。例如，在 put-blocks-in-bowls-unseen-colors、$n=1000$ 时，CLIPort (multi) 的表现从 $45.8$ 升至 $75.7$。

4.3 Real-Robot Experiments

4.3 真实机器人实验

Task	# Train (Samples)	# Test	Succ. %
Stack Blocks	05 (13)	10	70.0
Put Blocks in Bowl	05 (10)	10	65.0
Pack Objects	10 (31)	10	60.0
Move Rook	04 (29)	10	70.0
Fold Cloth	9 (9)	10	57.0
Read Text	02 (26)	10	55.0
Loop Rope	04 (12)	10	60.0
Sweep Beans	05 (23)	5	60.6
Pick Cherries	04 (26)	5	75.0

Table 2: Success rates (%) of a multi-task model trained an evaluated 9 real-world tasks (see Figure 1). Samples indicate total image-action pairs, e.g 1 in Figure 9.

任务	训练数（样本）	测试数	成功率 %
Stack Blocks	05 (13)	10	70.0
Put Blocks in Bowl	05 (10)	10	65.0
Pack Objects	10 (31)	10	60.0
Move Rook	04 (29)	10	70.0
Fold Cloth	9 (9)	10	57.0
Read Text	02 (26)	10	55.0
Loop Rope	04 (12)	10	60.0
Sweep Beans	05 (23)	5	60.6
Pick Cherries	04 (26)	5	75.0

表 2：在 9 个真实任务上训练并评测的多任务模型成功率（%）（见图 1）。样本指图像—动作对总数，例如图 9 中的 1。

We validated our results in hardware with a Franka Panda manipulator. See Appendix D for setup details. Table 2 reports success rates for a multi-task model trained and evaluated on 9 real-world tasks. Due to COVID restrictions, we could not conduct large-scale user-studies, so we report on small train (5-10 demos) and test sets (5-10 runs) per task. Overall, CLIPort (multi) is effective at few-shot learning with just 179 samples, and the performances roughly correspond to those in simulated experiments, with simple block manipulation tasks achieving $\sim 70\%$. We estimate that for more robust real-world performance at least 50 to 100 training demonstrations are necessary, as evident in Figure 3. Interestingly, we observed that the model sometimes exploits biases in the training data instead of learning to ground instructions. For instance, in Put Blocks in Bowl, the training set consisted of only one datapoint on “yellow blocks” being placed inside a “blue bowl”. This made it difficult to condition the model to place “yellow blocks” in non-blue bowls. But instances with just one or two examples where a colored block went to different colored bowls was sufficient to make the model pay attention to the language. In summary, unbiased datasets containing both a good coverage of expected skills and invariances, and a decent number of training demonstrations, are crucial for good real-world performance.

我们在 Franka Panda 机械臂上用硬件验证了结果。设定细节见附录 D。表 2 报告在 9 个真实任务上训练并评测的多任务模型成功率。受疫情限制，无法开展大规模用户研究，因而每任务仅报告较小的训练集（5–10 条演示）与测试集（5–10 次运行）。总体而言，CLIPort (multi) 用仅 179 个样本即可有效少样本学习，表现大致对应仿真实验，简单方块操作任务约 $\sim 70\%$。我们估计，要获得更稳健的真实表现，至少需要 50 到 100 条训练演示，如图 3 所示。有趣的是，我们观察到模型有时会利用训练数据中的偏差，而不是学会锚定指令。例如在 Put Blocks in Bowl 中，训练集里关于「黄色方块」放进「蓝色碗」只有一个数据点，这使模型很难被条件化为把「黄色方块」放到非蓝色碗中。但只要有一两个彩色方块放到不同颜色碗的例子，就足以让模型关注语言。总之，无偏数据集——既覆盖期望技能与不变性，又有足够训练演示——对良好真实表现至关重要。

Figure 4: Affordance predictions from CLIPort (multi) models in sim (left two) and real settings (right three). More examples in Appendix H.

图 4：仿真（左两列）与真实设定（右三列）中 CLIPort (multi) 模型的可供性预测。更多例子见附录 H。

5 Conclusion

5 结论

We introduced CLIPort, an end-to-end framework for language-conditioned fine-grained manipulation. Our experiments, specifically with multi-task models, indicate that data-driven approaches to generalization have yet to be fully-exploited in robotics. Coupled with the right action abstraction and spatio-semantic priors, end-to-end methods can quickly learn new skills without requiring top-down pipelines that need task-specific engineering.

我们提出了 CLIPort——面向语言条件精细操作的端到端框架。实验（尤其是多任务模型）表明，机器人学中数据驱动的泛化路径尚未被充分挖掘。配以恰当的动作抽象与时空—语义先验，端到端方法可以快速学会新技能，而无需依赖按任务定制工程的自上而下管线。

While CLIPort can solve a range of tabletop tasks, extending it to dexterous 6-DOF manipulation that goes beyond the two-step primitive remains a challenge. As such, it cannot handle complex partially-observable scenes, or output continuous control for multi-fingered hands, or predict task-completion (see Appendix I for an extended discussion). But overall, we are excited by the confluence of data and structural priors for building scalable and generalizable robotic systems.

尽管 CLIPort 能解决一系列桌面任务，将其扩展到超越两步原语的灵巧 6 自由度操作仍是挑战。因此，它尚不能处理复杂的部分可观测场景，也不能为多指手输出连续控制，或预测任务完成（扩展讨论见附录 I）。但总体而言，我们为数据与结构先验的汇合——用以构建可扩展、可泛化的机器人系统——感到兴奋。

Acknowledgments

All simulated experiments were facilitated through the Hyak computing cluster funded by the STF at the University of Washington. We thank Mohak Bhardwaj for help with the Franka setup at UW. We are also grateful to our colleagues Chris Xie, Jesse Thomason, and Valts Blukis for providing feedback on the initial draft. This work was funded in part by ONR under award #1140209-405780.

致谢

全部仿真实验借助华盛顿大学 STF 资助的 Hyak 计算集群完成。感谢 Mohak Bhardwaj 协助 UW 的 Franka 设定。亦感谢同事 Chris Xie、Jesse Thomason 与 Valts Blukis 对初稿的反馈。本研究部分由 ONR 资助，编号 #1140209-405780。

References

参考文献

[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020 [cs] , Feb. 2021.

[2] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL) , 2020.

[3] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019.

[4] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning (CoRL) , 2018.

[5] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212 , 2021.

[6] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In IEEE International Conference on Robotics and Automation (ICRA) , 2021.

[7] M. Shridhar and D. Hsu. Interactive visual grounding of referring expressions for human-robot interaction. In Proceedings of Robotics: Science and Systems (RSS) , 2018.

[8] C. Matuszek, L. Bo, L. Zettlemoyer, and D. Fox. Learning from unscripted deictic gesture and language for human-robot interactions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, 2014.

[9] M. Bollini, S. Tellex, T. Thompson, N. Roy, and D. Rus. Interpreting and executing recipes with a cooking robot. In Experimental Robotics , pages 481–495. Springer, 2013.

[10] D. K. Misra, J. Sung, K. Lee, and A. Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research (IJRR) , 35(1-3):281–300, 2016.

[11] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597–1607. PMLR, 2020.

[12] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9729–9738, 2020.

[13] J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems (NeuRIPS) , 2019.

[14] Y. C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision , pages 104–120. Springer, 2020.

[15] H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2019.

[16] D. H. Hubel and T. N. Wiesel. Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. Journal of neurophysiology , 28(2):229–289, 1965.

[17] M. Livingstone and D. Hubel. Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science , 240(4853):740–749, 1988.

[18] A. Derrington and P. Lennie. Spatial and temporal contrast sensitivities of neurones in lateral geniculate nucleus of macaque. The Journal of physiology , 357(1):219–240, 1984.

[19] J. J. Gibson. The ecological approach to visual perception: classic edition . Psychology Press, 2014.

[20] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. Mdetr–modulated detection for end-to-end multi-modal understanding. arXiv preprint arXiv:2104.12763 , 2021.

[21] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

[22] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. In Proceedings of Robotics: Science and Systems (RSS) , 2018.

[23] M. Zhu, K. G. Derpanis, Y. Yang, S. Brahmbhatt, M. Zhang, C. Phillips, M. Lecce, and K. Daniilidis. Single image 3d object detection and pose estimation for grasping. In 2014 IEEE International Conference on Robotics and Automation (ICRA) , pages 3936–3943. IEEE, 2014.

[24] A. Zeng, K.-T. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and J. Xiao. Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 1386–1383. IEEE, 2017.

[25] X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 3665–3671. IEEE, 2020.

[26] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. In Conference on Robot Learning (CoRL) , pages 1369–1378. PMLR, 2020.

[27] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference on Robot Learning (CoRL) , 2018.

[28] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters , 5(2):492–499, 2019.

[29] P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 9411–9418. IEEE, 2020.

[30] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. In International Symposium on Robotics Research (ISRR) , 2019.

[31] T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems (NeuRIPS) , 32:10724–10734, 2019.

[32] X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige. Keypose: Multi-view 3d labeling and keypoint estimation for transparent objects. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11602–11610, 2020.

[33] K. Zakka, A. Zeng, J. Lee, and S. Song. Form2fit: Learning shape priors for generalizable assembly from disassembly. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 9404–9410. IEEE, 2020.

[34] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters , 5(3):4978–4985, 2020.

[35] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to Manipulate Deformable Objects without Demonstrations. In Proceedings of Robotics: Science and Systems (RSS) , 2020.

[36] L. Yen-Chen, A. Zeng, S. Song, P. Isola, and T.-Y. Lin. Learning to see before learning to act: Visual pre-training for manipulation. In IEEE International Conference on Robotics and Automation (ICRA) , 2020.

[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeuRIPS) , 2017.

[38] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2018.

[39] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

[40] F. Yu, J. Tang, W. Yin, Y. Sun, H. Tian, H. Wu, and H. Wang. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934 , 2020.

[41] Y. Bisk, D. Yuret, and D. Marcu. Natural language communication with robots. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL) , pages 751–761, 2016.

[42] J. Thomason, S. Zhang, R. J. Mooney, and P. Stone. Learning to interpret natural language commands through human-robot dialog. In Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI) , 2015.

[43] J. Hatori, Y. Kikuchi, S. Kobayashi, K. Takahashi, Y. Tsuboi, Y. Unno, W. Ko, and J. Tan. Interactively picking real-world objects with unconstrained spoken language instructions. In Proceedings of International Conference on Robotics and Automation (ICRA) , 2018.

[44] Y. Chen, R. Xu, Y. Lin, and P. A. Vela. A Joint Network for Grasp Detection Conditioned on Natural Language Commands. arXiv:2104.00492 [cs] , Apr. 2021.

[45] V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding for mapping natural language instructions to robot control. In Conference on Robot Learning (CoRL) , 2020.

[46] C. Paxton, Y. Bisk, J. Thomason, A. Byravan, and D. Fox. Prospection: Interpretable plans from language by predicting the future. In International Conference on Robotics and Automation (ICRA) , pages 6942–6948. IEEE, 2019.

[47] S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2011.

[48] C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648 , 2020.

[49] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. arXiv preprint arXiv:1406.2199 , 2014.

[50] C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1933–1941, 2016.

[51] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR) , pages 6202–6211, 2019.

[52] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen. Slow-fast auditory streams for audio recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 855–859. IEEE, 2021.

[53] F. Xiao, Y. J. Lee, K. Grauman, J. Malik, and C. Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740 , 2020.

[54] A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3750–3757. IEEE, 2018.

[55] E. Jang, S. Vijayanarasimhan, P. Pastor, J. Ibarz, and S. Levine. End-to-end learning of semantic grasping. In Conference on Robot Learning (CoRL) , Proceedings of Machine Learning Research. PMLR, 2017.

[56] R. Kondor and S. Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning (ICML) , 2018.

[57] T. Cohen and M. Welling. Group equivariant convolutional networks. In International conference on machine learning (ICML) , 2016.

[58] D. Misra, A. Bennett, V. Blukis, E. Niklasson, M. Shatkhin, and Y. Artzi. Mapping instructions to actions in 3d environments with visual goal prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018.

[59] G. Goh, N. C. †, C. V. †, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah. Multimodal neurons in artificial neural networks. Distill , 2021. doi: 10.23915/distill.00030 . https://distill.pub/2021/multimodal-neurons.

[60] E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.

[61] Google scanned objects dataset, 2020. URL https://app.ignitionrobotics.org/GoogleResearch/fuel/collections/Google%20Scanned%20Objects .

[62] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770–778, 2016.

[63] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lee. 12-in-1: Multi-task vision and language representation learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

[64] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 , 2019.

[65] V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.

[66] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248–255. Ieee, 2009.

[67] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.12584 , 2019.

[68] D. Ding, F. Hill, A. Santoro, and M. Botvinick. Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv preprint arXiv:2012.08508 , 2020.

[69] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 610–623, 2021.

Appendix A Task Details

附录 A 任务细节

	precise	multimodal	multi-step	unseen	unseen	unseen	language
Task	placing	placing	sequencing	poses	colors	objects	instruction
put-blocks-in-bowls-seen-colors*∗*	✗	✓	✗	✓	✗	✗	goal
put-blocks-in-bowls-unseen-colors*∗*	✗	✓	✗	✓	✓	✗	goal
assembling-kits-seq-seen-colors	✓	✓	✓	✓	✗	✓	step
assembling-kits-seq-unseen-colors	✓	✓	✓	✓	✓	✓	step
packing-unseen-shapes	✗	✓	✗	✓	✓	✓	goal
stack-block-pyramid-seq-seen-colors	✓	✓	✓	✓	✗	✗	step
stack-block-pyramid-seq-unseen-colors	✓	✓	✓	✓	✓	✗	step
towers-of-hanoi-seq-seen-colors	✓	✓	✓	✓	✗	✗	step
towers-of-hanoi-seq-unseen-colors	✓	✓	✓	✓	✓	✗	step
packing-box-pairs-seen-colors*∗§*	✓	✓	✓	✓	✗	✓	goal
packing-box-pairs-unseen-colors*∗§*	✓	✓	✓	✓	✓	✓	goal
packing-seen-google-objects-seq*§*	✗	✓	✓	✓	✗	✗	step
packing-unseen-google-objects-seq*§*	✗	✓	✓	✓	✓	✓	step
packing-seen-google-objects-group*∗§*	✗	✓	✗	✓	✗	✗	goal
packing-unseen-google-objects-group*∗§*	✗	✓	✗	✓	✓	✓	goal
align-rope*∗†*	✓	✓	✓	✓	✗	✗	goal
separating-piles-seen-colors*∗†*	✓	✓	✓	✓	✗	✗	goal
separating-piles-unseen-colors*∗†*	✓	✓	✓	✓	✓	✗	goal

Table 3: Language-conditioned tasks in Ravens [2] with their associated challenges.

	精确	多模态	多步	未见	未见	未见	语言
任务	放置	放置	排序	位姿	颜色	物体	指令
put-blocks-in-bowls-seen-colors*∗*	✗	✓	✗	✓	✗	✗	goal
put-blocks-in-bowls-unseen-colors*∗*	✗	✓	✗	✓	✓	✗	goal
assembling-kits-seq-seen-colors	✓	✓	✓	✓	✗	✓	step
assembling-kits-seq-unseen-colors	✓	✓	✓	✓	✓	✓	step
packing-unseen-shapes	✗	✓	✗	✓	✓	✓	goal
stack-block-pyramid-seq-seen-colors	✓	✓	✓	✓	✗	✗	step
stack-block-pyramid-seq-unseen-colors	✓	✓	✓	✓	✓	✗	step
towers-of-hanoi-seq-seen-colors	✓	✓	✓	✓	✗	✗	step
towers-of-hanoi-seq-unseen-colors	✓	✓	✓	✓	✓	✗	step
packing-box-pairs-seen-colors*∗§*	✓	✓	✓	✓	✗	✓	goal
packing-box-pairs-unseen-colors*∗§*	✓	✓	✓	✓	✓	✓	goal
packing-seen-google-objects-seq*§*	✗	✓	✓	✓	✗	✗	step
packing-unseen-google-objects-seq*§*	✗	✓	✓	✓	✓	✓	step
packing-seen-google-objects-group*∗§*	✗	✓	✗	✓	✗	✗	goal
packing-unseen-google-objects-group*∗§*	✗	✓	✗	✓	✓	✓	goal
align-rope*∗†*	✓	✓	✓	✓	✗	✗	goal
separating-piles-seen-colors*∗†*	✓	✓	✓	✓	✗	✗	goal
separating-piles-unseen-colors*∗†*	✓	✓	✓	✓	✓	✗	goal

表 3：Ravens [2] 中的语言条件任务及其相关挑战。

We extend the Ravens benchmark [2] to 10 language-conditioned. 8 out of 10 tasks have two evaluation variants, denoted by seen and unseen in their names. See Table A for an overview of the challenges associated with each task and split. Figure 5 presents the full list of attributes, shapes, and objects across seen and unseen splits. All tasks use hand-coded experts to generate expert demonstrations. These experts use privileged state information from the simulator along with pre-specified heuristics to complete the tasks. We refer the reader to the original Transporter paper [2] for details regarding these experts. The following is a description of each language-conditioned task:

我们将 Ravens 基准 [2] 扩展为 10 个语言条件任务。其中 8/10 具有两个评测变体，名称中以 seen 与 unseen 区分。各任务与划分的挑战概览见表 A。图 5 给出 seen 与 unseen 划分上的完整属性、形状与物体列表。所有任务用手工编码的专家生成演示；这些专家利用仿真器的特权状态信息与预设启发式完成任务。专家细节见原版 Transporter 论文 [2]。以下是各语言条件任务的描述：

Figure 5: Attributes and Objects: Attributes and objects across seen and unseen splits. Shapes objects are from Transporter [2]. Other tabletop objects are from the Google Scanned Objects dataset [61]

图 5：属性与物体： seen 与 unseen 划分上的属性与物体。形状物体来自 Transporter [2]。其他桌面物体来自 Google Scanned Objects 数据集 [61]。

A.1 Align Rope

Example: Figure 1(a).

示例： 图 1(a)。

Task: Manipulate a deformable rope to connect its end-points between two corners of a 3-sided square. There are four possible combinations for aligning the rope: “front left tip to front right tip”. “front right tip to back right corner”, “front left tip to back left corner”, and “back right corner to back left corner”. Here ‘front’ and ‘back’ refer to canonical positions on the 3-sided square. The poses of both the rope and 3-sided square are randomized for each task instance.

任务： 操作可变形绳，使其两端连接三边方框的两个角。对齐绳有四种可能组合：“front left tip to front right tip”、“front right tip to back right corner”、“front left tip to back left corner” 与 “back right corner to back left corner”。此处 front 与 back 指三边方框上的规范位置。绳与三边方框的位姿在每个任务实例中随机。

Objects: All align-rope instances contain a rope with 20 articulated beads and a 3-sided square.

物体： 所有 align-rope 实例含一条由 20 个关节珠组成的绳与一个三边方框。

Success Metric: The poses of all beads match the line segments between the two correct sides.

成功度量： 全部珠的位姿与两条正确边之间的线段一致。

A.2 Packing Unseen Shapes

Example: Figure 1(b).

示例： 图 1(b)。

Task: Place a specified shape in the brown box. Each task instance contains 1 shape to be picked along with 4 distractor shapes. The shape colors are randomized but have no relevance to the task. This task does not require precise placements and is mostly a test of the agent’s semantic understanding of arbitrary shapes.

任务： 把指定形状放进棕色箱子。每个任务实例含 1 个待抓形状与 4 个干扰形状。形状颜色随机，但与任务无关。该任务不要求精确放置，主要检验智能体对任意形状的语义理解。

Objects: packing-unseen-shapes is trained with seen shapes but evaluated on unseen shapes from Figure 5.

物体： packing-unseen-shapes 在 seen 形状上训练，在图 5 的 unseen 形状上评测。

Success Metric: The correct shape is inside the bounds of the brown box.

成功度量： 正确形状位于棕色箱子边界内。

A.3 Assembling Kits Seq

Example: Figure 1(c).

示例： 图 1(c)。

Task: Precisely place each specified shape in the specified hole following the order prescribed in the language instruction at each timestep. This is one of the hardest tasks in the benchmark requiring precise placements of unseen shapes of unseen colors and grounding spatial relationships like “the middle square hole” or “the bottom letter R hole”. Each task instance contains 5 shapes and a kit with randomized poses.

任务： 按每时刻语言指令规定的顺序，把每个指定形状精确放入指定孔位。这是基准中最难任务之一：需要未见颜色的未见形状的精确放置，并锚定如「中间的方孔」或「底部字母 R 孔」这类空间关系。每个任务实例含 5 个形状与一个位姿随机的套件。

Objects: Both assembling-kits-seq-seen-colors and assembling-kits-seq-unseen-colors are trained on seen shapes but evaluated on unseen shapes from Figure 5. However for color randomization, assembling-kits-seq-seen-colors is trained and evaluated on seen colors, and assembling-kits-seq-unseen-colors is trained with seen colors but evaluated on unseen colors from Figure 5.

物体： assembling-kits-seq-seen-colors 与 assembling-kits-seq-unseen-colors 均在 seen 形状上训练、在图 5 的 unseen 形状上评测。颜色方面，前者在 seen 颜色上训练与评测；后者在 seen 颜色上训练、在图 5 的 unseen 颜色上评测。

Success Metric: The pose of each shape matches the specified hole at the correct timestep. The final score is the total number of shapes that were placed in the correct pose at the correct timestep, divided by the total number of shapes in the scene (always 5).

成功度量： 每个形状在正确时刻的位姿与指定孔位一致。最终分数为在正确时刻放入正确位姿的形状数，除以场景中形状总数（恒为 5）。

A.4 Put Blocks in Bowl

Example: Figure 1(d).

示例： 图 1(d)。

Task: Place all blocks of a specified color in a bowl of specified color. Each bowl fits just one block and all scenes contain enough bowls achieve the goal. Each task instance contains several distractor blocks and bowls with randomized colors. The solutions to this task are multi-modal in that there could be several ways to place the blocks specified in the language goal. This task does not require precise placements and mostly tests an agent’s ability to ground color attributes.

任务： 把指定颜色的全部方块放进指定颜色的碗。每个碗只装一块，场景中碗的数量足以达成目标。每个任务实例含若干颜色随机的干扰方块与碗。解是多模态的：语言目标指定的方块可有多种放置方式。该任务不要求精确放置，主要检验智能体锚定颜色属性的能力。

Objects: put-blocks-in-bowl-seen-colors is trained and evaluated on seen colors from Figure 5 for both blocks and bowls. put-blocks-in-bowl-unseen-colors is trained on seen colors but evaluated on unseen colors from Figure 5 for both blocks and bowls.

物体： put-blocks-in-bowl-seen-colors 对方块与碗均在图 5 的 seen 颜色上训练与评测；put-blocks-in-bowl-unseen-colors 在 seen 颜色上训练，在图 5 的 unseen 颜色上评测。

Success Metric: All blocks of the specified color are within the bounds a bowl of the specified color. The final score is the total number of correct blocks in the correct bowls, divided by the total number of relevant color blocks in the scene.

成功度量： 指定颜色的全部方块位于指定颜色碗的边界内。最终分数为正确碗中正确方块数，除以场景中相关颜色方块总数。

A.5 Packing Box Pairs

Example: Figure 1(e).

示例： 图 1(e)。

Task: Tightly pack all the boxes of two specified colors inside the brown box. All scenes contain the exact number of relevant color blocks to fill the box completely, but also contain some distractor boxes of irrelevant colors. The sizes of the boxes and the brown box are randomized. The distractor objects have equivalent sizes to the relevant objects to make the task more difficult. Sometimes the scene only contains one of the two specified specified colors and the agent has to actively ignore the missing color. Overall, this task requires both semantic understanding of colors and precise spatial reasoning for tightly packing boxes of unknown sizes.

任务： 把两种指定颜色的全部盒子紧密装进棕色箱子。所有场景恰好含填满箱子所需数量的相关颜色块，同时还有一些无关颜色的干扰盒。盒子与棕色箱子尺寸随机；干扰物尺寸与相关物体相当，以增加难度。有时场景只含两种指定颜色之一，智能体须主动忽略缺失颜色。总体而言，该任务既需要颜色语义理解，也需要为未知尺寸盒子紧密装箱的精确空间推理。

Objects: Boxes with randomized widths and lengths and a brown box. packing-box-pairs-seen-colors is trained and evaluated on seen color boxes from Figure 5. packing-box-pairs-unseen-colors is trained on seen color boxes but evaluated on unseen color boxes from Figure 5.

物体： 宽度与长度随机的盒子以及一个棕色箱子。packing-box-pairs-seen-colors 在图 5 的 seen 颜色盒上训练与评测；packing-box-pairs-unseen-colors 在 seen 颜色盒上训练、在图 5 的 unseen 颜色盒上评测。

Success Metric: All blocks of the two specified colors are tightly packed inside the bounds of the brown box. The final score is the total volume of the correct color blocks inside the box, divided by the total volume of the relevant color blocks in the scene.

成功度量： 两种指定颜色的全部块紧密装在棕色箱子边界内。最终分数为箱内正确颜色块的总体积，除以场景中相关颜色块总体积。

A.6 Packing Google Objects Seq

Example: Figure 1(f).

示例： 图 1(f)。

Task: Place the specified objects in the brown box following the order prescribed in the language instruction at each timestep. This task does not require precise placements and mostly evaluates an agent’s ability to ground semantic object descriptions. All objects in a scene are unique without any duplicates. The poses of the objects and the box are randomized for each scene.

任务： 按每时刻语言指令规定的顺序，把指定物体放进棕色箱子。该任务不要求精确放置，主要评测智能体锚定语义物体描述的能力。场景中物体均唯一、无重复；物体与箱子位姿每场景随机。

Objects: packing-seen-google-objects-seq is trained and evaluated on all 56 objects in Figure 5. packing-unseen-google-objects-seq is trained on 37 seen objects but evaluated on 19 unseen objects in Figure 5.

物体： packing-seen-google-objects-seq 在图 5 全部 56 个物体上训练与评测；packing-unseen-google-objects-seq 在 37 个 seen 物体上训练、在 19 个 unseen 物体上评测。

Success Metric: Each specified object is within the bounds of the brown box at the correct timestep. The final score is the total volume of the correct objects placed inside the box at the correct timestep, divided by the total volume of the relevant objects.

成功度量： 每个指定物体在正确时刻位于棕色箱子边界内。最终分数为正确时刻放入箱内的正确物体总体积，除以相关物体总体积。

A.7 Packing Google Objects Group

Example: Figure 1(g).

示例： 图 1(g)。

Task: Place all objects of the specified category in the brown box. This task does not require precise placements or following a specific action sequence. Each scene contains objects of multiple categories with each category containing at least 2 duplicates. The task cannot be solved by counting the number of objects since there are distractor objects, each with 2 or more duplicates.

任务： 把指定类别的全部物体放进棕色箱子。不要求精确放置，也不要求遵循特定动作序列。每场景含多个类别，每类至少 2 个重复。由于存在各带 2 个及以上重复的干扰物，不能靠数物体个数求解。

Objects: packing-seen-google-objects-group is trained and evaluated on all 56 objects in Figure 5. packing-unseen-google-objects-group is trained on 37 seen objects but evaluated on 19 unseen objects in Figure 5.

物体： packing-seen-google-objects-group 在图 5 全部 56 个物体上训练与评测；packing-unseen-google-objects-group 在 37 个 seen 物体上训练、在 19 个 unseen 物体上评测。

Success Metric: All specified objects of a category are within the bounds of the brown box. The final score is the total volume of the correct objects in the box, divided by the total volume of the relevant objects of the specified category in the scene.

成功度量： 指定类别的全部物体位于棕色箱子边界内。最终分数为箱内正确物体总体积，除以场景中该指定类别相关物体总体积。

A.8 Stack Block Pyramid

Example: Figure 1(h).

示例： 图 1(h)。

Task: Build a pyramid of colored blocks in a color sequence specified through the step-by-step language instructions. Each task contains 6 blocks with randomized colors and 1 rectangular base, all initially placed at random poses.

任务： 按逐步语言指令给出的颜色序列搭建彩色方块金字塔。每个任务含 6 个颜色随机的方块与 1 个矩形底座，初始位姿均随机。

Objects: 6 blocks and 1 rectangular base. stack-block-pyramid-seq-seen-colors is trained and evaluated on seen color blocks from Figure 5. stack-block-pyramid-seq-unseen-colors is trained on seen color blocks but evaluated on unseen color blocks from Figure 5.

物体： 6 个方块与 1 个矩形底座。stack-block-pyramid-seq-seen-colors 在图 5 的 seen 颜色方块上训练与评测；stack-block-pyramid-seq-unseen-colors 在 seen 颜色方块上训练、在图 5 的 unseen 颜色方块上评测。

Success Metric: The pose of each block at the corresponding timestep matches the specified location. The final score is the total number of blocks in the correct pose at the correct timestep, divided by the total number of blocks (always 6).

成功度量： 对应时刻每个方块的位姿与指定位置一致。最终分数为正确时刻处于正确位姿的方块数，除以方块总数（恒为 6）。

A.9 Separating Piles

Example: Figure 1(i).

示例： 图 1(i)。

Task: Sweep the pile of blocks into the specified zone. Each scene contains two square zones: one relevant to the task, another as a distractor. The pile and zones are placed at random poses on the table.

任务： 把一堆方块扫进指定区域。每场景有两个方形区域：一个与任务相关，另一个为干扰。堆与区域在桌面上的位姿随机。

Objects: A pile of colored blocks and two squares. separating-piles-seen-colors is trained and evaluated on seen colors from Figure 5 for all blocks and squares. separating-piles-unseen-colors is trained on seen colors but evaluated on unseen colors from Figure 5 for all blocks and squares.

物体： 一堆彩色方块与两个方框。separating-piles-seen-colors 对方块与方框均在图 5 的 seen 颜色上训练与评测；separating-piles-unseen-colors 在 seen 颜色上训练、在图 5 的 unseen 颜色上评测。

Success Metric: All blocks are inside the bounds of the specified zone. The final score is the total number of blocks inside the correct zone, divided by the total number of blocks in the scene.