Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation

蒸馏特征场支撑少样本语言引导操作

William Shen ∗1 , Ge Yang ∗1,2 , Alan Yu 1 , Jansen Wong 1 , Leslie Pack Kaelbling 1 , Phillip Isola 1 1 MIT CSAIL, 2 Institute for Artificial Intelligence and Fundamental Interactions

Abstract

摘要

Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manipulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models. We present a few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using features distilled from a vision-language model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize to unseen expressions and novel categories of objects. Project website: https://f3rm.csail.mit.edu

自监督与语言监督的图像模型蕴含对世界的丰富知识，对泛化很重要。但许多机器人任务需要细致理解三维几何，而二维图像特征往往欠缺这一点。本文借蒸馏特征场把精确三维几何与二维基础模型的丰富语义结合起来，弥合机器人操作中的二维到三维鸿沟。我们提出面向六自由度抓取与放置的少样本学习方法，利用这些强空间与语义先验，在野外泛化到未见物体。用从视觉–语言模型 CLIP 蒸馏的特征，我们给出用自由文本自然语言指定待操作新物体的方式，并展示其能泛化到未见表述与新物体类别。项目主页：https://f3rm.csail.mit.edu

1 Introduction

1 引言

What form of scene representation would facilitate open-set generalization for robotic manipulation systems? Consider a warehouse robot trying to fulfill an order by picking up an item from cluttered storage bins filled with other objects. The robot is given a product manifest, which contains the text description it needs to identify the correct item. In scenarios like this, geometry plays an equally important role as semantics, as the robot needs to comprehend which parts of the object geometry afford a stable grasp. Undertaking such tasks in unpredictable environments — where items from a diverse set can deviate markedly from the training data, and can be hidden or jumbled amidst clutter — underscores the critical need for robust priors in both spatial and semantic understanding.

怎样的场景表示才能支撑机器人操作系统的开放集泛化？设想仓库机器人要根据订单从堆满其他物品的杂乱货箱里取出某件货。机器人拿到产品清单，其中的文字描述用于识别正确物品。这类场景里，几何与语义同等重要：机器人需要理解物体几何的哪些部位能提供稳定抓取。在不可预测环境中完成此类任务——物品来自多样集合、可与训练数据明显偏离，且可能被遮挡或混在杂乱中——凸显了空间与语义理解上都需要稳健先验。

In this paper, we study few-shot and language-guided manipulation, where a robot is expected to pick up novel objects given a few grasping demonstrations or text descriptions without having previously seen a similar item. Toward this goal, we build our system around pre-trained image embeddings, which have emerged as a reliable way to learn commonsense priors from internet-scale data [1, 2, 3].

本文研究少样本与语言引导操作：机器人仅凭少量抓取示范或文本描述抓取新物体，而此前未见过相似物品。为此，系统围绕预训练图像嵌入构建——这类嵌入已成为从互联网规模数据学习常识先验的可靠途径 [1, 2, 3]。

Figure 1 illustrates how our system works. The robot first scans a tabletop scene by taking a sequence of photos using an RGB camera mounted on a selfie stick (Figure 1, left). These photos are used to construct a neural radiance field (NeRF) of the tabletop, which, crucially, is trained to render not just RGB colors but also image features from a pre-trained vision foundation model [4, 1]. This produces a scene representation, called a Distilled Feature Field (DFF), that embeds knowledge from 2D feature maps into a 3D volume (Figure 1, middle). The robot then references demonstrations and language instructions to grasp objects specified by a user (Figure 1, right).

图 1 示意系统如何工作。机器人先用自拍杆上的 RGB 相机连续拍照扫描桌面场景（图 1 左）。这些照片用于构建桌面的神经辐射场（NeRF）；关键在于，训练时不仅渲染 RGB 颜色，还渲染来自预训练视觉基础模型 [4, 1] 的图像特征。由此得到蒸馏特征场（Distilled Feature Field, DFF），把二维特征图中的知识嵌入三维体积（图 1 中）。机器人再参照示范与语言指令，抓取用户指定的物体（图 1 右）。

Distilled Feature Fields (DFFs) were introduced in computer graphics for tasks such as decomposing and editing images [5, 6]. The main contribution of this work is to study the use of DFFs instead for robotic manipulation. We evaluate the robot’s ability to generalize using features sourced from self-supervised vision transformers (DINO ViT, see [4]). These features have been shown to be effective out-of-the-box visual descriptors for dense correspondence [7]. We also source features from a vision-language model, CLIP [1], which is a strong zero-shot learner on various vision and visual question-answering tasks.

蒸馏特征场（DFF）最初在计算机图形学中用于分解与编辑图像等任务 [5, 6]。本文主要贡献是研究把 DFF 用于机器人操作。我们评估机器人用自监督视觉 Transformer（DINO ViT，见 [4]）特征泛化的能力；这些特征已被证明是开箱即用的稠密对应视觉描述子 [7]。我们也使用视觉–语言模型 CLIP [1] 的特征——CLIP 在多种视觉与视觉问答任务上是强零样本学习者。

One challenge that makes distilled feature fields unwieldy for robotics is the long time it takes to model each scene. To address this, we build upon the latest NeRF techniques, and employ hierarchical hashgrids to significantly reduce the modeling time [8, 9, 10]. When it comes to vision-language features, CLIP is trained to produce image-level features, whereas 3D feature distillation requires dense 2D descriptors. Our solution is to use the MaskCLIP [11] reparameterization trick, which extracts dense patch-level features from CLIP while preserving alignment with the language stream.

使蒸馏特征场难以用于机器人的一个挑战是为每个场景建模耗时过长。为此我们基于最新 NeRF 技术，用层次化哈希网格显著缩短建模时间 [8, 9, 10]。至于视觉–语言特征，CLIP 训练产出的是图像级特征，而三维特征蒸馏需要稠密二维描述子。我们的做法是用 MaskCLIP [11] 重参数化技巧，从 CLIP 提取稠密 patch 级特征，同时保持与语言流的对齐。

We demonstrate that Distilled Feature Fields enable open-ended scene understanding and can be leveraged by robots for 6-DOF object manipulation. We call this approach Feature Fields for Robotic Manipulation (F3RM). We present few-shot learning experiments on grasping and placing tasks, where our robot is able to handle open-set generalization to objects that differ significantly in shape, appearance, materials, and poses. We also present language-guided manipulation experiments where our robot grasps or places objects in response to free-text natural language commands. By taking advantage of the rich visual and language priors within 2D foundation models, our robot generalizes to new categories of objects that were not seen among the four categories used in the demonstrations.

我们表明蒸馏特征场能支撑开放式场景理解，并被机器人用于六自由度物体操作。该方法称为 Feature Fields for Robotic Manipulation（F3RM）。我们在抓取与放置任务上做少样本实验：机器人能处理形状、外观、材质与位姿显著不同的开放集物体。我们也做语言引导操作实验：机器人按自由文本自然语言指令抓取或放置物体。借助二维基础模型中的丰富视觉与语言先验，机器人能泛化到示范所用四类之外的新物体类别。

Figure 1: Distilled Feature Fields Enable Open-Ended Manipulation. (1) Robot uses a selfie stick to scan RGB images of the scene (camera frustums shown). (2) Extract patch-level dense features for the images from a 2D foundation model, and distill them into a feature field (PCA shown) along with modeling a NeRF. (3) We can query CLIP feature fields with language to generate heatmaps and infer 6-DOF grasps on novel objects given only ten demonstrations.

图 1：蒸馏特征场支撑开放式操作。（1）机器人用自拍杆扫描场景 RGB 图像（示出相机视锥）。（2）从二维基础模型提取图像的 patch 级稠密特征，并与 NeRF 一并蒸馏进特征场（示 PCA）。（3）可用语言查询 CLIP 特征场生成热图，并在仅有十次示范时推断新物体上的六自由度抓取。

2 Problem Formulation

2 问题形式化

We consider the class of manipulation problems that can be parameterized via a single rigid-body transformation $\mathbf{T}\in SE(3)$, and focus on grasping and placing tasks. We parameterize a 6-DOF grasp or place pose as $\mathbf{T}=(\mathbf{R},\mathbf{t})$ in the world frame (see Figure 2), where $\mathbf{R}$ is the rotation matrix, and $\mathbf{t}$ is the translation vector. In each scene, the robot is given a set of RGB images $\{\mathbf{I}\}$ with their corresponding camera poses.

我们考虑可用单个刚体变换 $\mathbf{T}\in SE(3)$ 参数化的操作问题类，并聚焦抓取与放置。把六自由度抓取或放置位姿参数化为世界系中的 $\mathbf{T}=(\mathbf{R},\mathbf{t})$（见图 2），其中 $\mathbf{R}$ 为旋转矩阵，$\mathbf{t}$ 为平移向量。每个场景中，机器人获得一组带对应相机位姿的 RGB 图像 $\{\mathbf{I}\}$。

Few-Shot Manipulation.

We aim to build robots that can manipulate objects given only a few demonstrations of a task, such as grasping a mug by its handle. During learning, each demonstration $D$ consists of the tuple $\langle\{\mathbf{I}\},\mathbf{T}^{*}\rangle,$ where $\{\mathbf{I}\}_{i=1}^{N}$ are $N$ RGB camera views of the scene and $\mathbf{T}^{*}$ is a pose that accomplishes the desired task. During testing, the robot is given multiple images $\{\mathbf{I}^{\prime}\}$ of a new scene which may contain distractor objects and clutter. The robot’s goal is to predict a pose $\mathbf{T}$ that achieves the task. We want to test for open-ended generalization: the new scene contains related but previously unseen objects that differ from the demo objects in shape, size, pose, and material.

少样本操作。

目标是构建仅凭少量任务示范（如抓住杯子手柄）就能操作物体的机器人。学习时，每次示范 $D$ 为元组 $\langle\{\mathbf{I}\},\mathbf{T}^{*}\rangle$，其中 $\{\mathbf{I}\}_{i=1}^{N}$ 为场景的 $N$ 个 RGB 相机视角，$\mathbf{T}^{*}$ 为完成期望任务的位姿。测试时，机器人获得新场景的多张图像 $\{\mathbf{I}^{\prime}\}$，其中可能含干扰物与杂乱；目标是预测完成任务的位姿 $\mathbf{T}$。我们希望检验开放式泛化：新场景含相关但此前未见、在形状、尺寸、位姿与材质上不同于示范物体的物体。

Open-Text Language-Guided Manipulation.

We extend few-shot manipulation to include open-text conditioning via natural language. Given a text description of an object, the robot’s objective is to grasp the objects that match this description. The robot has access to a few demonstrations for multiple object categories. During testing, the user provides the robot with a text query $L^{+}$ to specify which object to manipulate and negative texts $L^{-}$ to reject distractors. In practice, $L^{-}$ can be sampled automatically. The object category we care about at test time does not necessarily appear in the demonstrations. We explicitly seek open-ended generalization to new object categories, given just a few demonstrations limited to a small class of object categories.

开放文本语言引导操作。

我们将少样本操作扩展为用自然语言做开放文本条件。给定物体的文本描述，机器人目标是抓取匹配该描述的物体。机器人可获得多物体类别的少量示范。测试时，用户给出文本查询 $L^{+}$ 指定操作哪个物体，以及负文本 $L^{-}$ 以排除干扰物；实践中 $L^{-}$ 可自动采样。测试时关心的物体类别不必出现在示范中。我们明确追求对新物体类别的开放式泛化，而示范仅限少数类别。

3 Feature Fields for Robotic Manipulation (F3RM)

3 面向机器人操作的特征场（F3RM）

We present Feature Fields for Robotic Manipulation (F3RM), our approach for distilling pre-trained representations from vision and vision-language models into 3D feature fields for open-ended robotic manipulation. Doing so involves solving three separate problems. First, how to produce the feature field of a scene automatically at a reasonable speed; second, how to represent and infer 6-DOF grasping and placing poses; and finally, how to incorporate language guidance to enable open-text commands. We include a formal summary of NeRFs [12] in Appendix A.1.

我们提出 Feature Fields for Robotic Manipulation（F3RM）：把视觉与视觉–语言模型的预训练表示蒸馏进三维特征场，以支撑开放式机器人操作。这需要分别解决三件事：如何以合理速度自动生成场景特征场；如何表示并推断六自由度抓取与放置位姿；以及如何引入语言引导以支持开放文本指令。附录 A.1 给出 NeRF [12] 的形式化摘要。

3.1 Feature Field Distillation

3.1 特征场蒸馏

Distilled Feature Fields (DFFs) [5, 6] extend NeRFs by including an additional output to reconstruct dense 2D features from a vision model $\mathbf{f}_{\text{vis}}$. The feature field $\mathbf{f}$ is a function that maps a 3D position $\mathbf{x}$ to a feature vector $\mathbf{f}(\mathbf{x})$. We assume that $\mathbf{f}$ does not depend on the viewing direction $\mathbf{d}$.[^1] Supervision of this feature field $\mathbf{f}$ is provided through the rendered 2D feature maps, where each feature vector is given by the feature rendering integral between the near and far plane ($t_{n}$ and $t_{f}$):

蒸馏特征场（DFF） [5, 6] 在 NeRF 之外增加输出，以重建视觉模型 $\mathbf{f}_{\text{vis}}$ 的稠密二维特征。特征场 $\mathbf{f}$ 是把三维位置 $\mathbf{x}$ 映射到特征向量 $\mathbf{f}(\mathbf{x})$ 的函数。我们假定 $\mathbf{f}$ 不依赖于观察方向 $\mathbf{d}$。[^1] 对该特征场 $\mathbf{f}$ 的监督来自渲染出的二维特征图，每个特征向量由近远平面（$t_{n}$ 与 $t_{f}$）之间的特征渲染积分给出：

$$\mathbf{F}(\mathbf{r})=\int_{t_{n}}^{t_{f}}T(t)\sigma(\mathbf{r}_{t})\mathbf{f}(\mathbf{r}_{t})\,\mathrm{d}t\quad\text{with }\;T(t)\;=\exp\left({-\int_{t_{n}}^{t}\sigma(\mathbf{r}_{s})\,\mathrm{d}s}\right), \tag{1}$$

where $\mathbf{r}$ is the camera ray corresponding to a particular pixel, $t$ is the distance along the ray, $\sigma$ is the density field from NeRF, and $T(t)$ represents the accumulated transmission from the near plane $t_{n}$ to $t$. Observe this is similar to the volume rendering integral (Eq. 5).

其中 $\mathbf{r}$ 为对应某像素的相机射线，$t$ 为沿射线的距离，$\sigma$ 为 NeRF 的密度场，$T(t)$ 表示从近平面 $t_{n}$ 到 $t$ 的累积透射。可见这与体积渲染积分（式 5）类似。

Feature Distillation.

We begin with a set of $N$ 2D feature maps $\{\mathbf{I}^{f}_{i}\}_{i=1}^{N}$, where $\mathbf{I}^{f}=\mathbf{f}_{\text{vis}}(\mathbf{I})$ for each RGB image $\mathbf{I}$. We optimize $\mathbf{f}$ by minimizing the quadratic loss $\mathcal{L}_{\text{feat}}=\sum_{\mathbf{r}\in\mathcal{R}}\big\|\mathbf{\hat{F}}(\mathbf{r})-\mathbf{I}^{f}(\mathbf{r})\big\|_{2}^{2},$ where $\mathbf{I}^{f}(\mathbf{r})$ is the target feature vector from $\mathbf{I}^{f}$, and $\mathbf{\hat{F}}(\mathbf{r})$ is estimated by a discrete approximation of the feature rendering integral in Eq. 1.

特征蒸馏。

我们从一组 $N$ 张二维特征图 $\{\mathbf{I}^{f}_{i}\}_{i=1}^{N}$ 出发，其中对每张 RGB 图像 $\mathbf{I}$ 有 $\mathbf{I}^{f}=\mathbf{f}_{\text{vis}}(\mathbf{I})$。通过最小化二次损失 $\mathcal{L}_{\text{feat}}=\sum_{\mathbf{r}\in\mathcal{R}}\big\|\mathbf{\hat{F}}(\mathbf{r})-\mathbf{I}^{f}(\mathbf{r})\big\|_{2}^{2}$ 优化 $\mathbf{f}$，其中 $\mathbf{I}^{f}(\mathbf{r})$ 为 $\mathbf{I}^{f}$ 中的目标特征向量，$\mathbf{\hat{F}}(\mathbf{r})$ 由式 1 特征渲染积分的离散近似估计。

Extracting Dense Visual Features from CLIP.

The common practice for extracting dense features from ViTs is to use the key/value embeddings from the last layer before pooling [4]. While this approach has yielded great results for vision-only models on image segmentation and image-to-image correspondence tasks [7, 15], it removes the transformations needed to project the visual features into the shared feature space with the language stream in vision-language models such as CLIP [1, 16]. Re-aligning the dense features typically requires additional training, which negatively affects the model’s open-text generalization.

从 CLIP 提取稠密视觉特征。

从 ViT 提取稠密特征的常见做法是用池化前最后一层的 key/value 嵌入 [4]。这对仅视觉模型在图像分割与图像–图像对应任务上效果很好 [7, 15]，但去掉了把视觉特征投影到与语言流共享特征空间所需的变换，而 CLIP 等视觉–语言模型需要该变换 [1, 16]。重新对齐稠密特征通常需要额外训练，会损害模型的开放文本泛化。

Rather than using CLIP as an image-level feature (Alg. 1, App. A.2), we extract dense features from CLIP using the MaskCLIP reparameterization trick (Alg. 2) [11]. These features retain a sufficient alignment with the language embedding to support zero-shot language guidance in our experiments (see Fig.7). We present pseudocode for this technique in Appendix A.2. A second modification we apply is to interpolate the position encoding (see [4]) to accommodate larger images with arbitrary aspect ratios. This is needed because CLIP uses a small, fixed number of input patches from a square crop. These two techniques combined enable us to extract dense, high-resolution patch-level 2D features from RGB images at about $25$ frames per second and does not require fine-tuning CLIP.

我们不用 CLIP 作图像级特征（Alg. 1，附录 A.2），而用 MaskCLIP 重参数化技巧（Alg. 2） [11] 从 CLIP 提取稠密特征。这些特征与语言嵌入保持足够对齐，足以支撑实验中的零样本语言引导（见图 7）。该技巧的伪代码见附录 A.2。第二项修改是插值位置编码（见 [4]），以适配任意宽高比的更大图像——因为 CLIP 只使用来自方形裁剪的少量固定输入 patch。二者结合使我们能以约每秒 $25$ 帧从 RGB 图像提取稠密高分辨率 patch 级二维特征，且无需微调 CLIP。

Figure 2: Representing 6-DOF Poses. (a) Recording the gripper pose $\mathbf{T}^{*}$ in virtual reality (VR) on an example mug. (b) We approximate the continuous local field via a fixed set of query points in the gripper’s canonical frame. (c) We concatenate feature vectors at these query points, then average over $n$ (we use $n=2$) demonstrations. This gives a task embedding $\mathbf{Z}_{M}$ for the task $M$.

图 2：用特征场表示六自由度位姿。（a）在虚拟现实（VR）中于示例杯子上记录夹爪位姿 $\mathbf{T}^{*}$。（b）用夹爪规范系中固定一组查询点近似连续局部场。（c）拼接这些查询点处的特征向量，再对 $n$ 次示范求平均（我们取 $n=2$），得到任务 $M$ 的任务嵌入 $\mathbf{Z}_{M}$。

3.2 Representing 6-DOF Poses with Feature Fields

3.2 用特征场表示六自由度位姿

We wish to represent the pose of the gripper in a demonstration by the local 3D feature field in the gripper’s coordinate frame. We approximate this local context via a discrete set of query points and the feature vectors measured at each point. We sample a fixed set of $N_{q}$ query points $\mathcal{X}=\{\mathbf{x}\in\mathbb{R}^{3}\}_{N_{q}}$ in the canonical gripper frame for each task $M$ from a 3D Gaussian. We adjust its mean and variance manually to cover parts of the object we intend to target, as well as important context cues (e.g., body of the mug when grasping the handle) and free-space (Fig.2b). For a 6-DOF gripper pose $\mathbf{T}$, we sample the feature field $\mathbf{f}$ at each point in the query point cloud, transformed by $\mathbf{T}$ (Fig.2b).

我们希望用夹爪坐标系中的局部三维特征场表示示范中的夹爪位姿。用一组离散查询点及其处测得的特征向量近似该局部上下文。对每个任务 $M$，从三维高斯在规范夹爪系中采样固定的 $N_{q}$ 个查询点 $\mathcal{X}=\{\mathbf{x}\in\mathbb{R}^{3}\}_{N_{q}}$。手动调整均值与方差，以覆盖拟针对的物体部位，以及重要上下文线索（如抓手柄时的杯身）与自由空间（图 2b）。对六自由度夹爪位姿 $\mathbf{T}$，在经 $\mathbf{T}$ 变换后的查询点云各点处采样特征场 $\mathbf{f}$（图 2b）。

To account for the occupancy given by the local geometry, we weigh the features by their corresponding alpha values from the density field $\sigma$ of the NeRF model, integrated over the voxel. At a point $\mathbf{x}$ in the world frame, this produces the $\alpha$-weighted features

为计入局部几何给出的占据，用 NeRF 模型密度场 $\sigma$ 在体素上积分得到的对应 alpha 值对特征加权。在世界系点 $\mathbf{x}$ 处，得到 $\alpha$加权特征

$$\mathbf{f}_{\alpha}(\mathbf{x})=\alpha(\mathbf{x})\cdot\mathbf{f}(\mathbf{x})\text{,\;where\;}\alpha(\mathbf{x})=1-\exp(-\sigma(\mathbf{x})\cdot\delta)\in(0,1), \tag{2}$$

and $\delta$ is the distance between adjacent samples. We sample a set of features $\{\mathbf{f}_{\alpha}(\mathbf{x})\mid\mathbf{x}\in\mathbf{T}\mathcal{X}\}$ using the transformed query points $\mathbf{T}\mathcal{X}$, and concatenate along the feature-dimension into a vector, $\mathbf{z}_{\mathbf{T}}\in\mathbb{R}^{N_{q}\cdot|\mathbf{f}|}$. The query points $\mathcal{X}$ and demo embedding $\mathbf{z}_{\mathbf{T}}$ thus jointly encode the demo pose $\mathbf{T}$.

其中 $\delta$ 为相邻采样间距。用变换后的查询点 $\mathbf{T}\mathcal{X}$ 采样特征集 $\{\mathbf{f}_{\alpha}(\mathbf{x})\mid\mathbf{x}\in\mathbf{T}\mathcal{X}\}$，沿特征维拼接为向量 $\mathbf{z}_{\mathbf{T}}\in\mathbb{R}^{N_{q}\cdot|\mathbf{f}|}$。查询点 $\mathcal{X}$ 与示范嵌入 $\mathbf{z}_{\mathbf{T}}$ 因而共同编码示范位姿 $\mathbf{T}$。

We specify each manipulation task $M$ by a set of demonstrations $\{D\}$. We average $\mathbf{z}_{\mathbf{T}}$ over the demos for the same task to obtain a task embedding $\mathbf{Z}_{M}\in\mathbb{R}^{N_{q}\cdot|\mathbf{f}|}$ (Fig. 2c). This allows us to reject spurious features and focus on relevant parts of the feature space. This representation scheme is similar to the one used in Neural Descriptor Fields [17]. The main distinction is that NDF is trained from scratch on object point clouds, whereas our feature field is sourced from 2D foundation models that are trained over internet-scale datasets. The capabilities that emerge at this scale hold the potential for open-ended generalization beyond the few examples that appear in the demonstrations.

每个操作任务 $M$ 由一组示范 $\{D\}$ 指定。对同一任务的示范上 $\mathbf{z}_{\mathbf{T}}$ 求平均，得到任务嵌入 $\mathbf{Z}_{M}\in\mathbb{R}^{N_{q}\cdot|\mathbf{f}|}$（图 2c）。这样可抑制虚假特征、聚焦特征空间中的相关部分。该表示方案类似 Neural Descriptor Fields [17]。主要区别是：NDF 在物体点云上从头训练，而我们的特征场来自在互联网规模数据上训练的二维基础模型。这一规模涌现的能力有望支撑超越示范中少数例子的开放式泛化。

Inferring 6-DOF Poses.

Our inference procedure involves a coarse pre-filtering step for the translational DOFs, and an optimization-based fine-tuning step for the rotational DOFs. First, we sample a dense voxel grid over the workspace, where each voxel $\mathbf{v}$ has a grid-size $\delta$. We remove free space by rejecting voxels with alphas $\mathbf{\alpha}(\mathbf{v})<\epsilon_{\text{free}}$. We then remove voxels that are irrelevant to the task, using the cosine similarity between the voxel feature $\mathbf{f}_{\alpha}(\mathbf{v})$ and the task embedding $\mathbf{Z}_{M}$. To get the complete 6-DOF poses $\mathcal{T}=\{\mathbf{T}\}$, we uniformly sample $N_{r}$ rotations for each remaining voxel $\mathbf{v}$.

推断六自由度位姿。

推断流程包括对平移自由度的粗过滤，以及对旋转自由度的基于优化的细调。首先在工作空间上采样稠密体素网格，每个体素 $\mathbf{v}$ 边长为 $\delta$。用 $\mathbf{\alpha}(\mathbf{v})<\epsilon_{\text{free}}$ 剔除自由空间。再用体素特征 $\mathbf{f}_{\alpha}(\mathbf{v})$ 与任务嵌入 $\mathbf{Z}_{M}$ 的余弦相似度剔除与任务无关的体素。对每个剩余体素 $\mathbf{v}$ 均匀采样 $N_{r}$ 个旋转，得到完整六自由度位姿集 $\mathcal{T}=\{\mathbf{T}\}$。

Pose Optimization.

We optimize the initial poses with the following cost function

位姿优化。

用如下代价函数优化初始位姿

$$\mathcal{J}_{\textnormal{pose}}(\mathbf{T})=-\cos(\mathbf{z}_{\mathbf{T}},\mathbf{Z}_{M}) \tag{3}$$

using the Adam optimizer [18] to search for poses that have the highest similarity to the task embedding $\mathbf{Z}_{M}$. After each optimization step, we prune poses that have the highest costs. We also reject poses that are in collision by thresholding the number of overlapping voxels between a voxelized gripper model and the scene geometry. This leaves us with a ranked list of poses that we feed into a motion planner in PyBullet [19, 20]. We execute the highest-ranked grasp or place pose that has a valid motion plan. Observe that our scheme operates over the entire feature field, and does not rely on assumptions about objectness such as segmentation masks or object poses.

用 Adam 优化器 [18] 搜索与任务嵌入 $\mathbf{Z}_{M}$ 相似度最高的位姿。每步优化后剪枝代价最高的位姿。还通过对体素化夹爪模型与场景几何的重叠体素数设阈值，拒绝碰撞位姿。最终得到排序后的位姿列表，送入 PyBullet 中的运动规划器 [19, 20]。执行有有效运动规划且排名最高的抓取或放置位姿。注意该方案在整个特征场上操作，不依赖分割掩码或物体位姿等「物体性」假设。

3.3 Open-Text Language-Guided Manipulation

3.3 开放文本语言引导操作

[figure]skip=4pt

(a) Retrieving Demonstrations

[figure]skip=4pt

(a) 检索示范

Natural language offers a way to extend robotic manipulation to an open-set of objects, serving as an attractive alternative when photos of the target object are inaccurate or unavailable. In our language-guided few-shot manipulation pipeline, the learning procedure and the representation for the demonstrations remain consistent with Section 3.2. At test time, the robot receives open-text language queries from the user that specify the object of interest to manipulate. Our language-guided pose inference procedure comprises three steps (see Fig.3): (i) retrieving relevant demonstrations, (ii) initializing coarse grasps, and (iii) language-guided grasp pose optimization.

自然语言可把机器人操作扩展到开放物体集合，在目标物体照片不准或不可用时是有吸引力的替代。语言引导的少样本操作管线中，学习过程与示范表示与第 3.2 节一致。测试时，机器人收到用户指定待操作物体的开放文本查询。语言引导位姿推断含三步（见图 3）：（i）检索相关示范；（ii）初始化粗抓取；（iii）语言引导的抓取位姿优化。

Retrieving Relevant Demonstrations.

We select the two demonstrations whose average feature $\mathbf{F}_{d}$ (averaged among the query points of each demo pose $\mathbf{T}^{*}$) is closest to the text embedding $\mathbf{q}=\text{emb}_{\text{CLIP}}(L^{+})$ (Fig.3(a)). We found that using the positive query text ($L^{+}$) alone is sufficient. This means finding the demonstration that maximizes the cosine similarity $\cos(\mathbf{q},\mathbf{F}_{d})$. Note that the objects used in the demonstrations do not have to come from the same category as the target object. For instance, asking the robot to pick up the “measuring beaker” or “bowl” leads to the robot choosing the demonstration of picking up a mug by its lip (Fig.5).

检索相关示范。

选取平均特征 $\mathbf{F}_{d}$（对各示范位姿 $\mathbf{T}^{*}$ 的查询点求平均）与文本嵌入 $\mathbf{q}=\text{emb}_{\text{CLIP}}(L^{+})$ 最近的两次示范（图 3(a)）。我们发现仅用正查询文本（$L^{+}$）即足够，即找最大化余弦相似度 $\cos(\mathbf{q},\mathbf{F}_{d})$ 的示范。注意示范物体不必与目标物体同类。例如，要求机器人拿起「量杯」或「碗」时，会选用按杯沿拿起杯子的示范（图 5）。

Initializing Grasp Proposals.

We speed up grasp pose inference by first running a coarse proposal step where we filter out regions in the feature field that are irrelevant to the text query. We start by sampling a dense voxel grid among the occupied regions by masking out free space (see Sec.3.2). Afterward, we prune down the number of voxels by keeping those more similar to the positive query $L^{+}$ than any one of the negative queries $L^{-}$. Formally, let $\mathbf{q}_{i}^{-}=\text{emb}_{\text{CLIP}}(L_{i}^{-})\mid i\in\{1,\dots,n\}$ be the text embeddings of the negative queries. We compute the softmax over the pair-wise cosine similarity between the voxel’s feature $\mathbf{f}_{\alpha}(\mathbf{v})$ and the ensemble $[\mathbf{q},\mathbf{q}^{-}_{1},\mathbf{q}^{-}_{2},\dots,\mathbf{q}^{-}_{n}]$, and identify the closest negative query $\mathbf{q}^{-}$. We remove voxels that are closer to $\mathbf{q}^{-}$ than the positive query $\mathbf{q}$. The cosine similarity between the voxel embedding and $[\mathbf{q},\mathbf{q}^{-}]$ pair forms a binomial distribution that allows us to reject voxels that have $\geq 50\%$ probability of being associated with the negative query. Finally, to get the set of initial poses $\mathcal{T}=\{\mathbf{T}\}$, we sample $N_{r}$ rotations for each remaining voxel.

初始化抓取提案。

为加速抓取位姿推断，先做粗提案：过滤特征场中与文本查询无关的区域。先在占据区域采样稠密体素网格并掩掉自由空间（见第 3.2 节）。再保留相对任一负查询 $L^{-}$ 更相似于正查询 $L^{+}$ 的体素。形式化地，令 $\mathbf{q}_{i}^{-}=\text{emb}_{\text{CLIP}}(L_{i}^{-})\mid i\in\{1,\dots,n\}$ 为负查询文本嵌入。对体素特征 $\mathbf{f}_{\alpha}(\mathbf{v})$ 与集成 $[\mathbf{q},\mathbf{q}^{-}_{1},\mathbf{q}^{-}_{2},\dots,\mathbf{q}^{-}_{n}]$ 的成对余弦相似度做 softmax，找出最近负查询 $\mathbf{q}^{-}$。剔除比正查询 $\mathbf{q}$ 更接近 $\mathbf{q}^{-}$ 的体素。体素嵌入与 $[\mathbf{q},\mathbf{q}^{-}]$ 对的余弦相似度形成二项分布，可拒绝与负查询关联概率 $\geq 50\%$ 的体素。最后对每个剩余体素采样 $N_{r}$ 个旋转，得到初始位姿集 $\mathcal{T}=\{\mathbf{T}\}$。

Language-Guided Grasp Pose Optimization.

To incorporate language guidance, we first compute $\mathcal{J}_{\text{pose}}$ from Eq.3 using the two demonstrations retrieved in the first step. We then assign a lower cost to regions that are more similar to the language query $\mathbf{q}$ by computing a language-guidance weight $C_{\mathbf{q}}=\underset{\mathbf{x}\in\mathbf{T}\mathcal{X}}{\text{mean}}[\mathbf{q}\otimes\mathbf{f}_{\alpha}(\mathbf{x})]$, and multiply it with $\mathcal{J}_{\textnormal{pose}}$ (Fig.3(b))

语言引导的抓取位姿优化。

为引入语言引导，先用第一步检索到的两次示范按式 3 计算 $\mathcal{J}_{\text{pose}}$。再通过对更相似于语言查询 $\mathbf{q}$ 的区域赋更低代价，计算语言引导权重 $C_{\mathbf{q}}=\underset{\mathbf{x}\in\mathbf{T}\mathcal{X}}{\text{mean}}[\mathbf{q}\otimes\mathbf{f}_{\alpha}(\mathbf{x})]$，并与 $\mathcal{J}_{\textnormal{pose}}$ 相乘（图 3(b)）

$$\mathcal{J}_{\text{lang}}(\mathbf{T})=\underset{\mathbf{x}\in\mathbf{T}\mathcal{X}}{\text{mean}}\Bigl[\mathbf{q}\otimes\mathbf{f}_{\alpha}(\mathbf{x})\Bigr]\cdot\mathcal{J}_{\textnormal{pose}}(\mathbf{T}). \tag{4}$$

The first term, $C_{\mathbf{q}}$, is the normalized cosine similarity between the text embedding $\mathbf{q}$ and the average $\alpha$-weighted query point feature for a pose $\mathbf{T}$. We iteratively update the pose $\mathbf{T}$ via gradient descent while pruning using the procedure from Section 3.2 till convergence.

第一项 $C_{\mathbf{q}}$ 是文本嵌入 $\mathbf{q}$ 与位姿 $\mathbf{T}$ 的平均 $\alpha$ 加权查询点特征之间的归一化余弦相似度。我们用梯度下降迭代更新位姿 $\mathbf{T}$，并按第 3.2 节流程剪枝，直至收敛。

4 Results

4 结果

[figure]skip=4pt

[figure]skip=4pt

(a) Grasp mug (lip)

[figure]skip=4pt

[figure]skip=4pt

(a) 抓杯子（杯沿）

4.1 Learning to Grasp from Demonstrations

4.1 从示范学习抓取

We consider five 6-DOF grasping and placing tasks and provide two demonstrations per task (Fig.5). To label the demonstrations, we load a NeRF-reconstructed point cloud into virtual reality, and use a hand controller to move the gripper to the desired pose (Fig.2a). We compare the performance of three types of distilled features: (1) DINO ViT, (2) CLIP ViT, and (3) CLIP ResNet. We consider three baselines, including (1) using density $\sigma$ from the NeRF, (2) the intermediate NeRF features, and (3) the RGB color value as features, and compare against MIRA [21], a recent work which uses NeRFs to render orthographic viewpoints for pixel-wise affordance predictions. For each task, we evaluate in ten scenes that contain novel objects in arbitrary poses and distractor objects. The novel objects belong to the same or related object category as the demo objects, but differ in shape, size, material and appearance. We reset the scenes to about the same configuration for each compared method. We include the full details on the experimental setup in Appendix A.4.

我们考虑五项六自由度抓取与放置任务，每项给两次示范（图 5）。标注示范时，把 NeRF 重建的点云载入虚拟现实，用手持控制器把夹爪移到期望位姿（图 2a）。比较三类蒸馏特征：（1）DINO ViT，（2）CLIP ViT，（3）CLIP ResNet。基线包括（1）用 NeRF 密度 $\sigma$，（2）NeRF 中间特征，（3）用 RGB 颜色作特征；并对照 MIRA [21]——近期工作用 NeRF 渲染正交视角做逐像素可操作性预测。对每项任务，在十个含任意位姿新物体与干扰物的场景中评测。新物体与示范物体同类或相关，但形状、尺寸、材质与外观不同。各对比方法把场景重置到大致相同配置。实验设置完整细节见附录 A.4。

We present the success rates in Table 1 and examples of robot executions in Figure 5. While the baselines using density, RGB color values, or intermediate features from NeRF achieve respectable performance, they struggle to identify the semantic category of the objects we care about, especially in complex scenes with more distractors. We find that DINO and CLIP feature fields exhibit impressive generalization capabilities and have complementary advantages. The DINO ViT has a good part-level understanding of object geometry with $7/19$ failure cases caused by inaccuracies in the grasp rotations and occasionally, the translations. In comparison, $21/27$ failures for CLIP ViT and ResNet combined may be attributed to this issue. We find that CLIP favors semantic and categorical information over geometric features, which are essential for grasping and placing objects. DINO, on the other hand, struggles with distinguishing target objects from distractor objects that contain similar visual appearance to the objects used in the demonstrations. CLIP struggles less in this regard. The fusion between semantic features and detailed 3D geometry offers a way to model multiple objects piled tightly together: in Figure 6(b), for instance, a caterpillar toy is buried under other toys. Figure 6(c) shows our robot grasping the caterpillar, and dragging it from the bottom of the pile.

成功率见表 1，机器人执行例子见图 5。用密度、RGB 或 NeRF 中间特征的基线虽有可观表现，但难以识别我们关心的物体语义类别，尤其在干扰物更多的复杂场景。我们发现 DINO 与 CLIP 特征场展现出可观的泛化，且优势互补。DINO ViT 对物体几何的部位级理解较好，$7/19$ 次失败由抓取旋转（偶有平移）不准造成。相比之下，CLIP ViT 与 ResNet 合计 $21/27$ 次失败可归因于此。CLIP 更偏语义与类别信息而非几何特征，而抓取与放置需要几何。DINO 则难以把目标物与外观类似示范物的干扰物区分开；CLIP 在这方面更稳。语义特征与细致三维几何的融合，也能处理紧堆在一起的多物体：如图 6(b)，毛毛虫玩具被其他玩具压住；图 6(c) 显示机器人抓住毛毛虫并从堆底拖出。

[figure]skip=4pt

(a) Demonstration (1 of 2)

[figure]skip=4pt

(a) 示范（2 次中的 1 次）

	Mug	Mug	Caterpillar	Screwdriver	Cup	Total
	lip	handle	ear	handle	on rack
MIRA [21]	$1/10$	$2/10$	$6/10$	$3/10$	$3/10$	$15/50$
Density	$5/10$	$5/10$	$\mathbf{10/10}$	$2/10$	$5/10$	$27/50$
Intermediate	$2/10$	$2/10$	$1/10$	$3/10$	$1/10$	$9/50$
RGB	$4/10$	$3/10$	$9/10$	$1/10$	$4/10$	$21/50$
DINO ViT	$5/10$	$4/10$	$8/10$	$6/10$	$\mathbf{8/10}$	$31/50$
CLIP ViT	$7/10$	$\mathbf{7/10}$	$8/10$	$6/10$	$6/10$	$34/50$
CLIP ResNet	$\mathbf{9/10}$	$6/10$	$9/10$	$\mathbf{8/10}$	$7/10$	$\mathbf{39/50}$

Color	7/10
Material	7/10
Relational	4/10
General	4/10
OOD	9/10
Total	31/50

Table 1: Success rates on grasping and placing tasks. We compare the success rates over ten evaluation scenes given two demonstrations for each task. We consider a run successful if the robot grasps or places the correct corresponding object part for the task.

	杯子	杯子	毛毛虫	螺丝刀	杯	合计
	杯沿	手柄	耳朵	手柄	挂架上
MIRA [21]	$1/10$	$2/10$	$6/10$	$3/10$	$3/10$	$15/50$
密度	$5/10$	$5/10$	$\mathbf{10/10}$	$2/10$	$5/10$	$27/50$
中间特征	$2/10$	$2/10$	$1/10$	$3/10$	$1/10$	$9/50$
RGB	$4/10$	$3/10$	$9/10$	$1/10$	$4/10$	$21/50$
DINO ViT	$5/10$	$4/10$	$8/10$	$6/10$	$\mathbf{8/10}$	$31/50$
CLIP ViT	$7/10$	$\mathbf{7/10}$	$8/10$	$6/10$	$6/10$	$34/50$
CLIP ResNet	$\mathbf{9/10}$	$6/10$	$9/10$	$\mathbf{8/10}$	$7/10$	$\mathbf{39/50}$

颜色	7/10
材质	7/10
关系	4/10
一般	4/10
OOD	9/10
合计	31/50

表 1：抓取与放置任务成功率。 每项任务两次示范，在十个评测场景上比较成功率。若机器人抓到或放到任务对应的正确物体部位，则计为成功。

4.2 Language-Guided Object Manipulation

4.2 语言引导物体操作

We set up $13$ table-top scenes to study the feasibility of using open-text language and CLIP feature fields for designating objects to manipulate. We reuse the ten demonstrations from the previous section (Sec. 4.1), which span four object categories (see Fig.5). We include three types of objects in our test scenes: (1) novel objects from the same categories as the demonstrations, (2) out-of-distribution (OOD) objects from new categories that share similar geometry as the demonstrated items (e.g., bowls, measuring beakers, utensils), and (3) distractor items that we desire the system ignore. Success metric: we consider a language query successful if the robot stably grasps the target object and places it in a plastic bin at a known location. We include more details in Appendix A.7.

我们搭建 $13$ 个桌面场景，研究用开放文本语言与 CLIP 特征场指定待操作物体的可行性。复用上一节（第 4.1 节）的十次示范，覆盖四类物体（见图 5）。测试场景含三类物体：（1）与示范同类的新物体；（2）与示范物品几何相近、来自新类别的分布外（OOD）物体（如碗、量杯、餐具）；（3）希望系统忽略的干扰物。成功标准：若机器人稳定抓住目标物并放到已知位置的塑料箱中，则该语言查询计为成功。更多细节见附录 A.7。

We break down the success rates by category in Table 2, and show the robot’s execution sequence for an example scene in Figure 7 (video). This scene contained eleven objects, four were sourced from the YCB object dataset (the apple, black marker, mango, and a can of SPAM), the rest collected from the lab and bought online. We present five successful grasps (Figure 7). The robot failed to grasp the stainless steel jug by its handle due to a small error in the grasp rotation. This is a typical failure case — six out of $19$ failures stem from these poor grasp predictions with rotational or translational errors. The remaining $13/19$ failed grasps are due to CLIP features behaving like a bag-of-words and struggling to capture relationships, attributes, and ordinal information within sentences [22]. For instance, in queries such as “black screwdriver” and “mug on a can of spam,” CLIP paid more attention to other black objects and the can of spam, respectively. We found that retrieving demos via text occasionally (particularly in the rack scene) benefits from prompt engineering.

按类别拆开的成功率见表 2，示例场景中机器人执行序列见图 7（视频）。该场景含十一个物体，四个来自 YCB（苹果、黑色马克笔、芒果、一罐 SPAM），其余来自实验室与网购。图 7 展示五次成功抓取。机器人未能按手柄抓住不锈钢壶，因抓取旋转有小误差——这是典型失败：$19$ 次失败中六次源于旋转或平移不佳的抓取预测。其余 $13/19$ 次失败因 CLIP 特征表现得像词袋，难以捕捉句中关系、属性与序数信息 [22]。例如对「黑色螺丝刀」「放在 SPAM 罐上的杯子」，CLIP 更关注其他黑色物体或 SPAM 罐本身。我们发现用文本检索示范偶尔（尤其在挂架场景）受益于提示工程。

In total, our robot succeeds in $31$ out of $50$ language queries, including both fairly general queries (e.g., mug, drying rack) and ones that specify properties, such as color, material, and spatial relations (e.g., screwdriver on the block). Notably, our robot generalizes to out-of-distribution object categories including bowls, rolls of tape, whiteboard markers and utensils using demonstrations only on mugs and screwdrivers. Although this success rate is far from practical for industrial use, our overall strategy of using 2D visual priors for 3D scene understanding can leverage the rapid advancements in VLMs, which hold significant potential for improving performance.

合计在 $50$ 条语言查询中成功 $31$ 条，既包括较一般查询（如杯子、晾干架），也包括指定颜色、材质与空间关系等属性的查询（如「在方块上的螺丝刀」）。值得注意的是，仅用杯子与螺丝刀上的示范，机器人就能泛化到碗、胶带卷、白板笔与餐具等分布外类别。虽远未达工业实用成功率，但「用二维视觉先验做三维场景理解」的整体策略可借 VLM 的快速进步继续抬升表现。

Figure 7: Language-Guided Manipulation Execution. (Top Row) Heatmaps given the language queries. (Bottom Row) Robot executing grasps sequentially without rescanning. CLIP can behave like a bag-of-words, as shown by the bleed to the blue bowl for “blue screwdriver.”

图 7：语言引导操作执行。（上行）给定语言查询的热图。（下行）不重新扫描、顺序执行抓取。CLIP 可表现得像词袋，如「蓝色螺丝刀」时向蓝色碗渗色。

5 Related Work

5 相关工作

Open-Ended Generalization via Language.

A number of prior work use natural language for task-specification and long-horizon planning in applications such as tabletop and mobile manipulation [23, 24, 25, 26] and navigation [27, 28]. A recent line of work seeks to replicate the success of vision and language models in robotics, by jointly pre-training large foundation models on behavior data [29, 30]. The goal of our work is different. F3RM seeks to incorporate geometric information with any pre-trained features by lifting imagery and language prior knowledge into 3D.

经语言的开放式泛化。

若干先前工作用自然语言做任务指定与长程规划，用于桌面与移动操作 [23, 24, 25, 26] 以及导航 [27, 28]。近期一脉工作试图把视觉与语言模型的成功复制到机器人，在行为数据上联合预训练大型基础模型 [29, 30]。本文目标不同：F3RM 试图把几何信息与任意预训练特征结合，把图像与语言先验抬升到三维。

Dense 2D Visual Descriptors.

[31, 32] use dynamic 3D reconstruction from RGB-D videos to provide association labels between pixels from different video frames. Dense Object Nets eliminate the need for dynamic reconstruction, using multi-view RGB-D images of static scenes in the context of robotics [13, 33]. NeRF-Supervision extends descriptor learning to thin structures and reflective materials via NeRFs for 3D reconstruction [14]. In contrast, recent work indicates that excellent dense correspondence can emerge at a larger scale without the need for explicit supervision [4, 34].

稠密二维视觉描述子。

[31, 32] 用 RGB-D 视频的动态三维重建为不同帧像素提供关联标签。Dense Object Nets 在机器人情境下取消动态重建，改用静态场景的多视角 RGB-D [13, 33]。NeRF-Supervision 经 NeRF 做三维重建，把描述子学习扩展到细结构与反光材质 [14]。相对地，近期工作表明大规模下可涌现出色稠密对应而无需显式监督 [4, 34]。

Geometric Aware Representations for Robotics.

Geometric understanding is an essential part of mapping [35, 36, 37], grasping [38, 39, 17, 40], and legged locomotion [41]. These work either require direct supervision from 3D data such as point clouds, or try to learn representations from posed 2D images or videos [13, 14]. [21, 42, 43] leverage neural scene representations to take advantage of their ability to handle reflective or transparent objects and fine geometry. Our work incorporates pre-trained vision foundation models to augment geometry with semantics.

面向机器人的几何感知表示。

几何理解是建图 [35, 36, 37]、抓取 [38, 39, 17, 40] 与足式运动 [41] 的关键部分。这些工作或需点云等三维数据直接监督，或试图从带位姿的二维图像/视频学习表示 [13, 14]。[21, 42, 43] 利用神经场景表示处理反光/透明物体与精细几何。本文引入预训练视觉基础模型，用语义增强几何。

3D Feature Fields.

A number of recent work integrate 2D foundation models with 3D neural fields in contexts other than robotic manipulation [44, 45, 46, 47, 48, 49, 50]. See Appendix A.8 for a detailed overview. Our work shares many similarities with LERF [51]. However, unlike the multi-scale image-level features used in LERF, we extract dense patch-level features from CLIP. Additionally, we take a step further by exploring the utilization of feature fields for robotic manipulation.

三维特征场。

若干近期工作在机器人操作以外的情境把二维基础模型与三维神经场结合 [44, 45, 46, 47, 48, 49, 50]。详细综述见附录 A.8。本文与 LERF [51] 有许多相似处；但不同于 LERF 使用的多尺度图像级特征，我们从 CLIP 提取稠密 patch 级特征，并进一步探索把特征场用于机器人操作。

6 Conclusion

6 结论

We have illustrated a way to combine 2D visual priors with 3D geometry to achieve open-ended scene understanding for few-shot and language-guided robot manipulation. Without fine-tuning, Distilled Feature Fields enable out-of-the-box generalization over variations in object categories, material, and poses. When the features are sourced from vision-language models, distilled feature fields offer language-guidance at various levels of semantic granularity.

我们展示了一种把二维视觉先验与三维几何结合、以实现少样本与语言引导机器人操作的开放式场景理解的途径。无需微调，蒸馏特征场即可在物体类别、材质与位姿变化上开箱泛化。当特征来自视觉–语言模型时，蒸馏特征场还能在多种语义粒度上提供语言引导。

Limitations.

Our system takes $1\text{m}\ 40\text{s}$ to collect $50$ images of the scene, and $90\text{s}$ to model the NeRF and feature field. This highlights the need to develop generalizable NeRFs that can recover geometry quickly with just a few views [9, 43], opening the possibility for closed-loop dynamic manipulation. More generally, novel view synthesis is a generative process not too different from image generation with GANs [52] and diffusion models [53]. These alternatives, to which our philosophy equally applies, hold promise for solving general-purpose visual and geometric understanding.

局限。

系统采集场景的 $50$ 张图像需约 $1$ 分 $40$ 秒，建模 NeRF 与特征场需 $90$ 秒。这凸显需要开发能用极少视角快速恢复几何的可泛化 NeRF [9, 43]，从而打开闭环动态操作的可能。更一般地，新视角合成是一种生成过程，与 GAN [52] 与扩散模型 [53] 的图像生成并无本质不同。这些替代路线同样适用我们的思路，有望服务通用视觉与几何理解。

Acknowledgement

致谢

We gratefully acknowledge support from Amazon.com Service LLC, Award #2D-06310236; from the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/); from NSF grant 2214177; from AFOSR grant FA9550-22-1-0249; from ONR MURI grant N00014-22-1-2740; from ARO grant W911NF-23-1-0034; from the MIT-IBM Watson AI Lab; and from the MIT Quest for Intelligence. The authors also thank Tomás Lozano-Pérez, Lin Yen-Chen and Anthony Simeonov for their advice; Boyuan Chen for initial discussions; Rachel Holladay for her extensive help with setting up the robot; and Tom Silver for providing feedback on an earlier draft.

我们感谢 Amazon.com Service LLC 奖项 #2D-06310236；美国国家科学基金会合作协议 PHY-2019786（The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/）；NSF 资助 2214177；AFOSR 资助 FA9550-22-1-0249；ONR MURI 资助 N00014-22-1-2740；ARO 资助 W911NF-23-1-0034；MIT-IBM Watson AI Lab；以及 MIT Quest for Intelligence 的支持。作者亦感谢 Tomás Lozano-Pérez、Lin Yen-Chen 与 Anthony Simeonov 的建议；Boyuan Chen 的初期讨论；Rachel Holladay 在搭建机器人上的大量帮助；以及 Tom Silver 对早期草稿的反馈。

References

参考文献

[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning . PMLR, 2021.

[2] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. OpenCLIP, July 2021. URL https://doi.org/10.5281/zenodo.5143773 .

[3] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling Up Visual and Vision-Language Representation Learning with Noisy Text Supervision. In International Conference on Machine Learning . PMLR, 2021.

[4] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

[5] V. Tschernezki, I. Laina, D. Larlus, and A. Vedaldi. Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations. In Proceedings of the International Conference on 3D Vision (3DV) , 2022.

[6] S. Kobayashi, E. Matsumoto, and V. Sitzmann. Decomposing NeRF for Editing via Feature Field Distillation. In Advances in Neural Information Processing Systems , volume 35, 2022.

[7] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel. Deep ViT Features as Dense Visual Descriptors. ECCVW What is Motion For? , 2022.

[8] T. Müller, A. Evans, C. Schied, and A. Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics (ToG) , 2022.

[9] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

[10] M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: A Modular Framework for Neural Radiance Field Development. In ACM SIGGRAPH 2023 Conference Proceedings , SIGGRAPH ’23, 2023.

[11] C. Zhou, C. C. Loy, and B. Dai. Extract free dense labels from clip. In European Conference on Computer Vision (ECCV) , 2022.

[12] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In European Conference on Computer Vision (ECCV) , 2020.

[13] P. R. Florence, L. Manuelli, and R. Tedrake. Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. In Proceedings of the 2nd Conference on Robot Learning (CoRL) , June 2018.

[14] L. Yen-Chen, P. Florence, J. T. Barron, T.-Y. Lin, A. Rodriguez, and P. Isola. NeRF-Supervision: Learning dense object descriptors from neural radiance fields. In IEEE Conference on Robotics and Automation (ICRA) , 2022.

[15] M. Hamilton, Z. Zhang, B. Hariharan, N. Snavely, and W. T. Freeman. Unsupervised Semantic Segmentation by Distilling Feature Correspondences. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=SaKO6z6Hl0c .

[16] Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

[17] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitzmann. Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation. In IEEE Conference on Robotics and Automation (ICRA) , 2022.

[18] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

[19] E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.org , 2016.

[20] C. Garrett. PyBullet Planning. https://github.com/caelan/pybullet-planning , 2018.

[21] L. Yen-Chen, P. Florence, A. Zeng, J. T. Barron, Y. Du, W.-C. Ma, A. Simeonov, A. R. Garcia, and P. Isola. MIRA: Mental imagery for robotic affordances. In Conference on Robot Learning (CoRL) , 2022.

[22] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou. When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It? In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=KRLUvxh8uaX .

[23] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL) , 2021.

[24] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL) , 2022.

[25] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language-driven representation learning for robotics. In Robotics: Science and Systems (RSS) , 2023.

[26] S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor. Language-Conditioned Imitation Learning for Robot Manipulation Tasks. Advances in Neural Information Processing Systems , 33, 2020.

[27] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, B. Zitkovich, F. Xia, C. Finn, et al. Open-world object manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303.00905 , 2023.

[28] D. Shah, B. Osinski, B. Ichter, and S. Levine. LM-nav: Robotic navigation with large pre-trained models of language, vision, and action. In 6th Annual Conference on Robot Learning , 2022. URL https://openreview.net/forum?id=UW5A3SweAH .

[29] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. PaLM-E: An Embodied Multimodal Language Model. arXiv preprint arXiv:2303.03378 , Mar. 2023.

[30] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-1: Robotics Transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , Dec. 2022.

[31] C. B. Choy, J. Gwak, S. Savarese, and M. Chandraker. Universal Correspondence Network. In Advances in Neural Information Processing Systems , June 2016.

[32] T. Schmidt, R. Newcombe, and D. Fox. Self-supervised visual descriptor learning for dense correspondence. IEEE Robotics and Automation Letters (RA-L) , Apr. 2017.

[33] P. Florence, L. Manuelli, and R. Tedrake. Self-Supervised Correspondence in Visuomotor Policy Learning. IEEE Robotics and Automation Letters (RA-L) , 5(2):492–499, Apr. 2020.

[34] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. arXiv:2304.07193 , 2023.

[35] J. J. Leonard and H. F. Durrant-Whyte. Simultaneous map building and localization for an autonomous mobile robot. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , volume 3, pages 1442–1447, 1991.

[36] H. Durrant-Whyte, D. Rye, and E. Nebot. Localization of Autonomous Guided Vehicles. In Robotics Research , pages 613–625. Springer London, 1996.

[37] H. Durrant-Whyte and T. Bailey. Simultaneous localization and mapping: part i. IEEE Robotics and Automation Magazine , 13(2):99–110, 2006. doi: 10.1109/MRA.2006.1638022 .

[38] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. In Robotics: Science and Systems (RSS) , 2017.

[39] X. Yan, J. Hsu, M. Khansari, Y. Bai, A. Pathak, A. Gupta, J. Davidson, and H. Lee. Learning 6-dof grasping interaction via deep geometry-aware 3d representations. In IEEE International Conference on Robotics and Automation (ICRA) , pages 3766–3773. IEEE, 2018.

[40] J. Urain, N. Funk, J. Peters, and G. Chalvatzaki. SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion. IEEE International Conference on Robotics and Automation (ICRA) , 2023.

[41] R. Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

[42] J. Kerr, L. Fu, H. Huang, Y. Avigal, M. Tancik, J. Ichnowski, A. Kanazawa, and K. Goldberg. Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent Objects. In 6th Annual Conference on Robot Learning , 2022. URL https://openreview.net/forum?id=Bxr45keYrf .

[43] Q. Dai, Y. Zhu, Y. Geng, C. Ruan, J. Zhang, and H. Wang. GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF. In IEEE International Conference on Robotics and Automation (ICRA) , 2023.

[44] H.-Y. F. Tung, R. Cheng, and K. Fragkiadaki. Learning spatial common sense with geometry-aware recurrent networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2595–2603, 2019.

[45] H.-Y. F. Tung, Z. Xian, M. Prabhudesai, S. Lal, and K. Fragkiadaki. 3D-OES: Viewpoint-invariant object-factorized environment simulators. In Proceedings of the 4th Conference on Robot Learning (2020) , 2020.

[46] S. Peng, K. Genova, C. M. Jiang, A. Tagliasacchi, M. Pollefeys, and T. Funkhouser. OpenScene: 3D Scene Understanding with Open Vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

[47] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. In Robotics: Science and Systems (RSS) , 2023.

[48] B. Bolte, A. S. Wang, J. Yang, M. Mukadam, M. Kalakrishnan, and C. Paxton. USA-Net: Unified Semantic and Affordance Representations for Robot Memory. ArXiv , abs/2304.12164, 2023.

[49] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual Language Maps for Robot Navigation. In IEEE International Conference on Robotics and Automation (ICRA) , London, UK, 2023.

[50] K. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, S. Li, G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, J. Tenenbaum, C. de Melo, M. Krishna, L. Paull, F. Shkurti, and A. Torralba. Conceptfusion: Open-set multimodal 3d mapping. In Robotics: Science and Systems (RSS) , 2023.

[51] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. In International Conference on Computer Vision (ICCV) , 2023.

[52] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems , volume 27, 2014.

[53] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=PxTIG12RRHS .

[54] J. T. Kajiya and B. P. Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics , 18(3):165–174, 1984.

[55] N. Max. Optical models for direct volume rendering. IEEE Trans. Vis. Comput. Graph. , 1(2):99–108, June 1995.

[56] J. L. Schönberger and J.-M. Frahm. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

[57] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise View Selection for Unstructured Multi-View Stereo. In European Conference on Computer Vision (ECCV) , 2016.

[58] C.-H. Lin, W.-C. Ma, A. Torralba, and S. Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5741–5751, 2021.

[59] Z. Wang, S. Wu, W. Xie, M. Chen, and V. A. Prisacariu. NeRF − ⁣ − -- : Neural Radiance Fields Without Known Camera Parameters. arXiv preprint arXiv:2102.07064 , 2021.

[60] A. Simeonov, Y. Du, L. Yen-Chen, , A. Rodriguez, L. P. Kaelbling, T. L. Perez, and P. Agrawal. SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields. In Proceedings of the 6th Conference on Robot Learning (CoRL) . PMLR, 2022.

[61] V. Zhong, T. Rocktäschel, and E. Grefenstette. RTFM: Generalising to New Environment Dynamics via Reading. In International Conference on Learning Representations , 2020. URL https://arxiv.org/abs/1910.08210 .

[62] D. Bahdanau, F. Hill, J. Leike, E. Hughes, A. Hosseini, P. Kohli, and E. Grefenstette. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=H1xsSjC9Ym .

[63] R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision . Cambridge University Press, Mar. 2004.

[64] A. W. Harley, S. K. Lakshmikanth, F. Li, X. Zhou, H.-Y. F. Tung, and K. Fragkiadaki. Learning from unlabelled videos using contrastive predictive neural 3d mapping. arXiv preprint arXiv:1906.03764 , 2019.

[65] K. Mazur, E. Sucar, and A. Davison. Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understanding. In IEEE Conference on Robotics and Automation (ICRA) , 2023.

[66] L. Li, Z. Shen, Z. Wang, L. Shen, and L. Bo. Compressing volumetric radiance fields to 1 mb. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4222–4231, 2023.

Appendix

附录

Appendix A.1 Neural Radiance Fields (NeRFs)

附录 A.1 神经辐射场（NeRF）

Neural radiance fields [12] model a scene as a 6D, vector-valued continuous function that maps from a position $\mathbf{x}=(x,y,z)$ and a normalized viewing direction $\mathbf{d}=(d_{x},d_{y},d_{z})$, to the differential density $\sigma$ and emitted color $(r,g,b)$. In practice, this is achieved via two neural networks which partially share parameters: 1) the density network $\sigma(\mathbf{x})$ which depends only on the position $\mathbf{x}$; and 2) the color network $\mathbf{c}(\mathbf{x},\mathbf{d})$ which depends on both the position $\mathbf{x}$ and viewing direction $\mathbf{d}$.

神经辐射场 [12] 把场景建模为六维向量值连续函数：从位置 $\mathbf{x}=(x,y,z)$ 与归一化观察方向 $\mathbf{d}=(d_{x},d_{y},d_{z})$ 映射到微分密度 $\sigma$ 与发射颜色 $(r,g,b)$。实践中由部分共享参数的两个网络实现：1）仅依赖位置 $\mathbf{x}$ 的密度网络 $\sigma(\mathbf{x})$；2）依赖位置 $\mathbf{x}$ 与观察方向 $\mathbf{d}$ 的颜色网络 $\mathbf{c}(\mathbf{x},\mathbf{d})$。

Novel-View Synthesis.

NeRF synthesizes an image by casting a ray $\mathbf{r}$ from the camera origin $\mathbf{o}$ through the center of each pixel. Points along the ray are parameterized as $\mathbf{r}_{t}=\mathbf{o}+t\mathbf{d}$, where $t$ is the distance of the point to the camera origin $\mathbf{o}$. The color $\mathbf{C}(\mathbf{r})$ of the ray $\mathbf{r}$ between the near and far scene bounds $t_{n}$ and $t_{f}$ is given by the volume rendering integral [54]

新视角合成。

NeRF 从相机原点 $\mathbf{o}$ 经每个像素中心投射射线 $\mathbf{r}$ 来合成图像。沿射线的点参数化为 $\mathbf{r}_{t}=\mathbf{o}+t\mathbf{d}$，其中 $t$ 为点到相机原点 $\mathbf{o}$ 的距离。射线 $\mathbf{r}$ 在场景近远界 $t_{n}$ 与 $t_{f}$ 之间的颜色 $\mathbf{C}(\mathbf{r})$ 由体积渲染积分给出 [54]

$$\mathbf{C}(\mathbf{r})=\int_{t_{n}}^{t_{f}}T(t)\sigma(\mathbf{r}_{t})\mathbf{c}(\mathbf{r}_{t},\mathbf{d})\,\mathrm{d}t,\quad T(t)=\exp\left({-\int_{t_{n}}^{t}\sigma(\mathbf{r}_{s})\,\mathrm{d}s}\right), \tag{5}$$

where $T(t)$ is the accumulated transmittance along the ray from $\mathbf{r}_{t_{n}}$ to $\mathbf{r}_{t}$.

其中 $T(t)$ 为沿射线从 $\mathbf{r}_{t_{n}}$ 到 $\mathbf{r}_{t}$ 的累积透射率。

Modeling a Scene with NeRFs.

For a scene, we are given a dataset of $N$ RGB images $\{\mathbf{I}\}_{i=1}^{N}$ with camera poses $\{\mathbf{T}\}_{i=1}^{N}$. At each iteration, we sample a batch of rays $\mathcal{R}\sim\{\mathbf{T}\}_{i=1}^{N}$ and optimize $\sigma$ and $\mathbf{c}$ by minimizing the photometric loss $\mathcal{L}_{\text{rgb}}=\sum_{\mathbf{r}\in\mathcal{R}}\|\ \mathbf{\hat{C}}(\mathbf{r})-\mathbf{I}(\mathbf{r})\|_{2}^{2},$ where $\mathbf{I}(\mathbf{r})$ is the RGB value of the pixel corresponding to ray $\mathbf{r}\in\mathcal{R}$, and $\mathbf{\hat{C}}(\mathbf{r})$ is the color estimated by the model using a discrete approximation of Equation 5 [12, 55].

用 NeRF 建模场景。

对一个场景，给定 $N$ 张 RGB 图像 $\{\mathbf{I}\}_{i=1}^{N}$ 及其相机位姿 $\{\mathbf{T}\}_{i=1}^{N}$。每步从 $\{\mathbf{T}\}_{i=1}^{N}$ 采样一批射线 $\mathcal{R}$，通过最小化光度损失 $\mathcal{L}_{\text{rgb}}=\sum_{\mathbf{r}\in\mathcal{R}}\|\ \mathbf{\hat{C}}(\mathbf{r})-\mathbf{I}(\mathbf{r})\|_{2}^{2}$ 优化 $\sigma$ 与 $\mathbf{c}$，其中 $\mathbf{I}(\mathbf{r})$ 为射线 $\mathbf{r}\in\mathcal{R}$ 对应像素的 RGB，$\mathbf{\hat{C}}(\mathbf{r})$ 为由式 5 离散近似估计的颜色 [12, 55]。

Appendix A.2 Dense 2D Feature Extraction via MaskCLIP

附录 A.2 经 MaskCLIP 的稠密二维特征提取

We provide pseudo code for the MaskCLIP method [11] for extracting dense, patch-level features from the CLIP model [1] below. Algorithm 1 is the computation graph of the last layer of vanilla CLIP. Algorithm 2 is MaskCLIP’s modified graph. Note that the two linear transformations via $W_{\text{v}}$ and $W_{\text{out}}$ can be fused into a single convolution operation. We provide our feature extraction code in our GitHub repository (https://github.com/f3rm/f3rm).

下面给出从 CLIP 模型 [1] 提取稠密 patch 级特征的 MaskCLIP 方法 [11] 伪代码。算法 1 是原版 CLIP 最后一层的计算图；算法 2 是 MaskCLIP 的改写图。注意经 $W_{\text{v}}$ 与 $W_{\text{out}}$ 的两次线性变换可融为单次卷积。特征提取代码见 GitHub 仓库（https://github.com/f3rm/f3rm）。

1 def forward(x):
2 q, k, v = W_qkv @ self.ln_1(x)
3 v = (q[:1] * k).softmax(dim=-1) * v
4 x = x + W_out @ v
5 x = x + self.mlp(self.ln_2(x))
6 return x[:1] # the CLS token

Algorithm 1 Image Feature (Original)

1 def forward(x):
2 q, k, v = W_qkv @ self.ln_1(x)
3 v = (q[:1] * k).softmax(dim=-1) * v
4 x = x + W_out @ v
5 x = x + self.mlp(self.ln_2(x))
6 return x[:1] # the CLS token

算法 1 图像特征（原版）

1 def forward(x):
2 v = W_v @ self.ln_1(x)
3 z = W_out @ v
4 return z[1:] # all but the CLS token

Algorithm 2 Dense Features (MaskCLIP [11])

1 def forward(x):
2 v = W_v @ self.ln_1(x)
3 z = W_out @ v
4 return z[1:] # all but the CLS token

算法 2 稠密特征（MaskCLIP [11]）

Appendix A.3 Feature Fields

附录 A.3 特征场

Implementation Details.

Memory for caching the 2D feature map is a significant system bottleneck that does not appear with RGB reconstruction because high-dimensional features, up-scaled to the RGB image resolution, can grow to more than $40$ GB for a standard NeRF dataset. We solve this issue by reconstructing patch-level feature maps without up-scaling them to pixel resolution. We speed up our feature distillation by building off newer NeRF implementations using hierarchical hash grids [8] based on Nerfacto [10].

实现细节。

缓存二维特征图的内存是显著系统瓶颈——RGB 重建没有这个问题——因为高维特征上采样到 RGB 分辨率后，对标准 NeRF 数据集可超过 $40$ GB。我们的解决办法是重建 patch 级特征图而不上采样到像素分辨率。特征蒸馏基于 Nerfacto [10] 的层次哈希网格 [8] 等较新 NeRF 实现以加速。

Feature Field Quality.

F3RM benefits from neural feature fields’ ability to reconstruct detailed 3D geometry. We offer such an example in Figure A8. Notice the difference in resolution, between the source 2D feature map (middle), and the final feature field.

特征场质量。

F3RM 受益于神经特征场重建细致三维几何的能力。图 A8 给出一例。注意源二维特征图（中）与最终特征场之间的分辨率差异。

(a) RGB Image

(a) RGB 图像

A.3.1 Ablation on Feature Field Architecture

A.3.1 特征场架构消融

We implement our feature field as a hierarchical hash grid [8] that takes a 3D position $\mathbf{x}$ as input, and outputs the feature vector. We compare this against a MLP head that takes the intermediate features output by NeRF as input, which is similar to the architectures in [5, 6]. We first train a NeRF on images collected by the robot of a tabletop scene, then distill features for $10000$ steps over $3$ seeds.

我们把特征场实现为以三维位置 $\mathbf{x}$ 为输入、输出特征向量的层次哈希网格 [8]。对照是以 NeRF 中间特征为输入的 MLP 头，类似 [5, 6] 中的架构。先在机器人采集的桌面场景图像上训练 NeRF，再在 $3$ 个种子上各蒸馏特征 $10000$ 步。

Figure A9 shows that the hash grid architecture achieves a lower mean squared error (MSE), because it is able to capture higher-frequency signals and finer details. While the difference in the MSE seems marginal between these two architectures, the hash grid-based architecture qualitatively results in significantly more well-defined semantic boundaries between objects as shown in Figure A10.

图 A9 显示哈希网格架构达到更低均方误差（MSE），因其能捕捉更高频信号与更细细节。虽 MSE 差异看似不大，但如图 A10，哈希网格在定性上给出物体间更清晰的语义边界。

(a) CLIP Feature Mean Squared Error

(a) CLIP 特征均方误差

Figure A10: Comparing Feature Field Architectures. We show the similarity heatmaps for the language query “metal mug” on the scene shown in Fig.7. (a) When using the MLP head, regions unrelated to the metal mug exhibit high similarity, as shown by the red bleed onto objects including the red screwdriver and tip of the whiteboard marker. (b) In contrast, the hash grid architecture results in significantly less bleed and more well-defined semantic boundaries between objects.

图 A10：比较特征场架构。 在图 7 所示场景上，对语言查询「metal mug」给出相似度热图。（a）用 MLP 头时，与金属杯无关的区域仍呈高相似度，红渗色到红色螺丝刀与白板笔尖等物体。（b）相对地，哈希网格渗色显著更少，物体间语义边界更清晰。

Appendix A.4 Experimental Setup

附录 A.4 实验设置

We provide details about our experimental setup used across our experiments for learning to grasp from demonstrations and language-guided object manipulation.

下面给出从示范学习抓取与语言引导物体操作实验中共用的实验设置细节。

Physical Setup.

We collect RGB images with a RealSense D415 camera (the depth sensor is not used) mounted on a selfie stick. The selfie stick is used to increase the coverage of the workspace, as a wrist-mounted camera can only capture a small area of the workspace due to kinematic limitations. We program a Franka Panda arm to pick up the selfie stick from a magnetic mount, scan $50\times 1280\times 720$ RGB images of the scene following a fixed trajectory of three helical passes at different heights, and place the selfie stick back on the mount.

物理设置。

用固定在自拍杆上的 RealSense D415 采集 RGB 图像（不使用深度传感器）。自拍杆用于扩大工作空间覆盖：腕部相机因运动学限制往往只能拍到一小块区域。我们编程让 Franka Panda 机械臂从磁吸座取下自拍杆，按固定轨迹做三圈不同高度的螺旋扫描，采集 $50\times 1280\times 720$ 张场景 RGB，再把自拍杆放回座上。

To calibrate the camera poses, we run COLMAP [56, 57] on a scan of a dedicated calibration scene with calibration markers placed by the robot at known poses. We use these objects to solve for the transformation from the COLMAP coordinate system to the world coordinate system. These camera poses are reused on subsequent scans. Given that the true camera poses vary due to small differences in how the robot grasps the selfie-stick, we optimize them as part of NeRF modeling to minimize any errors and improve reconstruction quality [10, 58, 59].

为标定相机位姿，在机器人把标定标记摆到已知位姿的专用标定场景上跑 COLMAP [56, 57]。用这些物体求解从 COLMAP 坐标系到世界坐标系的变换。后续扫描复用这些相机位姿。由于机器人抓自拍杆的细微差异会使真实相机位姿变动，我们在 NeRF 建模中一并优化它们，以减小误差、提升重建质量 [10, 58, 59]。

Labeling Demonstrations.

We label demonstrations in virtual reality (VR) using a web-based 3D viewer based on Three.js that we developed which supports the real-time rendering of NeRFs, point clouds, and meshes. Given a NeRF of the demonstration scene, we sample a point cloud and export it into the viewer. We open the viewer in a Meta Quest 2 headset to visualize the scene, and move a gripper to the desired 6-DOF pose using the hand controllers (see Fig.2a).

标注示范。

在虚拟现实（VR）中用我们开发的、基于 Three.js 的网页三维查看器标注示范；该查看器支持 NeRF、点云与网格的实时渲染。给定示范场景的 NeRF，采样点云并导出到查看器。在 Meta Quest 2 头显中打开查看器观察场景，用手持控制器把夹爪移到期望的六自由度位姿（见图 2a）。

NeRF and Feature Field Modeling.

Feature Type	Resolution
DINO ViT	$98\times 55$
CLIP ViT	$42\times 24$
CLIP ResNet	$24\times 14$

Table 3: Feature Map Resolutions. Resolutions of the features output by the vision models given a $1280\times 720$ RGB image.

NeRF 与特征场建模。

特征类型	分辨率
DINO ViT	$98\times 55$
CLIP ViT	$42\times 24$
CLIP ResNet	$24\times 14$

表 3：特征图分辨率。 给定 $1280\times 720$ RGB 图像时，各视觉模型输出特征的分辨率。

We downscale the images to $640\times 480$ to speed up modeling of the RGB NeRF, and use the original $1280\times 720$ images as input to the vision model for dense feature extraction. We optimize the NeRF and feature field sequentially for $2000$ steps each, which takes at most $90$s (average is $80$s) on a NVIDIA RTX 3090, including the time to load the vision model into memory and extract features.

将图像缩到 $640\times 480$ 以加速 RGB NeRF 建模，并用原始 $1280\times 720$ 图像输入视觉模型做稠密特征提取。NeRF 与特征场各顺序优化 $2000$ 步，在 NVIDIA RTX 3090 上至多 $90$ 秒（平均 $80$ 秒），含把视觉模型载入内存并提取特征的时间。

In our experiments, we distill the features at their original feature map resolution which is significantly smaller than the RGB images (see Table 3). We achieve this by transforming the camera intrinsics to match the feature map resolutions, and sampling rays based on this updated camera model. The specific models we used were dino_vits8 for DINO ViT, ViT-L/14@336px for CLIP ViT, and RN50x64 for CLIP ResNet.

实验中我们在显著小于 RGB 图像的原始特征图分辨率上蒸馏特征（见表 3）。做法是把相机内参变换到匹配特征图分辨率，并按更新后的相机模型采样射线。所用具体模型为：DINO ViT 用 dino_vits8，CLIP ViT 用 ViT-L/14@336px，CLIP ResNet 用 RN50x64。

Appendix A.5 Ablation on Number of Training Views

附录 A.5 训练视角数量消融

Although our robot scans $50$ images per scene in our experiments, we demonstrate that it is possible to use a significantly smaller number of views for NeRF and feature field modeling without a significant loss in quality. To investigate this, we ablate the number of training images by evenly subsampling from the $50$ scanned images and modeling a NeRF and feature field.

虽实验中每场景扫描 $50$ 张图，但我们表明可用显著更少视角建模 NeRF 与特征场而不致严重掉质量。为此，从 $50$ 张扫描图中均匀子采样，消融训练图像数量并建模 NeRF 与特征场。

Figure A11 qualitatively compares the RGB, depth, and segmentation heatmaps. We observe an increase in floaters as we reduce the number of training images, with approximately $20$ images being the lower bound before a drastic decline in quality.

图 A11 定性比较 RGB、深度与分割热图。随训练图像减少，漂浮伪影增多；约 $20$ 张是质量骤降前的下界。

Figure A11: Ablating Number of Training Views. We qualitatively compare feature fields trained on different numbers of views. (Top Row) Segmentation heatmap for “Baymax” from a CLIP feature field overlaid on the RGB image from NeRF. (Bottom Row) Depth map rendered from NeRF.

图 A11：消融训练视角数。 定性比较不同视角数训练的特征场。（上行）CLIP 特征场上「Baymax」分割热图叠在 NeRF 的 RGB 上。（下行）NeRF 渲染的深度图。

Appendix A.6 Learning to Grasp from Demonstrations

附录 A.6 从示范学习抓取

Sampling Query Points.

We use $N_{q}=100$ query points across all the tasks shown in Fig.5. As other works have observed [17, 60], the downstream performance can vary significantly across different samples of the query points. To address this issue, we sample five sets of query points over different seeds for each task, and run the grasp optimization procedure across a set of test scenes used for method development. We select the query points that achieved the highest success rate on the test scenes. The covariance of the Gaussian is manually tuned to fit the task.

采样查询点。

图 5 所示全部任务使用 $N_{q}=100$ 个查询点。如其他工作所观察 [17, 60]，下游表现可随查询点采样显著变化。为此，对每项任务用不同种子采样五组查询点，并在方法开发用的测试场景上跑抓取优化；选取测试场景成功率最高的查询点。高斯协方差按任务手动调节。

Grasp Pose Optimization.

We first discuss how we initialize the grasp poses. We consider a tabletop workspace of size $0.7\times 0.8\times 0.35$ meters, and sample a dense voxel grid over the workspace with voxels of size $\delta=0.0075\text{m}$ (we use $0.005\text{m}$ for the cup on racks experiment), where each voxel $\mathbf{v}=(x,y,z)$ represents the translation for a grasp pose.

抓取位姿优化。

先说明如何初始化抓取位姿。考虑尺寸 $0.7\times 0.8\times 0.35$ 米的桌面工作空间，以边长 $\delta=0.0075$ m 的体素在其上采样稠密网格（杯挂架实验用 $0.005$ m），每个体素 $\mathbf{v}=(x,y,z)$ 表示抓取平移。

Next, we compute the alpha value $\alpha(\mathbf{v})$ for each voxel using the NeRF density network $\sigma$, and filter out voxels with $\alpha(\mathbf{v})<\epsilon_{\text{free}}=0.1$. This removes approximately $98\%$ of voxels by ignoring free space. The cosine similarity of the voxel features $\mathbf{f}_{\alpha}(\mathbf{v})$ is thresholded with the task embedding $\mathbf{Z}_{M}$ to further filter out voxels. This threshold is adjusted depending on the task and type of feature distilled, and typically cuts down $80\%$ of the remaining voxels. Finally, we uniformly sample $N_{r}=8$ rotations for each voxel to get the initial grasp proposals $\mathcal{T}$.

接着用 NeRF 密度网络 $\sigma$ 计算每个体素的 alpha $\alpha(\mathbf{v})$，过滤 $\alpha(\mathbf{v})<\epsilon_{\text{free}}=0.1$ 的体素，从而忽略自由空间、去掉约 $98\%$ 体素。再用体素特征 $\mathbf{f}_{\alpha}(\mathbf{v})$ 与任务嵌入 $\mathbf{Z}_{M}$ 的余弦相似度设阈值进一步过滤；阈值随任务与蒸馏特征类型调整，通常再砍掉剩余体素的约 $80\%$。最后对每个体素均匀采样 $N_{r}=8$ 个旋转，得到初始抓取提案 $\mathcal{T}$。

We minimize Equation 3 to find the grasp pose that best matches the demonstrations using the Adam optimizer [18] for $50$ steps with a learning rate of $5\text{e-}3$. This entire procedure takes $15$s on average, but could easily be sped up.

用 Adam [18] 最小化式 3，学习率 $5\text{e-}3$，共 $50$ 步，以找到最匹配示范的抓取位姿。整段流程平均约 $15$ 秒，但很容易再加速。

Grasp Execution.

We reject grasp poses which cause collisions by checking the overlap between a voxelized mesh of the Panda gripper and NeRF geometry by querying the density field $\sigma$. We input the ranked list of grasp poses into an inverse kinematics solver and BiRRT motion planner in PyBullet [19, 20], and execute the highest-ranked grasp with a feasible motion plan.

抓取执行。

通过查询密度场 $\sigma$，检查体素化的 Panda 夹爪网格与 NeRF 几何的重叠，以拒绝碰撞抓取。把排序后的抓取位姿送入 PyBullet 中的逆运动学求解器与 BiRRT 运动规划器 [19, 20]，执行有可行运动规划且排名最高的抓取。

A.6.1 Baselines

A.6.1 基线

We provide implementation details of the four baselines used in our few-shot imitation learning experiments. The first three baselines use NeRF-based outputs as features for the query point-based pose optimization:

下面给出少样本模仿学习实验中四个基线的实现细节。前三个基线把 NeRF 输出当作查询点位姿优化的特征：

1. Density: we use the alpha $\alpha\in(0,1)$ values for NeRF density to ensure the values are scaled consistently through different scenes, as the density values output by the density field $\sigma$ are unbounded.
2. Intermediate Features: we use the features output by the intermediate density embedding MLP in Nerfacto [10], which have a dimensionality of $15$.
3. RGB: we use $[r,g,b,\alpha]$ as the feature for this baseline. $\alpha$ is used to ensure that this baseline pays attention to both the color and geometry, as we found that using RGB only with the alpha-weighted feature field $\mathbf{f}_{\alpha}$ (Eq.2) collapsed RGB values to $(0,0,0)$ for free space, which corresponds to the color black.

1. 密度：使用 NeRF 密度的 alpha $\alpha\in(0,1)$，以保证不同场景数值尺度一致，因为密度场 $\sigma$ 的输出无界。
2. 中间特征：使用 Nerfacto [10] 中间密度嵌入 MLP 的输出特征，维数为 $15$。
3. RGB：使用 $[r,g,b,\alpha]$ 作特征。加入 $\alpha$ 是为了让基线同时关注颜色与几何——我们发现仅用 RGB、再配 alpha 加权特征场 $\mathbf{f}_{\alpha}$（式 2）时，自由空间的 RGB 会塌成 $(0,0,0)$，对应黑色。

MIRA Baseline.

The fourth baseline we consider is Mental Imagery for Robotic Affordances (MIRA) [21], a NeRF-based framework for 6-DOF pick-and-place from demonstrations that renders orthographic views for pixel-wise affordance prediction. Orthographic rendering ensures that an object has the same size regardless of its distance to the camera, and is used to complement the translation equivariance of the Fully Convolutional Network (FCN) for predicting affordances.

MIRA 基线。

第四个基线是 Mental Imagery for Robotic Affordances（MIRA） [21]：基于 NeRF 的六自由度抓取–放置框架，从示范渲染正交视角做逐像素可操作性预测。正交渲染保证物体尺寸与到相机距离无关，用以配合全卷积网络（FCN）的平移等变性来预测可操作性。

MIRA formulates each pixel in a rendered orthographic view as a 6-DOF action $\mathbf{T}=(\mathbf{R},\mathbf{t})$, with the orientation of the view defining the rotation $\mathbf{R}$ and the estimated depth from NeRF defining the translation $\mathbf{t}$. The FCN is trained to predict the pixels in the rendered views corresponding to the demonstrated 6-DOF actions, and reject pixels sampled from a set of negative views. During inference, MIRA renders several orthographic views of the scene and selects the pixel that has the maximum affordance across all views.

MIRA 把渲染正交视角中的每个像素表述为六自由度动作 $\mathbf{T}=(\mathbf{R},\mathbf{t})$：视角朝向定义旋转 $\mathbf{R}$，NeRF 估计的深度定义平移 $\mathbf{t}$。FCN 训练为预测渲染视角中对应示范六自由度动作的像素，并拒绝从一组负视角采样的像素。推断时，MIRA 渲染场景的多个正交视角，选取所有视角中可操作性最大的像素。

In our experiments, we train a separate FCN for $20000$ steps for each task in Fig.5 specified by two demonstrations, and sample negative pixels from datasets consisting solely of distractor objects. We use data augmentation following [21]’s provided implementation and apply random SE(2) transforms to the training views. Given a test scene, we scan $50$ RGB images as described in Appendix A.4, render $360$ orthographic viewpoints randomly sampled over an upper hemisphere looking towards the center of the workspace, and infer a 6-DOF action.

实验中，对图 5 每项由两次示范指定的任务单独训练一个 FCN，共 $20000$ 步，并从仅含干扰物的数据集采样负像素。数据增强遵循 [21] 提供的实现，对训练视角施加随机 SE(2) 变换。给定测试场景，按附录 A.4 扫描 $50$ 张 RGB，在朝向工作空间中心的上半球随机采样渲染 $360$ 个正交视角，并推断六自由度动作。

MIRA was designed for suction cup grippers and does not predict end-effector rotations. We attempted to learn this rotation, but found that the policy failed to generalize. To address this issue and give MIRA the best chance of success, we manually select the best end-effector rotation to achieve the task. We additionally find that MIRA often selects floater artifacts from NeRF, and manually filter these predictions out along with other unreasonable grasps (e.g., grasping the table itself).

MIRA 面向吸盘夹爪设计，不预测末端旋转。我们尝试学习该旋转，但策略未能泛化。为给 MIRA 最好成功机会，我们手动选择完成任务的最佳末端旋转。另外发现 MIRA 常选中 NeRF 的漂浮伪影，我们手动滤掉这些预测以及其他不合理抓取（例如抓桌子本身）。

(a) Test Scene with a Screwdriver

(a) 含螺丝刀的测试场景

Given that MIRA is trained from scratch given just two demonstrations, we find that it struggles to generalize and is easily confused by floaters and distractors despite data augmentations and negative samples. MIRA additionally reasons over 2.5D by using the rendered RGB and depth from NeRF as inputs to the FCN, while our query point-based formulation reasons explicitly over 3D. Because of this, we observe that MIRA can fail when there are occlusions or distractor objects that look like the demonstration objects from certain viewpoints. For example, one of the demonstrations for the screwdriver task was a top-down grasp on a screwdriver standing vertically in a rack (Fig.4(b)). Figure A12 depicts a test scene for the screwdriver grasping task where MIRA incorrectly selects the silicone brush as it looks similar to a screwdriver from a top-down 2.5D view. DINO and CLIP ResNet feature fields successfully grasp the screwdriver in this scene, highlighting the benefits of using pretrained features and reasoning explicitly over 3D geometry.

鉴于 MIRA 仅凭两次示范从头训练，我们发现即便有数据增强与负样本，它仍难泛化，易被漂浮物与干扰物迷惑。MIRA 还以 NeRF 渲染的 RGB 与深度作 FCN 输入，在 2.5D 上推理；而我们的查询点方案在三维上显式推理。因此，当存在遮挡或从某些视角看起来像示范物的干扰物时，MIRA 可能失败。例如，螺丝刀任务的一次示范是对竖直插在架子上的螺丝刀做自上而下抓取（图 4(b)）。图 A12 给出该任务测试场景：MIRA 误选硅胶刷，因其在自上而下的 2.5D 视角中像螺丝刀。DINO 与 CLIP ResNet 特征场在该场景成功抓住螺丝刀，凸显预训练特征与显式三维几何推理的好处。

A.6.2 DINO Failure Cases

A.6.2 DINO 失败情形

Our experiments show that DINO struggles with distractor objects which have high feature similarity to the demonstrations, despite not representing the objects and their parts we care about (Fig.13(b)). We observe that DINO has the tendency to overfit to color. On the other hand, CLIP struggles far less with distractors due to its stronger semantic understanding (Fig.13(c)).

实验表明，即便干扰物并不对应我们关心的物体及其部位，只要与示范特征相似度高，DINO 仍会受其困扰（图 13(b)）。我们观察到 DINO 有过拟合颜色的倾向。另一方面，CLIP 因更强语义理解，受干扰物影响小得多（图 13(c)）。

(a) Novel Scene

(a) 新场景

Appendix A.7 Language-Guided Manipulation

附录 A.7 语言引导操作

For the language-guided experiments, we distilled CLIP ViT features from ViT-L/14@336px. We reuse the $10$ demonstrations from the learning to grasp from demonstrations section and their associated $N_{q}=100$ query points, and sample $N_{r}=8$ rotations for each voxel when initializing the grasp proposals. We minimize the language-guided cost function in Equation 4 for $200$ steps with Adam using a learning rate of $2\times 10^{-3}$.

语言引导实验中，我们从 ViT-L/14@336px 蒸馏 CLIP ViT 特征。复用「从示范学习抓取」一节的 $10$ 次示范及其关联的 $N_{q}=100$ 查询点；初始化抓取提案时对每个体素采样 $N_{r}=8$ 个旋转。用 Adam 以学习率 $2\times 10^{-3}$ 最小化式 4 的语言引导代价，共 $200$ 步。

Retrieving Demonstrations via Text

In practice, we compare the user’s embedded text query $\mathbf{q}$ with the task embedding $\mathbf{Z}_{M}$ for each task $M$ which is specified by two demonstrations. We randomly sample object names as our negatives ($L^{-}$).

经文本检索示范

实践中，把用户嵌入的文本查询 $\mathbf{q}$ 与每个由两次示范指定的任务 $M$ 的任务嵌入 $\mathbf{Z}_{M}$ 比较。负样本（$L^{-}$）随机采样物体名称。

Inference Time.

The inference time to optimize for a grasp pose given a language query is $6.9$ seconds on average. We did not make any substantial attempts to speed this up, but note that reducing the number of optimization steps (we use $200$ steps but observe convergence usually within $50$-$100$ steps), pruning more aggressively, and improving the implementation will significantly reduce inference time.

推断时间。

给定语言查询优化抓取位姿的平均推断时间为 $6.9$ 秒。我们未做大幅加速，但减少优化步数（我们用 $200$ 步，通常 $50$–$100$ 步即收敛）、更激进剪枝与改进实现将显著缩短推断时间。

Appendix A.8 Additional Related Work

附录 A.8 补充相关工作

We provide a more comprehensive overview of the related work discussed in Section 5.

下面更全面地综述第 5 节讨论的相关工作。

Open-Ended Generalization via Language.

A number of prior work use natural language for task-specification and long-horizon planning in applications such as tabletop and mobile manipulation [23, 26], navigation [27, 28], and more generally, sequential decision-making in games [61, 62]. A recent line of work seeks to replicate the success of vision and language models in robotics, by jointly pre-training large foundation models on behavior data [29, 30]. One can refer to [25] for a more comprehensive coverage. The goal of our work is different. We seek to find a way to incorporate geometric information with any pre-trained features. F3RM is a model-agnostic approach for lifting imagery and language prior knowledge into 3D. In this regard, we are more closely connected to CLIPort [23], which focuses on top-down grasping, and PerAct [24], which trains a voxel grid conditioned behavior transformer from scratch given waypoints in motion trajectories.

经语言的开放式泛化。

若干先前工作用自然语言做任务指定与长程规划，用于桌面与移动操作 [23, 26]、导航 [27, 28]，以及更一般的游戏中序贯决策 [61, 62]。近期一脉工作试图把视觉与语言模型的成功复制到机器人，在行为数据上联合预训练大型基础模型 [29, 30]。更全面覆盖可参考 [25]。本文目标不同：我们寻找把几何信息与任意预训练特征结合的途径。F3RM 是与模型无关的、把图像与语言先验抬升到三维的方法。就此而言，我们与聚焦自上而下抓取的 CLIPort [23]、以及从运动轨迹航点从头训练体素条件行为 Transformer 的 PerAct [24] 更接近。

Semantic Understanding and Dense 2D Visual Descriptors.

Learning visual descriptors for dense correspondence estimation is a fundamental problem in computer vision. Among the earliest works, [31] and [32] used dynamic 3D reconstruction from RGB-D videos to provide labels of association between pixels from different video frames. Dense Object Nets tackle this problem in the context of robotics [13], and eliminate the need for dynamic reconstruction using multi-view RGB-D images of static scenes [33]. NeRF-Supervision [14] leverages NeRFs for 3D reconstruction, extending descriptor learning to thin structures and reflective materials that pose challenges for depth sensors. Unlike these prior works, recent work in vision foundation models shows that self-supervised vision transformers offer excellent dense correspondence [4]. When scaled to a large, curated dataset, such models offer phenomenal few-shot performance on various dense prediction tasks [34].

语义理解与稠密二维视觉描述子。

学习用于稠密对应估计的视觉描述子是计算机视觉基本问题。最早工作中，[31] 与 [32] 用 RGB-D 视频的动态三维重建为不同帧像素提供关联标签。Dense Object Nets 在机器人情境下处理该问题 [13]，并用静态场景的多视角 RGB-D 取消对动态重建的需求 [33]。NeRF-Supervision [14] 用 NeRF 做三维重建，把描述子学习扩展到深度传感器难处理的细结构与反光材质。与这些先前工作不同，视觉基础模型的近期工作表明自监督视觉 Transformer 可提供出色稠密对应 [4]。缩放到大规模精选数据时，此类模型在多种稠密预测任务上给出惊人的少样本表现 [34]。

Geometric Aware Representations for Robotics.

Geometric understanding has been a long-standing problem in computer vision [63] and is an essential and mission-critical part of mapping and navigation [35, 36, 37], grasping [38, 39, 17, 40], and legged locomotion [41]. These work either require direct supervision from 3D data such as Lidar point clouds, or try to learn representations from posed 2D images or videos [14]. More recently, the robot grasping community has experimented with neural scene representations to take advantage of their ability to handle reflective or transparent objects and fine geometry [21, 42, 43]. Our work incorporates pre-trained vision and vision-language foundation models to augment geometry with semantics.

面向机器人的几何感知表示。

几何理解是计算机视觉的长期问题 [63]，也是建图与导航 [35, 36, 37]、抓取 [38, 39, 17, 40] 与足式运动 [41] 的关键、攸关任务成败的部分。这些工作或需 Lidar 点云等三维数据直接监督，或试图从带位姿的二维图像/视频学习表示 [14]。更近地，机器人抓取社区试验神经场景表示，以利用其处理反光/透明物体与精细几何的能力 [21, 42, 43]。本文引入预训练视觉与视觉–语言基础模型，用语义增强几何。

3D Feature Fields in Vision and Robotics

A number of recent work integrate 2D foundation models with 3D neural fields in contexts other than robotic manipulation. For example, [44, 64] train geometry-aware recurrent neural networks to lift 2D features into 3D feature grids using sensed or predicted depth for improved 3D object detection and segmentation. 3D-OES [45] extend this to learn object dynamics models using graph neural networks over the 3D feature maps. OpenScene [46] and CLIP-Fields [47] distill 2D features into a neural field by sampling points from existing 3D point clouds or meshes. USA-Net [48] and VLMap [49] build 3D feature maps from RGB-D images and VLMs for navigation. Our approach uses RGB images and integrates geometric modeling via NeRF with feature fusion in a single pipeline. ConceptFusion [50] uses surfels to represent the features, which requires $50-100s$ GB per scene. Distilled feature fields offer a significant reduction in space [65]. Each of our scenes requires between $60-120$ MBs and could be further compressed using lower-rank approximations [66]. Our work shares many similarities with LERF [51]. However, unlike the image-level features used in LERF, which are derived from a hierarchy of image crops, we extract dense, patch-level features from CLIP. Additionally, we take a step further by exploring the utilization of these feature fields for robotic manipulation.

视觉与机器人中的三维特征场

若干近期工作在机器人操作以外情境把二维基础模型与三维神经场结合。例如，[44, 64] 训练几何感知循环神经网络，用感知或预测深度把二维特征抬升到三维特征网格，以改进三维物体检测与分割。3D-OES [45] 将其扩展为在三维特征图上用图神经网络学习物体动力学。OpenScene [46] 与 CLIP-Fields [47] 从已有三维点云或网格采样点，把二维特征蒸馏进神经场。USA-Net [48] 与 VLMap [49] 从 RGB-D 与 VLM 构建三维特征图用于导航。我们的方法用 RGB 图像，并在单一管线中把经 NeRF 的几何建模与特征融合结合。ConceptFusion [50] 用 surfel 表示特征，每场景需 $50$–$100$s GB。蒸馏特征场显著节省空间 [65]：我们每个场景约需 $60$–$120$ MB，还可用低秩近似进一步压缩 [66]。本文与 LERF [51] 有许多相似处；但不同于 LERF 由多尺度图像裁剪得到的图像级特征，我们从 CLIP 提取稠密 patch 级特征，并进一步探索把这些特征场用于机器人操作。