Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Abstract

摘要

22

22

21

21

I Introduction

I 引言

A central lesson from advances in machine learning and artificial intelligence is that large-scale learning from diverse datasets can enable capable AI systems by providing for general-purpose pretrained models. In fact, large-scale general-purpose models typically trained on large and diverse datasets can often outperform their narrowly targeted counterparts trained on smaller but more task-specific data. For instance, open-vocab classifiers (e.g., CLIP [1]) trained on large datasets scraped from the web tend to outperform fixed-vocabulary models trained on more limited datasets, and large language models [2, 3] trained on massive text corpora tend to outperform systems that are only trained on narrow task-specific datasets. Increasingly, the most effective way to tackle a given narrow task (e.g., in vision or NLP) is to adapt a general-purpose model. However, these lessons are difficult to apply in robotics: any single robotic domain might be too narrow, and while computer vision and NLP can leverage large datasets sourced from the web, comparably large and broad datasets for robotic interaction are hard to come by. Even the largest data collection efforts still end up with datasets that are a fraction of the size and diversity of benchmark datasets in vision (5-18M) [4, 5] and NLP (1.5B-4.5B) [6, 7]. More importantly, such datasets are often still narrow along some axes of variation, either focusing on a single environment, a single set of objects, or a narrow range of tasks. How can we overcome these challenges in robotics and move the field of robotic learning toward large data regime that has been so successful in other domains?

机器学习与人工智能进展的一条核心经验是：在多样化数据上大规模学习，能通过提供通才预训练模型来造就有能力的 AI 系统。事实上，通常在大规模多样数据上训练的通用大模型，往往优于在更小但更任务专用的数据上训练的 窄域 对应物。例如，在网络大规模数据上训练的开放词表分类器（如 CLIP [1]）常优于在更受限数据上训练的固定词表模型；在海量文本上训练的大语言模型 [2, 3] 也常优于只在窄任务数据上训练的系统。越来越多地，解决给定窄任务（视觉或 NLP）的最有效方式是适配一个通才模型。然而这些经验很难直接搬到机器人：任一单一机器人域都可能过窄；计算机视觉与 NLP 能利用网络大规模数据，而规模与广度可比的机器人交互数据却难求。即便最大的数据采集努力，所得数据集相对视觉（5-18M）[4, 5] 与 NLP（1.5B-4.5B）[6, 7] 基准仍只是零头，更重要的是往往在某些变化轴上仍然很窄——只盯一个环境、一组物体或一小段任务范围。我们如何克服这些挑战，把机器人学习推向其他领域已证明成功的大数据体制？

Inspired by the generalization made possible by pretraining large vision or language models on diverse data, we take the perspective that the goal of training generalizable robot policies requires X-embodiment training, i.e., with data from multiple robotic platforms. While each individual robotic learning dataset might be too narrow, the union of all such datasets provides a better coverage of variations in environments and robots. Learning generalizable robot policies requires developing methods that can utilize X-embodiment data, tapping into datasets from many labs, robots, and settings. Even if such datasets in their current size and coverage are insufficient to attain the impressive generalization results that have been demonstrated by large language models, in the future, the union of such data can potentially provide this kind of coverage. Because of this, we believe that enabling research into X-embodiment robotic learning is critical at the present juncture .

受在多样数据上预训练大规模视觉或语言模型所带来泛化的启发，我们采取的视角是：训练可泛化机器人策略的目标需要 跨本体（X-embodiment）训练 ，也就是使用来自多种机器人平台的数据。单个机器人学习数据集或许过窄，但所有此类数据的并集能更好覆盖环境与机器人的变化。学习可泛化策略，需要发展能利用跨本体数据、打通众多实验室、机器人与设定的方法。即便以当前规模与覆盖，这类数据还不足以复现大语言模型那类惊人泛化，未来其并集仍有望提供此类覆盖。因此， 我们相信：在当下节点推动跨本体机器人学习研究至关重要 。

Following this rationale, we have two goals: (1) Evaluate whether policies trained on data from many different robots and environments enjoy the benefits of positive transfer, attaining better performance than policies trained only on data from each evaluation setup. (2) Organize large robotic datasets to enable future research on X-embodiment models.

据此，我们有两个目标： （1） 评估在多机器人、多环境数据上训练的策略是否享受正向迁移，表现优于只在各评估设定数据上训练的策略； （2） 整理大规模机器人数据集，支撑未来的跨本体模型研究。

9

9

22

22

II Related Work

II 相关工作

Transfer across embodiments. A number of prior works have studied methods for transfer across robot embodiments in simulation [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22] and on real robots [23, 24, 25, 26, 27, 28, 29]. These methods often introduce mechanisms specifically designed to address the embodiment gap between different robots, such as shared action representations [14, 30], incorporating representation learning objectives [17, 26], adapting the learned policy on embodiment information [30, 31, 11, 18, 15], and decoupling robot and environment representations [24]. Prior work has provided initial demonstrations of X-embodiment training [27] and transfer [25, 32, 29] with transformer models. We investigate complementary architectures and provide complementary analyses, and, in particular, study the interaction between X-embodiment transfer and web-scale pretraining. Similarly, methods for transfer across human and robot embodiments also often employ techniques for reducing the embodiment gap, i.e. by translating between domains or learning transferable representations [33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]. Alternatively, some works focus on sub-aspects of the problem such as learning transferable reward functions [44, 17, 45, 46, 47, 48], goals [49, 50], dynamics models [51], or visual representations [52, 53, 54, 55, 56, 57, 58, 59] from human video data. Unlike most of these prior works, we directly train a policy on X-embodiment data, without any mechanisms to reduce the embodiment gap, and observe positive transfer by leveraging that data.

跨本体迁移 。 多项先前工作研究了仿真 [10-22] 与真实机器人 [23-29] 上的跨本体迁移方法。它们常引入专门缩小不同机器人之间本体差距的机制，如共享动作表示 [14, 30]、加入表示学习目标 [17, 26]、按本体信息适配所学策略 [30, 31, 11, 18, 15]、解耦机器人与环境表示 [24]。已有工作初步展示了用 Transformer 做跨本体训练 [27] 与迁移 [25, 32, 29]。我们考察互补架构并给出互补分析，尤其研究跨本体迁移与网络规模预训练的相互作用。类似地，人—机本体迁移方法也常采用缩小本体差距的技术，例如域间翻译或学习可迁移表示 [33-43]。另一些工作聚焦子问题，如从人类视频学习可迁移奖励 [44, 17, 45-48]、目标 [49, 50]、动力学模型 [51] 或视觉表示 [52-59]。与多数先前工作不同，我们直接在跨本体数据上训练策略，不引入缩小本体差距的机制，并观察到借助这些数据的正向迁移。

Large-scale robot learning datasets. The robot learning community has created open-source robot learning datasets, spanning grasping [60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], pushing interactions [72, 73, 74, 23], sets of objects and models [75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85], and teleoperated demonstrations [86, 87, 88, 89, 90, 8, 91, 92, 93, 94, 95]. With the exception of RoboNet [23], these datasets contain data of robots of the same type, whereas we focus on data spanning multiple embodiments. The goal of our data repository is complementary to these efforts: we process and aggregate a large number of prior datasets into a single, standardized repository, called Open X-Embodiment, which shows how robot learning datasets can be shared in a meaningul and useful way.

大规模机器人学习数据集 。 机器人学习社区已创建开源数据集，覆盖抓取 [60-71]、推挤交互 [72-74, 23]、物体与模型集合 [75-85] 以及遥操作演示 [86-95]。除 RoboNet [23] 外，这些数据多只含同一类型机器人；我们则聚焦跨多种本体的数据。本数据仓库的目标与这些努力互补：我们把大量先前数据加工并汇总为单一、标准化的仓库 Open X-Embodiment，展示机器人学习数据如何以有意义且可用的方式共享。

Language-conditioned robot learning. Prior work has aimed to endow robots and other agents with the ability to understand and follow language instructions [96, 97, 98, 99, 100, 101], often by learning language-conditioned policies [45, 102, 103, 104, 105, 40, 106, 8]. We train language-conditioned policies via imitation learning like many of these prior works but do so using large-scale multi-embodiment demonstration data. Following previous works that leverage pre-trained language embeddings [107, 45, 108, 103, 40, 109, 110, 8, 111, 112] and pre-trained vision-language models [113, 114, 115, 9] in robotic imitation learning, we study both forms of pre-training in our experiments, specifically following the recipes of RT-1 [8] and RT-2 [9].

语言条件机器人学习 。 先前工作致力于让机器人及其他智能体理解并遵循语言指令 [96-101]，常通过学习语言条件策略 [45, 102-106, 8]。我们像许多先前工作一样用模仿学习训练语言条件策略，但使用大规模多本体演示数据。沿用在机器人模仿学习中利用预训练语言嵌入 [107, 45, 108, 103, 40, 109-112] 与预训练视觉—语言模型 [113-115, 9] 的路线，实验中两种预训练都做了研究，具体遵循 RT-1 [8] 与 RT-2 [9] 的配方。

Fig. 2: RT-1-X and RT-2-X both take images and a text instruction as input and output discretized end-effector actions. RT-1-X is an architecture designed for robotics, with a FiLM [116] conditioned EfficientNet [117] and a Transformer [118]. RT-2-X builds on a VLM backbone by representing actions as another language, and training action text tokens together with vision-language data.

图 2：RT-1-X 与 RT-2-X 均以图像与文本指令为输入，输出离散化的末端执行器动作。RT-1-X 是面向机器人的架构，含 FiLM [116] 条件化的 EfficientNet [117] 与 Transformer [118]。RT-2-X 建立在 VLM 骨干上，把动作也表示成一种语言，并与视觉—语言数据一起训练动作文本 token。

III The Open X-Embodiment Repository

III Open X-Embodiment 仓库

We introduce the Open X-Embodiment Repository (robotics-transformer-x.github.io) - an open-source repository which includes large-scale data along with pre-trained model checkpoints for X-embodied robot learning research. More specifically, we provide and maintain the following open-source resources to the broader community:

我们推出 Open X-Embodiment 仓库（robotics-transformer-x.github.io）——一个开源仓库，包含面向跨本体机器人学习研究的 大规模数据 与 预训练模型检查点 。更具体地，我们向更广泛社区提供并维护以下开源资源：

22

22

21

21

III-A The Open X-Embodiment Dataset

III-A Open X-Embodiment 数据集

22

22

III-B Dataset Analysis

III-B 数据集分析

Fig. 1 analyzes the Open X-Embodiment Dataset. Fig. 1(a) shows the breakdown of datasets by robot embodiments, with the Franka robot being the most common. This is reflected in the number of distinct scenes (based on dataset metadata) per embodiment (Fig. 1(b)), where Franka dominates. Fig. 1(c) shows the breakdown of trajectories per embodiment. To further analyze the diversity, we use the language annotations present in our data. We use the PaLM language model [3] to extract objects and behaviors from the instructions. Fig. 1(d,e) show the diversity of skills and objects. While most skills belong to the pick-place family, the long tail of the dataset contains skills like “wiping” or “assembling”. Additionally, the data covers a range of household objects, from appliances to food items and utensils.

图 1 分析 Open X-Embodiment 数据集。图 1(a) 按机器人本体拆分数据集，Franka 最为常见；这反映在各本体的不同场景数上（基于数据集元数据，图 1(b)），Franka 占主导。图 1(c) 给出各本体轨迹数。为进一步分析多样性，我们利用数据中的语言标注，用 PaLM [3] 从指令中抽取物体与行为。图 1(d,e) 展示技能与物体的多样性：多数技能属 pick-place 家族，但长尾含「擦拭」「装配」等；数据还覆盖从电器到食材与餐具等家居物体。

IV RT-X Design

IV RT-X 设计

To evaluate how much X-embodiment training can improve the performance of learned policies on individual robots, we require models that have sufficient capacity to productively make use of such large and heterogeneous datasets. To that end, our experiments will build on two recently proposed Transformer-based robotic policies: RT-1 [8] and RT-2 [9]. We briefly summarize the design of these models in this section, and discuss how we adapted them to the X-embodiment setting in our experiments.

为评估跨本体训练能在多大程度上提升各台机器人上学得策略的表现，我们需要有足够容量、能有效利用如此大规模且异质数据的模型。因此实验建立在两种近期提出的基于 Transformer 的机器人策略之上：RT-1 [8] 与 RT-2 [9]。本节简要回顾其设计，并讨论我们如何把它们适配到跨本体设定。

IV-A Data format consolidation

IV-A 数据格式整合

x

x

IV-B Policy architectures

IV-B 策略架构

We consider two model architectures in our experiments: (1) RT-1 [8], an efficient Transformer-based architecture designed for robotic control, and (2) RT-2 [9] a large vision-language model co-fine-tuned to output robot actions as natural language tokens. Both models take in a visual input and natural language instruction describing the task, and output a tokenized action. For each model, the action is tokenized into 256 bins uniformly distributed along each of eight dimensions; one dimension for terminating the episode and seven dimensions for end-effector movement. Although both architectures are described in detail in their original papers [8, 9], we provide a short summary of each below:

实验考虑两种模型架构：（1）RT-1 [8]，面向机器人控制的高效 Transformer 架构；（2）RT-2 [9]，与输出为自然语言 token 的机器人动作共同微调的大规模视觉—语言—动作模型。两者都以视觉输入与描述任务的自然语言指令为输入，并输出经 tokenization 的动作。对每个模型，动作在八个维度上各均匀分成 256 个桶进行 tokenization：一维用于终止回合，七维用于末端执行器运动。尽管原论文 [8, 9] 有详细描述，下文仍给出简短摘要：

15

15

50\%

50\%

RT-2 [9] is a family of large vision-language- action models (VLAs) trained on Internet-scale vision and language data along with robotic control data. RT-2 casts the tokenized actions to text tokens, e.g., a possible action may be “1 128 91 241 5 101 127”. As such, any pretrained vision-language model (VLM [121, 122, 123]) can be finetuned for robotic control, thus leveraging the backbone of VLMs and transferring some of their generalization properties. In this work, we focus on the RT-2-PaLI-X variant [121] built on a backbone of a visual model, ViT [124], and a language model, UL2 [125], and pretrained primarily on the WebLI [121] dataset.

RT-2 [9] 是一族在互联网规模视觉与语言数据以及机器人控制数据上训练的大规模视觉—语言— 动作 模型（VLA）。RT-2 把经 tokenization 的动作写成文本 token，例如可能的动作是「1 128 91 241 5 101 127」。因此任何预训练视觉—语言模型（VLM [121-123]）都可微调用于机器人控制，从而借用 VLM 骨干并迁移其部分泛化性质。本文聚焦 RT-2-PaLI-X 变体 [121]：骨干为视觉模型 ViT [124] 与语言模型 UL2 [125]，主要在 WebLI [121] 上预训练。

IV-C Training and inference details

IV-C 训练与推理细节

Both models use a standard categorical cross-entropy objective over their output space (discrete buckets for RT-1 and all possible language tokens for RT-2).

两个模型都在输出空间上使用标准类别交叉熵目标（RT-1 为离散桶，RT-2 为全部可能的语言 token）。

9

9

At inference time, each model is run at the rate required for the robot (3-10 Hz), with RT-1 run locally and RT-2 hosted on a cloud service and queried over the network.

推理时，各模型以机器人所需速率运行（3-10 Hz）：RT-1 本地运行，RT-2 托管在云服务上经网络查询。

13\%

Evaluation Setting	Bridge	Bridge	RT-1 paper 6 skills
Evaluation Location	IRIS (Stanford)	RAIL Lab (UCB)	Google Robotic Lab
Robot Embodiment	WidowX	WidowX	Google Robot
Original Method	LCBC [95]	LCBC [95]	-
Original Method	$13\%$	$13\%$	-
RT-1	$40\%$	30%	92%
RT-1-X	$27\%$	$27\%$	$73\%$
RT-2-X (55B)	50%	30%	91%

13\%

评估设定	Bridge	Bridge	RT-1 论文 6 技能
评估地点	IRIS (Stanford)	RAIL Lab (UCB)	Google Robotic Lab
机器人本体	WidowX	WidowX	Google Robot
Original Method	LCBC [95]	LCBC [95]	-
Original Method	$13\%$	$13\%$	-
RT-1	$40\%$	30%	92%
RT-1-X	$27\%$	$27\%$	$73\%$
RT-2-X (55B)	50%	30%	91%

V Experimental Results

V 实验结果

Our experiments answer three questions about the effect of X-embodiment training: (1) Can policies trained on our X-embodiment dataset effectively enable positive transfer, such that co-training on data collected on multiple robots improves performance on the training task? (2) Does co-training models on data from multiple platforms and tasks improve generalization to new, unseen tasks? (3) What is the influence of different design dimensions, such as model size, model architecture or dataset composition, on performance and generalization capabilities of the resulting policy? To answer these questions we conduct the total number of 3600 evaluation trials across 6 different robots.

实验回答关于跨本体训练效应的三个问题：（1）在我们的跨本体数据上训练的策略能否有效实现正向迁移，使多机器人数据上的共训练提升训练任务表现？（2）在多平台、多任务数据上共训练，能否改进对新未见任务的泛化？（3）模型规模、架构或数据组成等不同设计维度，对所得策略的表现与泛化能力有何影响？为回答这些问题，我们在 6 种不同机器人上共进行 3600 次评估试验。

V-A In-distribution performance across different embodiments

V-A 不同本体上的分布内表现

To assess the ability of RT-X models to learn from X-embodiment data, we evaluate performance on in-distribution tasks. We split our evaluation into two types: evaluation on domains that have small-scale datasets (Fig. 3), where we would expect transfer from larger datasets to significantly improve performance, and evaluation on domains that have large-scale datasets (Table I), where we expect further improvement to be more challenging. Note that we use the same robotics data training mixture (defined in Sec. IV-C) for all the evaluations presented in this section. For small-scale dataset experiments, we use Kitchen Manipulation [128], Cable Routing [129], NYU Door Opening [130], AUTOLab UR5 [132], and Robot Play [134]. We use the same evaluation and robot embodiment as in the respective publications. For large-scale dataset experiments, we consider Bridge [95] and RT-1 [8] for in-distribution evaluation and use their respective robots: WidowX and Google Robot.

为评估 RT-X 模型从跨本体数据学习的能力，我们在分布内任务上评测，并分成两类：小规模数据集域（图 3）——预期从更大数据集迁移会显著提升；大规模数据集域（表 I）——进一步提升更具挑战。注意本节所有评估使用同一机器人数据 训练 混合（第 IV-C 节定义）。小规模数据实验使用 Kitchen Manipulation [128]、Cable Routing [129]、NYU Door Opening [130]、AUTOLab UR5 [132] 与 Robot Play [134]，评估与机器人本体与各自原论文一致。大规模数据实验考虑 Bridge [95] 与 RT-1 [8] 做分布内评估，分别使用 WidowX 与 Google Robot。

For each small dataset domain, we compare the performance of the RT-1-X model, and for each large dataset we consider both the RT-1-X and RT-2-X models. For all experiments, the models are co-trained on the full X-embodiment dataset. Throughout this evaluation we compare with two baseline models: (1) The model developed by the creators of the dataset trained only on that respective dataset. This constitutes a reasonable baseline insofar as it can be expected that the model has been optimized to work well with the associated data; we refer to this baseline model as the Original Method model. (2) An RT-1 model trained on the dataset in isolation; this baseline allows us to assess whether the RT-X model architectures have enough capacity to represent policies for multiple different robot platforms simultaneously, and whether co-training on multi-embodiment data leads to higher performance.

对每个小数据域，我们比较 RT-1-X；对每个大数据域，同时考虑 RT-1-X 与 RT-2-X。全部实验中模型都在完整跨本体数据上共训练。整段评估与两个基线对比：（1）由数据集创建者开发、只在该数据集上训练的模型——可预期其已针对相关数据优化，称为 Original Method ；（2）只在该数据集上单独训练的 RT-1——用于评估 RT-X 架构是否有足够容量同时表示多机器人平台策略，以及多本体共训练是否带来更高表现。

Small-scale dataset domains (Fig. 3). RT-1-X outperforms Original Method trained on each of the robot-specific datasets on 4 of the 5 datasets, with a large average improvement, demonstrating domains with limited data benefit substantially from co-training on X-embodiment data.

小规模数据集域 （图 3）。RT-1-X 在 5 个数据集中的 4 个上优于只在各机器人专用数据上训练的 Original Method，平均提升很大，表明数据有限的域从跨本体共训练中显著受益。

27.3\%

Row	Model	Size	History Length	Dataset	Co-Trained w/ Web	Initial Checkpoint	Emergent Skills Evaluation	RT-2 Generalization Evaluation
(1)	RT-2	55B	none	Google Robot action	Yes	Web-pretrained	$27.3\%$	62%
(2)	RT-2-X	55B	none	Robotics data	Yes	Web-pretrained	75.8%	61%
(3)	RT-2-X	55B	none	Robotics data except Bridge	Yes	Web-pretrained	$42.8\%$	$54\%$
(4)	RT-2-X	5B	2	Robotics data	Yes	Web-pretrained	$44.4\%$	$52\%$
(5)	RT-2-X	5B	none	Robotics data	Yes	Web-pretrained	$14.5\%$	$30\%$
(6)	RT-2-X	5B	2	Robotics data	No	From scratch	$0\%$	$1\%$
(7)	RT-2-X	5B	2	Robotics data	No	Web-pretrained	$48.7\%$	$47\%$

27.3\%

行	模型	规模	历史长度	数据集	与网络共训练	初始检查点	涌现技能评估	RT-2 泛化评估
(1)	RT-2	55B	无	Google Robot 动作	是	网络预训练	$27.3\%$	62%
(2)	RT-2-X	55B	无	机器人数据	是	网络预训练	75.8%	61%
(3)	RT-2-X	55B	无	机器人数据（除 Bridge）	是	网络预训练	$42.8\%$	$54\%$
(4)	RT-2-X	5B	2	机器人数据	是	网络预训练	$44.4\%$	$52\%$
(5)	RT-2-X	5B	无	机器人数据	是	网络预训练	$14.5\%$	$30\%$
(6)	RT-2-X	5B	2	机器人数据	否	从零	$0\%$	$1\%$
(7)	RT-2-X	5B	2	机器人数据	否	网络预训练	$48.7\%$	$47\%$

Large-scale dataset domains (Table I). In the large-dataset setting, the RT-1-X model does not outperform the RT-1 baseline trained on only the embodiment-specific dataset, which indicates underfitting for that model class. However, the larger RT-2-X model outperforms both the Original Method and RT-1 suggesting that X-robot training can improve performance in the data-rich domains, but only when utilizing a sufficiently high-capacity architecture.

大规模数据集域 （表 I）。在大数据设定下，RT-1-X 并未超过只在本体专用数据上训练的 RT-1 基线，表明该模型类存在欠拟合。然而更大的 RT-2-X 同时超过 Original Method 与 RT-1，说明跨机器人训练也能提升数据充裕域的表现，但前提是使用足够高容量的架构。

V-B Improved generalization to out-of-distribution settings

V-B 改进对分布外设定的泛化

We now examine how X-embodiment training can enable better generalization to out-of-distribution settings and more complex and novel instructions. These experiments focus on the high-data domains, and use the RT-2-X model.

接着考察跨本体训练如何改进对分布外设定以及更复杂、更新颖指令的泛化。这些实验聚焦高数据域，并使用 RT-2-X。

Unseen objects, backgrounds and environments. We first conduct the same evaluation of generalization properties as proposed in [9], testing for the ability to manipulate unseen objects in unseen environments and against unseen backgrounds. We find that RT-2 and RT-2-X perform roughly on par (Table II, rows (1) and (2), last column). This is not unexpected, since RT-2 already generalizes well (see [9]) along these dimensions due to its VLM backbone.

未见物体、背景与环境 。 我们首先按 [9] 提出的方式评估泛化：在未见环境、未见背景下操作未见物体的能力。发现 RT-2 与 RT-2-X 大致持平（表 II，行 (1)(2) 最后一列）。这并不意外：凭借 VLM 骨干，RT-2 在这些维度上已泛化良好（见 [9]）。

\sim 3\times

3\times

Our next ablation involves removing the Bridge dataset from RT-2-X training: Row (3) shows the results for RT-2-X that includes all data used for RT-2-X except the Bridge dataset. This variation significantly reduces performance on the hold-out tasks, suggesting that transfer from the WidowX data may indeed be responsible for the additional skills that can be performed by RT-2-X with the Google Robot.

下一步消融是从 RT-2-X 训练中去掉 Bridge：行 (3) 是包含 RT-2-X 全部数据、唯独不含 Bridge 的变体。该变体在保留任务上表现显著下降，暗示来自 WidowX 数据的迁移可能确实是 Google Robot 上 RT-2-X 额外技能的来源。

V-C Design decisions

V-C 设计决策

55B

最后，我们对表现最好的 RT-2-X 做消融，衡量不同设计决策对泛化能力的影响，见表 II。加入短图像历史显著提升泛化（行 (4) 对行 (5)）。与 RT-2 论文 [9] 结论类似，基于网络的预训练对大模型取得高性能至关重要（行 (4) 对行 (6)）。我们也注意到 55B 模型在涌现技能上的成功率显著高于 5B（行 (2) 对行 (4)），表明更高容量带来跨机器人数据集更强的迁移。与先前 RT-2 发现不同，共微调与微调在涌现技能与泛化评估上表现相近（行 (4) 对行 (7)），我们归因于 RT-2-X 所用机器人数据远比先前机器人数据集更多样。

VI Discussion, Future Work, and Open Problems

VI 讨论、未来工作与开放问题

22

21

Fig. 4: To assess transfer between embodiments, we evaluate the RT-2-X model on out-of-distribution skills. These skills are in the Bridge dataset, but not in the Google Robot dataset (the embodiment they are evaluated on).

图 4：为评估 本体之间 的迁移，我们在分布外技能上评估 RT-2-X。这些技能出现在 Bridge 数据中，但不在 Google Robot 数据中（即它们被评估的本体）。

While RT-X demonstrates a step towards a X-embodied robot generalist, many more steps are needed to make this future a reality. Our experiments do not consider robots with very different sensing and actuation modalities. They do not study generalization to new robots, and provide a decision criterion for when positive transfer does or does not happen. Studying these questions is an important future work direction. This work serves not only as an example that X-robot learning is feasible and practical, but also provide the tools to advance research in this direction in the future.

尽管 RT-X 朝跨本体机器人通才迈出一步，要实现这一未来仍需许多步骤。实验未考虑传感与驱动模态差异很大的机器人；也未研究向新机器人的泛化，也未给出正向迁移何时发生、何时不发生的判定准则。研究这些问题是重要的未来方向。本文不仅作为跨机器人学习可行且实用的示例，也为未来推进该方向的研究提供了工具。

References

参考文献

[41] S. Bahl, A. Gupta, and D. Pathak, “Human-to-robot imitation in the wild,” Robotics: Science and Systems (RSS) , 2022. [42] M. Ding, Y. Xu, Z. Chen, D. D. Cox, P. Luo, J. B. Tenenbaum, and C. Gan, “Embodied concept learner: Self-supervised learning of concepts and mapping through instruction following,” in Conference on Robot Learning . PMLR, 2023, pp. 1743-1754. [43] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak, “Affordances from human videos as a versatile representation for robotics,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2023, pp. 13 778-13 790. [44] P. Sermanet, K. Xu, and S. Levine, “Unsupervised perceptual rewards for imitation learning,” arXiv preprint arXiv:1612.06699 , 2016. [45] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg, “Concept2Robot: Learning manipulation concepts from instructions and human demonstrations,” in Proceedings of Robotics: Science and Systems (RSS) , 2020. [46] A. S. Chen, S. Nair, and C. Finn, “Learning generalizable robotic reward functions from “in-the-wild” human videos,” arXiv preprint arXiv:2103.16817 , 2021. [47] S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang, “Graph inverse reinforcement learning from diverse videos,” in Conference on Robot Learning . PMLR, 2023, pp. 55-66. [48] M. Alakuijala, G. Dulac-Arnold, J. Mairal, J. Ponce, and C. Schmid, “Learning reward functions for robotic manipulation by observing humans,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 5006-5012. [49] Y. Zhou, Y. Aytar, and K. Bousmalis, “Manipulator-independent representations for visual imitation,” 2021. [50] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar, “Mimicplay: Long-horizon imitation learning by watching human play,” in Conference on Robot Learning , 2023.

[111] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “ChatGPT for robotics: Design principles and model abilities,” Microsoft Auton. Syst. Robot. Res , vol. 2, p. 20, 2023. [112] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, “VoxPoser: Composable 3d value maps for robotic manipulation with language models,” arXiv preprint arXiv:2307.05973 , 2023. [113] M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in Conference on Robot Learning . PMLR, 2022, pp. 894-906. [114] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, B. Zitkovich, F. Xia, C. Finn et al. , “Open-world object manipulation using pre-trained vision-language models,” arXiv preprint arXiv:2303.00905 , 2023. [115] Y. Mu, Q. Zhang, M. Hu, W. Wang, M. Ding, J. Jin, B. Wang, J. Dai, Y. Qiao, and P. Luo, “EmbodiedGPT: Vision-language pre-training via embodied chain of thought,” arXiv preprint arXiv:2305.15021 , 2023. [116] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” 2017. [117] M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in International conference on machine learning . PMLR, 2019, pp. 6105-6114. [118] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017. [119] S. Ramos, S. Girgin, L. Hussenot, D. Vincent, H. Yakubovich, D. Toyama, A. Gergely, P. Stanczyk, R. Marinier, J. Harmsen, O. Pietquin, and N. Momchev, “RLDS: an ecosystem to generate, share and use datasets in reinforcement learning,” 2021. [120] D. Cer, Y. Yang, S. yi Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, Y.-H. Sung, B. Strope, and R. Kurzweil, “Universal sentence encoder,” 2018.

[131] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu, “Viola: Imitation learning for vision-based manipulation with object proposal priors,” 2023. [132] L. Y. Chen, S. Adebola, and K. Goldberg, “Berkeley UR5 demonstration dataset,” https://sites.google.com/view/berkeley-ur5/home . [133] G. Zhou, V. Dean, M. K. Srirama, A. Rajeswaran, J. Pari, K. Hatch, A. Jain, T. Yu, P. Abbeel, L. Pinto, C. Finn, and A. Gupta, “Train offline, test online: A real robot learning benchmark,” 2023. [134] “Task-agnostic real world robot play,” https://www.kaggle.com/datasets/oiermees/taco-robot .