OpenVLA: An Open-Source Vision-Language-Action Model

OpenVLA: An Open-Source Vision-Language-Action Model

OpenVLA：一个开源的视觉-语言-动作模型

Moo Jin Kim Karl Pertsch Siddharth Karamcheti Ted Xiao Ashwin Balakrishna Suraj Nair Rafael Rafailov Ethan Foster Grace Lam Pannag Sanketi Quan Vuong Thomas Kollar Benjamin Burchfiel Russ Tedrake Dorsa Sadigh Sergey Levine Percy Liang Chelsea Finn https://openvla.github.io

Moo Jin Kim*、Karl Pertsch*、Siddharth Karamcheti*、Ted Xiao、Ashwin Balakrishna、Suraj Nair、Rafael Rafailov、Ethan Foster、Grace Lam、Pannag Sanketi、Quan Vuong、Thomas Kollar、Benjamin Burchfiel、Russ Tedrake、Dorsa Sadigh、Sergey Levine、Percy Liang、Chelsea Finn https://openvla.github.io

*: denotes equal contribution. Correspondence to: moojink@stanford.edu, pertsch@berkeley.edu, skaramcheti@stanford.edu. 1 Stanford University, 2 UC Berkeley, 3 Toyota Research Institute, 4 Google Deepmind, 5 Physical Intelligence, 6 MIT. †Work done in part while at Google Deepmind.

*：同等贡献。通讯地址：moojink@stanford.edu、pertsch@berkeley.edu、skaramcheti@stanford.edu。1 斯坦福大学，2 加州大学伯克利分校，3 丰田研究院，4 Google DeepMind，5 Physical Intelligence，6 麻省理工学院。†部分工作在 Google DeepMind 期间完成。

Abstract

摘要

Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, and outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4% We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets.

在互联网规模的视觉-语言数据和多样化机器人示教数据上联合预训练的大型策略，有可能改变我们教机器人新技能的方式：与其从零训练新行为，不如微调这类视觉-语言-动作（VLA）模型，得到鲁棒、可泛化的视觉运动控制策略。然而，VLA 在机器人领域的广泛应用一直面临困难：1）现有 VLA 大多是封闭的，公众无法获取；2）先前工作没有探索如何为新任务高效微调 VLA，而这正是推广应用的关键一环。针对这些挑战，我们推出 OpenVLA：一个 70 亿参数的开源 VLA，在 97 万条多样化的真实世界机器人示教上训练。OpenVLA 以 Llama 2 语言模型为基础，结合一个融合了 DINOv2 与 SigLIP 预训练特征的视觉编码器。得益于更丰富的数据多样性和新的模型组件，OpenVLA 在通用操作上表现出色：在 29 个任务和多种机器人本体上，其绝对任务成功率比 RT-2-X（550 亿参数）等封闭模型高 16.5%，而参数量只有后者的七分之一。我们进一步表明，OpenVLA 可以被有效地微调到新的场景，在涉及多个物体的多任务环境中泛化效果尤其好，语言落地（language grounding）能力很强，并且比 Diffusion Policy 等表达力强的从零模仿学习方法高出 20.4%。我们还探讨了计算效率：作为一项独立贡献，我们表明 OpenVLA 可以借助现代低秩适配方法在消费级 GPU 上微调，并通过量化高效部署，而不损失下游成功率。最后，我们发布了模型检查点、微调用的 notebook，以及我们的 PyTorch 代码库；该代码库内置支持在 Open X-Embodiment 数据集上大规模训练 VLA。

Figure 1: We present OpenVLA, a 7B-parameter open-source vision-language-action model (VLA), trained on 970k robot episodes from the Open X-Embodiment dataset [1]. OpenVLA sets a new state of the art for generalist robot manipulation policies. It supports controlling multiple robots out of the box and can be quickly adapted to new robot domains via parameter-efficient fine-tuning. The OpenVLA checkpoints and PyTorch training pipeline are fully open-source and models can be downloaded and fine-tuned from HuggingFace.

图 1：我们推出 OpenVLA，一个 70 亿参数的开源视觉-语言-动作模型（VLA），在 Open X-Embodiment 数据集 [1] 的 97 万个机器人回合上训练。OpenVLA 为通用机器人操作策略树立了新的最高水平。它开箱即可控制多种机器人，并能通过参数高效微调快速适配新的机器人领域。OpenVLA 的检查点和 PyTorch 训练流水线完全开源，模型可以从 HuggingFace 下载并微调。

1 Introduction

1 引言

A key weakness of learned policies for robotic manipulation is their inability to generalize beyond their training data: while existing policies trained for individual skills or language instructions have the capacity to extrapolate behaviors to new initial conditions such as object positions or lighting [2, 3], they lack robustness to scene distractors or novel objects [4, 5] and struggle to execute unseen task instructions [6, 7]. Yet beyond robotics, existing foundation models for vision and language such as CLIP [8], SigLIP [9], and Llama 2 [10] are capable of these types of generalization and more, stemming from the priors captured by their Internet-scale pretraining datasets. While reproducing this scale of pretraining for robotics is still an open challenge — even the largest robot manipulation datasets [1, 11] only have 100K to 1M examples - this imbalance suggests an opportunity: using existing foundation models for vision and language as a core building block for training robotic policies that can generalize to objects, scenes, and tasks beyond their training data.

用于机器人操作的学习型策略有一个关键弱点：无法泛化到训练数据之外。现有的、针对单个技能或语言指令训练的策略，能够把行为外推到新的初始条件，例如物体位置或光照的变化 [2, 3]，但它们对场景干扰物或新物体缺乏鲁棒性 [4, 5]，也难以执行没见过的任务指令 [6, 7]。而在机器人领域之外，CLIP [8]、SigLIP [9]、Llama 2 [10] 等现有的视觉与语言基础模型具备这类泛化能力乃至更多，这源自它们互联网规模的预训练数据所蕴含的先验。在机器人领域复现这种规模的预训练仍是一个未解决的难题——即便最大的机器人操作数据集 [1, 11] 也只有 10 万到 100 万个样本——但这种不平衡也提示了一个机会： 把现有的视觉与语言基础模型作为核心构件 ，来训练能够泛化到训练数据之外的物体、场景和任务的机器人策略。

Towards this goal, existing work has explored integrating pretrained language and vision-language models for robotic representation learning [12, 13, 14] and as a component in modular systems for task planning and execution [15, 16]. More recently, they have been used for directly learning vision-language-action models [VLAs; 7, 1, 17, 18] for control. VLAs provide a direct instantiation of using pretrained vision-and-language foundation models for robotics, directly fine-tuning visually-conditioned language models (VLMs) such as PaLI [19, 20] to generate robot control actions. By building off of strong foundation models trained on Internet-scale data, VLAs such as RT-2 [7] demonstrate impressive robustness results, as well as an ability to generalize to novel objects and tasks, setting a new standard for generalist robot policies. Yet, there are two key reasons preventing the widespread use of existing VLAs: 1) current models [7, 1, 17, 18] are closed, with limited visibility into model architecture, training procedures, and data mixture, and 2) existing works do not provide best practices for deploying and adapting VLAs to new robots, environments, and tasks — especially on commodity hardware (e.g., consumer-grade GPUs). We argue that to develop a rich foundation for future research and development, robotics needs open-source, generalist VLAs that support effective fine-tuning and adaptation, akin to the existing ecosystem around open-source language models [21, 22, 23, 24].

为此，已有工作探索了把预训练的语言模型和视觉-语言模型用于机器人表征学习 [12, 13, 14]，以及作为模块化系统中负责任务规划和执行的组件 [15, 16]。最近，它们又被用来直接学习用于控制的视觉-语言-动作模型 [VLAs; 7, 1, 17, 18]。VLA 是把预训练视觉-语言基础模型用于机器人的一种直接实现：直接微调 PaLI [19, 20] 等视觉条件语言模型（VLM），让它生成机器人控制动作。借助在互联网规模数据上训练的强大基础模型，RT-2 [7] 等 VLA 表现出令人印象深刻的鲁棒性，并能泛化到新的物体和任务，为通用机器人策略树立了新标准。然而，有两个关键原因阻碍了现有 VLA 的广泛使用：1）当前的模型 [7, 1, 17, 18] 是 封闭 的，模型架构、训练流程和数据配比都不透明；2）现有工作没有给出把 VLA 部署并适配 到新机器人、新环境和新任务的最佳实践——尤其是在普通硬件（例如消费级 GPU）上。我们认为，要为未来的研究与开发打下丰厚的基础，机器人领域需要支持有效微调与适配的开源通用 VLA，就像围绕开源语言模型已经形成的生态 [21, 22, 23, 24] 那样。

To this end, we introduce OpenVLA, a 7B-parameter open-source VLA that establishes a new state of the art for generalist robot manipulation policies.¹ OpenVLA consists of a pretrained visually-conditioned language model backbone that captures visual features at multiple granularities, fine-tuned on a large, diverse dataset of 970k robot manipulation trajectories from the Open-X Embodiment [1] dataset — a dataset that spans a wide range of robot embodiments, tasks, and scenes. As a product of increased data diversity and new model components, OpenVLA outperforms the 55B-parameter RT-2-X model [7, 1], the prior state-of-the-art VLA, by 16.5% absolute success rate across 29 evaluation tasks on the WidowX and Google Robot embodiments. We additionally investigate efficient fine-tuning strategies for VLAs, a new contribution not explored in prior work, across 7 diverse manipulation tasks spanning behaviors from object pick-and-place to cleaning a table. We find that fine-tuned OpenVLA policies clearly outperform fine-tuned pretrained policies such as Octo [5]. Compared to from-scratch imitation learning with diffusion policies [3], fine-tuned OpenVLA shows substantial improvement on tasks involving grounding language to behavior in multi-task settings with multiple objects. Following these results, we are the first to demonstrate the effectiveness of compute-efficient fine-tuning methods leveraging low-rank adaptation [LoRA; 26] and model quantization [27] to facilitate adapting OpenVLA models on consumer-grade GPUs instead of large server nodes without compromising performance. As a final contribution, we open-source all models, deployment and fine-tuning notebooks, and the OpenVLA codebase for training VLAs at scale, with the hope that these resources enable future work exploring and adapting VLAs for robotics.

为此，我们推出 OpenVLA，一个 70 亿参数的开源 VLA，它为通用机器人操作策略树立了新的最高水平。¹ OpenVLA 由一个预训练的视觉条件语言模型骨干组成，能够捕捉多种粒度的视觉特征，并在来自 Open-X Embodiment [1] 数据集的 97 万条大规模、多样化机器人操作轨迹上微调；该数据集涵盖了广泛的机器人本体、任务和场景。得益于更高的数据多样性和新的模型组件，OpenVLA 在 WidowX 和 Google 机器人两种本体的 29 个评测任务上，绝对成功率比此前最先进的 VLA——550 亿参数的 RT-2-X 模型 [7, 1]——高 16.5%。我们还研究了 VLA 的高效微调策略 ，这是先前工作未曾探索的新贡献，覆盖 7 个多样的操作任务，行为从物体抓放到擦桌子不等。我们发现，微调后的 OpenVLA 策略明显优于微调后的 Octo [5] 等预训练策略。与使用扩散策略 [3] 的从零模仿学习相比，微调后的 OpenVLA 在多物体、多任务、需要把语言落实到行为的任务上有显著提升。在这些结果之后，我们首次证明了利用低秩适配 [LoRA; 26] 和模型量化 [27] 的计算高效微调方法是有效的，使 OpenVLA 模型可以在消费级 GPU 而不是大型服务器节点上完成适配，且不牺牲性能。最后一项贡献是，我们开源了全部模型、部署与微调 notebook，以及用于大规模训练 VLA 的 OpenVLA 代码库，希望这些资源能推动未来探索和适配机器人 VLA 的工作。

¹ OpenVLA uses multiple pretrained model components: SigLIP [9] and DinoV2 [25] vision encoders and a Llama 2 [10] language model backbone. For all three models, weights are open, but not their training data or code. We release training data, code and model weights for reproducing OpenVLA on top of these components.

¹ OpenVLA 使用了多个预训练模型组件：SigLIP [9] 和 DinoV2 [25] 视觉编码器，以及 Llama 2 [10] 语言模型骨干。这三个模型的权重都是开放的，但训练数据和代码没有公开。我们发布了在这些组件之上复现 OpenVLA 所需的训练数据、代码和模型权重。

2 Related Work

2 相关工作

Visually-Conditioned Language Models Visually-conditioned language models (VLMs), which are trained on Internet-scale data to generate natural language from input image(s) and language prompts, have been adopted for myriad applications from visual question answering [28, 29, 30, 31] to object localization [32, 33]. One of the key advances fueling recent VLMs are model architectures that bridge features from pretrained vision encoders [8, 9, 25] with pretrained language models [10, 23, 34, 35, 36], directly building on advances in both computer vision and natural language modelling to create powerful multimodal models. While early work explored various architectures for cross-attending between vision and language features [37, 38, 39, 40, 41], new open-source VLMs [42, 43, 20, 44] have converged on a simpler “patch-as-token” approach, in which patch features from pretrained visual transformers are treated as tokens, and are then projected into the input space of a language model. This simplicity makes it easy to repurpose existing tools for training language models at scale for VLM training. We employ these tools in our work to scale VLA training, and specifically use VLMs from Karamcheti et al. [44] as our pretrained backbone, as they are trained from multi-resolution visual features, fusing low-level spatial information from DINOv2 [25] with higher-level semantics from SigLIP [9] to aid in visual generalization.

视觉条件语言模型 视觉条件语言模型（VLM）在互联网规模的数据上训练，根据输入图像和语言提示生成自然语言，已被用于从视觉问答 [28, 29, 30, 31] 到物体定位 [32, 33] 的众多应用。推动近期 VLM 发展的关键进展之一，是把预训练视觉编码器 [8, 9, 25] 的特征与预训练语言模型 [10, 23, 34, 35, 36] 连接起来的模型架构，它直接建立在计算机视觉和自然语言建模两方面的进展之上，造就了强大的多模态模型。早期工作探索了多种在视觉与语言特征之间做交叉注意力的架构 [37, 38, 39, 40, 41]，而新的开源 VLM [42, 43, 20, 44] 已收敛到一种更简单的“图块即词元（patch-as-token）”方法：把预训练视觉 Transformer 的图块特征当作词元，再投影到语言模型的输入空间。这种简单性使得大规模训练语言模型的现有工具很容易被改用于 VLM 训练。我们在工作中借助这些工具来扩展 VLA 训练，具体采用 Karamcheti 等人 [44] 的 VLM 作为预训练骨干，因为它们基于 多分辨率视觉特征 训练，融合了 DINOv2 [25] 的低层空间信息和 SigLIP [9] 的高层语义，有助于视觉泛化。

Generalist Robot Policies A recent trend in robotics works towards training multi-task “generalist” robot policies [45, 46, 47, 6, 2, 48, 49] on large diverse robot datasets [50, 51, 45, 52, 53, 54, 55, 6, 2, 56, 49, 11, 1], spanning many different robot embodiments [57, 53, 58, 59, 60, 61, 62, 63, 64, 65, 1, 5, 66]. Notably, Octo [5] trains a generalist policy that can control multiple robots out-of-the-box and allows for flexible fine-tuning to new robot setups. A key difference between these approaches and OpenVLA is the model architecture. Prior works like Octo typically compose pretrained components such as language embeddings or visual encoders with additional model components initialized from scratch [6, 2, 5], learning to “stitch” them together during the course of policy training. Unlike these works, OpenVLA adopts a more end-to-end approach, directly fine-tuning VLMs to generate robot actions by treating them as tokens in the language model vocabulary. Our experimental evaluation shows that this simple yet scalable pipeline substantially boosts performance and generalization ability over prior generalist policies.

通用机器人策略 机器人领域近来的一个趋势，是在大规模、多样化的机器人数据集 [50, 51, 45, 52, 53, 54, 55, 6, 2, 56, 49, 11, 1] 上训练多任务“通用”机器人策略 [45, 46, 47, 6, 2, 48, 49]，覆盖许多不同的机器人本体 [57, 53, 58, 59, 60, 61, 62, 63, 64, 65, 1, 5, 66]。值得注意的是，Octo [5] 训练了一个开箱即可控制多种机器人、并能灵活微调到新机器人配置的通用策略。这些方法与 OpenVLA 的一个关键区别在于模型架构。Octo 等先前工作通常把语言嵌入或视觉编码器等预训练组件，与额外的从零初始化的模型组件组合起来 [6, 2, 5]，在策略训练过程中学习把它们“缝合”在一起。与这些工作不同，OpenVLA 采用更端到端的方式：把机器人动作当作语言模型词表中的词元，直接微调 VLM 来生成动作。我们的实验评估表明，这条简单而可扩展的流水线相比先前的通用策略大幅提升了性能和泛化能力。

Vision-Language-Action Models A number of works have explored the use of VLMs for robotics, e.g., for visual state representations [12, 13], object detection [67], high-level planning [16], and for providing a feedback signal [68, 69, 70, 71]. Others integrate VLMs directly into end-to-end visuomotor manipulation policies [14, 15], but incorporate significant structure into the policy architecture or require calibrated cameras, which limits their applicability. A number of recent works have explored similar recipes to ours and directly fine-tuned large pretrained VLMs for predicting robot actions [7, 1, 72, 73, 17, 18, 74]. Such models are often referred to as vision-language-action models (VLAs), since they fuse robot control actions directly into VLM backbones. This has three key benefits: (1) it performs alignment of pretrained vision and language components on a large, Internet-scale vision-language dataset, (2) the use of a generic architecture, not custom-made for robot control, allows us to leverage the scalable infrastructure underlying modern VLM training [75, 76, 77] and scale to training billion-parameter policies with minimal code modifications, and (3) it provides a direct pathway for robotics to benefit from the rapid improvements in VLMs. Existing works on VLAs either focus on training and evaluating in single robot or simulated setups [72, 73, 78, 74] and thus lack generality, or are closed and do not support efficient fine-tuning to new robot setups [7, 1, 17, 18].

视觉-语言-动作模型 许多工作探索了 VLM 在机器人中的用途，例如用于视觉状态表征 [12, 13]、物体检测 [67]、高层规划 [16]，以及提供反馈信号 [68, 69, 70, 71]。另一些工作把 VLM 直接整合进端到端的视觉运动操作策略 [14, 15]，但在策略架构中引入了大量结构，或者需要标定过的相机，这限制了它们的适用范围。近来有不少工作探索了与我们类似的做法，直接微调大型预训练 VLM 来预测机器人动作 [7, 1, 72, 73, 17, 18, 74]。这类模型通常被称为视觉-语言-动作模型（VLA），因为它们把机器人控制动作直接融合进 VLM 骨干。这样做有三个关键好处：（1）预训练的视觉和语言组件已在大规模、互联网级的视觉-语言数据集上完成对齐；（2）使用并非为机器人控制专门定制的通用架构，使我们能利用现代 VLM 训练背后的可扩展基础设施 [75, 76, 77]，只需极少的代码修改就能扩展到训练数十亿参数的策略；（3）它为机器人领域直接受益于 VLM 的快速进步提供了一条通路。现有的 VLA 工作要么只在单一机器人或仿真环境中训练和评测 [72, 73, 78, 74]，因而缺乏通用性；要么是封闭的，不支持高效微调到新的机器人配置 [7, 1, 17, 18]。

Most closely related, RT-2-X [1] trains a 55B-parameter VLA policy on the Open X-Embodiment dataset and demonstrates state-of-the-art generalist manipulation policy performance. However, our work differs from RT-2-X in multiple important aspects: (1) by combining a strong open VLM backbone with a richer robot pretraining dataset, OpenVLA outperforms RT-2-X in our experiments while being an order of magnitude smaller; (2) we thoroughly investigate fine-tuning of OpenVLA models to new target setups, while RT-2-X does not investigate the fine-tuning setting; (3) we are the first to demonstrate the effectiveness of modern parameter-efficient fine-tuning and quantization approaches for VLAs; and (4) OpenVLA is the first generalist VLA that is open-source and thus supports future research on VLA training, data mixtures, objectives, and inference.

与我们最相近的是 RT-2-X [1]，它在 Open X-Embodiment 数据集上训练了一个 550 亿参数的 VLA 策略，展示了最先进的通用操作策略性能。不过，我们的工作在几个重要方面与 RT-2-X 不同：（1）通过把强大的开放 VLM 骨干与更丰富的机器人预训练数据集结合，OpenVLA 在我们的实验中优于 RT-2-X，而规模小了一个数量级；（2）我们深入研究了把 OpenVLA 模型微调到新目标配置的问题，而 RT-2-X 没有研究微调场景；（3）我们首次证明了现代参数高效微调和量化方法对 VLA 的有效性；（4）OpenVLA 是第一个开源的通用 VLA，因此能支持未来关于 VLA 训练、数据配比、目标函数和推理的研究。

3 The OpenVLA Model

3 OpenVLA 模型

We introduce the OpenVLA model, a 7B-parameter vision-language-action model (VLA) trained on 970k robot demonstrations from the Open X-Embodiment dataset [1]. There are many, largely unexplored, questions around best practices for developing VLA models, e.g., what are the best model backbones, datasets, and hyperparameters to use for training. Below, we detail our approach for developing OpenVLA and summarize our key learnings. Concretely, we first provide a brief overview of modern VLMs, which form the backbone of OpenVLA (Section 3.1); then describe our basic training recipe and dataset (Section 3.2 and Section 3.3); discuss key design decisions (Section 3.4); and provide details of the used infrastructure for training and inference (Section 3.5).

我们介绍 OpenVLA 模型：一个在 Open X-Embodiment 数据集 [1] 的 97 万条机器人示教上训练的 70 亿参数视觉-语言-动作模型（VLA）。关于开发 VLA 模型的最佳实践，还有许多基本未被探索的问题，例如训练时最好用什么模型骨干、数据集和超参数。下面我们详细介绍开发 OpenVLA 的方法，并总结关键经验。具体来说，我们先简要概述构成 OpenVLA 骨干的现代 VLM（第 3.1 节），然后介绍基本训练方案和数据集（第 3.2 节和第 3.3 节），讨论关键的设计决策（第 3.4 节），最后给出训练与推理所用基础设施的细节（第 3.5 节）。

Figure 2: OpenVLA model architecture. Given an image observation and a language instruction, the model predicts 7-dimensional robot control actions. The architecture consists of three key components: (1) a vision encoder that concatenates Dino V2 [25] and SigLIP [79] features, (2) a projector that maps visual features to the language embedding space, and (3) the LLM backbone, a Llama 2 7B-parameter large language model [10].

图 2： OpenVLA 模型架构 。给定一张图像观测和一条语言指令，模型预测 7 维的机器人控制动作。该架构由三个关键组件组成：（1）一个 视觉编码器 ，把 Dino V2 [25] 和 SigLIP [79] 的特征拼接起来；（2）一个 投影器 ，把视觉特征映射到语言嵌入空间；（3） LLM 骨干 ，即 70 亿参数的大语言模型 Llama 2 [10]。

3.1 Preliminaries: Vision-Language Models

3.1 预备知识：视觉-语言模型

The architecture of most recent VLMs [42, 43, 20, 44] consists of three main parts (see Fig. 2): (1) a visual encoder that maps image inputs to a number of “image patch embeddings”, (2) a projector that takes the output embeddings of the visual encoder and maps them into the input space of a language model, and (3) a large language model (LLM) backbone . During VLM training, the model is trained end-to-end with a next text token prediction objective on paired or interleaved vision and language data curated from various Internet sources.

近期大多数 VLM [42, 43, 20, 44] 的架构由三个主要部分组成（见图 2）：（1）一个 视觉编码器 ，把输入图像映射为若干“图像图块嵌入”；（2）一个 投影器 ，接收视觉编码器的输出嵌入并把它们映射到语言模型的输入空间；（3）一个 大语言模型（LLM）骨干 。在 VLM 训练中，模型以预测下一个文本词元为目标，在从各种互联网来源整理的成对或交错的视觉-语言数据上端到端训练。

In this work, we build on the Prismatic-7B VLM [44]. Prismatic follows the same standard architecture described above, with a 600M-parameter visual encoder, a small 2-layer MLP projector, and a 7B-parameter Llama 2 language model backbone [10]. Notably, Prismatic uses a two-part visual encoder, consisting of pretrained SigLIP [79] and DinoV2 [25] models. Input image patches are passed separately through both encoders and the resulting feature vectors are concatenated channel-wise. In contrast to the more commonly used vision encoders such as CLIP- [80] or SigLIP-only encoders, the addition of DinoV2 features has been shown to be helpful for improved spatial reasoning [44], which can be particularly helpful for robot control.

在这项工作中，我们以 Prismatic-7B VLM [44] 为基础。Prismatic 采用上述标准架构，包含一个 6 亿参数的 视觉编码器 、一个小型两层 MLP 投影器 ，以及一个 70 亿参数的 Llama 2 语言模型骨干 [10]。值得注意的是，Prismatic 使用 两部分 组成的视觉编码器，由预训练的 SigLIP [79] 和 DinoV2 [25] 模型构成。输入图块分别经过两个编码器，得到的特征向量按通道拼接。与更常用的 CLIP [80] 或仅 SigLIP 的视觉编码器相比，加入 DinoV2 特征已被证明有助于提升空间推理能力 [44]，这对机器人控制可能尤其有帮助。

SigLIP, DinoV2, and Llama 2 do not release details about their training data, which likely consists of trillions of tokens of Internet-sourced image-text, image-only, and text-only data respectively. The Prismatic VLM is fine-tuned on top of these components using the LLaVA 1.5 data mixture [43], which contains a total of approximately 1M image-text and text-only data samples from open-source datasets [81, 82, 29, 83, 42].

SigLIP、DinoV2 和 Llama 2 都没有公布训练数据的细节，这些数据很可能分别是数万亿词元的互联网图文数据、纯图像数据和纯文本数据。Prismatic VLM 在这些组件之上用 LLaVA 1.5 的数据配比 [43] 进行微调，该配比总共包含约 100 万个来自开源数据集 [81, 82, 29, 83, 42] 的图文和纯文本样本。

3.2 OpenVLA Training Procedure

3.2 OpenVLA 训练流程

To train OpenVLA, we fine-tune a pretrained Prismatic-7B VLM backbone for robot action prediction (see Fig. 2). We formulate the action prediction problem as a “vision-language” task, where an input observation image and a natural language task instruction are mapped to a string of predicted robot actions [7]. To enable the VLM’s language model backbone to predict robot actions, we represent the actions in the output space of the LLM by mapping continuous robot actions to discrete tokens used by the language model’s tokenizer. Following Brohan et al. [7], we discretize each dimension of the robot actions separately into one of 256 bins. For each action dimension, we set the bin width to uniformly divide the interval between the 1st and 99th quantile of the actions in the training data. Using quantiles instead of the min-max bounds Brohan et al. [7] used allows us to ignore outlier actions in the data that could otherwise drastically expand the discretization interval and reduce the effective granularity of our action discretization.

为训练 OpenVLA，我们微调一个预训练的 Prismatic-7B VLM 骨干来预测机器人动作（见图 2）。我们把动作预测问题表述为一个“视觉-语言”任务：把输入的观测图像和自然语言任务指令映射为一串预测的机器人动作 [7]。为了让 VLM 的语言模型骨干能够预测机器人动作，我们把连续的机器人动作映射为语言模型分词器所用的离散词元，从而在 LLM 的输出空间中表示动作。沿用 Brohan 等人 [7] 的做法，我们把机器人动作的每一维分别离散化到 256 个区间（bin）之一。对每个动作维度，区间宽度设为把训练数据中动作的第 1 和第 99 百分位数之间的区间均匀等分。用分位数而不是 Brohan 等人 [7] 所用的最小-最大边界，可以忽略数据中的离群动作；否则这些离群值可能大幅撑大离散化区间，降低动作离散化的有效粒度。

\in[0\dots 255]

N

3.3 Training Data

3.3 训练数据

The goal in constructing the OpenVLA training dataset is to capture a large diversity of robot embodiments, scenes, and tasks. This enables the final model to control various robots out of the box and admits efficient fine-tuning to new robot setups. We leverage the Open X-Embodiment dataset [1] (OpenX) as a base to curate our training dataset. The full OpenX dataset, at the time of writing, consists of more than 70 individual robot datasets, with more than 2M robot trajectories, that were pooled into a coherent and easy-to-use data format in a large community effort. To make training on this data practical, we apply multiple steps of data curation to the raw dataset.

构建 OpenVLA 训练数据集的目标，是覆盖大量多样的机器人本体、场景和任务。这样最终模型既能开箱控制多种机器人， 又 能高效微调到新的机器人配置。我们以 Open X-Embodiment 数据集 [1]（OpenX）为基础来整理训练数据集。在撰写本文时，完整的 OpenX 数据集包含 70 多个单独的机器人数据集、超过 200 万条机器人轨迹，这些数据经过一项大型社区合作被汇集成统一、易用的数据格式。为了让在这些数据上训练变得可行，我们对原始数据集做了多步数据整理。

The goals of this curation are to ensure (1) a coherent input and output space across all training datasets, and (2) a balanced mix of embodiments, tasks, and scenes in the final training mixture.² To address (1), we follow [1, 5] and restrict our training dataset to contain only manipulation datasets with at least one 3rd person camera and use single-arm end-effector control. For (2), we leverage the data mixture weights of Octo [5] for all datasets that pass the first round of filtering. Octo heuristically down-weights or removes less diverse datasets and up-weights datasets with larger task and scene diversity; see Octo Model Team et al. [5] for details.

整理的目标是保证：（1）所有训练数据集的输入和输出空间一致；（2）最终训练配比中本体、任务和场景的平衡。² 针对（1），我们沿用 [1, 5] 的做法，只保留至少有一个第三人称相机、使用单臂末端执行器控制的操作数据集。针对（2），对所有通过第一轮筛选的数据集，我们沿用 Octo [5] 的数据配比权重。Octo 启发式地降低或移除多样性较低的数据集的权重，提高任务和场景多样性较大的数据集的权重；细节见 Octo Model Team 等人 [5]。

² Octo [5] demonstrated training across datasets with heterogeneous sensory inputs. While very promising, we leave an investigation of VLA training across heterogeneous sensor modalities and action spaces to future work.

² Octo [5] 展示了在传感输入异构的数据集之间进行训练。这虽然很有前景，但我们把跨异构传感模态和动作空间的 VLA 训练研究留待未来工作。

We also experimented with incorporating a few additional datasets into our training mixture that were added to the OpenX dataset since the release of Octo, including the DROID dataset [11], although at a conservative mixture weight of 10%. In practice, we found that the action token accuracy on DROID remained low throughout training, suggesting a larger mixture weight or model may be required to fit its diversity in the future. To not jeopardize the quality of the final model, we removed DROID from the data mixture for the final third of training. We provide a complete overview of the used datasets and mixture weights in Appendix A.

我们还尝试把 Octo 发布后才加入 OpenX 的几个数据集纳入训练配比，其中包括 DROID 数据集 [11]，不过只给了 10% 这一保守的配比权重。实际中我们发现，DROID 上的动作词元准确率在整个训练过程中都很低，这表明将来可能需要更大的配比权重或更大的模型才能拟合它的多样性。为了不影响最终模型的质量，我们在训练的最后三分之一阶段把 DROID 从数据配比中移除了。所用数据集和配比权重的完整概览见附录 A。

3.4 OpenVLA Design Decisions

3.4 OpenVLA 的设计决策

When developing the OpenVLA model, we explored various design decisions in smaller-scale experiments before starting the final model training run. Concretely, we trained and evaluated OpenVLA models on BridgeData V2 [6] for our initial experiments, instead of training on the full OpenX mixture, to increase iteration speed and reduce computational cost. We summarize key learnings from these explorations below.

在开发 OpenVLA 模型时，我们先在较小规模的实验中探索了各种设计决策，再开始最终的模型训练。具体来说，初期实验中我们在 BridgeData V2 [6] 上训练和评测 OpenVLA 模型，而不是在完整的 OpenX 配比上训练，以加快迭代速度、降低计算成本。下面总结这些探索得到的关键经验。

VLM Backbone. Initially, we experimented with multiple VLM backbones. Apart from Prismatic [44], we tested fine-tuning IDEFICS-1 [84] and LLaVA [85] for robot action prediction. We found that LLaVA and IDEFICS-1 performed comparably on tasks with only one object in the scene, but LLaVA demonstrated stronger language grounding in tasks that involved multiple objects in the scene and required the policy to manipulate the correct object, i.e., the object specified in the language instruction. Concretely, LLaVA improved upon IDEFICS-1 by 35% in absolute success rate, averaged across five language grounding tasks in a BridgeData V2 sink environment. The fine-tuned Prismatic VLM policy achieved further improvements, outperforming the LLaVA policy by roughly 10% in absolute success rate across both simple single-object tasks and multi-object, language grounding tasks. We attribute this performance delta to improved spatial reasoning capabilities afforded by the fused SigLIP-DinoV2 backbones (see Section 3.1). In addition to the performance enhancements, Prismatic also provides a modular and easy-to-use codebase, so we ultimately chose it to be the backbone for the OpenVLA model.

VLM 骨干 。起初我们试验了多种 VLM 骨干。除 Prismatic [44] 外，我们还测试了微调 IDEFICS-1 [84] 和 LLaVA [85] 来预测机器人动作。我们发现，在场景中只有一个物体的任务上，LLaVA 和 IDEFICS-1 表现相当；但在场景中有多个物体、需要策略操作 正确 物体（即语言指令中指定的物体）的任务上，LLaVA 表现出更强的语言落地能力。具体而言，在 BridgeData V2 水槽环境的五个语言落地任务上取平均，LLaVA 的绝对成功率比 IDEFICS-1 高 35%。微调后的 Prismatic VLM 策略又有进一步提升，在简单的单物体任务和多物体语言落地任务上，绝对成功率都比 LLaVA 策略高出约 10%。我们把这一性能差距归因于融合 SigLIP-DinoV2 骨干带来的更强空间推理能力（见第 3.1 节）。除了性能提升，Prismatic 还提供了模块化、易用的代码库，因此我们最终选它作为 OpenVLA 模型的骨干。

224\times 224

224\times 224

Fine-Tuning Vision Encoder. Prior work on VLMs found that freezing vision encoders during VLM training typically leads to higher performance [44]. Intuitively, a frozen vision encoder may better preserve the robust features learned from its Internet-scale pretraining. However, we found fine-tuning the vision encoder during VLA training to be crucial for good VLA performance. We hypothesize that the pretrained vision backbone may not capture sufficient fine-grained spatial details about important parts of the scene to enable precise robotic control.

微调视觉编码器 。先前关于 VLM 的工作发现，VLM 训练时冻结视觉编码器通常能带来更高的性能 [44]。直观上，冻结的视觉编码器可能更好地保留其在互联网规模预训练中学到的鲁棒特征。然而我们发现，在 VLA 训练中微调视觉编码器对取得良好的 VLA 性能至关重要。我们推测，预训练的视觉骨干可能没有捕捉到场景关键部位足够细粒度的空间细节，不足以支撑精确的机器人控制。

Training Epochs. Typical LLM or VLM training runs complete at most one or two epochs through their training dataset. In contrast, we found it important for VLA training to iterate through the training dataset significantly more times, with real robot performance continually increasing until training action token accuracy surpasses 95%. Our final training run completes 27 epochs through its training dataset.

训练轮数 。典型的 LLM 或 VLM 训练最多在训练数据集上跑一到两个轮次（epoch）。相比之下，我们发现 VLA 训练需要在训练数据集上迭代多得多的次数：真实机器人上的性能会持续提高，直到训练动作词元准确率超过 95%。我们最终的训练在训练数据集上完成了 27 个轮次。

Learning Rate. We swept the learning rate across multiple orders of magnitude for VLA training, and achieved the best results using a fixed learning rate of 2e-5 (the same learning rate used during VLM pretraining [44]). We did not find learning rate warmup to provide benefits.

学习率 。我们在多个数量级范围内扫描了 VLA 训练的学习率，使用 2e-5 的固定学习率（与 VLM 预训练 [44] 时相同）取得了最好的结果。我们没有发现学习率预热（warmup）带来好处。

3.5 Infrastructure for Training and Inference

3.5 训练与推理基础设施

The final OpenVLA model is trained on a cluster of 64 A100 GPUs for 14 days, or a total of 21,500 A100-hours, using a batch size of 2048. During inference, OpenVLA requires 15GB of GPU memory when loaded in bfloat16 precision (i.e., without quantization) and runs at approximately 6Hz on one NVIDIA RTX 4090 GPU (without compilation, speculative decoding, or other inference speed-up tricks). We can further reduce the memory footprint of OpenVLA during inference via quantization, without compromising performance in real-world robotics tasks, as shown in Section 5.4. We report inference speed on various consumer- and server-grade GPUs in Fig. 6. For convenience, we implement a remote VLA inference server to allow real-time remote streaming of action predictions to the robot - removing the requirement of having access to a powerful local compute device to control the robot. We release this remote inference solution as part of our open-source code release (Section 4).

最终的 OpenVLA 模型在 64 块 A100 GPU 组成的集群上训练了 14 天，共计 21,500 A100 小时，批大小为 2048。推理时，OpenVLA 以 bfloat16 精度加载（即不量化）需要 15GB 显存，在一块 NVIDIA RTX 4090 GPU 上的运行速度约为 6Hz（未使用编译、推测解码或其他推理加速技巧）。如第 5.4 节所示，我们还可以通过量化进一步降低 OpenVLA 推理时的显存占用，而不影响真实机器人任务上的性能。图 6 报告了在各种消费级和服务器级 GPU 上的推理速度。为方便起见，我们实现了一个远程 VLA 推理服务器，可以把动作预测实时远程流式传给机器人——这样控制机器人时就不必在本地配备强大的计算设备。我们把这一远程推理方案作为开源代码的一部分发布（第 4 节）。

4 The OpenVLA Codebase

4 OpenVLA 代码库

Along with our model, we release the OpenVLA codebase, a modular PyTorch codebase for training VLA models (see https://openvla.github.io). It scales from fine-tuning VLAs on individual GPUs to training billion-parameter VLAs on multi-node GPU clusters, and supports modern techniques for large transformer model training such as automatic mixed precision (AMP, PyTorch [75]), FlashAttention [76], and fully sharded data parallelism (FSDP, Zhao et al. [77]). Out of the box, the OpenVLA codebase has full support for training on the Open X dataset, integrates with HuggingFace’s [21] AutoModel class, and supports LoRA fine-tuning [26] and quantized model inference [88, 27].

随模型一起，我们发布了 OpenVLA 代码库，一个用于训练 VLA 模型的模块化 PyTorch 代码库（见 https://openvla.github.io）。它既能在单块 GPU 上微调 VLA，也能扩展到在多节点 GPU 集群上训练数十亿参数的 VLA，并支持大型 Transformer 模型训练的现代技术，例如自动混合精度（AMP，PyTorch [75]）、FlashAttention [76] 和全分片数据并行（FSDP，Zhao 等人 [77]）。OpenVLA 代码库开箱即完整支持在 Open X 数据集上训练，与 HuggingFace [21] 的 AutoModel 类集成，并支持 LoRA 微调 [26] 和量化模型推理 [88, 27]。

5 Experiments

5 实验

The goal of our experimental evaluations is to test OpenVLA’s ability to serve as a powerful multi-robot control policy out of the box, as well as be a good initialization for fine-tuning to new robot tasks. Concretely, we aim to answer the following questions:

我们实验评估的目标，是检验 OpenVLA 能否开箱即用地充当强大的多机器人控制策略，以及能否作为微调到新机器人任务的良好初始化。具体来说，我们希望回答以下问题：

How does OpenVLA compare to prior generalist robot policies, when evaluating on multiple robots and various types of generalization? Can OpenVLA be effectively fine-tuned on a new robot setup and task, and how does it compare to state-of-the-art data-efficient imitation learning approaches? Can we use parameter-efficient fine-tuning and quantization to reduce the computational requirements for training and inference of OpenVLA models and make them more accessible? What are the performance-compute trade-offs?

在多种机器人和多类泛化上评测时，OpenVLA 与先前的通用机器人策略相比表现如何？ OpenVLA 能否有效地微调到新的机器人配置和任务上？与最先进的数据高效模仿学习方法相比如何？ 能否用参数高效微调和量化来降低 OpenVLA 模型训练和推理的计算需求，让它更容易使用？性能与计算之间的权衡是怎样的？

5.1 Direct Evaluations on Multiple Robot Platforms

5.1 在多个机器人平台上的直接评测

\pm

\pm

Robot Setups and Tasks. We evaluate OpenVLA’s performance “out-of-the-box” on two robot embodiments: the WidowX robot from the BridgeData V2 evaluations [6] (see Fig. 1, left) and the mobile manipulation robot from the RT-1 and RT-2 evaluations [2, 7] (“Google robot”; see Fig. 1, middle). Both platforms have been extensively used in prior works for evaluating generalist robot policies [2, 7, 1, 5]. We define a comprehensive set of evaluation tasks in each environment that covers various axes of generalization, such as visual (unseen backgrounds, distractor objects, colors/appearances of objects); motion (unseen object positions/orientations); physical (unseen object sizes/shapes); and semantic (unseen target objects, instructions, and concepts from the Internet) generalization. We also assess language conditioning ability in scenes with multiple objects, testing whether the policy can manipulate the correct target object, as specified in the user’s prompt. See bottom row of Fig. 3 and Fig. 4 for example task images in the BridgeData V2 and Google robot evaluations, respectively. Overall, we evaluated each method in 170 rollouts (17 tasks with 10 trials each) for BridgeData V2 experiments and 60 rollouts (12 tasks with 5 trials each) for Google robot experiments. A detailed breakdown of all tasks and how they differ from the training data is in Appendix B. All evaluations in this and the following sections are conducted as A/B evaluations, using the same tasks with the same sets of initial robot and object states, to ensure fair comparison.

机器人配置与任务 。我们在两种机器人本体上评测 OpenVLA“开箱即用”的性能：BridgeData V2 评测 [6] 中的 WidowX 机器人（见图 1 左），以及 RT-1 和 RT-2 评测 [2, 7] 中的移动操作机器人（“Google 机器人”；见图 1 中）。两个平台都在先前工作中被广泛用于评测通用机器人策略 [2, 7, 1, 5]。我们在每个环境中定义了一套全面的评测任务，覆盖多个泛化维度，例如 视觉 泛化（未见过的背景、干扰物体、物体颜色/外观）、 运动 泛化（未见过的物体位置/朝向）、 物理 泛化（未见过的物体尺寸/形状）和 语义 泛化（未见过的目标物体、指令以及来自互联网的概念）。我们还在多物体场景中评估语言条件能力，测试策略能否按用户提示操作正确的目标物体。BridgeData V2 和 Google 机器人评测的示例任务图像分别见图 3 和图 4 的底行。总的来说，BridgeData V2 实验中每种方法评测 170 次（17 个任务，每个 10 次），Google 机器人实验中评测 60 次（12 个任务，每个 5 次）。所有任务的详细说明以及它们与训练数据的差异见附录 B。本节及后续各节的所有评测都以 A/B 方式进行，使用相同的任务和相同的机器人与物体初始状态集合，以保证比较公平。

Comparisons. We compare OpenVLA’s performance to three prior generalist manipulation policies: RT-1-X [1], RT-2-X [1], and Octo [5]. RT-1-X (35M parameters) and Octo (93M parameters) are transformer policies trained from scratch on subsets of the OpenX dataset; Octo is the state-of-the-art model among open-source manipulation policies. RT-2-X (55B parameters) is a state-of-the-art, closed-source VLA that leverages Internet-pretrained vision and language backbones.

对比方法 。我们把 OpenVLA 的性能与三种先前的通用操作策略比较： RT-1-X [1]、 RT-2-X [1] 和 Octo [5]。 RT-1-X （3500 万参数）和 Octo （9300 万参数）是在 OpenX 数据集子集上从零训练的 Transformer 策略；Octo 是开源操作策略中最先进的模型。 RT-2-X （550 亿参数）是最先进的闭源 VLA，利用了互联网预训练的视觉和语言骨干。

The results are summarized in Fig. 3 for BridgeData V2 evaluations and Fig. 4 for Google robot evaluations (per-task breakdown in Appendix, Table 4 and Table 6). We find that both RT-1-X and Octo struggle on the tested tasks, often failing to manipulate the correct object, especially when distractors are present, and in some cases causing the robot to wave its arm around aimlessly. Note that our evaluations test even larger degrees of generalization than the evaluations performed in those prior works to challenge the Internet-pretrained VLA models. Thus, lower performance of models without Internet pretraining is expected. RT-2-X clearly outperforms both RT-1-X and Octo, demonstrating the benefits of large, pretrained VLMs for robotics.

BridgeData V2 评测的结果汇总在图 3，Google 机器人评测的结果汇总在图 4（逐任务细分见附录表 4 和表 6）。我们发现 RT-1-X 和 Octo 在所测任务上都表现吃力，常常操作错物体，尤其在有干扰物时；有时还会让机器人漫无目的地挥动手臂。需要注意，为了考验互联网预训练的 VLA 模型，我们的评测所测试的泛化程度比那些先前工作中的评测更大。因此，没有互联网预训练的模型表现较差是意料之中的。RT-2-X 明显优于 RT-1-X 和 Octo，展示了大型预训练 VLM 对机器人的好处。

\pm

\pm

Notably, OpenVLA performs comparably to RT-2-X on Google robot evaluations and significantly outperforms RT-2-X on BridgeData V2 evaluations despite being an order of magnitude smaller (7B vs. 55B parameters). Qualitatively, we find that both RT-2-X and OpenVLA exhibit markedly more robust behaviors than the other tested models, such as approaching the correct object when distractor objects are present, properly orienting the robot’s end-effector to align with the orientation of the target object, and even recovering from mistakes such as insecurely grasping objects (see https://openvla.github.io for qualitative rollout examples). RT-2-X achieves higher performance in semantic generalization tasks, as shown in Fig. 3, which is expected given that it uses larger-scale Internet pretraining data and is co-fine-tuned with both robot action data and Internet pretraining data to better preserve the pretraining knowledge, rather than being fine-tuned solely on robot data, like OpenVLA. However, OpenVLA performs comparably or better in all other task categories in both BridgeData V2 and Google robot evaluations. The performance difference can be attributed to a combination of factors: we curated a much larger training dataset for OpenVLA with 970k trajectories (vs. 350k for RT-2-X); we performed more careful cleaning of the training dataset and, e.g., filtered out all-zero actions in the Bridge dataset (see Appendix C for a detailed discussion); and OpenVLA uses a fused vision encoder that combines pretrained semantic and spatial features. See Appendix D for ablation analyses of these components.

值得注意的是，OpenVLA 在 Google 机器人评测上与 RT-2-X 表现相当，在 BridgeData V2 评测上则显著优于 RT-2-X，尽管规模小了一个数量级（70 亿对 550 亿参数）。定性地看，我们发现 RT-2-X 和 OpenVLA 都比其他被测模型表现出明显更鲁棒的行为，例如在有干扰物时仍能接近正确的物体，正确调整机器人末端执行器的朝向以对齐目标物体的朝向，甚至能从没抓稳物体之类的错误中恢复（定性的试验示例见 https://openvla.github.io）。如图 3 所示，RT-2-X 在语义泛化任务上性能更高，这在意料之中：它使用了更大规模的互联网预训练数据，并且同时用机器人动作数据和互联网预训练数据做联合微调，以更好地保留预训练知识，而不是像 OpenVLA 那样只在机器人数据上微调。不过，在 BridgeData V2 和 Google 机器人评测的所有其他任务类别上，OpenVLA 的表现都相当或更好。这一性能差异可以归因于几方面因素的结合：我们为 OpenVLA 整理了大得多的训练数据集，包含 97 万条轨迹（RT-2-X 为 35 万条）；我们对训练数据集做了更仔细的清洗，例如过滤掉了 Bridge 数据集中的全零动作（详细讨论见附录 C）；此外 OpenVLA 使用融合视觉编码器，结合了预训练的语义特征 和 空间特征。这些组件的消融分析见附录 D。

5.2 Data-Efficient Adaptation to New Robot Setups

5.2 数据高效地适配新的机器人配置

While prior works mainly focused on directly evaluating VLAs “out-of-the-box” [16, 7, 1], effective fine-tuning of VLA models to new tasks and robot setups is largely unexplored, yet is key for their widespread adoption. In this section, we investigate OpenVLA’s ability to be quickly adapted to a new real-world robot setup. (See Appendix E for fine-tuning experiments in simulation.)

先前工作主要关注直接“开箱即用”地评测 VLA [16, 7, 1]，而把 VLA 模型有效地 微调 到新任务和新机器人配置上，基本没有被研究过，但这恰恰是 VLA 得到广泛应用的关键。本节我们研究 OpenVLA 能否快速适配到新的 真实世界 机器人配置。（仿真中的微调实验见附录 E。）

Robot setups and tasks. We test a simple fine-tuning recipe for the OpenVLA model: full fine-tuning of all model parameters, using small datasets with 10-150 demonstrations of a target task (see Fig. 5; we explore parameter-efficient fine-tuning approaches in Section 5.3). We test OpenVLA in two setups: Franka-Tabletop, a stationary, table-mounted Franka Emika Panda 7-DoF robot arm; and Franka-DROID, the Franka robot arm setup from the recently released DROID dataset [11], mounted on a movable standing desk. The setups use 5Hz and 15 Hz non-blocking controllers, respectively. We choose Franka robot arms as the target embodiment for our fine-tuning experiments since they are widely used in the robot learning community and thus a likely “target” of OpenVLA fine-tuning. We test on setups with different control frequencies to test OpenVLA’s applicability to a range of use cases.

机器人配置与任务 。我们为 OpenVLA 模型测试一种简单的微调方案：全参数微调，使用目标任务的 10-150 条示教组成的小数据集（见图 5；参数高效微调方法在第 5.3 节探讨）。我们在两种配置中测试 OpenVLA： Franka-Tabletop ，一台固定安装在桌上的 Franka Emika Panda 7 自由度机械臂；以及 Franka-DROID ，即最近发布的 DROID 数据集 [11] 中的 Franka 机械臂配置，安装在可移动的升降桌上。两种配置分别使用 5Hz 和 15Hz 的非阻塞控制器。我们选择 Franka 机械臂作为微调实验的目标本体，因为它在机器人学习社区中被广泛使用，因而很可能成为 OpenVLA 微调的“目标”。我们在控制频率不同的配置上测试，以检验 OpenVLA 对多种使用场景的适用性。

\pm

\pm

Comparisons. We compare to Diffusion Policy [3], a state-of-the-art data-efficient imitation learning approach, trained from scratch. We also compare to Diffusion Policy (matched), a version of Diffusion Policy that matches the input and output specifications of OpenVLA.³ Additionally, we evaluate Octo [5] fine-tuned on the target dataset, since it is currently the best generalist policy that supports fine-tuning (fine-tuning of RT-2-X is not supported through its inference API). We also fine-tune OpenVLA on the same target dataset, and the resulting policy is denoted by OpenVLA . Finally, as an ablation experiment, we compare to OpenVLA (scratch), where we directly fine-tune the underlying base Prismatic VLM on the target robot setup - rather than fine-tuning the OpenX-pretrained OpenVLA model - to assess the benefit of large-scale robot pretraining.

对比方法 。我们与从零训练的 Diffusion Policy [3] 比较，它是最先进的数据高效模仿学习方法。我们还与 Diffusion Policy (matched) 比较，这是一个输入和输出规格与 OpenVLA 一致的 Diffusion Policy 版本。³ 此外，我们评测在目标数据集上微调的 Octo [5]，因为它是目前支持微调的最佳通用策略（RT-2-X 的推理 API 不支持微调）。我们也在同一目标数据集上微调 OpenVLA，得到的策略记作 OpenVLA 。最后，作为消融实验，我们与 OpenVLA (scratch) 比较：直接在目标机器人配置上微调底层的 Prismatic 基础 VLM，而不是微调经过 OpenX 预训练的 OpenVLA 模型，以评估大规模机器人预训练的好处。

T

T

We present the results in Fig. 5 (per-task breakdown in Appendix, Table 7). We find that both versions of Diffusion Policy are competitive with or outperform the generalist policies Octo and OpenVLA on narrower single-instruction tasks like “Put Carrot in Bowl” and “Pour Corn into Pot”, but the pretrained generalist policies perform better in more diverse fine-tuning tasks that involve multiple objects in the scene and require language conditioning. OpenX pretraining for Octo and OpenVLA enables the models to better adapt to these more diverse tasks where language grounding is important; we see evidence for this in the lower performance of OpenVLA (scratch).

结果见图 5（逐任务细分见附录表 7）。我们发现，在“把胡萝卜放进碗里”“把玉米倒进锅里”这类较窄的单指令任务上，两个版本的 Diffusion Policy 都与通用策略 Octo 和 OpenVLA 相当或更好；但在场景中有多个物体、需要语言条件的更多样化的微调任务上，预训练的通用策略表现更好。OpenX 预训练使 Octo 和 OpenVLA 能更好地适应这些语言落地很重要的多样化任务；OpenVLA (scratch) 较低的性能为此提供了证据。

Overall, we find that OpenVLA achieves the highest average performance. Notably, most prior works achieve strong performance only in either narrow single-instruction or diverse multi-instruction tasks, resulting in widely varying success rates. OpenVLA is the only approach that achieves at least 50% success rate across all tested tasks, suggesting that it can be a strong default option for imitation learning tasks, particularly if they involve a diverse set of language instructions. For narrower but highly dexterous tasks, Diffusion Policy still shows smoother and more precise trajectories; incorporating action chunking and temporal smoothing, as implemented in Diffusion Policy, may help OpenVLA attain the same level of dexterity and may be a promising direction for future work (see Section 6 for a detailed discussion of current limitations).

总体而言，我们发现 OpenVLA 取得了最高的平均性能。值得注意的是，大多数先前工作 要么 只在狭窄的单指令任务上、 要么 只在多样化的多指令任务上表现强劲，导致成功率波动很大。OpenVLA 是唯一在所有被测任务上成功率都至少达到 50% 的方法，表明它可以作为模仿学习任务的强有力默认选择，尤其是当任务涉及多样的语言指令时。对于较窄但对灵巧性要求很高的任务，Diffusion Policy 仍然表现出更平滑、更精确的轨迹；引入 Diffusion Policy 中实现的动作分块（action chunking）和时间平滑，可能帮助 OpenVLA 达到同样的灵巧水平，这可能是未来工作的一个有前景的方向（当前局限的详细讨论见第 6 节）。

5.3 Parameter-Efficient Fine-Tuning

5.3 参数高效微调

The full fine-tuning runs of OpenVLA in the previous section used 8 A100 GPUs for 5-15 hours per task (depending on the dataset size) to achieve high performance. While this is substantially less compute than what is required for VLA pretraining, in this section we explore even more compute- and parameter-efficient fine-tuning approaches and investigate their effectiveness.

上一节中 OpenVLA 的全参数微调，每个任务用 8 块 A100 GPU 训练 5-15 小时（取决于数据集大小）才达到高性能。虽然这比 VLA 预训练所需的计算少得多，但本节我们探索计算和参数上更高效的微调方法，并考察它们的效果。

\times 10^{6}

Strategy	Success Rate	Train Params ( $\times 10^{6}$ )	VRAM (batch 16)
Full FT	69.7 $\pm$ 7.2 %	7,188.1	163.3 GB*
Last layer only	30.3 $\pm$ 6.1 %	465.1	51.4 GB
Frozen vision	47.0 $\pm$ 6.9 %	6,760.4	156.2 GB*
Sandwich	62.1 $\pm$ 7.9 %	914.2	64.0 GB
LoRA, rank=32	68.2 $\pm$ 7.5%	97.6	59.7 GB
rank=64	68.2 $\pm$ 7.8%	195.2	60.5 GB

\times 10^{6}

策略	成功率	训练参数量（ $\times 10^{6}$ ）	显存（批大小 16）
全参数微调	69.7 $\pm$ 7.2 %	7,188.1	163.3 GB*
仅最后一层	30.3 $\pm$ 6.1 %	465.1	51.4 GB
冻结视觉编码器	47.0 $\pm$ 6.9 %	6,760.4	156.2 GB*
三明治微调	62.1 $\pm$ 7.9 %	914.2	64.0 GB
LoRA，秩=32	68.2 $\pm$ 7.5%	97.6	59.7 GB
秩=64	68.2 $\pm$ 7.8%	195.2	60.5 GB

r

r

r=32

r=32

⁴ In Section 5.3 and Section 5.4, we experiment with a version of the OpenVLA model that is pretrained with a smaller robot data mixture (the same OpenX dataset mixture as Octo) and has a slightly smaller architecture which only uses a SigLIP [79] vision backbone instead of the fused DinoSigLIP encoder. We find that this simpler architecture still achieves strong performance in both fine-tuning tasks and “out-of-the-box” tasks.

⁴ 在第 5.3 节和第 5.4 节中，我们实验的是 OpenVLA 模型的一个版本：它用更小的机器人数据配比（与 Octo 相同的 OpenX 数据配比）预训练，架构也略小，只使用 SigLIP [79] 视觉骨干，而不是融合的 DinoSigLIP 编码器。我们发现这种更简单的架构在微调任务和“开箱即用”任务上仍有强劲的表现。

5.4 Memory-Efficient Inference via Quantization

5.4 通过量化实现省显存的推理

\spadesuit

\spadesuit

\pm

Precision	Bridge Success	VRAM
bfloat16	71.3 $\pm$ 4.8%	16.8 GB
int8	58.1 $\pm$ 5.1%	10.2 GB
int4	71.9 $\pm$ 4.7%	7.0 GB

\pm

精度	Bridge 成功率	显存
bfloat16	71.3 $\pm$ 4.8%	16.8 GB
int8	58.1 $\pm$ 5.1%	10.2 GB
int4	71.9 $\pm$ 4.7%	7.0 GB

<

OpenVLA 是一个 70 亿参数的模型，推理时比 Octo（参数不到 1 亿）等先前的开源通用策略占用更多显存。我们遵循 LLM 部署的最佳实践，以 bfloat16 精度保存和加载 OpenVLA 进行推理（我们的默认方式），这把显存占用减半，使我们能在只有 16GB 显存的 GPU 上部署 OpenVLA。本节我们测试能否利用为部署 LLM 而开发的现代量化技术 [88, 27]，进一步降低策略推理所需的显存，扩大 VLA 策略的可及性。这些方法以更低的精度加载网络权重，用可能降低的推理速度和精度换取更低的显存需求。

Concretely, we investigate serving the OpenVLA model with 8-bit and 4-bit precision on 8 representative BridgeData V2 tasks. We report memory footprint and rollout performance in Table 2. We also report achievable control frequencies on various consumer- and server-grade GPUs in Fig. 6. We observe that 8-bit quantization slows down inference across most GPUs, due to the overhead of the added quantization operations. 4-bit inference achieves higher throughput, since reduced GPU memory transfer compensates for the quantization overhead.

具体来说，我们在 8 个有代表性的 BridgeData V2 任务上研究以 8 比特和 4 比特精度部署 OpenVLA 模型。表 2 报告了显存占用和试验性能。图 6 还报告了在各种消费级和服务器级 GPU 上可达到的控制频率。我们观察到，由于新增量化操作的开销，8 比特量化在大多数 GPU 上会拖慢推理。4 比特推理的吞吐更高，因为 GPU 显存传输量的减少抵消了量化开销。

As a result of the reduced inference speed, we observe a substantial performance decrease with 8-bit quantization: on the A5000 GPU we use for our evaluations, we can only run the model at 1.2Hz, which significantly changes the system dynamics compared to the training dataset for the 5Hz non-blocking controller used in the BridgeData V2 tasks.⁵ Notably, 4-bit quantization results in similar performance as bfloat16 half-precision inference despite requiring less than half the amount of GPU memory. 4-bit quantized models can run at 3Hz on the A5000, thus more closely matching the system dynamics during data collection.

由于推理速度降低，我们观察到 8 比特量化导致性能大幅下降：在我们评测所用的 A5000 GPU 上，模型只能以 1.2Hz 运行，与 BridgeData V2 任务所用的 5Hz 非阻塞控制器下采集的训练数据相比，系统动力学发生了显著变化。⁵ 值得注意的是，4 比特量化的性能与 bfloat16 半精度推理相近，而所需 GPU 显存不到后者的一半。4 比特量化模型在 A5000 上能以 3Hz 运行，因而更接近数据采集时的系统动力学。

⁵ We attribute the performance loss to low inference speed, since both 8-bit and 4-bit quantization achieve comparable token accuracy to bfloat16 inference when evaluated offline on training data. See Section D.4 for supporting details.

⁵ 我们把性能损失归因于推理速度低，因为在训练数据上离线评估时，8 比特和 4 比特量化的词元准确率都与 bfloat16 推理相当。支持细节见 D.4 节。

6 Discussion and Limitations

6 讨论与局限

In this work, we presented OpenVLA, a state-of-the-art, open-source vision-language-action model that obtains strong performance for cross-embodiment robot control out-of-the-box. We also demonstrated that OpenVLA can be easily adapted to new robot setups via parameter-efficient fine-tuning techniques.

在这项工作中，我们推出了 OpenVLA，一个最先进的开源视觉-语言-动作模型，开箱即可在跨本体机器人控制上取得强劲性能。我们还证明了 OpenVLA 可以借助参数高效微调技术轻松适配到新的机器人配置。

The current OpenVLA model has several limitations. First, it currently only supports single-image observations. In reality, real-world robot setups are heterogeneous, with a wide range of possible sensory inputs [5]. Expanding OpenVLA to support multiple image and proprioceptive inputs as well as observation history is an important avenue for future work. Exploring the use of VLMs pretrained on interleaved image and text data may facilitate such flexible-input VLA fine-tuning.

当前的 OpenVLA 模型有若干局限。首先，它目前只支持单张图像观测。而现实中的机器人配置是异构的，可能的传感输入范围很广 [5]。扩展 OpenVLA 以支持多张图像、本体感知输入以及观测历史，是未来工作的重要方向。探索在 交错 图文数据上预训练的 VLM，可能有助于实现这种输入灵活的 VLA 微调。

Secondly, improving the inference throughput of OpenVLA is critical to enable VLA control for high-frequency control setups such as ALOHA [90], which runs at 50Hz. This will also enable testing VLAs on more dexterous, bi-manual manipulation tasks than what we investigated in this work. Exploring the use of action chunking or alternative inference-time optimization techniques such as speculative decoding [91] offer potential remedies.

其次，提高 OpenVLA 的推理吞吐，对于在 ALOHA [90]（以 50Hz 运行）这类高频控制配置上实现 VLA 控制至关重要。这也将使我们能在比本文所研究的更灵巧的双臂操作任务上测试 VLA。探索动作分块或推测解码 [91] 等其他推理时优化技术，提供了可能的解决办法。

Additionally, there is room for further performance improvements. While OpenVLA outperforms prior generalist policies, it does not yet offer very high reliability on the tested tasks, typically achieving <90% success rate.

此外，性能还有进一步提升的空间。虽然 OpenVLA 优于先前的通用策略，但它在所测任务上的可靠性还不够高，成功率通常低于 90%。

Finally, due to compute limitations, many VLA design questions remain underexplored: What effect does the size of the base VLM have on VLA performance? Does co-training on robot action prediction data and Internet-scale vision-language data substantially improve VLA performance? What visual features are best-suited for VLA models? We hope that the release of the OpenVLA model and codebase will enable the community to jointly investigate these questions.

最后，由于算力限制，许多 VLA 设计问题仍未被充分探索：基础 VLM 的规模对 VLA 性能有什么影响？在机器人动作预测数据和互联网规模视觉-语言数据上联合训练，能否显著提升 VLA 性能？哪些视觉特征最适合 VLA 模型？我们希望 OpenVLA 模型和代码库的发布能让社区共同研究这些问题。

Acknowledgments We are grateful to the Toyota Research Institute for providing significant funding and compute resources required to carry out this research. We also thank the Stanford Center for Research on Foundation Models for providing additional compute resources and Google DeepMind for alpha access to the RT-2-X API for our evaluations. We acknowledge additional support from Volkswagen, Physical Intelligence, ONR grants N00014-22-1-2621 and N00014-22-1-2293, the National Science Foundation through IIS-2246811, and DARPA ANSR.

致谢 感谢丰田研究院（Toyota Research Institute）为开展本研究提供了大量资金和计算资源。我们还感谢斯坦福基础模型研究中心（Stanford Center for Research on Foundation Models）提供额外的计算资源，以及 Google DeepMind 为我们的评测提供 RT-2-X API 的 alpha 访问权限。我们感谢大众汽车（Volkswagen）、Physical Intelligence、ONR 资助 N00014-22-1-2621 和 N00014-22-1-2293、美国国家科学基金会 IIS-2246811 以及 DARPA ANSR 提供的额外支持。

References

参考文献

[71] S. Sontakke, J. Zhang, S. Arnold, K. Pertsch, E. Bıyık, D. Sadigh, C. Finn, and L. Itti. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems , 36, 2024. [72] J. Huang, S. Yong, X. Ma, X. Linghu, P. Li, Y. Wang, Q. Li, S.-C. Zhu, B. Jia, and S. Huang. An embodied generalist agent in 3d world. In Proceedings of the International Conference on Machine Learning (ICML) , 2024. [73] X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y. Jing, W. Zhang, H. Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378 , 2023. [74] H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y. Du, Y. Hong, and C. Gan. 3d-vla: 3d vision-language-action generative world model. arXiv preprint arXiv:2403.09631 , 2024. [75] PyTorch. Automatic mixed precision. URL https://pytorch.org/docs/stable/amp.html . [76] T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 , 2023. [77] Y. Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277 , 2023. [78] N. Dorka, C. Huang, T. Welschehold, and W. Burgard. What matters in employing vision language models for tokenizing actions in robot control? In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024 . [79] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11975-11986, 2023. [80] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

[81] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, 2018. [82] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021. [83] O. Sidorov, R. Hu, M. Rohrbach, and A. Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 742-758. Springer, 2020. [84] H. Face. Introducing idefics: An open reproduction of state-of-the-art visual langage model. Hugging Face Blog , 2024. [85] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024. [86] B. McKinzie, Z. Gan, J.-P. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah, X. Du, F. Peng, F. Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611 , 2024. [87] J. Lin, H. Yin, W. Ping, Y. Lu, P. Molchanov, A. Tao, H. Mao, J. Kautz, M. Shoeybi, and S. Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07533 , 2023. [88] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems , 35:30318-30332, 2022. [89] NVIDIA. Tensorrt-llm. URL https://github.com/NVIDIA/TensorRT-LLM . [90] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705 , 2023.

[91] Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning , pages 19274-19286. PMLR, 2023. [92] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022. [93] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task agnostic offline reinforcement learning. In Proceedings of the 6th Conference on Robot Learning (CoRL) , 2022. [94] O. Mees, J. Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , London, UK, 2023. [95] S. Dass, J. Yapeter, J. Zhang, J. Zhang, K. Pertsch, S. Nikolaidis, and J. J. Lim. CLVR jaco play dataset, 2023. URL https://github.com/clvrai/clvr_jaco_play_dataset . [96] J. Luo, C. Xu, X. Geng, G. Feng, K. Fang, L. Tan, S. Schaal, and S. Levine. Multi-stage cable routing through hierarchical imitation learning. arXiv preprint arXiv:2307.08927 , 2023. [97] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. RoboTurk: A crowdsourcing platform for robotic skill learning through imitation. CoRR , abs/1811.02790, 2018. URL http://arxiv.org/abs/1811.02790 . [98] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors, 2023. [99] L. Y. Chen, S. Adebola, and K. Goldberg. Berkeley UR5 demonstration dataset. https://sites.google.com/view/berkeley-ur5/home . [100] G. Zhou, V. Dean, M. K. Srirama, A. Rajeswaran, J. Pari, K. Hatch, A. Jain, T. Yu, P. Abbeel, L. Pinto, C. Finn, and A. Gupta. Train offline, test online: A real robot learning benchmark, 2023.

[101] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters , 2023. [102] S. Belkhale, Y. Cui, and D. Sadigh. Hydra: Hybrid robot actions for imitation learning. arxiv , 2023. [103] Y. Zhu, P. Stone, and Y. Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters , 7(2):4126-4133, 2022. [104] Z. J. Cui, Y. Wang, N. M. M. Shafiullah, and L. Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.10047 , 2022. [105] M. Heo, Y. Lee, D. Lee, and J. J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In Robotics: Science and Systems , 2023. [106] G. Yan, K. Wu, and X. Wang. ucsd kitchens Dataset. August 2023. [107] S. Nasiriany, T. Gao, A. Mandlekar, and Y. Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Conference on Robot Learning (CoRL) , 2022. [108] H. Liu, S. Nasiriany, L. Zhang, Z. Bao, and Y. Zhu. Robot learning on the job: Human-in-the-loop autonomy and learning during deployment. In Robotics: Science and Systems (RSS) , 2023. [109] G. Quere, A. Hagengruber, M. Iskandar, S. Bustamante, D. Leidner, F. Stulp, and J. Vogel. Shared Control Templates for Assistive Robotics. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , page 7, Paris, France, 2020. [110] S. Saxena, M. Sharma, and O. Kroemer. Multi-resolution sensing for real-time control with vision-language models. In 7th Annual Conference on Robot Learning , 2023. URL https://openreview.net/forum?id=WuBv9-IGDUA .

[111] R. Shah, R. Martín-Martín, and Y. Zhu. MUTEX: Learning unified policies from multimodal task specifications. In 7th Annual Conference on Robot Learning , 2023. URL https://openreview.net/forum?id=PwqiqaaEzJ . [112] X. Zhu, R. Tian, C. Xu, M. Ding, W. Zhan, and M. Tomizuka. Fanuc manipulation: A dataset for learning-based manipulation with fanuc mate 200id robot. 2023. [113] R. Mendonca, S. Bahl, and D. Pathak. Structured world models from human videos. CoRL , 2023. [114] J. Luo, C. Xu, F. Liu, L. Tan, Z. Lin, J. Wu, P. Abbeel, and S. Levine. Fmb: a functional manipulation benchmark for generalizable robotic learning. arXiv preprint arXiv:2401.08553 , 2024. [115] N. M. M. Shafiullah, A. Rai, H. Etukuru, Y. Liu, I. Misra, S. Chintala, and L. Pinto. On bringing robots home, 2023. [116] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems , 36, 2024. [117] V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.

Appendix A Data Mixture Details

附录 A 数据配比细节

We list our used data mixture in Table 3. The mixture mostly follows [5], with a few additional datasets.

表 3 列出了我们使用的数据配比。该配比大体沿用 [5]，另外加入了几个数据集。

OpenVLA Training Dataset Mixture Fractal [92] 12.7% Kuka [45] 12.7% Bridge[47, 6] 13.3% Taco Play [93, 94] 3.0% Jaco Play [95] 0.4% Berkeley Cable Routing [96] 0.2% Roboturk [97] 2.3% Viola [98] 0.9% Berkeley Autolab UR5 [99] 1.2% Toto [100] 2.0% Language Table [101] 4.4% Stanford Hydra Dataset [102] 4.4% Austin Buds Dataset [103] 0.2% NYU Franka Play Dataset [104] 0.8% Furniture Bench Dataset [105] 2.4% UCSD Kitchen Dataset [106] <0.1% Austin Sailor Dataset [107] 2.2% Austin Sirius Dataset [108] 1.7% DLR EDAN Shared Control [109] <0.1% IAMLab CMU Pickup Insert [110] 0.9% UTAustin Mutex [111] 2.2% Berkeley Fanuc Manipulation [112] 0.7% CMU Stretch [113] 0.2% BC-Z [55] 7.5% FMB Dataset [114] 7.1% DobbE [115] 1.4% DROID [11] 10.0%⁶ Table 3: OpenVLA training data mixture using datasets from the Open X-Embodiment dataset [1], following [5] with a few additions.

OpenVLA Training Dataset Mixture
Fractal [92]	12.7%
Kuka [45]	12.7%
Bridge[47, 6]	13.3%
Taco Play [93, 94]	3.0%
Jaco Play [95]	0.4%
Berkeley Cable Routing [96]	0.2%
Roboturk [97]	2.3%
Viola [98]	0.9%
Berkeley Autolab UR5 [99]	1.2%
Toto [100]	2.0%
Language Table [101]	4.4%
Stanford Hydra Dataset [102]	4.4%
Austin Buds Dataset [103]	0.2%
NYU Franka Play Dataset [104]	0.8%
Furniture Bench Dataset [105]	2.4%
UCSD Kitchen Dataset [106]	<0.1%
Austin Sailor Dataset [107]	2.2%
Austin Sirius Dataset [108]	1.7%
DLR EDAN Shared Control [109]	<0.1%
IAMLab CMU Pickup Insert [110]	0.9%
UTAustin Mutex [111]	2.2%
Berkeley Fanuc Manipulation [112]	0.7%
CMU Stretch [113]	0.2%
BC-Z [55]	7.5%
FMB Dataset [114]	7.1%
DobbE [115]	1.4%
DROID [11]	10.0%⁶

OpenVLA 训练数据配比 Fractal [92] 12.7% Kuka [45] 12.7% Bridge[47, 6] 13.3% Taco Play [93, 94] 3.0% Jaco Play [95] 0.4% Berkeley Cable Routing [96] 0.2% Roboturk [97] 2.3% Viola [98] 0.9% Berkeley Autolab UR5 [99] 1.2% Toto [100] 2.0% Language Table [101] 4.4% Stanford Hydra Dataset [102] 4.4% Austin Buds Dataset [103] 0.2% NYU Franka Play Dataset [104] 0.8% Furniture Bench Dataset [105] 2.4% UCSD Kitchen Dataset [106] <0.1% Austin Sailor Dataset [107] 2.2% Austin Sirius Dataset [108] 1.7% DLR EDAN Shared Control [109] <0.1% IAMLab CMU Pickup Insert [110] 0.9% UTAustin Mutex [111] 2.2% Berkeley Fanuc Manipulation [112] 0.7% CMU Stretch [113] 0.2% BC-Z [55] 7.5% FMB Dataset [114] 7.1% DobbE [115] 1.4% DROID [11] 10.0%⁶ 表 3：OpenVLA 的训练数据配比，使用 Open X-Embodiment 数据集 [1] 中的数据集，沿用 [5] 并做了少量补充。

OpenVLA 训练数据配比
Fractal [92]	12.7%
Kuka [45]	12.7%
Bridge[47, 6]	13.3%
Taco Play [93, 94]	3.0%
Jaco Play [95]	0.4%
Berkeley Cable Routing [96]	0.2%
Roboturk [97]	2.3%
Viola [98]	0.9%
Berkeley Autolab UR5 [99]	1.2%
Toto [100]	2.0%
Language Table [101]	4.4%
Stanford Hydra Dataset [102]	4.4%
Austin Buds Dataset [103]	0.2%
NYU Franka Play Dataset [104]	0.8%
Furniture Bench Dataset [105]	2.4%
UCSD Kitchen Dataset [106]	<0.1%
Austin Sailor Dataset [107]	2.2%
Austin Sirius Dataset [108]	1.7%
DLR EDAN Shared Control [109]	<0.1%
IAMLab CMU Pickup Insert [110]	0.9%
UTAustin Mutex [111]	2.2%
Berkeley Fanuc Manipulation [112]	0.7%
CMU Stretch [113]	0.2%
BC-Z [55]	7.5%
FMB Dataset [114]	7.1%
DobbE [115]	1.4%
DROID [11]	10.0%⁶

⁶ We remove DROID for the last third of training due to slow learning progress (see Section 3.3) and re-distribute its mixture weights across all other datasets.

⁶ 由于学习进展缓慢，我们在训练的最后三分之一阶段移除了 DROID（见第 3.3 节），并把它的配比权重重新分配给其他所有数据集。

Appendix B Evaluation Tasks and Detailed Results

附录 B 评测任务与详细结果

In this section, we provide more details on the BridgeData V2 WidowX and Google robot evaluations discussed in Section 5.1, as well as the Franka-Tabletop and Franka-DROID fine-tuning evaluations discussed in Section 5.2.

本节提供第 5.1 节讨论的 BridgeData V2 WidowX 和 Google 机器人评测，以及第 5.2 节讨论的 Franka-Tabletop 和 Franka-DROID 微调评测的更多细节。

B.1 BridgeData V2 WidowX Evaluation Details

B.1 BridgeData V2 WidowX 评测细节

Here we focus specifically on BridgeData V2 evaluations discussed in Section 5.1.

这里专门介绍第 5.1 节讨论的 BridgeData V2 评测。

B.1.1 BridgeData V2 Evaluation Tasks As described in Section 5.1, we evaluate each generalist robot manipulation policy on 17 tasks with 10 trials each. In this section, we provide details on the task categories and individual tasks.

B.1.1 BridgeData V2 评测任务 如第 5.1 节所述，我们在 17 个任务上评测每个通用机器人操作策略，每个任务 10 次试验。本节给出任务类别和各个任务的细节。

Figure 7: BridgeData V2 WidowX robot evaluation tasks. We evaluate every generalist robot policy on 4 types out-of-distribution (OOD) generalization tasks: visual, motion, physical, and semantic (as defined in Section 5.1). Every pair of images shows the start state and an example end state after the robot completes the task. We also rigorously assess language grounding in the 3 tasks shown in the bottom 3 rows, by changing the prompt while fixing the initial state and testing whether the policy can approach the correct target object.

图 7： BridgeData V2 WidowX 机器人评测任务 。我们在 4 类分布外（OOD）泛化任务上评测每个通用机器人策略： 视觉 、 运动 、 物理 和 语义 泛化（定义见第 5.1 节）。每对图像展示起始状态，以及机器人完成任务后的一个示例结束状态。我们还在最下面 3 行所示的 3 个任务中严格评估 语言落地 能力：固定初始状态、改变提示语，测试策略能否接近正确的目标物体。

In total, we evaluate on 5 visual generalization tasks, 2 motion generalization tasks, 3 physical generalization tasks, 4 semantic generalization tasks, and 3 language grounding tasks. Note that all tasks we evaluate on introduce some form of distribution shift since we are unable to procure the exact objects used in the original dataset (other distribution shifts naturally arise as we reproduce a real-world test environment originally constructed at a different location; see Section B.1.2 for a detailed discussion on such distribution shifts). All 17 tasks are depicted in Fig. 7. Each rollout is marked as a failure (0) or success (1). In some more difficult tasks, we record partial successes (0.5); we describe the conditions for partial credit in the task descriptions below.

我们总共评测 5 个视觉泛化任务、2 个运动泛化任务、3 个物理泛化任务、4 个语义泛化任务和 3 个语言落地任务。注意，我们评测的所有任务都引入了某种形式的分布偏移，因为我们无法获得原始数据集中使用的那些物体（我们在别处重建一个原本搭建在另一个地点的真实测试环境，也会自然产生其他分布偏移；关于这类分布偏移的详细讨论见 B.1.2 节）。全部 17 个任务见图 7。每次试验记为失败（0）或成功（1）。在一些较难的任务中，我们记录部分成功（0.5）；获得部分分数的条件在下面的任务描述中说明。

Below we describe each of the 17 tasks, in the order shown in Fig. 7:

下面按图 7 中的顺序逐一描述这 17 个任务：

B.1.2 Comparing Evaluation Tasks to Original BridgeData V2 Training Data We conduct our evaluations in a sink environment used in the original BridgeData V2 dataset [6]. We reproduce the environment to match the original environment in the BridgeData V2 dataset with rough approximations for the robot’s location relative to the sink, as well as the camera’s placement relative to the scene. Given the lack of precise measurements of these positions in the original dataset, we are unable to reproduce the exact environment setup, and natural distribution shifts arise due to slightly different robot, sink, and camera placements. In addition, since we evaluate robot policies in a different location than where the training demonstrations were collected from, other natural distribution shifts arise. For example, the lighting conditions and background (e.g., visible areas behind the sink) are inevitably different than what was seen in the training dataset. Furthermore, we are unable to procure the exact set of objects used in the original BridgeData V2 dataset, so there are distribution shifts between the objects used at train time and those used at test time.

B.1.2 评测任务与原始 BridgeData V2 训练数据的比较 我们在原始 BridgeData V2 数据集 [6] 所用的一个水槽环境中进行评测。我们重建了这个环境，使之与 BridgeData V2 数据集中的原始环境相匹配，对机器人相对水槽的位置以及相机相对场景的摆放做了粗略近似。由于原始数据集中缺少这些位置的精确测量，我们无法重现 完全一致 的环境配置，机器人、水槽和相机的摆放略有不同，自然会产生分布偏移。此外，由于我们评测机器人策略的地点与采集训练示教的地点不同，还会产生其他自然的分布偏移。例如，光照条件和背景（例如水槽后面可见的区域）不可避免地与训练数据集中看到的不同。再者，我们无法获得原始 BridgeData V2 数据集中所用的那套物体，因此训练时和测试时所用的物体之间存在分布偏移。

Despite all these challenges, we find that certain generalist policies, such as OpenVLA and RT-2-X, can still generalize and perform various tasks fairly reliably “out-of-the-box”. Other generalist policies, such as RT-1-X and Octo, can also complete some tasks, though they struggle when tested with more difficult generalization tasks in our BridgeData V2 evaluation suite.

尽管有这些挑战，我们发现某些通用策略，例如 OpenVLA 和 RT-2-X，仍然能够“开箱即用”地泛化，并相当可靠地完成各种任务。其他通用策略，例如 RT-1-X 和 Octo，也能完成部分任务，但在我们 BridgeData V2 评测套件中较难的泛化任务上表现吃力。

The original BridgeData V2 dataset includes demonstrations of the following seven tasks in this specific sink environment: “Flip Pot Upright”, “Put Carrot on Plate”, “Put Cup from Counter (or Drying Rack) into Sink”, “Put Eggplant into Pot”, “Put Knife on Cutting Board”, “Put Spoon in Pot”, and “Turn Lever Vertical to Front”. See Fig. 8 for samples images of all these tasks from the original dataset. Note that all training demonstrations collected in this environment are initialized such that the robot’s end-effector is positioned directly above the target object in the beginning of the episode. (However, this is not the case across all environments in the BridgeData V2 dataset; in some other environments, the robot is initialized farther away from the target object, so it must horizontally reach for the object first before manipulating it.)

原始 BridgeData V2 数据集在这个水槽环境中包含以下七个任务的示教：“把锅翻正”“把胡萝卜放到盘子上”“把杯子从台面（或沥水架）放进水槽”“把茄子放进锅里”“把刀放到砧板上”“把勺子放进锅里”和“把拉杆转到竖直朝前”。原始数据集中所有这些任务的样例图像见图 8。注意，在这个环境中采集的所有训练示教，在回合开始时都把机器人末端执行器初始化在目标物体正上方。（不过，并不是 BridgeData V2 数据集的所有环境都如此；在其他一些环境中，机器人被初始化在离目标物体较远的位置，因此必须先水平伸向物体，再操作它。）

Figure 8: Original BridgeData V2 sink environment tasks. Images from sample demonstrations in the sink environment from the original BridgeData V2 dataset reveal that all demonstrations in this environment were initialized such that the robot’s end-effector was positioned immediately above the target object. Note that these initial states are different from the initial states we use in our BridgeData V2 evaluation tasks shown in Fig. 7. In our evaluations, we always initialize the robot’s end-effector to a fixed location above the sink, rather than positioning it directly above the target object (except for one task: “Put Eggplant into Pot (Easy Version)”).

图 8： 原始 BridgeData V2 水槽环境中的任务 。原始 BridgeData V2 数据集水槽环境中的示教样例图像显示，这个环境中所有示教的初始状态都是机器人末端执行器位于目标物体正上方。注意，这些初始状态与我们在图 7 所示的 BridgeData V2 评测任务中所用的初始状态不同。在我们的评测中，我们总是把机器人末端执行器初始化在水槽上方的一个固定位置，而不是直接位于目标物体上方（只有一个任务例外：“把茄子放进锅里（简单版）”）。

In our BridgeData V2 evaluation suite, only one task - “Put Eggplant into Pot (Easy Version”) - is initialized with the robot’s end-effector hovering directly over the target object; in all 16 other tasks, the end-effector is initialized at a fixed location above the sink such that the robot must horizontally reach towards the object. This initial condition, in combination with the distribution shifts we introduce in the various types of OOD generalization in our evaluation suite, challenges the generalist policies and requires a high degree of robustness in order to complete the tasks successfully. Hence, the success rates for policies like RT-1-X and Octo are lower than what is reported in prior works. However, we find that other policies such as RT-2-X and OpenVLA still achieve relatively strong performance despite all these distribution shifts and challenges.

在我们的 BridgeData V2 评测套件中，只有一个任务——“把茄子放进锅里（简单版）”——在初始化时让机器人末端执行器悬停在目标物体正上方；在其他全部 16 个任务中，末端执行器都被初始化在水槽上方的一个固定位置，机器人必须水平伸向物体。这种初始条件，加上我们在评测套件中各类 OOD 泛化里引入的分布偏移，对通用策略构成挑战，要成功完成任务需要很高的鲁棒性。因此，RT-1-X 和 Octo 等策略的成功率低于先前工作中报告的数字。不过我们发现，RT-2-X 和 OpenVLA 等其他策略尽管面临这些分布偏移和挑战，仍取得了相对强劲的性能。

B.1.3 Detailed BridgeData V2 Evaluation Results See Table 4 for the full BridgeData V2 WidowX evaluation results. The number of successes for each method, out of 10 trials, is listed for each of 17 tasks. OpenVLA achieves strongest performance in the majority of the tasks and has the highest aggregate success rate among the generalist policies. RT-2-X also shows good performance, outperforming RT-1-X and Octo, though it does not perform as well as OpenVLA. RT-1-X and Octo generally experience difficulty in these generalization tasks.

B.1.3 BridgeData V2 详细评测结果 完整的 BridgeData V2 WidowX 评测结果见表 4。表中列出了每种方法在 17 个任务中每个任务 10 次试验里的成功次数。OpenVLA 在大多数任务上表现最强，在通用策略中综合成功率最高。RT-2-X 也表现不错，优于 RT-1-X 和 Octo，但不如 OpenVLA。RT-1-X 和 Octo 在这些泛化任务上普遍表现吃力。

\pm

Category	Task	# Trials	RT-1-X # Successes	Octo # Successes	RT-2-X # Successes	OpenVLA (ours) # Successes
Visual gen	Put Eggplant into Pot (Easy Version)	10	1	5	7	10
Visual gen	Put Eggplant into Pot	10	0	1	5	10
Visual gen	Put Cup from Counter into Sink	10	1	1	0	7
Visual gen	Put Eggplant into Pot (w/ Clutter)	10	1	3.5	6	7.5
Visual gen	Put Yellow Corn on Pink Plate	10	1	4	8	9
Motion gen	Lift Eggplant	10	3	0.5	6.5	7.5
Motion gen	Put Carrot on Plate (w/ Height Change)	10	2	1	4.5	4.5
Physical gen	Put Carrot on Plate	10	1	0	1	8
Physical gen	Flip Pot Upright	10	2	6	5	8
Physical gen	Lift AAA Battery	10	0	0	2	7
Semantic gen	Move Skull into Drying Rack	10	1	0	5	5
Semantic gen	Lift White Tape	10	3	0	0	1
Semantic gen	Take Purple Grapes out of Pot	10	6	0	5	4
Semantic gen	Stack Blue Cup on Pink Cup	10	0.5	0	5.5	4.5
Language grounding	Put {Eggplant, Red Bottle} into Pot	10	2.5	4	8.5	7.5
Language grounding	Lift {Cheese, Red Chili Pepper}	10	1.5	2.5	8.5	10
Language grounding	Put {Blue Cup, Pink Cup} on Plate	10	5	5.5	8.5	9.5
		Mean Success Rate	18.5 $\pm$ 2.7%	20.0 $\pm$ 2.6%	50.6 $\pm$ 3.5%	70.6 $\pm$ 3.2%

\pm

类别	任务	试验次数	RT-1-X 成功次数	Octo 成功次数	RT-2-X 成功次数	OpenVLA (ours) 成功次数
视觉泛化	把茄子放进锅里（简单版）	10	1	5	7	10
视觉泛化	把茄子放进锅里	10	0	1	5	10
视觉泛化	把杯子从台面放进水槽	10	1	1	0	7
视觉泛化	把茄子放进锅里（有杂物）	10	1	3.5	6	7.5
视觉泛化	把黄玉米放到粉色盘子上	10	1	4	8	9
运动泛化	举起茄子	10	3	0.5	6.5	7.5
运动泛化	把胡萝卜放到盘子上（高度变化）	10	2	1	4.5	4.5
物理泛化	把胡萝卜放到盘子上	10	1	0	1	8
物理泛化	把锅翻正	10	2	6	5	8
物理泛化	举起 AAA 电池	10	0	0	2	7
语义泛化	把骷髅放进沥水架	10	1	0	5	5
语义泛化	举起白色胶带	10	3	0	0	1
语义泛化	把紫葡萄从锅里拿出来	10	6	0	5	4
语义泛化	把蓝杯子叠到粉杯子上	10	0.5	0	5.5	4.5
语言落地	把 {茄子, 红瓶子} 放进锅里	10	2.5	4	8.5	7.5
语言落地	举起 {奶酪, 红辣椒}	10	1.5	2.5	8.5	10
语言落地	把 {蓝杯子, 粉杯子} 放到盘子上	10	5	5.5	8.5	9.5
		平均成功率	18.5 $\pm$ 2.7%	20.0 $\pm$ 2.6%	50.6 $\pm$ 3.5%	70.6 $\pm$ 3.2%

Additionally, in Table 5, we provide the full evaluation results for the quantized inference experiments that were summarized in Table 2. For these evaluations, we test policies on 8 representative BridgeData V2 tasks spanning all task categories in the full evaluation suite.

此外，我们在表 5 中给出量化推理实验的完整评测结果，其汇总见表 2。这些评测在 8 个有代表性的 BridgeData V2 任务上测试策略，这些任务覆盖完整评测套件中的所有任务类别。

\pm

Category	Task	# Trials	bfloat16 # Successes	int8 # Successes	int4 # Successes
Visual gen	Put Eggplant into Pot (Easy Version)	10	9	7	9
Visual gen	Put Eggplant into Pot	10	7	7	7
Visual gen	Put Cup from Counter into Sink	10	5	3	7
Motion gen	Lift Eggplant	10	6	4	7.5
Physical gen	Put Carrot on Plate	10	6	5	7
Physical gen	Lift AAA Battery	10	7	5	3
Semantic gen	Take Purple Grapes out of Pot	10	8	8	9
Language grounding	Put {Eggplant, Red Bottle} into Pot	10	9	7.5	8
		Mean Success Rate	71.3 $\pm$ 4.8%	58.1 $\pm$ 5.1%	71.9 $\pm$ 4.7%

\pm

类别	任务	试验次数	bfloat16 成功次数	int8 成功次数	int4 成功次数
视觉泛化	把茄子放进锅里（简单版）	10	9	7	9
视觉泛化	把茄子放进锅里	10	7	7	7
视觉泛化	把杯子从台面放进水槽	10	5	3	7
运动泛化	举起茄子	10	6	4	7.5
物理泛化	把胡萝卜放到盘子上	10	6	5	7
物理泛化	举起 AAA 电池	10	7	5	3
语义泛化	把紫葡萄从锅里拿出来	10	8	8	9
语言落地	把 {茄子, 红瓶子} 放进锅里	10	9	7.5	8
		平均成功率	71.3 $\pm$ 4.8%	58.1 $\pm$ 5.1%	71.9 $\pm$ 4.7%

B.2 Google Robot Evaluation Details

B.2 Google 机器人评测细节

In this section, we provide more details on the Google robot evaluations introduced in Section 5.1.

本节提供第 5.1 节介绍的 Google 机器人评测的更多细节。

B.2.1 Google Robot Evaluation Tasks Figure 9: Google robot evaluation tasks. We evaluate every generalist robot policy on in-distribution tasks and out-of-distribution (OOD) generalization tasks. OOD tasks involve unseen backgrounds, target objects, instructions/object relations, and semantic concepts (e.g., photos from the Internet that do not appear in robot action data).

B.2.1 Google 机器人评测任务 图 9： Google 机器人评测任务 。我们在分布内任务和分布外（OOD）泛化任务上评测每个通用机器人策略。OOD 任务涉及未见过的背景、目标物体、指令/物体关系，以及语义概念（例如机器人动作数据中没有出现过的互联网照片）。

On the Google robot, we evaluate each generalist robot policy on 12 tasks with 5 rollouts each, for a total of 60 rollouts. The first five tasks test on in-distribution conditions, and the last seven tasks test on more difficult out-of-distribution (OOD) conditions. All tasks are depicted in Fig. 9. Each rollout is marked as a failure (0) or success (1).

在 Google 机器人上，我们在 12 个任务上评测每个通用机器人策略，每个任务 5 次试验，共 60 次。前五个任务在分布内条件下测试，后七个任务在更难的分布外（OOD）条件下测试。所有任务见图 9。每次试验记为失败（0）或成功（1）。

We describe the 12 tasks below:

下面描述这 12 个任务：

⁷ See Appendix of Brohan et al. [7] for a detailed list of OOD conditions in Google robot evaluations.

⁷ Google 机器人评测中 OOD 条件的详细列表见 Brohan 等人 [7] 的附录。

\pm

Category	Task	# Trials	RT-1-X # Successes	Octo # Successes	RT-2-X # Successes	OpenVLA (ours) # Successes
In-distribution	Pick Coke Can	5	5	1	5	5
In-distribution	Move Apple near Green Can	5	3	3	3	5
In-distribution	Move Blue Chip Bag near Apple	5	0	3	4	5
In-distribution	Place Coke Can Upright	5	0	0	4	4
In-distribution	Open Middle Drawer	5	0	4	2	3
OOD	Move Orange near Brown Chip Bag	5	1	2	5	5
OOD	Pick Pepsi Can	5	3	0	5	4
OOD	Pick Banana	5	5	3	5	5
OOD	Pick Green Cup	5	1	0	5	5
OOD	Place Apple on Plate	5	0	0	4	4
OOD	Place Banana in Pan	5	0	0	2	4
OOD	Move Coke Can near Taylor Swift	5	2	0	3	2
		Mean Success Rate	33.3 $\pm$ 6.1%	26.7 $\pm$ 5.8%	78.3 $\pm$ 5.4%	85.0 $\pm$ 4.6%

\pm

类别	任务	试验次数	RT-1-X 成功次数	Octo 成功次数	RT-2-X 成功次数	OpenVLA (ours) 成功次数
分布内	拿起可乐罐	5	5	1	5	5
分布内	把苹果移到绿色罐子旁	5	3	3	3	5
分布内	把蓝色薯片袋移到苹果旁	5	0	3	4	5
分布内	把可乐罐立起来	5	0	0	4	4
分布内	打开中间的抽屉	5	0	4	2	3
分布外	把橙子移到棕色薯片袋旁	5	1	2	5	5
分布外	拿起百事可乐罐	5	3	0	5	4
分布外	拿起香蕉	5	5	3	5	5
分布外	拿起绿色杯子	5	1	0	5	5
分布外	把苹果放到盘子上	5	0	0	4	4
分布外	把香蕉放进平底锅	5	0	0	2	4
分布外	把可乐罐移到泰勒·斯威夫特附近	5	2	0	3	2
		平均成功率	33.3 $\pm$ 6.1%	26.7 $\pm$ 5.8%	78.3 $\pm$ 5.4%	85.0 $\pm$ 4.6%

Full results for the Google robot evaluations are shown in Table 6. Overall, we find that RT-1-X and Octo experience difficulty on the evaluation tasks; they are often unable to achieve a single success out of five trials in several tasks. On the other hand, RT-2-X and OpenVLA demonstrate strong performance, completing every task at least two times out of five trials; these two VLA policies perform comparably with each other on this particular evaluation suite.

Google 机器人评测的完整结果见表 6。总体而言，我们发现 RT-1-X 和 Octo 在评测任务上表现吃力；在若干任务上，它们五次试验连一次都没能成功。另一方面，RT-2-X 和 OpenVLA 表现强劲，每个任务五次试验中至少完成两次；这两个 VLA 策略在这套评测上表现相当。

B.3 Data-Efficient Adaptation Experiment Details

B.3 数据高效适配实验细节

In this section, we provide more details on the data-efficient adaptation experiments discussed in Section 5.2, where we investigate the effectiveness of fine-tuned OpenVLA policies on new robot setups such as Franka-Tabletop and Franka-DROID.

本节提供第 5.2 节讨论的数据高效适配实验的更多细节，该实验研究微调后的 OpenVLA 策略在 Franka-Tabletop 和 Franka-DROID 等新机器人配置上的效果。

B.3.1 Franka-Tabletop and Franka-DROID Tasks We collect 10-150 demonstrations of each of seven tasks. The first six tasks correspond to a robot setup which we denote as “Franka-Tabletop” (Franka Emika Panda robot mounted on top of a table), and the final task corresponds to a robot setup which we call “Franka-DROID”.

B.3.1 Franka-Tabletop 与 Franka-DROID 任务 我们为七个任务各采集 10-150 条示教。前六个任务对应我们称为“Franka-Tabletop”的机器人配置（安装在桌面上的 Franka Emika Panda 机器人），最后一个任务对应我们称为“Franka-DROID”的机器人配置。

In the Franka-Tabletop setup, the first three of six tasks correspond to single-instruction tasks and are narrow, while the last three tasks correspond to multi-instruction tasks in which multiple objects are present in the scene and the robot must manipulate the correct one depending on the language instruction.

在 Franka-Tabletop 配置中，六个任务里的前三个是单指令任务，范围较窄；后三个是多指令任务，场景中有多个物体，机器人必须根据语言指令操作正确的那个。

Figure 10: Franka-Tabletop fine-tuning tasks. Franka-Tabletop tasks used in the data-efficient adaptation experiments in Section 5.2 and described in detail in Fig. 10 are depicted above. The first three of six tasks, shown in the top three rows, only involve a single instruction, while the last three tasks in the bottom three rows involve multiple objects and instructions (the instructions specify the target object or target location). The first column shows sample initial states matching the training data distribution, while the second column shows out-of-distribution (OOD) initial states (e.g., unseen backgrounds, target objects, distractors, and object positions/orientations). Every policy in Section 5.2 is evaluated with 10-12 rollouts on in-distribution tasks and 5-6 rollouts on OOD tasks.

图 10： Franka-Tabletop 微调任务 。上图展示了第 5.2 节数据高效适配实验中使用、并在图 10 中详细描述的 Franka-Tabletop 任务。六个任务中的前三个（上面三行）只涉及单条指令，后三个任务（下面三行）涉及多个物体和多条指令（指令指定目标物体或目标位置）。第一列展示与训练数据分布一致的示例初始状态，第二列展示分布外（OOD）初始状态（例如未见过的背景、目标物体、干扰物以及物体位置/朝向）。第 5.2 节中的每个策略在分布内任务上评测 10-12 次，在 OOD 任务上评测 5-6 次。 译注：图注中“在图 10 中详细描述”指向图 10 自身，原文如此；任务的文字描述实际在 B.3.1 节正文中。

Below we describe each of the six Franka-Tabletop tasks shown in Fig. 10:

下面描述图 10 所示的六个 Franka-Tabletop 任务：

For every Franka-Tabletop task, we evaluate each method with 10-12 in-distribution trials and 5-6 OOD generalization trials. The in-distribution and OOD test conditions are depicted in Fig. 10 (second column).

对每个 Franka-Tabletop 任务，我们用 10-12 次分布内试验和 5-6 次 OOD 泛化试验评测每种方法。分布内和 OOD 测试条件见图 10（第二列）。

We describe the OOD test conditions for each of the six tasks below:

下面描述六个任务各自的 OOD 测试条件：

Put Carrot in Bowl (OOD): An eggplant (unseen object) replaces the carrot. Pour Corn into Pot (OOD): An unseen brown tablecloth covers the tabletop. Flip Pot Upright (OOD): An unseen white tablecloth covers the tabletop Move <object> onto Plate (OOD): A set of three unseen objects are placed on the table. Knock <object> Over (OOD): Two unseen distractor objects (red plastic cup and brown box) are positioned behind the set of three seen objects. Cover <object> with Towel (OOD): The three objects on the table are placed upside-down and at unseen positions.

把胡萝卜放进碗里（OOD）：用一根茄子（未见过的物体）代替胡萝卜。 把玉米倒进锅里（OOD）：桌面上铺一块未见过的棕色桌布。 把锅翻正（OOD）：桌面上铺一块未见过的白色桌布。 把 <object> 移到盘子上（OOD）：桌上放一组三个未见过的物体。 把 <object> 推倒（OOD）：在三个见过的物体后面放两个未见过的干扰物（红色塑料杯和棕色盒子）。 用毛巾盖住 <object>（OOD）：桌上的三个物体倒扣放置，并处在未见过的位置。

Finally, in the Franka-DROID environment, we experiment with one task and variants of it: Wipe Table (see Fig. 11). In this task, the robot’s goal is to grab the brush and sweep all three small brown objects into the dustpan. We collect 70 demonstrations for this task for the training dataset, varying the positions of all the objects.

最后，在 Franka-DROID 环境中，我们用一个任务及其变体做实验： 擦桌子 （见图 11）。在这个任务中，机器人的目标是抓起刷子，把三个棕色小物体全部扫进簸箕。我们为训练数据集采集了这个任务的 70 条示教，每次改变所有物体的位置。

Figure 11: Franka-DROID fine-tuning task. The “Wipe Table” task shown here is the final task used in the data-efficient adaptation experiments in Section 5.2. The left image shows the initial conditions for an in-distribution trial. The right image shows an out-of-distribution trial in which unseen distractor objects are present on the table. To fully complete the task, the robot must grab the brush and sweep all three objects into the dustpan.

图 11： Franka-DROID 微调任务 。这里展示的“擦桌子”任务是第 5.2 节数据高效适配实验中使用的最后一个任务。左图展示一次分布内试验的初始条件。右图展示一次分布外试验，桌上有未见过的干扰物体。要完全完成任务，机器人必须抓起刷子，把三个物体全部扫进簸箕。

At test time, we evaluate on in-distribution conditions matching the training data (Fig. 11, left), as well as out-of-distribution (OOD) conditions in which distractor objects are also present in the scene on the table (Fig. 11, right). Since there are various possible outcomes for each trial, we define a scoring rubric as follows: The maximum score for each trial is 2 points. The policy receives the full 2 points if the robot sweeps all three objects into the dustpan. It receives 1 point for successfully sweeping one or two objects into the dustpan. Otherwise, it receives 0 points. We evaluate each policy with 18 in-distribution trials and 12 OOD trials, so each policy receives an aggregate score out of 60 points.

测试时，我们既在与训练数据一致的分布内条件下评测（图 11 左），也在场景中桌上还有干扰物体的分布外（OOD）条件下评测（图 11 右）。由于每次试验可能有多种结果，我们定义如下评分标准：每次试验满分 2 分。如果机器人把三个物体全部扫进簸箕，策略得满分 2 分；成功把一个或两个物体扫进簸箕得 1 分；否则得 0 分。我们用 18 次分布内试验和 12 次 OOD 试验评测每个策略，因此每个策略的总分满分为 60 分。

B.3.2 Detailed Franka-Tabletop and Franka-DROID Evaluation Results Full evaluation results for both Franka-Tabletop and Franka-DROID evaluations are shown in Table 7. We evaluate the methods discussed in Section 5.2. We find that Diffusion Policy demonstrates strong performance on the single-instruction Franka-Tabletop tasks (e.g., “Put Carrot in Bowl” and “Pour Corn in Pot”), outperforming other methods. However, OpenVLA and Octo achieve higher performance in the more diverse multi-instruction tasks (“Move <object> onto Plate”, “Knock <object> Over”, and “Cover <object> with Towel”). In the Franka-DROID environment, OpenVLA obtains best results. Overall, we find that OpenVLA achieves the highest average performance across both tasks.

B.3.2 Franka-Tabletop 与 Franka-DROID 详细评测结果 Franka-Tabletop 和 Franka-DROID 评测的完整结果见表 7。我们评测第 5.2 节讨论的方法。我们发现，Diffusion Policy 在单指令的 Franka-Tabletop 任务上（例如“把胡萝卜放进碗里”和“把玉米倒进锅里”）表现强劲，优于其他方法。然而在更多样化的多指令任务（“把 <object> 移到盘子上”“把 <object> 推倒”和“用毛巾盖住 <object>”）上，OpenVLA 和 Octo 性能更高。在 Franka-DROID 环境中，OpenVLA 取得了最好的结果。总体而言，我们发现 OpenVLA 在两类任务上的平均性能最高。

\pm

		# trials	Diffusion Policy	Diffusion Policy (matched)	Octo	OpenVLA (scratch)	OpenVLA (ours)
Franka-Tabletop (5Hz)	“Put Carrot in Bowl” (in-distribution)	10	90.0%	80.0%	40.0%	70.0%	70.0%
	“Put Carrot in Bowl” (OOD)	5	20.0%	0.0%	20.0%	0.0%	40.0%
	“Pour Corn into Pot” (in-distribution)	10	100.0%	90.0%	0.0%	10.0%	50.0%
	“Pour Corn into Pot” (OOD)	5	80.0%	60.0%	0.0%	20.0%	60.0%
	“Flip Pot Upright” (in-distribution)	10	100.0%	85.0%	40.0%	85.0%	100.0%
	“Flip Pot Upright” (OOD)	5	50.0%	20.0%	0.0%	40.0%	80.0%
	“Move <object> onto Plate” (in-distribution)	12	25.0%	25.0%	41.7%	8.3%	75.0%
	“Move <object> onto Plate” (OOD)	6	8.3%	33.3%	8.3%	33.3%	58.3%
	“Knock <object> Over” (in-distribution)	12	33.3%	25.0%	83.3%	75.0%	75.0%
	“Knock <object> Over” (OOD)	6	16.7%	16.7%	33.3%	58.3%	83.3%
	“Cover <object> with Towel” (in-distribution)	12	16.7%	20.8%	91.7%	41.7%	50.0%
	“Cover <object> with Towel” (OOD)	6	16.7%	33.3%	91.7%	50.0%	50.0%
	Average		48.5 $\pm$ 4.9%	43.4 $\pm$ 4.7%	43.4 $\pm$ 4.4%	43.4 $\pm$ 4.6%	67.2 $\pm$ 4.0%
Franka-DROID (15Hz)	“Wipe Table” (in-distribution)	18	50.0%	27.8%	52.8%	25.0%	55.6%
	“Wipe Table” + Distractors (OOD)	12	12.5%	25.0%	16.7%	16.7%	62.5%
	Average		35.0 $\pm$ 8.0%	26.7 $\pm$ 7.5%	38.3 $\pm$ 8.5%	21.7 $\pm$ 6.6%	58.3 $\pm$ 7.2%

\pm

		试验次数	Diffusion Policy	Diffusion Policy（matched）	Octo	OpenVLA（scratch）	OpenVLA（本文）
Franka-Tabletop (5Hz)	“把胡萝卜放进碗里”（分布内）	10	90.0%	80.0%	40.0%	70.0%	70.0%
	“把胡萝卜放进碗里”（OOD）	5	20.0%	0.0%	20.0%	0.0%	40.0%
	“把玉米倒进锅里”（分布内）	10	100.0%	90.0%	0.0%	10.0%	50.0%
	“把玉米倒进锅里”（OOD）	5	80.0%	60.0%	0.0%	20.0%	60.0%
	“把锅翻正”（分布内）	10	100.0%	85.0%	40.0%	85.0%	100.0%
	“把锅翻正”（OOD）	5	50.0%	20.0%	0.0%	40.0%	80.0%
	“把 <object> 移到盘子上”（分布内）	12	25.0%	25.0%	41.7%	8.3%	75.0%
	“把 <object> 移到盘子上”（OOD）	6	8.3%	33.3%	8.3%	33.3%	58.3%
	“把 <object> 推倒”（分布内）	12	33.3%	25.0%	83.3%	75.0%	75.0%
	“把 <object> 推倒”（OOD）	6	16.7%	16.7%	33.3%	58.3%	83.3%
	“用毛巾盖住 <object>”（分布内）	12	16.7%	20.8%	91.7%	41.7%	50.0%
	“用毛巾盖住 <object>”（OOD）	6	16.7%	33.3%	91.7%	50.0%	50.0%
	平均		48.5 $\pm$ 4.9%	43.4 $\pm$ 4.7%	43.4 $\pm$ 4.4%	43.4 $\pm$ 4.6%	67.2 $\pm$ 4.0%
Franka-DROID (15Hz)	“擦桌子”（分布内）	18	50.0%	27.8%	52.8%	25.0%	55.6%
	“擦桌子” + 干扰物（OOD）	12	12.5%	25.0%	16.7%	16.7%	62.5%
	平均		35.0 $\pm$ 8.0%	26.7 $\pm$ 7.5%	38.3 $\pm$ 8.5%	21.7 $\pm$ 6.6%	58.3 $\pm$ 7.2%

Additionally, in Table 8, we show the detailed version of the parameter-efficient fine-tuning experiment results summarized in Table 1. In these experiments, we use a representative subset of two Franka-Tabletop tasks, with both in-distribution and OOD variants: one narrow single-instruction task (“Put Carrot in Bowl”) and one diverse multi-instruction task (“Move <object> onto Plate”). We use the same number of training demonstrations used in Section 5.2 (50 and 150, respectively), which is delineated in Section B.3.1.

此外，我们在表 8 中给出表 1 所汇总的参数高效微调实验结果的详细版本。在这些实验中，我们使用两个 Franka-Tabletop 任务组成的代表性子集，每个任务都有分布内和 OOD 两种变体：一个是狭窄的单指令任务（“把胡萝卜放进碗里”），一个是多样化的多指令任务（“把 <object> 移到盘子上”）。我们使用与第 5.2 节相同数量的训练示教（分别为 50 条和 150 条），具体见 B.3.1 节。

\pm

		# trials	Full FT	Last layer only	Frozen vision	Sandwich	LoRA, r=32	LoRA, r=64
Franka-Tabletop (5Hz)	“Put Carrot in Bowl” (in-distribution)	10	90.0	40.0	40.0	90.0	60.0	90.0
	“Put Carrot in Bowl” (OOD)	5	40.0	0.0	40.0	0.0	60.0	40.0
	“Move <object> onto Plate” (in-distribution)	12	79.2	33.3	50.0	75.0	75.0	62.5
	“Move <object> onto Plate” (OOD)	6	41.7	33.3	58.3	41.7	75.0	66.7
	Average		69.7 $\pm$ 7.2%	30.3 $\pm$ 6.1%	47.0 $\pm$ 6.9%	62.1 $\pm$ 7.9%	68.2 $\pm$ 7.5%	68.2 $\pm$ 7.8%

\pm

		试验次数	全参数微调	仅最后一层	冻结视觉编码器	三明治微调	LoRA，r=32	LoRA，r=64
Franka-Tabletop (5Hz)	“把胡萝卜放进碗里”（分布内）	10	90.0	40.0	40.0	90.0	60.0	90.0
	“把胡萝卜放进碗里”（OOD）	5	40.0	0.0	40.0	0.0	60.0	40.0
	“把 <object> 移到盘子上”（分布内）	12	79.2	33.3	50.0	75.0	75.0	62.5
	“把 <object> 移到盘子上”（OOD）	6	41.7	33.3	58.3	41.7	75.0	66.7
	平均		69.7 $\pm$ 7.2%	30.3 $\pm$ 6.1%	47.0 $\pm$ 6.9%	62.1 $\pm$ 7.9%	68.2 $\pm$ 7.5%	68.2 $\pm$ 7.8%

Appendix C RT-2-X vs. OpenVLA in BridgeData V2 Evaluations

附录 C BridgeData V2 评测中的 RT-2-X 与 OpenVLA

In this section, we provide additional details on RT-2-X vs. OpenVLA comparisons in BridgeData V2 evaluations discussed in Section 5.1. As discussed previously, OpenVLA is pretrained on a larger subset of OpenX data than RT-2-X and uses a fused SigLIP-DinoV2 vision backbone rather than a single visual encoder. However, in addition to these factors, we believe that OpenVLA’s significant improvement upon RT-2-X specifically in BridgeData V2 evaluations (as shown in Fig. 3) also stems from more careful preprocessing of the Bridge dataset.

本节补充第 5.1 节讨论的 BridgeData V2 评测中 RT-2-X 与 OpenVLA 比较的更多细节。如前所述，OpenVLA 在比 RT-2-X 更大的 OpenX 数据子集上预训练，并使用融合的 SigLIP-DinoV2 视觉骨干而非单一视觉编码器。不过，除了这些因素之外，我们认为 OpenVLA 在 BridgeData V2 评测中相对 RT-2-X 的显著提升（如图 3 所示）还源于对 Bridge 数据集更仔细的预处理。

During the development of the OpenVLA model, we discovered that the original version of the BridgeData V2 dataset contained many transitions with all-zero (no-op) actions. For instance, in every demonstration, an all-zero action was recorded as the ground-truth action in the first timestep. Consequently, training a highly expressive VLA model on the original dataset without any data preprocessing led to a policy that frequently predicted all-zero actions and froze during evaluations. Therefore, we simply filtered out the first transition in every demonstration when training the OpenVLA model, and this was sufficient for mitigating the freezing behavior in most cases.

在开发 OpenVLA 模型的过程中，我们发现原始版本的 BridgeData V2 数据集包含许多全零（无操作，no-op）动作的转移。例如，每条示教的第一个时间步都把一个全零动作记录为真值动作。因此，不做任何数据预处理、直接在原始数据集上训练一个表达力很强的 VLA 模型，会得到一个经常预测全零动作、在评测中卡住不动的策略。于是，我们在训练 OpenVLA 模型时简单地过滤掉每条示教的第一个转移，这在大多数情况下足以缓解卡住的行为。

However, the RT-2-X model was trained without such data preprocessing, so it often suffers the aforementioned freezing behavior if deployed out of the box without modifying the model querying procedure - which severely deteriorates rollout performance. Since this is a proprietary model that is infeasible for us to re-train (e.g., with our preprocessed version of the BridgeData V2 dataset), we mitigated this issue by simply querying the second-most-likely action from the model, since the first-most-likely action was often all zeros while the second-most-likely action was not. (Note that this is the same workaround that was applied by the developers of the RT-2-X model for BridgeData V2 evaluations reported in the Open X-Embodiment experiments [1].) This workaround led to much stronger RT-2-X performance on BridgeData V2 evaluations - though we believe that it is still suboptimal compared to re-training the model on the preprocessed version of the dataset.

然而，RT-2-X 模型训练时没有做这种数据预处理，所以如果不修改模型的查询方式就开箱部署，它常常出现上述卡住的行为——这会严重损害试验表现。由于这是一个专有模型，我们无法重新训练它（例如用我们预处理过的 BridgeData V2 数据集），我们缓解这一问题的办法是直接从模型中查询 概率第二高 的动作，因为概率最高的动作常常是全零，而概率第二高的动作不是。（注意，这与 RT-2-X 模型的开发者在 Open X-Embodiment 实验 [1] 中报告 BridgeData V2 评测时所用的变通办法相同。）这一变通办法使 RT-2-X 在 BridgeData V2 评测上的表现强了很多——不过我们认为，与在预处理过的数据集上重新训练模型相比，这仍不是最优的。

We also tried to dynamically query RT-2-X, i.e., by first sampling the first-most-likely action and then sampling the second-most-likely action if the first one was all zeros. However, we empirically found that dynamic querying led to worse performance than simply querying the second-most-likely action at all times. We hypothesize that this is due to a change in the robot’s dynamics that arises from dynamic querying: pausing in the middle of a trajectory to re-query the model leads to slight interruptions in the robot’s movement due to non-neglible latency in the querying pipeline, and this leads to subtle performance degradation. Therefore, we report the performance of RT-2-X when always querying the second-most-likely action, as done in the Open X-Embodiment project [1].

我们也尝试过 动态地 查询 RT-2-X，即先采样概率最高的动作，如果它是全零，再采样概率第二高的动作。然而我们在实验中发现，动态查询的性能比始终查询概率第二高的动作更差。我们推测这是因为动态查询改变了机器人的动力学：由于查询流水线存在不可忽略的延迟，在轨迹中途停下来重新查询模型会让机器人的运动出现轻微中断，从而导致细微的性能下降。因此，我们报告的是 RT-2-X 始终查询概率第二高动作时的性能，与 Open X-Embodiment 项目 [1] 的做法一致。

Appendix D Additional Experiments and Ablations

附录 D 补充实验与消融

In this section, we conduct several additional experiments to analyze the effects of individual components of the OpenVLA model architecture and training scheme, as well as provide quantitative evidence for claims made in earlier sections of this work. We aim to answer the following questions:

本节我们进行若干补充实验，分析 OpenVLA 模型架构和训练方案中各个组件的作用，并为本文前面各节的论断提供定量证据。我们希望回答以下问题：

How important is OpenX training and how does it impact OpenVLA’s performance (Section D.1)? What effect does using a fused SigLIP-DinoV2 vision encoder have on OpenVLA’s performance, compared to using a SigLIP-only vision encoder (Section D.2)? Is it better to fine-tune or freeze the vision encoder in OpenVLA (Section D.3)? How do the quantized inference results discussed in Section 5.3 change when policy performance is disentangled from model inference speed (Section D.4)?

OpenX 训练有多重要？它如何影响 OpenVLA 的性能（D.1 节）？ 与仅用 SigLIP 的视觉编码器相比，使用融合的 SigLIP-DinoV2 视觉编码器对 OpenVLA 的性能有什么影响（D.2 节）？ 在 OpenVLA 中，微调视觉编码器好还是冻结它好（D.3 节）？ 当把策略性能与模型推理速度分离开时，第 5.3 节讨论的量化推理结果会如何变化（D.4 节）？ 译注：量化推理实验实际在第 5.4 节，此处及 D.4 节开头的“第 5.3 节”原文如此。

We discuss the experimental setup and results addressing each of the above questions sequentially in the following sections.

下面各节依次讨论回答上述问题的实验设置和结果。

D.1 OpenX Training Data Ablation Experiments

D.1 OpenX 训练数据消融实验

As discussed in Section 3.3, OpenVLA is trained on a large dataset of robot embodiments, scenes, and tasks from the Open X-Embodiment dataset [1] (OpenX). In this section, we ablate the OpenX mixture and train a VLA policy solely on one robot dataset, to assess the impact of OpenX training on policy performance. Note that we have already observed the negative effect of ablating OpenX training in the fine-tuning regime, as discussed in Section 5.2 (see OpenVLA (Scratch)), but we discuss additional experiments on another robot embodiment in this section to provide more supporting evidence.

如第 3.3 节所述，OpenVLA 在来自 Open X-Embodiment 数据集 [1]（OpenX）、涵盖多种机器人本体、场景和任务的大型数据集上训练。本节我们消融 OpenX 配比，只在一个机器人数据集上训练 VLA 策略，以评估 OpenX 训练对策略性能的影响。注意，我们已经在微调场景中观察到去掉 OpenX 训练的负面影响，见第 5.2 节的讨论（见 OpenVLA (Scratch)），但本节讨论在另一种机器人本体上的补充实验，以提供更多支持证据。

Experimental setup and tasks. We compare the original OpenVLA model with OpenVLA-Bridge, which is produced by taking the same pretrained VLM as OpenVLA (Prismatic VLM [44]) and fine-tuning it solely on BridgeData V2 [6] rather than the entire OpenX training mixture discussed in Appendix A. We evaluate OpenVLA and OpenVLA-Bridge on a subset of 8 representative tasks from the BridgeData V2 WidowX robot evaluation suite discussed in Section B.1.1. The tasks are listed in Table 9.

实验设置与任务 。我们把原始 OpenVLA 模型与 OpenVLA-Bridge 比较：后者取与 OpenVLA 相同的预训练 VLM（Prismatic VLM [44]），只在 BridgeData V2 [6] 上微调，而不是在附录 A 所述的完整 OpenX 训练配比上微调。我们在 B.1.1 节所述 BridgeData V2 WidowX 机器人评测套件中选出 8 个有代表性的任务组成的子集上评测 OpenVLA 和 OpenVLA-Bridge。任务列于表 9。

Results. Results for the OpenX training mixture ablation are shown in Table 9. By comparing OpenVLA with OpenVLA-Bridge, we see that performance drops drastically (reduction of 30 percent in absolute success rate), which demonstrates the importance of OpenX pretraining on final policy performance. Although the language grounding performance is not impacted, we observe performance reduction across all generalization categories. This result suggests that the large diversity of scenes, objects, and tasks in the OpenX training mixture is essential for unlocking improved generalization capabilities in the OpenVLA model.

结果 。OpenX 训练配比的消融结果见表 9。比较 OpenVLA 和 OpenVLA-Bridge 可以看到性能急剧下降（绝对成功率降低 30 个百分点），这表明 OpenX 预训练对最终策略性能的重要性。虽然语言落地性能没有受到影响，但我们观察到所有泛化类别上的性能都有所下降。这一结果表明，OpenX 训练配比中场景、物体和任务的高度多样性，对于释放 OpenVLA 模型更强的泛化能力至关重要。

\pm

Category	Task	# Trials	OpenVLA # Successes	OpenVLA-Bridge # Successes	OpenVLA-Bridge-SigLIP # Successes
Visual gen	Put Eggplant into Pot (Easy Version)	10	10	8	8
Visual gen	Put Eggplant into Pot	10	10	2	3
Visual gen	Put Cup from Counter into Sink	10	7	4	2
Motion gen	Lift Eggplant	10	7.5	5.5	6.5
Physical gen	Put Carrot on Plate	10	8	4	1
Physical gen	Lift AAA Battery	10	7	2	2
Semantic gen	Take Purple Grapes out of Pot	10	4	3	3
Language grounding	Put {Eggplant, Red Bottle} into Pot	10	7.5	8	7
		Mean Success Rate	76.3 $\pm$ 4.8%	45.6 $\pm$ 5.6%	40.6 $\pm$ 5.5%

\pm

类别	任务	试验次数	OpenVLA 成功次数	OpenVLA-Bridge 成功次数	OpenVLA-Bridge-SigLIP 成功次数
视觉泛化	把茄子放进锅里（简单版）	10	10	8	8
视觉泛化	把茄子放进锅里	10	10	2	3
视觉泛化	把杯子从台面放进水槽	10	7	4	2
运动泛化	举起茄子	10	7.5	5.5	6.5
物理泛化	把胡萝卜放到盘子上	10	8	4	1
物理泛化	举起 AAA 电池	10	7	2	2
语义泛化	把紫葡萄从锅里拿出来	10	4	3	3
语言落地	把 {茄子, 红瓶子} 放进锅里	10	7.5	8	7
		平均成功率	76.3 $\pm$ 4.8%	45.6 $\pm$ 5.6%	40.6 $\pm$ 5.5%

D.2 Dual vs. Single Vision Encoder Experiments

D.2 双视觉编码器与单视觉编码器实验

The OpenVLA model architecture consists of a fused vision backbone that combines the SigLIP [9] and DinoV2 [25] encoders. In this section, we ablate the DinoV2 component to assess the importance of using a dual vision encoder.

OpenVLA 的模型架构包含一个融合 SigLIP [9] 和 DinoV2 [25] 编码器的视觉骨干。本节我们消融 DinoV2 组件，以评估使用双视觉编码器的重要性。

Experimental setup and tasks. We instantiate a model, OpenVLA-Bridge-SigLIP, which is a version of OpenVLA that is trained only on BridgeData V2 and consists of only the SigLIP encoder as the vision backbone. We compare this model with the OpenVLA-Bridge model discussed in the previous section (Section D.1), which shares the same model architecture as the original OpenVLA model and is only trained on Bridge robot data. Therefore, the only difference between OpenVLA-Bridge-SigLIP and OpenVLA-Bridge is that the former omits the DinoV2 encoder in the vision backbone. We evaluate these models on the same subset of 8 Bridge tasks described in the previous section.

实验设置与任务 。我们实例化一个模型 OpenVLA-Bridge-SigLIP ，它是只在 BridgeData V2 上训练、视觉骨干只有 SigLIP 编码器的 OpenVLA 版本。我们把它与上一节（D.1 节）讨论的 OpenVLA-Bridge 模型比较，后者与原始 OpenVLA 模型架构相同，只在 Bridge 机器人数据上训练。因此，OpenVLA-Bridge-SigLIP 与 OpenVLA-Bridge 的唯一区别是前者在视觉骨干中省略了 DinoV2 编码器。我们在上一节所述的同一组 8 个 Bridge 任务上评测这些模型。

Results. Results for the dual vision encoder ablation are shown in Table 9. The drop in performance from OpenVLA-Bridge to OpenVLA-Bridge-SigLIP implies that additionally including the DinoV2 encoder in the vision backbone improves policy performance. However, the 5 percent reduction in performance here is not as significant as the 30 percent drop in performance observed from ablating OpenX training. The low-level spatial features represented in DinoV2 appear to aid generalization in only some cases.

结果 。双视觉编码器的消融结果见表 9。从 OpenVLA-Bridge 到 OpenVLA-Bridge-SigLIP 的性能下降说明，在视觉骨干中额外加入 DinoV2 编码器能提升策略性能。不过，这里 5 个百分点的性能下降，远不如去掉 OpenX 训练时观察到的 30 个百分点那样显著。DinoV2 所表示的低层空间特征似乎只在部分情况下有助于泛化。

D.3 Fine-Tuned vs. Frozen Vision Encoder Experiments

D.3 微调视觉编码器与冻结视觉编码器实验

As discussed in Section 3.4, prior work on VLMs observed higher performance from freezing the vision encoder than fine-tuning its parameters [44]. However, when training OpenVLA, we fine-tuned all 7B parameters in the model, including the SigLIP-DinoV2 vision backbone, as we discovered early on during development that fine-tuning the vision encoder led to higher-performing VLAs — a finding which held across various pretrained VLMs and model architectures. We discuss details of such findings below.

如第 3.4 节所述，先前关于 VLM 的工作观察到冻结视觉编码器比微调其参数性能更高 [44]。然而在训练 OpenVLA 时，我们微调了模型的全部 70 亿参数，包括 SigLIP-DinoV2 视觉骨干，因为我们在开发早期就发现，微调视觉编码器能得到性能更高的 VLA——这一发现在多种预训练 VLM 和模型架构上都成立。下面讨论这些发现的细节。

Experimental setup and tasks. In this section, we report the performance of two VLA policies produced by fine-tuning two different pretrained models from the Prismatic VLMs [44] repository on BridgeData V2. The two pretrained models are named SigLIP ViT-SO 224px and LLaVa v1.5 7B (Reproduction); see Karamcheti et al. [44] for details on their architectures and training mixtures. We evaluate both policies on various Bridge tasks shown in Table 10. Note that the evaluation configurations here differ from previously discussed Bridge evaluations, so the results are not directly comparable to results in other similar experiments.

实验设置与任务 。本节报告两个 VLA 策略的性能，它们分别由 Prismatic VLMs [44] 仓库中的两个不同预训练模型在 BridgeData V2 上微调得到。这两个预训练模型名为 SigLIP ViT-SO 224px 和 LLaVa v1.5 7B (Reproduction) ；它们的架构和训练配比详见 Karamcheti 等人 [44]。我们在表 10 所示的多个 Bridge 任务上评测这两个策略。注意，这里的评测配置与前面讨论的 Bridge 评测不同，因此结果不能与其他类似实验的结果直接比较。

Results. Results for the fine-tuned vs. frozen vision encoder experiments are shown in Table 10. We find that for both VLAs tested, fine-tuning the vision encoder leads to significantly higher success rates across various tasks. Qualitatively, in some cases, deploying the frozen vision encoder policies leads to unstable robot behaviors that are clearly suboptimal. Consequently, we decided early on during development to not conduct further experimentation with frozen vision encoders.

结果 。微调与冻结视觉编码器的实验结果见表 10。我们发现，对两个被测 VLA 来说，微调视觉编码器都能在多个任务上带来显著更高的成功率。定性地看，在某些情况下，部署冻结视觉编码器的策略会导致机器人行为不稳定，明显不理想。因此，我们在开发早期就决定不再对冻结视觉编码器做进一步实验。

Task # Trials SigLIP ViT-SO 224px: Frozen Vision # Successes SigLIP ViT-SO 224px: Fine-Tuned # Successes LLaVa v1.5 7B (Reproduction): Frozen Vision # Successes LLaVa v1.5 7B (Reproduction): Fine-Tuned # Successes Put Eggplant into Pot 1071059 Put Corn on Plate 1010909 Mean Success Rate 85952590 Put { Eggplant, Red Bottle } into Pot 424 - 3 Put { Blue Cup, Pink Cup } on Plate 400 - 0 Lift { Cheese, Red Chili Pepper } 403 - 2 Put { Strawberry, Lime } into Pot 410 - 3 Move { Sushi, Grapes } 434 - 3 Mean Success Rate 3055 - 55 Table 10: Fine-tuned vs. frozen vision encoder experiment results. We evaluate the performance of fine-tuning (“Fine-Tuned”) vs. freezing the vision encoder (“Frozen Vision”) in two VLA policies built on top of two different pretrained VLMs from the Prismatic VLMs [44] repository. BridgeData V2 WidowX tasks shown here are performed in the same sink environment used for other Bridge experiments in this work (however, the initial environment configurations here differ, as these evaluations were conducted at an earlier stage in the project). We find that fine-tuning the vision encoder is crucial to obtain good policy performance. Certain frozen vision encoder evaluations were discontinued due to very poor (near-zero) performance and unstable robot behaviors. Among the evaluations where both frozen vision and fine-tuned approaches are tested, fine-tuning the vision encoder leads to 80.0% average success versus 46.7% average success from leaving it frozen.

Task	# Trials	SigLIP ViT-SO 224px: Frozen Vision # Successes	SigLIP ViT-SO 224px: Fine-Tuned # Successes	LLaVa v1.5 7B (Reproduction): Frozen Vision # Successes	LLaVa v1.5 7B (Reproduction): Fine-Tuned # Successes
Put Eggplant into Pot	10	7	10	5	9
Put Corn on Plate	10	10	9	0	9
	Mean Success Rate	85	95	25	90
Put { Eggplant, Red Bottle } into Pot	4	2	4	–	3
Put { Blue Cup, Pink Cup } on Plate	4	0	0	–	0
Lift { Cheese, Red Chili Pepper }	4	0	3	–	2
Put { Strawberry, Lime } into Pot	4	1	0	–	3
Move { Sushi, Grapes }	4	3	4	–	3
	Mean Success Rate	30	55	–	55

任务 试验次数 SigLIP ViT-SO 224px：冻结视觉编码器 成功次数 SigLIP ViT-SO 224px：微调 成功次数 LLaVa v1.5 7B (Reproduction)：冻结视觉编码器 成功次数 LLaVa v1.5 7B (Reproduction)：微调 成功次数 把茄子放进锅里 1071059 把玉米放到盘子上 1010909 平均成功率 85952590 把 { 茄子, 红瓶子 } 放进锅里 424 - 3 把 { 蓝杯子, 粉杯子 } 放到盘子上 400 - 0 举起 { 奶酪, 红辣椒 } 403 - 2 把 { 草莓, 青柠 } 放进锅里 410 - 3 移动 { 寿司, 葡萄 } 434 - 3 平均成功率 3055 - 55 表 10： 微调与冻结视觉编码器的实验结果 。我们在两个 VLA 策略中评测微调视觉编码器（“微调”）与冻结视觉编码器（“冻结视觉编码器”）的性能，这两个策略分别建立在 Prismatic VLMs [44] 仓库中两个不同的预训练 VLM 之上。这里的 BridgeData V2 WidowX 任务在本文其他 Bridge 实验所用的同一个水槽环境中进行（不过初始环境配置不同，因为这些评测是在项目较早阶段进行的）。我们发现微调视觉编码器对获得良好的策略性能至关重要。部分冻结视觉编码器的评测因性能极差（接近零）且机器人行为不稳定而中止。在同时测试了冻结和微调两种方式的评测中，微调视觉编码器的平均成功率为 80.0%，而保持冻结的平均成功率为 46.7%。 译注：原表为两级表头（SigLIP ViT-SO 224px 与 LLaVa v1.5 7B (Reproduction) 各占两列），此处合并为一级表头；原文两组任务各有一行“平均成功率”。

任务	试验次数	SigLIP ViT-SO 224px：冻结视觉编码器成功次数	SigLIP ViT-SO 224px：微调成功次数	LLaVa v1.5 7B (Reproduction)：冻结视觉编码器成功次数	LLaVa v1.5 7B (Reproduction)：微调成功次数
把茄子放进锅里	10	7	10	5	9
把玉米放到盘子上	10	10	9	0	9
	平均成功率	85	95	25	90
把 { 茄子, 红瓶子 } 放进锅里	4	2	4	–	3
把 { 蓝杯子, 粉杯子 } 放到盘子上	4	0	0	–	0
举起 { 奶酪, 红辣椒 }	4	0	3	–	2
把 { 草莓, 青柠 } 放进锅里	4	1	0	–	3
移动 { 寿司, 葡萄 }	4	3	4	–	3
	平均成功率	30	55	–	55

D.4 Additional Quantized Inference Experiments: Disentangling Policy Performance and Model Inference Speed

D.4 补充量化推理实验：把策略性能与模型推理速度分离开

In Section 5.3, we evaluated OpenVLA with different levels of precision at inference time: half precision (bfloat16), 8-bit quantization, and 4-bit quantization. 8-bit quantization led to lower BridgeData V2 performance relative to the other two approaches, and we hypothesized that the reduction in performance was caused by lower model inference speed from the operations used in 8-bit quantization. In this section, we conduct experiments to assess the veracity of this claim.

在第 5.3 节中，我们用不同的推理精度评测了 OpenVLA：半精度（bfloat16）、8 比特量化和 4 比特量化。8 比特量化在 BridgeData V2 上的性能低于另外两种方式，我们推测性能下降是由 8 比特量化所用的运算导致模型推理速度降低造成的。本节我们通过实验来检验这一说法是否属实。

Specifically, we evaluate OpenVLA again with the three different levels of precision listed above, but now with blocking control . In other words, each action is fully executed on the robot before the next one is predicted by the policy and executed by the controller. This scheme controls system dynamics across methods with varying amounts of latency and thus allows us to test the quality of a policy’s action predictions, independent of its prediction speed. Effectively, the precision levels that have higher throughput - bfloat16 and 4-bit quantization - are forced to run slower to match the dynamics observed when deploying OpenVLA with 8-bit precision. Therefore, we expect OpenVLA’s performance with 8-bit precision to match the performance of bfloat16 and 4-bit precision under blocking control.

具体来说，我们再次用上述三种精度评测 OpenVLA，但这次使用 阻塞控制 。也就是说，每个动作都在机器人上完全执行完毕，策略才预测下一个动作并交给控制器执行。这种方式在延迟各不相同的方法之间控制住了系统动力学，因此可以独立于预测速度来检验策略动作预测的质量。实际上，吞吐更高的精度——bfloat16 和 4 比特量化——被迫放慢运行，以匹配以 8 比特精度部署 OpenVLA 时观察到的动力学。因此我们预期，在阻塞控制下，OpenVLA 以 8 比特精度运行的性能会与 bfloat16 和 4 比特精度相当。

Experimental setup and tasks. We report the performance of OpenVLA with blocking control and quantized inference on the same subset of 8 BridgeData V2 tasks used in Section D.1 and Section D.2.

实验设置与任务 。我们在 D.1 节和 D.2 节所用的同一组 8 个 BridgeData V2 任务上，报告 OpenVLA 在阻塞控制和量化推理下的性能。

Results. Quantized inference experiment results with blocking control are shown in Table 11. Unlike in Table 2, where 8-bit quantization led to the worst rollout performance due to low inference speed, here we observe that 8-bit quantization performs comparably to bfloat16 precision and 4-bit quantization given that we evaluate with blocking control to remove the influence of varying inference speeds on task performance. This confirms our hypothesis about the effect of inference speed on 8-bit quantization performance in previous experiments (when using non-blocking control). We also see no substantial performance degradation when using the lowest precision, 4-bit, as also observed in Section 5.3.

结果 。阻塞控制下的量化推理实验结果见表 11。在表 2 中，8 比特量化因推理速度低而试验表现最差；而这里由于我们用阻塞控制评测，消除了推理速度差异对任务性能的影响，我们观察到 8 比特量化的表现与 bfloat16 精度和 4 比特量化相当。这证实了我们关于先前实验中（使用非阻塞控制时）推理速度影响 8 比特量化性能的假设。我们还看到，使用最低的 4 比特精度时没有明显的性能下降，这与第 5.3 节的观察一致。

\pm

Category	Task	# Trials	bfloat16 # Successes	int8 # Successes	int4 # Successes
Visual gen	Put Eggplant into Pot (Easy Version)	10	10	10	10
Visual gen	Put Eggplant into Pot	10	9	10	10
Visual gen	Put Cup from Counter into Sink	10	5	5	3
Motion gen	Lift Eggplant	10	8	7	7.5
Physical gen	Put Carrot on Plate	10	10	10	10
Physical gen	Lift AAA Battery	10	3	6	4
Semantic gen	Take Purple Grapes out of Pot	10	2	2	2
Language grounding	Put {Eggplant, Red Bottle} into Pot	10	9	9.5	8.5
		Mean Success Rate	70.0 $\pm$ 5.1%	74.4 $\pm$ 4.9%	68.8 $\pm$ 5.2%

\pm

类别	任务	试验次数	bfloat16 成功次数	int8 成功次数	int4 成功次数
视觉泛化	把茄子放进锅里（简单版）	10	10	10	10
视觉泛化	把茄子放进锅里	10	9	10	10
视觉泛化	把杯子从台面放进水槽	10	5	5	3
运动泛化	举起茄子	10	8	7	7.5
物理泛化	把胡萝卜放到盘子上	10	10	10	10
物理泛化	举起 AAA 电池	10	3	6	4
语义泛化	把紫葡萄从锅里拿出来	10	2	2	2
语言落地	把 {茄子, 红瓶子} 放进锅里	10	9	9.5	8.5
		平均成功率	70.0 $\pm$ 5.1%	74.4 $\pm$ 4.9%	68.8 $\pm$ 5.2%

Appendix E LIBERO Simulation Experiments

附录 E LIBERO 仿真实验

Our previous discussions in Section 5.2 and Section 5.3 focused on adapting OpenVLA to novel real-world robot setups and tasks. This section explores adapting OpenVLA to simulated robot setups and tasks, specifically utilizing the LIBERO benchmark [116]. Our experimentation in simulation offers two key advantages:

前面第 5.2 节和第 5.3 节的讨论集中于把 OpenVLA 适配到新的 真实世界 机器人配置和任务。本节探索把 OpenVLA 适配到 仿真 机器人配置和任务，具体使用 LIBERO 基准 [116]。仿真实验有两个关键优势：

Demonstration of versatility: We show that OpenVLA, despite having been pretrained exclusively on real-world robot data, can effectively adapt to simulated domains, overcoming potential disparities between real-world and simulated environments and dynamics. Enhanced accessibility and reproducibility: Integration of OpenVLA into a publicly available simulation platform makes our model more accessible to other researchers, especially those who may not have access to robotic hardware. Additionally, simulated experiments are more easily reproduced than their real-world counterparts.

展示通用性：我们表明，OpenVLA 尽管完全在真实世界机器人数据上预训练，仍能有效适配仿真领域，克服真实环境与仿真环境及其动力学之间可能存在的差异。 提高可及性和可复现性：把 OpenVLA 集成到公开可用的仿真平台中，使其他研究者更容易使用我们的模型，尤其是那些没有机器人硬件的研究者。此外，仿真实验比真实世界实验更容易复现。

We discuss the experimental setup in Section E.1 and the results in Section E.2. We release the materials required to reproduce the experiments along with the OpenVLA codebase.

我们在 E.1 节讨论实验设置，在 E.2 节讨论结果。复现这些实验所需的材料随 OpenVLA 代码库一起发布。

E.1 LIBERO Simulation Experimental Setup

E.1 LIBERO 仿真实验设置

Simulation setup and tasks. The LIBERO benchmark [116] consists of four task suites designed for studying lifelong learning in robotic manipulation, and the original paper therefore investigates both forward and backward transfer to a variety of tasks. In our experiments, we focus solely on supervised fine-tuning on the target task suite, measuring the performance of various policies trained via behavioral cloning on successful demonstrations of the tasks.

仿真设置与任务 。LIBERO 基准 [116] 由四个任务套件组成，用于研究机器人操作中的终身学习，因此原论文同时研究了对多种任务的前向和后向迁移。在我们的实验中，我们只关注在目标任务套件上的监督微调，衡量通过行为克隆在任务成功示教上训练的各种策略的性能。

We perform experiments with the following four task suites, which each contain 10 tasks with 50 human-teleoperated demonstrations each:

我们用以下四个任务套件做实验，每个套件包含 10 个任务，每个任务有 50 条人类遥操作示教：

LIBERO-Spatial consists of the same set of objects but different layouts, and tests the model’s understanding of spatial relationships. LIBERO-Object consists of the same scene layouts but different objects, and tests the model’s understanding of object types. LIBERO-Goal consists of the same objects and layouts but different task goals, and tests the model’s knowledge of different task-oriented behaviors. LIBERO-Long (also called LIBERO-10) consists of long-horizon tasks with diverse objects, layouts, and tasks.

LIBERO-Spatial 由同一组物体、不同布局构成，测试模型对空间关系的理解。 LIBERO-Object 由相同的场景布局、不同物体构成，测试模型对物体类型的理解。 LIBERO-Goal 由相同的物体和布局、不同的任务目标构成，测试模型对不同任务导向行为的掌握。 LIBERO-Long （也称 LIBERO-10 ）由包含多样物体、布局和任务的 长时程 任务构成。

We make the following modifications to each of the training datasets above:

我们对上述每个训练数据集做如下修改：

256\times 256

256\times 256

r=32

r=32

⁸ We use the implementation of Diffusion Policy that is described in the DROID dataset paper [11], which conditions action generation on DistilBERT [117] language embeddings of the task label.

⁸ 我们使用 DROID 数据集论文 [11] 中描述的 Diffusion Policy 实现，它以任务标签的 DistilBERT [117] 语言嵌入作为动作生成的条件。

Evaluation details. To ensure lower variance in the experimental results, all methods are evaluated across 500 trials for each task suite, and the reported performance is the average success rate over three random seeds (resulting in 1500 total trials per statistic). Although we modify the training datasets, as described earlier, we do not change the test environments but rather use the same initial environment configurations provided by the original LIBERO benchmark.

评测细节 。为了降低实验结果的方差，所有方法在每个任务套件上都评测 500 次，报告的性能是三个随机种子上的平均成功率（因此每个统计量共 1500 次试验）。虽然我们如前所述修改了训练数据集，但我们没有改动测试环境，而是使用原始 LIBERO 基准提供的同一套初始环境配置。

E.2 LIBERO Simulation Experimental Results

E.2 LIBERO 仿真实验结果

We present the LIBERO experimental results in Table 12. Importantly, we observe that OpenVLA can be effectively adapted to tasks in the LIBERO simulation environments, as it obtains highest average success rate and rank among the tested methods. However, we find that the overall margin between OpenVLA and the other methods are tighter here than in the real-world fine-tuning experiments discussed in Section 5.2. We attribute this to the fact that OpenVLA was pretrained with purely real-world robot data and no simulation data, which suggests that fine-tuning the model on simulated robot tasks may not be as effective as fine-tuning it on real-world tasks due to the domain gap between simulated and real-world environments and dynamics. We see evidence for this notion in the results obtained by Octo - another policy pretrained on large amounts of real-world robot data - which also only achieves a small boost in overall performance relative to a simple, strong baseline such as Diffusion Policy trained from scratch. We expect increased gains in performance for the pretrained and fine-tuned methods if simulation data is added to the pretraining data mixture.

LIBERO 实验结果见表 12。重要的是，我们观察到 OpenVLA 能有效适配 LIBERO 仿真环境中的任务，在被测方法中取得最高的平均成功率和平均排名。不过我们发现，这里 OpenVLA 与其他方法之间的总体差距比第 5.2 节讨论的真实世界微调实验中更小。我们把这归因于 OpenVLA 只用真实世界机器人数据预训练、没有使用仿真数据，这表明由于仿真与真实环境及其动力学之间存在领域差异，在仿真机器人任务上微调该模型可能不如在真实世界任务上微调有效。Octo 的结果为这一看法提供了证据：它是另一个在大量真实世界机器人数据上预训练的策略，相对于从零训练的 Diffusion Policy 这种简单而强的基线，总体性能也只有小幅提升。我们预期，如果在预训练数据配比中加入仿真数据，预训练后再微调的方法会获得更大的性能提升。

\uparrow

Method	Spatial SR ( $\uparrow$ )	Spatial Rank ( $\downarrow$ )	Object SR ( $\uparrow$ )	Object Rank ( $\downarrow$ )	Goal SR ( $\uparrow$ )	Goal Rank ( $\downarrow$ )	Long SR ( $\uparrow$ )	Long Rank ( $\downarrow$ )	Average SR ( $\uparrow$ )	Average Rank ( $\downarrow$ )
Diffusion Policy from scratch	78.3 $\pm$ 1.1%	3	92.5 $\pm$ 0.7%	1	68.3 $\pm$ 1.2%	3	50.5 $\pm$ 1.3%	3	72.4 $\pm$ 0.7%	2.5
Octo fine-tuned	78.9 $\pm$ 1.0%	2	85.7 $\pm$ 0.9%	3	84.6 $\pm$ 0.9%	1	51.1 $\pm$ 1.3%	2	75.1 $\pm$ 0.6%	2
OpenVLA fine-tuned (ours)	84.7 $\pm$ 0.9%	1	88.4 $\pm$ 0.8%	2	79.2 $\pm$ 1.0%	2	53.7 $\pm$ 1.3%	1	76.5 $\pm$ 0.6%	1.5

\uparrow

方法	Spatial 成功率（ $\uparrow$ ）	Spatial 排名（ $\downarrow$ ）	Object 成功率（ $\uparrow$ ）	Object 排名（ $\downarrow$ ）	Goal 成功率（ $\uparrow$ ）	Goal 排名（ $\downarrow$ ）	Long 成功率（ $\uparrow$ ）	Long 排名（ $\downarrow$ ）	平均成功率（ $\uparrow$ ）	平均排名（ $\downarrow$ ）
从零训练的 Diffusion Policy	78.3 $\pm$ 1.1%	3	92.5 $\pm$ 0.7%	1	68.3 $\pm$ 1.2%	3	50.5 $\pm$ 1.3%	3	72.4 $\pm$ 0.7%	2.5
微调后的 Octo	78.9 $\pm$ 1.0%	2	85.7 $\pm$ 0.9%	3	84.6 $\pm$ 0.9%	1	51.1 $\pm$ 1.3%	2	75.1 $\pm$ 0.6%	2
微调后的 OpenVLA（本文）	84.7 $\pm$ 0.9%	1	88.4 $\pm$ 0.8%	2	79.2 $\pm$ 1.0%	2	53.7 $\pm$ 1.3%	1	76.5 $\pm$ 0.6%	1.5

OpenVLA: An Open-Source Vision-Language-Action Model

OpenVLA：一个开源的视觉-语言-动作模型

Abstract

摘要

1 Introduction

1 引言

2 Related Work

2 相关工作

Visually-Conditioned Language Models

视觉条件语言模型

Generalist Robot Policies

通用机器人策略

Vision-Language-Action Models

视觉-语言-动作模型

3 The OpenVLA Model

3 OpenVLA 模型

3.1 Preliminaries: Vision-Language Models

3.1 预备知识：视觉-语言模型

3.2 OpenVLA Training Procedure

3.2 OpenVLA 训练流程

3.3 Training Data

3.3 训练数据

3.4 OpenVLA Design Decisions

3.4 OpenVLA 的设计决策

3.5 Infrastructure for Training and Inference

3.5 训练与推理基础设施

4 The OpenVLA Codebase

4 OpenVLA 代码库

5 Experiments

5 实验

5.1 Direct Evaluations on Multiple Robot Platforms

5.1 在多个机器人平台上的直接评测

5.2 Data-Efficient Adaptation to New Robot Setups

5.2 数据高效地适配新的机器人配置

5.3 Parameter-Efficient Fine-Tuning

5.3 参数高效微调

5.4 Memory-Efficient Inference via Quantization

5.4 通过量化实现省显存的推理

6 Discussion and Limitations

6 讨论与局限

Acknowledgments

致谢

References

参考文献

Appendix A Data Mixture Details

附录 A 数据配比细节

Appendix B Evaluation Tasks and Detailed Results

附录 B 评测任务与详细结果

B.1 BridgeData V2 WidowX Evaluation Details

B.1 BridgeData V2 WidowX 评测细节

B.1.1 BridgeData V2 Evaluation Tasks

B.1.1 BridgeData V2 评测任务

B.1.2 Comparing Evaluation Tasks to Original BridgeData V2 Training Data

B.1.2 评测任务与原始 BridgeData V2 训练数据的比较

B.1.3 Detailed BridgeData V2 Evaluation Results

B.1.3 BridgeData V2 详细评测结果

B.2 Google Robot Evaluation Details

B.2 Google 机器人评测细节

B.2.1 Google Robot Evaluation Tasks

B.2.1 Google 机器人评测任务

B.2.2 Detailed Google Robot Evaluation Results

B.2.2 Google 机器人详细评测结果

B.3 Data-Efficient Adaptation Experiment Details

B.3 数据高效适配实验细节

B.3.1 Franka-Tabletop and Franka-DROID Tasks

B.3.1 Franka-Tabletop 与 Franka-DROID 任务

B.3.2 Detailed Franka-Tabletop and Franka-DROID Evaluation Results

B.3.2 Franka-Tabletop 与 Franka-DROID 详细评测结果

Appendix C RT-2-X vs. OpenVLA in BridgeData V2 Evaluations

附录 C BridgeData V2 评测中的 RT-2-X 与 OpenVLA

Appendix D Additional Experiments and Ablations

附录 D 补充实验与消融

D.1 OpenX Training Data Ablation Experiments

D.1 OpenX 训练数据消融实验

D.2 Dual vs. Single Vision Encoder Experiments

D.2 双视觉编码器与单视觉编码器实验

D.3 Fine-Tuned vs. Frozen Vision Encoder Experiments

D.3 微调视觉编码器与冻结视觉编码器实验

D.4 Additional Quantized Inference Experiments: Disentangling Policy Performance and Model Inference Speed

D.4 补充量化推理实验：把策略性能与模型推理速度分离开

附录 A　数据配比细节

附录 B　评测任务与详细结果

附录 C　BridgeData V2 评测中的 RT-2-X 与 OpenVLA

附录 D　补充实验与消融

附录 E　LIBERO 仿真实验