RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies

RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies

RoboArena：通用机器人策略的分布式真实世界评测

Pranav Atreya Karl Pertsch Tony Lee Moo Jin Kim Arhan Jain Artur Kuramshin Clemens Eppner Cyrus Neary Edward Hu Fabio Ramos Jonathan Tremblay Kanav Arora Kirsty Ellis Luca Macesanu Marcel Torne Villasevil Matthew Leonard Meedeum Cho Ozgur Aslan Shivin Dass Jie Wang William Reger Xingfang Yuan Xuning Yang Abhishek Gupta Dinesh Jayaraman Glen Berseth Kostas Daniilidis Roberto Martin-Martin Youngwoon Lee Percy Liang Chelsea Finn Sergey Levine https://robo-arena.github.io

作者：Pranav Atreya、Karl Pertsch、Tony Lee、Moo Jin Kim、Arhan Jain、Artur Kuramshin、Clemens Eppner、Cyrus Neary、Edward Hu、Fabio Ramos、Jonathan Tremblay、Kanav Arora、Kirsty Ellis、Luca Macesanu、Marcel Torne Villasevil、Matthew Leonard、Meedeum Cho、Ozgur Aslan、Shivin Dass、Jie Wang、William Reger、Xingfang Yuan、Xuning Yang、Abhishek Gupta、Dinesh Jayaraman、Glen Berseth、Kostas Daniilidis、Roberto Martin-Martin、Youngwoon Lee、Percy Liang、Chelsea Finn、Sergey Levine。项目主页： https://robo-arena.github.io （译注：本对照依据 arXiv:2506.18123v2（2025 年 11 月 29 日修订版），原文以 CC BY 4.0 许可发布；中文为本站翻译。图片未收录，只保留图注。）

*: Core contributors, detailed contributions in Appendix A. Correspondence to: pranavatreya@berkeley.edu, pertsch@berkeley.edu, tonyhlee@stanford.edu ¹UC Berkeley, ²Stanford University, ³University of Washington, ⁴University of Montreal, ⁵NVIDIA, ⁶University of Pennsylvania, ⁷UT Austin, ⁸Yonsei University

*：核心贡献者，详细贡献见附录 A。通讯邮箱：pranavatreya@berkeley.edu、pertsch@berkeley.edu、tonyhlee@stanford.edu ¹加州大学伯克利分校，²斯坦福大学，³华盛顿大学，⁴蒙特利尔大学，⁵NVIDIA，⁶宾夕法尼亚大学，⁷得克萨斯大学奥斯汀分校，⁸延世大学

Abstract

摘要

Comprehensive, unbiased, and comparable evaluation of modern generalist policies is uniquely challenging: existing approaches for robot benchmarking typically rely on heavy standardization, either by specifying fixed evaluation tasks and environments, or by hosting centralized “robot challenges”, and do not readily scale to evaluating generalist policies across a broad range of tasks and environments. In this work, we propose RoboArena, a new approach for scalable evaluation of generalist robot policies in the real world. Instead of standardizing evaluations around fixed tasks, environments, or locations, we propose to crowd-source evaluations across a distributed network of evaluators. Importantly, evaluators can freely choose the tasks and environments they evaluate on, enabling easy scaling of diversity, but they are required to perform double-blind evaluations over pairs of policies. Then, by aggregating preference feedback from pairwise comparisons across diverse tasks and environments, we can derive a ranking of policies. We instantiate our approach across a network of evaluators at seven academic institutions using the DROID robot platform. Through more than 600 pairwise real-robot evaluation episodes across seven generalist policies, we demonstrate that our crowd-sourced approach can more accurately rank the performance of existing generalist policies than conventional, centralized evaluation approaches, while being more scalable, resilient, and trustworthy. We open our evaluation network to the community and hope that it can enable more accessible comparisons of generalist robot policies.

对现代通用策略进行全面、无偏且可比较的评测，是一项格外困难的挑战：现有的机器人基准测试方法通常依赖高度标准化，要么规定固定的评测任务和环境，要么举办集中式的“机器人挑战赛”，难以扩展到在大范围任务和环境上评测通用策略。本文提出 RoboArena，一种在真实世界中可扩展地评测通用机器人策略的新方法。我们不再围绕固定的任务、环境或地点来标准化评测，而是提出在一个分布式评测者网络中众包评测。关键在于，评测者可以自由选择评测所用的任务和环境，从而轻松扩大多样性，但他们必须对 成对 的策略进行双盲评测。然后，通过汇总在多样任务和环境上成对比较得到的偏好反馈，我们就能导出策略的排名。我们在七所学术机构的评测者网络上、使用 DROID 机器人平台实现了这一方法。通过对七个通用策略进行的 600 多个成对真实机器人评测回合，我们表明：与传统的集中式评测方法相比，这种众包方法能更准确地对现有通用策略的性能进行排序，同时更具可扩展性、更有韧性、也更可信。我们向社区开放这一评测网络，希望它能让通用机器人策略之间的比较更容易进行。

Keywords: Generalist Robot Policy Evaluation, Crowd-Sourced Evaluation

关键词 ：通用机器人策略评测，众包评测

Figure 1: We present RoboArena, a distributed real-world evaluation framework for generalist robot policies. Instead of standardizing environments and tasks, RoboArena aggregates crowd-sourced pairwise A/B policy evaluations across a broad spectrum of environments and tasks to derive a global policy ranking. Its decentralized design makes RoboArena a scalable, comprehensive, and trustworthy framework for generalist robot policy evaluation. We open-source an instantiation of RoboArena on the DROID robot platform [1] and invite community members to participate, both by contributing policies and running evaluations.

图 1：我们提出 RoboArena，一个面向通用机器人策略的分布式真实世界评测框架。RoboArena 不对环境和任务做标准化，而是汇总在大范围环境和任务上众包得到的成对 A/B 策略评测，导出全局策略排名。其去中心化设计使 RoboArena 成为一个可扩展、全面且可信的通用机器人策略评测框架。我们开源了 RoboArena 在 DROID 机器人平台 [1] 上的一个实现，并邀请社区成员通过提交策略和运行评测来参与。

1 Introduction

1 引言

Modern robot policies are increasingly general, and able to perform a wide range of tasks across many environments [2, 3, 4, 5, 6]. With the increased generality a new challenge arises: how can we accurately measure and compare the performance of such generalist policies? Conventional approaches to robot evaluation and benchmarking rely on comparing policies across tightly standardized sets of environments and tasks [7, 8, 9, 10], and are typically restricted to only a handful of tasks across a small number of scenes [11, 12, 13, 14]. As such, they are not well-suited to provide comprehensive performance evaluations for policies that are designed to perform across a much broader spectrum of initial conditions. At the same time, existing evaluation approaches are challenging to scale: guaranteeing comparable conditions across a large number of real-world tasks and environments faces many practical challenges, from accurately reproducing scene layouts and lighting conditions, to maintaining large, centralized fleets of evaluation robots, and manufacturing differences between said robots. As such, the development of increasingly general robot policies requires us to rethink the way we evaluate robot policies.

现代机器人策略越来越通用，能够在许多环境中完成大量不同的任务 [2, 3, 4, 5, 6]。通用性提高之后出现了一个新挑战：如何准确地衡量和比较这类通用策略的性能？传统的机器人评测和基准测试方法依赖在严格标准化的环境和任务集合上比较策略 [7, 8, 9, 10]，而且通常只局限于少数几个场景中的少量任务 [11, 12, 13, 14]。因此，对于那些设计用来在宽得多的初始条件范围内工作的策略，它们并不适合提供全面的性能评测。同时，现有评测方法难以扩展：要在大量真实世界任务和环境中保证条件可比，会遇到许多实际困难，从准确复现场景布局和光照条件，到维护庞大的集中式评测机器人队伍，再到这些机器人之间的制造差异。因此，越来越通用的机器人策略的发展，要求我们重新思考评测机器人策略的方式。

At first glance, broadly capable robot policies seem to compound the reproducibility and scalability challenges faced by robot evaluations and benchmarks in the past. Yet, in this work, we argue that increasingly general robot policies offer an opportunity for an alternative approach to robot evaluation, that has the promise to address many of the challenges of prior evaluation efforts. Realizing this potential, however, requires rethinking how we run robot evaluations: instead of standardization and centralized evaluation “challenges”, we argue that decentralization and an evaluation approach that embraces the non-stationarity of the physical world offer a promising alternative.

乍看之下，能力宽泛的机器人策略似乎会加剧以往机器人评测和基准测试所面临的可复现性和可扩展性难题。然而在本文中我们认为，越来越通用的机器人策略为另一种评测方式提供了 机会 ，有望解决以往评测工作的许多难题。但要实现这一潜力，就需要重新思考如何开展机器人评测：我们认为，相比 标准化 和集中式的评测“挑战赛”， 去中心化 以及一种接纳物理世界非平稳性的评测方式，是一个有前景的替代方案。

To this end, we propose RoboArena, a distributed real-world evaluation framework for generalist robot policies. Inspired by recent crowdsourced benchmarks for evaluating generalist language or vision-language models like Chatbot Arena [15] or GenAI Arena [16], RoboArena relies on a decentralized network of evaluators that perform pairwise, double-blind comparisons of policies in whichever scene and on whatever task they deem suitable. The evaluator then provides a preference for which of the two policies performed better, along with a free-form language explanation. Our evaluation algorithm aggregates a large number of such pairwise comparisons into a global policy ranking, as well as a set of qualitative characteristics, strengths and weaknesses for each policy. This decentralized design allows RoboArena to be open-ended in the number of environments and tasks policies are evaluated on, robust, since many entities across the decentralized network contribute via double-blind evaluations to the final score, and scalable, with dozens of evaluators asynchronously contributing policy evaluations at any time of day. By not relying on identical initial conditions beyond the horizon of a single pairwise policy comparison, RoboArena foregoes many of the practical challenges prior robot evaluation frameworks faced.

为此，我们提出 RoboArena，一个面向通用机器人策略的分布式真实世界评测框架。受近来用于评测通用语言模型或视觉-语言模型的众包基准（如 Chatbot Arena [15] 和 GenAI Arena [16]）启发，RoboArena 依靠一个去中心化的评测者网络，由评测者在 任何 他们认为合适的 场景 和 任务 上，对策略进行 成对 、 双盲 的比较。评测者随后给出两个策略中哪一个表现更好的偏好，并附上一段自由形式的语言解释。我们的评测算法把大量这样的成对比较汇总为一个全局策略排名，并为每个策略提炼出一组定性特征及其优缺点。这种去中心化设计使 RoboArena 在评测所用的环境和任务数量上是 开放式的 ；它是 稳健的 ，因为去中心化网络中的许多参与方都通过双盲评测为最终得分作出贡献；它也是 可扩展的 ，任何时候都可以有数十名评测者异步地贡献策略评测。由于 RoboArena 只要求在单次成对策略比较的范围内初始条件相同，它避开了以往机器人评测框架所面临的许多实际困难。

In this paper, we outline our distributed evaluation protocol, including algorithms for aggregating pairwise policy comparisons into global policy rankings, and for extracting qualitative policy characteristics via LLM-assisted analysis of evaluation results. We then instantiate RoboArena on the DROID robot platform [1], on which modern policies can out-of-the-box generalize to new scenes and tasks [5]. Through decentralized evaluations of 7 generalist policies across 7 universities and a total of 612 pairwise comparisons, we demonstrate that RoboArena provides more accurate performance rankings of generalist policies than the conventional, centralized evaluation scheme used in prior work, when compared to an “oracle” policy ranking computed via exhaustive evaluation of all policies on all tested tasks. At the same time, RoboArena matches the episode efficiency of standard evaluation, i.e., the same number of evaluations, when distributed across the RoboArena evaluator network, lead to higher quality policy rankings. In addition to the evaluation framework, our work also provides the most comprehensive evaluations of generalist robot policies to date (4284 evaluation episodes across numerous tasks and scenes), and highlights the limitations of current policies.

本文阐述我们的分布式评测协议，包括把成对策略比较汇总为全局策略排名的算法，以及借助 LLM 辅助分析评测结果来提取定性策略特征的算法。随后，我们在 DROID 机器人平台 [1] 上实现了 RoboArena；在这个平台上，现代策略可以开箱即用地泛化到新场景和新任务 [5]。通过在 7 所大学对 7 个通用策略进行去中心化评测，共计 612 次成对比较，我们表明：以对所有策略在所有测试任务上进行穷举评测得到的“神谕”（oracle）策略排名为参照，RoboArena 给出的通用策略性能排名比以往工作所用的传统集中式评测方案更准确。同时，RoboArena 在回合效率上与标准评测相当，即相同数量的评测分布到 RoboArena 评测者网络上之后，能得到质量更高的策略排名。除评测框架之外，我们的工作还提供了迄今为止对通用机器人策略最全面的评测（在大量任务和场景中共 4284 个评测回合），并揭示了当前策略的局限。

2 Related Work

2 相关工作

Simulated and real-world robot evaluation. Simulated evaluations offers perfect reproducibility, and can be efficiently parallelized. Thus, numerous simulated robot benchmarks have been proposed over the years [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]. However, simulated environments are an imperfect replica of the real world, and even works that optimize for visual and physical fidelity [29] are often limited in the diversity of tasks they support, and sometimes fail to accurately reflect real-world policy performance [30]. In the real world, benchmarks try to ensure reproducibility through detailed manuals for reproducing environment setups [7, 8, 9, 10, 31, 32], enabling remote access to centrally hosted evaluations [33, 12, 13, 30], or organizing in-person challenges [34, 11, 35, 14, 36]. However, to ensure reproducibility of initial conditions and make centralized evaluation feasible, these approaches typically restrict evaluations to a narrow set of tasks and environments, or even a specific challenge date. In this work, we propose an alternative approach for real-world robot evaluation based on crowd-sourcing pairwise policy comparisons across a decentralized network of evaluators. Our approach can cover a wider distribution of tasks and environments, and is inherently more resilient and scalable than existing, centralized evaluation approaches.

仿真与真实世界中的机器人评测。 仿真评测具有完美的可复现性，并且可以高效地并行化。因此，多年来人们提出了大量仿真机器人基准 [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]。然而，仿真环境只是真实世界的不完美复制品，即使是针对视觉和物理逼真度做了优化的工作 [29]，其支持的任务多样性也往往有限，有时还不能准确反映策略在真实世界中的性能 [30]。在真实世界中，基准测试试图通过以下方式保证可复现性：提供复现环境布置的详细手册 [7, 8, 9, 10, 31, 32]，允许远程访问集中托管的评测 [33, 12, 13, 30]，或组织线下挑战赛 [34, 11, 35, 14, 36]。但为了保证初始条件可复现、让集中评测可行，这些方法通常把评测限制在一小组任务和环境上，甚至限定在某个特定的挑战赛日期。本文提出另一种真实世界机器人评测方法，它基于在一个 去中心化 评测者网络中众包成对策略比较。我们的方法可以覆盖更广的任务和环境分布，而且天生比现有的集中式评测方法更有韧性、更可扩展。

Generalist robot policies. Recently, robotics has seen a trend towards training generalist robot policies [37, 38, 2, 39, 40, 3, 41, 42, 43, 44, 45, 5, 6, 46, 47] on diverse datasets of robot experience [48, 49, 50, 2, 1, 51, 52, 53, 54]. Notably, multiple works have demonstrated policies that can perform tasks across many environments out of the box [55, 56, 4, 5]. At the same time, conventional approaches to robot evaluation struggle to comprehensively evaluate the performance of such generalist policies, since they are cumbersome to scale to broad distributions of tasks and environments, motivating our distributed evaluation approach in this work.

通用机器人策略。 近来，机器人领域出现了在多样化机器人经验数据集 [48, 49, 50, 2, 1, 51, 52, 53, 54] 上训练 通用 机器人策略 [37, 38, 2, 39, 40, 3, 41, 42, 43, 44, 45, 5, 6, 46, 47] 的趋势。值得注意的是，已有多项工作展示了能够 开箱即用 地在许多环境中执行任务的策略 [55, 56, 4, 5]。与此同时，传统的机器人评测方法难以全面评测这类通用策略的性能，因为要把它们扩展到宽广的任务和环境分布上非常繁琐，这正是我们在本文中提出分布式评测方法的动机。

Crowd-sourced benchmarks. In recent years, crowd-sourced benchmarks have gained popularity, from language modeling [15], to generative image and video modeling [16], or even for specialized legal model training [57]. In robotics, Dasari et al. [58] proposed a benchmark that aggregates policy ranking results across institutions to obtain more comprehensive evaluations of learning algorithms, but it required retraining of policies for every new institution and task at hand, and thus remained limited to a small number of tested environments and tasks. In contrast, our work proposes a distributed benchmark for generalist robot policies, which can be evaluated out-of-the-box, and thus makes it feasible to evaluate on a much broader distribution of tasks and environments.

众包基准。 近年来，众包基准越来越流行，从语言建模 [15]，到图像和视频生成建模 [16]，甚至专门的法律模型训练 [57]。在机器人领域，Dasari 等人 [58] 提出了一个跨机构汇总策略排名结果的基准，以获得对学习 算法 更全面的评测；但它要求每到一个新机构、每遇到一个新任务都重新训练策略，因此仍局限于少量测试环境和任务。与之不同，我们的工作提出一个面向 通用机器人策略 的分布式基准，这些策略可以开箱即用地评测，因此可以在宽得多的任务和环境分布上进行评测。

3 Decentralized Robot Policy Evaluation via Pairwise Comparison

3 通过成对比较进行去中心化的机器人策略评测

Conventional robot policy evaluations aim to standardize the conditions under which policies are compared as much as possible, typically by defining fixed sets of tasks and scenes policies should be run on, and by trying to closely match initial conditions like lighting, camera angles, or scene background [7, 9, 50, 59]. As we discussed in Section 1, reproducing such standardized conditions in the real world is extremely challenging, particularly when evaluations should be run not just in one central place, but across institutions.

传统的机器人策略评测力求尽可能 标准化 比较策略时的条件，通常的做法是定义策略应当运行的固定任务和场景集合，并尽量严格匹配光照、相机角度或场景背景等初始条件 [7, 9, 50, 59]。正如第 1 节所讨论的，在真实世界中复现这种标准化条件极其困难，尤其是当评测不只在一个中心地点、而是跨机构进行的时候。

In this section, we introduce an alternative approach for robot evaluation that foregoes repeatedly evaluating policies in standardized settings, and instead relies on decentralized, pair-wise, double-blind policy comparisons . Crucially, our evaluation approach does not prescribe what scene or task a policy pair should be evaluated in, but instead aggregates a large number of pairwise comparisons across many tasks and scenes to establish a policy ranking. This decentralized approach has multiple benefits: it is open-ended, since we do not standardize tasks and environments, thus broadening coverage; it is robust, since no single entity can easily sway results in double-blind, decentralized evaluations; it is scalable, since many institutions can collaborate on the evaluations; and it is adaptable, since tasks can naturally adapt to the frontier of policy capabilities.

本节介绍另一种机器人评测方法：它不再在标准化设置中反复评测策略，而是依靠 去中心化、成对、双盲的策略比较 。关键在于，我们的评测方法 不 规定一对策略应在什么场景或任务中评测，而是汇总在 许多 任务和场景上进行的大量成对比较来建立策略排名。这种去中心化方法有多重好处：它是 开放式的 ，因为我们不对任务和环境做标准化，从而扩大了覆盖面；它是 稳健的 ，因为在双盲的去中心化评测中，没有哪个单一参与方能轻易左右结果；它是 可扩展的 ，因为许多机构可以协作完成评测；它也是 可适应的 ，因为任务可以自然地跟随策略能力的前沿而变化。

In the remainder of this section, we will first describe our evaluation procedure (Section 3.1), then detail how we aggregate pairwise comparisons into global policy rankings (Section 3.2), and finally describe tools for extracting qualitative policy characteristics from the evaluation data (Section 3.3).

本节其余部分首先描述我们的评测流程（第 3.1 节），然后详述如何把成对比较汇总为全局策略排名（第 3.2 节），最后介绍从评测数据中提取定性策略特征的工具（第 3.3 节）。

3.1 Policy Evaluation Protocol

3.1 策略评测协议

\Pi

\Pi

\pi_{1\dots N}\in\Pi

\pi_{1\dots N}\in\Pi

E

E

E

E

T

T

3.2 Computing Global Policy Rankings

3.2 计算全局策略排名

N

N

\pi_{A}

\pi_{A}

\pi_{A}

\pi_{A}

\theta=(\theta_{1}\dots\theta_{N})

\theta=(\theta_{1}\dots\theta_{N})

p(\pi_{A}>\pi_{B})=\sum_{t=1}^{T}\nu_{t}\cdot\sigma(\theta_{A}+\psi_{A_{t}}-\tau_{t})\cdot(1-\sigma(\theta_{B}+\psi_{B_{t}}-\tau_{t})) \tag{1}

\tau,\nu

\tau,\nu

\theta,\tau,\nu

\theta,\tau,\nu

3.3 Extracting Qualitative Policy Characteristics

3.3 提取定性的策略特征

Figure 2: Pipeline for extracting qualitative policy characteristics from RoboArena’s rich evaluation data. We use a VLMs to categorize scenes and tasks, and then use an LLM to aggregate information across a large number of evaluation rollouts into a policy report that summarizes qualitative strengths and weaknesses, and cites concrete evaluation rollout videos as evidence.

图 2：从 RoboArena 丰富的评测数据中提取 定性 策略特征的流程。我们用 VLM 对场景和任务分类，再用 LLM 把大量评测回合的信息汇总成一份 策略报告 ，总结定性的优缺点，并引用具体的评测回合视频作为证据。

Estimating qualitative policy characteristics, e.g., a model’s ability to follow language or perform multi-step tasks, is crucial to guide future research. Typically, researchers develop an intuition for such qualitative characteristics by running all evaluations for a given policy themselves. Yet, in a distributed evaluation setup we need new tools to synthesize such nuanced insights from hundreds of policy rollout videos, task instructions, and evaluator language feedback. To this end, we experiment with using large language and vision-language models (LLMs, VLMs) to assist with the analysis.

估计 定性 的策略特征，例如模型遵循语言指令或执行多步任务的能力，对指导后续研究至关重要。通常，研究者通过亲自运行某个策略的全部评测来形成对这类定性特征的直觉。然而在分布式评测设置下，我们需要新的工具，从数百个策略回合视频、任务指令和评测者的语言反馈中综合出这类细致的洞见。为此，我们尝试用大语言模型和视觉-语言模型（LLM、VLM）来辅助分析。

We provide an overview of our prototype analysis tool in Figure 2. We pass the first images of each evaluation video and the corresponding task instruction to a VLM (OpenAI GPT‑4.5) and ask it to categorize the task (e.g., pick-place vs. open-close vs. tool use) and describe the scene’s lighting, clutter, object visibility, etc. Then, for each policy in our pool, we use an LLM (OpenAI o3) to generate a policy report by summarizing preference annotations, categorization results, and free-form evaluator feedback for all evaluations. We instruct the LLM to compare performance to other policies along the task categories, and to extract qualitative policy characteristics from the language feedback. Importantly, the LLM is instructed to cite evaluation episodes as evidence for any claim in the report and we automatically annotate the report with videos from these rollouts to enable researchers to verify any claims. We experimentally verify our tool in Section 5.4, and include example policy reports in the supplementary data.

图 2 概述了我们的原型分析工具。我们把每段评测视频的首帧图像和对应的任务指令输入一个 VLM（OpenAI GPT‑4.5），让它对任务分类（例如抓取放置、开合、使用工具），并描述场景的光照、杂乱程度、物体可见度等。然后，针对策略池中的每个策略，我们用一个 LLM（OpenAI o3）汇总所有评测的偏好标注、分类结果和自由形式的评测者反馈，生成一份 策略报告 。我们指示 LLM 按任务类别把该策略的表现与其他策略相比较，并从语言反馈中提取定性的策略特征。重要的是，LLM 被要求为报告中的任何论断 引用 评测回合作为证据，我们还会自动在报告中附上这些回合的视频，方便研究者核实各项论断。我们在第 5.4 节通过实验验证了这一工具，并在补充数据中提供了策略报告示例。

4 The DROID-RoboArena Evaluation System

4 DROID-RoboArena 评测系统

We instantiate RoboArena in a prototype evaluation system on the DROID robot platform [1], which consists of a Franka Panda 7DoF robot arm, a Robotiq 2F-85 parallel-jaw gripper, a ZED-mini stereo wrist camera and one or multiple external ZED 2 stereo cameras (see Figure 3). We choose the DROID platform since it offers multiple attractive properties:

我们在 DROID 机器人平台 [1] 上实现了 RoboArena 的原型评测系统。该平台由一台 Franka Panda 七自由度机械臂、一个 Robotiq 2F-85 平行夹爪、一台 ZED-mini 立体腕部相机，以及一台或多台外部 ZED 2 立体相机组成（见图 3）。我们选择 DROID 平台，是因为它具有多项有吸引力的特点：

The Panda arm provides sufficient dexterity to perform a wide range of manipulation tasks across varied real-world environments. The robot is mounted on a height-adjustable, mobile table, which enables rapid reconfiguration of scenes and camera viewpoints. Most importantly, the platform is associated with the open-source DROID dataset [1], a large-scale, real-world dataset that supports training of generalizable robot policies. Finally, DROID setups are already deployed across multiple academic institutions worldwide, which is key to enabling distributed evaluation at scale.

Panda 机械臂具有足够的灵巧性，能够在各种真实世界环境中执行大量不同的操作任务。 机器人安装在可调高度的移动桌上，可以快速重新配置场景和相机视角。 最重要的是，该平台与开源的 DROID 数据集 [1] 相关联，这是一个支持训练 可泛化 机器人策略的大规模真实世界数据集。 最后，DROID 设备已经部署在全球多所学术机构中，这是实现大规模分布式评测的关键。

Figure 3: The DROID robot setup, which we use for the DROID-RoboArena evaluation system. Reproduced with permission from Khazatsky et al. [1].

图 3：DROID 机器人设备，我们用它构建 DROID-RoboArena 评测系统。经 Khazatsky 等人 [1] 许可转载。

We note that, while DROID is a convenient platform to develop a prototype distributed robot evaluation, the RoboArena evaluation approach readily extends to other robot embodiments and potentially even to evaluating cross-embodiment policies [2] across platforms. The remainder of this section, we provide details on our evaluation system design (Section 4.1) and outline safety considerations and incentive mechanisms that support open participation from the broader robotics community (Section 4.2).

需要指出，虽然 DROID 是开发分布式机器人评测原型的便利平台，但 RoboArena 评测方法可以直接扩展到其他机器人本体，甚至有可能用于跨平台评测跨本体策略 [2]。本节其余部分详细介绍我们的评测系统设计（第 4.1 节），并概述支持更广泛机器人社区开放参与的安全考虑和激励机制（第 4.2 节）。

4.1 RoboArena System Design

4.1 RoboArena 系统设计

Figure 4: The DROID-RoboArena system consists of a pool of remotely hosted policy servers, a pool of distributed evaluator “clients” with real robot setups, a database for storing evaluation results, and a central evaluation management server that orchestrates communication, aggregates the evaluation results, and computes a policy ranking.

图 4：DROID-RoboArena 系统由以下部分组成：一组远程托管的策略服务器、一组配有真实机器人设备的分布式评测者“客户端”、一个存储评测结果的数据库，以及一个中央评测管理服务器，它负责协调通信、汇总评测结果并计算策略排名。

We design a system for performing decentralized evaluation over a potentially large pool of policies, with a pool of evaluators that may span many different locations, countries, or even continents. Our system has four core components: policy inference servers, evaluation clients, an evaluation database, and a central evaluation server (see Figure 4).

我们设计了一个系统，用于对可能很大的策略池进行去中心化评测，评测者池可能分布在许多不同的地点、国家，甚至不同大洲。系统有四个核心组件：策略推理服务器、评测客户端、评测数据库和中央评测服务器（见图 4）。

Policy inference servers We host all policies in our pool on remote servers, instead of running them “on-premise” at the robot evaluation station. This has multiple benefits: effective resource utilization since multiple evaluators can share the same policy server, a lightweight evaluation client since no inference compute needs to be provided client-side, which lowers the barrier for contributing evaluations, and user-side control over policy servers since we assume users host their own policies when submitting them for evaluation, which ensures that policies are run correctly, equipped with sufficient inference compute, and proprietary models can be evaluated without sharing model weights. Remote hosting may, however, introduce additional latency during policy inference, but in practice we found this to be negligible for the static manipulation tasks on which we typically evaluate DROID policies. We acknowledge that future versions of RoboArena evaluations may need to support (at least partially) local inference to enable evaluation of more dynamic tasks.

策略推理服务器 我们把策略池中的所有策略托管在远程服务器上，而不是在机器人评测站“本地”运行。这样做有多重好处： 资源利用高效 ，因为多个评测者可以共享同一个策略服务器； 评测客户端轻量 ，因为客户端无需提供推理算力，降低了贡献评测的门槛； 用户自主控制 策略服务器，因为我们假设用户在提交策略评测时自行托管策略，这保证了策略被正确运行、配备足够的推理算力，而且专有模型无需共享模型权重即可接受评测。不过，远程托管可能在策略推理时引入额外的延迟，但在实践中我们发现，对于我们通常用来评测 DROID 策略的静态操作任务，这种延迟可以忽略不计。我们承认，未来版本的 RoboArena 评测可能需要支持（至少部分）本地推理，以便评测动态性更强的任务。

Evaluation clients We provide a client-side script that guides users through the evaluation protocol outlined in Section 3.1. It handles communication with the central evaluation server and the policy inference servers. Since no client-side inference compute is required, this script can be run on any computer that is connected to the Internet and the DROID robot.

评测客户端 我们提供一个客户端脚本，引导用户完成第 3.1 节所述的评测协议。它负责与中央评测服务器和策略推理服务器通信。由于客户端不需要推理算力，这个脚本可以在任何连接互联网和 DROID 机器人的计算机上运行。

Evaluation database This database hosts all evaluation results: instructions, scores and preference, natural language feedback, and the rollout videos. Information is uploaded by the clients.

评测数据库 该数据库存放所有评测结果：指令、分数和偏好、自然语言反馈以及回合视频。这些信息由客户端上传。

Central evaluation server We host a central evaluation server that assigns policies to evaluators, and keeps track of newly registered or deprecated policies in the policy pool and automatically cancels evaluations after a timeout, e.g., if evaluation clients crashed or lost network connection.

中央评测服务器 我们托管一个中央评测服务器，负责把策略分配给评测者，跟踪策略池中新注册或已弃用的策略，并在超时后自动取消评测，例如评测客户端崩溃或断网时。

4.2 Open-Sourcing RoboArena: Interfaces, Safety and Incentives

4.2 开源 RoboArena：接口、安全与激励

Our goal in designing the DROID-RoboArena is to provide it as a resource to the robotics community, to which anyone can submit policies and contribute evaluations. Importantly, DROID-RoboArena can enable researchers without access to a physical robot to train and evaluate real-world generalist robot policies - now all required resources are open-source: diverse, real-robot datasets [50, 2, 1, 54], scalable modeling frameworks [39, 3, 64, 65, 47], and DROID-RoboArena real-robot evaluations.

我们设计 DROID-RoboArena 的目标，是把它作为一项资源提供给机器人社区，任何人都可以向其提交策略、贡献评测。重要的是，DROID-RoboArena 能让没有实体机器人的研究者训练和评测真实世界的通用机器人策略——现在所需的全部资源都已开源：多样化的真实机器人数据集 [50, 2, 1, 54]、可扩展的建模框架 [39, 3, 64, 65, 47]，以及 DROID-RoboArena 真实机器人评测。

Making the DROID-RoboArena easy to use, safe, and self-sustaining requires a few additional elements. First, we make it easy to browse all performed evaluations and view existing policy leaderboards on our website. We plan to publish separate leaderboards for “all policies” and “open-source policies”; the latter are trained only on publicly available datasets, and provide weight access and details for how to reproduce training. Additionally, we design multiple safety layers to prevent policies from damaging robot evaluation hardware: first, we test that any newly submitted policy server complies with the expected input and output formats. Then we run said policy in a “test environment” across a few different scenes and tasks with a specifically trained evaluator that is able to quickly intervene if a policy runs the danger of acting unsafely and damaging the robot (akin to a test driver for autonomous vehicles). Only after a policy passed these tests, we add it to the general pool and distribute it to evaluators.

要让 DROID-RoboArena 易用、安全且能自我维持，还需要一些额外的要素。首先，我们让用户可以在网站上方便地浏览所有已完成的评测和现有的策略排行榜。我们计划分别发布“所有策略”和“开源策略”两个排行榜；后者只用公开可得的数据集训练，提供权重下载以及复现训练的细节。此外，我们设计了多层 安全措施 ，防止策略损坏机器人评测硬件：首先，我们检查任何新提交的策略服务器是否符合预期的输入和输出格式；然后，在一个“测试环境”中，由一名经过专门培训、能在策略可能做出不安全动作并损坏机器人时迅速干预的评测者，在几个不同的场景和任务上运行该策略（类似于自动驾驶汽车的测试驾驶员）。只有通过这些测试的策略，才会被加入公共池并分发给评测者。

Finally, to make the DROID-RoboArena self-sufficient in the long term, we implement an “evaluation credit” system, that balances evaluation supply and demand: for every pairwise policy evaluation that an evaluator runs, they receive a credit, which they can use to request an equal number of pairwise comparisons between their own policies and other policies from the pool. This way, evaluation effort for any individual is comparable to running evaluations for just their own policy on their own setup, but by agreeing to evaluate others’ policies through the decentralized benchmarking system, they effectively get a much broader evaluation coverage for the same effort. To support participation of researchers without access to a physical robot (and thus without the ability to contribute evaluations themselves), multiple institutions agreed to “sponsor” a base budget of evaluations that will be distributed among all users “free of charge”.

最后，为了让 DROID-RoboArena 长期自给自足，我们实现了一套“ 评测积分 ”系统来平衡评测的供给与需求：评测者每运行一次成对策略评测，就获得一个积分，可以用它请求同等数量的、在自己的策略与池中其他策略之间进行的成对比较。这样，任何个人付出的评测工作量，与只在自己的设备上评测自己的策略相当；但通过同意在去中心化基准系统中评测他人的策略，他们以同样的付出获得了宽广得多的评测覆盖。为了支持没有实体机器人（因而无法自己贡献评测）的研究者参与，多家机构同意“赞助”一定的基础评测额度，“免费”分配给所有用户。

5 Experiments

5 实验

The goal of our experimental evaluation is to answer the following questions: (1) How does RoboArena compare to conventional robot evaluation approaches for ranking the performance of generalist robot policies? (2) How sample efficient is RoboArena in ranking robot policies? (3) Can RoboArena extract qualitative insights about policy performance beyond aggregate performance?

我们的实验评测旨在回答以下问题：(1) 在对通用机器人策略的性能排序方面，RoboArena 与传统机器人评测方法相比如何？(2) RoboArena 对机器人策略排序的 样本效率 如何？(3) 除总体性能之外，RoboArena 能否提取关于策略性能的 定性 洞见？

5.1 Experimental Setup

5.1 实验设置

\pi_{0}

\pi_{0}

Comparison. We establish an “oracle” policy ranking by exhaustively evaluating all policies on all tested tasks and comparing their average progress scores. This was done by asking evaluators to score the remaining five policies after each A/B evaluation on the same task. In total, we use 4284 evaluations to compute this oracle ranking. We compare RoboArena to a conventional robot policy evaluation approach, which tests all policies on a fixed, tightly standardized set of evaluation tasks in a narrower set of environments. Concretely, we compare to the DROID evaluation procedure used in Pertsch et al. [5], which consists of 17 tasks and 44 episodes per policy and is representative of typical robot evaluations ([3, 68], see the appendix of Pertsch et al. [5] for a detailed list of tasks).

对比方式 。我们通过让 所有 策略在 所有 测试任务上进行穷举评测、比较它们的平均进度分数，建立一个“神谕”策略排名。具体做法是：每次 A/B 评测之后，请评测者在同一任务上再为其余五个策略打分。我们总共用 4284 次评测来计算这一神谕排名。我们把 RoboArena 与一种传统的机器人策略评测方法作比较，后者在一个范围较窄的环境集合中、用一组固定且严格标准化的评测任务测试所有策略。 具体来说，我们与 Pertsch 等人 [5] 所用的 DROID 评测流程作比较 ，它包含 17 个任务，每个策略 44 个回合，代表了典型的机器人评测（[3, 68]；详细任务列表见 Pertsch 等人 [5] 的附录）。

5.2 RoboArena Accurately Ranks Policy Performances

5.2 RoboArena 能准确地对策略性能排序

Figure 5: Left : Examples of RoboArena evaluations. Evaluations span a diverse set of scenes and tasks. Right : “Oracle” policy ranking, aggregated from progress scores of 4284 evaluation rollouts.

图 5： 左 ：RoboArena 评测示例。评测覆盖多样的场景和任务。 右 ：由 4284 个评测回合的进度分数汇总得到的“神谕”策略排名。

Figure 6: Policy rankings from RoboArena pairwise comparisons correlate significantly better with oracle rankings than conventional robot evaluation approaches (“Regular”). Our task-aware ranking approach (“TASK”) leads to the best ranking. Ranking with progress scores (“PROG”) also proves effective, but may lose more nuanced information about policy performance.

图 6：由 RoboArena 成对比较得到的策略排名，与神谕排名的相关性明显好于传统机器人评测方法（“Regular”）。我们的任务感知排名方法（“TASK”）得到最好的排名。用进度分数排名（“PROG”）也很有效，但可能丢失关于策略性能的更细致信息。

In total, our RoboArena evaluation covers 4284 policy rollouts across dozens of scenes and hundreds of task instructions. We show a few examples in Figure 5, left, and many more in Appendix C. To our knowledge, this is the most extensive evaluation of generalist policies to date. While the remainder of this section is devoted to evaluating our RoboArena evaluation framework itself, we provide a detailed analysis of the underlying evaluation data, including strengths and weaknesses of current generalist policies, in Appendix F.

我们的 RoboArena 评测总共覆盖了 4284 个策略回合，涉及数十个场景和数百条任务指令。图 5 左展示了少量示例，附录 C 中还有更多。据我们所知，这是迄今为止对通用策略最广泛的评测。本节其余部分专门评测 RoboArena 评测框架本身，而对底层评测数据的详细分析，包括当前通用策略的优缺点，见附录 F。

r

r

\pi_{0}

\pi_{0}

We also test using an average of the progress scores for each of the pairwise policy comparisons to compute a global ranking (“PROG”). The results in Figure 6 show that this strategy achieves good correlation to the oracle as well. However, ranking policies based on progress scores alone can miss more nuanced feedback on policy performance. We find that evaluators regularly score both policies in an A/B comparison with the same progress score, yet express clear preference for one policy over the other, e.g., because it acted more swiftly or confidently (see Appendix D for examples). Thus, progress-based and preference-based rankings are complementary, and should both be reported.

我们还测试了用每次成对策略比较的进度分数平均值来计算全局排名（“PROG”）。图 6 的结果表明，这一策略与神谕排名也有良好的相关性。然而，只根据进度分数为策略排序，可能会遗漏关于策略性能的更细致反馈。我们发现，评测者经常在一次 A/B 比较中给两个策略打 相同 的进度分数，却明确表达出对其中一个的 偏好 ，例如因为它动作更迅速或更果断（示例见附录 D）。因此，基于进度的排名和基于偏好的排名是互补的，两者都应报告。

5.3 RoboArena Evaluation is Sample Efficient

5.3 RoboArena 评测具有样本效率

Figure 7: Rank correlation as a function of number of evaluation episodes. RoboArena converges to high-quality rankings within just 100 pairwise comparisons.

图 7：排名相关性随评测回合数的变化。RoboArena 只需 100 次成对比较就能收敛到高质量的排名。

We investigate how many evaluations are required to produce an accurate ranking using RoboArena: we compute rankings for differently-sized, random subsets of our full evaluation data and report ranking accuracy as a function of the number of evaluation episodes in Figure 7. We find that RoboArena converges to high-quality rankings within just 100 pairwise comparisons, matching the convergence speed of conventional robot evaluations, while providing significantly more accurate rankings. The quality of RoboArena rankings further improves as more comparisons are collected. This suggests, that distributed RoboArena evaluations offer an appealing alternative to regular policy evaluations.

我们考察用 RoboArena 得到准确排名需要多少次评测：对完整评测数据中不同大小的随机子集计算排名，并在图 7 中报告排名准确度随评测回合数的变化。我们发现，RoboArena 只需 100 次成对比较就能收敛到高质量的排名，收敛速度与传统机器人评测相当，同时给出的排名明显更准确。随着收集到的比较增多，RoboArena 排名的质量还会进一步提高。这表明，分布式的 RoboArena 评测是常规策略评测的一个有吸引力的替代方案。

5.4 Extracting qualitative policy characteristics

5.4 提取定性的策略特征

Model Task Category Acc. GPT-4o 94.6% Figure 8: Task categorization accuracy (448 samples).

Model	Task Category Acc.
GPT-4o	94.6%

模型 任务类别准确率 GPT-4o 94.6% 图 8：任务分类准确率（448 个样本）。 （译注：此表中的模型名原文为 GPT-4o，而第 3.3 节正文写的是 GPT‑4.5，两处不一致，均按原文照录。）

模型	任务类别准确率
GPT-4o	94.6%

In this section, we evaluate the quality of the LLM/VLM-assisted policy reports (Section 3.3). First, we evaluate the VLM category predictions by comparing them to category assignments made by a human expert, and we find that they are approximately 95% accurate (Figure 8).

本节评测 LLM/VLM 辅助生成的 策略报告 （第 3.3 节）的质量。首先，我们把 VLM 的类别预测与人类专家的类别标注相比较，以此评测前者，发现其准确率约为 95%（图 8）。

\pi_{0}

\pi_{0}

\pi_{0}

\pi_{0}

6 Discussion

6 讨论

We introduced RoboArena, a distributed framework for evaluating generalist robot policies. We have shown that by aggregating evaluations across a decentralized network of evaluators, each running pairwise policy comparisons on many different tasks and scenes, RoboArena can generate more accurate policy performance rankings than conventional, centralized evaluation approaches, while retaining high evaluation sample efficiency. We have also introduced prototype tools for LLM-assisted analysis of the evaluation results. We will open-source our RoboArena evaluation framework and give other researchers access for contributing policies and evaluations, in an effort to make evaluations of generalist robot policies more comparable.

我们提出了 RoboArena，一个用于评测通用机器人策略的分布式框架。我们表明，通过汇总去中心化评测者网络中的评测——每位评测者在许多不同的任务和场景上进行成对策略比较——RoboArena 能比传统的集中式评测方法得到更准确的策略性能排名，同时保持很高的评测样本效率。我们还提出了用 LLM 辅助分析评测结果的原型工具。我们将开源 RoboArena 评测框架，并向其他研究者开放提交策略和贡献评测的通道，以期让通用机器人策略的评测更具可比性。

7 Limitations

7 局限

Cross-embodiment. While RoboArena is a general approach for robot evaluation, the experimental evaluations in this paper have focused on the DROID platform [1], since it was well-suited for developing our evaluation framework. However, there is a growing interest in developing cross-embodiment policies, that can not only operate across many scenes and tasks, but also across robot embodiments. Future work should investigate how RoboArena can be extended to diverse robot embodiments and still support policies that may only be evaluatable on specific embodiments.

跨本体 。虽然 RoboArena 是一种通用的机器人评测方法，但本文的实验评测集中在 DROID 平台 [1] 上，因为它很适合用来开发我们的评测框架。然而，人们对开发 跨本体 策略的兴趣日益增长，这类策略不仅能在许多场景和任务中工作，还能跨越不同的机器人本体。未来的工作应研究如何把 RoboArena 扩展到多样的机器人本体，同时仍能支持那些可能只能在特定本体上评测的策略。

Controlled experimentation. The design of RoboArena, which is focused on decentralized evaluation without restrictions on tasks or scenes, makes it challenging to perform experiments that only vary a single condition at a time (e.g. only camera angle, or only object position). As such, RoboArena is complementary to targeted, smaller-scale evaluations that focus on individual axes of generalization [72].

受控实验 。RoboArena 的设计侧重于不限制任务或场景的去中心化评测，这使得一次只改变 单一 条件（例如只改变相机角度，或只改变物体位置）的实验难以进行。因此，RoboArena 与聚焦于单个泛化维度的、有针对性的小规模评测 [72] 是互补的。

Adversarial evaluators. While RoboArena’s distributed, double-blind evaluation scheme gives it an inherent robustness against individual influencing, we have not investigated its robustness to intentionally adversarial evaluators that try to temper with evaluation results, for example by providing random preference ratings or intentionally misleading language feedback. Future work should investigate how distributed robot evaluation approaches can be hardened against such tampering.

对抗性评测者 。虽然 RoboArena 的分布式双盲评测方案使它天生能抵御个体的影响，但我们尚未研究它对蓄意对抗的评测者的稳健性，这类评测者可能试图篡改评测结果，例如给出随机的偏好评分或故意误导的语言反馈。未来的工作应研究如何加固分布式机器人评测方法，使其能抵御此类篡改。

Over-optimization and Gotthart’s Law. A common wisdom is that a measure, e.g., for model performance, ceases to be a good measure when it becomes a target. This potential for over-optimization, also known as Gotthart’s law, is innate to any measure, and has recently been shown to also impact crowd-sourced evaluations [73], even if they may inherently be more robust to such over-optimization than static benchmarks. While in robotics, the current (limited) performance of policies makes such over-optimization less likely, future work should critically examine whether evaluations with approaches like RoboArena remain well-correlated with perceived real-world policy performance.

过度优化与 Gotthart 定律 。一个常见的看法是：一项度量（例如模型性能的度量）一旦成为目标，就不再是好的度量。这种过度优化的可能性，也称 Gotthart 定律，是任何度量所固有的，而且最近已有研究表明它同样会影响众包评测 [73]，尽管众包评测可能天生比静态基准更能抵御此类过度优化。虽然在机器人领域，策略目前（有限）的性能使这种过度优化不太可能发生，但未来的工作应审慎检验：用 RoboArena 这类方法得到的评测结果，是否仍与人们感知到的真实世界策略性能保持良好的相关性。 （译注：此定律通常称为 Goodhart 定律，原文拼作 “Gotthart”，按原文照录。）

Acknowledgments We thank Siyi Huang, Ellie Huynh, Emiliano Adrian Sanchez, Justin Kang, and Sarah Kunda for help with evaluating policies. We also thank Kyle Stachowicz for help with repairing the DROID robot station during development. This research was partly supported by RAI, ONR N00014-25-1-2060, ONR N00014-22-1-2621, ONR N00014-22-12677NSF, NFS IIS-2150826, NSF CAREER 2239301, NSF 2331783, DARPA HR00112490428, DARPA TIAMAT HR00112490421, the Amazon Science Hub, the Toyota Research Institute, the Army Research Lab, and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (RS-2024-00333634). We acknowlegde compute support from the Google TPU Research cloud (TRC). We also want to acknowledge funding support from Natural Sciences and Engineering Research Council of Canada, Fonds de recherche du Québec and The Canadian Institute for Advanced Research (CIFAR) and compute support from Digital Research Alliance of Canada, Mila IDT, and NVidia. We would also like to thank John Edwards Leadership Fund (CFI) for funding the purchase of part of the hardware for this project.

致谢 我们感谢 Siyi Huang、Ellie Huynh、Emiliano Adrian Sanchez、Justin Kang 和 Sarah Kunda 帮助评测策略。我们也感谢 Kyle Stachowicz 在开发期间帮助修理 DROID 机器人工作站。本研究部分得到以下资助：RAI、ONR N00014-25-1-2060、ONR N00014-22-1-2621、ONR N00014-22-12677NSF、NFS IIS-2150826、NSF CAREER 2239301、NSF 2331783、DARPA HR00112490428、DARPA TIAMAT HR00112490421、Amazon Science Hub、丰田研究院、美国陆军研究实验室，以及由韩国政府（MSIT）资助的韩国国家研究基金会（NRF）项目（RS-2024-00333634）。我们感谢 Google TPU Research Cloud（TRC）提供的算力支持。我们还感谢加拿大自然科学与工程研究理事会、魁北克研究基金（Fonds de recherche du Québec）和加拿大高等研究院（CIFAR）的资助，以及加拿大数字研究联盟（Digital Research Alliance of Canada）、Mila IDT 和 NVIDIA 的算力支持。我们也感谢 John Edwards 领导力基金（CFI）资助购买了本项目的部分硬件。

References

参考文献

\pi_0

[61] A. E. Elo. The proposed uscf rating system, its development, theory, and applications. Chess life , 22(8):242-247, 1967. [62] E. Zermelo. Die berechnung der turnier-ergebnisse als ein maximumproblem der wahrscheinlichkeitsrechnung. Mathematische Zeitschrift , 29(1):436-460, 1929. [63] R. R. Davidson. On extending the bradley-terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association , 65(329):317-328, 1970. [64] A. Z. Ren. Open pi zero: An open source raspberry pi zero. https://github.com/allenzren/open-pi-zero, 2025. [65] P. Intelligence. Openpi. https://github.com/Physical-Intelligence/openpi, 2025. [66] L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 , 2024. [67] F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen. Finite scalar quantization: Vq-vae made simple, 2023. URL https://arxiv.org/abs/2309.15505 . [68] M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine. Robotic control via embodied chain-of-thought reasoning. In Conference on Robot Learning , 2024. [69] S. Lee, Y. Wang, H. Etukuru, H. J. Kim, N. M. M. Shafiullah, and L. Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403.03181 , 2024. [70] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research , page 02783649241273668, 2023.

[71] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. pi0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054 , 2025. [72] J. Gao, S. Belkhale, S. Dasari, A. Balakrishna, D. Shah, and D. Sadigh. A taxonomy for evaluating generalist robot policies. arXiv preprint arXiv:2503.01238 , 2025. [73] The Register. Regarding Meta, Llama 4, and Cheating, 2025. URL https://www.theregister.com/2025/04/08/meta_llama4_cheating/ .

Appendix A Contributions

附录 A 贡献

RoboArena system design and implementation : Pranav Atreya, Karl Pertsch, Tony Lee

RoboArena 系统设计与实现 ：Pranav Atreya、Karl Pertsch、Tony Lee

Experiment design and analysis : Pranav Atreya, Karl Pertsch, Tony Lee

实验设计与分析 ：Pranav Atreya、Karl Pertsch、Tony Lee

Policy evaluation : Pranav Atreya, Tony Lee, Moo Jin Kim, Karl Pertsch, Arhan Jain, Artur Kuramshin, Cyrus Neary, Edward Hu, Kanav Arora, Kirsty Ellis, Luca Macesanu, Matthew Leonard, Meedeum Cho, Ozgur Aslan, Shivin Dass, Jie Wang, William Reger, Xingfang Yuan

策略评测 ：Pranav Atreya、Tony Lee、Moo Jin Kim、Karl Pertsch、Arhan Jain、Artur Kuramshin、Cyrus Neary、Edward Hu、Kanav Arora、Kirsty Ellis、Luca Macesanu、Matthew Leonard、Meedeum Cho、Ozgur Aslan、Shivin Dass、Jie Wang、William Reger、Xingfang Yuan

Simulated evaluation environments : Arhan Jain, Karl Pertsch, Xuning Yang, Clemens Eppner, Fabio Ramos, Jonathan Tremblay

仿真评测环境 ：Arhan Jain、Karl Pertsch、Xuning Yang、Clemens Eppner、Fabio Ramos、Jonathan Tremblay

Paper writing : Karl Pertsch, Pranav Atreya, Tony Lee

论文写作 ：Karl Pertsch、Pranav Atreya、Tony Lee

Project coordination : Karl Pertsch, Pranav Atreya

项目协调 ：Karl Pertsch、Pranav Atreya

Advising : Sergey Levine, Chelsea Finn, Percy Liang, Abhishek Gupta, Dinesh Jayaraman, Glen Berseth, Kostas Daniilidis, Roberto Martin-Martin, Youngwoon Lee

指导 ：Sergey Levine、Chelsea Finn、Percy Liang、Abhishek Gupta、Dinesh Jayaraman、Glen Berseth、Kostas Daniilidis、Roberto Martin-Martin、Youngwoon Lee

Appendix B Policy Ranking EM Procedure

附录 B 策略排名的 EM 流程

Here we give additional details on our proposed policy ranking algorithm, introduced in section 3.2. We first describe the underlying probabilistic model, derive all necessary gradients and Hessians, then present the complete EM pseudocode. We then show how the algorithm can be easily modified to also use partial-success information when present, outline the algorithms behind the other ranking procedures discussed in the paper, and then list all hyperparameters.

这里补充说明第 3.2 节提出的策略排名算法。我们先描述底层的概率模型，推导所需的全部梯度和 Hessian，然后给出完整的 EM 伪代码。接着说明如何简单修改该算法，使其在有部分成功信息时也加以利用；概述文中讨论的其他排名方法背后的算法；最后列出全部超参数。

B.1 Model Definition

B.1 模型定义

N

N

\Pi=\{\pi_{1},\pi_{2},\dots,\pi_{N}\}.

n

n

y_{n}\;\in\;\{0,1,2\},\quad 0=\text{loss},\;1=\text{tie},\;2=\text{win},\quad n=1,\dots,M.

y_{n}\;\in\;\{0,1,2\},\quad 0=\text{负},\;1=\text{平},\;2=\text{胜},\quad n=1,\dots,M.

M

M

\mathcal{D}_{p}=\bigl\{\,\bigl(i_{n},j_{n},y_{n}\bigr)\bigr\}_{n=1}^{M}.

At no point do we assume the true “task” identifiers are observed; instead we infer a small number of latent “task‐buckets” that capture varying difficulty levels.

我们从不假设真实的“任务”标识可以观测到；相反，我们推断少量潜在的“任务桶”，用来刻画不同的难度水平。

T

T

\nu=(\nu_{1},\dots,\nu_{T}),\quad\nu_{t}\geq 0,\;\;\sum_{t=1}^{T}\nu_{t}=1,

t

t

\pi_{p}

\pi_{p}

\theta=(\theta_{1},\dots,\theta_{N}),\quad\psi=\{\psi_{p,t}\}_{p=1..N,\;t=1..T},\quad\tau=(\tau_{1},\dots,\tau_{T}).

\theta_{p}

\theta_{p}

\nu_{\rm tie}\in(0,1)

\nu_{\rm tie}\in(0,1)

n

n

z_{i_{n},t}=\theta_{i_{n}}+\psi_{i_{n},t}-\tau_{t},\quad z_{j_{n},t}=\theta_{j_{n}}+\psi_{j_{n},t}-\tau_{t}.

\sigma(z)=1/(1+e^{-z})

\sigma(z)=1/(1+e^{-z})

q_{i_{n},t}=\sigma(z_{i_{n},t}),\quad q_{j_{n},t}=\sigma(z_{j_{n},t}).

i_{n}

t

P(y_{n}=2\mid t)=q_{i_{n},t}\,\bigl(1-q_{j_{n},t}\bigr),\quad P(y_{n}=0\mid t)=\bigl(1-q_{i_{n},t}\bigr)\,q_{j_{n},t},

P(y_{n}=1\mid t)=2\,\nu_{\rm tie}\,\sqrt{q_{i_{n},t}(1-q_{i_{n},t})\,q_{j_{n},t}(1-q_{j_{n},t})}\,,

t

t

P(y_{n})=\sum_{t=1}^{T}\nu_{t}\;P\!\bigl(y_{n}\mid t\bigr).

This completes the model definition.

模型定义至此完成。

B.2 Gradient & Hessian Derivation

B.2 梯度与 Hessian 的推导

Below we derive all first and second derivatives needed for the clipped‐Newton M‐step. We begin by writing the expected complete‐data objective and then expand gradients and Hessians for each parameter block.

下面推导截断牛顿 M 步所需的全部一阶和二阶导数。我们先写出完全数据的期望目标函数，再对每个参数块展开梯度和 Hessian。

\Theta=(\theta,\psi,\tau,\nu,\nu_{\rm tie})

\Theta=(\theta,\psi,\tau,\nu,\nu_{\rm tie})

\gamma_{n,t}=P\bigl(t\mid y_{n};\Theta^{\rm old}\bigr)=\frac{\nu_{t}\,P(y_{n}\mid t;\Theta^{\rm old})}{\sum_{u=1}^{T}\nu_{u}\,P(y_{n}\mid u;\Theta^{\rm old})}.

\lambda_{\theta},\lambda_{\psi}

\lambda_{\theta},\lambda_{\psi}

Q(\Theta)=\sum_{n=1}^{M}\sum_{t=1}^{T}\gamma_{n,t}\,\Bigl[\log\nu_{t}+\log P(y_{n}\mid t)\Bigr]\;-\;\frac{\lambda_{\theta}}{2}\sum_{p=1}^{N}\theta_{p}^{2}\;-\;\frac{\lambda_{\psi}}{2}\sum_{p=1}^{N}\sum_{t=1}^{T}\psi_{p,t}^{2}.

\theta_{p}

\theta_{p}

\frac{\partial Q}{\partial\theta_{p}}=\sum_{n\in\mathcal{I}_{p}}\sum_{t=1}^{T}\gamma_{n,t}\,\frac{\partial}{\partial z_{i_{n},t}}\log P(y_{n}\mid t)-\sum_{n\in\mathcal{J}_{p}}\sum_{t=1}^{T}\gamma_{n,t}\,\frac{\partial}{\partial z_{j_{n},t}}\log P(y_{n}\mid t)-\lambda_{\theta}\,\theta_{p}.

\frac{\partial Q}{\partial\theta_{p}}=\sum_{n\in\mathcal{I}_{p}}\sum_{t=1}^{T}\gamma_{n,t}\,\frac{\partial}{\partial z_{i_{n},t}}\log P(y_{n}\mid t)-\sum_{n\in\mathcal{J}_{p}}\sum_{t=1}^{T}\gamma_{n,t}\,\frac{\partial}{\partial z_{j_{n},t}}\log P(y_{n}\mid t)-\lambda_{\theta}\,\theta_{p}.

y_{n}=2

y_{n}=2

\frac{\partial}{\partial z_{i,t}}\log P(y_{n}=2\mid t)=1-\sigma(z_{i,t}),\qquad\frac{\partial}{\partial z_{j,t}}\log P(y_{n}=2\mid t)=-\,\sigma(z_{j,t}),

with analogous expressions in the loss or tie case.

负或平局的情形有类似的表达式。

\theta_{p}

\theta_{p}

\frac{\partial^{2}Q}{\partial\theta_{p}^{2}}=-\,\sum_{n\in\mathcal{I}_{p}}\sum_{t}\gamma_{n,t}\,\sigma(z_{i,t})(1-\sigma(z_{i,t}))-\sum_{n\in\mathcal{J}_{p}}\sum_{t}\gamma_{n,t}\,\sigma(z_{j,t})(1-\sigma(z_{j,t}))-\lambda_{\theta}.

\psi_{p,t}

\psi_{p,t}

\frac{\partial Q}{\partial\psi_{p,t}}=\sum_{n:i_{n}=p}\gamma_{n,t}\,\frac{\partial\log P}{\partial z_{i_{n},t}}-\sum_{n:j_{n}=p}\gamma_{n,t}\,\frac{\partial\log P}{\partial z_{j_{n},t}}-\lambda_{\psi}\,\psi_{p,t},

\frac{\partial^{2}Q}{\partial\psi_{p,t}^{2}}=-\sum_{n:i_{n}=p}\gamma_{n,t}\,\sigma(z_{i,t})(1-\sigma(z_{i,t}))-\sum_{n:j_{n}=p}\gamma_{n,t}\,\sigma(z_{j,t})(1-\sigma(z_{j,t}))-\lambda_{\psi}.

\tau_{t}

\tau_{t}

\frac{\partial Q}{\partial\tau_{t}}=-\sum_{n=1}^{M}\gamma_{n,t}\,\Bigl[\tfrac{\partial}{\partial z_{i,t}}\log P(y_{n}\mid t)+\tfrac{\partial}{\partial z_{j,t}}\log P(y_{n}\mid t)\Bigr],

\frac{\partial^{2}Q}{\partial\tau_{t}^{2}}=-\sum_{n=1}^{M}\gamma_{n,t}\,\Bigl[\sigma(z_{i,t})(1-\sigma(z_{i,t}))+\sigma(z_{j,t})(1-\sigma(z_{j,t}))\Bigr].

All of these gradient and Hessian terms are used in the M‐step clipped‐Newton updates described in Algorithm 2.

以上所有梯度和 Hessian 项都用于算法 2 所述的 M 步截断牛顿更新。

\{(i_{n},j_{n},y_{n})\}_{n=1}^{M}

\{(i_{n},j_{n},y_{n})\}_{n=1}^{M}

B.3 Including Partial‐Success Information

B.3 纳入部分成功信息

\,s^{(i)}_{n},s^{(j)}_{n}\in[0,1]

\,s^{(i)}_{n},s^{(j)}_{n}\in[0,1]

B.4 Baseline Ranking Algorithms

B.4 基线排名算法

Below we summarize the simpler ranking methods we compare against, including both classic preference‐only approaches and a simple partial‐success approach.

下面概述我们用作对比的较简单的排名方法，既包括经典的只用偏好的方法，也包括一种简单的部分成功方法。

Bradley-Terry MLE (offline). In the standard Bradley-Terry maximum‐likelihood estimator, we posit

Bradley-Terry 最大似然估计（离线）。 在标准的 Bradley-Terry 最大似然估计中，我们假定

P(y_{n}=2)\;=\;\sigma\bigl(\theta_{i_{n}}-\theta_{j_{n}}\bigr),

\theta

\theta

\theta_{p}\;\leftarrow\;\theta_{p}\;+\;\eta\sum_{n:\,i_{n}=p}\Bigl[y_{n}-\sigma\bigl(\theta_{i_{n}}-\theta_{j_{n}}\bigr)\Bigr]\;-\;\eta\sum_{n:\,j_{n}=p}\Bigl[y_{n}-\sigma\bigl(\theta_{i_{n}}-\theta_{j_{n}}\bigr)\Bigr],

\eta

\eta

A

A

\theta_{A}\;\leftarrow\;\theta_{A}\;+\;K\,\Bigl(y_{AB}-\sigma(\theta_{A}-\theta_{B})\Bigr),\quad\theta_{B}\;\leftarrow\;\theta_{B}\;-\;K\,\Bigl(y_{AB}-\sigma(\theta_{A}-\theta_{B})\Bigr),

K

K

p

p

\bar{s}_{p}\;=\;\frac{1}{N_{p}}\sum_{n:\,i_{n}=p\text{ or }j_{n}=p}\bigl[s_{n}^{(p)}\bigr],

\bar{s}_{p}\;=\;\frac{1}{N_{p}}\sum_{n:\,i_{n}=p\text{ 或 }j_{n}=p}\bigl[s_{n}^{(p)}\bigr],

N_{p}

N_{p}

B.5 Hyperparameter Table

B.5 超参数表

\{50,100,200\}

Parameter	Default	Search range
EM_ITERS	60	$\{50,100,200\}$
# buckets $T$	60	$\{10...200\}$
step_clip	1.0	$[0.1,\,10]$
l2_theta	$10^{-2}$	$\{10^{-3},10^{-2},10^{-1}\}$
l2_psi	$10^{-2}$	$\{10^{-3},10^{-2},10^{-1}\}$
step_decay	0.99	$\{0.9,0.99,0.999\}$
tol	$10^{-4}$	$\{10^{-5},10^{-4}\}$

\{50,100,200\}

参数	默认值	搜索范围
EM_ITERS	60	$\{50,100,200\}$
桶数 $T$	60	$\{10...200\}$
step_clip	1.0	$[0.1,\,10]$
l2_theta	$10^{-2}$	$\{10^{-3},10^{-2},10^{-1}\}$
l2_psi	$10^{-2}$	$\{10^{-3},10^{-2},10^{-1}\}$
step_decay	0.99	$\{0.9,0.99,0.999\}$
tol	$10^{-4}$	$\{10^{-5},10^{-4}\}$

B.6 Simulating Shifts in the Task Distribution and Policy Set

B.6 模拟任务分布和策略集合的变化

The intent of the RoboArena platform is for it to serve as a long-running evaluation and benchmarking resource for the robotics community. In such a setting, it is likely that the policy set will evolve over time, with weaker policies getting dropped and stronger policies getting added to the pool. At the same time, as the capabilities of policies evolve, evaluators will likely adjust the tasks they set up for evaluation to better test the new policy capabilities.

RoboArena 平台的目的是作为机器人社区一项长期运行的评测和基准资源。在这种情形下，策略集合很可能随时间演变：较弱的策略被淘汰，更强的策略被加入池中。同时，随着策略能力的演进，评测者很可能会调整他们设置的评测任务，以更好地检验新的策略能力。

We test whether the RoboArena ranking model will remain valid and performant under distribution shifts over time. We rerun our rankings with artificial drift in task difficulty (as judged by average per-task progress scores), starting with easy tasks and moving towards harder tasks over time, while at the same time phasing out weaker policies and adding stronger policies to the evaluation pool over time (again judged by per-policy progress scores). This simulates RoboArena’s development as a community benchmark with increasingly capable policies and increasingly challenging task distributions. The results in Table 2 show that even under these more challenging circumstances, RoboArena evaluations retain significantly higher correlation with oracle scores than “Regular” evaluations in a single laboratory.

我们检验 RoboArena 的排名模型在随时间发生的分布变化下是否仍然有效、表现良好。我们在人为制造的任务难度漂移（以各任务的平均进度分数衡量）下重新计算排名：从简单任务开始，随时间逐渐转向更难的任务；同时随时间逐步淘汰较弱的策略、加入更强的策略（同样以各策略的进度分数衡量）。这模拟了 RoboArena 作为社区基准的发展过程：策略能力越来越强，任务分布越来越有挑战性。表 2 的结果表明，即使在这种更具挑战性的情况下，RoboArena 评测与神谕分数的相关性仍明显高于在单个实验室进行的“Regular”评测。

r(\uparrow)

Eval Approach	Pearson $r(\uparrow)$	MMRV $(\downarrow)$
Regular	0.692	0.141
RoboArena w/ dist. shifts (ours)	0.838	0.058

r(\uparrow)

评测方式	Pearson $r(\uparrow)$	MMRV $(\downarrow)$
Regular	0.692	0.141
有分布变化的 RoboArena（本文）	0.838	0.058

Appendix C Evaluation Data Breakdown

附录 C 评测数据细分

Here we depict in detail the evaluation data collected by RoboArena, emphasizing it’s diversity and scale. To the best of our knowledge, the evaluations done thus far with RoboArena constitute the most comprehensive evaluations of generalist policies to date.

这里详细展示 RoboArena 收集的评测数据，着重说明其多样性和规模。据我们所知，迄今为止用 RoboArena 完成的评测构成了对通用策略最全面的评测。

Figure 10 depicts on the top the diversity in the verb making up the task command, and on the bottom the wide range of object classes that are being interacted with during the evaluation episodes. Generalist policies are uniquely data demanding when it comes to evaluations, as to properly assess a policy’s performance as a generalist, the policy must be queried with a diverse array of tasks. Indeed, figure 10 shows that the RoboArena evaluation procedure permits the required diversity in task commands and objects. Figure 11 similarly shows a representation of the diversity of RoboArena evaluations with respect to the set of scenes upon which evaluations were performed. The environments sampled were chosen randomly from the pool of all evaluation episodes.

图 10 上半部分展示了构成任务指令的 动词 的多样性，下半部分展示了评测回合中被操作物体类别的广泛范围。在评测方面，通用策略对数据的需求格外大，因为要恰当评估一个策略作为通用策略的性能，就必须用多样的任务去考它。图 10 确实表明，RoboArena 的评测流程能提供所需的任务指令和物体多样性。图 11 同样展示了 RoboArena 评测在所用场景集合方面的多样性。所展示的环境是从全部评测回合中随机抽取的。

Figure 10: Top: bar chart of the most common verbs used in the task commands, along with their frequencies in the evaluation data collected. The task commands exhibit a diverse array of instructions, including ”uncapping”, ”unplugging”, ”finding”, ”dusting”, etc. Bottom: visualization of the diversity of object categories being commanded to be interacted with, bucketed by major objects types.

图 10： 上 ：任务指令中最常用动词的柱状图，以及它们在收集到的评测数据中的频次。任务指令涵盖多种多样的指示，包括“拔掉盖子”“拔掉插头”“寻找”“擦灰”等。 下 ：被要求操作的物体类别的多样性可视化，按主要物体类型分组。

Figure 11: Environments in RoboArena are diverse, due to RoboArena’s distributed nature and the eschewing of a standardization of tasks. Here we depict 32 sample environments used for RoboArena across the network of participating institutions. Evaluators were encouraged to scale diversity, leading to a visible heterogeneity of environments. Even when the physical location of the robot was the same, lighting, camera viewpoints, tablecloths, and objects were often altered significantly, again made easy by the fact that exact scenes and tasks need not be standardized.

图 11：由于 RoboArena 的分布式特性以及不对任务做标准化，RoboArena 中的环境非常多样。这里展示了参与机构网络中 RoboArena 所用的 32 个环境样例。评测者被鼓励扩大多样性，因此环境呈现出明显的异质性。即使机器人的物理位置相同，光照、相机视角、桌布和物体也经常发生明显变化，这同样得益于无需对确切的场景和任务做标准化。

Appendix D Nuances in Policy Preference Labels

附录 D 策略偏好标签中的细微差别

Evaluators participating in the RoboArena evaluation framework are asked to provide a rough estimate of partial success of each rollout on the prescribed task, but also critically, a preference label specifying whether they liked policy A versus B on the same task. As we will outline here, this preference feedback can contain quite a bit of information beyond what is present in the partial success feedback. Concretely, figure 3 depicts a few examples where partial success feedback was the same for policies A and B, yet the evaluator marked a clear preference for either A or B.

参与 RoboArena 评测框架的评测者需要粗略估计每个回合在指定任务上的部分成功程度，但同样关键的是，还要给出一个偏好标签，说明在同一任务上他们更喜欢策略 A 还是 B。正如下文将要说明的，这种偏好反馈可能包含相当多部分成功反馈中所没有的信息。具体来说，图 3 列出了几个例子： 策略 A 和 B 的部分成功反馈相同 ，但评测者对 A 或 B 表达了明确的偏好。 （译注：此处所指应为下面的表 3，原文写作 “figure 3”，按原文照录。）

Session ID Preferred Policy Long‐form Feedback 748 A Although both policies don’t put the cloth on the screwdriver, policy A places the cloth close to the banana, while policy B does not seem close to either object. 674 B Both policies completed the entire task but policy B did it on the first try. After the first grasp and lift, it feels like policy A dropped the mouse prematurely. It then picked it up again and moved it further. 660 B Policy B initially hesitated to move long distance but later transitioned to effective and rapid movements. Meanwhile, policy A also succeeded at the task, but it exhibited more sluggish movements. 649 A Both A and B picked up the cup instead of pushing, and both then placed it on the table. After letting go, A returned to a starting pose while B kept repeatedly grabbing the cup, which is sub‐optimal. 648 B While both did take the object out of the bowl (qualifying for full score), B placed the object on the table area next to the bowl. This is the more natural thing to do. 641 A Both policy A and policy B almost solved the task completely. However, policy A displayed more decisive motions with less corrective behavior while policy B solved the task by chance after multiple attempts. 579 A Both policies were able to close the drawer most of the way. However, policy A went straight for the drawer and didn’t stop until it was closed. Policy B got distracted by the plastic food items halfway through (though it eventually returned to close the drawer). Table 3: Examples of preference feedback for A/B pairs for which partial success feedback was equal for A and B, illustrating information beyond binary partial‐success scores.

Session ID	Preferred Policy	Long‐form Feedback
748	A	Although both policies don’t put the cloth on the screwdriver, policy A places the cloth close to the banana, while policy B does not seem close to either object.
674	B	Both policies completed the entire task but policy B did it on the first try. After the first grasp and lift, it feels like policy A dropped the mouse prematurely. It then picked it up again and moved it further.
660	B	Policy B initially hesitated to move long distance but later transitioned to effective and rapid movements. Meanwhile, policy A also succeeded at the task, but it exhibited more sluggish movements.
649	A	Both A and B picked up the cup instead of pushing, and both then placed it on the table. After letting go, A returned to a starting pose while B kept repeatedly grabbing the cup, which is sub‐optimal.
648	B	While both did take the object out of the bowl (qualifying for full score), B placed the object on the table area next to the bowl. This is the more natural thing to do.
641	A	Both policy A and policy B almost solved the task completely. However, policy A displayed more decisive motions with less corrective behavior while policy B solved the task by chance after multiple attempts.
579	A	Both policies were able to close the drawer most of the way. However, policy A went straight for the drawer and didn’t stop until it was closed. Policy B got distracted by the plastic food items halfway through (though it eventually returned to close the drawer).

会话 ID 更偏好的策略 详细反馈 748 A 虽然两个策略都没有把布放到螺丝刀上，但策略 A 把布放在了靠近香蕉的地方，而策略 B 看起来离两个物体都不近。 674 B 两个策略都完成了整个任务，但策略 B 第一次就做成了。第一次抓起并举起之后，感觉策略 A 过早地把鼠标放下了。它随后又把鼠标拿起来，移得更远。 660 B 策略 B 起初在长距离移动时有些犹豫，但后来转为有效而快速的动作。与此同时，策略 A 也完成了任务，但动作更迟缓。 649 A A 和 B 都把杯子拿了起来而不是推，然后都把它放在桌上。松手后，A 回到了起始姿态，而 B 一直反复去抓杯子，这并不理想。 648 B 虽然两者都把物体从碗里拿了出来（够得上满分），但 B 把物体放在了碗旁边的桌面上。这是更自然的做法。 641 A 策略 A 和策略 B 都几乎完全解决了任务。但策略 A 的动作更果断、纠正行为更少，而策略 B 是在多次尝试后碰巧完成的。 579 A 两个策略都把抽屉推到几乎关上。但策略 A 径直去推抽屉，直到关上才停。策略 B 中途被塑料食物玩具分散了注意力（尽管最终还是回去关上了抽屉）。 表 3：A、B 两个策略的部分成功反馈相同时的偏好反馈示例，说明了二值部分成功分数之外的信息。

会话 ID	更偏好的策略	详细反馈
748	A	虽然两个策略都没有把布放到螺丝刀上，但策略 A 把布放在了靠近香蕉的地方，而策略 B 看起来离两个物体都不近。
674	B	两个策略都完成了整个任务，但策略 B 第一次就做成了。第一次抓起并举起之后，感觉策略 A 过早地把鼠标放下了。它随后又把鼠标拿起来，移得更远。
660	B	策略 B 起初在长距离移动时有些犹豫，但后来转为有效而快速的动作。与此同时，策略 A 也完成了任务，但动作更迟缓。
649	A	A 和 B 都把杯子拿了起来而不是推，然后都把它放在桌上。松手后，A 回到了起始姿态，而 B 一直反复去抓杯子，这并不理想。
648	B	虽然两者都把物体从碗里拿了出来（够得上满分），但 B 把物体放在了碗旁边的桌面上。这是更自然的做法。
641	A	策略 A 和策略 B 都几乎完全解决了任务。但策略 A 的动作更果断、纠正行为更少，而策略 B 是在多次尝试后碰巧完成的。
579	A	两个策略都把抽屉推到几乎关上。但策略 A 径直去推抽屉，直到关上才停。策略 B 中途被塑料食物玩具分散了注意力（尽管最终还是回去关上了抽屉）。

We further found experimentally that for 11% of all A/B evaluations, the preference feedback did not agree with the partial success feedback, either because partial feedback was equal but the preference was not, or partial feedback and preference feedback displayed opposing trends. Thus, preference feedback is a uniquely rich data source for policy evaluations.

我们还通过实验发现，在全部 A/B 评测中有 11% 的偏好反馈与部分成功反馈不一致：要么部分成功反馈相同而偏好不同，要么部分成功反馈与偏好反馈呈现相反的趋势。因此，偏好反馈是策略评测中一种格外丰富的数据来源。

Figure 12: We observe that policies tend to succeed on tasks involving direct object manipulation, such as placing objects into containers (examples in green), but often fail in tasks that require tool use or nuanced semantic understanding, such as wiping, cleaning or following detailed multi-step instructions (examples in red).

图 12：我们观察到，策略往往能在涉及直接操作物体的任务上成功，例如把物体放进容器（绿色示例），但在需要使用工具或细致语义理解的任务上常常失败，例如擦拭、清洁或遵循详细的多步指令（红色示例）。

Appendix E Policy Performance Reports

附录 E 策略性能报告

\pi_{0}

\pi_{0}

Appendix F Strengths and Weaknesses of the Evaluated Generalist Policies

附录 F 被评测通用策略的优势与弱点

To our knowledge, the evaluation done in this work is the most comprehensive evaluation of generalist policies to date. Analyzing the head-to-head comparisons across a diverse set of real-world tasks, we identify consistent behavioral patterns and failure modes exhibited by current policies. In this section, we synthesize the general strengths and weaknesses observed across all evaluated policies and the trends that emerge across policy families.

据我们所知，本文所做的评测是迄今为止对通用策略最全面的评测。通过分析在一组多样的真实世界任务上的正面对决，我们识别出当前策略表现出的一致的行为模式和失败模式。本节综合所有被评测策略中观察到的总体优势和弱点，以及各策略家族之间呈现的趋势。

A primary strength of these generalist policies is their ability to operate in diverse viewpoints, lighting conditions, and background appearances. Across the evaluated policies, we observe that tasks involving direct object manipulation (e.g., pick-and-place, pushing, toppling, and simple open/close tasks) are more reliably solved than those requiring tool use, cloth manipulation or complex semantic understanding (Figure 12). In particular, these policies tend to perform better when goals are straightforward and visually grounded but struggle when the task demands precision alignment, multi-step reasoning or specific attribute perception like object class or color. Tasks involving deformable objects (e.g., folding, draping) and actions with tools (e.g., wiping, scooping) remain key challenges, often resulting in partial motion or outright frozen behavior. The weaknesses highlight the gaps in generalization, robustness, and physical interaction across various manipulation tasks.

这些通用策略的一个主要优势，是能在多样的视角、光照条件和背景外观下工作。在被评测的策略中，我们观察到，涉及直接操作物体的任务（例如抓取放置、推、推倒，以及简单的开/关任务）比需要使用工具、操作布料或复杂语义理解的任务更能可靠地完成（图 12）。尤其是，当目标直接且有视觉依据时，这些策略往往表现更好；但当任务要求精确对准、多步推理或对物体类别、颜色等特定属性的感知时，它们就会遇到困难。涉及可变形物体的任务（例如折叠、披盖）和使用工具的动作（例如擦拭、舀取）仍是主要挑战，常常导致动作只做到一部分，或者直接僵住不动。这些弱点凸显了在各类操作任务上，泛化、稳健性和物理交互方面的差距。

\pi_{0}

\pi_{0}

Policy Overview pi0_fast_droid is a fast, reactive manipulation policy that rarely idles and generally succeeds at single-object pick-and-place routines. When the goal object is visually salient it plans a direct path, grasps confidently, and often retries after a failed grasp. The policy copes reasonably well with mild clutter and can manipulate flexible objects such as towels or cloths. Limitations become evident whenever the task demands fine tool control, accurate alignment (e.g., insertions or stacking), deliberate inaction, or multi-step reasoning. In those settings the controller may oscillate, freeze, or grasp the wrong item, and it occasionally terminates without releasing the object it is holding.

策略概览 pi0_fast_droid 是一个快速、反应灵敏的操作策略，很少空转，在单物体抓取放置流程上大体能够成功。当目标物体在视觉上很显眼时，它会规划一条直接路径，抓取果断，抓取失败后常常会重试。该策略能较好地应对轻度杂乱，也能操作毛巾、布料等柔性物体。一旦任务要求精细的工具控制、精确对准（例如插入或堆叠）、有意的不动作或多步推理，其局限就会显现。在这些情况下，控制器可能来回摆动、僵住或抓错物体，偶尔还会在没有松开所持物体的情况下结束。

Comparative Performance (head-to-head) Pick and Place - Across dozens of episodes pi0_fast_droid beat or tied the competing policy more often than it lost. It routinely grasped the correct item and reached the target location, while rival policies either froze or mis-grasped (e.g., ducks into cups, cups into bowls, blocks into trays). Cover / Drape / Fold - The policy consistently outperformed its counterpart; it was usually the only agent to lift cloth or achieve partial folding, whereas competitors often merely poked at the fabric. Sorting / Classification - In repeated colour-based sorting tasks pi0_fast_droid identified target colours correctly and placed them, while the alternative policies hesitated or selected wrong colours. Move / Slide - When required to slide or reposition an object, the policy succeeded with smoother, faster trajectories; rival controllers tended to overshoot or stall. Tool Use - Performance lagged behind the comparison agent: pi0_fast_droid lost or tied in most erasing, wiping, stirring or “clean the table” episodes, whereas the other policies executed smoother tool contact and fewer redundant motions. Open / Close - The policy underperformed; it either failed to latch onto handles or to finish closing motions, while the competing agent completed the same drawer or cabinet tasks more reliably. Object Manipulation - When asked to re-orient blocks, pour or open bottles, the policy frequently lost or tied; the competitor could align objects more precisely or at least avoid freezing. Group / Organize / Stack - Stacking success was mixed; pi0_fast_droid often placed items near the correct spot but the rival policy achieved proper alignment more often, leading to several losses.

比较表现（正面对决） 抓取放置——在数十个回合中，pi0_fast_droid 胜过或战平对手策略的次数多于落败次数。它通常能抓住正确的物体并到达目标位置，而对手策略要么僵住，要么抓错（例如把鸭子放进杯子、把杯子放进碗、把积木放进托盘）。 覆盖 / 披盖 / 折叠——该策略始终胜过对手；它通常是唯一能把布提起来或完成部分折叠的智能体，而对手往往只是戳一戳布料。 分拣 / 分类——在反复进行的按颜色分拣任务中，pi0_fast_droid 能正确识别目标颜色并放置到位，而其他策略犹豫不决或选错颜色。 移动 / 滑动——在需要滑动或重新放置物体时，该策略以更平滑、更快速的轨迹完成任务；对手控制器则往往过冲或停滞。 使用工具——表现落后于对比智能体：在大多数擦除、擦拭、搅拌或“清理桌面”回合中，pi0_fast_droid 输掉或战平，而其他策略的工具接触更平稳、多余动作更少。 开 / 关——该策略表现欠佳；它要么无法扣住把手，要么无法完成关闭动作，而对手智能体能更可靠地完成同样的抽屉或柜门任务。 物体操作——在被要求调整积木朝向、倒水或打开瓶子时，该策略经常输掉或战平；对手能更精确地对准物体，或至少不会僵住。 归组 / 整理 / 堆叠——堆叠的成功情况参差不齐；pi0_fast_droid 常把物体放在正确位置附近，但对手策略更常实现正确对准，导致它输了好几次。

Strengths Robust grasping of familiar rigid objects; e.g., picked the cup and placed it into a basket smoothly, and removed a block from a box despite flaps in the way. Re-attempt behaviour: after missing the bowl the first time it re-planned, re-grasped, and completed the pineapple placement. Effective colour/shape recognition that supports sorting and non-red discrimination tasks. Cloth manipulation: successfully folded a towel and achieved full coverage over a piggy-bank while the opponent merely poked. Quick object search sweeps that cover shelves and table surfaces before freezing competitors, e.g., “find the creeper toy”.

优势 对熟悉的刚性物体抓取稳健；例如平稳地拿起杯子放进篮子，以及在盒盖挡住的情况下仍从盒子里取出积木。 重试行为：第一次没对准碗之后，它重新规划、重新抓取，完成了放菠萝的任务。 有效的颜色/形状识别，支撑分拣以及“非红色”辨别任务。 布料操作：成功折叠了一条毛巾，并把一个存钱罐完全盖住，而对手只是戳了戳。 快速的物体搜索扫视，能在对手僵住之前覆盖货架和桌面，例如“找到苦力怕玩具”。

Weaknesses Frequent freezing or limited exploration after one failed attempt, stalling entire episodes. Tool control deficiencies: could not keep the eraser in contact with the board or open a water bottle despite grasping the cap. Mis-identification of targets in clutter, e.g., placed tape in the wrong plate and grasped the robot instead of the marker meant to hit it. Difficulty with precision insertions (portafilter into grinder, small cubes onto stacks) leading to losses. Does not always release objects after placement, leaving grasped items hovering and tasks incomplete.

弱点 一次尝试失败后经常僵住或探索有限，使整个回合停滞。 工具控制不足：无法让板擦保持与白板接触；抓住了瓶盖却打不开水瓶。 在杂乱环境中认错目标，例如把胶带放进了错误的盘子，以及抓起了机器人，而不是用来敲击它的记号笔。 难以完成精确插入（把咖啡手柄装进磨豆机、把小方块放到堆上），因而落败。 放置后并不总会松开物体，使被抓物体悬在空中、任务无法完成。

Instruction Following Handles colour and spatial qualifiers well (“blue cup into yellow bowl” succeeded). Fails at “do absolutely nothing” - still moved despite the explicit negation. Interprets minor typos (“non-read” \to non-red) correctly and complied. Multi-step or relational instructions are followed inconsistently; it dropped the towel but never folded it in “place carrot then fold towel”. Occasionally ignores action verbs and grasps the wrong reference (picked the robot instead of using the marker to hit it).

指令遵循 能很好地处理颜色和空间限定语（“把蓝杯子放进黄碗”成功了）。 在“什么都不要做”上失败——尽管有明确的否定，仍然动了。 能正确理解小的拼写错误（“non-read”\to non-red）并照做。 对多步或关系型指令的遵循不稳定；在“放胡萝卜然后折毛巾”中，它把毛巾放下了，却从没折叠。 偶尔忽略动作动词，抓错指代对象（抓起了机器人，而不是用记号笔去敲它）。

Reasoning Scene reasoning strengths: correctly inferred colour grouping goals and located all cups quickly in cluttered scenes. Weaknesses: often violates order constraints (tried to stack before tray placement) or stops after partial completion (emptied only one item then froze). Spatial reasoning is sometimes coarse; the controller places objects “near” rather than “inside/on top”, leading to almost-correct states that still lose.

推理 场景推理的优势：能正确推断按颜色归组的目标，并在杂乱场景中迅速找到所有杯子。 弱点：经常违反顺序约束（在放托盘之前就试图堆叠），或在部分完成后就停下（只清空了一个物体就僵住了）。空间推理有时较粗糙；控制器把物体放在“附近”而不是“里面/上面”，导致几乎正确却仍然落败的状态。

Manipulation Skills Grasping: robust with medium-sized rigid objects (cups, blocks, towels). Placing: accurate onto large, open targets; less precise for narrow targets or stacking (frequent mis-alignment of tape rolls). Stacking/Inserting: partial success, but alignment errors common; blocks often dropped from height. Tool manipulation: weak torque control when erasing, wiping or turning caps; slips off tools or hovers without making contact. Recovery: will back-off and re-grasp after a miss instead of giving up. Release: occasionally forgets to open gripper after placement.

操作技能 抓取：对中等大小的刚性物体（杯子、积木、毛巾）稳健。 放置：放到大而开阔的目标上很准确；对狭窄目标或堆叠则不够精确（胶带卷经常没对准）。 堆叠/插入：部分成功，但对准误差常见；积木经常从高处掉落。 工具操作：擦除、擦拭或拧盖子时力矩控制弱；从工具上滑脱，或悬停而不接触。 恢复：没抓到后会后退并重新抓取，而不是放弃。 松开：放置后偶尔忘记张开夹爪。

Robustness to Scene Variations Handles moderate clutter and distractors, succeeding in busy kitchen and office scenes. Cloth backgrounds, patterned tables, and partial occlusions rarely confuse its perception. Sensitive to low-light episodes: performance degraded in dim “Place cup right side up” and “Put the yellow ducks in mug” scenes. Wrist-camera occlusions sometimes cause it to mis-localise small targets (e.g., screwdriver into mug task).

对场景变化的稳健性 能应对中等程度的杂乱和干扰物，在繁忙的厨房和办公室场景中取得成功。 布料背景、带图案的桌面和部分遮挡很少干扰其感知。 对低光照回合敏感：在昏暗的“把杯子正着放”和“把黄色小鸭子放进马克杯”场景中性能下降。 腕部相机被遮挡有时会导致它对小目标定位错误（例如把螺丝刀放进马克杯的任务）。

Common Failure Modes Freezing after first error or mid-air hover with object still grasped. Grasping the correct item but never releasing it into the goal. Selecting a distractor of similar colour/shape (tape vs. stapler, carrot vs. duck). Over-shooting and colliding with cabinets or shelving. Tool tasks: pushes or nudges the tool instead of forming a stable grasp, leading to repeated but ineffective motions. Misinterpreting passive, negated, or multi-step instructions (moved during “do not move”, folded towel step omitted, etc.).

常见失败模式 第一次出错后僵住，或抓着物体悬停在半空。 抓住了正确的物体，却从不把它放到目标处。 选中颜色/形状相似的干扰物（胶带与订书机、胡萝卜与鸭子）。 过冲并撞上柜子或货架。 工具任务：推动或轻碰工具而不是形成稳定抓取，导致反复而无效的动作。 误解被动、否定或多步指令（在“不要动”时动了，漏掉了折毛巾这一步，等等）。

Overall, pi0_fast_droid provides a solid baseline for everyday pick-and-place and cloth-handling tasks, but would benefit from improved tool manipulation control, tighter release logic, and more deliberate high-level planning for multi-step or precision-alignment scenarios.

总体而言，pi0_fast_droid 为日常的抓取放置和布料处理任务提供了一个扎实的基线，但若能改进工具操作控制、收紧松开逻辑，并在多步或精确对准场景中进行更审慎的高层规划，将会受益。

\pi_{0}

\pi_{0}

We are evaluating a policy named POLICY NAME deployed on a robot arm to perform various manipulation tasks. This policy was compared head-to-head against other policies across multiple episodes. Each episode includes: A session ID A task description and the task category it belongs to. The possible task categories are: Pick and Place, Open / Close, Move / Slide, Knock Over / Topple, Cover / Drape / Fold, Group / Organize / Stack, Find / Search, Minimal or No Action, Object Manipulation, Sorting / Classification, Tool Use. A scene and task analysis Head-to-head evaluation results Using the episode data provided, generate a structured and comprehensive summary report in the format below: Policy Overview A brief paragraph summarizing the general behavior, capabilities, and limitations of the policy. Comparative Performance How the policy performed in head-to-head comparisons against other policies across the different task categories. For each task category, create a bullet point with a discussion of how the policy consistently outperformed or underperformed compared to all the other policies. Make sure in this section that every claim about the policy is with respect to other competing policies. When making a claim, always mention how the other policies performed in comparison to the current policy. Do not discuss the policy in isolation. Don’t mention a task category unless there is evidence of the policy performing well or poorly in that category across multiple episodes. Make your claims based on overall performance or underperformance for specific task categories rather than individual episodes. There is no need to reference specific session IDs in this section (no <ref> tags).

我们正在评测一个名为 POLICY NAME 的策略，它部署在一台机械臂上，执行各种操作任务。 该策略在多个回合中与其他策略进行了正面对决。每个回合包括： 一个会话 ID 一段任务描述及其所属的任务类别。可能的任务类别有：抓取放置、开 / 关、移动 / 滑动、推倒 / 翻倒、覆盖 / 披盖 / 折叠、归组 / 整理 / 堆叠、寻找 / 搜索、极少或无动作、物体操作、分拣 / 分类、使用工具。 一段场景与任务分析 正面对决的评测结果 请根据所提供的回合数据，按以下格式生成一份 结构化且全面的总结报告 ： 策略概览 用一段简短的话总结该策略的总体行为、能力和局限。 比较表现 该策略在不同任务类别上与其他策略正面对决时的表现。为每个任务类别写一个要点，讨论该策略相对于所有其他策略是如何一贯地表现更好或更差的。确保本节中关于该策略的每一项论断都是相对于其他竞争策略而言的。提出论断时，始终说明其他策略与当前策略相比表现如何。不要孤立地讨论该策略。除非有证据表明该策略在某个任务类别的多个回合中表现好或差，否则不要提及该类别。依据该策略在特定任务类别上的总体表现优劣提出论断，而不是依据个别回合。本节无需引用具体的会话 ID（不要使用 <ref> 标签）。

Strengths Bullet-pointed list of notable strengths in manipulation behavior or general reliability. Mention the task categories the policy is good at (if any) instead of basing a claim on a single instance. Focus on generalizable behaviors like smooth trajectories, robust grasping, or adaptability. Use concrete examples and session ID citations. Weaknesses Bullet-pointed list of recurring limitations or error patterns. Mention the task categories the policy is poor at instead of basing a claim on a single instance. Mention issues such as fine motor control, object confusion, multi-step failure, etc. Include session ID references with <ref> tags. Instruction Following Analyze how well the policy understands and executes task instructions. Note sensitivity to language structure, ability to follow negated or relational commands, issues with ambiguous phrasing, ability to handle typos, etc. Cite session-specific evidence. Reasoning Evaluate the policy’s ability to reason about both the scene context (e.g., spatial relationships, object visibility) and the text instruction (e.g., goal inference, conditional logic). Mention cases where reasoning appears strong or deficient. Use <ref> tags to support your analysis. Manipulation Skills Describe the physical performance of the policy: grasping, placing, stacking, inserting, pouring, drawer use, and recovery from errors. Use examples to show when skills succeed or fail. Robustness to Scene Variations Assess the policy’s performance under different lighting, clutter levels, object positions, and camera views. Note any sensitivities to occlusion or distractors, etc. Common Failure Modes List frequently observed failures (e.g., freezing mid-task, grabbing wrong item, failing passive commands). Provide short descriptions and supporting citations.

优势 以要点列表列出在操作行为或总体可靠性方面值得注意的优势。说明该策略擅长的任务类别（如有），而不是根据单个实例提出论断。关注可推广的行为，例如平滑的轨迹、稳健的抓取或适应能力。使用具体例子并引用会话 ID。 弱点 以要点列表列出反复出现的局限或错误模式。说明该策略不擅长的任务类别，而不是根据单个实例提出论断。提及精细运动控制、物体混淆、多步失败等问题。用 <ref> 标签附上会话 ID 引用。 指令遵循 分析该策略对任务指令的理解和执行情况。注意其对语言结构的敏感性、遵循否定或关系型指令的能力、对含糊措辞的处理问题、处理拼写错误的能力等。引用具体会话的证据。 推理 评估该策略对 场景上下文 （例如空间关系、物体可见性）和 文本指令 （例如目标推断、条件逻辑）进行推理的能力。指出推理显得强或不足的情形。用 <ref> 标签支持你的分析。 操作技能 描述该策略的物理表现：抓取、放置、堆叠、插入、倾倒、使用抽屉以及从错误中恢复。用例子说明这些技能何时成功、何时失败。 对场景变化的稳健性 评估该策略在不同光照、杂乱程度、物体位置和相机视角下的表现。注意它对遮挡或干扰物等的敏感性。 常见失败模式 列出经常观察到的失败（例如任务中途僵住、抓错物体、无法执行被动指令）。给出简短描述和支持性的引用。

Instructions: When referring to a session, always cite the full session ID (UUID format, e.g., 16e5bbda-57c1-4e58-a24a-b39ee8142d41) exactly as provided. Do not shorten, truncate or modify it in any way. Always wrap session IDs inside <ref> ... </ref> tags. Example: <ref> 16e5bbda-57c1-4e58-a24a-b39ee8142d41 </ref> Try to cite as many session IDs as possible to support your claims, but only if they are relevant to the point you’re making. Avoid generalizing from a single episode unless there is clear evidence of a pattern. Keep the tone analytical and professional, emphasizing repeatable behaviors and insights. Do not invent session IDs. Only use session IDs present in the provided episode reports. There is no need to mention the specific number of episodes and wins/losses/ties in head-to-head evaluations in this report. The individual episode reports are as follows: ========== Episode Report #1 ========== \dots

说明： 提到某个会话时，始终完整引用会话 ID（UUID 格式，例如 16e5bbda-57c1-4e58-a24a-b39ee8142d41），与所给内容完全一致。不要缩短、截断或以任何方式修改。 始终把会话 ID 放在 <ref> ... </ref> 标签内。例如： <ref> 16e5bbda-57c1-4e58-a24a-b39ee8142d41 </ref> 尽可能多地引用会话 ID 来支持你的论断，但只限与所论述要点相关的会话。 除非有明确证据表明存在某种模式，否则不要从单个回合进行概括。 保持分析性和专业的语气，强调可重复的行为和洞见。 不要编造会话 ID。只使用所提供的回合报告中出现的会话 ID。 本报告中无需提及正面对决的具体回合数和胜/负/平次数。 各回合报告如下： ========== 回合报告 #1 ========== \dots

Figure 14: The prompt template used to generate the full policy performance reports.

图 14：用于生成完整策略性能报告的提示模板。 （译注：图 14、图 15 中的提示词原文为英文，此处的中文仅供理解。）

Given the following full evaluation report of a robot manipulation policy, generate a concise, high-quality summary that captures the main findings from sections 1 through 9. Each bullet should summarize the corresponding section in a few sentence fragments, focusing on the most important points. Avoid excessive detail, ensure clarity and correctness. Stick to the facts presented in the full report. Use the following format exactly: Policy Overview: <summary> Comparative Performance: <summary> Strengths: <summary> Weaknesses: <summary> Instruction Following: <summary> Reasoning: <summary> Manipulation Skills: <summary> Robustness to Scene Variations: <summary> Common Failure Modes: <summary> Place a line break between each bullet point. Don’t output anything before or after the bullet points. Here is the full report to summarize: FULL REPORT

下面是一份机器人操作策略的完整评测报告，请生成一份简洁、高质量的摘要，概括第 1 至第 9 节的主要发现。 每个要点用几个短句概括对应的一节，聚焦最重要的内容。避免过多细节，确保清晰、正确。严格依据完整报告中给出的事实。 严格使用以下格式： 策略概览： <summary> 比较表现： <summary> 优势： <summary> 弱点： <summary> 指令遵循： <summary> 推理： <summary> 操作技能： <summary> 对场景变化的稳健性： <summary> 常见失败模式： <summary> 每个要点之间空一行。不要在要点之前或之后输出任何内容。 以下是需要概括的完整报告： FULL REPORT

Figure 15: The prompt template used to generate a concise summary of the full policy performance report.

图 15：用于把完整策略性能报告概括为简短摘要的提示模板。

RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies

RoboArena：通用机器人策略的分布式真实世界评测

Abstract

摘要

1 Introduction

1 引言

2 Related Work

2 相关工作

Simulated and real-world robot evaluation.

仿真与真实世界中的机器人评测。

Generalist robot policies.

通用机器人策略。

Crowd-sourced benchmarks.

众包基准。

3 Decentralized Robot Policy Evaluation via Pairwise Comparison

3 通过成对比较进行去中心化的机器人策略评测

3.1 Policy Evaluation Protocol

3.1 策略评测协议

3.2 Computing Global Policy Rankings

3.2 计算全局策略排名

3.3 Extracting Qualitative Policy Characteristics

3.3 提取定性的策略特征

4 The DROID-RoboArena Evaluation System

4 DROID-RoboArena 评测系统

4.1 RoboArena System Design

4.1 RoboArena 系统设计

4.2 Open-Sourcing RoboArena: Interfaces, Safety and Incentives

4.2 开源 RoboArena：接口、安全与激励

5 Experiments

5 实验

5.1 Experimental Setup

5.1 实验设置

5.2 RoboArena Accurately Ranks Policy Performances

5.2 RoboArena 能准确地对策略性能排序

5.3 RoboArena Evaluation is Sample Efficient

5.3 RoboArena 评测具有样本效率

5.4 Extracting qualitative policy characteristics

5.4 提取定性的策略特征

6 Discussion

6 讨论

7 Limitations

7 局限

Acknowledgments

致谢

References

参考文献

Appendix A Contributions

附录 A 贡献

Appendix B Policy Ranking EM Procedure

附录 B 策略排名的 EM 流程

B.1 Model Definition

B.1 模型定义

Observed data.

观测数据。

Latent tasks and mixture weights.

潜在任务与混合权重。

Per‐policy and per‐bucket parameters.

逐策略与逐桶参数。

Tie‐model parameter.

平局模型参数。

Link function and conditional likelihood.

连接函数与条件似然。

B.2 Gradient & Hessian Derivation

B.2 梯度与 Hessian 的推导

Expected complete‐data objective.

完全数据的期望目标函数。

Gradient w.r.t. θp\theta_{p}θp​.

对 θp\theta_{p}θp​ 的梯度。

Hessian w.r.t. θp\theta_{p}θp​.

对 θp\theta_{p}θp​ 的 Hessian。

Gradients/Hessians for ψp,t\psi_{p,t}ψp,t​.

ψp,t\psi_{p,t}ψp,t​ 的梯度与 Hessian。

Gradients/Hessians for τt\tau_{t}τt​.

τt\tau_{t}τt​ 的梯度与 Hessian。

B.3 Including Partial‐Success Information

B.3 纳入部分成功信息

B.4 Baseline Ranking Algorithms

B.4 基线排名算法

Bradley–Terry MLE (offline).

Bradley–Terry 最大似然估计（离线）。

Gradient w.r.t. $\theta_{p}$ .

对 $\theta_{p}$ 的梯度。

Hessian w.r.t. $\theta_{p}$ .

对 $\theta_{p}$ 的 Hessian。

Gradients/Hessians for $\psi_{p,t}$ .

$\psi_{p,t}$ 的梯度与 Hessian。

Gradients/Hessians for $\tau_{t}$ .

$\tau_{t}$ 的梯度与 Hessian。