RoboJEPA: Scaling Robotic Latent World Models

RoboJEPA：缩放机器人潜空间世界模型

Artem Zholus¹·²·³·⁴∗, Nicolas Beltran-Velez¹, Jianhao Yuan¹, Sarath Chandar²·³·⁴, Tushar Nagarajan¹, Daniel Severo¹, Koustuv Sinha¹, Michal Drozdzal¹, Adriana Romero Soriano¹, Jeannette Bohg¹†, Nicolas Ballas¹†, Mahmoud Assran¹†

¹ FAIR at Meta  ² Chandar Research Lab  ³ Mila - Quebec AI Institute  ⁴ Polytechnique Montréal
∗ Work done at Meta  † Joint last author

Abstract

摘要

Latent world models have shown a remarkable ability to predict future states and to plan in the real world. In practice, however, we lack a principled way to estimate how their capabilities scale with model size, data, and compute, an open problem that slows progress in the field. In this work we present RoboJEPA, a world model based on the Joint Embedding Predictive Architecture (JEPA) and trained on a large-scale dataset spanning 12 robotic embodiments. We show that RoboJEPA's imagination error, the error of its latent rollouts, follows a second-order power law in compute, allowing us to predict model quality well beyond the scale at which the law is fit. We further show that downstream robotic planning performance improves predictably with compute, and that imagination error is strongly correlated with it, making it a reliable proxy for real-robot evaluation. Finally, we demonstrate that latent world models can be deployed zero-shot as robotic agents, planning toward a single goal image to solve tasks requiring long-horizon planning on real hardware. We release all model checkpoints together with our training and robot deployment code. To our knowledge, this is the first work to establish scaling laws for multi-embodiment robotic world models trained on real robot data, and RoboJEPA, at 8B parameters, is the largest JEPA predictor model trained to date.

潜空间世界模型已展现出预测未来状态、并在真实世界中做规划的能力。但实践中，我们仍缺少一套原则性办法，去估计这些能力如何随模型规模、数据与算力缩放——这一缺口拖慢了领域进展。本文提出 RoboJEPA：基于联合嵌入预测架构（Joint Embedding Predictive Architecture，JEPA）的世界模型，并在覆盖 12 种机器人本体的大规模数据上训练。我们表明，RoboJEPA 的想象误差（潜空间滚动误差）随算力服从二阶幂律，因而可在拟合尺度之外外推模型质量。我们进一步表明，下游机器人规划表现随算力可预期地提升，且想象误差与之强相关，可作为真机评测的可靠代理。最后，我们展示潜空间世界模型可零样本部署为机器人智能体：仅朝向一张目标图像规划，即可在真机上完成需要长地平规划的任务。我们公开全部模型检查点以及训练与真机部署代码。据我们所知，这是首个在真实机器人数据上、为多本体机器人世界模型建立缩放定律的工作；8B 参数的 RoboJEPA 也是迄今最大的 JEPA 预测器。

Date: October 7, 2026
Correspondence: Artem Zholus at artem.zholus@mila.quebec, Jeannette Bohg at jeannettebohg@meta.com
Code: https://github.com/facebookresearch/robo_jepa
Website: https://robojepa.github.io

Figure 1: RoboJEPA scales predictably from offline prediction to real-robot control. Left: the world model’s forward-prediction (ℓ 1 $\ell_{1}$) error, averaged over the DROID and RoboCasa held-out sets, follows a scaling law in training compute. Middle: downstream planning success in RoboCasa improves predictably with compute across tasks of increasing difficulty. Right: on the real Franka robot, the success rate improves with model size across Grasp, Object Lift, and Pick and Place. Actions are sampled from a uniform distribution and prioritized using the World Model during planning.

图 1：RoboJEPA 从离线预测到真机控制均可预期地缩放。左：世界模型前向预测（$\ell_{1}$）误差，在 DROID 与 RoboCasa 留出集上取平均，随训练算力服从缩放定律。中：RoboCasa 上的下游规划成功率随算力可预期提升，且随任务难度递增。右：真机 Franka 上，Grasp、Object Lift、Pick and Place 的成功率随模型规模上升。规划时动作从均匀分布采样，再由世界模型排序。

1 Introduction

1 引言

Figure 2: World-model imagination improves with scale. Decoded imagination rollouts on an AgiBot episode for RoboJEPA at $0.3$ B, $2$ B, and $8$ B parameters, compared against the ground-truth (GT) video (bottom row). Each row shows the same rollout sampled at equal-time steps; larger models track the true dynamics and preserve object and contact detail markedly better. All world models share the same encoder (V-JEPA 2.1-G) and decoder.

图 2：世界模型想象随规模改善。在 AgiBot 片段上，将 RoboJEPA 在 $0.3$B、$2$B、$8$B 参数下的解码想象滚动与真值（GT）视频（底行）对照。每行是同一滚动在等时间间隔上的采样；更大模型更贴合真实动力学，并明显更好地保留物体与接触细节。所有世界模型共用同一编码器（V-JEPA 2.1-G）与解码器。

Scaling has been one of the primary engines of recent progress in artificial intelligence. What makes scaling particularly actionable is predictability: empirical scaling laws relate a model’s loss to the amount of data, parameters, and compute used to train it, so that the return on an additional order of magnitude of compute can be forecast before it is spent (Kaplan et al., 2020; Hoffmann et al., 2022). This predictability turns scaling from a gamble into a science, it tells practitioners how large a model to train for a given budget, how much data to pair it with, and when returns will diminish. Scaling laws of this kind have been established across language models (Kaplan et al., 2020; Hoffmann et al., 2022), generative modeling of images, video, and other modalities (Henighan et al., 2020), or semantic classification of images (Zhai et al., 2022). Comparatively little is known about whether, and how, the same scaling predictability holds for models of the physical world.

缩放是近年人工智能进展的主要引擎之一。使其真正可操作的，是可预测性：经验缩放定律把模型损失与训练所用的数据量、参数量、算力量联系起来，从而在再多花一个数量级算力之前，就能预估回报（Kaplan 等，2020；Hoffmann 等，2022）。这种可预测性把缩放从赌运气变成一门可规划的科学：它告诉实践者在给定预算下该训多大模型、配多少数据、回报何时递减。同类定律已在语言模型（Kaplan 等，2020；Hoffmann 等，2022）、图像/视频等生成建模（Henighan 等，2020）以及图像语义分类（Zhai 等，2022）上建立。相比之下，同样的缩放可预测性是否、以及如何适用于物理世界的模型，所知仍很少。

We study this question for world models in robotics. A world model is a learned model of environment dynamics that an agent can roll out to imagine future states and plan (Ha and Schmidhuber, 2018; Hafner et al., 2023). We focus on latent world models that predict and plan in a latent representation space (Hafner et al., 2020; LeCun, 2022; Assran et al., 2025). Latent world models have shown the ability to plan toward goals specified at test time without the need of a task-specific policy (Assran et al., 2025). Previous latent world models are typically trained at a single scale, and how their quality changes with data, parameters, and compute has not been characterized (Assran et al., 2025; Hafner et al., 2025; Zhang et al., 2026). Other studies, focusing on scaling, either target generative video and interactive-environment models outside of robot control (Bruce et al., 2024; Google DeepMind, 2024; Hu et al., 2023), or scale policies rather than world models (Lin et al., 2025; Sartor and Thompson, 2024; Ai et al., 2025). In contrast, our goal is to establish scaling laws for latent world models. This matters particularly in robotics, where high-quality interaction data is scarce and computationally expensive to collect: knowing which capabilities a world model will have at a given training-compute budget lets practitioners decide whether to invest in a larger model, more demonstrations, or more compute, and reveals when a data budget has been saturated.

我们在机器人世界模型上研究这一问题。世界模型是学到的环境动力学模型，智能体可对其滚动以想象未来状态并做规划（Ha 与 Schmidhuber，2018；Hafner 等，2023）。我们聚焦在潜表示空间中预测与规划的潜空间世界模型（Hafner 等，2020；LeCun，2022；Assran 等，2025）。潜空间世界模型已能在测试时朝给定目标规划，而无需任务专属策略（Assran 等，2025）。既往潜空间世界模型多在单一规模训练，其质量如何随数据、参数与算力变化尚未被刻画（Assran 等，2025；Hafner 等，2025；Zhang 等，2026）。另一些聚焦缩放的研究，要么面向机器人控制之外的生成视频与交互环境模型（Bruce 等，2024；Google DeepMind，2024；Hu 等，2023），要么缩放的是策略而非世界模型（Lin 等，2025；Sartor 与 Thompson，2024；Ai 等，2025）。相比之下，我们的目标是为潜空间世界模型建立缩放定律。这在机器人领域尤其重要：高质量交互数据稀缺且采集昂贵；若能预知给定训练算力预算下世界模型会具备哪些能力，实践者便可决定是加大模型、增加演示，还是增加算力，并判断数据预算何时已饱和。

To this end, this work introduces RoboJEPA, a family of action-conditioned latent joint-embedding predictive world models. We keep the training recipe deliberately simple to focus on the scaling behavior. RoboJEPA is a feed-forward transformer that predicts the next latent state deterministically in a single forward pass. It is trained with a plain ℓ 1 $\ell_{1}$ regression loss in the frozen encoder’s latent space. RoboJEPA operates in the representation space of a frozen V-JEPA 2.1 encoder (Assran et al., 2025; Mur-Labadia et al., 2026) and is trained on a unified mixture of 23 public manipulation datasets spanning 12 embodiments and 26 different action spaces. We study model scaling of the world model, ranging parameter count from 22M up to 8B, while holding the representation encoder fixed, so our laws characterize how a plannable dynamics model scales on top of a shared representation. We build scaling laws estimating the compute-efficient frontier of the forward-prediction evaluation loss as a function of the training compute (Kaplan et al., 2020; Pearce and Song, 2024).

为此，本文提出 RoboJEPA：一族动作条件的潜空间联合嵌入预测世界模型。训练配方刻意保持简单，以便聚焦缩放行为。RoboJEPA 是一个前馈 Transformer，在单次前向中确定性预测下一潜状态；在冻结编码器的潜空间上以普通 $\ell_{1}$ 回归损失训练。它运行在冻结的 V-JEPA 2.1 编码器表示空间中（Assran 等，2025；Mur-Labadia 等，2026），并在涵盖 12 种本体、26 种动作空间的 23 个公开操作数据集的统一混合上训练。我们缩放的是世界模型本身：参数量从 22M 到 8B，同时固定表示编码器，从而使定律刻画的是在共享表示之上、可规划动力学模型如何缩放。我们拟合前向预测评估损失相对训练算力的算力高效前沿（Kaplan 等，2020；Pearce 与 Song，2024）。

Our central finding is that robotic world-model quality scales predictably with compute. First, the latent forward-prediction loss on out-of-distribution data follows a scaling law L ⁡ (C) = A ​ C α − γ ​ ln ⁡ C + E $L(C)=A\,C^{\alpha-\gamma\ln C}+E$, where $C$ is the training compute, $E$ is the irreducible loss, and A, α, γ $A,\alpha,\gamma$ are scaling-law parameters that we estimate alongside $E$. This holds on both real-robot holdout (DROID (Khazatsky et al., 2024)) data and simulated holdout (RoboCasa (Nasiriany et al., 2026)) data. Second, this improvement on the offline data transfers to downstream planning performance on a robot: on a suite of RoboCasa planning tasks, downstream success improves predictably with training compute, so that a world model’s latent forward-prediction loss is a strong predictor of its online planning performance. For example, only models trained with more than 10 22 $10^{22}$ FLOPs of training compute (at sizes 2B, 4B, and 8B) can solve object-interaction tasks in RoboCasa via non-greedy planning, while smaller models trained with less compute cannot (see Figure 15).

核心发现是：机器人世界模型质量随算力可预期地缩放。第一，分布外数据上的潜空间前向预测损失服从缩放定律 $L(C)=A\,C^{\alpha-\gamma\ln C}+E$，其中 $C$ 为训练算力，$E$ 为不可约损失，$A,\alpha,\gamma$ 与 $E$ 一并估计。这在真机留出（DROID，Khazatsky 等，2024）与仿真留出（RoboCasa，Nasiriany 等，2026）上均成立。第二，离线改进会迁移到机器人下游规划：在一组 RoboCasa 规划任务上，下游成功率随训练算力可预期提升，因而潜空间前向预测损失是在线规划表现的强预测因子。例如，只有训练算力超过 $10^{22}$ FLOPs 的模型（规模 2B、4B、8B）才能在 RoboCasa 上通过非贪心规划解决物体交互任务，而算力更小的模型做不到（见图 15）。

We deploy RoboJEPA on real and simulated robots, planning toward a single goal image that defines the final state of the environment. Across two robot platforms, we perform planning in over 50,000 $50{,}000$ evaluation episodes. We keep the planning hyperparameters fixed for each robot, varying only the planning horizon for harder tasks. We perform planning on a set of real world manipulation tasks that include grasping, lifting, and pick-and-place. In simulation, our planning spans tasks of increasing difficulty from reaching to object pushing. We observe an improvement trend in the success rate on the real robot as we increase the model size. Qualitatively, we also observe that the behavior of the planning policies changes as the model grows: for example, in object-manipulation tasks, only models with 2B parameters or more keep the gripper closed after grasping to complete the task, while smaller models often accidentally open it, dropping the object.

我们在真机与仿真机器人上部署 RoboJEPA，朝向一张定义环境终态的目标图像做规划。在两个机器人平台上，规划评估超过 $50{,}000$ 个回合。每个机器人的规划超参固定，仅在更难任务上加长规划地平线。真机操作任务包括抓取、举起与抓取放置；仿真则覆盖从到达到位姿到推物等难度递增的任务。我们观察到真机成功率随模型规模上升。定性上，规划策略行为也随规模变化：例如在物体操作任务中，只有 2B 及以上模型会在抓取后保持夹爪闭合以完成任务，更小模型常误开夹爪把物体丢掉。

Finally, to support open research and reproducibility, we release all model checkpoints together with the training and robot-deployment code. To the best of our knowledge, this is the first work to establish compute-optimal scaling laws for action-conditioned robotic world models trained on large-scale, multi-embodiment real robot data, and the first scaling law for JEPA with verified extrapolation. We also train the largest JEPA predictor to date, at 8B parameters.

最后，为支持开放研究与复现，我们公开全部模型检查点以及训练与真机部署代码。据我们所知，这是首个在大规模、多本体真实机器人数据上为动作条件机器人世界模型建立算力最优缩放定律的工作，也是首个带经验证外推的 JEPA 缩放定律。我们还训练了迄今最大的 JEPA 预测器，规模为 8B 参数。

2 Method and Data

2 方法与数据

To study the scaling behavior of robotic world models, we require a model family that scales with training compute. To this end we introduce RoboJEPA, a family of action-conditioned latent world models. The goal of RoboJEPA models is to predict how the latent features of a scene will evolve under the robot’s actions. Formally, at timestep $t$, given a history of past visual features z ≤ t $z_{\leq t}$ (where z t $z_{t}$ is the latent encoding of a given visual input x t $x_{t}$), proprioceptive states s ≤ t $s_{\leq t}$, and actions a ≤ t $a_{\leq t}$, our models are functions g θ $g_{\theta}$ that deterministically predict the features of the next frame in one neural network forward pass:

要研究机器人世界模型的缩放行为，需要一族能随训练算力缩放的模型。为此我们引入 RoboJEPA：动作条件的潜空间世界模型族。目标是预测场景潜特征在机器人动作作用下如何演化。形式化地，在时刻 $t$，给定历史视觉特征 $z_{\leq t}$（$z_{t}$ 是视觉输入 $x_{t}$ 的潜编码）、本体状态 $s_{\leq t}$ 与动作 $a_{\leq t}$，模型 $g_{\theta}$ 在一次网络前向中确定性预测下一帧特征：

Planning in RoboJEPA is specified by a goal image encoded once into goal features z g $z_{g}$, and at every timestep $t$ we search for the $H$ -step action sequence whose imagined latent rollout comes closest to the goal:

RoboJEPA 的规划由一张目标图像指定：先编码为目标特征 $z_{g}$；在每个时刻 $t$，搜索 $H$ 步动作序列，使其想象出的潜空间滚动最接近目标：

The rest of this section describes the design and training of this family of world models and how to use them for planning, with the goal of subsequently studying their scaling behavior.

本节余下部分介绍这族世界模型的设计、训练以及如何用于规划，以便随后研究其缩放行为。

2.1 RoboJEPA Architecture

2.1 RoboJEPA 架构

Figure 3: RoboJEPA training. The frozen V-JEPA 2.1 encoder $f$ (Mur-Labadia et al., 2026) maps the input multiview frames x t (1: V) $x_{t}^{(1:V)}$ to a grid of features. The predictor g θ $g_{\theta}$ consumes per-step sequences of multiview visual (z t (1: V) $z_{t}^{(1:V)}$), action (a t $a_{t}$) and proprioceptive state (s t $s_{t}$) tokens, conditioned on view and embodiment identifiers, and predicts the next-step features z ^ t + 1 $\hat{z}_{t+1}$. Training minimises two losses: (left) the teacher-forcing loss, the per-token ℓ 1 $\ell_{1}$ distance between predicted features and the next-step features from the encoder (Equation 5); and (right) the rollout loss, which attempts to predict the same target sequence as the teacher-forcing loss, but given fewer input frames. The model is given the full training sequence of actions but only a prefix of the encoded features, and predicts the remaining features autoregressively, with a KV-cache. This way the model learns to make accurate long term predictions.

图 3：RoboJEPA 训练。冻结的 V-JEPA 2.1 编码器 $f$（Mur-Labadia 等，2026）把多视角输入帧 $x_{t}^{(1:V)}$ 映射为特征网格。预测器 $g_{\theta}$ 消费逐步的多视角视觉（$z_{t}^{(1:V)}$）、动作（$a_{t}$）与本体状态（$s_{t}$）词元，并以视角与本体标识为条件，预测下一步特征 $\hat{z}_{t+1}$。训练最小化两项损失：（左）教师强制损失，即预测特征与编码器下一步特征的逐词元 $\ell_{1}$ 距离（式 5）；（右）滚动损失：目标序列与教师强制相同，但只给更少的输入帧。模型拿到完整动作序列与编码特征前缀，借助 KV 缓存自回归预测剩余特征，从而学会做准确的长期预测。

The family of RoboJEPA world models builds on the action-conditioned recipe of V-JEPA 2 (Assran et al., 2025) and V-JEPA 2.1 (Mur-Labadia et al., 2026). This subsection introduces the architecture of the RoboJEPA predictor. The full list of differences between RoboJEPA and V-JEPA 2-AC is summarised in Table 4 and detailed in Appendix A.

RoboJEPA 世界模型族建立在 V-JEPA 2（Assran 等，2025）与 V-JEPA 2.1（Mur-Labadia 等，2026）的动作条件配方之上。本小节介绍 RoboJEPA 预测器架构。与 V-JEPA 2-AC 的完整差异列于表 4，细节见附录 A。

Scaling world models across robot datasets is challenging because each dataset comes with a different number of camera views, proprioceptive state size, and controller type and frequency. Therefore, we aim to build a world model architecture that can work with any combination of views, controller type, and proprioceptive information. At each timestep $t$, a frozen V-JEPA 2.1-G encoder maps up to $V$ simultaneous camera views at resolution H × W $H\times W$, each spanning $T$ frames, denoted x t (1: V) = (x t (1), …, x t (V)) $x_{t}^{(1:V)}=(x_{t}^{(1)},\dots,x_{t}^{(V)})$, to the latent features z t (1: V) = (z t (1), …, z t (V)) $z_{t}^{(1:V)}=(z_{t}^{(1)},\dots,z_{t}^{(V)})$, where each z t (v) $z_{t}^{(v)}$ is an H p × W p $H_{p}\times W_{p}$ grid of D e $D_{e}$ -dimensional patch embeddings with D e = 1664 $D_{e}=1664$ and H p = H / 16 $H_{p}=H/16$, W p = W / 16 $W_{p}=W/16$. We use V-JEPA 2.1 as the encoder because early in the work we found that it leads to successful and predictable scaling of the world model, whereas V-JEPA 2 does not (we hypothesize why in Appendix K). Alongside these features, each timestep provides the action a t ∈ ℝ d a $a_{t}\in\mathbb{R}^{d_{a}}$ and proprioceptive state s t ∈ ℝ d s $s_{t}\in\mathbb{R}^{d_{s}}$, while each input is associated with a robot type operating in a particular action space, identified by the robot id e ∈ ℰ $e\in\mathcal{E}$, and a per-view camera type c (v) ∈ 𝒞 $c^{(v)}\in\mathcal{C}$. Here, ℰ $\mathcal{E}$ and 𝒞 $\mathcal{C}$ are finite sets of known robot ids and camera types. Next we describe how we tokenize inputs, and the world model architecture.

跨机器人数据集缩放世界模型很难：各数据集的相机数、本体状态维度、控制器类型与频率往往不同。因此我们希望架构能兼容任意视角数、控制器类型与本体信息组合。在时刻 $t$，冻结的 V-JEPA 2.1-G 编码器把分辨率 $H\times W$、各含 $T$ 帧、最多 $V$ 路同时视角$x_{t}^{(1:V)}=(x_{t}^{(1)},\dots,x_{t}^{(V)})$ 映射为潜特征 $z_{t}^{(1:V)}=(z_{t}^{(1)},\dots,z_{t}^{(V)})$，其中每个 $z_{t}^{(v)}$ 是 $H_{p}\times W_{p}$ 的 $D_{e}$ 维补丁嵌入网格，$D_{e}=1664$，$H_{p}=H/16$，$W_{p}=W/16$。选用 V-JEPA 2.1 是因为早期实验显示它能支撑世界模型成功且可预期的缩放，而 V-JEPA 2 不行（假说见附录 K）。此外每步还提供动作 $a_{t}\in\mathbb{R}^{d_{a}}$ 与本体状态 $s_{t}\in\mathbb{R}^{d_{s}}$；输入对应在特定动作空间中运行的机器人类型（机器人 id $e\in\mathcal{E}$）以及每路相机类型 $c^{(v)}\in\mathcal{C}$。$\mathcal{E}$、$\mathcal{C}$ 为已知机器人 id 与相机类型的有限集。接下来说明如何词元化输入以及世界模型架构。

Table 1: Compressed view of the $23$ -dataset, $26$ -action-space mixture. Video hours sum the durations of all recorded videos. Action hours measure the total duration of teleoperation or policy execution, counted once regardless of the number of cameras. The per-dataset breakdown with resolutions and source FPS is in Table 7; sampling weights are in Table 8; and semantic-camera mappings are in Table 9. We report dataset sizes that we were able to collect and verify, which may be smaller than those reported in the original papers.

表 1：23 个数据集、26 种动作空间混合的压缩视图。视频小时数累计所有录制视频时长；动作小时数累计遥操作或策略执行时长，不论相机数只计一次。分数据集分辨率与源 FPS 见附表 7；采样权重见表 8；语义相机映射见表 9。我们报告的是能够收集并核验到的规模，可能小于原论文所报。

Tokenization. We encode each modality separately and interleave the results into a single flat sequence. We refer to these encoded multimodal vectors as tokens. The motivation of our tokenization scheme is to let the model learn all temporal dependencies only from a structured sequence of tokens. To tokenize each z t (v) $z_{t}^{(v)}$, we project each patch from D e $D_{e}$ to the world-model width $D$ with a single linear layer, giving a visual token tensor of shape [ T, V, H p, W p, D ] $[T,V,H_{p},W_{p},D]$. Actions and proprioceptive states are mapped to $D$ by linear encoders (specific to robot id) applied to a t $a_{t}$ and s t $s_{t}$.

词元化。各模态分别编码，再交错成一条扁平序列；这些多模态向量称为词元。动机是让模型只从结构化词元序列学习全部时间依赖。对每个 $z_{t}^{(v)}$，用单层线性把每个补丁从 $D_{e}$ 投到世界模型宽度 $D$，得到形状 $[T,V,H_{p},W_{p},D]$ 的视觉词元张量。动作与本体状态由（按机器人 id 专用的）线性编码器映射到 $D$。

We interleave these tokens by timestep,

我们按时间步交错这些词元，

World Model Architecture. RoboJEPA performs the next embedding prediction via a deterministic model. RoboJEPA uses the vision transformer (Dosovitskiy et al., 2021) as the backbone and we parameterize the prediction such that each spatial input token of z t (1: V) $z_{t}^{(1:V)}$ predicts the corresponding spatial token of z t + 1 (1: V) $z_{t+1}^{(1:V)}$. We use the frame-level causal attention mask so that queries that correspond to timestep $t$ have only access to keys and values at step $t$ or earlier. The model stacks $L$ blocks that form a standard residual stream of ViT. The architecture of each block follows the standard recipe for modern variants of ViTs: we use pre-normalization, Rotary Position Encoding (Su et al., 2024), RMSNorm normalization (Zhang and Sennrich, 2019), GELU feed-forward block (Hendrycks and Gimpel, 2016), and QK-normalization (Dehghani et al., 2023). We found QK-normalization to be of particular importance for training stability when scaling the dataset size and the model size.

世界模型架构。RoboJEPA 用确定性模型做下一嵌入预测。骨干为视觉 Transformer（Dosovitskiy 等，2021），并参数化使得 $z_{t}^{(1:V)}$ 的每个空间输入词元预测 $z_{t+1}^{(1:V)}$ 的对应空间词元。使用帧级因果注意力掩码，使对应时刻 $t$ 的查询只能访问 $t$ 及更早的键与值。模型堆叠 $L$ 个构成标准 ViT 残差流的块；每块遵循现代 ViT 配方：预归一化、旋转位置编码（Su 等，2024）、RMSNorm（Zhang 与 Sennrich，2019）、GELU 前馈（Hendrycks 与 Gimpel，2016）以及 QK 归一化（Dehghani 等，2023）。我们发现，在放大数据集与模型时，QK 归一化对训练稳定性尤为重要。

Factorized RoPE and view conditioning. The spatio-temporal position information is supplied by a multi-axis Rotary Position Embedding (Su et al., 2024) that splits each head’s channels into four equal sub-blocks of width d h / 4 $d_{h}/4$, one for each of the four axes (t, i, j, v) $(t,i,j,v)$ for time, height, width, and view, where d h $d_{h}$ is the head dimension. Our version of RoPE rotates each independently by its own per-axis frequency (Section A.1). Action and state tokens have a temporal index but no spatial index, i.e. the 4D index of the action a t $a_{t}$ would be (t, 0, 0, 0) $(t,0,0,0)$. We combine this 4D RoPE with a per-layer learnable view bias applied after RoPE. The idea is that the 4D RoPE informs the model that tokens belong to different views, while the view bias informs the model which content each view holds (e.g. left camera, wrist camera; Section A.2). This allows us to randomize the order of views and at the same time provide the information which views the model works with.

因子化 RoPE 与视角条件。时空位置由多轴旋转位置嵌入（Su 等，2024）提供：把每个头的通道均分为四个宽为 $d_{h}/4$ 的子块，分别对应时间、高、宽、视角四轴 $(t,i,j,v)$，$d_{h}$ 为头维。我们的 RoPE 按各轴自有频率独立旋转（附录 A.1）。动作与状态词元只有时间索引、无空间索引，例如动作 $a_{t}$ 的 4D 索引为 $(t,0,0,0)$。4D RoPE 之后再加每层可学习的视角偏置：前者告诉模型词元属于不同视角，后者告诉模型各视角装的是什么内容（如左侧相机、腕部相机；附录 A.2）。这样可随机打乱视角顺序，同时仍告知模型在处理哪些视角。

Attention pattern. We use joint space-time-view attention and do not factorize it, meaning that each token can have access to potentially all other tokens in the space-time-view sequence, unless restricted by the attention mask. However, using no mask would allow past token activations access future information that they need to predict. For example, token activations of z t $z_{t}$ will have access to tokens of z t + 1 $z_{t+1}$ in self-attention. Therefore, we use a timestep-causal mask (Assran et al., 2025; Po et al., 2025; PAN Team et al., 2025). That is, a query associated with timestep $t$ has access to keys only at timestep $t$ or less. This includes attention queries and keys associated with different views, actions, and states. In addition to that, we make the attention local by letting queries only access keys that are at most $8$ timesteps in the past (Assran et al., 2025; Po et al., 2025; PAN Team et al., 2025). This design helps the model extrapolate to predicting longer sequences of embeddings because no query will ever attend to a history longer than the model was trained on, including during long autoregressive prediction. For implementation details on the attention masking, see Section A.3.

注意力模式。我们使用联合的空间–时间–视角注意力且不做因子化，即除非被掩码限制，每个词元原则上可访问序列中所有其他词元。但若无掩码，过去词元激活会看到它本应预测的未来信息——例如 $z_{t}$ 的激活在自注意力中可访问 $z_{t+1}$。因此采用时间步因果掩码（Assran 等，2025；Po 等，2025；PAN Team 等，2025）：与时刻 $t$ 相关的查询只能访问时刻 $\leq t$ 的键，涵盖不同视角、动作与状态。此外把注意力局部化：查询最多回看过去 $8$ 个时间步（Assran 等，2025；Po 等，2025；PAN Team 等，2025）。这有助于外推到更长嵌入序列——即便长自回归预测时，查询也不会看到超过训练所见的历史。掩码实现细节见附录 A.3。

Outputs. After feeding the entire sequence of tokens through the backbone, we project tokens that correspond to z (1: V) 1: T $z^{(1:V)}_{1:T}$ back to the encoder space. We map the output from $D$ dimensions to D e $D_{e}$ dimensions and apply layer normalisation. The idea is that since both input and output features are normalized, this strenghtens the learning signal and lowers the gap between the training and the autoregressive generation. We do not use any learnable modules to predict states and we simply compute the proprioceptive state s t $s_{t}$ either through integration, i.e. s t + 1 = s t + a t $s_{t+1}=s_{t}+a_{t}$, or s t = 0 $s_{t}=0$ (for example when working with commanded actions with non-trivial robot dynamics).

输出。整段词元过完骨干后，把对应 $z^{(1:V)}_{1:T}$ 的词元投回编码器空间：从 $D$ 维映到 $D_{e}$ 维并做层归一化。由于输入与输出特征都归一化，可加强学习信号并缩小训练与自回归生成之间的差距。我们不用可学习模块预测状态，而是通过积分 $s_{t+1}=s_{t}+a_{t}$ 计算本体状态，或在非平凡动力学的指令动作设定下取 $s_{t}=0$。

Training objective. We train the world model to predict the next embedding and, to minimize exposure bias, we additionally train it on its own outputs over potentially long horizons. Our objective therefore has two loss components. The first component trains the model to predict every next embedding along a sequence of encoded multiview frames. The second component uses the same objective, but the model is given only a prefix of fewer encoded frames (but the entire sequence of actions); it then predicts the missing embeddings autoregressively (Equations 1 and 2) and is trained on them with the same objective. We use a key-value cache to speed up this autoregressive prediction (Sections 2.2 and H). While the first loss component ensures that the model predicts the representations of a fixed encoder, the second makes the world model robust to its own errors and suppresses error accumulation. Both components use the same per-token ℓ 1 $\ell_{1}$ objective, and the total loss is their average:

训练目标。世界模型学习预测下一嵌入；为减轻暴露偏差，还在可能的长地平上对自身输出训练。因此目标含两项：第一项让模型沿编码多视角帧序列预测每一个下一嵌入；第二项目标相同，但只给更短的编码帧前缀（动作序列仍完整），再自回归预测缺失嵌入（式 1、2）并以同一目标训练。自回归用键值缓存加速（第 2.2 节与附录 H）。第一项保证模型对齐固定编码器表示；第二项使世界模型对自身误差鲁棒并抑制误差累积。两项都用同一逐词元 $\ell_{1}$ 目标，总损失取其平均：

2.2 Data, Training and Inference

2.2 数据、训练与推理

Data. Training a world model that generalizes across robot embodiments and scenes requires both scale and diversity. We assemble $23$ public manipulation datasets spanning $12$ robot platforms—single- and dual-arm robots and humanoids—across real and simulated data. The compiled dataset contains 15,022 $15{,}022$ hours of video, including 6,692 $6{,}692$ hours synchronized with actions; some demonstrations are recorded from multiple cameras. Some platforms record with only one camera (e.g. most of Open-X Embodiment data), whereas others use two or three cameras (e.g. DROID and RoboSet). In addition, the datasets were collected with different robot action spaces, camera poses, sensor resolutions, and frame rates. Therefore, to build a data-scalable world model, we unify our training datasets and formulate the architecture and training recipe to be general enough to work with any robotic dataset. Table 1 summarizes the mixture, and Appendix B provides further details about its action spaces, unification, and sampling.

数据。要让世界模型跨本体与场景泛化，需要规模与多样性。我们汇集 23 个公开操作数据集，覆盖 12 个机器人平台——单臂、双臂与人形，含真机与仿真。汇总后约有 $15{,}022$ 小时视频，其中 $6{,}692$ 小时与动作同步；部分演示有多路相机。有的平台只录一路相机（如多数 Open-X Embodiment），有的用两到三路（如 DROID、RoboSet）。各数据集的动作空间、相机位姿、传感器分辨率与帧率也不同。因此为得到可随数据缩放的世界模型，我们统一训练数据，并把架构与训练配方做成足以适配任意机器人数据集。表 1 概括该混合；附录 B 补充动作空间、统一方式与采样细节。

Figure 4: Two-view imagination scales in simulation. First $60$ s of a RoboCasa rollout (two agent-view cameras, stacked within each cell) imagined by RoboJEPA at $0.3$ B, $2$ B, and $8$ B parameters. Larger models keep the two views mutually consistent and preserve manipulation detail markedly deeper into the rollout. All world models share the same encoder (V-JEPA 2.1-G) and decoder.

图 4：仿真中双视角想象随规模改善。RoboCasa 滚动前 $60$ 秒（两路 agent 视角，格内上下叠放），由 $0.3$B、$2$B、$8$B 的 RoboJEPA 想象。更大模型在更长滚动中仍保持两视角一致，并更好保留操作细节。所有世界模型共用同一编码器（V-JEPA 2.1-G）与解码器。

Training. We train a series of RoboJEPA models that range in size from $22$ M to $8$ B parameters and in training compute from 2 × 10 19 $2{\times}10^{19}$ to 9.5 × 10 22 $9.5{\times}10^{22}$ FLOPs. For all models, we use AdamW (Loshchilov and Hutter, 2019) with a weight decay of $0.1$ and global gradient-norm clipping at $1.0$. We linearly warm up the learning rate, hold it constant at a peak of 5 × 10 − 5 $5\times 10^{-5}$ for most of training, and then linearly decay it to zero over a short cooldown phase at the end (Hu et al., 2024). This allows us to overlap the flat learning-rate stage across models trained with different amounts of compute: we run the flat stage only once and perform learning-rate annealing from intermediate checkpoints. For the detailed description of the training configuration and hyperparameters, please refer to Appendix C.

训练。我们训练一系列 RoboJEPA，参数从 $22$M 到 $8$B，训练算力从 $2{\times}10^{19}$ 到 $9.5{\times}10^{22}$ FLOPs。全部使用 AdamW（Loshchilov 与 Hutter，2019），权重衰减 $0.1$，全局梯度范数裁剪 $1.0$。学习率线性预热，在峰值 $5\times 10^{-5}$ 保持平坦一段，再在末端短冷却阶段线性衰减到零（Hu 等，2024）。这样可在不同算力预算的模型之间复用平坦学习率阶段：平坦阶段只跑一次，再从中间检查点做学习率退火。训练配置与超参见附录 C。

The training recipe of RoboJEPA is designed to facilitate scaling the model on as much data as is available, which requires training on many heterogeneous robotic datasets at once. A central issue with such heterogeneous data is that some datasets supply only a single view while others provide multiple views, often at different resolutions. We therefore train the model to work with a variable number of views and a variable resolution. To achieve this, we partition the GPUs into groups, where all GPUs within a group work with the same number of views and the same resolution. In practice this often leads to imbalanced step times across groups, which is why we balance these times by using different batch sizes for different groups. Our initial scaling-law experiments were done at a resolution of 240 × 320 $240\times 320$ with only two GPU groups: one working with single-view data and the other with dual-view data. However, eventually we found it beneficial to include high-resolution GPU groups, resulting in four groups: single-view low-resolution (240 × 320 $240\times 320$), dual-view low-resolution, single-view high-resolution (480 × 640 $480\times 640$), and dual-view high-resolution. We describe the GPU grouping and batch-size balancing in detail in Sections C.1 and C.2.

RoboJEPA 的训练配方旨在尽量吃下可得数据，因而需同时训练大量异构机器人数据集。核心难点是：有的数据集只有单视角，有的多视角，分辨率还常不一致。于是我们训练模型适应可变视角数与可变分辨率：把 GPU 分成若干组，组内视角数与分辨率相同。实践中各组步进时间往往不均，故用不同批量大小来平衡。最初缩放实验在 $240\times 320$ 分辨率、两组 GPU（单视角 / 双视角）上完成；后来发现加入高分辨率组更有利，变成四组：单视角低分辨率（$240\times 320$）、双视角低分辨率、单视角高分辨率（$480\times 640$）、双视角高分辨率。GPU 分组与批量平衡细节见附录 C.1、C.2。

Figure 5: DROID holdout (real Franka, unseen scenes).

图 5：DROID 留出集（真机 Franka，未见场景）。

Figure 6: RoboCasa target scenes (simulation).

图 6：RoboCasa 目标场景（仿真）。

Figure 7: World-model scaling laws. Out-of-distribution forward-prediction L 1 $L_{1}$ error versus training compute for RoboJEPA models from 22M to 8B parameters (color), evaluated on a real-robot DROID holdout and simulated RoboCasa scenes. The compute-optimal frontier (lower envelope) follows the second-order power law L ⁡ (C) = E + A ​ C α − γ ​ ln ⁡ C $L(C){=}E{+}A\,C^{\alpha-\gamma\ln C}$ (dashed).

图 7：世界模型缩放定律。RoboJEPA 从 22M 到 8B（按颜色）在真机 DROID 留出与仿真 RoboCasa 场景上的分布外前向预测 $L_{1}$ 误差对训练算力。算力最优前沿（下包络）服从二阶幂律 $L(C){=}E{+}A\,C^{\alpha-\gamma\ln C}$（虚线）。

Inference and Planning. RoboJEPA performs planning by predicting the future in the latent space and searching over the action sequence that brings that future to the desired goal. We employ receding-horizon CEM (Rubinstein, 1997; de Boer et al., 2005) as our planning algorithm with image goals; importantly, we use only a single image goal that represents the final state of the task, and we do not use subgoals. Notably, we run all planning with the same set of hyperparameters across embodiments and tasks. This process requires running thousands of forward passes of the world model, which is prohibitively expensive if the full history is recomputed every step. Therefore, we employ a rolling KV-cache that reuses the keys and values of past steps in self-attention. This way, at every step the model only processes a single frame, action, and proprioceptive vector, while the past information is provided by the cache (see Appendix H). Additionally, we employ PyTorch compilation to further speed up inference. However, in the typical planning scenario we need to run compiled inference during planning on the real robot, and since first-time compilation is time-consuming, running planning with compilation can become prohibitively slow in practice. This is why we build a simple inference engine that precompiles the inference computation and saves it to disk, and at planning time simply loads it back (see Section I.2). Finally, since CEM planning is heavily parallelizable, we connect the robot directly to the GPU cluster and run the planning on multiple GPUs. This way, we make the planning fast enough to be practical: while it is not real-time, it takes only several seconds to perform a multi-step planning even for an $8$ B model.

推理与规划。RoboJEPA 在潜空间预测未来，并搜索能把未来带到目标的动作序列。规划算法采用带图像目标的滚动地平 CEM（Rubinstein，1997；de Boer 等，2005）；重要的是只用一张表示任务终态的图像目标，不用子目标。跨本体与任务使用同一套规划超参。该过程需要对世界模型前向数千次；若每步重算全部历史，代价过高。因此采用滚动 KV 缓存复用过去步的键值：每步只处理单帧、动作与本体向量，历史由缓存提供（附录 H）。另用 PyTorch 编译加速推理。但真机规划时首次编译很慢，现场编译可能不切实际，于是我们做了简单推理引擎：预先编译并落盘，规划时直接加载（附录 I.2）。CEM 高度可并行，我们把机器人直连 GPU 集群、多卡跑规划，使多步规划即便对 $8$B 模型也只需数秒——虽非实时，但已实用。

2.3 Diffusion Decoder

2.3 扩散解码器

To qualitatively understand the capabilities of our world models, we train a diffusion decoder that produces pixel outputs from the predicted features; its goal is to visualize the forward predictions of the world model. The decoder is a diffusion model (Ho et al., 2020; Song et al., 2021) that denoises the continuous video tokens of the Cosmos tokenizer (NVIDIA, 2025) 1 1 1 Cosmos-CV4x8x8. spatially conditioned on the V-JEPA 2.1 features. The purpose of using a generative model here is that the decoder does not merely convert the V-JEPA features into Cosmos tokens, but also imagines unseen motion. During the training of the decoder, we compute V-JEPA features with a frame skip of $4$ frames, whereas the Cosmos tokens compress the same frames without any frame skip. As a result, the Cosmos tokens contain more information than the original V-JEPA feature frames, and this extra information — precisely, the $3$ RGB frames between consecutive frames encoded by the V-JEPA encoder — must be generated. With such a decoder, we can run forward prediction with the world model at, for example, $5$ Hz actions, producing latent features at the same rate, and then use the decoder to expand them into an RGB stream at $20$ FPS that covers exactly the same motion. Full architecture, training, and sampling details are in Appendix D.

为定性理解世界模型能力，我们训练一个扩散解码器，把预测特征译回像素，用于可视化前向预测。解码器是扩散模型（Ho 等，2020；Song 等，2021），对 Cosmos tokenizer（NVIDIA，2025；Cosmos-CV4x8x8）的连续视频词元去噪，并以 V-JEPA 2.1 特征做空间条件。采用生成模型的目的是：解码器不只是把 V-JEPA 特征转成 Cosmos 词元，还要想象未见运动。训练解码器时，V-JEPA 特征按隔 $4$ 帧计算，而 Cosmos 词元压缩同一批帧时不隔帧，因而 Cosmos 词元信息更密——多出的正是 V-JEPA 相邻编码帧之间的 $3$ 帧 RGB，必须由解码器生成。有了它，例如可在 $5$ Hz 动作上跑世界模型前向、同速率得到潜特征，再解码成覆盖同一运动的 $20$ FPS RGB 流。架构、训练与采样细节见附录 D。

3 Experiments

3 实验

3.1 World Model Scaling Laws

3.1 世界模型缩放定律

Our goal is to empirically find the concrete functional form of the scaling law that allows us to extrapolate predictions about the performance of the RoboJEPA model trained at larger scales. We consider four candidate functional forms. First, the standard power law L ⁡ (C) = E + A ​ C α $L(C)=E+A\,C^{\alpha}$ (Henighan et al., 2020), whose exponent α $\alpha$ is constant. Second, a generalization whose exponent itself changes linearly with compute, L ⁡ (C) = E + A ​ C α − γ ​ ln ⁡ C $L(C)=E+A\,C^{\alpha-\gamma\ln C}$; we refer to the former as a first-order and to the latter as a second-order power law 2 2 2 It is easy to show that, for the latter, the dependency between ln ⁡ (L − E) $\ln(L-E)$ and ln ⁡ C $\ln C$ is a quadratic function of ln ⁡ C $\ln C$.. Finally, the Broken (Caballero et al., 2022) and Unified (Caballero et al., 2026) Neural Scaling Laws. Here, $C$ is the total training compute (in FLOPs) and L ⁡ (C) $L(C)$ is the world model’s forward-prediction error, i.e. the per-token ℓ 1 $\ell_{1}$ distance between the predicted and the encoder features, averaged over a $9$ -step latent rollout on held-out data. Among these four forms, we find that RoboJEPA predictably follows the second-order power law.

目标是经验地找出能外推更大规模 RoboJEPA 表现的具体缩放定律函数形式。我们考虑四种候选。第一，标准幂律 $L(C)=E+A\,C^{\alpha}$（Henighan 等，2020），指数 $\alpha$ 为常数。第二，指数本身随算力线性变化的推广式 $L(C)=E+A\,C^{\alpha-\gamma\ln C}$；前者称一阶幂律，后者称二阶幂律（易证：对后者，$\ln(L-E)$ 与 $\ln C$ 的关系是 $\ln C$ 的二次函数）。最后是 Broken（Caballero 等，2022）与 Unified（Caballero 等，2026）神经缩放定律。这里 $C$ 为总训练算力（FLOPs），$L(C)$ 为世界模型前向预测误差，即预测特征与编码器特征的逐词元 $\ell_{1}$ 距离，在留出数据上对 $9$ 步潜空间滚动取平均。四种形式中，RoboJEPA 可预期地服从二阶幂律。

Setup. We evaluate each model trained at each compute budget by its forward-prediction L 1 $L_{1}$ error, L ⁡ (C) $L(C)$, on held-out data. Importantly, this is the same per-token L 1 $L_{1}$ objective the model is trained on and that the planner minimizes at deployment (Equation 5). We report L ⁡ (C) $L(C)$ on two embodiments, a DROID holdout dataset collected on a real robot and a simulated RoboCasa suite. Evaluation is always on scenes unseen during training, always uses two views: one side camera (left or right) and one wrist camera, and predicts nine future steps from an initial observation under the recorded action sequence, giving a ten-step sequence in total. Further evaluation details are provided in Appendix J. For each model at each training duration we compute the total training compute $C$ exactly, by counting the FLOPs of every operation in one training step and multiplying by the number of steps taken across all stages. We plot the prediction error for every model and training duration against its total training compute. At each compute budget, we retain the lowest error across model sizes. These best-achievable errors form the compute-optimal frontier as a function of the training compute. Finally, we fit each candidate curve to the frontier points of the smaller models (22M–2B) and select as our scaling law the one that best extrapolates to the held-out 4B and 8B models.

设置。对每个算力预算下训练的模型，用留出数据上的前向预测 $L_{1}$ 误差 $L(C)$ 评估。重要的是：这与训练目标、以及部署时规划器最小化的逐词元 $L_{1}$ 目标相同（式 5）。我们在两种本体上报告 $L(C)$：真机采集的 DROID 留出，以及仿真 RoboCasa 套件。评估始终在训练未见场景上，始终用两路视角（一侧视左/右 + 腕部），并从初始观测出发、在记录动作序列下预测未来九步，合共十步序列。更多评估细节见附录 J。对每个模型、每个训练时长，精确统计总训练算力 $C$：单步所有运算的 FLOPs 乘以各阶段步数。把各模型、各训练时长的预测误差对总算力作图；每个算力预算保留跨规模最低误差，这些点构成算力最优前沿。最后用较小模型（22M–2B）的前沿点拟合各候选曲线，并选对外留出的 4B、8B 外推最好的作为缩放定律。

Table 2: Variants of parametric curves as scaling-law candidates for the L 1 $L_{1}$ rollout error of RoboJEPA world models. Curve fits use models sized $22$ M– $2$ B parameters; we report held-out extrapolation error at 4B and 8B for DROID and RoboCasa. The BNSL and UNSL functional forms and their fitted parameter values are in Appendix J.

表 2：RoboJEPA 世界模型 $L_{1}$ 滚动误差的缩放定律候选参数曲线。拟合用 $22$M–$2$B 模型；报告在 DROID 与 RoboCasa 上对 4B、8B 的留出外推误差。BNSL、UNSL 形式与拟合参数见附录 J。

Results. Table 2 compares the four candidate curves by how well a fit on the smaller models (22M–2B) extrapolates to the held-out 4B and 8B models. The second-order power law extrapolates best, with a held-out error of $0.6$ and 1.4 × 10 − 3 $1.4\times 10^{-3}$ on DROID and RoboCasa respectively, roughly $2$ – 3 × $3\times$ lower than the standard power law ($2.0$ and 3.7 × 10 − 3 $3.7\times 10^{-3}$) and below both BNSL and UNSL. We therefore adopt it as our scaling law. Figure 7 plots the forward-prediction error against training compute for all models. The lower envelope across model sizes forms the compute-optimal frontier, and no model size lies on the frontier entirely. In addition, Figure 7 plots the scaling law computed on all models as the dashed line. Interestingly, the two best candidates, the second-order law and BNSL, estimate nearly the same irreducible error, ≈ 0.2 $\approx 0.2$ on DROID and ≈ 0.17 $\approx 0.17$ on RoboCasa, indicating that RoboJEPA is close to saturation on both evaluations under its current training and data budget.

结果。表 2 比较四种候选：在小模型（22M–2B）上拟合后，对留出 4B、8B 的外推优劣。二阶幂律外推最好，DROID / RoboCasa 留出误差分别为 $0.6$ 与 $1.4\times 10^{-3}$，大约比标准幂律（$2.0$ 与 $3.7\times 10^{-3}$）低 $2$–$3\times$，也低于 BNSL 与 UNSL。因此采用它作为缩放定律。图 7 画出全部模型的前向预测误差对训练算力；跨规模下包络即算力最优前沿，没有任何单一规模全程落在前沿上。图 7 虚线还给出用全部模型算出的缩放定律。有趣的是，两个最佳候选——二阶定律与 BNSL——估计的不可约误差几乎相同：DROID 约 $0.2$，RoboCasa 约 $0.17$，说明在当前训练与数据预算下 RoboJEPA 在两项评估上已接近饱和。

3.2 Downstream Planning Scaling

3.2 下游规划缩放

Figure 8: The normalized return for the pose reach task in RoboCasa environment evaluated in unseen kitchens.

图 8：RoboCasa 未见厨房中位姿到达任务的归一化回报。

Figure 9: The normalized return for the task of pose reach while holding an object.

图 9：持物位姿到达任务的归一化回报。

Figure 10: Downstream planning success scales with training compute. Median success rate on the reach task versus total training compute, following a power-law fit.

图 10：下游规划成功随训练算力缩放。到达任务中位成功率对总算力，并给出幂律拟合。

Having identified an improvement trend in offline performance of world models, we hypothesize that this improvement translates to the world model’s physical understanding of the new environment in which it operates. For example, is the action conditioning accurately mapped to the correct movements of the robot in the representation space? Are the grasping and object interaction actions correctly represented by the world model in response to incoming action? We test this hypothesis via online planning with image goals. Section 3.1 introduced the predictable improvement in offline forward-prediction, and we verify that this improvement translates into a better success rate.

在确认离线表现有改善趋势后，我们假设这一改善会转化为世界模型对新环境的物理理解——例如动作条件是否准确映射到表示空间中的正确运动？抓取与物体交互是否被正确表示？我们用带图像目标的在线规划检验该假设。第 3.1 节给出了离线前向预测的可预期改善，这里验证它是否带来更高成功率。

To test this hypothesis, we build a set of interactive environments on top of RoboCasa 365 (Nasiriany et al., 2026), each isolating one basic capability, and test whether more training compute improves it. In this section we focus on greedy tasks, which can be solved by one-step planning where simply moving toward the goal at every step is enough. (i) 3D understanding (pose reaching): reach a target end-effector pose in as few steps as possible. (ii) Object understanding (pose reaching while holding an object): reach a target pose while holding a grasped object, so the planner must additionally search over gripper actions to avoid dropping it. Harder tasks that require non-greedy, longer-horizon planning — such as moving around an obstacle or pushing an object to a target — are studied in Section 3.3. Full environment and reward details are in Section G.4.

为此，我们在 RoboCasa 365（Nasiriany 等，2026）上构建一组交互环境，各隔离一项基本能力，检验更多训练算力是否提升它。本节聚焦贪心任务：一步规划、每步朝目标移动即可解决。（i）三维理解（位姿到达）：尽快到达目标末端位姿。（ii）物体理解（持物位姿到达）：持已抓物体到达目标位姿，规划器还需搜索夹爪动作以免掉落。需要非贪心、更长地平的更难任务——绕障或推物到目标——见第 3.3 节。环境与奖励细节见附录 G.4。

We use the same range of RoboJEPA checkpoints as in Section 3.1 and run each of them in our RoboCasa environments with closed-loop planning (Mayne et al., 2000), where the model re-plans toward the goal image after every executed action (full protocol in Section G.4). Figure 10 shows the evaluation results, measured by the simulator’s normalized cumulative reward on the performed trajectory (Section G.4). Just like with the offline performance, we see a clear improvement trend of the downstream performance as we scale the training compute. For both tasks, models form a clear performance frontier, with larger models surpassing smaller ones as the training compute grows. For example, the simplest end-effector 3D movement capabilities emerge after around 10 20 $10^{20}$ FLOPs while the basic capability of moving while holding an object emerges at around 3 × 10 20 $3\times 10^{20}$ FLOPs of training, both of which correspond to training $50$ M– $100$ M-parameter models on the full multi-embodiment dataset. Due to the simplicity of these tasks, performance saturates early. Therefore, for building robotic world models, we recommend first building downstream evaluations that reflect the desired capabilities, then estimating the minimal training compute required to achieve them.

我们使用与第 3.1 节相同范围的 RoboJEPA 检查点，在 RoboCasa 环境中做闭环规划（Mayne 等，2000）：每执行一个动作后重新朝目标图像规划（完整协议见附录 G.4）。图 10 给出评估结果，指标为仿真器对轨迹的归一化累计奖励（附录 G.4）。与离线表现一样，下游表现随训练算力清晰改善；两任务上都形成明显性能前沿，算力增大时大模型超越小模型。例如，最简单的末端三维运动能力大约在 $10^{20}$ FLOPs 后出现；持物移动大约在 $3\times 10^{20}$ FLOPs——均对应在完整多本体数据上训练 $50$M–$100$M 参数模型。因任务简单，表现较早饱和。因此，构建机器人世界模型时，我们建议先建立反映目标能力的下游评估，再估计达到这些能力所需的最小训练算力。

3.3 Better Scaling for Long Horizon Planning

3.3 长地平规划的更好缩放

With the established transfer from offline performance to online greedy performance in Sections 3.1 and 3.2, we now ask whether the same offline-to-online connection holds for non-greedy, longer-horizon tasks that cannot be solved in one step. We design two such tasks in RoboCasa. The first is an obstacle-reach task: the goal is again to reach a target end-effector position, but we place a large object (a kitchen appliance) as an obstacle so that a direct straight-line motion cannot reach the goal. The second is an object-push task, where an object is placed on the table in front of the robot and the goal is to move it toward a target position, which requires the arm to descend to the object, push it, and ascend back. Full task definitions are in Section G.4.

在第 3.1–3.2 节建立离线到在线贪心表现的迁移后，我们问：同一离线–在线联系是否也适用于无法一步解决的非贪心、更长地平任务。我们在 RoboCasa 设计两任务。其一为绕障到达：目标仍是到达末端位置，但放置大型障碍物（厨电），使直线运动无法到达。其二为推物：桌前放置物体，目标是将其推向目标位置，需要手臂下降、推动再抬起。完整定义见附录 G.4。

We first deploy the models trained with the recipe of Section 3.1 on the obstacle-reach task. Figure 11 shows that the final reward correlates with training compute, but the correlation is weak and no clean compute

我们先把第 3.1 节配方训练的模型部署到绕障到达。图 11 显示最终奖励与训练算力相关，但相关偏弱，且没有干净的算力

Figure 11: Short-horizon prediction training on obstacle-reach. Final reward versus training compute. The reward correlates with compute, but no clean frontier emerges and smaller models often beat larger ones.

图 11：短地平预测训练在绕障到达上的表现。最终奖励对训练算力。奖励与算力相关，但未形成干净前沿，小模型常胜过大模型。

frontier emerges: smaller models often beat larger ones (for example, the $300$ M planner reaches a higher final reward than the $1$ B one). Unlike the greedy tasks of Section 3.2, predictable scaling does not emerge for longer-horizon tasks, which we attribute to a mismatch between training and deployment. Planning on these tasks relies on accurate long-horizon prediction, yet the models were trained with only a short autoregressive rollout (K = 2 $K=2$ steps). We therefore align training with how the model is used at test time and increase the training rollout to K = 10 $K=10$ steps during the cooldown, together with higher-resolution inputs. We refer to the two cooldown recipes as short-horizon prediction training (K = 2 $K=2$, 240 × 320 $240\times 320$) and long-horizon prediction training (K = 10 $K=10$, a mix of 240 × 320 $240\times 320$ and 480 × 640 $480\times 640$). Retraining every model from scratch this way would be prohibitively computationally expensive, so we run the long-horizon cooldown only from the flat-learning-rate pretraining checkpoints (Section 2.2): the long flat-learning-rate stage only needs to learn the coarse dynamics cheaply, and the computationally expensive long-rollout, high-resolution signal is spent where it matters, in the final annealing phase.

前沿：小模型常胜过大模型（例如 $300$M 规划器最终奖励高于 $1$B）。与第 3.2 节贪心任务不同，长地平任务未出现可预期缩放，我们归因于训练与部署错配：这些任务依赖准确的长地平预测，而模型训练时自回归滚动只有 $K=2$ 步。于是我们让训练对齐测试用法：在冷却阶段把训练滚动加到 $K=10$ 步，并提高输入分辨率。两种冷却配方分别称为短地平预测训练（$K=2$，$240\times 320$）与长地平预测训练（$K=10$，混合 $240\times 320$ 与 $480\times 640$）。从头重训全部模型代价过高，因此长地平冷却只从平坦学习率预训练检查点接着跑（第 2.2 节）：漫长平坦阶段只需廉价学到粗动力学，昂贵的长滚动、高分辨率信号花在最终退火阶段。

Figure 12: Imagination error scaling for short- vs. long-horizon prediction training.

图 12：短地平与长地平预测训练的想象误差缩放。

Figure 13: Obstacle-reach final reward, long-horizon prediction training.

图 13：长地平预测训练下的绕障到达最终奖励。

Figure 14: Longer-horizon prediction training improves scaling and downstream long-horizon planning. (a) Second-order scaling laws of the imagination error on DROID and RoboCasa for the two cooldown recipes. (b) Obstacle-reach final reward versus total training compute for the long-horizon (K = 10 $K=10$) models.

图 14：更长地平预测训练改善缩放与下游长地平规划。(a) 两种冷却配方在 DROID 与 RoboCasa 上的想象误差二阶缩放定律。(b) 长地平（$K=10$）模型的绕障到达最终奖励对总算力。

Figure 14 compares the two recipes by their scaling laws on DROID and RoboCasa. Long-horizon prediction training is more computationally expensive per step, but it becomes more compute-efficient beyond about 10 21 $10^{21}$ FLOPs of total training compute (around $300$ M– $1$ B parameters at the compute-optimal frontier) and reaches a substantially lower forward-prediction error. It also lowers the irreducible error of the fit, indicating a genuine reduction in error accumulation.

图 14 用 DROID 与 RoboCasa 上的缩放定律比较两种配方。长地平预测训练每步更贵，但在总算力约超过 $10^{21}$ FLOPs 后更算力高效（算力最优前沿大约对应 $300$M–$1$B 参数），并达到明显更低的前向预测误差；拟合的不可约误差也更低，表明误差累积被实质压低。

Figure 15: Downstream success rate on the RoboCasa push task versus training compute, for the long-horizon (K = 10 $K=10$) models.

图 15：长地平（$K=10$）模型在 RoboCasa 推物任务上的下游成功率对训练算力。

Deploying the non-greedy planning models on the obstacle-reach task (Figure 14) significantly improves the trend: the final reward now strongly follows the predictable scaling pattern. Small models (below $300$ M parameters) cap at around $0.3$, which corresponds to the end-effector remaining stuck on the wrong side of the obstacle. The obstacle-avoidance capability then takes off around 10 21 $10^{21}$ FLOPs and saturates around 10 22 $10^{22}$ FLOPs. This is much later than the end-effector capabilities of Section 3.2, which emerged around 10 20 $10^{20}$ – 3 × 10 20 $3\times 10^{20}$ FLOPs, indicating that modeling interaction with the static scene geometry requires considerably more training.

把非贪心规划模型部署到绕障到达（图 14）后，趋势显著改善：最终奖励强力跟随可预期缩放模式。小模型（低于 $300$M）封顶约在 $0.3$，对应末端卡在障碍物错误一侧。绕障能力约在 $10^{21}$ FLOPs 起跳，约在 $10^{22}$ FLOPs 饱和——远晚于第 3.2 节约 $10^{20}$–$3\times 10^{20}$ FLOPs 出现的末端能力，说明建模与静态场景几何的交互需要多得多的训练。

Finally, the object-push task assesses the object interaction understanding (through non-greedy planning) by the world model: because the robot’s start and goal poses are identical (so the side and wrist views change only through the object’s position), the model must understand that pushing the object is what changes the scene. Figure 15 shows the success rate, defined simply as whether the object moved toward its target position (Section G.4). Interestingly, no model achieves a positive success rate until around 10 22 $10^{22}$ FLOPs, after which all models begin to succeed, suggesting that fine-grained understanding of surrounding objects emerges after certain training computation regardless of the model size. In summary, these results align in a natural order in which capabilities appear with more compute: first the 3D control of the end-effector, then manipulating a grasped object, then the geometry of the static scene, and finally the dynamics of objects in the scene.

最后，推物任务通过非贪心规划检验世界模型的物体交互理解：机器人起止位姿相同（侧视与腕视只因物体位置变化），模型必须理解「推动物体」才是改变场景的原因。图 15 给出成功率，定义为物体是否朝目标位置移动（附录 G.4）。有趣的是，直到约 $10^{22}$ FLOPs 之前没有任何模型取得正成功率，之后各规模都开始成功，暗示对周围物体的细粒度理解在达到一定训练算力后出现，而与模型规模关系较弱。综上，能力随算力出现的自然顺序是：末端三维控制 → 持物操作 → 静态场景几何 → 场景中物体动力学。

3.4 Real World Robotic Capabilities

3.4 真机机器人能力

Figure 16: Real-world deployment on DROID. Success rates across model sizes for three manipulation tasks; the dashed line shows their average.

图 16：DROID 真机部署。三任务成功率随模型规模变化；虚线为平均。

Our ultimate goal is to showcase the capabilities of RoboJEPA in real-world settings. As a step towards this goal, we deploy RoboJEPA models ranging from 22M to 8B parameters on the DROID (Khazatsky et al., 2024) platform with a single-arm Franka robot and two cameras: an external camera mounted to the left of the arm and a wrist camera using the standard DROID mount. To deploy RoboJEPA via planning, we record an image goal on the robot by executing a predefined robot motion and manually placing the object in the grasping position so the robot can interact with it. We provide only a single final image goal, requiring longer-horizon, non-greedy planning. We evaluate three tasks: Grasp, Object Lift, and Pick and Place. We use the same planning hyperparameters across tasks and model sizes, evaluating the highest-compute checkpoint at each size. Setup, task definitions, episode counts, scoring criteria, and planning hyperparameters are detailed in Sections G.1 to G.3.

最终目标是展示 RoboJEPA 在真实世界中的能力。作为一步，我们在 DROID（Khazatsky 等，2024）平台上部署 22M 到 8B 的模型：单臂 Franka，外加左侧外置相机与标准 DROID 腕部相机。通过规划部署时，先执行预设运动并手动把物体放到可抓位置，从而在机器人上录制目标图像。只给一张最终目标图，因而需要更长地平的非贪心规划。评估三任务：Grasp、Object Lift、Pick and Place。跨任务与规模使用同一规划超参，并取各规模最高算力检查点。设置、任务定义、回合数、评分与超参见附录 G.1–G.3。

To contextualize the RoboJEPA results, we also evaluate the Vision-Language-Action (VLA) models π 0 $\pi_{0}$ -FAST (Pertsch et al., 2025) and π 0.5 $\pi_{0.5}$ (Black et al., 2025). However, their deployment differs in goal specification (text instructions versus a single goal image) and training (DROID fine-tuning for the VLA baselines versus pretraining on a mixture that includes DROID for RoboJEPA). Neither RoboJEPA nor the VLA baselines receive task-specific fine-tuning. These differences make the VLA results contextual references rather than directly comparable baselines. Figure 16 shows the scaling trend. For each task, all RoboJEPA models, including very small models, demonstrate non-zero success rates. Overall, success rates show a clear trend of improving with model size and generally decreasing with task complexity. This picture does not hold for the reference models, which work best on Pick and Place, the hardest task, but degrade significantly on the others. For example, π 0.5 $\pi_{0.5}$ achieves a 53 % $53\%$ success rate on Pick and Place but 0 % $0\%$ on Object Lift. We hypothesize that this difference may be an artifact of training, as verbs such as “lift” are heavily underrepresented in the DROID dataset, which focuses mostly on pick-and-place tasks. Due to the planning nature of the agent and the task specification as an image goal, RoboJEPA models are not susceptible to this type of degradation. Qualitatively, behavior transitioned gradually from smooth but less adaptive execution in smaller models to more reliable recovery from mistakes in larger models (Section G.7). Table 3 reports task progress and success rates for the largest RoboJEPA models and VLA references.

为给 RoboJEPA 结果提供语境，我们也评估视觉–语言–动作（VLA）模型 $\pi_{0}$-FAST（Pertsch 等，2025）与 $\pi_{0.5}$（Black 等，2025）。但二者在目标形式（文本指令 vs 单张目标图）与训练（VLA 基线做 DROID 微调 vs RoboJEPA 在含 DROID 的混合上预训练）上不同。双方都未做任务专属微调。这些差异使 VLA 结果只宜作语境参考，而非直接可比基线。图 16 显示缩放趋势：各任务上所有 RoboJEPA——包括很小的模型——都有非零成功率；总体随规模上升、随任务难度下降。参考模型则不然：它们在最难的 Pick and Place 最好，其他任务显著变差。例如 $\pi_{0.5}$ 在 Pick and Place 上 53%，在 Object Lift 上 0%。我们猜测这可能是训练伪影：DROID 以抓取放置为主，「lift」等动词严重不足。RoboJEPA 以规划 + 图像目标指定任务，不易受此类退化。定性上，行为从较小模型较平滑但欠自适应，逐步过渡到较大模型更能从失误中恢复（附录 G.7）。表 3 报告最大 RoboJEPA 与 VLA 参考的任务进度与成功率。

Table 3: Real-robot task progress and success rates (%). Full results are in Table 12.

表 3：真机任务进度与成功率（%）。完整结果见表 12。

3.5 Qualitative Analysis

3.5 定性分析

So far we have quantified the world model through its prediction loss and downstream planning success; we now inspect how the prediction improves qualitatively by decoding its latent rollouts back to pixels. To do that, we use the diffusion decoder introduced in Section 2.3. The decoder serves purely as a visualization tool and is trained only on the frozen V-JEPA 2.1-G encoder features (see Appendix D). Note that all visualizations use the same encoder and decoder and vary only the RoboJEPA world model size. We perform forward prediction given a single frame (or a pair of frames) and a sequence of robot actions, and each sequence of predicted embeddings is decoded back to pixels. We run the prediction across a range of settings — low- and high-resolution, single- and dual-view — and report them separately (Figures 2 and 4; full set in Appendix E); the decoder itself is trained at a single 240 × 320 $240{\times}320$ resolution. These visualizations show that the forward-prediction improves with model scale in several concrete ways: larger models track the true dynamics more accurately over the rollout, preserve object identity and fine manipulation detail, and keep multiple camera views mutually consistent. Several behaviors are effectively emergent with scale — gripper–object contact and structured deformation such as folding appear only once the predictor is large enough, whereas smaller models blur these interactions away. Finally, we finetune the $8$ B version of RoboJEPA on DROID to imagine all three camera views simultaneously at high ($720$ p) resolution (Section C.4; Figure 17). These rollouts demonstrate that RoboJEPA performs accurate object manipulation even at very high resolution, and they exhibit strong cross-view information transfer, with content from the side (exterior) cameras propagating into the wrist-camera imagination.

至此我们用预测损失与下游规划成功量化了世界模型；现在把潜空间滚动解码回像素，定性观察预测如何随规模改善。使用第 2.3 节的扩散解码器：它仅作可视化，且只在冻结的 V-JEPA 2.1-G 编码器特征上训练（附录 D）。所有可视化共用同一编码器与解码器，只改 RoboJEPA 世界模型规模。给定单帧（或一对帧）与一段机器人动作做前向预测，再把预测嵌入序列解码为像素。我们在低/高分辨率、单/双视角等多种设定下分别报告（图 2、图 4；完整集见附录 E）；解码器本身在单一 $240{\times}320$ 分辨率训练。可视化显示前向预测随规模在几方面具体改善：更大模型在滚动中更贴合真实动力学、更好保留物体身份与细操作细节，并保持多相机视角相互一致。若干行为近乎随规模涌现——夹爪–物体接触、折叠等结构化形变，只有预测器足够大才出现，较小模型会把这些交互糊掉。最后，我们在 DROID 上微调 $8$B 版 RoboJEPA，使其在高分辨率（$720$p）同时想象三路相机（附录 C.4；图 17）。这些滚动表明 RoboJEPA 在很高分辨率下仍能做准确物体操作，并展现强跨视角信息迁移：侧视（外置）内容会渗入腕部相机的想象。

4 Related Work

4 相关工作

Joint-Embedding Predictive Architectures. Joint-Embedding Predictive Architectures (JEPAs) were proposed as a path towards self-supervised models that learn abstract representations of the world by predicting in a latent space rather than at the pixel level (LeCun, 2022). I-JEPA (Assran et al., 2023) instantiated this recipe for images by predicting the representations of masked image regions from visible context, avoiding pixel-level reconstruction and the resulting collapse to low-level texture. V-JEPA extended the framework to video by predicting masked spatio-temporal feature regions and showed that the resulting features capture motion and temporal structure (Bardes et al., 2024). Assran et al. (2025) scaled feature prediction to over a million hours of internet video and, crucially for our setting, post-trained an action-conditioned predictor (V-JEPA 2-AC) on a small amount of unlabeled robot interaction data, enabling zero-shot Franka manipulation by latent planning against image goals. Our work, RoboJEPA, is similar to V-JEPA 2-AC in terms of the approach. We keep the V-JEPA encoder frozen and focus on the action-conditioned predictor as a world model. However, compared to V-JEPA 2 AC, we replace the small post-training corpus with a unified, large-scale mixture of robot demonstrations, therefore, the methods are not apples-to-apples comparable. To our knowledge, we provide the first systematic recipe for scaling V-JEPA-style action-conditioned predictors on large-scale, multi-embodiment robot data, and we characterize how predictor performance scales with parameters and demonstration tokens.

联合嵌入预测架构。联合嵌入预测架构（JEPA）被提出作为通向自监督模型的一条路径：在潜空间而非像素级预测，以学习世界的抽象表示（LeCun，2022）。I-JEPA（Assran 等，2023）把该配方实例化到图像：由可见上下文预测被掩图像区域的表示，避免像素重建及其向低层纹理坍缩。V-JEPA 将其扩展到视频，预测被掩时空特征区域，并表明所得特征捕捉运动与时间结构（Bardes 等，2024）。Assran 等（2025）把特征预测扩展到逾百万小时互联网视频，并——对我们的设定至关重要——在少量未标注机器人交互数据上后训练动作条件预测器（V-JEPA 2-AC），从而能以图像目标做潜空间规划、实现零样本 Franka 操作。本文 RoboJEPA 在方法上接近 V-JEPA 2-AC：冻结 V-JEPA 编码器，把动作条件预测器当作世界模型。但相对 V-JEPA 2-AC，我们用统一的大规模机器人演示混合取代小规模后训练语料，因而不能简单横向比较。据我们所知，我们首次给出在大规模、多本体机器人数据上缩放 V-JEPA 风格动作条件预测器的系统配方，并刻画预测器表现如何随参数与演示词元缩放。

World Models for Robotics. World models compress experience into a learned simulator of the environment that an agent can train and plan in (Ha and Schmidhuber, 2018). The Dreamer family pioneered this paradigm at scale: PlaNet learned latent dynamics for planning from pixels (Hafner et al., 2019), Dreamer used latent imagination to learn behaviors (Hafner et al., 2020), DreamerV2 introduced discrete latents to master Atari (Hafner et al., 2021), and DreamerV3 demonstrated a single set of hyperparameters that succeeds across a wide range of domains including 3D control (Hafner et al., 2023). DayDreamer applied Dreamer directly to physical robots and learned locomotion and manipulation in the real world (Wu et al., 2023), while Masked World Models showed that decoupling visual representation learning from dynamics modeling improves sample efficiency (Seo et al., 2023). More recent work has scaled video-based world models with transformer and diffusion backbones: IRIS (Micheli et al., 2023) and iVideoGPT (Wu et al., 2024a) treat world modeling as autoregressive token prediction; UniSim (Yang et al., 2024), Genie (Bruce et al., 2024), and GAIA-1 (Hu et al., 2023) train large generative simulators of interactive scenes; and the Cosmos platform releases physical-AI world foundation models at industrial scale (NVIDIA, 2025). In contrast to these pixel-space generators, RoboJEPA predicts in the representation space of a frozen V-JEPA encoder, which is what allows us to scale the predictor to billions of parameters on robot data without paying the computational cost of pixel-level reconstruction at every step.

机器人世界模型。世界模型把经验压缩成可训练、可规划的环境仿真器（Ha 与 Schmidhuber，2018）。Dreamer 系列把该范式做大：PlaNet 从像素学习潜动力学用于规划（Hafner 等，2019）；Dreamer 用潜空间想象学行为（Hafner 等，2020）；DreamerV2 引入离散潜变量以精通 Atari（Hafner 等，2021）；DreamerV3 展示同一套超参可跨越含三维控制的广泛领域（Hafner 等，2023）。DayDreamer 把 Dreamer 直接用于物理机器人，在真机上学习运动与操作（Wu 等，2023）；Masked World Models 则表明把视觉表示学习与动力学建模解耦可提高样本效率（Seo 等，2023）。更近的工作用 Transformer 与扩散骨干缩放基于视频的世界模型：IRIS（Micheli 等，2023）与 iVideoGPT（Wu 等，2024a）把世界建模当作自回归词元预测；UniSim（Yang 等，2024）、Genie（Bruce 等，2024）与 GAIA-1（Hu 等，2023）训练大型交互场景生成仿真器；Cosmos 平台则发布工业级物理 AI 世界基础模型（NVIDIA，2025）。与这些像素空间生成器不同，RoboJEPA 在冻结 V-JEPA 编码器的表示空间中预测，从而能在机器人数据上把预测器扩到数十亿参数，而不必每步支付像素级重建的计算代价。

Planning with Learned World Models. A central use of a world model is test-time planning. Classical model-based control methods such as PETS (Chua et al., 2018), model-based RL with neural network dynamics (Nagabandi et al., 2018), and visual foresight (Finn and Levine, 2017) optimize action sequences against a learned dynamics model. Sampling-based shooting methods, in particular the Cross-Entropy Method (Rubinstein, 1997; de Boer et al., 2005) and Model Predictive Path Integral control (Williams et al., 2017), remain the workhorses for high-dimensional action search. TD-MPC and TD-MPC2 combine latent dynamics, value learning, and CEM-style trajectory optimization for robust continuous control (Hansen et al., 2022; Hansen et al., 2024). DINO-WM (Zhou et al., 2025) learns task-agnostic dynamics in DINOv2 feature space from offline trajectories using an approximately $20$ M-parameter predictor, and uses CEM to plan toward image goals without additional task-specific training. RoboJEPA follows this line: at deployment, we use CEM in the latent space of the world model to score action sequences by how close their predicted features land to a goal image’s features, and execute the first action of the best plan in receding-horizon fashion.

用学到的世界模型做规划。世界模型的核心用途之一是测试时规划。经典基于模型的控制——如 PETS（Chua 等，2018）、带神经网络动力学的基于模型 RL（Nagabandi 等，2018）与视觉前瞻（Finn 与 Levine，2017）——相对学到的动力学优化动作序列。基于采样的射击法，尤其是交叉熵方法（Rubinstein，1997；de Boer 等，2005）与模型预测路径积分控制（Williams 等，2017），仍是高维动作搜索的主力。TD-MPC 与 TD-MPC2 结合潜动力学、价值学习与 CEM 风格轨迹优化，实现稳健连续控制（Hansen 等，2022；Hansen 等，2024）。DINO-WM（Zhou 等，2025）在 DINOv2 特征空间、从离线轨迹学习任务无关动力学（约 $20$M 参数预测器），并用 CEM 朝图像目标规划、无需额外任务训练。RoboJEPA 沿此路线：部署时在世界模型潜空间用 CEM，按预测特征与目标图像特征的接近程度给动作序列打分，并以滚动地平方式执行最优计划的首个动作。

Closely related to our work, Terver et al. (2026) investigate how architectural, training, and planning choices affect latent world-model performance, including model and data scaling, albeit at smaller scales. Their proposed DROID predictor has approximately $229$ M parameters and is trained on an 8,000 $8{,}000$ -trajectory dataset. We extend this investigation both by scaling to substantially larger world models (up to $8$ B parameters) and datasets (a mixture of approximately $2.87$ M trajectories across $12$ robot platforms) and by explicitly studying their scaling behavior through scaling laws.

与我们密切相关的是 Terver 等（2026）：他们研究架构、训练与规划选择如何影响潜空间世界模型表现，也包括模型与数据缩放，但尺度更小。其 DROID 预测器约 $229$M 参数，训练于约 $8{,}000$ 条轨迹。我们把调查扩展到显著更大的世界模型（至 $8$B）与数据（约 $2.87$M 条轨迹、跨 $12$ 个机器人平台的混合），并显式通过缩放定律研究缩放行为。

Robotic Foundation Models from Demonstrations. A complementary line of work scales policies directly on robot demonstrations rather than scaling world models. RT-1 (Brohan et al., 2023) and RT-2 (Zitkovich et al., 2023) train large transformer- and vision-language-action policies on broad manipulation data. Open X-Embodiment Collaboration et al. (2023) aggregate demonstrations across many embodiments to enable cross-embodiment generalist policies, and a wave of recent generalist models such as Octo (Ghosh et al., 2024), OpenVLA (Kim et al., 2024), and π 0 $\pi_{0}$ (Black et al., 2024) push this behavior-cloning recipe further. RoboJEPA shares the same data substrate (in particular Open X-Embodiment, DROID (Khazatsky et al., 2024), RoboMIND (Wu et al., 2024b), BridgeData V2 (Walke et al., 2023), LeRobot (Cadene et al., 2024), and AgiBot World (AgiBot-World-Contributors et al., 2025)), but trains a goal-agnostic, action-conditioned predictive model instead of a task-conditioned policy. The same world model can then be used to evaluate or improve any policy, and to plan zero-shot for new goals at test time.

来自演示的机器人基础模型。互补路线是直接在机器人演示上缩放策略，而非缩放世界模型。RT-1（Brohan 等，2023）与 RT-2（Zitkovich 等，2023）在广泛操作数据上训练大型 Transformer 与视觉–语言–动作策略。Open X-Embodiment Collaboration 等（2023）汇聚多本体演示以支持跨本体通用策略；Octo（Ghosh 等，2024）、OpenVLA（Kim 等，2024）与 $\pi_{0}$（Black 等，2024）等近期通用模型进一步推进行为克隆配方。RoboJEPA 共享同一数据底座（尤其 Open X-Embodiment、DROID、RoboMIND、BridgeData V2、LeRobot、AgiBot World），但训练的是目标无关、动作条件的预测模型，而非任务条件策略。同一个世界模型随后可用于评估或改进任意策略，并在测试时为零样本新目标做规划。

Scaling Laws. Power-law scaling of model loss with parameters, data, and compute has been documented extensively for language models (Kaplan et al., 2020), generative modeling more broadly (Henighan et al., 2020), and vision transformers (Zhai et al., 2022), with compute-optimal allocations of model and data established by Chinchilla (Hoffmann et al., 2022). In robotics, however, scaling laws are largely an open question because high-quality interaction data is scarce. We provide, to our knowledge, the first systematic scaling study for action-conditioned world models trained on large-scale, multi-embodiment real robot demonstrations, characterizing how forward-prediction loss scales with predictor parameters and demonstration tokens, and showing that this scaling behavior transfers to downstream planning performance.

缩放定律。模型损失相对参数、数据与算力的幂律缩放已在语言模型（Kaplan 等，2020）、更广的生成建模（Henighan 等，2020）与视觉 Transformer（Zhai 等，2022）上大量记载；Chinchilla（Hoffmann 等，2022）确立了模型与数据的算力最优分配。但在机器人领域，因高质量交互数据稀缺，缩放定律大体仍是开放问题。据我们所知，我们首次对在大规模、多本体真实机器人演示上训练的动作条件世界模型做系统缩放研究，刻画前向预测损失如何随预测器参数与演示词元缩放，并表明该缩放行为迁移到下游规划表现。

Scaling world models. A growing body of work reports that world models improve with scale. In autonomous driving, the GAIA line of latent-diffusion world models has been scaled to billions of parameters for controllable simulation and offline evaluation (Russell et al., 2025; Wayve, 2025; Wayve, 2026), though technical details beyond GAIA-2 are not publicly available. Pearce et al. (2024) fit power laws for pre-trained agents and world models and, in their appendix, report scaling laws for a robotic world model, but their study is otherwise centered on game and simulated environments rather than large-scale real-world multi-embodiment robot data. Sato et al. (2023) study how world-model quality varies with model size; IRASim (Zhu et al., 2025) reports empirical scaling trends for a fine-grained video world model for manipulation; PointWorld (Huang et al., 2026) shows that a 3D point-flow world model improves with more data and parameters; and DINO-world (Baldassarre et al., 2025) scales video prediction in the feature space of a frozen DINOv2 encoder using predictors with up to $1.1$ B parameters. Closest in spirit, τ 0 $\tau_{0}$ -VLA (Cai et al., 2026) scales test-time computation of a hierarchical policy using a world model to score candidate subtasks, and Dyna-2 (Dyna Robotics, 2026) scales a world-action model over a million hours of human and robot video and measures action -prediction quality. RoboJEPA differs from these along two axes: we scale training compute of a pure world model and measure its future-embedding (imagination) accuracy, and we fit an explicit scaling law rather than an empirical trend. This is a very important distinction, as an empirical scaling trend only establishes that a metric improves as one adds parameters, data, or compute, but says nothing about how much it will improve, whether returns have begun to saturate, or whether the next unit of budget is better spent on model size or on data. A scaling law, by contrast, is predictive: it extrapolates performance well beyond the scale at which it was fit, turning these decisions into quantitative forecasts. Establishing such predictive laws for action-conditioned robotic world models—and connecting them to downstream planning—is, to our knowledge, what sets this work apart.

缩放世界模型。越来越多工作报告世界模型随规模改善。自动驾驶中，GAIA 系列潜扩散世界模型已扩到数十亿参数，用于可控仿真与离线评估（Russell 等，2025；Wayve，2025；Wayve，2026），但 GAIA-2 之外的技术细节未公开。Pearce 等（2024）为预训练智能体与世界模型拟合幂律，并在附录报告机器人世界模型的缩放定律，但主体仍围绕游戏与仿真环境，而非大规模真实多本体机器人数据。Sato 等（2023）研究世界模型质量如何随模型大小变化；IRASim（Zhu 等，2025）报告细粒度视频世界模型用于操作的经验缩放趋势；PointWorld（Huang 等，2026）表明三维点流世界模型随数据与参数改善；DINO-world（Baldassarre 等，2025）在冻结 DINOv2 特征空间缩放视频预测，预测器至 $1.1$B。精神上最接近的是 $\tau_{0}$-VLA（Cai 等，2026）用世界模型给候选子任务打分以缩放分层策略的测试时计算，以及 Dyna-2（Dyna Robotics，2026）在逾百万小时人机视频上缩放世界–动作模型并度量动作预测质量。RoboJEPA 在两轴上不同：我们缩放的是纯世界模型的训练算力并度量其未来嵌入（想象）精度；我们拟合的是显式缩放定律而非经验趋势。这一区分很重要：经验趋势只说明加参数/数据/算力时指标变好，却不回答会好多少、回报是否开始饱和、下一单位预算该投模型还是数据。缩放定律则具有预测性：可在拟合尺度之外外推表现，把这些决策变成定量预报。据我们所知，为动作条件机器人世界模型建立此类可预测定律、并将其与下游规划相连，正是本文的独特之处。

5 Conclusion

5 结论

We introduced RoboJEPA, a family of action-conditioned latent world models trained on a large-scale, multi-embodiment robotic data collection in the latent space of V-JEPA 2.1. RoboJEPA predicts the next embedding in a single forward pass, and we train it to predict the future conditioned on past embeddings and future actions. We presented a recipe for scaling the predictor that works from 22M to 8B parameters, the latter being the largest JEPA predictor model trained to date. We then established scaling laws that predict the world-modeling error as we scale training compute. These take the form of a second-order power law L ⁡ (C) = E + A ​ C α − γ ​ ln ⁡ C $L(C)=E+A\,C^{\alpha-\gamma\ln C}$. We established these laws for both a real-world (DROID) and a simulated (RoboCasa) embodiment. Finally, we demonstrated that offline prediction scaling laws correlate with the online downstream performance of world models, evaluated via planning toward a single image goal, which makes the offline imagination error a reliable proxy for real-robot evaluation and spares much of its computational cost. We also found that distinct manipulation capabilities emerge at their own compute thresholds, in a natural order from end-effector control, to manipulating a grasped object, to reasoning about static scene geometry, and finally to the dynamics of objects in the scene. On a real robot, we showed that the policy induced by world-model planning improves in success rate and in qualitative behavior as the world model scales. We also built a diffusion decoder to visualize the predicted action-conditioned future in pixel space, and used it to show that qualitatively better robotic predictions emerge with scale. We release all model checkpoints together with our training and robot-deployment code.

我们提出 RoboJEPA：在 V-JEPA 2.1 潜空间、于大规模多本体机器人数据上训练的动作条件潜空间世界模型族。RoboJEPA 单次前向预测下一嵌入，并在过去嵌入与未来动作条件下训练以预测未来。我们给出从 22M 到 8B 的预测器缩放配方，后者是迄今最大的 JEPA 预测器。随后建立随训练算力预测世界建模误差的缩放定律，形式为二阶幂律 $L(C)=E+A\,C^{\alpha-\gamma\ln C}$，并在真机（DROID）与仿真（RoboCasa）本体上分别确立。我们还表明，离线预测缩放定律与朝向单张图像目标规划所评估的在线下游表现相关，从而使离线想象误差成为真机评测的可靠代理，并省去大量算力开销。我们还发现，不同操作能力在各自的算力阈值涌现，顺序自然：末端控制 → 持物操作 → 静态场景几何 → 场景中物体动力学。在真机上，世界模型规划诱导的策略随世界模型缩放，在成功率与定性行为上都改善。我们还构建扩散解码器，在像素空间可视化动作条件的预测未来，并用以展示随规模涌现的定性更优机器人预测。我们公开全部模型检查点以及训练与真机部署代码。

Our study also has limitations that mark natural next steps. We scaled the predictor while keeping the V-JEPA 2.1 encoder frozen, so our laws describe a dynamics model on top of a fixed representation rather than the encoder and predictor jointly. Because we train for multiple epochs over a fixed corpus, our fits already sit close to data saturation, so pushing the frontier further will likely require more and more diverse interaction data rather than parameters alone. RoboJEPA also makes no use of text, planning instead toward a single image goal, and it makes no use of a learned policy proposal during planning, sampling actions from a uniform distribution and prioritizing them with the world model.

本研究亦有局限，标出自然的下一步。我们在冻结 V-JEPA 2.1 编码器的同时缩放预测器，故定律描述的是固定表示之上的动力学模型，而非编码器与预测器联合缩放。由于在固定语料上多轮训练，拟合已接近数据饱和；要继续推进前沿，更可能需要更多样的交互数据，而非仅堆参数。RoboJEPA 也不使用文本，而是朝向单张图像目标规划；规划时也不使用学到的提议策略，而是从均匀分布采样动作再由世界模型排序。

References

参考文献

1X Technologies (2024) 1X Technologies. 1X world model dataset. https://huggingface.co/datasets/1x-technologies/world_model_raw_data, 2024.

AgiBot-World-Contributors et al. (2025) AgiBot-World-Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, et al. AgiBot World Colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025.

Ai et al. (2025) Bo Ai, Liu Dai, et al. Towards embodiment scaling laws in robot locomotion. In Conference on Robot Learning (CoRL), 2025.

Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Empirical Methods in Natural Language Processing (EMNLP), 2023.

Ansel et al. (2024) Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, C. K. Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster machine learning through dynamic Python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2024. 10.1145/3620665.3640366.

Assran et al. (2023) Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

Assran et al. (2025) Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025.

Baldassarre et al. (2025) Federico Baldassarre, Marc Szafraniec, Basile Terver, Vasil Khalidov, Francisco Massa, Yann LeCun, Patrick Labatut, Maximilian Seitzer, and Piotr Bojanowski. Back to the features: DINO as a foundation for video world models. arXiv preprint arXiv:2507.19468, 2025. https://arxiv.org/abs/2507.19468.

Bardes et al. (2024) Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. In Transactions on Machine Learning Research (TMLR), 2024.

Bharadhwaj et al. (2024) Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. RoboAgent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In IEEE International Conference on Robotics and Automation (ICRA), 2024.

Black et al. (2024) Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π 0 $\pi_{0}$: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024.

Black et al. (2025) Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. π 0.5 $\pi_{0.5}$: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025.

Bordes et al. (2025) Florian Bordes, Quentin Garrido, Justine T Kao, Adina Williams, Michael Rabbat, and Emmanuel Dupoux. IntPhys 2: Benchmarking intuitive physics understanding in complex synthetic environments. arXiv preprint arXiv:2506.09849, 2025.

Brohan et al. (2023) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems (RSS), 2023.

Bruce et al. (2024) Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel. Genie: Generative interactive environments. In International Conference on Machine Learning (ICML), 2024.

Caballero et al. (2022) Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. arXiv preprint arXiv:2210.14891, 2022.

Caballero et al. (2026) Ethan Caballero, Priyank Jaini, David Krueger, and Irina Rish. Unified neural scaling laws. arXiv preprint arXiv:2605.26248, 2026.

Cadene et al. (2024) Remi Cadene, Simon Alibert, Alexander Soare, Quentin Gallouedec, Adil Zouitine, Steven Palma, Pepijn Kooijmans, Michel Aractingi, Mustafa Shukor, Dana Aubakirova, Martino Russi, Francesco Capuano, Caroline Pascal, Jade Choghari, Khalil Meftah, Maxime Ellerbach, Jess Moss, and Thomas Wolf. LeRobot: State-of-the-art machine learning for real-world robotics in PyTorch. https://github.com/huggingface/lerobot, 2024.

Cai et al. (2026) Xiaowei Cai, Yunuo Cai, Bingao Chen, Jingxiao Chen, Zhi Chen, Siyuan Feng, et al. τ 0 $\tau_{0}$ -VLA: a hierarchical robot foundation model with world-model-guided test-time computation. arXiv preprint arXiv:2608.16885, 2026.

Chen et al. (2024a) Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 24081–24125, 2024a. 10.52202/079017-0759. https://proceedings.neurips.cc/paper_files/paper/2024/hash/2aee1c4159e48407d68fe16ae8e6e49e-Abstract-Conference.html.

Chen et al. (2024b) Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. PixArt- α $\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representations (ICLR), 2024b.

Chua et al. (2018) Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems (NeurIPS), 2018.

Dao (2024) Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.

Dao et al. (2022) Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

Dasari et al. (2019) Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. RoboNet: Large-scale multi-robot learning. In Conference on Robot Learning (CoRL), 2019.

de Boer et al. (2005) Pieter-Tjerk de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134(1):19–67, 2005.

Dehghani et al. (2023) Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf, Mario Lučić, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (ICML), 2023.

Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.

Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.

Dyna Robotics (2026) Dyna Robotics. Dyna-2: A 1-million-hour scaling law for world-action models. https://www.dyna.co/dyna-2, 2026. Accessed: 2026-09-04.

Ebert et al. (2022) Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. In Robotics: Science and Systems (RSS), 2022.

Finn and Levine (2017) Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In IEEE International Conference on Robotics and Automation (ICRA), 2017.

Ghosh et al. (2024) Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Quan Vuong, Ted Xiao, Pannag R Sanketi, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, 2024. 10.15607/RSS.2024.XX.090. https://www.roboticsproceedings.org/rss20/p090.html.

Google DeepMind (2024) Google DeepMind. Genie 2: A large-scale foundation world model. https://deepmind.google/blog/genie-2-a-large-scale-foundation-world-model/, 2024. Technical report / blog post.

Ha and Schmidhuber (2018) David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems (NeurIPS), 2018.

Hafner et al. (2019) Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning (ICML), 2019.

Hafner et al. (2020) Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), 2020.

Hafner et al. (2021) Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with discrete world models. In International Conference on Learning Representations (ICLR), 2021.

Hafner et al. (2023) Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.

Hafner et al. (2025) Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. arXiv preprint arXiv:2509.24527, 2025.

Hansen et al. (2022) Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In International Conference on Machine Learning (ICML), 2022.

Hansen et al. (2024) Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024.

Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.

Henighan et al. (2020) Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020.

Ho and Salimans (2022) Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. https://arxiv.org/abs/2207.12598.

Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.

Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 30016–30030, 2022. 10.52202/068431-2176. https://proceedings.neurips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract.html.

Hoogeboom et al. (2023) Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning (ICML), 2023.

Hu et al. (2023) Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. GAIA-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023.

Hu et al. (2024) Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024.

Huang et al. (2026) Wenlong Huang, Yu-Wei Chao, Arsalan Mousavian, Ming-Yu Liu, Dieter Fox, Kaichun Mo, et al. PointWorld: Scaling 3d world models for in-the-wild robotic manipulation. arXiv preprint arXiv:2601.03782, 2026.

Jang et al. (2022) Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-Z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning (CoRL), 2022.

Kalashnikov et al. (2018) Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 651–673, 2018. https://proceedings.mlr.press/v87/kalashnikov18a.html.

Kalashnikov et al. (2021) Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. MT-Opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212, 2021.

Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.

Khazatsky et al. (2024) Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O’Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. DROID: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024.

Kim et al. (2024) Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning (CoRL), 2024.

LeCun (2022) Yann LeCun. A path towards autonomous machine intelligence. Open Review, 2022. https://openreview.net/forum?id=BZ5a1r-kVsf.

Lin et al. (2025) Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. In International Conference on Learning Representations (ICLR), 2025.

Lin et al. (2024) Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024.

Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.

Luo et al. (2024) Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel, and Sergey Levine. FMB: A functional manipulation benchmark for generalizable robotic learning. The International Journal of Robotics Research, 2024.

Lynch et al. (2023) Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. In IEEE Robotics and Automation Letters (RA-L), 2023.

Mayne et al. (2000) D. Q. Mayne, J. B. Rawlings, C. V. Rao, and P. O. M. Scokaert. Constrained model predictive control: Stability and optimality. Automatica, 36(6):789–814, 2000. 10.1016/S0005-1098(99)00214-9.

Micheli et al. (2023) Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In International Conference on Learning Representations (ICLR), 2023.

Mur-Labadia et al. (2026) Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Mike Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-JEPA 2.1: Unlocking dense features in video self-supervised learning. arXiv preprint arXiv:2603.14482, 2026.

Nagabandi et al. (2018) Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In IEEE International Conference on Robotics and Automation (ICRA), 2018.

Nasiriany et al. (2024) Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), 2024.

Nasiriany et al. (2026) Soroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri, and Yuke Zhu. RoboCasa365: A large-scale simulation framework for training and benchmarking generalist robots. arXiv preprint arXiv:2603.04356, 2026.

Nichol and Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICML), 2021.

Nilaksh et al. (2026) Nilaksh, Saurav Jha, Artem Zholus, and Sarath Chandar. Reconstruction or semantics? what makes a latent space useful for robotic world models. arXiv preprint arXiv:2605.06388, 2026. https://arxiv.org/abs/2605.06388.

NVIDIA (2025) NVIDIA. Cosmos world foundation model platform for physical AI. arXiv preprint arXiv:2501.03575, 2025.

Open X-Embodiment Collaboration et al. (2023) Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open X-embodiment: Robotic learning datasets and RT-X models. arXiv preprint arXiv:2310.08864, 2023.

PAN Team et al. (2025) PAN Team, Jiannan Xiang, Yi Gu, Zihan Liu, Zeyu Feng, Qiyue Gao, Yiyan Hu, Benhao Huang, Guangyi Liu, Yichi Yang, Kun Zhou, et al. PAN: A world model for general, interactable, and long-horizon world simulation. arXiv preprint arXiv:2511.09057, 2025. https://arxiv.org/abs/2511.09057v1.

Pearce and Song (2024) Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. arXiv preprint arXiv:2406.12907, 2024.

Pearce et al. (2024) Tim Pearce, Tabish Rashid, Dave Bignell, Raluca Georgescu, Sam Devlin, and Katja Hofmann. Scaling laws for pre-training agents and world models. arXiv preprint arXiv:2411.04434, 2024.

Peebles and Xie (2023) William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

Pertsch et al. (2025) Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747, 2025.

Po et al. (2025) Ryan Po, Yotam Nitzan, Richard Zhang, Berlin Chen, Tri Dao, Eli Shechtman, Gordon Wetzstein, and Xun Huang. Long-context state-space video world models. arXiv preprint arXiv:2505.20171, 2025.

Punzo et al. (2026) Samuele Punzo, Niccolò Caselli, Ippokratis Pantelidis, Francesco Massafra, Salvatore Lo Sardo, and Mohammadreza Salehi. How do video foundation models encode intuitive physics? probing across pretraining paradigms. arXiv preprint arXiv:2606.09646, 2026.

Rubinstein (1997) Reuven Y. Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1):89–112, 1997.

Russell et al. (2025) Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. GAIA-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523, 2025.

Sartor and Thompson (2024) Sebastian Sartor and Neil C. Thompson. Neural scaling laws in robotics. arXiv preprint arXiv:2405.14005, 2024.

Sato et al. (2023) Makoto Sato, Ryosuke Unno, Masahiro Negishi, Koudai Tabata, Taiju Watanabe, Junnosuke Kamohara, Taiga Kume, Ryo Okada, Yusuke Iwasawa, and Yutaka Matsuo. Scaling laws of model size for world models. In Proceedings of the Annual Conference of the Japanese Society for Artificial Intelligence (JSAI), 2023. 10.11517/pjsai.JSAI2023.0_2G5OS21e02. In Japanese.

Seo et al. (2023) Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning (CoRL), 2023.

Song et al. (2021) Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021.

Su et al. (2024) Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.

Terver et al. (2026) Basile Terver, Tsung-Yen Yang, Jean Ponce, Adrien Bardes, and Yann LeCun. What drives success in physical planning with joint-embedding predictive world models? Transactions on Machine Learning Research, 2026. https://openreview.net/forum?id=cHZn5Gdh8e.

Walke et al. (2023) Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. BridgeData V2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.

Wayve (2025) Wayve. GAIA-3: Scaling world models to power safety and evaluation. https://wayve.ai/thinking/gaia-3/, 2025. Accessed: 2026-09-04.

Wayve (2026) Wayve. GAIA-4: Multimodal world models powering closed-loop simulation for safe and scalable autonomy. https://wayve.ai/thinking/gaia-4/, 2026. Accessed: 2026-09-04.

Williams et al. (2017) Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M. Rehg, Byron Boots, and Evangelos A. Theodorou. Information theoretic MPC for model-based reinforcement learning. In IEEE International Conference on Robotics and Automation (ICRA), 2017. 10.1109/ICRA.2017.7989202.

Wu et al. (2024a) Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. iVideoGPT: Interactive VideoGPTs are scalable world models. In Advances in Neural Information Processing Systems (NeurIPS), 2024a.

Wu et al. (2024b) Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, Zhen Zhao, Guangyu Li, Zhao Jin, Lecheng Wang, Jilei Mao, Xinhua Wang, Shichao Fan, Ning Liu, Pei Ren, Qiang Zhang, Yaoxu Lyu, Mengzhen Liu, Jingyang He, Yulin Luo, Zeyu Gao, Chenxuan Li, Chenyang Gu, Yankai Fu, Di Wu, Xingyu Wang, Sixiang Chen, Zhenyu Wang, Pengju An, Siyuan Qian, Shanghang Zhang, and Jian Tang. RoboMIND: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877, 2024b. https://arxiv.org/abs/2412.13877v1.

Wu et al. (2023) Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. DayDreamer: World models for physical robot learning. In Conference on Robot Learning (CoRL), 2023.

Yang et al. (2024) Sherry Yang, Yilun Du, Seyed Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In International Conference on Learning Representations (ICLR), pages 45210–45234, 2024. https://proceedings.iclr.cc/paper_files/paper/2024/hash/c4d66eae503694424123b93ac0fbaf17-Abstract-Conference.html.

Zhai et al. (2022) Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

Zhang and Sennrich (2019) Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems (NeurIPS), 2019.

Zhang et al. (2026) Wancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis, Harsh Sutaria, Mido Assran, Randall Balestriero, Amir Bar, Adrien Bardes, Yann LeCun, et al. Hierarchical planning with latent world models. arXiv preprint arXiv:2604.03208, 2026.

Zhou et al. (2025) Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 79115–79135. PMLR, 2025. https://proceedings.mlr.press/v267/zhou25t.html.

Zhu et al. (2025) Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. IRASim: A fine-grained world model for robot manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.

Zitkovich et al. (2023) Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Jie Tan, Marc Toussaint, and Kourosh Darvish, editors, Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 2165–2183. PMLR, 06–09 Nov 2023. https://proceedings.mlr.press/v229/zitkovich23a.html.

Appendix A Model Architecture: Additional Details

附录 A 模型架构：补充细节

Expands Section 2.1. We collect here the implementation-level details of the predictor block: the channel-pair derivation behind factorised RoPE (Section A.1); the per-block view bias under grouped-query attention (Section A.2); the per-embodiment input projection (Section A.4); the embodiment-id breakdown that yields | ℰ | = 26 $|\mathcal{E}|{=}26$ (Section A.5); and the predictor scaling sweep across all eight scales used in the scaling law of Section 3 (Section A.6). All notation is shared with Section 2.1; in particular, ℓ $\ell$ is reserved for the predictor layer index and τ $\tau$ for autoregressive rollout steps.

扩展第 2.1 节。此处汇总预测器块的实现级细节：因子化 RoPE 背后的通道对推导（A.1）；分组查询注意力下的逐块视角偏置（A.2）；按本体的输入投影（A.4）；给出 $|\mathcal{E}|{=}26$ 的本体 id 分解（A.5）；以及第 3 节缩放定律所用全部八个规模的预测器扫描（A.6）。记号与第 2.1 节一致：尤其 $\ell$ 专指预测器层索引，$\tau$ 专指自回归滚动步。

Table 4 summarises how the RoboJEPA predictor differs from the action-conditioned V-JEPA 2 predictor (V-JEPA 2-AC) (Assran et al., 2025) it builds on. The two share the same backbone family and the same next-frame prediction parameterisation; RoboJEPA changes the normalisation, position encoding, and the conditioning that supports cross-view, cross-embodiment world modeling.

表 4 概括 RoboJEPA 预测器相对其所基于的动作条件 V-JEPA 2 预测器（V-JEPA 2-AC）（Assran 等，2025）有何不同。二者共享同一骨干族与同一下一帧预测参数化；RoboJEPA 改动了归一化、位置编码，以及支撑跨视角、跨本体世界建模的条件化。

Table 4: Architectural comparison between the action-conditioned V-JEPA 2 predictor (V-JEPA 2-AC) (Assran et al., 2025) and the RoboJEPA predictor. Both share the V-JEPA 2 backbone and the next-frame latent-prediction parameterisation; RoboJEPA adds RMSNorm and QK-norm for training stability at scale, a fourth (view) RoPE axis, and explicit per-embodiment and per-view conditioning for cross-embodiment, multi-view world modeling.

表 4：动作条件 V-JEPA 2 预测器（V-JEPA 2-AC）（Assran 等，2025）与 RoboJEPA 预测器的架构对比。二者共享 V-JEPA 2 骨干与下一帧潜预测参数化；RoboJEPA 增加 RMSNorm 与 QK-norm 以利大规模训练稳定，增加第四（视角）RoPE 轴，以及显式的按本体、按视角条件，以支持跨本体、多视角世界建模。

A.1 Factorised multi-axis RoPE

A.1 因子化多轴 RoPE

Each per-head channel space { 0, …, d h − 1 } $\{0,\dots,d_{h}-1\}$ partitions into four contiguous, equal-size sub-blocks of width d ax = d h / 4 $d_{\mathrm{ax}}{=}d_{h}/4$, one per spatio-temporal axis. Fix the enumeration 𝒜 max = { t, i, j, v } $\mathcal{A}_{\max}\!=\!\{t,i,j,v\}$ for the time, height, width, and view axes, and let j α $j_{\alpha}$ denote the starting channel of axis α $\alpha$, so

每个头的通道空间 $\{0,\dots,d_{h}-1\}$ 被划分为四个连续、等宽子块，宽 $d_{\mathrm{ax}}{=}d_{h}/4$，各对应一个时空轴。固定枚举 $\mathcal{A}_{\max}\!=\!\{t,i,j,v\}$ 表示时间、高、宽、视角轴，并以 $j_{\alpha}$ 记轴 $\alpha$ 的起始通道，于是

Under multi-view training (V max = 2 $V_{\max}{=}2$) we use the full 4-axis partition 𝒜 = { t, i, j, v } $\mathcal{A}\!=\!\{t,i,j,v\}$: the $v$ -axis sub-block contributes its own rotation, and viewpoint identity is additionally supplied by the per-block additive bias of Section A.2 (the two are complementary). For purely single-view runs (V = 1 $V{=}1$) the $v$ -axis sub-block carries no rotation and is left as the identity. Action and state tokens are detached from the visual slab before the RoPE rotation is applied: only the time component of RoPE rotates them (indexed by their step id), so that a single proprioceptive token per step does not spuriously inherit a spatial position.

多视角训练（$V_{\max}{=}2$）时使用完整四轴划分 $\mathcal{A}\!=\!\{t,i,j,v\}$：$v$ 轴子块贡献自有旋转，视角身份另由 A.2 节的逐块加性偏置提供（二者互补）。纯单视角（$V{=}1$）时 $v$ 轴子块不旋转、保持恒等。动作与状态词元在施加 RoPE 前从视觉板分离：只对它们施加 RoPE 的时间分量（按步 id 索引），以免每步单个本体词元错误继承空间位置。

A.2 Per-block view bias under grouped-query attention

A.2 分组查询注意力下的逐块视角偏置

The view information is encoded into the model in two ways. First, the view position encoding disambiguates different views of tokens with the same spatio-temporal position. This is achieved by rotating the view-axis sub-block (d h / 4 $d_{h}/4$ channels per head). However, the view position only informs the model that two different views are distinct without any encoding of the type of the content (e.g. the model has to know which view represents the side camera looking from the left and which one is mounted on the wrist). One way of dealing with it is to just let the model learn this end-to-end; however, we decided to directly inject the view information as a learnable bias added across the full head width of visual-token queries and keys after the four-axis RoPE rotation:

视角信息以两种方式编入模型。其一，视角位置编码区分具有相同时空位置的不同视角词元，做法是旋转视角轴子块（每头 $d_{h}/4$ 通道）。但视角位置只告诉模型两视角不同，并不编码内容类型（例如须知哪路是左侧侧视、哪路是腕部）。一端到端学习也可以，但我们选择在四轴 RoPE 之后，直接把视角信息作为可学习偏置加到视觉词元查询与键的整个头宽上：

The strong-view conditioning of Equation 9 is a zero-initialised, layer-wise additive bias indexed by the semantic camera id c (v) ∈ 𝒱 $c^{(v)}\!\in\!\mathcal{V}$ (left, right, wrist, front, back, head, top), implemented as U ℓ ∈ ℝ | 𝒱 | × d h $U_{\ell}\!\in\!\mathbb{R}^{|\mathcal{V}|\times d_{h}}$ for each ℓ ∈ { 1, …, N layers } $\ell\!\in\!\{1,\dots,N_{\text{layers}}\}$ and added to all d h $d_{h}$ channels of every visual token’s queries and keys after the RoPE rotation of Equation 8. Concretely, let Q ℓ ∈ ℝ N × h q × d h $Q_{\ell}\!\in\!\mathbb{R}^{N\times h_{q}\times d_{h}}$ and K ℓ ∈ ℝ N × h k ​ v × d h $K_{\ell}\!\in\!\mathbb{R}^{N\times h_{kv}\times d_{h}}$ be the post-RoPE query and key tensors of layer ℓ $\ell$ for a per-step token bag of length $N$, and let b ℓ (v):= U ℓ ​ [ c (v) ] ∈ ℝ d h $b_{\ell}^{(v)}\!:=\!U_{\ell}[c^{(v)}]\!\in\!\mathbb{R}^{d_{h}}$ be the bias gathered for view $v$. Under grouped-query attention (Ainslie et al., 2023) with h q $h_{q}$ query heads and h k ​ v $h_{kv}$ KV heads sharing the same head dimension d h $d_{h}$, the bias is added by the broadcast rule

式 9 的强视角条件是零初始化、按层的加性偏置，由语义相机 id $c^{(v)}\!\in\!\mathcal{V}$（left、right、wrist、front、back、head、top）索引，实现为每层 $\ell\!\in\!\{1,\dots,N_{\text{layers}}\}$ 的$U_{\ell}\!\in\!\mathbb{R}^{|\mathcal{V}|\times d_{h}}$，并在式 8 的 RoPE 旋转之后加到每个视觉词元查询与键的全部 $d_{h}$ 通道。具体地，令 $Q_{\ell}\!\in\!\mathbb{R}^{N\times h_{q}\times d_{h}}$、$K_{\ell}\!\in\!\mathbb{R}^{N\times h_{kv}\times d_{h}}$ 为层 $\ell$、长度 $N$ 的逐步词元袋在 RoPE 后的查询与键张量，并令 $b_{\ell}^{(v)}\!:=\!U_{\ell}[c^{(v)}]\!\in\!\mathbb{R}^{d_{h}}$ 为视角 $v$ 取到的偏置。在分组查询注意力（Ainslie 等，2023）下，$h_{q}$ 个查询头与 $h_{kv}$ 个共享头维 $d_{h}$ 的 KV 头按广播规则相加偏置

This per-layer additive bias supplements the $v$ -axis RoPE rotation of Equation 6: under our default strong-view configuration both signals are active, the rotation contributes a relative-view phase to attention scores, and the bias contributes a d h $d_{h}$ -wide per-layer per-camera offset on top of it. The combination gives a stable handle on viewpoint identity for multi-view planning (Section 2.2); zero initialisation of U ℓ $U_{\ell}$ makes loading a checkpoint trained without strong view conditioning a no-op at step $0$.

该逐层加性偏置补充式 6 的 $v$ 轴 RoPE 旋转：在默认强视角配置下两信号同时生效，旋转为注意力分数贡献相对视角相位，偏置再叠加 $d_{h}$ 宽的逐层、逐相机偏移。组合为多视角规划提供稳定的视角身份把手（第 2.2 节）；$U_{\ell}$ 零初始化使得加载未用强视角条件训练的检查点在第 $0$ 步是空操作。

A.3 Attention masking

A.3 注意力掩码

We use a similar attention mask to V-JEPA 2-AC (Assran et al., 2025) but extend it to multiple views: attention is timestep-level causal and restricted to a local temporal window of $8$ timesteps, while within a step all V ​ H p ​ W p $VH_{p}W_{p}$ visual tokens (plus the action and state tokens) attend omnidirectionally (any-to-any). More precisely, the rasterized sequence of tokens has T × N tok $T\times N_{\text{tok}}$ tokens in total and the token at index $k$ can attend to tokens with indices $j$ such that ⌊ k N tok ⌋ − 7 ⩽ ⌊ j N tok ⌋ ⩽ ⌊ k N tok ⌋ $\lfloor\frac{k}{N_{\text{tok}}}\rfloor-7\leqslant\lfloor\frac{j}{N_{\text{tok}}}\rfloor\leqslant\lfloor\frac{k}{N_{\text{tok}}}\rfloor$. Schematically, this means that packing the sequence as $T$ frame-blocks of size N tok $N_{\mathrm{tok}}$ given by Equation 20, the attention mask between blocks is banded lower-triangular: each block of size N tok × N tok $N_{\mathrm{tok}}{\times}N_{\mathrm{tok}}$ is fully attended within the diagonal band of width W t $W_{t}$ and zero outside it. This trick makes the computational cost of attention grow linearly as a function of the step number and it alleviates the problem of the position extrapolation – the predicted content quality depends only on the accumulated error, not the ability of the model to extrapolate, because it never observes a sequence longer than the window size. This is a classical trick for language models and world models (Assran et al., 2025; Po et al., 2025; PAN Team et al., 2025). The bounded window keeps the per-step computational cost of attention constant and makes the predictor KV-cacheable in 𝒪 ⁡ (W t ⋅ V ⋅ H p ⋅ W p) $\mathcal{O}(W_{t}\cdot V\cdot H_{p}\cdot W_{p})$ keys per layer; Section 2.2 uses this at deployment.

我们使用与 V-JEPA 2-AC（Assran 等，2025）类似的注意力掩码，但扩展到多视角：注意力在时间步级因果，并限制在 $8$ 个时间步的局部时间窗内；同一步内全部 $VH_{p}W_{p}$ 个视觉词元（外加动作与状态词元）全互连（any-to-any）。更精确地，光栅化词元序列共有 $T\times N_{\text{tok}}$ 个词元，索引 $k$ 的词元可关注满足$\lfloor\frac{k}{N_{\text{tok}}}\rfloor-7\leqslant\lfloor\frac{j}{N_{\text{tok}}}\rfloor\leqslant\lfloor\frac{k}{N_{\text{tok}}}\rfloor$ 的索引 $j$。示意上，按式 20 把序列打成大小为 $N_{\mathrm{tok}}$ 的 $T$ 个帧块后，块间注意力掩码是带宽下三角：每个 $N_{\mathrm{tok}}{\times}N_{\mathrm{tok}}$ 块在宽为 $W_{t}$ 的对角带内全互连，带外为零。该技巧使注意力计算量随步数线性增长，并缓解位置外推问题——预测内容质量只依赖累积误差，而不依赖模型外推能力，因为它从未见到超过窗口的序列。这是语言模型与世界模型中的经典做法（Assran 等，2025；Po 等，2025；PAN Team 等，2025）。有界窗口使每步注意力代价恒定，并使预测器可按每层 $\mathcal{O}(W_{t}\cdot V\cdot H_{p}\cdot W_{p})$ 键做 KV 缓存；第 2.2 节在部署时使用。

A.4 Per-embodiment input projection

A.4 按本体的输入投影

Every embodiment $e$ owns its own input weight matrix W a (e) $W_{a}^{(e)}$ in a single multi-embodiment linear projection. The raw action vector a t ∈ ℝ d a $a_{t}\!\in\!\mathbb{R}^{d_{a}}$ is zero-padded to a common width d a max $d_{a}^{\max}$ jointly large enough to absorb the widest action convention in our mixture (Section 2.2; for the 1B run d a max = d s max = 64 $d_{a}^{\max}{=}d_{s}^{\max}{=}64$, set by the chunked DROID joint_vel action-key variant which produces an ⋅ = 64 $8\!\cdot\!8\!=\!64$ -D mega-action) before projection,

每个本体 $e$ 在单一多本体线性投影中拥有自己的输入权重矩阵 $W_{a}^{(e)}$。原始动作向量 $a_{t}\!\in\!\mathbb{R}^{d_{a}}$ 先零填充到共同宽度 $d_{a}^{\max}$，该宽度足以吸收混合中最宽的动作约定（第 2.2 节；1B 实验取 $d_{a}^{\max}{=}d_{s}^{\max}{=}64$，由分块 DROID joint_vel 动作键变体产生的 $8\cdot 8=64$ 维巨型动作设定），再投影，

The action token participates in self-attention as a regular token: the projected vector ActTok ⁡ (a t, e) $\mathrm{ActTok}(a_{t},e)$ is concatenated to the per-step token bag and attends to (and is attended by) all visual and state tokens of the same step, and all visual / action / state tokens within the local temporal window W t $W_{t}$. The state token participates symmetrically. Token-style conditioning, in contrast to additive action embedding, lets the predictor express step-dependent gating of action effects (e.g., the gripper acting differently early vs. late in a grasp) without inflating the per-step input dimensionality.

动作词元作为普通词元参与自注意力：投影向量 $\mathrm{ActTok}(a_{t},e)$ 拼入逐步词元袋，与同一步全部视觉与状态词元互相关注，并在局部时间窗 $W_{t}$ 内与视觉/动作/状态词元互相关注。状态词元对称参与。相对加性动作嵌入，词元式条件使预测器能表达动作效应的步依赖门控（例如抓取早段与晚段夹爪作用不同），而不抬高每步输入维数。

A.5 Robot-id breakdown

A.5 机器人 id 分解

The training mixture spans $12$ physical robotic platforms (the Total row of Table 1). We treat each (platform, action-source) pair as a distinct robot id, which yields | ℰ | = 26 $|\mathcal{E}|{=}26$ ids, split into $12$ default state-delta ids (one per platform) plus $14$ extra splits for those datasets that ship more than one synchronized controller stream or distinct state convention not collapsed by the action-key pipeline of Section B.7. Table 5 lists the visible breakdown (# $\#$ / Platform / Notes); the internal embodiment_name strings used by the training config are collected in the table footnote.

训练混合覆盖 $12$ 个物理机器人平台（表 1 合计行）。我们将每个（平台，动作源）对视为不同机器人 id，得到 $|\mathcal{E}|{=}26$ 个 id：其中 $12$ 个默认状态差分 id（每平台一个），外加 $14$ 个额外拆分——对应那些提供不止一路同步控制器流、或状态约定无法被 B.7 节动作键管线合并的数据集。表 5 列出可见分解（# / 平台 / 备注）；训练配置使用的内部 embodiment_name 字符串见该表脚注。

Table 5: Robot-id breakdown used in this work. The first $12$ rows are default state-delta ids, one per physical platform. The next $14$ rows are extra (platform, action-source) or (platform, state-convention) splits. Total robot ids: | ℰ | = + = 26 $|\mathcal{E}|\!=\!12\!+\!14\!=\!26$.

表 5：本工作使用的机器人 id 分解。前 $12$ 行为默认状态差分 id，每物理平台一个；后 $14$ 行为额外的（平台，动作源）或（平台，状态约定）拆分。机器人 id 总数：$|\mathcal{E}|\!=\!12\!+\!14\!=\!26$。

A.6 Predictor scaling sweep

A.6 预测器缩放扫描

For the scaling study of Section 3, we train $22$ M, $50$ M, $100$ M, $300$ M, $1$ B, $2$ B, $4$ B, and $8$ B variants with the same block recipe (pre-norm GQA + GELU MLP, factorised 4-axis RoPE on (t, i, j, v) $(t,i,j,v)$ supplemented by the per-block view bias of Equation 9, encoder, padded action / state widths, local temporal window, optimizer schedule). The eight variants differ only in N layers $N_{\text{layers}}$, $D$, and the GQA head counts (h q, h k ​ v) $(h_{q},\,h_{kv})$ (Table 6); the head dimension d h = 128 $d_{h}{=}128$ and the MLP hidden ratio 4 ​ D $4D$ are held constant. The KV head count matches the query head count at the smallest scales and approaches h k ​ v = h q / 2 $h_{kv}=h_{q}/2$ as model size increases.

为第 3 节缩放研究，我们训练 $22$M、$50$M、$100$M、$300$M、$1$B、$2$B、$4$B、$8$B 变体，块配方相同（预归一化 GQA + GELU MLP、在 $(t,i,j,v)$ 上的因子化四轴 RoPE 外加式 9 的逐块视角偏置、编码器、填充后的动作/状态宽度、局部时间窗、优化器日程）。八个变体仅在 $N_{\text{layers}}$、$D$ 与 GQA 头数 $(h_{q},\,h_{kv})$ 上不同（表 6）；头维 $d_{h}{=}128$ 与 MLP 隐比 $4D$ 固定。最小规模时 KV 头数等于查询头数，随模型增大趋近 $h_{kv}=h_{q}/2$。

Table 6: Hyperparameters of the RoboJEPA predictor at eight scales (zero-init view bias). Only depth N layers $N_{\text{layers}}$, width $D$, and head counts (h q, h k ​ v) $(h_{q},h_{kv})$ change across scales; everything else (encoder, factorised RoPE on (t, i, j, v) $(t,i,j,v)$, post-RoPE full-head-width view bias, padded action / state widths, local temporal window W t $W_{t}$, attention kernel, output projection, training loss) is held fixed. Predictor parameter counts are approximate D 2 ​ N layers $D^{2}N_{\text{layers}}$ -dominated estimates.

表 6：八个规模下 RoboJEPA 预测器超参（视角偏置零初始化）。跨规模只改深度 $N_{\text{layers}}$、宽度 $D$ 与头数 $(h_{q},h_{kv})$；其余（编码器、$(t,i,j,v)$ 因子化 RoPE、RoPE 后全头宽视角偏置、填充动作/状态宽度、局部时间窗 $W_{t}$、注意力核、输出投影、训练损失）固定。预测器参数量约为由 $D^{2}N_{\text{layers}}$ 主导的估计。

Appendix B Data: Additional Details

附录 B 数据：补充细节

Expands Section 2.2. Heterogeneous robot data become a single token stream by being normalized onto a common temporal grid (Section B.6), a common padded action / state width (Section B.7), and a common semantic camera vocabulary (Section B.9). The on-disk format (Section B.3) and the non-trivial per-dataset conversions needed to reach it (Section B.4) make this routing reproducible, the view-mixed-rank mixture (Section B.5) fixes the sampling, and the full per-dataset breakdown lives in Section B.2.

扩展第 2.2 节。异构机器人数据经归一化到共同时间网格（B.6）、共同填充动作/状态宽度（B.7）与共同语义相机词表（B.9）后，成为单一词元流。磁盘格式（B.3）与到达该格式所需的非平凡逐数据集转换（B.4）使路由可复现；视角混合秩混合（B.5）固定采样；完整分数据集分解见 B.2。

B.1 Dataset mixture and unification

B.1 数据集混合与统一

Actions.

动作。

For most datasets, the per-step action is the difference between consecutive states (which we refer to as the state-delta action): whether it is an end-effector (EEF) absolute pose, absolute joint angles, a commanded motion, or any combination of those for multiarm or wheeled robots. The same demonstration can often be described through several such parameterisations at once (for example joint angle control or EEF pose control), and each one shows the world model the same behavior from a different control perspective, acting as an action-level augmentation. We exploit this: we found it beneficial to train on different action parameterisations of the same embodiment whenever they are available. For example, with the DROID dataset arm we cannot only train on state-delta of the end-effector but also on the velocities of the end-effector or joint angles. Early in the work, we found that this improves the world model at a fixed training compute budget.

对多数数据集，逐步动作是相邻状态之差（我们称为状态差分动作）：不论是末端（EEF）绝对位姿、绝对关节角、指令运动，还是多臂/轮式机器人上的任意组合。同一演示常可同时用多种参数化描述（如关节角控制或 EEF 位姿控制），每一种从不同控制视角展示同一行为，相当于动作级增强。我们利用这一点：只要可得，就在同一本体的不同动作参数化上训练。例如对 DROID 机械臂，不仅可以训末端状态差分，也可训末端速度或关节角。工作早期发现，在固定训练算力下这能改善世界模型。

In addition to that, different embodiments provide a much richer potential learning signal for world models since they expose manipulation through different action spaces, each following its own unique dynamics and a set of constraints. Therefore, the world model has more potential to learn the underlying physics and dynamics of the world than with one embodiment. We treat each (platform, action-space) pair as a distinct embodiment id, giving | ℰ | = 26 $|\mathcal{E}|{=}26$ over the $12$ physical platforms (26 = 12 $26=12$ default state-delta ids + 14 $+14$ extra splits; full enumeration in Table 5).

此外，不同本体通过各自动作空间暴露操作，各有独特动力学与约束集，为世界模型提供更丰富的潜在学习信号，从而比单一本体更有机会学到底层物理与动力学。我们将每个（平台，动作空间）对视为不同本体 id，在 $12$ 个物理平台上得到 $|\mathcal{E}|{=}26$（$26=12$ 个默认状态差分 id + $14$ 个额外拆分；完整枚举见表 5）。

We use a lean per-embodiment linear projection of actions to facilitate learning a shared latent action tokenization within the world model. The dimensionality of the actions also varies. For the single-arm end-effector delta poses, we use seven dimensions that represent a $6$ D translation+rotation and gripper closure. For dual arm platforms (e.g. AgileX in RoboMind or AgiBot World), this gets doubled since we have to control two arms independently. For wheeled platforms, the action space is extended by the commanded actions for the wheeled navigation - for example the RoboCasa platform reserves four actions for navigation atop the standard $7$ D EEF control and 1X reserves $2$ D for navigation (see more details in Tables 7 and B.7).

我们用精简的按本体线性动作投影，以利于在世界模型内学习共享潜动作词元化。动作维数也因平台而异。单臂末端差分位姿用七维表示 $6$D 平移+旋转与夹爪闭合；双臂平台（如 RoboMind 的 AgileX 或 AgiBot World）则加倍，因需独立控制两臂。轮式平台再扩展导航指令动作——例如 RoboCasa 在标准 $7$D EEF 控制之上预留四维导航，1X 预留 $2$D 导航（详见附表 7 与 B.7）。

Unification and sampling.

统一与采样。

To facilitate efficient training, we convert all datasets into the unified data format. Since datasets are collected with different sensor frequencies, loading continuous chunks of data would often lead to an overly short horizon for prediction. Additionally, different datasets have different motion velocities due to differences in data collection mechnisms – often different teleoperated platforms scale human motion to a faster or slower motion of the robot. Therefore, we employ different frame sampling frequencies for different datasets which we report in Table 7 (see Section B.6 for details). Each frame is processed via the image embedding of the V-JEPA 2.1 recipe (Mur-Labadia et al., 2026), whose features performed strongly in policy-based evaluations of latent diffusion world models (Nilaksh et al., 2026) on Bridge V2 (Walke et al., 2023). To correctly represent the total action between sampled frames, we either use the total measured state delta action or the concatenated chunk of commanded actions between sampled frames when working with commanded actions. The world model does not have a well-defined notion of modeling frequency. Instead, we simply let the world model integrate the total motion represented by the action into the next predicted latent state.

为高效训练，我们将全部数据集转为统一数据格式。由于采集传感器频率不同，加载连续数据块常导致预测地平过短；不同数据集因采集机制差异，运动速度也不同——遥操作平台常把人类运动缩放到更快或更慢的机器人运动。因此我们对不同数据集采用不同帧采样频率，见表 7（细节见 B.6）。每帧按 V-JEPA 2.1 配方（Mur-Labadia 等，2026）做图像嵌入；其特征在 Bridge V2（Walke 等，2023）上潜扩散世界模型的策略评估中表现强（Nilaksh 等，2026）。为正确表示采样帧之间的总动作，我们对测量到的状态差分取总和，或在指令动作设定下把采样帧之间的指令动作块拼接。世界模型没有定义良好的建模频率概念；我们只是让它把动作所表示的总运动积分进下一预测潜状态。

B.2 Per-dataset breakdown

B.2 分数据集分解

Table 1 collapses the OXE, AgiBot World, and RoboMind sub-corpora into single rows. Table 7 gives the full per-dataset breakdown with resolutions and source FPS.

表 1 把 OXE、AgiBot World、RoboMind 子语料各压成一行。表 7 给出含分辨率与源 FPS 的完整分数据集分解。

Table 7: Full per-dataset breakdown of the $23$ -dataset, $26$ -embodiment mixture. “Act. hrs” counts only frames with synchronized proprioception. The compressed main-text view is Table 1; per-rank sampling weights are in Table 8.

表 7：$23$ 数据集、$26$ 本体混合的完整分数据集分解。「Act. hrs」只计有同步本体传感的帧。正文压缩视图见表 1；按秩采样权重见表 8。

B.3 Standardised on-disk format

B.3 标准化磁盘格式

Every source dataset is re-packaged into a single object-storage layout so that the dataloader does not need per-source branching beyond a thin adapter that emits the standard tuple consumed by the predictor. Each trajectory exposes three groups: an MP4-encoded camera stream per view (cameras/), HDF5 datasets holding the native proprioceptive vector (trajectory/observation/robot_state/), and, when the source ships a controller stream, raw controller commands (trajectory/action/). Episode-level metadata (e.g. DROID building / labs fields used by the negative filter that drops setup-room recordings) is recorded in a per-dataset aggregated_metadata.json; a per-dataset index maps episode ids to S3 paths, episode lengths, and per-camera availability so the sampler operates on this index without touching object storage. Per-clip sampling reads T = 16 $T{=}16$ frames at the world-model rate (Section B.6) and $V$ cameras (Section B.9); intrinsics, extrinsics, and uncalibrated metadata are loaded as NaN-filled placeholders and discarded by the predictor in this configuration.

每个源数据集被重新打包到统一对象存储布局，使数据加载器除一层薄适配器外无需按源分支；适配器发出预测器消费的标准元组。每条轨迹暴露三组：每视角一路 MP4 编码相机流（cameras/）、保存原生本体向量的 HDF5（trajectory/observation/robot_state/），以及源若提供控制器流时的原始控制器指令（trajectory/action/）。回合级元数据（如 DROID 用于丢弃布置间录制的 building/labs 负过滤字段）记在逐数据集 aggregated_metadata.json；逐数据集索引把回合 id 映到 S3 路径、回合长度与每相机可用性，使采样器无需触碰对象存储即可在索引上工作。逐片段采样以世界模型速率（B.6）读 $T{=}16$ 帧与 $V$ 路相机（B.9）；内外参与未标定元数据以填 NaN 占位加载，本配置下由预测器丢弃。

B.4 Per-dataset postprocessing

B.4 分数据集后处理

Beyond the common repackaging of Section B.3, several datasets require non-trivial numerical, geometric, or video conversions before they can enter the shared token stream. We describe these here grouped by the kind of transformation; the trivial relabelling of fields onto the common schema is omitted.

在 B.3 的共同重打包之外，若干数据集在进入共享词元流前还需非平凡的数值、几何或视频转换。此处按变换类型分组描述；字段到共同模式的平凡重标注略去。

Video decoding and re-encoding.

视频解码与重编码。

Every source is emitted as one MP4 per view at its native frame rate (Table 7). Most OXE (Open X-Embodiment Collaboration et al., 2023) sources ship RLDS/TFDS records with per-step RGB arrays, which we re-encode to MP4 (e.g. $1$ fps for RoboNet (Dasari et al., 2019), $5$ fps for Bridge (Ebert et al., 2022) /MT-Opt (Kalashnikov et al., 2021) /Language-Table (Lynch et al., 2023), $10$ fps for RT-1 (Brohan et al., 2023) /QT-Opt (Kalashnikov et al., 2018) /BC-Z (Jang et al., 2022) /FMB (Luo et al., 2024)). RoboMind (Wu et al., 2024b) stores each frame as a JPEG byte-string inside HDF5: we decode every frame individually, falling back to a raw-buffer reshape (720 × 1280 $720{\times}1280$ or 480 × 640 $480{\times}640$) when the JPEG header is missing, and correct the channel order from BGR to RGB for the real Franka streams (the simulated streams are already RGB). For DROID (Khazatsky et al., 2024) we discard the stereo/SVO recordings and keep the rectified monocular left, right, and wrist MP4 streams. For sources with multiple nominal cameras we detect and drop all-zero (absent) camera streams so they are not written as empty views (e.g. the unused auxiliary cameras in Bridge V2 (Walke et al., 2023)).

每个源按原生帧率（表 7）为每视角写出一个 MP4。多数 OXE（Open X-Embodiment Collaboration 等，2023）源以含逐步 RGB 数组的 RLDS/TFDS 记录交付，我们重编码为 MP4（例如 RoboNet $1$ fps；Bridge/MT-Opt/Language-Table $5$ fps；RT-1/QT-Opt/BC-Z/FMB $10$ fps）。RoboMind 在 HDF5 内以 JPEG 字节串存每帧：我们逐帧解码，缺 JPEG 头时回退为原始缓冲重排形（$720{\times}1280$ 或 $480{\times}640$），并对真机 Franka 流把通道序从 BGR 改为 RGB（仿真流已是 RGB）。对 DROID 丢弃立体/SVO 录制，保留校正后的单目左、右、腕部 MP4。对名义多相机源，检测并丢弃全零（缺失）相机流，以免写成空视角（如 Bridge V2 中未用的辅助相机）。

Forward kinematics for joint-only sources.

仅关节源的正向运动学。

Roboset (Bharadhwaj et al., 2024) and LeRobot (Cadene et al., 2024) record only joint encoders, with no end-effector pose. We reconstruct the end-effector Cartesian pose with a differentiable forward-kinematics model of the corresponding arm (a Franka Panda for Roboset, the SO-101 URDF for LeRobot), applied to both the state trajectory and, for Roboset, the commanded-joint trajectory to recover a Cartesian action. LeRobot joint encoders are first converted from degrees to radians. For Roboset we additionally apply constant per-joint offsets (a + π / 2 $+\pi/2$ shift on the wrist joints of both state and action, and a − π / 2 $-\pi/2$ shift on one action joint) so that the joint zero convention matches the URDF used by the kinematics model.

Roboset 与 LeRobot 只记录关节编码器、无末端位姿。我们用对应臂的可微正向运动学模型重建末端笛卡尔位姿（Roboset 用 Franka Panda，LeRobot 用 SO-101 URDF），同时作用于状态轨迹，并对 Roboset 的指令关节轨迹作用以恢复笛卡尔动作。LeRobot 关节编码器先从度转为弧度。对 Roboset 还施加常值关节偏移（状态与动作腕关节各 $+\pi/2$，某一动作关节 $-\pi/2$），使关节零位约定与运动学模型所用 URDF 一致。

Orientation representation.

朝向表示。

Sources that store the end-effector orientation as a quaternion (RT-1, QT-Opt, MT-Opt, FMB, the simulated RoboMind Franka split) and the forward-kinematics outputs above are all converted to an x ​ y ​ z $xyz$ Euler parameterisation, so that the entire mixture shares a single $3$ -D translation + $+$ $3$ -D Euler orientation convention for end-effector poses.

以四元数存储末端朝向的源（RT-1、QT-Opt、MT-Opt、FMB、仿真 RoboMind Franka 拆分）以及上述正向运动学输出，全部转为 $xyz$ 欧拉参数化，使整个混合对末端位姿共享单一的 $3$D 平移 + $3$D 欧拉朝向约定。

Gripper normalisation.

夹爪归一化。

Gripper signals are placed on a common [ 0, 1 ] $[0,1]$ closedness scale. Roboset state grippers are divided by their observed open width ($0.8305$) and clipped to [ 0, 1 ] $[0,1]$, and its command grippers are affinely mapped from [ − 1, 1 ] $[-1,1]$ to [ 0, 1 ] $[0,1]$. WidowX sources (Bridge and Bridge V2) invert and clip the raw gripper signal, g ← 1 − clip ⁡ (g, 0, 1) $g\leftarrow 1-\mathrm{clip}(g,0,1)$, so that larger values consistently denote a more closed gripper.

夹爪信号放到共同的 $[0,1]$ 闭合度尺度。Roboset 状态夹爪除以观测到的张开宽度（$0.8305$）并裁到 $[0,1]$；其指令夹爪从 $[-1,1]$ 仿射映到 $[0,1]$。WidowX 源（Bridge 与 Bridge V2）反转并裁剪原始夹爪信号：$g\leftarrow 1-\mathrm{clip}(g,0,1)$，使更大值始终表示更闭合。

Multi-arm and humanoid assembly.

多臂与人形组装。

Bimanual and humanoid embodiments are assembled into a single per-step vector. For the RoboMind AgileX dual-arm platform we concatenate the left and right end-effector poses and the left and right arm joints, taking each arm’s last joint as its gripper. Bridge (v1) actions, which are stored as separate translation and rotation-delta fields, are concatenated into a single delta-pose action vector, and BC-Z end-effector poses are assembled from separately stored translation and axis-angle observations.

双臂与人形本体组装为单一逐步向量。对 RoboMind AgileX 双臂平台，我们拼接左右末端位姿与左右臂关节，并以各臂末关节为夹爪。Bridge（v1）动作以分开的平移与旋转差分字段存储，拼成单一差分位姿动作向量；BC-Z 末端位姿由分开存储的平移与轴角观测组装。

B.5 View-mixed-rank training: mixture composition

B.5 视角混合秩训练：混合组成

The 1B run uses W ddp = 256 $W_{\mathrm{ddp}}{=}256$ data-parallel ranks (32 nodes × $\times$ 8 H100s), partitioned into two equally sized view-groups:

1B 实验使用 $W_{\mathrm{ddp}}{=}256$ 个数据并行秩（32 节点 $\times$ 8 块 H100），均分为两个等大视角组：

• Two-view group (ranks 0 ​ … ​ 127 $0{\dots}127$): each rank loads V g = 2 $V_{g}{=}2$ camera views per clip with B g = 2 $B_{g}{=}2$ clips per microbatch and gradient accumulation K g = 1 $K_{g}{=}1$, contributing $4$ view-clips per rank per step, i.e. $512$ view-clips group-wide.

• 双视角组（秩 $0{\dots}127$）：每秩每片段加载 $V_{g}{=}2$ 路相机，微批 $B_{g}{=}2$ 片段、梯度累积 $K_{g}{=}1$，每秩每步贡献 $4$ 个视角–片段，即全组 $512$ 个视角–片段。

One-view group (ranks 128 ​ … ​ 255 $128{\dots}255$): each rank loads V g = 1 $V_{g}{=}1$ view per clip with B g = 2 $B_{g}{=}2$, K g = 1 $K_{g}{=}1$, contributing $2$ view-clips per rank per step, i.e. $256$ view-clips group-wide.

• 单视角组（秩 $128{\dots}255$）：每秩每片段加载 $V_{g}{=}1$ 路，微批 $B_{g}{=}2$、$K_{g}{=}1$，每秩每步贡献 $2$ 个视角–片段，即全组 $256$ 个视角–片段。

This yields the effective per-step batch of $768$ view-clips ($512$ clips) derived in Section C.1 (Equation 13). The two groups share predictor parameters and step the optimizer in lockstep.

由此得到第 C.1 节（式 13）导出的有效逐步批量：$768$ 个视角–片段（$512$ 个片段）。两组共享预测器参数，并同步步进优化器。

Table 8 lists the per-(dataset, action-source) sampling weights for both groups in a single table, together with the per-row total. The top block lists datasets that ship two reliably informative external cameras and feed both groups; the lower block lists single-camera sources that feed only the 1-view group. For datasets appearing in both groups, the 1-view rank uses a contracted camera regex selected so that the rank still sees a representative external view (e.g. left/right exterior for DROID excludes the wrist; AgiBot World Robotiq is restricted to its head-mounted RGB stream).

表 8 在一张表中列出两组的逐（数据集，动作源）采样权重及每行合计。上块为提供两路可靠外置相机、同时喂入两组的数据集；下块为只喂入单视角组的单相机源。对同时出现在两组的数据集，单视角秩使用收缩后的相机正则，使该秩仍看到代表性外置视角（例如 DROID 的左/右外置排除腕部；AgiBot World Robotiq 限制为头戴 RGB 流）。

Table 8: Sampling weights for the 1-view and 2-view groups of the 1B run. The top block lists datasets that ship two reliably informative external cameras and feed both groups; the lower block lists single-camera sources that feed only the 1-view group. DROID and RoboCasa appear under multiple (platform, action-source) pairs — each one is a distinct embodiment id with its own per-embodiment input projection (Section B.7) — but all variants share the same underlying source episodes.

表 8：1B 实验单视角与双视角组的采样权重。上块为两路可靠外置相机、同时喂入两组的数据集；下块为只喂单视角组的单相机源。DROID 与 RoboCasa 以多个（平台，动作源）对出现——每一对是不同本体 id，各有按本体输入投影（B.7）——但所有变体共享同一底层源回合。

Effective per-step mixture.

有效逐步混合。

At the clip level the split is 50 / 50 $50/50$; at the token-with-loss level the run sees 2 3 $\tfrac{2}{3}$ of the signal at V = 2 $V{=}2$ (the V = 2 $V{=}2$ clips contribute two view-clips each). Both view counts therefore appear in every optimizer step, so the per-block view conditioning (Equation 9) and the factorised RoPE of Section 2.1 are jointly exposed to both regimes rather than separated by curriculum.

在片段层面划分为 $50/50$；在带损失的词元层面，运行看到 $\tfrac{2}{3}$ 的信号来自 $V{=}2$（因 $V{=}2$ 片段各贡献两个视角–片段）。因此每个优化器步都同时见到两种视角数，逐块视角条件（式 9）与第 2.1 节因子化 RoPE 被联合暴露于两种体制，而非靠课程分隔。

B.6 Frame rate, decimation and the per-frame mask

B.6 帧率、抽稀与逐帧掩码

Every dataset is normalized to a target world-model rate f wm $f_{\text{wm}}$ that determines the temporal stride between consecutive frames in a clip. Source FPS varies by two orders of magnitude across the mixture (Table 7): $1$ Hz for OXE RoboNet, $5$ – $10$ Hz for most OXE sources, $20$ – $30$ Hz for RoboCasa / RoboMind / AgiBot / LeRobot, $60$ Hz for DROID. The loader decimates by the integer factor c = ⌈ f raw / f wm ⌉ $c\;=\;\lceil f_{\text{raw}}/f_{\text{wm}}\rceil$ and indexes frames as np.arange(start_frame, end_frame, c), so one predictor step always corresponds to a fixed 1 / f wm $1/f_{\text{wm}}$ of robot wall-clock regardless of source FPS. Trajectories whose effective length falls below T = 16 $T{=}16$ are right-padded with zero frames, and a per-frame mask

每个数据集归一化到目标世界模型速率 $f_{\text{wm}}$，决定片段内相邻帧的时间跨度。源 FPS 在混合中跨两个数量级（表 7）：OXE RoboNet 为 $1$ Hz，多数 OXE 源为 $5$–$10$ Hz，RoboCasa/RoboMind/AgiBot/LeRobot 为 $20$–$30$ Hz，DROID 为 $60$ Hz。加载器用整数因子 $c\;=\;\lceil f_{\text{raw}}/f_{\text{wm}}\rceil$ 抽稀，并以 np.arange(start_frame, end_frame, c) 取帧，使无论源 FPS 如何，预测器一步始终对应固定的 $1/f_{\text{wm}}$ 机器人墙钟时间。有效长度不足 $T{=}16$ 的轨迹右侧用零帧填充，并使用逐帧掩码

Rate bridge f wm → f ctrl $f_{\text{wm}}\!\to\!f_{\text{ctrl}}$ at deployment.

部署时的速率桥接 $f_{\text{wm}}\!\to\!f_{\text{ctrl}}$。

At deployment the Franka controller of Section 2.2 runs at f ctrl = 10 $f_{\text{ctrl}}{=}10$ Hz while the world model operates at f wm = 5 $f_{\text{wm}}{=}5$ Hz. The planner replans on every other controller tick and the $7$ -D end-effector delta returned by the planner is split into f ctrl / f wm = 2 $f_{\text{ctrl}}/f_{\text{wm}}{=}2$ equal sub-steps emitted back-to-back; this keeps training and deployment aligned on the same $200$ ms predictor stride and isolates the controller from the world-model rate.

部署时第 2.2 节 Franka 控制器以 $f_{\text{ctrl}}{=}10$ Hz 运行，世界模型以 $f_{\text{wm}}{=}5$ Hz 运行。规划器每隔一个控制器节拍重规划，并将返回的 $7$ 维末端差分拆成 $f_{\text{ctrl}}/f_{\text{wm}}{=}2$ 个相等子步背靠背发出；从而使训练与部署对齐同一 $200$ ms 预测器步长，并把控制器与世界模型速率隔离。

B.7 Action conditioning details

B.7 动作条件细节

The mixture exposes two complementary action-vector pipelines per (dataset, action-source) pair. Each pair is a distinct embodiment id from the | ℰ | = 26 $|\mathcal{E}|{=}26$ registry consumed by the predictor’s per-embodiment input projection (Section 2.1); duplicates of the same physical platform under different action sources (e.g. the six DROID embodiments listed in Table 8) share encoder inputs but learn distinct W a (e) $W_{a}^{(e)}$, W s (e) $W_{s}^{(e)}$ slices.