Learning Interactive Real-World Simulators

学习可交互的真实世界模拟器

Sherry Yang    Yilun Du  Seyed Kamyar Seyed Ghasemipour  Jonathan Tompson    Leslie Kaelbling  Dale Schuurmans  Pieter Abbeel Affiliation: UC Berkeley  Google DeepMind  MIT   University of Alberta sherryy@{berkeley.edu, google.com}

Abstract

摘要

Generative models trained on internet data have revolutionized how text, image, and video content can be created. Perhaps the next milestone for generative models is to simulate realistic experience in response to actions taken by humans, robots, and other interactive agents. Applications of a real-world simulator range from controllable content creation in games and movies, to training embodied agents purely in simulation that can be directly deployed in the real world. We explore the possibility of learning a universal simulator (UniSim) of real-world interaction through generative modeling. We first make the important observation that natural datasets available for learning a real-world simulator are often rich along different dimensions (e.g., abundant objects in image data, densely sampled actions in robotics data, and diverse movements in navigation data). With careful orchestration of diverse datasets, each providing a different aspect of the overall experience, we can simulate the visual outcome of both high-level instructions such as “open the drawer” and low-level controls such as “move by $\Delta x,\Delta y$” from otherwise static scenes and objects. We use the simulator to train both high-level vision-language policies and low-level reinforcement learning policies, each of which can be deployed in the real world in zero shot after training purely in simulation. We also show that other types of intelligence such as video captioning models can benefit from training with simulated experience, opening up even wider applications. Video demos can be found at https://universal-simulator.github.io.

在互联网数据上训练的生成模型，已经彻底改变了文本、图像与视频内容的创作方式。生成模型的下一个里程碑，或许是根据人类、机器人及其他交互式智能体采取的动作，模拟出逼真的体验。真实世界模拟器的应用，从游戏与电影中的可控内容创作，到完全在仿真中训练、可直接部署到真实世界的具身智能体，不一而足。我们探索通过生成建模学习真实世界交互的通用模拟器（UniSim）的可能性。我们首先指出：可用于学习真实世界模拟器的自然数据集，往往在不同维度上各自丰富（例如图像数据中的大量物体、机器人数据中的稠密采样动作、导航数据中的多样运动）。通过精心编排这些多样数据集——各自提供整体体验的不同侧面——我们可以从原本静态的场景与物体出发，既模拟「打开抽屉」这类高层指令的视觉结果，也模拟「移动 $\Delta x,\Delta y$」这类低层控制的视觉结果。我们用该模拟器训练高层视觉–语言策略与低层强化学习策略；二者均仅在仿真中训练，即可零样本部署到真实世界。我们还表明，视频描述等其他类型智能也能从模拟体验训练中受益，从而打开更广的应用空间。视频演示见 https://universal-simulator.github.io。

1 Introduction

1 引言

Generative models trained on internet data can now produce highly realistic text (OpenAI, 2023),image (Ramesh et al., 2022), and video (Ho et al., 2022a). Perhaps the ultimate goal of generative models is to be able to simulate the visual effects of a wide variety of actions, from how cars are driven on a street to how furniture and meals are prepared. With a real-world simulator, humans can “interact” with diverse scenes and objects, robots can learn from simulated experience without risking physical damage, and a vast amount of “real-world” like data can be simulated to train other types of machine intelligence.

在互联网数据上训练的生成模型，现已能生成高度逼真的文本（OpenAI, 2023）、图像（Ramesh et al., 2022）与视频（Ho et al., 2022a）。生成模型的终极目标，或许是能模拟多种多样动作的视觉效果——从街上汽车如何行驶，到家具与餐食如何被准备。有了真实世界模拟器，人类可以与多样场景与物体「交互」，机器人可从模拟体验中学习而不冒物理损坏风险，并可模拟出大量「类真实世界」数据以训练其他类型的机器智能。

One roadblock to building this simulator lies in the datasets — different datasets cover different information that have to be brought together to simulate realistic experience. For instance, paired text-image data from the internet contains rich scenes and objects but little movement (Schuhmann et al., 2022; Zhai et al., 2022), video captioning and question answering data contain rich high-level descriptions but little low-level movement detail (Xu et al., 2016; Krishna et al., 2017), human activity data contains rich human action but little mechanical motion (Miech et al., 2019; Grauman et al., 2022), and robotics data contains rich robot action but are limited in quantity (Dasari et al., 2019; Mandlekar et al., 2018). Since different datasets are curated by different industrial or research communities for different purposes, divergence in information is natural and hard to overcome, posing difficulties to a real-world simulator that seeks to capture all visual aspects of the world.

构建这一模拟器的障碍之一在于数据集——不同数据集覆盖不同信息，必须拼到一起才能模拟逼真体验。例如，互联网配对文图数据含有丰富场景与物体，但几乎没有运动（Schuhmann et al., 2022; Zhai et al., 2022）；视频描述与问答数据含有丰富高层描述，但缺少低层运动细节（Xu et al., 2016; Krishna et al., 2017）；人类活动数据含有丰富人类动作，但机械运动很少（Miech et al., 2019; Grauman et al., 2022）；机器人数据含有丰富机器人动作，但数量有限（Dasari et al., 2019; Mandlekar et al., 2018）。由于不同数据集由不同工业或研究社区为不同目的整理，信息发散是自然且难以克服的，这给试图捕捉世界全部视觉方面的真实世界模拟器带来困难。

In this work, we propose to combine a wealth of data in a conditional video generation framework to instantiate a universal simulator (UniSim)[^1]. Under a unified action-in-video-out interface, the simulator enables rich interaction through fine-grained motion control of otherwise static scenes and objects. To support long-horizon repeated interactions, we formulate the simulator as an observation prediction model that can be rolled out autoregressively to support consistent simulation across video generation boundaries.

本工作提出：在条件视频生成框架中组合大量数据，实例化一个通用模拟器（UniSim）[^1]。在统一的「动作入–视频出」接口下，模拟器通过对原本静态的场景与物体做细粒度运动控制，实现丰富交互。为支持长时程反复交互，我们把模拟器表述为可自回归展开的观测预测模型，从而在视频生成边界之间保持一致仿真。

While the potential applications of the simulator are broad, we demonstrate three specific use cases. We first show how the simulator enables a vision-language policy to perform long-horizon goal-conditioned tasks through hindsight relabeling of simulated experience (Andrychowicz et al., 2017). In addition to learning high-level vision-language policies, we illustrate how the simulator can enable learning low-level control policies by leveraging model-based reinforcement learning (RL) (Sutton, 1988). Both the high-level vision-language policy and the low-level control policy, while trained purely in simulation, can generalize to real robot settings. This is enabled by using the simulator that is nearly visually indistinguishable from the real world, achieving one step towards bridging the sim-to-real gap in embodied learning (Rusu et al., 2017). Furthermore, we can simulate rare events where data collection is expensive or dangerous (e.g., crashes in self-driving cars). Such simulated videos can then be used to improve other machine intelligence such as rare event detectors, suggesting broad applications of UniSim beyond embodied learning. The main contributions can be summarized as follows:

尽管模拟器的潜在应用很广，我们展示三种具体用例。首先说明模拟器如何通过对模拟体验做事后重标注（hindsight relabeling）（Andrychowicz et al., 2017），使视觉–语言策略完成长时程目标条件任务。除高层视觉–语言策略外，我们还说明模拟器如何借助基于模型的强化学习（RL）（Sutton, 1988）学习低层控制策略。高层视觉–语言策略与低层控制策略虽仅在仿真中训练，却都能泛化到真实机器人设定。这得益于模拟器在视觉上几乎与真实世界难以区分，朝弥合具身学习中的仿真到真实鸿沟迈出一步（Rusu et al., 2017）。此外，我们可以模拟数据采集昂贵或危险的稀有事件（例如自动驾驶中的碰撞）。此类模拟视频可用来改进稀有事件检测器等其他机器智能，表明 UniSim 在具身学习之外仍有广阔应用。主要贡献概括如下：

• • We take the first step toward building a universal simulator of real-world interaction by combining diverse datasets rich in along different dimensions — e.g., objects, scenes, actions, motions, language, and motor controls — in a unified action-in-video-out generative framework.
• • We formulate the action-in-video-out framework as an observation prediction model conditioned on finite history and parametrized by a video diffusion model. We illustrate that the observation prediction model can be rolled out autoregressively to obtain consistent and long-horizon videos.
• • We illustrate how the simulator can enable both high-level language policies, low-level control policies, and video captioning models to generalize to the real world when trained purely in simulation, thereby bridging the sim-to-real gap.

• • 我们迈出构建真实世界交互通用模拟器的第一步：在统一的「动作入–视频出」生成框架中，组合在物体、场景、动作、运动、语言与电机控制等不同维度上各自丰富的多样数据集。
• • 我们将「动作入–视频出」框架表述为以有限历史为条件、由视频扩散模型参数化的观测预测模型，并说明该模型可自回归展开以得到一致的长时程视频。
• • 我们说明模拟器如何使高层语言策略、低层控制策略与视频描述模型在仅于仿真中训练时仍能泛化到真实世界，从而弥合仿真到真实的鸿沟。

Figure 1: A universal simulator (UniSim). The simulator of the real-world learns from broad data with diverse information including objects, scenes, human activities, motions in navigation and manipulation, panorama scans, and simulations and renderings.

图 1：通用模拟器（UniSim）。真实世界模拟器从含有多样信息的广泛数据中学习，包括物体、场景、人类活动、导航与操作中的运动、全景扫描，以及仿真与渲染。

2 Learning an Interactive Real-World Simulator

2 学习可交互的真实世界模拟器

Figure 2: Training and inference of UniSim. UniSim is a video diffusion model trained to predict the next (variable length) set of observation frames ($o_{t}$) given observations from the past (e.g., $o_{t-1}$) and action input $a_{t-1}$. UniSim can handle temporally extended actions in various modalities such as motor controls ($\Delta x_{1},\Delta\omega_{1},\Delta x_{2},...$), language descriptions (“wipe table”), and actions extracted from camera motions and other sources. Each dotted arrow indicates concatenating the initial noise sample for the next video segment with the previous frame.

图 2：UniSim 的训练与推理。UniSim 是一个视频扩散模型，在给定过去观测（例如 $o_{t-1}$）与动作输入 $a_{t-1}$ 时，预测下一段（可变长度）观测帧集合（$o_{t}$）。UniSim 可处理多种模态的时序扩展动作，例如电机控制（$\Delta x_{1},\Delta\omega_{1},\Delta x_{2},...$）、语言描述（「擦桌子」）以及从相机运动等来源提取的动作。每条虚线箭头表示把下一段视频的初始噪声样本与上一帧拼接。

We define a simulator of the real world as a model that, given some state of the world (e.g., an image frame), can take in some action as input, and produce the visual consequence of the action (in the form of a video) as output. Learning such a simulator is hard, since different actions have different formats (e.g.,language instructions, robot controls, camera movements) and videos have different frame rates. Nevertheless, we propose specific strategies for processing each type of data to unify the action space and align videos of variable lengths to actions in Section 2.1. With a unified action space, we then train an action-conditioned video generation model to fuse information across datasets through a universal interface relating actions to videos in Section 2.2.

我们将真实世界模拟器定义为：给定世界的某种状态（例如一帧图像），可接受某种动作作为输入，并产出该动作的视觉后果（以视频形式）作为输出的模型。学习这样的模拟器很难，因为不同动作格式不同（例如语言指令、机器人控制、相机运动），视频帧率也不同。尽管如此，我们在第 2.1 节提出处理各类数据的具体策略，以统一动作空间并把可变长度视频与动作对齐。有了统一动作空间后，我们在第 2.2 节训练动作条件视频生成模型，通过把动作与视频关联起来的通用接口，融合跨数据集的信息。

2.1 Orchestrating Diverse Datasets

2.1 编排多样数据集

Below, we highlight diverse information in different datasets and propose ways to process actions into a common format (see all datasets used to train UniSim in Appendix B).

下文强调不同数据集中的多样信息，并提出把动作处理成统一格式的方法（训练 UniSim 所用全部数据集见附录 B）。

• • Simulated execution and renderings. While annotating actions for real-world videos is expensive, simulation engines such as Habitat (Savva et al., 2019) can render a wide variety of actions. We use datasets previously collected from these simulators, i.e., Habitat object navigation with HM3D (Ramakrishnan et al., 2021) and Language Table Data from Lynch et al. (2023) to train UniSim. We extract text descriptions as actions when available. For simulated continuous control actions, we encode them via language embeddings and concatenate the text embeddings with discretized control values.
• • Real robot data. An increasing amount of video data of real-robot executions paired with task descriptions such as the Bridge Data (Ebert et al., 2021) and data that enabled RT-1 and RT-2 (Brohan et al., 2022) are becoming increasingly available. Despite low-level control actions often being different across robots, the task descriptions can serve as high-level actions in UniSim. We further include discretize continuous controls actions when available similar to simulated robotics data.
• • Human activity videos. Rich human activity data such as Ego4D (Grauman et al., 2022), EPIC-KITCHENS (Damen et al., 2018), and Something-Something V2 (Goyal et al., 2017) have been curated. Different from low-level robot controls, these activities are high-level actions that humans take to interact with the world. But these actions are often provided as labels for video classification or activity recognition tasks (Goyal et al., 2017). In this case, we convert the video labels into text actions. In addition, we subsample the videos to construct chunks of observations at a frame rate that captures meaningful actions.
• • Panorama scans. There exists a wealth of 3D scans such as Matterport3D (Chang et al., 2017). These static scans do not contain actions. We construct actions (e.g., turn left) by truncating panorama scans and utilize the information of camera poses between two images.
• • Internet text-image data. Paired text-image datasets such as LAION (Schuhmann et al., 2021) contain static images of a variety of objects without actions. However, the captions often contain motion information such as “a person walking”. To use image data in UniSim, we treat individual images as single-frame videos and image captions as actions.

• • 仿真执行与渲染。 为真实世界视频标注动作成本高昂，但 Habitat（Savva et al., 2019）等仿真引擎可渲染多种多样的动作。我们使用此前从这些模拟器收集的数据集，即带 HM3D 的 Habitat 物体导航（Ramakrishnan et al., 2021）以及 Lynch et al.（2023）的 Language Table 数据来训练 UniSim。有文本描述时将其提取为动作。对仿真中的连续控制动作，我们经语言嵌入编码，并把文本嵌入与离散化控制值拼接。
• • 真实机器人数据。 越来越多带任务描述的真实机器人执行视频数据变得可用，例如 Bridge Data（Ebert et al., 2021）以及支撑 RT-1 与 RT-2 的数据（Brohan et al., 2022）。尽管不同机器人的低层控制动作往往不同，任务描述可作为 UniSim 中的高层动作。有连续控制时，我们也类似仿真机器人数据那样进一步纳入离散化连续控制动作。
• • 人类活动视频。 已整理出 Ego4D（Grauman et al., 2022）、EPIC-KITCHENS（Damen et al., 2018）与 Something-Something V2（Goyal et al., 2017）等丰富人类活动数据。与低层机器人控制不同，这些活动是人类与世界交互的高层动作，但常作为视频分类或活动识别任务的标签提供（Goyal et al., 2017）。此时我们把视频标签转为文本动作。此外，我们对视频子采样，按能捕捉有意义动作的帧率构造观测块。
• • 全景扫描。 存在 Matterport3D（Chang et al., 2017）等大量三维扫描。这些静态扫描不含动作。我们通过截取全景扫描并利用两幅图像之间的相机位姿信息来构造动作（例如向左转）。
• • 互联网文图数据。 LAION（Schuhmann et al., 2021）等配对文图数据集包含多样物体的静态图像而无动作。但字幕常含运动信息，例如「一个人在走路」。为在 UniSim 中使用图像数据，我们把单张图像当作单帧视频，把图像字幕当作动作。

For each of these datasets, we process text tokens into continuous representations using T5 language model embeddings (Raffel et al., 2020) concatenated with low-level actions such as robot controls. This serves as the final unified action space of our simulator.

对上述每个数据集，我们用 T5 语言模型嵌入（Raffel et al., 2020）把文本词元处理为连续表示，并与机器人控制等低层动作拼接。这构成我们模拟器最终的统一动作空间。

2.2 Simulating Long-Horizon Interactions through Observation Prediction

2.2 通过观测预测模拟长时程交互

With observations from different environments that have been converted to videos, and actions of different formats that have been converted to continuous embeddings, we can formulate interactions with many real-world environments as interacting with a universal simulator. We then formulate the universal simulator as an observation prediction model that predicts observations conditioned on actions and previous observations as shown in Figure 2. We finally show that this observation prediction model can be parametrized using video diffusion.

在把不同环境中的观测转为视频、把不同格式的动作转为连续嵌入之后，我们可以把与许多真实世界环境的交互表述为与通用模拟器的交互。随后，我们把通用模拟器表述为以动作与先前观测为条件预测观测的观测预测模型，如图 2 所示。最后说明该观测预测模型可用视频扩散来参数化。

Simulating Real-World Interactions. We define an observation space $O$ and an action space $A$ which capture the videos and actions described in Section 2.1. At a specific interactive step $t$, an agent, having observed a set of history frames $h_{t-1}\in O$, decides on some temporally extended action $a_{t-1}\in A$, which can be resolved into a sequence of low-level robot commands to be executed in the real world. During the execution, the next set of video frames $o_{t}\in O$ are captured from the real world. The goal of a simulator is to predict $o_{t}$ from $h_{t-1}$ and $a_{t-1}$. We can formulate this prediction problem as learning an observation prediction model $p(o_{t}|h_{t-1},a_{t-1})$. While an ideal predictive model should condition on all information of the past, i.e., $(o_{0},a_{0}\ldots,a_{t-2},o_{t-1})$, through some recurrent state, we found conditioning on a finite set of frames (e.g., frames from the most recent interaction, $o_{t-1}$) greatly simplifies the modeling problem. To simulate long interactions, we can sample from the observation prediction model $p(o_{t}|h_{t-1},a_{t-1})$ autoregressively conditioned on the previously sampled observations. One advantage of this observation prediction model is that the simulator stays the same across all tasks and can be used in combination with any reward function, which can be separately learned. The learned reward function can then be used to optimize policies $\pi(a_{t}|h_{t})$ using existing decision making algorithms such as planning and RL, as we will illustrate in Section 4.1 and Section 4.2.

模拟真实世界交互。 我们定义观测空间 $O$ 与动作空间 $A$，以刻画第 2.1 节所述视频与动作。在特定交互步 $t$，智能体在观测到一组历史帧 $h_{t-1}\in O$ 后，决定某个时序扩展动作 $a_{t-1}\in A$，该动作可被解析为可在真实世界执行的一串低层机器人指令。执行期间，从真实世界捕获下一段视频帧 $o_{t}\in O$。模拟器的目标是从 $h_{t-1}$ 与 $a_{t-1}$ 预测 $o_{t}$。我们可将该预测问题表述为学习观测预测模型 $p(o_{t}|h_{t-1},a_{t-1})$。理想预测模型应条件于过去全部信息，即 $(o_{0},a_{0}\ldots,a_{t-2},o_{t-1})$，并通过某种循环状态实现；但我们发现，条件于有限帧集合（例如最近一次交互的帧 $o_{t-1}$）可大幅简化建模。为模拟长交互，可对观测预测模型 $p(o_{t}|h_{t-1},a_{t-1})$ 以先前采样的观测为条件做自回归采样。该观测预测模型的一个优点是：模拟器跨所有任务保持不变，并可与任意奖励函数组合使用，而奖励可单独学习。学到的奖励函数随后可用规划与 RL 等既有决策算法优化策略 $\pi(a_{t}|h_{t})$，我们将在第 4.1 与 4.2 节说明。

Parametrizing and Training the Simulator. We parametrize $p(o_{t}|h_{t-1},a_{t-1})$ using diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) as an instantiation of UniSim outlined in Figure 2. Specifically, the reverse process learns a denoising model $\epsilon_{\theta}(o_{t}^{(k)},k|h_{t-1},a_{t-1})$ that, conditioned on the history, generates the next observationfrom initial noise samples using $K$ denoising steps. In practice, we only use previous video frames and omit previous actions as history, and concatenate previous video frames with initial noise samples $o^{(K)}_{t}\sim\mathcal{N}(0,I)$ channelwise to serve as conditional inputs to the denoising model. To condition on an action $a_{t-1}$, we leverage classifier-free guidance (Ho & Salimans, 2022). The final $\overline{\mathcal{T}}(o_{t}|h_{t-1},a_{t-1})$ is parametrized by the variance schedule:

参数化并训练模拟器。 我们用扩散模型（Sohl-Dickstein et al., 2015; Ho et al., 2020）参数化 $p(o_{t}|h_{t-1},a_{t-1})$，作为图 2 所示 UniSim 的实例。具体地，反向过程学习去噪模型 $\epsilon_{\theta}(o_{t}^{(k)},k|h_{t-1},a_{t-1})$，在历史条件下用 $K$ 步去噪从初始噪声样本生成下一观测。实践中我们只用先前视频帧、省略先前动作作为历史，并把先前视频帧与初始噪声样本 $o^{(K)}_{t}\sim\mathcal{N}(0,I)$ 在通道维拼接，作为去噪模型的条件输入。为条件于动作 $a_{t-1}$，我们使用无分类器引导（Ho & Salimans, 2022）。最终 $\overline{\mathcal{T}}(o_{t}|h_{t-1},a_{t-1})$ 由方差日程参数化：

$$\epsilon_{\theta}(o_{t}^{(k)},k|h_{t-1},a_{t-1})=(1+\eta)\epsilon_{\theta}(o_{t}^{(k)},k|h_{t-1},a_{t-1})-\eta\epsilon_{\theta}(o_{t},k|h_{t-1}), \tag{1}$$

where $\eta$ controls action conditioning strength. With this parametrization, we train $\epsilon_{\theta}$ by minimizing

其中 $\eta$ 控制动作条件强度。在该参数化下，我们通过最小化下式训练 $\epsilon_{\theta}$

$$\mathcal{L}_{\text{MSE}}=\left\|\mathbf{\epsilon}-\epsilon_{\theta}\Big(\sqrt{1-\beta^{(k)}}o_{t}+\sqrt{\beta^{(k)}}\mathbf{\epsilon},\,k\Big|h_{t-1},a_{t-1}\Big)\right\|^{2},$$

where $\epsilon\sim\mathcal{N}(0,I)$, and $\beta^{(k)}\in\mathbb{R}$ are a set of $K$ different noise levels for each $k\in[1,K]$. Given the learned $\epsilon_{\theta}$, an observation $o_{t}$ can be generated by sampling from the initial distribution $o^{(K)}_{t}\sim\mathcal{N}(0,I)$ and iteratively denoising according to the following process for $k$ from $K$ to $0$

其中 $\epsilon\sim\mathcal{N}(0,I)$，$\beta^{(k)}\in\mathbb{R}$ 是每个 $k\in[1,K]$ 的一组 $K$ 个不同噪声水平。给定学到的 $\epsilon_{\theta}$，可从初始分布 $o^{(K)}_{t}\sim\mathcal{N}(0,I)$ 采样，再对 $k$ 从 $K$ 到 $0$ 按下述过程迭代去噪，从而生成观测 $o_{t}$

$$o^{(k-1)}_{t}=\alpha^{(k)}(o^{(k)}_{t}-\gamma^{(k)}\epsilon_{\theta}(o^{(k)}_{t},k|h_{t-1},a_{t-1}))+\xi,\quad\xi\sim\mathcal{N}\bigl(0,\sigma_{k}^{2}I\bigl), \tag{2}$$

where $\gamma^{(k)}$ is the denoising step size, $\alpha^{(k)}$ is a linear decay on the current denoised sample, and $\sigma_{k}$ is a time varying noise level that depends on $\alpha^{(k)}$ and $\beta^{(k)}$.

其中 $\gamma^{(k)}$ 是去噪步长，$\alpha^{(k)}$ 是对当前去噪样本的线性衰减，$\sigma_{k}$ 是依赖 $\alpha^{(k)}$ 与 $\beta^{(k)}$ 的时变噪声水平。

Architecture and Training. We use the video U-Net architecture (Ho et al., 2022b) to implement UniSim by employing interleaved temporal and spatial attention and convolution layers in both the downsampling and upsampling passes. For history conditioning, we replicate the conditioning frames at all future frame indices, and concatenate the conditioning frames with the noise sample for each of the future frame to serve as input to the U-Net. UniSim model has 5.6B parameters and requires 512 TPU-v3 and 20 days to train on all data. See more details in Appendix C.

结构与训练。 我们用视频 U-Net 结构（Ho et al., 2022b）实现 UniSim，在下采样与上采样通路中交错使用时间与空间注意力及卷积层。对历史条件，我们在所有未来帧索引处复制条件帧，并把条件帧与每个未来帧的噪声样本拼接后作为 U-Net 输入。UniSim 模型有 56 亿参数，在全部数据上训练需 512 块 TPU-v3、历时 20 天。更多细节见附录 C。

3 Simulating Real-World Interactions

3 模拟真实世界交互

We now demonstrate emulating real-world manipulation and navigation environments by simulating both action-rich and long-horizon interactions for both humans and robots.

我们现在展示：通过为人类与机器人模拟动作丰富且长时程的交互，来仿真真实世界操作与导航环境。

Figure 3: Action-rich simulations. UniSim can support manipulation actions such as “cut carrots”, “wash hands”, and “pickup bowl” from the same initial frame (top left) and other navigation actions.

图 3：动作丰富的仿真。 UniSim 可支持从同一初始帧（左上）出发的「切胡萝卜」「洗手」「拿起碗」等操作动作，以及其他导航动作。

3.1 Action-Rich, Long-Horizon, and Diverse Interactions

3.1 动作丰富、长时程与多样交互

Action-Rich Simulation. We first demonstrate action-rich interactions through natural language actions. Figure 3 shows simulation of human manipulation and navigation starting from the same initial observation (left-most column). We can instruct a person in the initial frame to perform various kitchen tasks (top left), press different switches (top right), or navigate scenes (bottom). The model only trained on generic internet data, without action-rich manipulation data such as EPIC-KITCHENS (Damen et al., 2018), fails to simulate action-rich manipulations (Appendix F).

动作丰富的仿真。 我们首先通过自然语言动作展示动作丰富的交互。图 3 展示从同一初始观测（最左列）出发的人类操作与导航仿真。我们可以指示初始帧中的人执行各种厨房任务（左上）、按不同开关（右上）或在场景中导航（下）。仅在通用互联网数据上训练、不含 EPIC-KITCHENS（Damen et al., 2018）等动作丰富操作数据的模型，无法模拟动作丰富的操作（附录 F）。

Figure 4: Long-horizon simulations. UniSim sequentially simulates 8 interactions autoregressively. The simulated interactions maintain temporal consistency across long-horizon interactions, correctly preserving objects and locations (can on counter in column 2-7, orange in drawer in column 4-5).

图 4：长时程仿真。 UniSim 自回归地顺序模拟 8 次交互。模拟交互在长时程上保持时间一致性，正确保留物体与位置（第 2–7 列柜台上的罐头，第 4–5 列抽屉中的橙子）。

Long-Horizon Simulation. Next, we illustrate 8 sequential interactions in Figure 4. We condition the simulation of each interaction on previous observations and new language action as described in Section 2.2. UniSim successfully preserves objects manipulated by previous instructions (e.g., the orange and can are preserved in the drawers in Columns 4, 5, 7, 8 after being put in the drawers). See additional long-horizon interactions in Appendix A.1.

长时程仿真。 接下来，我们在图 4 中展示 8 次顺序交互。按第 2.2 节所述，每次交互的仿真条件于先前观测与新的语言动作。UniSim 成功保留先前指令操作过的物体（例如橙子与罐头在放入抽屉后，在第 4、5、7、8 列仍保留在抽屉中）。更多长时程交互见附录 A.1。

Diversity and Stochasticity in the Simulator. UniSim can also support highly diverse and stochastic environment transitions, e.g., diverse objects being revealed after removing the towel on top (Figure 5 left), diverse object colors and locations (cups and pens in Figure 5 right), and real-world variabilities such as change in camera angles. Flexibility in diffusion models promotes simulation of highly stochastic environments that cannot be controlled by actions, so that a policy can learn to only control the controllable part (Yang et al., 2022).

模拟器中的多样性与随机性。 UniSim 也可支持高度多样且随机的环境转移，例如揭开上方毛巾后露出的多样物体（图 5 左）、多样物体颜色与位置（图 5 右的杯子与笔），以及相机角度变化等真实世界可变性。扩散模型的灵活性有助于模拟无法由动作完全控制的高度随机环境，从而使策略学会只控制可控部分（Yang et al., 2022）。

Figure 5: Diverse and stochastic simulations. On the left, we use text to specify the object being revealed by suffixing “uncovering” with the object name. On the right, we only specify “put cup” or “put pen”, and cups and pens of different colors are sampled as a result of the stochastic sampling process during video generation.

图 5：多样且随机的仿真。 左侧，我们用文本指定被揭开的物体，在「uncovering」后接物体名。右侧，我们只指定「put cup」或「put pen」，不同颜色的杯子与笔由视频生成中的随机采样过程采样得到。

3.2 Ablation and Analysis

3.2 消融与分析

Condition	FID $\downarrow$	FVD $\downarrow$	IS $\uparrow$	CLIP $\uparrow$
1 frame	59.47	315.69	3.03	22.55
4 distant	34.89	237	3.43	22.62
4 recent	34.63	211.3	3.52	22.63

Table 1: Ablations of history conditioning using FVD, FID, and Inception score, and CLIP score on Ego4D. Conditioning on multiple frames is better than on a single frame, and recent history has an edge over distant history.

Condition	FID $\downarrow$	FVD $\downarrow$	IS $\uparrow$	CLIP $\uparrow$
1 frame	59.47	315.69	3.03	22.55
4 distant	34.89	237	3.43	22.62
4 recent	34.63	211.3	3.52	22.63

表 1：历史条件消融，在 Ego4D 上用 FVD、FID、Inception 分数与 CLIP 分数评估。条件于多帧优于单帧，近期历史优于过远历史。

Frame Conditioning Ablations. We ablate over choices of past frames to condition on using a validation split of the Ego4D dataset (Grauman et al., 2022), which contains egocentric movement requiring proper handling of observation history. We compare UniSim conditioned on different numbers of past frames in Table 1. Conditioning on 4 frames is better than conditioning on a single frame, but conditioning on history that is too far in the past (4 frames with exponentially increasing distances) can hurt performance. Increasing the number of conditioning frames beyond 4 did not further improve performance on Ego4D, but it could be helpful for applications that require memory from distant past (e.g., navigation for retrieval).

帧条件消融。 我们在含自我中心运动、因而需要妥善处理观测历史的 Ego4D 数据集（Grauman et al., 2022）验证划分上，消融过去帧条件的选择。表 1 比较条件于不同数量过去帧的 UniSim。条件于 4 帧优于单帧，但条件于过远历史（指数递增间距的 4 帧）会损害性能。在 Ego4D 上把条件帧数增加到超过 4 并未进一步提升，但对需要远期记忆的应用（例如检索式导航）可能有帮助。

Simulating Low-Data Domains. During joint training of UniSim on diverse data, we found that naïvely combining datasets of highly varying size can result in low generation quality in low-data domains. While we can increase the weight of these domains in the data mixture during training, we found that attaching a domain identifier such as the name of the dataset to the actions being conditioned on improves generation quality in low-data domains, as shown in Figure 6. While such domain identifier improves in-distribution generation quality, we found domain-specific identifiers to hurt generalization to other domains, and should only be applied with the test domain is in distribution of the training domain.

低数据域的仿真。 在多样数据上联合训练 UniSim 时，我们发现朴素合并规模差异很大的数据集，会在低数据域导致生成质量偏低。虽然可以在训练时提高这些域在数据混合中的权重，但我们发现：把数据集名称等域标识附到所条件的动作上，能提升低数据域的生成质量，如图 6 所示。域标识虽改善分布内生成质量，却会损害向其他域的泛化，因此仅应在测试域属于训练域分布内时使用。

4 Applications of UniSim

4 UniSim 的应用

We now demonstrate how UniSim can be used to train other types of machine intelligence such as vision-language policies, RL agents, and vision-language models through simulating highly realistic experiences.

我们现在展示如何用 UniSim 通过模拟高度逼真的体验，来训练视觉–语言策略、RL 智能体与视觉–语言模型等其他类型的机器智能。

4.1 Training Long-Horizon Vision-Language Policies through Hindsight Labeling.

4.1 通过事后标注训练长时程视觉–语言策略

Language models and vision language models (VLM) have recently been used as policies that can operate in image or text based observation and action spaces (Du et al., 2023b; Driess et al., 2023; Brohan et al., 2023). One major challenge in learning such agents lies in the need for large amounts of language action labels. The labor intensity in data collection only increases as tasks increase in horizon and complexity. UniSim can generate large amounts of training data for VLM policies through hindsight relabeling.

语言模型与视觉语言模型（VLM）近期被用作可在基于图像或文本的观测与动作空间中运行的策略（Du et al., 2023b; Driess et al., 2023; Brohan et al., 2023）。学习此类智能体的一大挑战在于需要大量语言动作标签。随着任务时程与复杂度上升，数据采集的劳动强度只会增加。UniSim 可通过事后重标注为 VLM 策略生成大量训练数据。

Setup and Baseline. We use data from the Language Table environment (Lynch & Sermanet, 2020) for learning geometric rearrangements of blocks on a table. We train an image-goal conditioned VLM policy to predict language instructions and the motor controls from the start and goal images using the PALM-E architecture (Driess et al., 2023) (See data and model details in Appendix D.1). For the baseline, the goal is set to the last frame of the original short-horizon trajectories. During each evaluation run, we set the long-horizon goal by modifying the location of 3-4 blocks, and measure the blocks’ distance to their goal states after executing 5 instructions using the VLM policy. We define the reduction in distance to goal (RDG) metric as

设定与基线。 我们使用 Language Table 环境（Lynch & Sermanet, 2020）的数据学习桌面方块的几何重排。我们用 PALM-E 结构（Driess et al., 2023）训练以图像目标为条件的 VLM 策略，从起始与目标图像预测语言指令与电机控制（数据与模型细节见附录 D.1）。对基线，目标设为原始短时程轨迹的最后一帧。每次评估中，我们通过修改 3–4 个方块的位置设定长时程目标，并在用 VLM 策略执行 5 条指令后测量方块到目标状态的距离。我们将到目标距离缩减（RDG）指标定义为

$$\text{RDG}=\frac{\|s_{0}-s_{\text{goal}}\|_{2}-\|s_{T}-s_{\text{goal}}\|_{2}}{\|s_{0}-s_{\text{goal}}\|_{2}}, \tag{3}$$

where $s_{T}$ represents the underlying block locations after executing the policy, $s_{0}$ and $s_{\text{goal}}$ represents the initial and goal block locations.

其中 $s_{T}$ 表示执行策略后的底层方块位置，$s_{0}$ 与 $s_{\text{goal}}$ 表示初始与目标方块位置。

Figure 7: Long-horizon simulation. A VLM poliy generates high-level language actions (first row) which are executed in the simulator (middle row) similar to how they are executed in the real world (bottom row) using the Language Table robot. The VLM trained on data from the simulator complete long-horizon tasks by successfully moving three blocks (blue, green, yellow) to match their target location in the goal image.

图 7：长时程仿真。 VLM 策略生成高层语言动作（第一行），在模拟器中执行（中行），方式类似于在真实世界用 Language Table 机器人执行（底行）。在模拟器数据上训练的 VLM 通过成功移动三个方块（蓝、绿、黄）使其匹配目标图像中的位置，完成长时程任务。

Generating Hindsight Data with the Simulator. To use the simulator for long-horizon tasks, we draw inspiration from hindsight relabeling (Rauber et al., 2019). Specifically, we create a total of 10k long-horizon trajectories from the simulator by doing rollouts in the simulator 3-5 times per trajectory, where each rollout corresponds to one scripted language instruction. We then use the final frame from each long-horizon rollout as a goal input and the scripted language instructions as supervision for training the VLM policy.

用模拟器生成事后数据。 为把模拟器用于长时程任务，我们借鉴事后重标注（Rauber et al., 2019）。具体地，我们在模拟器中共创建 1 万条长时程轨迹，每条轨迹展开 3–5 次，每次展开对应一条脚本化语言指令。然后把每条长时程展开的最终帧作为目标输入，把脚本化语言指令作为监督，以训练 VLM 策略。

Results on Real-Robot Evaluation. Despite the VLM policy only being trained on simulated data, it is able to produce effective high-level language actions given an initial and goal image from the real Language Table domain where the data for training the simulator was collected. The simulator can simulate video trajectories from the initial real observation, from which robot actions are recovered using an inverse dynamics model and executed on the real robot. Figure 7 shows that the language actions produced by the VLM, the generated videos from the simulator according to the language actions, and the executions on the real robot. We see that the simulated video trajectory is successfully translated to robot actions in the real world. See additional results from the long-horizon VLM policy in Appendix A.2.

真实机器人评估结果。 尽管 VLM 策略仅在模拟数据上训练，给定来自真实 Language Table 域（收集训练模拟器数据的同一域）的初始与目标图像，它仍能产出有效的高层语言动作。模拟器可从初始真实观测模拟视频轨迹，再用逆动力学模型恢复机器人动作并在真实机器人上执行。图 7 展示 VLM 产出的语言动作、按语言动作由模拟器生成的视频，以及真实机器人上的执行。可见模拟视频轨迹被成功翻译为真实世界中的机器人动作。更多长时程 VLM 策略结果见附录 A.2。

Results on Simulated Evaluation. In addition to testing the language instructions and simulated video by converting video trajectory into robot actions executed on the real robot, we also conduct simulator based evaluation to compare the reduction in distance to goal (RDG) of the VLM policy using generated long-horizon data to using the original short-horizon data in Table 3. The VLM trained using long-horizon generated data performs 3-4 times better than using the original data in completing long-horizon goal-conditioned tasks.

仿真评估结果。 除了把语言指令与模拟视频转为机器人动作并在真实机器人上执行外，我们还做基于模拟器的评估，在表 3 中比较用生成的长时程数据与原始短时程数据训练的 VLM 策略的到目标距离缩减（RDG）。用长时程生成数据训练的 VLM 在完成长时程目标条件任务上比用原始数据好 3–4 倍。

	RDG (moved)	RDG (all)
VLM-BC	0.11 $\pm$ 0.13	0.07 $\pm$ 0.11
Simulator-Hindsight	0.34 $\pm 0.13$	0.34 $\pm$ 0.13

	Succ. rate (all)	Succ. rate (pointing)
VLA-BC	0.58	0.12
Simulator-RL	0.81	0.71

Table 2: Evaluation of long-horizon actions. Reduction in distance to goal (RDG) defined in Equation 3 across 5 evaluation runs of VLM trained using simulated long-horizon data (bottom row) compared to VLM trained on original short-horizon data (top row). Using the simulator performs much better both in RGD of moved blocks (left) and RGD in all blocks (right).

	RDG (moved)	RDG (all)
VLM-BC	0.11 $\pm$ 0.13	0.07 $\pm$ 0.11
Simulator-Hindsight	0.34 $\pm 0.13$	0.34 $\pm$ 0.13

	Succ. rate (all)	Succ. rate (pointing)
VLA-BC	0.58	0.12
Simulator-RL	0.81	0.71

表 2：长时程动作评估。 式 (3) 定义的到目标距离缩减（RDG），比较用模拟长时程数据训练的 VLM（底行）与用原始短时程数据训练的 VLM（顶行）在 5 次评估中的表现。使用模拟器在已移动方块的 RDG（左）与全部方块的 RDG（右）上都好得多。

4.2 Real-World Simulator for Reinforcement Learning

4.2 面向强化学习的真实世界模拟器

Reinforcement learning (RL) has achieved superhuman performance on difficult tasks such as playing Go and Atari games (Silver et al., ; Mnih et al., 2015), but has limited real world applications due, among other reasons, to the lack of a realistic environment simulator (Dulac-Arnold et al., 2019). We investigate whether the simulator can enable effective training of RL agents by providing the agent with a realistic simulator that can be accessed in parallel.

强化学习（RL）已在围棋与 Atari 游戏等困难任务上达到超人类表现（Silver et al., ; Mnih et al., 2015），但真实世界应用有限，原因之一是缺乏逼真的环境模拟器（Dulac-Arnold et al., 2019）。我们研究模拟器能否通过为智能体提供可并行访问的逼真模拟器，从而有效训练 RL 智能体。

Setup. We finetune the PaLI 3B vision-language model (Chen et al., 2022b) to predict low-level control actions (joint movements in $\Delta x,\Delta y$) from an image observation and a task description (e.g., “move the blue cube to the right”) using behavioral cloning (BC) to serve as the low-level control policy and the baseline, which we call the vision-language-action (VLA) policy similar to Brohan et al. (2023). Because UniSim can take low-level control actions as input, we can directly conduct model-based rollouts in the simulator using control actions generated by VLA policy. To acquire reward information, we use the number of steps-to-completion from the training data as a proxy reward to train a model that maps the current observation to learned reward. We then use the REINFORCE algorithm (Williams, 1992) to optimize the VLA policy, treating the rollouts from the simulator as the on-policy rollouts from the real environment and use the learned reward model to predict rewards from simulated rollouts. See details of RL training in Appendix D.2.

设定。 我们微调 PaLI 3B 视觉–语言模型（Chen et al., 2022b），使其从图像观测与任务描述（例如「把蓝色方块移到右边」）预测低层控制动作（$\Delta x,\Delta y$ 关节运动），用行为克隆（BC）作为低层控制策略与基线；我们称之为视觉–语言–动作（VLA）策略，类似 Brohan et al.（2023）。由于 UniSim 可接受低层控制动作作为输入，我们可直接用 VLA 策略生成的控制动作在模拟器中做基于模型的展开。为获取奖励信息，我们用训练数据中的完成剩余步数作为代理奖励，训练一个把当前观测映射到所学奖励的模型。随后用 REINFORCE 算法（Williams, 1992）优化 VLA 策略，把模拟器展开当作真实环境的同策略展开，并用所学奖励模型从模拟展开预测奖励。RL 训练细节见附录 D.2。

Figure 8: [Top] Simulation from low-level controls. UniSim supports low-level control actions as inputs to move endpoint horizontally, vertically, and diagonally. [Bottom] Real-robot execution of an RL policy trained in simulation and zero-shot onto the real Language Table task. The RL policy can successfully complete the task of “moving blue cube to green circle”.

图 8：[上] 由低层控制驱动的仿真。UniSim 支持以低层控制动作为输入，水平、垂直与对角地移动末端。[下] 在仿真中训练并零样本部署到真实 Language Table 任务的 RL 策略的真实机器人执行。该 RL 策略可成功完成「把蓝色方块移到绿色圆圈」任务。

Results. We first do a sanity check on simulating real-robot executions by applying low-level control actions (e.g., $\Delta x=0.05,\delta y=0.05$) repeatedly for 20-30 environment steps to move the endpoint left, right, down, up, and diagonally in Figure 8 (top two rows). We see that the simulated rollouts capture both the endpoint movements and the physics of collision. To compare the RL policy trained in simulation to the BC policy, we qualitatively assessed the simulated rollouts in the simulator. Table 3 shows that RL training significantly improves the performance of the VLA policy across a wide set of tasks, especially in tasks such as “point to blue block”. We then directly deploy the RL policy trained in the simulator onto the real robot in zero-shot, and observe successful task executions as shown in Figure 8 (bottom row). Additional results on real robot can be found in Appendix A.3.

结果。 我们先做真实机器人执行仿真的健全性检查：反复施加低层控制动作（例如 $\Delta x=0.05,\delta y=0.05$）20–30 个环境步，使末端左、右、下、上与对角移动，见图 8（上两行）。可见模拟展开既捕捉末端运动，也捕捉碰撞物理。为比较仿真中训练的 RL 策略与 BC 策略，我们对模拟器中的展开做定性评估。表 3 显示 RL 训练在广泛任务上显著提升 VLA 策略表现，尤其在「指向蓝色方块」等任务上。随后我们把仿真中训练的 RL 策略零样本直接部署到真实机器人，并观察到成功的任务执行，见图 8（底行）。更多真实机器人结果见附录 A.3。

4.3 Realistic Simulator for Broader Vision-Language Tasks

4.3 面向更广视觉–语言任务的逼真模拟器

	Activity	MSR-VTT	VATEX	SMIT
No finetune	15.2	21.91	13.31	9.22
Activity	54.90	24.88	36.01	16.91
Simulator	46.23	27.63	40.03	20.58

Table 4: VLM trained in UniSim to perform video captioning tasks. CIDEr scores for PaLI-X finetuned only on simulated data from UniSim compared to no finetuning and finetuning on true video data from ActivityNet Captions. Finetuning only on simulated data has a large advantage over no finetuning and transfers better to other tasks than finetuning on true data.

	Activity	MSR-VTT	VATEX	SMIT
No finetune	15.2	21.91	13.31	9.22
Activity	54.90	24.88	36.01	16.91
Simulator	46.23	27.63	40.03	20.58

表 4：在 UniSim 中训练的 VLM 执行视频描述任务。仅在 UniSim 模拟数据上微调的 PaLI-X 的 CIDEr 分数，对比无微调与在 ActivityNet Captions 真实视频数据上微调。仅在模拟数据上微调相对无微调有大幅优势，且向其他任务的迁移优于在真实数据上微调。

UniSim can generate training data for other machine-learning subproblems. This is especially useful when natural data is rare or difficult to collect (e.g., footage of crimes or accidents). We provide such a proof-of-concept by training vision-language models on purely generated data from UniSim, and observe significant performance benefits in video captioning.

UniSim 可为其他机器学习子问题生成训练数据。当自然数据稀少或难以收集时（例如犯罪或事故录像）尤其有用。我们给出概念验证：在 UniSim 纯生成数据上训练视觉–语言模型，并在视频描述上观察到显著性能收益。

Setup. We finetune PaLI-X (Chen et al., 2023), a VLM with 55B parameters pretrained on a broad set of image, video, and language tasks, to caption a set of videos generated by UniSim using texts from the training split of ActivityNet Captions (Krishna et al., 2017). We measure the CIDEr score of the finetuned model on the test split of ActivityNet Captions as well as other captioning tasks following the same setup as Chen et al. (2023). See finetuning details of PaLI-X in Appendix D.3.

设定。 我们微调 PaLI-X（Chen et al., 2023）——一个在广泛图像、视频与语言任务上预训练、含 550 亿参数的 VLM——使其描述 UniSim 用 ActivityNet Captions（Krishna et al., 2017）训练划分文本生成的一组视频。我们按 Chen et al.（2023）相同设定，在 ActivityNet Captions 测试划分及其他描述任务上测量微调模型的 CIDEr 分数。PaLI-X 微调细节见附录 D.3。

Results. We compare PaLI-X finetuned on purely generated videos to pretrained PaLI-X without finetuning and PaLI-X finetuned on original ActivityNet Captions in Table 4. Purely finetuning on generated data drastically improves the captioning performance from no finetuning at all on ActivityNet (15.2 to 46.23), while achieving 84% performance of finetuning on true data. Furthermore, PaLI-X finetuned on generated data transfers better to other captioning tasks such as MSR-VTT (Xu et al., 2016), VATEX (Wang et al., 2019), and SMIT (Monfort et al., 2021) than PaLI-X finetuned on true data, which tends to overfit to ActivityNet. These results suggest that UniSim can serve as an effective data generator for improving broader vision-language models.

结果。 我们在表 4 中比较仅在生成视频上微调的 PaLI-X、无微调的预训练 PaLI-X，以及在原始 ActivityNet Captions 上微调的 PaLI-X。仅在生成数据上微调大幅提升 ActivityNet 上的描述表现（从 15.2 到 46.23），达到真实数据微调表现的 84%。此外，在生成数据上微调的 PaLI-X 向 MSR-VTT（Xu et al., 2016）、VATEX（Wang et al., 2019）与 SMIT（Monfort et al., 2021）等其他描述任务的迁移，优于容易过拟合 ActivityNet 的真实数据微调模型。这些结果表明，UniSim 可作为有效的数据生成器，以改进更广的视觉–语言模型。

5 Related Work

5 相关工作

Internet-Scale Generative Models. Language models trained on internet text succeed at text-based tasks (OpenAI, 2023; Anil et al., 2023) but not physical tasks, which requires perception and control. Internet-scale generative models can synthesize realistic images and videos (Wu et al., 2021; Ho et al., 2022a; Singer et al., 2022; Yang et al., 2023; Blattmann et al., 2023), but have mostly been applied to generative media (Zhang et al., 2023) as opposed to empowering sophisticated agents capable of multi-turn interactions. Du et al. (2023a) shows video generation can serve as policies, but the major bottleneck for policy learning often lies in limited access to real-world environments (Dulac-Arnold et al., 2019). We focus on this exact bottleneck by learning universal simulators of the real world, enabling realistic and unlimited “environment” access for training sophisticated agents interactively.

互联网规模生成模型。 在互联网文本上训练的语言模型擅长基于文本的任务（OpenAI, 2023; Anil et al., 2023），但不擅长需要感知与控制的物理任务。互联网规模生成模型可合成逼真图像与视频（Wu et al., 2021; Ho et al., 2022a; Singer et al., 2022; Yang et al., 2023; Blattmann et al., 2023），但大多被用于生成媒体（Zhang et al., 2023），而非赋能能进行多轮交互的复杂智能体。Du et al.（2023a）表明视频生成可充当策略，但策略学习的主要瓶颈往往在于真实世界环境访问有限（Dulac-Arnold et al., 2019）。我们正聚焦这一瓶颈：学习真实世界的通用模拟器，为交互式训练复杂智能体提供逼真且无限的「环境」访问。

Learning World Models. Learning an accurate dynamics model in reaction to control inputs has been a long-standing challenge in system identification (Ljung & Glad, 1994), model-based reinforcement learning (Sutton, 1991), and optimal control Åström & Wittenmark (1973); Bertsekas (1995). Most systems choose to learn one dynamics model per system in the lower dimensional state space as opposed to in the pixel space (Ferns et al., 2004; Achille & Soatto, 2018; Lesort et al., 2018; Castro, 2020), which, despite being a simpler modeling problem, limits knowledge sharing across systems. With large transformer architectures, learning image-based world models has become plausible (Hafner et al., 2020; Chen et al., 2022a; Seo et al., 2022; Micheli et al., 2022; Wu et al., 2022; Hafner et al., 2023), but mostly in games or simulated domains with visually simplistic and abundant data. In generative modeling of videos, previous works have leveraged text prompts (Yu et al., 2023; Zhou et al., 2022), driving motions (Siarohin et al., 2019; Wang et al., 2022), 3D geometries (Weng et al., 2019; Xue et al., 2018), physical simulations (Chuang et al., 2005), frequency information (Li et al., 2023), and user annotations (Hao et al., 2018) to introduce movements into videos. However, they focus on generating domain specific videos (e.g., for self-driving) as opposed to building a universal simulator that can be used to further improve other agents. The amount of control over generated videos in these existing work is also limited, as they do not treat video generation as a dynamics modeling problem like in our work.

学习世界模型。 学习对控制输入作出反应的准确动力学模型，是系统辨识（Ljung & Glad, 1994）、基于模型的强化学习（Sutton, 1991）与最优控制（Åström & Wittenmark, 1973; Bertsekas, 1995）中的长期挑战。多数系统选择在较低维状态空间而非像素空间为每个系统学习一个动力学模型（Ferns et al., 2004; Achille & Soatto, 2018; Lesort et al., 2018; Castro, 2020）；尽管建模更简单，却限制了跨系统知识共享。有了大型 Transformer 结构，学习基于图像的世界模型变得可行（Hafner et al., 2020; Chen et al., 2022a; Seo et al., 2022; Micheli et al., 2022; Wu et al., 2022; Hafner et al., 2023），但多在视觉简单、数据充裕的游戏或仿真域。在视频生成建模中，先前工作借助文本提示（Yu et al., 2023; Zhou et al., 2022）、驱动运动（Siarohin et al., 2019; Wang et al., 2022）、三维几何（Weng et al., 2019; Xue et al., 2018）、物理仿真（Chuang et al., 2005）、频率信息（Li et al., 2023）与用户标注（Hao et al., 2018）为视频引入运动。但它们聚焦生成特定域视频（例如自动驾驶），而非构建可进一步改进其他智能体的通用模拟器。这些工作对生成视频的控制量也有限，因为它们不像我们那样把视频生成当作动力学建模问题。

6 Limitations and Conclusion

6 局限与结论

We have shown it is possible to learn a simulator of the real world in response to various action inputs ranging from texts to robot controls. UniSim can simulate visually realistic experiences for interacting with humans and training autonomous agents. We hope UniSim will instigate broad interest in learning and applying real-world simulators to improve machine intelligence. Our simulator has a few limitations that call for future work:

我们已表明：可以学习一个对从文本到机器人控制等多种动作输入作出响应的真实世界模拟器。UniSim 可为与人类交互以及训练自主智能体模拟视觉上逼真的体验。我们希望 UniSim 激发对学习与应用真实世界模拟器以改进机器智能的广泛兴趣。我们的模拟器有若干局限，有待未来工作：

• • Hallucination. When an action is unrealistic given the scene (e.g., “wash hands” is given to a tabletop robot), we observe hallucinations (e.g., the table turns into a sink or the view turns away from the tabletop robot and a sink shows up). Ideally, we want UniSim to detect actions that are not possible to simulate as opposed to hallucinating unrealistic outcomes.
• • Limited memory. The simulator conditioned on a few frames of the recent history cannot capture long-term memory (e.g., an apple in a drawer could disappear when the drawer is opened if putting the apple in the drawer is not a part of the history for conditioning). How much history to condition on depends on the application of the (e.g., whether the simulator will be used for policy learning in a near-Markov setting or question answering that requires long-term memory).
• • Limited out-of-domain generalization. This is especially true for domains that are not represented in the training data. For instance, the simulator is mostly trained on 4 robot morphologies, and its ability to generalize to an unseen robot is limited. Further scaling up training data could help, as the training data is nowhere near all the video data available on the internet.
• • Visual simulation only. Our simulator is not suitable for environments where actions do not cause visual observation change (e.g., different forces in grasping a static cup). A true universal simulator should capture all aspects of the world beyond visual experience (e.g., sound, sensory, etc).

• • 幻觉。 当动作相对场景不现实时（例如给桌面机器人「洗手」），我们观察到幻觉（例如桌子变成水槽，或视角从桌面机器人移开并出现水槽）。理想情况下，我们希望 UniSim 能检测无法模拟的动作，而不是幻觉出不现实的结果。
• • 有限记忆。 仅条件于近期历史若干帧的模拟器无法捕捉长期记忆（例如若把苹果放进抽屉不在条件历史中，打开抽屉时抽屉里的苹果可能消失）。应条件多少历史取决于应用（例如模拟器用于近马尔可夫设定下的策略学习，还是需要长期记忆的问答）。
• • 有限的域外泛化。 对训练数据未覆盖的域尤其如此。例如模拟器主要在 4 种机器人形态上训练，向未见机器人泛化的能力有限。进一步扩大训练数据可能有帮助，因为训练数据远未接近互联网上全部可用视频。
• • 仅视觉仿真。 对我们的模拟器而言，动作不引起视觉观测变化的环境并不合适（例如抓静止杯子时的不同力）。真正的通用模拟器应捕捉超越视觉体验的世界各方面（例如声音、传感等）。

References

参考文献

Appendix

附录

In this Appendix we provide additional qualitative results on long-horizon simulation of human and robot interactions (Section A.1), long-horizon VLM policies (Section A.2), and low-level RL policies (Section A.3) that work on real robot. We also provided details on the dataset used to train UniSim in Section B, the model architecture and training details of UniSim in Section C, and the details of the three experimental setups for applications of UniSim in Section D. Finally, we provide failed examples when UniSim is not jointly trained on broad datasets (Section F). Video demos can be found at anonymous-papers-submissions.github.io

本附录提供人类与机器人长时程交互仿真（第 A.1 节）、长时程 VLM 策略（第 A.2 节）以及可在真实机器人上工作的低层 RL 策略（第 A.3 节）的额外定性结果。我们还在第 B 节给出训练 UniSim 所用数据集细节，第 C 节给出 UniSim 的模型结构与训练细节，第 D 节给出 UniSim 三类应用实验设定的细节。最后，我们给出未在广泛数据集上联合训练时 UniSim 的失败示例（第 F 节）。视频演示见 anonymous-papers-submissions.github.io

Achille & Soatto (2018) Alessandro Achille and Stefano Soatto. A separation principle for control in the age of deep learning. Annual Review of Control, Robotics, and Autonomous Systems , 1:287–307, 2018.

Anderson et al. (2018) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 3674–3683, 2018.

Andrychowicz et al. (2017) Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems . 2017.

Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 , 2023.

Bertsekas (1995) Dimitri P. Bertsekas. Dynamic Programming and Optimal Control . Athena Scientific, 1995.

Blattmann et al. (2023) Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. arXiv preprint arXiv:2304.08818 , 2023.

Brohan et al. (2022) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022.

Brohan et al. (2023) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 , 2023.

Castro (2020) Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pp. 10069–10076, 2020.

Chang et al. (2017) Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 , 2017.

Chen et al. (2022a) Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Transdreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.09481 , 2022a.

Chen et al. (2022b) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 , 2022b.

Chen et al. (2023) Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565 , 2023.

Chuang et al. (2005) Yung-Yu Chuang, Dan B Goldman, Ke Colin Zheng, Brian Curless, David H Salesin, and Richard Szeliski. Animating pictures with stochastic motion textures. In ACM SIGGRAPH 2005 Papers , pp. 853–860. 2005.

Çiçek et al. (2016) Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19 , pp. 424–432. Springer, 2016.

Damen et al. (2018) Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV) , pp. 720–736, 2018.

Dasari et al. (2019) Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv:1910.11215 , 2019.

Driess et al. (2023) Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

Du et al. (2023a) Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation, 2023. URL https://arxiv. org/abs/2302.00111 , 2023a.

Du et al. (2023b) Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.06692 , 2023b.

Dulac-Arnold et al. (2019) Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901 , 2019.

Ebert et al. (2021) Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396 , 2021.

Ferns et al. (2004) Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI , volume 4, pp. 162–169, 2004.

Goyal et al. (2017) Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision , pp. 5842–5850, 2017.

Grauman et al. (2022) Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 18995–19012, 2022.

Hafner et al. (2020) Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193 , 2020.

Hafner et al. (2023) Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 , 2023.

Hao et al. (2018) Zekun Hao, Xun Huang, and Serge Belongie. Controllable video generation with sparse trajectories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 7854–7863, 2018.

Ho & Salimans (2022) Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , 2020.

Ho et al. (2022a) Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022a.

Ho et al. (2022b) Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022b.

Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pp. 4904–4916. PMLR, 2021.

Krishna et al. (2017) Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision , pp. 706–715, 2017.

Lesort et al. (2018) Timothée Lesort, Natalia Díaz-Rodríguez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks , 108:379–392, 2018.

Li et al. (2023) Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics, 2023.

Ljung & Glad (1994) Lennart Ljung and Torkel Glad. Modeling of dynamic systems . Prentice-Hall, Inc., 1994.

Lynch & Sermanet (2020) Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648 , 2020.

Lynch et al. (2023) Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters , 2023.

Mandlekar et al. (2018) Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning , pp. 879–893. PMLR, 2018.

Micheli et al. (2022) Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. arXiv preprint arXiv:2209.00588 , 2022.

Miech et al. (2019) Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision , pp. 2630–2640, 2019.

Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529–533, 2015.

Monfort et al. (2021) Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14871–14881, 2021.

OpenAI (2023) OpenAI. Gpt-4 technical report, 2023.

Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21(140):1–67, 2020.

Ramakrishnan et al. (2021) Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2021. URL https://arxiv.org/abs/2109.08238 .

Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Rauber et al. (2019) Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, and Jürgen Schmidhuber. Hindsight policy gradients. In International Conference on Learning Representations , 2019.

Rusu et al. (2017) Andrei A Rusu, Matej Večerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. In Conference on robot learning , pp. 262–270. PMLR, 2017.

Savva et al. (2019) Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision , pp. 9339–9347, 2019.

Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278–25294, 2022.

Seo et al. (2022) Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning , pp. 19561–19579. PMLR, 2022.

Siarohin et al. (2019) Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2377–2386, 2019.

(56) David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815 .

Singer et al. (2022) Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022.

Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pp. 2256–2265. PMLR, 2015.

Sutton (1988) Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning , 1988.

Sutton (1991) Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin , 2(4):160–163, 1991.

Tassa et al. (2018) Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Tay et al. (2022) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. Ul2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations , 2022.

Wang et al. (2021) Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 6847–6857, 2021.

Wang et al. (2019) Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 4581–4591, 2019.

Wang et al. (2022) Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate images via latent space navigation. arXiv preprint arXiv:2203.09043 , 2022.

Weng et al. (2019) Chung-Yi Weng, Brian Curless, and Ira Kemelmacher-Shlizerman. Photo wake-up: 3d character animation from a single photo. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 5908–5917, 2019.

Williams (1992) Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229–256, 1992.

Wu et al. (2021) Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806 , 2021.

Wu et al. (2022) Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05861 , 2022.

Xu et al. (2016) Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 5288–5296, 2016.

Xue et al. (2018) Tianfan Xue, Jiajun Wu, Katherine L Bouman, and William T Freeman. Visual dynamics: Stochastic future generation via layered cross convolutional networks. IEEE transactions on pattern analysis and machine intelligence , 41(9):2236–2250, 2018.

Yang et al. (2022) Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Separating what you can control from what you cannot. arXiv preprint arXiv:2210.13435 , 2022.

Yang et al. (2023) Mengjiao Yang, Yilun Du, Bo Dai, Dale Schuurmans, Joshua B Tenenbaum, and Pieter Abbeel. Probabilistic adaptation of text-to-video models. arXiv preprint arXiv:2306.01872 , 2023.

Yu et al. (2023) Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 18456–18466, 2023.

Zhai et al. (2022) Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 12104–12113, 2022.

Zhang et al. (2023) Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.07909 , 2023.

Zhou et al. (2022) Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018 , 2022.

Åström & Wittenmark (1973) Karl J. Åström and Björn Wittenmark. Adaptive control of linear time-invariant systems. Automatica , 9(6):551–564, 1973.

Appendix A Additional Results

附录 A 额外结果

A.1 Additional Long-Horizon Interaction

A.1 额外长时程交互

Figure 9: Additional results on long-horizon interaction with humans and robots similar to Figure 4. UniSim can generate consistent video rollouts across 3-4 high-level language actions.

图 9：类似图 4 的人类与机器人长时程交互额外结果。UniSim 可在 3–4 个高层语言动作上生成一致的视频展开。

A.2 Additional Real-Robot Results for Long-Horizon Language Policy

A.2 长时程语言策略的额外真实机器人结果

Figure 10: Additional results (similar to Figure 7) on applying UniSim to train vision-language policies to complete long-horizon tasks. VLM finetuned with hindsight labeled data is able to generate long-horizon instructions that moves two or three blocks successfully to match their location in the goal image.

图 10：类似图 7、把 UniSim 用于训练视觉–语言策略以完成长时程任务的额外结果。用事后标注数据微调的 VLM 能生成长时程指令，成功移动两到三个方块以匹配目标图像中的位置。

A.3 Additional Results on Learning RL Policy in UniSim

A.3 在 UniSim 中学习 RL 策略的额外结果

Figure 11: First real observations and last simulated observations of rolling out the RL policy trained in UniSim.

图 11：在 UniSim 中训练的 RL 策略展开时的首帧真实观测与末帧模拟观测。

Figure 12: First real observations and last real observations of executing the RL policy trained from UniSim in the real world in zero-shot. Middle plot also shows the output of the learned reward model (steps-to-completion) during policy execution, where step 0 corresponds to the top plot (initial observation) and step 70 corresponds to the bottom plot (final observation).

图 12：把从 UniSim 训练的 RL 策略零样本在真实世界执行时的首帧与末帧真实观测。中间图还展示策略执行期间所学奖励模型（完成剩余步数）的输出，其中第 0 步对应上图（初始观测），第 70 步对应下图（最终观测）。

Appendix B Datasets

附录 B 数据集

We provide the datasets used to train UniSim below, including dataset name, number of training examples (approximate), and weight in the data mixture. Miscellaneous data are collections of datasets that have not been published. Some of these datasets have been processed into train and validation split, hence the number of training examples may differ from the original data size. When text are available in the original dataset, we use T5 language model embeddings (Raffel et al., 2020) to preprocess the text into continuous representations. When low-level controls are available in the original dataset, we encode them both as text and normalize then discretize them into 4096 bins contatenated with language embeddings (if present). The choice of mixture weights are either 0.1 or 0.05 without careful tuning. How data mixture weights affect simulation performance is an interesting line of future work.

我们在下表给出训练 UniSim 所用数据集，包括数据集名称、训练样本数（约数）以及在数据混合中的权重。杂项数据是尚未发表的数据集合。部分数据集已处理为训练与验证划分，故训练样本数可能与原始数据规模不同。原数据集有文本时，我们用 T5 语言模型嵌入（Raffel et al., 2020）把文本预处理为连续表示。原数据集有低层控制时，我们既把它们编码为文本，也归一化后离散化为 4096 个箱，再与语言嵌入（若有）拼接。混合权重取 0.1 或 0.05，未经仔细调参。数据混合权重如何影响仿真性能是有趣的未来方向。

	Dataset	# Examples	Weight
Simulation	Habitat HM3D (Ramakrishnan et al., 2021)	710	0.1
Language Table sim (Lynch & Sermanet, 2020)	160k	0.05
Real Robot	Bridge Data (Ebert et al., 2021)	2k	0.05
RT-1 data (Brohan et al., 2022)	70k	0.1
Language Table real (Lynch & Sermanet, 2020)	440k	0.05
Miscellaneous robot videos	133k	0.05
Human activities	Ego4D (Grauman et al., 2022)	3.5M	0.1
Something-Something V2 (Goyal et al., 2017)	160k	0.1
EPIC-KITCHENS (Damen et al., 2018)	25k	0.1
Miscellaneous human videos	50k	0.05
Panorama scan	Matterport Room-to-Room scans (Anderson et al., 2018)	3.5M	0.1
Internet text-image	LAION-400M (Schuhmann et al., 2021)	400M	0.05
ALIGN (Jia et al., 2021)	400M	0.05
Internet video	Miscellaneous videos	13M	0.05

Table 5: Dataset name, number of training examples, and mixture weights used for training UniSim.

	Dataset	# Examples	Weight
Simulation	Habitat HM3D (Ramakrishnan et al., 2021)	710	0.1
Language Table sim (Lynch & Sermanet, 2020)	160k	0.05
Real Robot	Bridge Data (Ebert et al., 2021)	2k	0.05
RT-1 data (Brohan et al., 2022)	70k	0.1
Language Table real (Lynch & Sermanet, 2020)	440k	0.05
Miscellaneous robot videos	133k	0.05
Human activities	Ego4D (Grauman et al., 2022)	3.5M	0.1
Something-Something V2 (Goyal et al., 2017)	160k	0.1
EPIC-KITCHENS (Damen et al., 2018)	25k	0.1
Miscellaneous human videos	50k	0.05
Panorama scan	Matterport Room-to-Room scans (Anderson et al., 2018)	3.5M	0.1
Internet text-image	LAION-400M (Schuhmann et al., 2021)	400M	0.05
ALIGN (Jia et al., 2021)	400M	0.05
Internet video	Miscellaneous videos	13M	0.05

表 5：训练 UniSim 所用的数据集名称、训练样本数与混合权重。

Appendix C Architecture and Training

附录 C 结构与训练

We the 3D U-Net architecture (Çiçek et al., 2016; Ho et al., 2022b) to parametrize UniSim video model. We apply the spatial downsampling pass followed by the spatial upsampling pass with skip connections to the downsampling pass activations with interleaved 3D convolution and attention layers as in the standard 3D U-Net. The video models in UniSim consist of one history conditioned video prediction model as the base and two additional spatial super-resolution models similar to Ho et al. (2022a). The history conditioned base model operates at temporal and spatial resolution $[16,24,40]$, and the two spatial super-resolution models operate at spatial resolution $[24,40]\rightarrow[48,80]$ and $[48,80]\rightarrow[192,320]$, respectively. To condition the base video model on the history, we take 4 frames from the previous video segment and concatenate them channelwise to the noise samples inputted to the U-Net. We employ temporal attention for the forward model to allow maximum modeling flexibility but temporal convolution to the super-resolution models for efficiency reasons similar to Ho et al. (2022a). The model and training hyperparamters of UniSim are summarized in Table 6.

我们用三维 U-Net 结构（Çiçek et al., 2016; Ho et al., 2022b）参数化 UniSim 视频模型。我们先做空间下采样，再做带跳跃连接到下采样激活的空间上采样，并像标准三维 U-Net 那样交错使用三维卷积与注意力层。UniSim 中的视频模型由一个以历史为条件的视频预测基模型与两个额外的空间超分辨率模型组成，类似 Ho et al.（2022a）。以历史为条件的基模型工作在时间与空间分辨率 $[16,24,40]$，两个空间超分辨率模型分别工作在 $[24,40]\rightarrow[48,80]$ 与 $[48,80]\rightarrow[192,320]$。为使基视频模型条件于历史，我们从先前视频段取 4 帧，在通道维拼接到输入 U-Net 的噪声样本上。前向模型使用时间注意力以获得最大建模灵活性，超分辨率模型则因效率原因使用时间卷积，类似 Ho et al.（2022a）。UniSim 的模型与训练超参数汇总于表 6。

Hyperparameter	Value
Base channels	1024
Optimizer	Adam ($\beta_{1}=0.9,\beta_{2}=0.99$)
Channel multipliers	1, 2, 4
Learning rate	0.0001
Blocks per resolution	3
Batch size	256
Attention resolutions	6, 12, 24
Num attention heads	16, 16, 8
Conditioning embedding dimension	4096
Conditioning embedding MLP layers: 4
Conditioning token length	64
EMA	0.9999
Dropout	0.1
Training hardware	512 TPU-v3 chips
Training steps	1000000
Diffusion noise schedule	cosine
Noise schedule log SNR range	[-20, 20]
Sampling timesteps	256
Sampling log-variance interpolation	$\gamma=0.1$
Weight decay	0.0
Prediction target	$\epsilon$

Table 6: Hyperparameters for training UniSim diffusion model.

Hyperparameter	Value
Base channels	1024
Optimizer	Adam ($\beta_{1}=0.9,\beta_{2}=0.99$)
Channel multipliers	1, 2, 4
Learning rate	0.0001
Blocks per resolution	3
Batch size	256
Attention resolutions	6, 12, 24
Num attention heads	16, 16, 8
Conditioning embedding dimension	4096
Conditioning embedding MLP layers: 4
Conditioning token length	64
EMA	0.9999
Dropout	0.1
Training hardware	512 TPU-v3 chips
Training steps	1000000
Diffusion noise schedule	cosine
Noise schedule log SNR range	[-20, 20]
Sampling timesteps	256
Sampling log-variance interpolation	$\gamma=0.1$
Weight decay	0.0
Prediction target	$\epsilon$

表 6：训练 UniSim 扩散模型的超参数。

Appendix D Details of Experimental Setups

附录 D 实验设定细节

D.1 Details of Learning Long-Horizon Policy

D.1 学习长时程策略的细节

Language Table Dataset and environment. The Language Table (Lynch & Sermanet, 2020) dataset consists of 160k simulated trajectories and 440k real trajectories where each trajectory contains a language instruction (e.g., “move blue cube to the right”), a sequence of visuomotor controls, and a sequence of image frames corresponding to the execution of the task. The original trajectories have short horizons (e.g., only moving one block).

Language Table 数据集与环境。 Language Table（Lynch & Sermanet, 2020）数据集包含 16 万条仿真轨迹与 44 万条真实轨迹，每条轨迹含一条语言指令（例如「把蓝色方块移到右边」）、一串视觉运动控制，以及对应任务执行的一串图像帧。原始轨迹时程较短（例如只移动一个方块）。

PALM-E VLM Policy. We modify the original PALM-E 12B model (Driess et al., 2023) to condition on a goal image as additional input before decoding the text actions. The VLM is finetuned on either the original short horizon data or the long horizon simulated data using 64 TPUv3 chips for 1 day. The supervision for short-horizon baseline is the single step language instruction in the original data, whereas the supervision for long-horizon UniSim data is the scripted long-horizon language instructions chained together that generated the video data. Other model architecture and training details follow Driess et al. (2023).

PALM-E VLM 策略。 我们修改原始 PALM-E 120 亿参数模型（Driess et al., 2023），使其在解码文本动作前额外条件于目标图像。VLM 在原始短时程数据或长时程模拟数据上微调，使用 64 块 TPUv3、历时 1 天。短时程基线的监督是原始数据中的单步语言指令；长时程 UniSim 数据的监督是串联起来生成视频数据的脚本化长时程语言指令。其他模型结构与训练细节遵循 Driess et al.（2023）。

Simulated evaluation. In setting up goal in the simulated environments, a subset of 3-4 blocks (randomly selected) are moved by 0.05, 0.1, or 0.2 along the x,y axes (randomly selected). The original observation space has $x\in[0.15,0.6]$ and $y\in[-0.3048,0.3048]$. So the modification of goal location corresponds to meaningful block movements. For executing the long-horizon VLM policy trained on UniSim data, we first sample one language instruction from the VLM, predict a video of 16 frames, and use a separately trained inverse dynamics model similar to Du et al. (2023a) to recover the low-level control actions, which we found to slightly outperform directly regressing on control actions from language outputs of the VLM. We execute 5 instructions in total, and measure the final distance to goal according to the ground truth simulator state. We 5 evaluations each with a different random seed for sampling the initial state and resetting the goal, and report the mean and standard error in Table 3.

仿真评估。 在仿真环境中设定目标时，随机选取 3–4 个方块子集，沿 x、y 轴（随机选取）移动 0.05、0.1 或 0.2。原始观测空间为 $x\in[0.15,0.6]$、$y\in[-0.3048,0.3048]$，因此目标位置修改对应有意义的方块移动。对在 UniSim 数据上训练的长时程 VLM 策略，我们先从 VLM 采样一条语言指令，预测 16 帧视频，再用类似 Du et al.（2023a）单独训练的逆动力学模型恢复低层控制动作；我们发现这略优于直接从 VLM 语言输出回归控制动作。我们共执行 5 条指令，并按真值模拟器状态测量最终到目标距离。我们做 5 次评估，每次用不同随机种子采样初始状态并重置目标，在表 3 中报告均值与标准误差。

D.2 Details of RL Policy Training

D.2 RL 策略训练细节

Stage 1 (Supervised Learning) Model Architecture The PaLI 3B model trained on Language-Table uses a Vision Transformer architecture G/14 (Zhai et al., 2022) to process images, and the encoder-decoder architecture of UL2 language model (Tay et al., 2022) for encoding task descriptions and decoding tokens which can represent language, control actions, or other values of interest (described below). Objectives In the first stage of training, using a dataset of demonstrations, we finetune the pretrained PaLI 3B vision language model checkpoint (Chen et al., 2022b) with the following tasks:

阶段 1（监督学习） 模型结构 在 Language-Table 上训练的 PaLI 3B 模型用 Vision Transformer 结构 G/14（Zhai et al., 2022）处理图像，并用 UL2 语言模型（Tay et al., 2022）的编码器–解码器结构编码任务描述、解码可表示语言、控制动作或其他关注量的词元（见下）。目标 在训练第一阶段，我们用演示数据集按以下任务微调预训练 PaLI 3B 视觉语言模型检查点（Chen et al., 2022b）：

• • Behavioral Cloning: Given observations and task instruction, predict the demonstration action. The continuous actions of the Language-Table domain are discretized into the form “+1 -5”, and represented using extra tokens from the PaLI model’s token vocabulary. As an example, “+1 -5” is represented by the token sequence (, , , ).
• • Timestep to Success Prediction: Given observations and task instruction, predict how many timesteps are left until the end of episode (i.e. success). Similar to actions, the number of steps remaining is represented via extra tokens from the PaLI model’s token vocabulary.
• • Instruction Prediction: Given the first and last frame of an episode, predict the task instruction associated with that episode.

• • 行为克隆： 给定观测与任务指令，预测演示动作。Language-Table 域的连续动作被离散化为「+1 -5」形式，并用 PaLI 模型词表中的额外词元表示。例如，「+1 -5」表示为词元序列 (, , , )。
• • 到成功的时间步预测： 给定观测与任务指令，预测距回合结束（即成功）还剩多少时间步。与动作类似，剩余步数用 PaLI 模型词表中的额外词元表示。
• • 指令预测： 给定一个回合的首末帧，预测与该回合关联的任务指令。

We use learning rate 0.001, dropout rate 0.1, and batch size 128 to finetune the PaLI 3B model for 300k gradient steps with 1k warmup steps on both the simulated and real Language Table dataset similar to RT-2 Brohan et al. (2023).

我们用学习率 0.001、dropout 0.1、批量大小 128，在仿真与真实 Language Table 数据集上微调 PaLI 3B 模型 30 万梯度步、含 1 千预热步，类似 RT-2（Brohan et al., 2023）。

Stage 2 (RL Training) Reward Definition As mentioned above, during Stage 1, given an observation and goal, the PaLI model is finetuned to predict how many timesteps are left until the demonstration episode reaches a success state. Let us denote this function by $d(o,g)$. The reward we use during RL training is defined as $r(o_{t},a_{t},o_{t+1},g)=-[d(o_{t+1},g)-d(o_{t},g)]\cdot\mathcal{C}$, where $\mathcal{C}>0$ is a small constant used to stabilize training ($\mathcal{C}=5e-2$ in this work). Intuitively, this reward tracks if from timestep $t$ to $t+1$ the policy arrived closer to accomplishing the desired goal. Before starting Stage 2, we make a copy of the Stage 1 model checkpoint and keep it frozen to use as the reward model for RL training. Environment Definition To implement video generation as environment transitions, we expose the inference interface of the video generation model through remote procedure call, and use the DeepMind RL Environment API (also known as DM Env API) (Tassa et al., 2018) to wrap the remote procedure call in the step function of the environment. When the environment is reset to start a new episode, a goal instruction is randomly sampled from the ones available in the dataset of demonstrations used in Stage 1. RL Method We initialize the RL trained policy using the Stage 1 checkpoint, which as mentioned was also trained with a Behavioral Cloning objective. A collection of actor processes perform policy rollouts in the video generation environment, and add rewards to the trajectories using the reward model defined above. The policy is updated using the REINFORCE (Williams, 1992) objective, i.e. $\nabla_{\pi}\mathcal{L}(o_{t},a_{t},g)=\nabla_{\pi}\log\pi(a_{t}|o_{t},g)\cdot\left[\sum_{i=t}^{T}\gamma^{i-t}\cdot r(o_{i},a_{i},o_{i+1},g)\right]$, where $\mathcal{L}(o_{t},a_{t},g)$ represents the loss associated with the observation-action pair $(o_{t},a_{t})$ in an episode with the goal $g$. The actors are rate limited to prevent generated trajectories from being very off-policy. We report the hyperparameters associated with RL training in Table 7.

阶段 2（RL 训练） 奖励定义 如上所述，阶段 1 中给定观测与目标，PaLI 模型被微调以预测演示回合到达成功状态还剩多少时间步。记该函数为 $d(o,g)$。RL 训练中使用的奖励定义为 $r(o_{t},a_{t},o_{t+1},g)=-[d(o_{t+1},g)-d(o_{t},g)]\cdot\mathcal{C}$，其中 $\mathcal{C}>0$ 是用于稳定训练的小常数（本工作中 $\mathcal{C}=5e-2$）。直观上，该奖励跟踪从时间步 $t$ 到 $t+1$ 策略是否更接近完成期望目标。开始阶段 2 前，我们复制阶段 1 模型检查点并保持冻结，作为 RL 训练的奖励模型。环境定义 为实现作为环境转移的视频生成，我们通过远程过程调用暴露视频生成模型的推理接口，并用 DeepMind RL Environment API（亦称 DM Env API）（Tassa et al., 2018）把远程过程调用包装进环境的 step 函数。环境重置以开始新回合时，从阶段 1 所用演示数据集中随机采样一条目标指令。RL 方法 我们用阶段 1 检查点初始化 RL 训练策略，如前所述该检查点也曾用行为克隆目标训练。一组 actor 进程在视频生成环境中做策略展开，并用上述奖励模型为轨迹加奖励。策略用 REINFORCE（Williams, 1992）目标更新，即 $\nabla_{\pi}\mathcal{L}(o_{t},a_{t},g)=\nabla_{\pi}\log\pi(a_{t}|o_{t},g)\cdot\left[\sum_{i=t}^{T}\gamma^{i-t}\cdot r(o_{i},a_{i},o_{i+1},g)\right]$，其中 $\mathcal{L}(o_{t},a_{t},g)$ 表示目标为 $g$ 的回合中观测–动作对 $(o_{t},a_{t})$ 的损失。对 actor 限速，以防生成轨迹过于偏离策略。RL 训练相关超参数见表 7。

Hyperparameter	Value
Max steps per episode	100
Number of actor processes	64
Number of image history stack	2
Learner batch size	64
Discounting factor $\gamma$	0.9

Table 7: Hyperparameters for training the VLA RL policy using the ACME framework.

Hyperparameter	Value
Max steps per episode	100
Number of actor processes	64
Number of image history stack	2
Learner batch size	64
Discounting factor $\gamma$	0.9

表 7：用 ACME 框架训练 VLA RL 策略的超参数。

D.3 Details of Video Captioning

D.3 视频描述细节

Note that even though UniSim is a video based simulator trained to condition on past history, we can achieve text-only conditioning by inputting placeholder frames such as white images while increasing the classifier-free guidance strength on text. We found this to work well in generating videos purely from captions of ActivityNet Captions. For generating data to train VLMs, we take the training split of ActivityNet Captions which consists of 30,740 text-video examples after the 50/25/25% train/val1/val2 split as in Chen et al. (2023). For each of the 30,740 text, we generate 4 videos from UniSim, and use the text labels as supervision in finetuning PaLI-X. As a result, we have 4X amount of the original training data (in terms the number of videos). In addition, we found the generated videos to generally align better semantically than the original ActivityNet Captions videos, which could contain noise and ambiguous videos that could be labeled differently. We use ground truth temporal proposals at evaluation following Chen et al. (2023) and Krishna et al. (2017). Following Chen et al. (2023) and Wang et al. (2021), we use the val1 split for validation and val2 split for testing.

需注意：尽管 UniSim 是以过去历史为条件训练的基于视频的模拟器，我们可通过输入白图等占位帧并提高文本上的无分类器引导强度，实现纯文本条件。我们发现这在仅从 ActivityNet Captions 字幕生成视频时效果良好。为生成训练 VLM 的数据，我们取 ActivityNet Captions 训练划分，按 Chen et al.（2023）的 50/25/25% 训练/val1/val2 划分后共有 30740 条文视频示例。对这 30740 条文本中的每一条，我们从 UniSim 生成 4 段视频，并用文本标签作为微调 PaLI-X 的监督。因此我们拥有原训练数据 4 倍数量的视频。此外，我们发现生成视频在语义对齐上通常优于可能含噪声与歧义、标签可不唯一的原始 ActivityNet Captions 视频。评估时我们按 Chen et al.（2023）与 Krishna et al.（2017）使用真值时间提议。按 Chen et al.（2023）与 Wang et al.（2021），用 val1 划分做验证、val2 划分做测试。

Appendix E Additional Ablations

附录 E 额外消融

E.1 Ablations of Datasets

E.1 数据集消融

We conduct ablations on dataset used in UniSim by computing the FVD and CLIP scores over 1024 samples from the test split. We observe that including internet data and various activity and robot data performs the best. Removing the internet data led to significantly worse FVD, highlighting the importance of using internet data in UniSim.

我们通过对测试划分上 1024 个样本计算 FVD 与 CLIP 分数，对 UniSim 所用数据集做消融。我们观察到：纳入互联网数据以及各类活动与机器人数据表现最好。去掉互联网数据导致 FVD 显著变差，凸显互联网数据在 UniSim 中的重要性。

Dataset	FVD $\downarrow$	CLIP $\uparrow$
Internet only	219.62	22.27
Without internet	307.80	21.99
Universal simulator	211.30	22.63

Table 8: Ablations of datasets using FVD and CLIP score on the held-out test split. Including internet data and diverse human activity and robot data in UniSim achieves the best FVD and CLIP scores.

Dataset	FVD $\downarrow$	CLIP $\uparrow$
Internet only	219.62	22.27
Without internet	307.80	21.99
Universal simulator	211.30	22.63

表 8：数据集消融，在留出测试划分上用 FVD 与 CLIP 分数评估。在 UniSim 中纳入互联网数据以及多样人类活动与机器人数据，取得最佳 FVD 与 CLIP 分数。

E.2 Ablations of Model Size

E.2 模型规模消融

We conduct ablations on model size by computing the FVD and CLIP scores over 1024 samples from the test split. We found that while increasing the model size improves the video modeling performance, the amount of improvement measured by FVD plateaus as the model gets bigger, which is slightly disappointing from a scaling point of view.

我们通过对测试划分上 1024 个样本计算 FVD 与 CLIP 分数，对模型规模做消融。我们发现：增大模型规模会改善视频建模表现，但以 FVD 衡量的改进随模型变大而趋于平台，从缩放角度看略显遗憾。

Model size	FVD $\downarrow$	CLIP $\uparrow$
500M	277.85	22.08
1.6B	224.61	22.27
5.6B	211.30	22.63

Table 9: Ablations of model size using FVD and CLIP score on the held-out test split. The largest model achieves the best FVD and CLIP scores.

Model size	FVD $\downarrow$	CLIP $\uparrow$
500M	277.85	22.08
1.6B	224.61	22.27
5.6B	211.30	22.63

表 9：模型规模消融，在留出测试划分上用 FVD 与 CLIP 分数评估。最大模型取得最佳 FVD 与 CLIP 分数。

Appendix F Failed Simulations without Joint Training

附录 F 未经联合训练时的失败仿真

Figure 13: Failed environment simulation from the action “uncover bottle” without training on broad data as in UniSim. Top two videos are generated from only training on SSV2. Bottom two videos are generated from only training on generic internet data (without SSV2, EpicKitchen, Ego4D, and various robotics dataset).

图 13：未像 UniSim 那样在广泛数据上训练时，对动作「uncover bottle」的失败环境仿真。上两段视频仅在 SSV2 上训练生成；下两段仅在通用互联网数据上训练生成（不含 SSV2、EpicKitchen、Ego4D 与各类机器人数据集）。

Figure 14: When the text-to-video model behind UniSim is only trained on data from Brohan et al. (2022) as opposed incorporating broad data from the internet and other manipulation datasets, long-horizon interaction simulations fail half of the time (red text).

图 14：当 UniSim 背后的文生视频模型仅在 Brohan et al.（2022）的数据上训练，而未纳入互联网及其他操作数据集时，长时程交互仿真约有一半失败（红字）。