figures/hook.pdfA whole new world: Genie is capable of converting a variety of different prompts into interactive, playable environments that can be easily created, stepped into, and explored. This is made possible via a latent action interface, learned fully unsupervised from Internet videos. On the right we see a few generated steps for taking two latent actions. See more examples on our website.

figures/hook.pdf全新世界：Genie 能把多种提示转换成可交互、可玩的环境，便于创建、进入与探索。关键是潜在动作接口，完全从互联网视频无监督学得。右侧展示了执行两种潜在动作后生成的若干步。更多示例见我们的网站。

Genie: Generative Interactive Environments

Jake Bruce Affiliation: Equal contributions Affiliation: Google DeepMind    Michael Dennis Affiliation: Equal contributions Affiliation: Google DeepMind    Ashley Edwards Affiliation: Equal contributions Affiliation: Google DeepMind    Jack Parker-Holder Affiliation: Equal contributions Affiliation: Google DeepMind    Yuge (Jimmy) Shi Affiliation: Equal contributions Affiliation: Google DeepMind    Edward Hughes Affiliation: Google DeepMind    Matthew Lai Affiliation: Google DeepMind    Aditi Mavalankar Affiliation: Google DeepMind    Richie Steigerwald Affiliation: Google DeepMind    Chris Apps Affiliation: Google DeepMind    Yusuf Aytar Affiliation: Google DeepMind    Sarah Bechtle Affiliation: Google DeepMind    Feryal Behbahani Affiliation: Google DeepMind    Stephanie Chan Affiliation: Google DeepMind    Nicolas Heess Affiliation: Google DeepMind    Lucy Gonzalez Affiliation: Google DeepMind    Simon Osindero Affiliation: Google DeepMind    Sherjil Ozair Affiliation: Google DeepMind    Scott Reed Affiliation: Google DeepMind    Jingwei Zhang Affiliation: Google DeepMind    Konrad Zolna Affiliation: Google DeepMind    Jeff Clune Affiliation: Google DeepMind Affiliation: University of British Columbia    Nando de Freitas Affiliation: Google DeepMind    Satinder Singh Affiliation: Google DeepMind    Tim Rocktäschel Affiliation: Equal contributions Affiliation: Google DeepMind

Abstract

摘要

We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.

我们提出 Genie：首个以无标注互联网视频、无监督方式训练的生成式交互环境。模型可被提示，生成由文本、合成图、照片乃至草图描述的、动作可控的虚拟世界，种类近乎无穷。参数量 11B，可视为基础世界模型。它由时空视频分词器、自回归动力学模型，以及简单可扩展的潜在动作模型组成。即使用户在训练阶段没有真实动作标签，也没有世界模型文献中常见的领域专用约束，Genie 仍支持按帧在生成环境中行动。学到的潜在动作空间还可用来训练智能体去模仿未见视频中的行为，为未来通才智能体的训练打开路径。

1 1. Introduction

1 引言

keywords

The last few years have seen an emergence of generative AI, with models capable of generating novel and creative content. Driven by breakthroughs in architectures such as transformers (Vaswani et al., 2017), advances in hardware, and a recent focus on scaling models and datasets, we can now generate coherent, conversational language (Radford et al., 2018; Radford et al., 2019; Brown et al., 2020), as well as crisp and aesthetically pleasing images from a text prompt (Ramesh et al., 2021; Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022). Early signs indicate video generation will be yet another frontier, with recent results suggesting that such models may also benefit from scale (Hong et al., 2023; Ho et al., 2022a; Esser et al., 2023; Blattmann et al., 2023a). Still, there remains a gulf between the level of interactions and engagement of video generative models and language tools such as ChatGPT, let alone more immersive experiences.

关键词

近几年来，生成式 AI 兴起，模型已能生成新颖而有创意的内容。在 Transformer（Vaswani et al., 2017）等架构突破、硬件进步，以及模型与数据规模化的推动下，我们已能生成连贯、可对话的语言（Radford et al., 2018; Radford et al., 2019; Brown et al., 2020），也能从文本提示生成清晰、美观的图像（Ramesh et al., 2021; Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022）。早期迹象表明视频生成将成为下一前沿，近期结果也暗示这类模型同样受益于规模（Hong et al., 2023; Ho et al., 2022a; Esser et al., 2023; Blattmann et al., 2023a）。不过，视频生成模型在交互与参与感上，与 ChatGPT 一类语言工具仍有鸿沟，更谈不上沉浸式体验。

What if, given a large corpus of videos from the Internet, we could not only train models capable of generating novel images or videos, but entire interactive experiences? We propose generative interactive environments, a new paradigm for generative AI whereby interactive environments can be generated from a single text or image prompt. Our approach, Genie, is trained from a large dataset of over 200,000 hours of publicly available Internet gaming videos and, despite training without action or text annotations, is controllable on a frame-by-frame basis via a learned latent action space (see Table 1 for a comparison to other approaches). At 11B parameters, Genie exhibits properties typically seen in foundation models—it can take an unseen image as a prompt making it possible to create and play entirely imagined virtual worlds (e.g Figure 1).

若拥有大量互联网视频，我们能否不仅训练出生成新图像或视频的模型，还能生成整段交互体验？我们提出生成式交互环境：一种新的生成式 AI 范式，可从单条文本或图像提示生成可交互环境。我们的方法 Genie 在超过 20 万小时的公开互联网游戏视频上训练；尽管没有动作或文本标注，仍可通过学到的潜在动作空间按帧控制（与其他方法的对比见表 1）。参数量 11B 时，Genie 展现出基础模型常见的性质——可用未见过的图像作提示，从而创建并游玩完全想象出的虚拟世界（例如图 1）。

Figure 1: Diverse trajectories: Genie is a generative model that can be used as an interactive environment. The model can be prompted in various ways, either with a generated image (top) or a hand-drawn sketch (bottom). At each time step, the model takes a user-provided latent action to generate the next frame, producing trajectories with interesting and diverse character actions.

图 1：多样轨迹：Genie 是可作为交互环境使用的生成模型。提示方式多样，例如生成图像（上）或手绘草图（下）。每一步模型接收用户给出的潜在动作并生成下一帧，得到角色动作有趣且多样的轨迹。

Genie builds on ideas from state-of-the-art video generation models (Villegas et al., 2023; Gupta et al., 2023), with a core design choice being spatiotemporal (ST) transformers (Xu et al., 2020) which are used in all of our model components. Genie utilizes a novel video tokenizer, and extracts latent actions via a causal action model. Both the video tokens and latent actions are passed to a dynamics model, which autoregressively predicts the next frame using MaskGIT (Chang et al., 2022). We provide a rigorous scaling analysis of our architecture with respect to both batch and model size, which we vary from 40M to 2.7B parameters. The results show that our architecture scales gracefully with additional computational resources, leading to a final 11B parameter model. We train Genie on a filtered set of 30,000 hours of Internet gameplay videos from hundreds of 2D platformer games, producing a foundation world model for this setting.

Genie 借鉴了先进视频生成模型的思路（Villegas et al., 2023; Gupta et al., 2023），核心设计是在所有组件中使用时空（ST）Transformer（Xu et al., 2020）。Genie 采用新型视频分词器，并用因果动作模型提取潜在动作。视频 token 与潜在动作一并送入动力学模型，由 MaskGIT（Chang et al., 2022）自回归预测下一帧。我们对架构相对批大小与模型规模（40M 到 2.7B）做了严格的缩放分析；结果表明，增加算力时架构平滑扩展，最终得到 11B 参数模型。Genie 在经筛选的、来自数百款 2D 平台跳跃游戏的约 3 万小时互联网游玩视频上训练，形成该设定下的基础世界模型。

To demonstrate the generality of our approach, we also train a separate model on action-free robot videos from the RT1 dataset (Brohan et al., 2023), learning a generative environment with consistent latent actions. Finally, we show that latent actions learned from Internet videos can be used for inferring policies from unseen action-free videos of simulated reinforcement learning (RL) environments, indicating that Genie may hold the key to unlocking unlimited data for training the next generation of generalist agents (Open Ended Learning Team et al., 2021; Bauer et al., 2023; Reed et al., 2022; Clune, 2019).

为展示方法的通用性，我们还在 RT1 数据集（Brohan et al., 2023）的无动作机器人视频上另训一模型，得到潜在动作一致的生成式环境。最后我们表明：从互联网视频学到的潜在动作，可用于从未见过的、无动作标注的模拟强化学习（RL）环境视频中推断策略，说明 Genie 或能解锁近乎无限的数据，用于训练下一代通才智能体（Open Ended Learning Team et al., 2021; Bauer et al., 2023; Reed et al., 2022; Clune, 2019）。

Figure 2: Genie model training: Genie takes in $T$ frames of video as input, tokenizes them into discrete tokens $\bm{z}$ via the video tokenizer, and infers the latent actions $\tilde{\bm{a}}$ between each frame with the latent action model. Both are then passed to the dynamics model to generate predictions for the next frames in an iterative manner.

图 2：Genie 模型训练：Genie 以 $T$ 帧视频为输入，经视频分词器得到离散 token $\bm{z}$，并用潜在动作模型推断帧间潜在动作 $\tilde{\bm{a}}$。二者再送入动力学模型，迭代生成对后续帧的预测。

Model Class	Training Data	Controllability
World Models	Video + Actions	Frame-level
Video Models	Video + Text	Video-level
Genie	Video	Frame-level

Table 1: A new class of generative model: Genie is a novel video and world model that is controllable on a frame-by-frame basis, which requires only video data at train time.

模型类别	训练数据	可控性
世界模型	视频 + 动作	帧级
视频模型	视频 + 文本	视频级
Genie	视频	帧级

表 1：一类新的生成模型：Genie 是新颖的视频与世界模型，可按帧控制，训练时只需视频数据。

2 2. Methodology

2 方法

Genie is a generative interactive environment trained from video-only data. In this section we begin with preliminaries before explaining the main components of our model.

Genie 是仅用视频数据训练的生成式交互环境。本节先给出预备知识，再说明模型的主要组件。

Several components in the Genie architecture are based on the Vision Transformer (ViT) (Vaswani et al., 2017; Dosovitskiy et al., 2021). Notably, the quadratic memory cost of transformers poses challenges for videos, which can contain up to $O(10^{4})$ tokens. We thus adopt a memory efficient ST-transformer architecture (inspired by Xu et al. (2020), see Figure 3) across all model components, balancing model capacity with computational constraints.

Genie 架构中多个组件基于 Vision Transformer（ViT）（Vaswani et al., 2017; Dosovitskiy et al., 2021）。值得注意的是，Transformer 二次内存代价对视频构成挑战——视频可含多达 $O(10^{4})$ 个 token。因此我们在所有组件中采用节省内存的 ST-Transformer 架构（受 Xu et al. (2020) 启发，见图 3），在模型容量与算力约束之间取得平衡。

Figure 3: ST-transformer architecture. The architecture is composed of $L$ spatiotemporal blocks, each containing a spatial layer, temporal layer and feed-forward layer. Each color represents a single self-attention map, with the spatial layer attending over the $H\times W$ tokens from within a single time step, and temporal the same token from across the $T$ time steps.

图 3：ST-Transformer 架构。架构由 $L$ 个时空块组成，每块含空间层、时间层与前馈层。每种颜色表示一张自注意力图：空间层在同一时间步的 $H\times W$ 个 token 上注意力，时间层则对跨 $T$ 个时间步的同一 token 位置做注意力。

Unlike a traditional transformer where every token attends to all others, an ST-transformer contains $L$ spatiotemporal blocks with interleaved spatial and temporal attention layers, followed by a feed-forward layer (FFW) as standard attention blocks. The self-attention in the spatial layer attends over the $1\times H\times W$ tokens within each time step, and in the temporal layer attends over $T\times 1\times 1$ tokens across the $T$ time steps. Similar to sequence transformers, the temporal layer assumes a causal structure with a causal mask. Crucially, the dominating factor of computation complexity (i.e. the spatial attention layer) in our architecture scales linearly with the number of frames rather than quadratically, making it much more efficient for video generation with consistent dynamics over extended interactions. Further, note that in the ST block, we include only one FFW after both spatial and temporal components, omitting the post-spatial FFW to allow for scaling up other components of the model, which we observe to improve results significantly.

与传统 Transformer 中每个 token 关注所有其他 token 不同，ST-Transformer 含 $L$ 个时空块，空间与时间注意力层交错，其后接标准注意力块中的前馈层（FFW）。空间层自注意力覆盖每个时间步内的 $1\times H\times W$ 个 token；时间层自注意力覆盖跨 $T$ 个时间步的 $T\times 1\times 1$ 个 token。与序列 Transformer 类似，时间层采用因果掩码。关键是，计算复杂度的主导项（即空间注意力层）随帧数线性而非二次增长，更适合在长时间交互中保持一致动力学的视频生成。此外，在 ST 块中，我们只在空间与时间分量之后放一个 FFW，省略空间后的 FFW，以便把容量留给模型其他部分；我们观察到这能显著改善结果。

2.1 Model Components

2.1 模型组件

As shown in Figure 2, our model contains three key components: 1) a latent action model that infers the latent action $\bm{a}$ between each pair of frames and 2) a video tokenizer that converts raw video frames into discrete tokens $\bm{z}$ and 3) a dynamics model that, given a latent action and past frame tokens, predicts the next frame of the video. The model is trained in two phases following a standard autoregressive video generation pipeline: we train the video tokenizer first, which is used for the dynamics model. We then co-train the latent action model (directly from pixels) and the dynamics model (on video tokens).

如图 2 所示，模型含三个关键组件：1）在每对帧之间推断潜在动作 $\bm{a}$ 的潜在动作模型；2）将原始视频帧转为离散 token $\bm{z}$ 的视频分词器；3）给定潜在动作与历史帧 token、预测下一视频帧的动力学模型。训练分两阶段，遵循标准自回归视频生成流程：先训视频分词器，供动力学模型使用；再联合训练潜在动作模型（直接来自像素）与动力学模型（在视频 token 上）。

Latent Action Model (LAM) To achieve controllable video generation, we condition each future frame prediction on the action taken at the previous frame. However, such action labels are rarely available in videos from the Internet and action annotation can be costly to obtain. Instead, we learn latent actions in a fully unsupervised manner (see Figure 4).

潜在动作模型（LAM） 为实现可控视频生成，我们让每一未来帧的预测以先前帧所采取的动作为条件。但互联网视频很少带动作标签，标注也成本高昂。于是我们以完全无监督方式学习潜在动作（见图 4）。

Figure 4: Latent action model: learns actions $a_{t}$ unsupervised from unlabelled video frames.

图 4：潜在动作模型：从无标注视频帧无监督学习动作 $a_{t}$。

First, an encoder takes as inputs all previous frames $\bm{x}_{1:t}=(x_{1},\cdots x_{t})$ as well as the next frame $x_{t+1}$, and outputs a corresponding set of continuous latent actions $\tilde{\bm{a}}_{1:t}=(\tilde{a}_{1},\cdots\tilde{a}_{t})$. A decoder then takes all previous frames and latent actions as input and predicts the next frame $\hat{x}_{t+1}$.

首先，编码器以所有先前帧 $\bm{x}_{1:t}=(x_{1},\cdots x_{t})$ 以及下一帧 $x_{t+1}$ 为输入，输出对应的连续潜在动作 $\tilde{\bm{a}}_{1:t}=(\tilde{a}_{1},\cdots\tilde{a}_{t})$。解码器再以所有先前帧与潜在动作为输入，预测下一帧 $\hat{x}_{t+1}$。

To train the model, we leverage a VQ-VAE-based objective (van den Oord et al., 2017), which enables us to limit the number of predicted actions to a small discrete set of codes. We limit the vocabulary size $|A|$ of the VQ codebook, i.e. the maximum number of possible latent actions, to a small value to permit human playability and further enforce controllability (we use $|A|=8$ in our experiments). As the decoder only has access to the history and latent action, $\tilde{a}_{t}$ should encode the most meaningful changes between the past and the future for the decoder to successfully reconstruct the future frame. Note that this decoder exists only to give the LAM training signal. In fact, apart from the VQ codebook, the entire LAM is discarded at inference time and replaced with actions from the user.

训练时我们采用基于 VQ-VAE 的目标（van den Oord et al., 2017），从而把预测动作限制在一小组离散码上。我们将 VQ 码本词汇量 $|A|$（即可用潜在动作的最大数）限制为较小值，以便人类可玩并进一步强化可控性（实验中取 $|A|=8$）。由于解码器只能看到历史与潜在动作，$\tilde{a}_{t}$ 应编码过去与未来之间对成功重建未来帧最有意义的变化。注意该解码器仅用于给 LAM 提供训练信号。事实上，推理时除 VQ 码本外，整个 LAM 会被丢弃，改由用户动作替代。

We utilize our ST-transformer architecture for the latent action model. The causal mask in the temporal layer allows us to take the entire video $\bm{x}_{1:T}$ as input and generate all latent actions between each frame $\tilde{\bm{a}}_{1:T-1}$.

潜在动作模型采用我们的 ST-Transformer 架构。时间层中的因果掩码使我们能以整段视频 $\bm{x}_{1:T}$ 为输入，并生成帧间全部潜在动作 $\tilde{\bm{a}}_{1:T-1}$。

Video Tokenizer Following prior work (Villegas et al., 2023; Gupta et al., 2023; Yan et al., 2023), we compress videos into discrete tokens to reduce dimensionality and enable higher quality video generation (see Figure 5). We again make use of VQ-VAE, which takes in $T$ frames of video $\bm{x}_{1:T}=(x_{1},x_{2},\cdots,x_{T})\in\mathbb{R}^{T\times H\times W\times C}$ as input, generating discrete representations for each frame $\bm{z}_{1:T}=(z_{1},z_{2},\cdots,z_{T})\in\mathbb{I}^{T\times D}$, where $D$ is the size of the discrete latent space. The tokenizer is trained using a standard VQ-VQAE objective over the entire video sequence.

视频分词器 沿用先前工作（Villegas et al., 2023; Gupta et al., 2023; Yan et al., 2023），我们将视频压缩为离散 token，以降低维度并提高生成质量（见图 5）。同样使用 VQ-VAE：输入 $T$ 帧视频 $\bm{x}_{1:T}=(x_{1},x_{2},\cdots,x_{T})\in\mathbb{R}^{T\times H\times W\times C}$，为每帧生成离散表示 $\bm{z}_{1:T}=(z_{1},z_{2},\cdots,z_{T})\in\mathbb{I}^{T\times D}$，其中 $D$ 为离散潜空间大小。分词器以整段视频序列上的标准 VQ-VAE 目标训练。

Figure 5: Video tokenizer: a VQ-VAE with ST-transformer.

图 5：视频分词器：带 ST-Transformer 的 VQ-VAE。

Unlike prior works that focus on spatial-only compression in the tokenization phase (Hong et al., 2022; Wu et al., 2022; Gupta et al., 2023), we utilize the ST-transformer in both the encoder and decoder to incorporate temporal dynamics in the encodings, which improves the video generation quality. By the causal nature of the ST-transformer, each discrete encoding $z_{t}$ contains information from all previously seen frames of the video $\bm{x}_{1:t}$. Phenaki (Villegas et al., 2023) also uses a temporal-aware tokenizer, C-ViViT, but this architecture is compute intensive, as the cost grows quadratically with the number of frames—in comparison, our ST-transformer based tokenizer (ST-ViViT) is much more compute efficient with the dominating factor in its cost increasing linearly with the number of frames.

与分词阶段只做空间压缩的先前工作不同（Hong et al., 2022; Wu et al., 2022; Gupta et al., 2023），我们在编码器与解码器中都使用 ST-Transformer，使编码纳入时间动态，从而提升视频生成质量。由于 ST-Transformer 的因果性，每个离散编码 $z_{t}$ 含有此前所见全部帧 $\bm{x}_{1:t}$ 的信息。Phenaki（Villegas et al., 2023）也使用时间感知分词器 C-ViViT，但该架构计算密集，代价随帧数二次增长——相比之下，我们基于 ST-Transformer 的分词器（ST-ViViT）更省算力，主导代价随帧数近似线性增长。

Figure 6: Dynamics model: takes in video tokens and action embeddings, and predicts future masked video tokens.

图 6：动力学模型：接收视频 token 与动作嵌入，预测未来被掩码的视频 token。

Dynamics Model The dynamics model is a decoder-only MaskGIT (Chang et al., 2022) transformer (Figure 6). At each time step $t\in[1,T]$, it takes in the tokenized video $\bm{z}_{1:t-1}$ and stopgrad latent actions $\tilde{\bm{a}}_{1:t-1}$ and predicts the next frame tokens $\hat{z}_{t}$. We again utilize an ST-transformer, whose causal structure enables us to use tokens from all $(T-1)$ frames $\bm{z}_{1:T-1}$ and latent actions $\tilde{\bm{a}}_{1:T-1}$ as input, and generate predictions for all next frames $\hat{\bm{z}}_{2:T}$. The model is trained with a cross-entropy loss between the predicted tokens $\hat{\bm{z}}_{2:T}$ and ground-truth tokens $\bm{z}_{2:T}$. At train time we randomly mask the input tokens $\bm{z}_{2:T-1}$ according to a Bernoulli distribution masking rate sampled uniformly between $0.5$ and $1$. Note that a common practice for training world-models, including transformer-based models, is to concatenate the action at time $t$ to the corresponding frame (Micheli et al., 2023; Robine et al., 2023). However, we found that treating the latent actions as additive embeddings for both the latent action and dynamics models helped to improve the controllability of the generations.

动力学模型 动力学模型是仅解码器的 MaskGIT（Chang et al., 2022）Transformer（图 6）。在每个时间步 $t\in[1,T]$，它以分词后的视频 $\bm{z}_{1:t-1}$ 与 stopgrad 潜在动作 $\tilde{\bm{a}}_{1:t-1}$ 为输入，预测下一帧 token $\hat{z}_{t}$。我们再次使用 ST-Transformer：其因果结构使我们能以全部 $(T-1)$ 帧的 token $\bm{z}_{1:T-1}$ 与潜在动作 $\tilde{\bm{a}}_{1:T-1}$ 为输入，并生成对所有后续帧 $\hat{\bm{z}}_{2:T}$ 的预测。训练目标是预测 token $\hat{\bm{z}}_{2:T}$ 与真实 token $\bm{z}_{2:T}$ 之间的交叉熵。训练时按 Bernoulli 分布对输入 token $\bm{z}_{2:T-1}$ 随机掩码，掩码率在 $0.5$ 与 $1$ 之间均匀采样。训练世界模型（含基于 Transformer 的模型）的常见做法是把时刻 $t$ 的动作拼接到对应帧上（Micheli et al., 2023; Robine et al., 2023）。但我们发现，把潜在动作当作潜在动作模型与动力学模型中的加性嵌入，有助于提升生成的可控性。

2.2 Inference: Action-Controllable Video Generation

2.2 推理：动作可控的视频生成

Figure 7: Genie Inference: the prompt frame is tokenized, combined with the latent action taken by the user, and passed to the dynamics model for iterative generation. The predicted frame tokens are then decoded back to image space via the tokenizer’s decoder.

图 7：Genie 推理：提示帧经分词后，与用户采取的潜在动作结合，送入动力学模型做迭代生成。预测的帧 token 再经分词器解码器还原到图像空间。

We now describe how to use Genie for action-controllable video generation at inference time (see Figure 7). A player first prompts the model with an image $x_{1}$ that serves as the initial frame[^1]. The image is tokenized using the video encoder, yielding $z_{1}$. The player then specifies a discrete latent action $a_{1}$ to take by choosing any integer value within $[0,|A|)$.[^2] The dynamics model takes the frame tokens $z_{1}$ and corresponding latent action $\tilde{a}_{1}$, which is obtained by indexing into the VQ codebook with the discrete input $a_{1}$, to predict the next frame tokens $z_{2}$. This process is repeated to generate the rest of the sequence $\hat{\bm{z}}_{2:T}$ in an autoregressive manner as actions continue to be passed to the model, while tokens are decoded into video frames $\hat{\bm{x}}_{2:T}$ with the tokenizer’s decoder. Note that we can regenerate ground truth videos from the dataset by passing the model the starting frame and inferred actions from the video, or generate completely new videos (or trajectories) by changing the actions.

下面说明如何在推理时用 Genie 做动作可控的视频生成（见图 7）。玩家先以图像 $x_{1}$ 提示模型，作为初始帧[^1]。图像经视频编码器分词得到 $z_{1}$。玩家再指定离散潜在动作 $a_{1}$，取 $[0,|A|)$ 内任意整数。[^2] 动力学模型接收帧 token $z_{1}$ 以及对应潜在动作 $\tilde{a}_{1}$（由离散输入 $a_{1}$ 索引 VQ 码本得到），预测下一帧 token $z_{2}$。随着动作不断传入，该过程自回归重复，生成序列其余部分 $\hat{\bm{z}}_{2:T}$；token 再经分词器解码器变为视频帧 $\hat{\bm{x}}_{2:T}$。注意：把起始帧与从视频推断的动作传给模型，可重建数据集中的真实视频；改变动作则可生成全新视频（或轨迹）。

3 3. Experimental Results

3 实验结果

Figure 8: Scaling results. Left: Training curves for different model sizes, Middle: Final training loss for each model size, averaged over the last 300 updates, Right: Final training loss for a 2.3B model with different batch sizes.

图 8：缩放结果。左：不同模型规模的训练曲线；中：各规模最终训练损失（最后 300 次更新平均）；右：2.3B 模型在不同批大小下的最终训练损失。

Datasets We train Genie on a large-scale dataset collected from publicly available Internet videos of 2D Platformer games (referred to from here on as “Platformers”). We construct the Platformers dataset by filtering publicly available videos for keywords relating to platformers, yielding 55M 16s video clips at 10FPS, with 160x90 resolution. The final dataset contains 6.8M 16s video clips (30k hours), within an order of magnitude of other popular Internet video datasets (Wang et al., 2023; Bain et al., 2021). More details can be found in Section B.1. Unless otherwise specified, results are with a 11B-parameter model trained on this dataset.

数据集 我们在从公开互联网 2D 平台跳跃游戏视频收集的大规模数据集上训练 Genie（下文称 “Platformers”）。通过关键词过滤公开视频得到 5500 万段 16 秒、10FPS、160×90 分辨率的片段。最终数据集含 680 万段 16 秒片段（约 3 万小时），与其他流行互联网视频数据集量级接近（Wang et al., 2023; Bain et al., 2021）。更多细节见第 B.1 节。除非另有说明，结果均来自在该数据集上训练的 11B 参数模型。

To verify the generality of our method, we also consider the robotics datasets used to train RT1 Brohan et al. (2023), combining their dataset of ${\sim}130k$ robot demonstrations with a separate dataset of simulation data and the 209k episodes of real robot data from prior work (Kalashnikov et al., 2018). Note that we do not use actions from any of these datasets, and simply treat them as videos. For simplicity, from here on we refer to this dataset as “Robotics”.

为验证方法的通用性，我们还考虑用于训练 RT1 的机器人数据集 Brohan et al. (2023)：将其约 ${\sim}130k$ 条机器人演示，与另一份仿真数据以及先前工作中 209k 条真实机器人回合（Kalashnikov et al., 2018）合并。注意我们不使用其中任何动作，只当视频使用。下文简称该数据集为 “Robotics”。

Metrics We examine the video generation performance of Genie via two factors, namely video fidelity, i.e. the quality of video generation, and controllability, i.e. how much impact the latent actions have in video generation. For video fidelity we use the Frechet Video Distance (FVD), a video-level metric, which has been shown to have a high level of alignment to human evaluation on video quality (Unterthiner et al., 2019). For controllability, we devise a metric based on peak signal-to-noise ratio (PSNR) which we call $\Delta_{t}\text{PSNR}$, that measures how much the video generations differ when conditioned on latent actions inferred from ground-truth ($\hat{x}_{t}$) vs. sampled from a random distribution ($\hat{x}_{t}^{\prime}$):

指标 我们从两方面考察 Genie 的视频生成表现：视频保真度（生成质量）与可控性（潜在动作对生成的影响程度）。保真度用 Frechet Video Distance（FVD）这一视频级指标，它与人类对视频质量的评价高度一致（Unterthiner et al., 2019）。可控性方面，我们基于峰值信噪比（PSNR）设计指标 $\Delta_{t}\text{PSNR}$，衡量以从真实帧推断的潜在动作为条件生成的视频（$\hat{x}_{t}$）与以随机分布采样的潜在动作为条件生成的视频（$\hat{x}_{t}^{\prime}$）之间的差异：

$$\displaystyle\Delta_{t}\text{PSNR}=\text{PSNR}(x_{t},\hat{x}_{t})-\text{PSNR}(x_{t},\hat{x}_{t}^{\prime}),$$

where $x_{t}$ denotes the ground-truth frame at time $t$, $\hat{x}_{t}$ denotes the frame from latent actions $\tilde{\bm{a}}_{1:t}$ inferred from ground-truth frames, and $\hat{x}_{t}^{\prime}$ the same frame generated from a sequence of latent actions randomly sampled from a categorical distribution. As such, the greater $\Delta_{t}\text{PSNR}$ is, the more the video generated from random latent actions differs from ground-truth, which indicates a higher level of controllability from the latent actions. For all experiments we report $\Delta_{t}\text{PSNR}$ with $t=4$.

其中 $x_{t}$ 为时刻 $t$ 的真实帧，$\hat{x}_{t}$ 为由从真实帧推断的潜在动作 $\tilde{\bm{a}}_{1:t}$ 生成的帧，$\hat{x}_{t}^{\prime}$ 为由从类别分布随机采样的潜在动作序列生成的同一帧。因此 $\Delta_{t}\text{PSNR}$ 越大，随机潜在动作生成的视频与真实越不同，说明潜在动作的可控性越强。所有实验报告 $t=4$ 时的 $\Delta_{t}\text{PSNR}$。

Training Details Our video tokenizer uses 200M parameters, a patch size of 4 and a codebook with embedding size 32 and 1024 unique codes, which we found to be the most effective given the trade-off between reconstruction quality of the tokenizer and downstream performance of video prediction. The latent action model has 300M parameters, a patch size of 16, and a codebook with embedding size 32 and 8 unique codes (latent actions). For all modelling components we use a sequence length of 16 frames with an FPS of 10. Further, we employ bfloat16 and QK norm for training our dynamics model, which has been shown to stabilize training at large scale (Henry et al., 2020; Dehghani et al., 2023). At inference time, we perform 25 MaskGIT steps for the sampling of each frame with a temperature of 2 using random sampling. See Appendix C for more details.

训练细节 视频分词器约 2 亿参数，patch 大小为 4，码本嵌入维 32、共 1024 个码；在重建质量与下游视频预测表现的权衡下，这组设置最有效。潜在动作模型约 3 亿参数，patch 大小 16，码本嵌入维 32、共 8 个码（潜在动作）。各建模组件序列长度均为 16 帧、10 FPS。动力学模型训练使用 bfloat16 与 QK norm，已有工作表明其有助于大规模稳定训练（Henry et al., 2020; Dehghani et al., 2023）。推理时每帧做 25 步 MaskGIT 采样，温度 2，随机采样。更多细节见附录 C。

3.1 Scaling Results

3.1 缩放结果

In this section, we investigate the scaling behavior of our model. To this end, we conduct studies that explore the impact of both model size and batch size. See Appendix D for more details on architecture and compute usage.

本节考察模型的缩放行为。我们分别研究模型规模与批大小的影响。架构与算力细节见附录 D。

Figure 9: Playing from Image Prompts: We can prompt Genie with images generated by text-to-image models, hand-drawn sketches or real-world photos. In each case we show the prompt frame and a second frame after taking one of the latent actions four consecutive times. In each case we see clear character movement, despite some of the images being visually distinct from the dataset.

图 9：从图像提示游玩：可用文生图模型生成的图像、手绘草图或真实照片提示 Genie。每种情形展示提示帧，以及连续执行同一潜在动作四次后的第二帧。尽管部分图像视觉上与训练集差异明显，仍可见清晰的角色运动。

Scaling Model Size Given a fixed video tokenizer and action model architecture, we train a series of dynamics models ranging from 40M to 2.7B parameters. Figure 8 shows our architecture scales gracefully with model parameters, with each increase in size corresponding to a consistent decrease in the final training loss. This is a strong indication that our approach benefits from scaling, which we exploit with our main Genie model.

缩放模型规模 在固定视频分词器与动作模型架构下，我们训练一系列 40M 到 2.7B 参数的动力学模型。图 8 显示架构随参数平滑扩展：规模每增一档，最终训练损失都稳定下降。这强烈表明方法受益于缩放，我们也据此打造最终的 Genie 模型。

Scaling Batch Size We also investigate the effect of scaling the batch size, considering a 2.3B model with batch sizes of 128, 256, and 448, equating to 1.9M, 3.8M and 6.6M tokens. As shown in Figure 8, increasing the batch size leads to a similarly favorable gain in terms of model performance.

缩放批大小 我们还研究批大小的影响：在 2.3B 模型上比较批大小 128、256、448，对应约 190 万、380 万与 660 万 token。如图 8 所示，增大批大小同样带来有利的性能提升。

Genie Model It is clear that increasing both model size and batch size helps improve model performance. As a result, for our final model, we train a 10.1B dynamics model with a batch size of 512, for a total of 125k steps, using 256 TPUv5p. When combined with the tokenizer and action model this brings the total to 10.7B parameters, trained on 942B tokens, which we refer to as the Genie model. For our website, we train a larger decoder mapping tokens to 360p videos, adding additional parameters.

Genie 模型 增大模型规模与批大小都能改善表现。因此最终模型中，我们用批大小 512、共 125k 步、256 块 TPUv5p 训练 10.1B 动力学模型；加上分词器与动作模型，总参数约 10.7B，训练 token 约 942B，即 Genie 模型。网站演示另训了更大的解码器，把 token 映射到 360p 视频，从而增加额外参数。

3.2 Qualitative Results

3.2 定性结果

We now present qualitative results from the Genie model. We showcase a 11B parameter model trained on the Platformers dataset and a smaller model trained on the Robotics dataset. Our model generates high-quality, controllable videos across diverse domains. Notably, we qualitatively evaluate our Platformers-trained model using only out-of-distribution (OOD) image prompts, including those generated from text-to-image models, hand-drawn sketches, and even realistic photos. The ability to generalize to such significantly OOD inputs underscores the robustness of our approach and the value of training on large-scale data, which would not have been feasible with real actions as input.

下面给出 Genie 的定性结果。我们展示在 Platformers 上训练的 11B 模型，以及在 Robotics 上训练的较小模型。模型能在多样领域生成高质量、可控的视频。值得注意的是，对 Platformers 模型的定性评估仅使用分布外（OOD）图像提示，包括文生图模型生成的图像、手绘草图乃至真实照片。能泛化到如此显著 OOD 的输入，说明方法稳健，也体现了大规模数据训练的价值——若必须以真实动作为输入，这种规模往往不可行。

Figure 10: Learning to simulate deformable objects: we show frames from a ten step trajectory in the model, taking the same action. Genie is capable of learning the physical properties of objects such as bags of chips.

图 10：学习模拟可变形物体：展示模型中十步轨迹的若干帧，始终采取同一动作。Genie 能学到薯片袋等物体的物理性质。

Platformers-trained model Figure 9 showcases examples of our model’s generations prompted from OOD images, including (top row) images generated from Imagen2 (Ho et al., 2022a; van den Oord et al., ), (second row) hand-drawn sketches and (bottom row) real-world photos. Genie is able to bring these imagined worlds to life, as we see game-like behaviour when interacting with each example. We showcase more generations by our model in Appendix A, additionally highlighting the consistency of the latent actions.

Platformers 训练模型 图 9 展示由 OOD 图像提示得到的生成示例，包括（上排）Imagen2 生成图像（Ho et al., 2022a; van den Oord et al., ）、（中排）手绘草图与（下排）真实照片。Genie 能把这些想象世界“激活”，与各示例交互时可见类游戏行为。附录 A 有更多生成结果，并进一步突出潜在动作的一致性。

Figure 11: Emulating parallax, a common feature in platformer games. From this initial text-generated image, the foreground moves more than the near and far middle ground, while the background moves only slightly.

图 11：模拟视差，平台跳跃游戏的常见特征。从这张文本生成的初始图像出发，前景移动多于近与远中景，而背景几乎不动。

Another emergent capability of our model is its ability to understand 3D scenes and emulate parallax, which is commonly seen in platformer games. In Figure 11 we show an image generated by Imagen2, where taking a latent action moves the foreground at a different rate to the background (as indicated by the length of different colored arrows).

模型的另一涌现能力是理解三维场景并模拟视差——平台跳跃游戏中常见。图 11 给出 Imagen2 生成的图像：采取潜在动作时，前景与背景以不同速率移动（不同颜色箭头长度所示）。

Figure 12: Controllable, consistent latent actions in Robotics: trajectories beginning from three different starting frames from our Robotics dataset. Each column shows the resulting frame from taking the same latent action five times. Despite training without action labels, the same actions are consistent across varied prompt frames and have semantic meaning: down, up and left.

图 12：机器人设定中可控且一致的潜在动作：从 Robotics 数据集中三个不同起始帧出发的轨迹。每列展示连续执行同一潜在动作五次后的结果帧。尽管训练无动作标签，相同动作在不同提示帧间保持一致，并具有语义：下、上与左。

Robotics-trained model We trained a 2.5B-parameter model on the Robotics dataset using the same hyperparameters found to be best on Platformers, achieving an FVD of 82.7 on the test split. As shown in Figure 12, this model successfully learns distinct and consistent actions from video data, requiring neither text nor action labels (as in e.g. Yang et al. (2023)). Notably, our model learns not only the controls of the robotic arm but also the interactions and deformations of various objects (Figure 10). We believe this shows our approach presents a path to using larger video datasets from the Internet to create a foundational world model for robotics, with low-level controllable simulation that could be used for a variety of applications.

Robotics 训练模型 我们用在 Platformers 上表现最佳的超参，在 Robotics 数据集上训练 2.5B 参数模型，测试集 FVD 为 82.7。如图 12 所示，该模型仅从视频成功学到彼此区分且一致的动作，无需文本或动作标签（对比如 Yang et al. (2023)）。值得注意的是，模型不仅学会机械臂控制，还学会多种物体的交互与形变（图 10）。我们认为这表明：利用更大规模互联网视频数据，为机器人打造带低层可控仿真的基础世界模型，并服务多种应用，是一条可行路径。

3.3 Training Agents

3.3 训练智能体

We believe Genie could one day be used as a foundation world model for training generalist agents. In Figure 13 we show that the model can already be used for generating diverse trajectories in unseen RL environments given starting frames. We further investigate if latent actions learnt from Internet videos can be used for imitating behaviors from unseen videos. We use a frozen LAM to label a sequence of expert videos from a target environment with discrete latent actions and then train a policy that predicts the likelihood of the expert taking a latent action given an observation. We then use a small dataset with expert ground-truth actions for mapping latent to real actions (see Appendix E for more details).

我们相信 Genie 有朝一日可成为训练通才智能体的基础世界模型。图 13 表明：给定起始帧，模型已能在未见过的 RL 环境中生成多样轨迹。我们进一步考察：从互联网视频学到的潜在动作，能否用于模仿未见视频中的行为。我们用冻结的 LAM 为目标环境中的专家视频序列标注离散潜在动作，再训练策略，预测专家在给定观测下采取某潜在动作的概率。随后用一小份带真实专家动作的数据，把潜在动作映射到真实动作（细节见附录 E）。

Figure 13: Playing from RL environments: Genie can generate diverse trajectories given an image of an unseen RL environment.

图 13：从 RL 环境游玩：给定未见 RL 环境的图像，Genie 可生成多样轨迹。

We evaluate in both hard and easy settings of a procedurally generated 2D-platformer environment, CoinRun (Cobbe et al., 2020), and compare against an oracle behavioral cloning (BC) model that has access to expert actions as an upper bound, and a random agent as a lower bound (Figure 14). The LAM-based policy achieves the same score as the oracle given as few as 200 expert samples to adapt, despite almost certainly never seeing CoinRun before. This provides evidence that the learnt latent actions are consistent and meaningful for transfer, as the mapping from latent to real contains no information about the current observation.

我们在程序生成的 2D 平台跳跃环境 CoinRun（Cobbe et al., 2020）的困难与简单设定下评估，并与能访问专家动作的 oracle 行为克隆（BC）模型（上界）以及随机智能体（下界）比较（图 14）。基于 LAM 的策略在仅用约 200 条专家样本适配时，即可达到与 oracle 相同的分数——尽管几乎肯定从未见过 CoinRun。这为所学潜在动作可迁移且有意义提供了证据：因为从潜在到真实的映射并不包含当前观测信息。

Figure 14: BC results. Mean percentage of levels solved out of 100 samples, averaged over $5$ seeds with $95\%$ confidence intervals.

图 14：BC 结果。在 100 次采样中平均通关关卡百分比，对 $5$ 个种子取平均，并给出 $95\%$ 置信区间。

3.4 Ablation Studies

3.4 消融实验

Design choices for latent action model In designing our latent action model, we carefully considered the type of input to use. While we ultimately chose to use the original images (pixels), we evaluated this choice against the alternative of using tokenized images (replacing x with z in Figure 4). We refer to this alternative approach as the “token-input" model (see Table 2).

潜在动作模型的设计选择 设计潜在动作模型时，我们仔细考虑输入类型。最终选择原始图像（像素），并与使用分词图像的替代方案对比（图 4 中用 z 替换 x）。后者称为 “token-input” 模型（见表 2）。

While this model achieved a slightly lower FVD score on the Platformers dataset, it did not maintain this advantage on the Robotics dataset. More importantly, in both environments, the token-input model exhibited worse controllability (as measured by $\Delta_{t}\text{PSNR}$). This suggests that some information about video dynamics and movement might have been lost during tokenization, and as a result it is beneficial for the latent action model to take in raw videos as input.

该模型在 Platformers 上 FVD 略低，但在 Robotics 上未保持优势。更重要的是，两环境中 token-input 模型的可控性（以 $\Delta_{t}\text{PSNR}$ 衡量）都更差。这表明分词过程可能丢失部分关于视频动态与运动的信息，因此潜在动作模型以原始视频为输入更有利。

	Dataset	#Params	FVD ($\downarrow$)	$\Delta_{t}\text{PSNR}$($\uparrow$)
Token-input	Platformers	2.3B	38.8	1.33
Pixel-input (Genie)	Platformers	2.5B	40.1	1.91
Token-input	Robotics	1B	257.8	1.65
Pixel-input (Genie)	Robotics	1B	136.4	2.07

Table 2: Latent action model input ablation. We see that Genie achieves higher controllability.

	Dataset	#Params	FVD ($\downarrow$)	$\Delta_{t}\text{PSNR}$($\uparrow$)
Token-input	Platformers	2.3B	38.8	1.33
Pixel-input (Genie)	Platformers	2.5B	40.1	1.91
Token-input	Robotics	1B	257.8	1.65
Pixel-input (Genie)	Robotics	1B	136.4	2.07

表 2：潜在动作模型输入消融。可见 Genie 取得更高可控性。

Tokenizer architecture ablations We compare the performance of three choices of tokenizers, including 1) (spatial-only) ViT, 2) (spatial-temporal) ST-ViViT and 3) (spatial-temporal) C-ViViT (Table 3). For comparison we use similar number of parameters for all tokenizers, with patch size 10, batch size 128 and sequence length 16. We then train the same dynamics and latent action model on these three different tokenizers, and report their FVD as well as $\Delta_{t}\text{PSNR}$.

分词器架构消融 我们比较三种分词器：1）仅空间的 ViT；2）时空 ST-ViViT；3）时空 C-ViViT（表 3）。对比时各分词器参数量相近，patch 大小 10、批大小 128、序列长度 16。随后在三种分词器上训练相同的动力学与潜在动作模型，报告 FVD 与 $\Delta_{t}\text{PSNR}$。

	#Params	Memory	FVD ($\downarrow$)	$\Delta_{t}\text{PSNR}$($\uparrow$)
ViT	230M	0.3GB	114.5	1.39
C-ViViT (Villegas et al., 2023)	225M	1.6GB	272.7	1.37
ST-ViViT (ours)	205M	0.9GB	81.4	1.66

Table 3: Tokenizer architecture ablation: Our ST-ViViT architecture results in the best performing tokenizer.

	#Params	Memory	FVD ($\downarrow$)	$\Delta_{t}\text{PSNR}$($\uparrow$)
ViT	230M	0.3GB	114.5	1.39
C-ViViT (Villegas et al., 2023)	225M	1.6GB	272.7	1.37
ST-ViViT (ours)	205M	0.9GB	81.4	1.66

表 3：分词器架构消融：我们的 ST-ViViT 架构得到表现最佳的分词器。

Our proposed ST-ViViT architecture provides both improved video generation (FVD) and $\Delta_{t}\text{PSNR}$, for a reasonable trade-off in memory, as compared to to C-ViViT and the spatial-only ViT. This demonstrates its ability to generate videos of high fidelity and controllability, respectively. While C-ViViT employs a full space-time attention mechanism, resulting in significantly higher memory consumption compared to the other two architectures at the same parameter count, this does not translate to improved performance. In fact, C-ViViT exhibits a tendency towards overfitting, necessitating strong regularization during training, which might explain its considerably lower performance.

相对 C-ViViT 与仅空间 ViT，所提出的 ST-ViViT 在内存代价可接受的前提下，同时改善视频生成（FVD）与 $\Delta_{t}\text{PSNR}$，分别体现高保真与高可控。C-ViViT 采用完整时空注意力，同参数量下内存显著更高，却未带来更好表现；事实上它更易过拟合，训练需强正则，这或可解释其明显更差的结果。

4 4. Related Work

4 相关工作

World models Generative interactive environments can be considered a class of World Models (Ha and Schmidhuber, 2018; Oh et al., 2015), which enable next-frame prediction that is conditioned on action inputs (Nunes et al., 2020; Hafner et al., 2020; Hafner et al., 2021; Micheli et al., 2023; Robine et al., 2023; Kim et al., 2020; Kim et al., 2021; Bamford and Lucas, 2020; Chiappa et al., 2017; Pan et al., 2022; Eslami et al., 2018). Such models can be useful for training agents, as they can be used for learning policies without direct environment experience at agent training time. However, learning the models themselves typically requires action-conditioned data obtained directly from the environment. In contrast, our approach seeks to learn a world model in an unsupervised fashion from videos alone. Recently, there has been renewed emphasis on scaling world models. GAIA-1 (Hu et al., 2023) and UniSim (Yang et al., 2023) learn world models for autonomous driving and robotic manipulation respectively. These approaches require both text and action labels, while we focus on training from video-only data from publicly available Internet videos.

世界模型 生成式交互环境可视为一类世界模型（Ha and Schmidhuber, 2018; Oh et al., 2015），能在动作条件下做下一帧预测（Nunes et al., 2020; Hafner et al., 2020; Hafner et al., 2021; Micheli et al., 2023; Robine et al., 2023; Kim et al., 2020; Kim et al., 2021; Bamford and Lucas, 2020; Chiappa et al., 2017; Pan et al., 2022; Eslami et al., 2018）。这类模型有助于训练智能体：智能体训练期不必直接与环境交互即可学策略。但学习模型本身通常需要直接从环境获得的动作条件数据。相反，我们的方法试图仅从视频无监督地学习世界模型。近期世界模型缩放再度受关注。GAIA-1（Hu et al., 2023）与 UniSim（Yang et al., 2023）分别为自动驾驶与机器人操作学习世界模型。这些方法需要文本与动作标签，而我们聚焦于仅用公开互联网视频训练。

Video models Our work is related to video models, which typically condition on initial frames (or text) and predict the remaining frames in a video (Kalchbrenner et al., 2017; Clark et al., 2019; Finn et al., 2016; Luc et al., 2020; Lotter et al., 2017; Yan et al., 2021; Blattmann et al., 2023b; Walker et al., 2021; Le Moing et al., 2021; Höppe et al., 2022; Singer et al., 2023; Ho et al., 2022a; Ho et al., 2022b; Brooks et al., 2024; Yu et al., 2023). Our approach most resembles recent transformer based models such as Phenaki (Villegas et al., 2023), TECO (Yan et al., 2023) and MaskViT (Gupta et al., 2023), as we use MaskGIT (Chang et al., 2022) and an ST-Transformer (Xu et al., 2020) over tokenized images. While video models are becoming increasingly controllable (e.g. (Huang et al., 2022)), we seek a more agentic goal and explicitly learn a latent action space from data, allowing users or agents to “play” the model using latent action-conditioned predictions.

视频模型 本工作与视频模型相关：通常以初始帧（或文本）为条件，预测视频其余帧（Kalchbrenner et al., 2017; Clark et al., 2019; Finn et al., 2016; Luc et al., 2020; Lotter et al., 2017; Yan et al., 2021; Blattmann et al., 2023b; Walker et al., 2021; Le Moing et al., 2021; Höppe et al., 2022; Singer et al., 2023; Ho et al., 2022a; Ho et al., 2022b; Brooks et al., 2024; Yu et al., 2023）。我们的方法最接近近期基于 Transformer 的模型，如 Phenaki（Villegas et al., 2023）、TECO（Yan et al., 2023）与 MaskViT（Gupta et al., 2023），因为我们在分词图像上使用 MaskGIT（Chang et al., 2022）与 ST-Transformer（Xu et al., 2020）。尽管视频模型正变得更可控（如 Huang et al., 2022），我们追求更偏智能体的目标，并从数据中显式学习潜在动作空间，使用户或智能体能用潜在动作条件预测来“玩”模型。

Playable Video Generation Genie generalizes beyond Playable Video Generation (PVG) (Menapace et al., 2021), where latent actions are used for controlling world models learnt directly from videos (Menapace et al., 2021; Menapace et al., 2022). In contrast to Genie, PVG considers domain-specific static examples, rather than generating entirely new environments via prompting. Thus, scaling beyond this setting required non-trivial architectural changes, dropping inductive biases in exchange for a general method.

可玩视频生成 Genie 推广了可玩视频生成（PVG）（Menapace et al., 2021）：后者用潜在动作控制直接从视频学到的世界模型（Menapace et al., 2021; Menapace et al., 2022）。与 Genie 不同，PVG 面向领域特定的静态示例，而非通过提示生成全新环境。因此要超越该设定，需要非平凡的架构改动，用更通用的方法换掉归纳偏置。

Environment generation Our work is also related to Procedural Content Generation (Risi and Togelius, 2020a; Risi and Togelius, 2020b, PCG, e.g.) where machine learning has proven highly effective for generating game levels (Summerville et al., 2018), recently via language models that directly write game code (Sudhakaran et al., 2023; Todd et al., 2023). Language models themselves can also be considered to be interactive environments (Wong et al., 2023), albeit lacking a visual component. By contrast in our setting the levels can be learnt and generated directly from pixels, which enables us to utilize the diversity of Internet video data.

环境生成 本工作也与程序化内容生成相关（Risi and Togelius, 2020a; Risi and Togelius, 2020b, 例如 PCG）：机器学习已被证明能有效生成游戏关卡（Summerville et al., 2018），近来还有语言模型直接编写游戏代码（Sudhakaran et al., 2023; Todd et al., 2023）。语言模型本身也可视为交互环境（Wong et al., 2023），只是缺少视觉成分。相比之下，我们的设定可直接从像素学习并生成关卡，从而利用互联网视频数据的多样性。

Training agents with latent actions Prior works have used latent actions for imitation from observation (Edwards et al., 2019), planning (Rybkin* et al., 2019) and pre-training RL agents (Ye et al., 2022; Schmidt and Jiang, 2024). These approaches have similar objectives to our latent action model, though have not been applied at scale. VPT (Baker et al., 2022) is a recent approach that uses an inverse dynamics model learnt from human-provided action labeled data, to label Internet-scale videos with actions that can then be used for training a policy. We showed, in contrast, that we can use latent actions learnt from Internet videos to infer policies for arbitrary environments, avoiding the need for ground-truth actions that are costly and may not generalize.

用潜在动作训练智能体 先前工作已用潜在动作做从观测模仿（Edwards et al., 2019）、规划（Rybkin* et al., 2019）以及预训练 RL 智能体（Ye et al., 2022; Schmidt and Jiang, 2024）。这些方法与我们的潜在动作模型目标相近，但尚未大规模应用。VPT（Baker et al., 2022）近期用从人工提供的动作标注数据学到的逆动力学模型，为互联网规模视频打上动作标签，再用于训练策略。我们则表明：可用从互联网视频学到的潜在动作，为任意环境推断策略，从而避免昂贵且未必可泛化的真实动作需求。

5 5. Conclusion and Future Work

5 结论与未来工作

We proposed Genie, a new form of generative AI that enables anyone, even children, to dream up, create, and step into generated worlds as we can with human-designed simulated environments. Genie can be prompted to generate a diverse set of interactive and controllable environments despite training from video-only data.

我们提出 Genie：一种新的生成式 AI 形态，使任何人——甚至儿童——都能像使用人工设计的仿真环境一样，想象、创造并踏入生成世界。尽管仅用视频数据训练，Genie 仍可被提示生成多样、可交互且可控的环境。

There are clear improvements that can be made to the model. Genie inherits some of the weaknesses of other autoregressive transformer models, and can hallucinate unrealistic futures. And while we have made progress with spatiotemporal representations, we are still limited to 16 frames of memory which makes it challenging to get consistent environments over long horizons. Finally, Genie currently operates around 1FPS and requires future advances to achieve an efficient frame rate for interaction.

模型仍有明确改进空间。Genie 继承了其他自回归 Transformer 的部分弱点，可能幻觉出不真实的未来。尽管时空表示已有进展，记忆仍限于 16 帧，难以在长时程上保持环境一致。此外 Genie 目前约 1FPS，需要后续进展才能达到适合交互的帧率。

Still, we believe Genie opens up vast potential for future research. Given its generality, the model could be trained from an even larger proportion of Internet videos to simulate diverse, realistic, and imagined environments. Furthermore, we only briefly touched upon the capabilities of using Genie for training agents, but given that the lack of rich and diverse environments is one of the key limitations in RL, we could unlock new paths to creating more generally capable agents.

尽管如此，我们相信 Genie 为未来研究打开广阔空间。凭其通用性，模型可从更大比例的互联网视频训练，以仿真多样、真实与想象中的环境。此外我们只简要触及用 Genie 训练智能体的能力；而丰富多样环境的匮乏正是 RL 的关键瓶颈之一，这或能解锁通向更通才智能体的新路径。

Broader Impact

更广泛影响

Societal Impact Genie could enable a large amount of people to generate their own game-like experiences. This could be positive for those who wish to express their creativity in a new way, for example children who could design and step into their own imagined worlds. We also recognize that with significant advances, it will be critical to explore the possibilities of using this technology to amplify existing human game generation and creativity—and empowering relevant industries to utilize Genie to enable their next generation of playable world development.

社会影响 Genie 可能使大量人群生成自己的类游戏体验。对希望以新方式表达创造力的人——例如儿童设计并踏入想象世界——这可能是积极的。我们也认识到，随着能力显著提升，必须探索如何用该技术放大既有人类游戏创作与创意，并赋能相关产业用 Genie 推动下一代可玩世界开发。

Training Data and Weights: We have chosen not to release the trained model checkpoints, the model’s training dataset, or examples from that data to accompany this paper or the website. We would like to have the opportunity to further engage with the research (and video game) community and to ensure that any future such releases are respectful, safe and responsible.

训练数据与权重：我们选择不随论文或网站发布训练好的模型检查点、训练数据集或其中样本。我们希望有机会进一步与研究（以及电子游戏）社区沟通，确保未来任何此类发布都尊重、安全且负责任。

Reproducibility: We understand that it may be challenging for researchers with fewer computational to reproduce our main results. In order to mitigate this issue, we describe a smaller scale, fully reproducible example in Appendix F that can run on a single mid-range TPU (or GPU). Given that many design choices translate between the two settings, we believe this will make it possible for the broader community to investigate future architectural improvements as well as additional research directions resulting from our work.

可复现性：我们理解算力较少的研究者可能难以复现主要结果。为缓解这一问题，附录 F 描述了一个更小规模、可完全复现的示例，可在单块中端 TPU（或 GPU）上运行。鉴于许多设计选择可在两种设定间迁移，我们相信这将使更广泛社区能够探究后续架构改进，以及由本工作引出的其他研究方向。

Acknowledgements

致谢

We thank Mateusz Malinowski, Philip Ball and Louis Kirsch for reviewing a draft of our paper; Cassidy Hardin, David Bridson, Eric Lau, Lars Lowe Sjoesund, Lucas Smaira and Bernardo Avila Pires for help with our Platformers dataset; Ruben Villegas for valuable discussions on our video model training and evaluation; and Adrian Bolton, Rushil Mistry, Hannah Openshaw, Zoubin Ghahramani, Raia Hadsell, Koray Kavukcuoglu, Daan Wierstra, Doina Precup and Ed Hirst for strategic advice and guidance. We make use of the DeepMind Jax ecosystem (Babuschkin et al., 2010) and specifically thank Andy Brock for building the internal framework we used for our model training and Arthur Brussee who provided an initial interface that enabled us to “play” our models. Finally, thank you to Seneca and Caspian Clune for their creative sketches, potentially making them the youngest ever game designers.

感谢 Mateusz Malinowski、Philip Ball 与 Louis Kirsch 审阅论文初稿；感谢 Cassidy Hardin、David Bridson、Eric Lau、Lars Lowe Sjoesund、Lucas Smaira 与 Bernardo Avila Pires 协助 Platformers 数据集；感谢 Ruben Villegas 就视频模型训练与评估展开有价值讨论；感谢 Adrian Bolton、Rushil Mistry、Hannah Openshaw、Zoubin Ghahramani、Raia Hadsell、Koray Kavukcuoglu、Daan Wierstra、Doina Precup 与 Ed Hirst 提供战略建议与指导。我们使用 DeepMind Jax 生态（Babuschkin et al., 2010），并特别感谢 Andy Brock 构建用于模型训练的内部框架，以及 Arthur Brussee 提供最初让我们能够“玩”模型的接口。最后感谢 Seneca 与 Caspian Clune 富有创意的草图——或许使他们成为有史以来最年轻的游戏设计师。

Author Contributions

作者贡献

We list authors alphabetically by last name. Please direct all correspondence to Ashley Edwards (edwardsashley@google.com) and Jack Parker-Holder (jparkerholder@google.com).

Core Contributors

核心贡献者

• • Jake Bruce: project leadership, video tokenizer research, action model research, dynamics model research, scaling, model demo, infrastructure
• • Michael Dennis: dynamics model research, scaling, metrics, model demo, infrastructure
• • Ashley Edwards: genie concept, project leadership, action model research, agent training, model demo
• • Edward Hughes: dynamics model research, infrastructure
• • Matthew Lai: dataset curation, infrastructure
• • Aditi Mavalankar: action model research, metrics, agent training
• • Jack Parker-Holder: genie concept, project leadership, dynamics model research, scaling, dataset curation
• • Yuge (Jimmy) Shi: video tokenizer research, dynamics model research, dataset curation, metrics
• • Richie Steigerwald: dataset curation, metrics

• • Jake Bruce：项目领导、视频分词器研究、动作模型研究、动力学模型研究、缩放、模型演示、基础设施
• • Michael Dennis：动力学模型研究、缩放、指标、模型演示、基础设施
• • Ashley Edwards：Genie 构想、项目领导、动作模型研究、智能体训练、模型演示
• • Edward Hughes：动力学模型研究、基础设施
• • Matthew Lai：数据集策展、基础设施
• • Aditi Mavalankar：动作模型研究、指标、智能体训练
• • Jack Parker-Holder：Genie 构想、项目领导、动力学模型研究、缩放、数据集策展
• • Yuge (Jimmy) Shi：视频分词器研究、动力学模型研究、数据集策展、指标
• • Richie Steigerwald：数据集策展、指标

Partial Contributors and Advisors

部分贡献者与顾问

• • Chris Apps: project management
• • Yusuf Aytar: technical advice
• • Sarah Bechtle: technical advice
• • Feryal Behbahani: strategic advice
• • Stephanie Chan: technical advice
• • Jeff Clune: technical advice, strategic advice
• • Lucy Gonzalez: project management
• • Nicolas Heess: strategic advice
• • Simon Osindero: technical advice
• • Sherjil Ozair: technical advice
• • Scott Reed: technical advice
• • Jingwei Zhang: technical advice
• • Konrad Zolna: scaling, technical advice

• • Chris Apps：项目管理
• • Yusuf Aytar：技术建议
• • Sarah Bechtle：技术建议
• • Feryal Behbahani：战略建议
• • Stephanie Chan：技术建议
• • Jeff Clune：技术建议、战略建议
• • Lucy Gonzalez：项目管理
• • Nicolas Heess：战略建议
• • Simon Osindero：技术建议
• • Sherjil Ozair：技术建议
• • Scott Reed：技术建议
• • Jingwei Zhang：技术建议
• • Konrad Zolna：缩放、技术建议

Sponsors

赞助人

• • Nando de Freitas: strategic advice
• • Tim Rocktäschel: genie concept, project leadership
• • Satinder Singh: strategic advice

• • Nando de Freitas：战略建议
• • Tim Rocktäschel：Genie 构想、项目领导
• • Satinder Singh：战略建议

References

参考文献

Babuschkin et al. (2010) I. Babuschkin, K. Baumli, A. Bell, S. Bhupatiraju, J. Bruce, P. Buchlovsky, D. Budden, T. Cai, A. Clark, I. Danihelka, et al. The deepmind jax ecosystem, 2020. URL http://github. com/deepmind , 2010.

Bain et al. (2021) M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 1708–1718, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society. 10.1109/ICCV48922.2021.00175 .

Baker et al. (2022) B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems , 35:24639–24654, 2022.

Bamford and Lucas (2020) C. Bamford and S. M. Lucas. Neural game engine: Accurate learning ofgeneralizable forward models from pixels. In Conference on Games , 2020.

Bauer et al. (2023) J. Bauer, K. Baumli, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg, M. Chang, N. Clay, A. Collister, V. Dasagi, L. Gonzalez, K. Gregor, E. Hughes, S. Kashem, M. Loks-Thompson, H. Openshaw, J. Parker-Holder, S. Pathak, N. Perez-Nieves, N. Rakicevic, T. Rocktäschel, Y. Schroecker, S. Singh, J. Sygnowski, K. Tuyls, S. York, A. Zacherl, and L. M. Zhang. Human-timescale adaptation in an open-ended task space. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 1887–1935. PMLR, 23–29 Jul 2023.

Blattmann et al. (2023a) A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, V. Jampani, and R. Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023a.

Blattmann et al. (2023b) A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 22563–22575, 2023b.

Brohan et al. (2023) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems , 2023.

Brooks et al. (2024) T. Brooks, B. Peebles, C. Homes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. W. Y. Ng, R. Wang, and A. Ramesh. Video generation models as world simulators. 2024. URL https://openai.com/research/video-generation-models-as-world-simulators .

Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020.

Chang et al. (2022) H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11315–11325, June 2022.

Chiappa et al. (2017) S. Chiappa, S. Racaniere, D. Wierstra, and S. Mohamed. Recurrent environment simulators. In International Conference on Learning Representations , 2017.

Clark et al. (2019) A. Clark, J. Donahue, and K. Simonyan. Efficient video generation on complex datasets. CoRR , abs/1907.06571, 2019. URL http://arxiv.org/abs/1907.06571 .

Clune (2019) J. Clune. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv:1905.10985 , 2019.

Cobbe et al. (2020) K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning , pages 2048–2056, 2020.

Dehghani et al. (2023) M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, R. Jenatton, L. Beyer, M. Tschannen, A. Arnab, X. Wang, C. Riquelme Ruiz, M. Minderer, J. Puigcerver, U. Evci, M. Kumar, S. V. Steenkiste, G. F. Elsayed, A. Mahendran, F. Yu, A. Oliver, F. Huot, J. Bastings, M. Collier, A. A. Gritsenko, V. Birodkar, C. N. Vasconcelos, Y. Tay, T. Mensink, A. Kolesnikov, F. Pavetic, D. Tran, T. Kipf, M. Lucic, X. Zhai, D. Keysers, J. J. Harmsen, and N. Houlsby. Scaling vision transformers to 22 billion parameters. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 7480–7512. PMLR, 23–29 Jul 2023.

Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=YicbFdNTTy .

Edwards et al. (2019) A. Edwards, H. Sahni, Y. Schroecker, and C. Isbell. Imitating latent policies from observation. In International conference on machine learning , pages 1755–1763. PMLR, 2019.

Eslami et al. (2018) S. M. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, I. Danihelka, K. Gregor, D. P. Reichert, L. Buesing, T. Weber, O. Vinyals, D. Rosenbaum, N. Rabinowitz, H. King, C. Hillier, M. Botvinick, D. Wierstra, K. Kavukcuoglu, and D. Hassabis. Neural scene representation and rendering. Science , 360(6394):1204–1210, 2018. 10.1126/science.aar6170 .

Esser et al. (2023) P. Esser, J. Chiu, P. Atighehchian, J. Granskog, and A. Germanidis. Structure and content-guided video synthesis with diffusion models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Finn et al. (2016) C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS’16, page 64–72, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.

Gupta et al. (2023) A. Gupta, S. Tian, Y. Zhang, J. Wu, R. Martín-Martín, and L. Fei-Fei. Maskvit: Masked visual pre-training for video prediction. In The Eleventh International Conference on Learning Representations , 2023.

Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems , NeurIPS’18, pages 2455–2467, 2018.

Hafner et al. (2020) D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations , 2020.

Hafner et al. (2021) D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. In International Conference on Learning Representations , 2021.

He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770–778, 2016. 10.1109/CVPR.2016.90 .

Henry et al. (2020) A. Henry, P. R. Dachapally, S. S. Pawar, and Y. Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 4246–4253, Online, Nov. 2020. Association for Computational Linguistics. 10.18653/v1/2020.findings-emnlp.379 .

Ho et al. (2022a) J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen video: High definition video generation with diffusion models, 2022a.

Ho et al. (2022b) J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 8633–8646. Curran Associates, Inc., 2022b.

Hong et al. (2022) W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 , 2022.

Hong et al. (2023) W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=rB6TpjAuSRy .

Höppe et al. (2022) T. Höppe, A. Mehrjou, S. Bauer, D. Nielsen, and A. Dittadi. Diffusion models for video prediction and infilling. Transactions on Machine Learning Research , 2022. ISSN 2835-8856.

Hu et al. (2023) A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado. Gaia-1: A generative world model for autonomous driving, 2023.

Huang et al. (2022) J. Huang, Y. Jin, K. M. Yi, and L. Sigal. Layered controllable video generation. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVI , page 546–564, Berlin, Heidelberg, 2022. Springer-Verlag. ISBN 978-3-031-19786-4.

Jouppi et al. (2020) N. P. Jouppi, D. H. Yoon, G. Kurian, S. Li, N. Patil, J. Laudon, C. Young, and D. Patterson. A domain-specific supercomputer for training deep neural networks. Communications of the ACM , 63(7):67–78, 2020.

Kalashnikov et al. (2018) D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293 , 2018.

Kalchbrenner et al. (2017) N. Kalchbrenner, A. van den Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, and K. Kavukcuoglu. Video pixel networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1771–1779. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/kalchbrenner17a.html .

Kapturowski et al. (2018) S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations , 2018.

Kim et al. (2020) S. W. Kim, Y. Zhou, J. Philion, A. Torralba, and S. Fidler. Learning to simulate dynamic environments with gamegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Kim et al. (2021) S. W. Kim, J. Philion, A. Torralba, and S. Fidler. Drivegan: Towards a controllable high-quality neural simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5820–5829, June 2021.

Le Moing et al. (2021) G. Le Moing, J. Ponce, and C. Schmid. Ccvs: Context-aware controllable video synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 14042–14055. Curran Associates, Inc., 2021.

Lotter et al. (2017) W. Lotter, G. Kreiman, and D. Cox. Deep predictive coding networks for video prediction and unsupervised learning. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=B1ewdt9xe .

Luc et al. (2020) P. Luc, A. Clark, S. Dieleman, D. de Las Casas, Y. Doron, A. Cassirer, and K. Simonyan. Transformation-based adversarial video prediction on large-scale data. CoRR , abs/2003.04035, 2020.

Menapace et al. (2021) W. Menapace, S. Lathuilière, S. Tulyakov, A. Siarohin, and E. Ricci. Playable video generation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 10061–10070. Computer Vision Foundation / IEEE, 2021.

Menapace et al. (2022) W. Menapace, S. Lathuilière, A. Siarohin, C. Theobalt, S. Tulyakov, V. Golyanik, and E. Ricci. Playable environments: Video manipulation in space and time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Micheli et al. (2023) V. Micheli, E. Alonso, and F. Fleuret. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations , 2023.

Nunes et al. (2020) M. S. Nunes, A. Dehban, P. Moreno, and J. Santos-Victor. Action-conditioned benchmarking of robotic video prediction models: a comparative study. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 8316–8322, 2020. 10.1109/ICRA40945.2020.9196839 .

Oh et al. (2015) J. Oh, X. Guo, H. Lee, R. Lewis, and S. Singh. Action-conditional video prediction using deep networks in atari games. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 , NIPS’15, page 2863–2871, Cambridge, MA, USA, 2015. MIT Press.

Open Ended Learning Team et al. (2021) Open Ended Learning Team, A. Stooke, A. Mahajan, C. Barros, C. Deck, J. Bauer, J. Sygnowski, M. Trebacz, M. Jaderberg, M. Mathieu, N. McAleese, N. Bradley-Schmieg, N. Wong, N. Porcel, R. Raileanu, S. Hughes-Fitt, V. Dalibard, and W. M. Czarnecki. Open-ended learning leads to generally capable agents. CoRR , abs/2107.12808, 2021.

Oquab et al. (2023) M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Pan et al. (2022) M. Pan, X. Zhu, Y. Wang, and X. Yang. Iso-dream: Isolating and leveraging noncontrollable visual dynamics in world models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 23178–23191. Curran Associates, Inc., 2022.

Radford et al. (2018) A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. 2018.

Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Rajbhandari et al. (2020) S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1–16. IEEE, 2020.

Ramesh et al. (2021) A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8821–8831. PMLR, 18–24 Jul 2021.

Ramesh et al. (2022) A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents, 2022.

Reed et al. (2022) S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-maron, M. Giménez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent. Transactions on Machine Learning Research , 2022. ISSN 2835-8856. Featured Certification, Outstanding Certification.

Risi and Togelius (2020a) S. Risi and J. Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence , 2, 08 2020a. 10.1038/s42256-020-0208-z .

Risi and Togelius (2020b) S. Risi and J. Togelius. Procedural content generation: From automatically generating game levels to increasing generality in machine learning. Nature , 2020b.

Robine et al. (2023) J. Robine, M. Höftmann, T. Uelwer, and S. Harmeling. Transformer-based world models are happy with 100k interactions. In The Eleventh International Conference on Learning Representations , 2023.

Rombach et al. (2022) R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684–10695, June 2022.

Rybkin* et al. (2019) O. Rybkin*, K. Pertsch*, K. G. Derpanis, K. Daniilidis, and A. Jaegle. Learning what you can do before doing anything. In International Conference on Learning Representations , 2019.

Saharia et al. (2022) C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, R. Gontijo-Lopes, B. K. Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

Schmidt and Jiang (2024) D. Schmidt and M. Jiang. Learning to act without actions. In The Twelfth International Conference on Learning Representations , 2024.

Shoeybi et al. (2019) M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR , abs/1909.08053, 2019. URL http://arxiv.org/abs/1909.08053 .

Singer et al. (2023) U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, D. Parikh, S. Gupta, and Y. Taigman. Make-a-video: Text-to-video generation without text-video data. In The Eleventh International Conference on Learning Representations , 2023.

Sudhakaran et al. (2023) S. Sudhakaran, M. González-Duque, C. Glanois, M. Freiberger, E. Najarro, and S. Risi. Prompt-guided level generation. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation , pages 179–182, 2023.

Summerville et al. (2018) A. Summerville, S. Snodgrass, M. Guzdial, C. Holmgård, A. K. Hoover, A. Isaksen, A. Nealen, and J. Togelius. Procedural content generation via machine learning (PCGML). IEEE Trans. Games , 10(3):257–270, 2018.

Todd et al. (2023) G. Todd, S. Earle, M. U. Nasir, M. C. Green, and J. Togelius. Level generation through large language models. In Proceedings of the 18th International Conference on the Foundations of Digital Games , pages 1–8, 2023.

Torabi et al. (2018) F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954 , 2018.

Unterthiner et al. (2019) T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. FVD: A new metric for video generation, 2019.

(72) A. van den Oord, A. Razavi, B. Uria, Çağlar Ünlü, C. Nash, C. Wolff, C. Durkan, D. Ding, D. Górny, E. Gladchenko, F. Riedel, H. Qi, J. Kelly, J. Bauer, J. Donahue, J. Zhang, M. Malinowski, M. Bińkowski, P. Luc, R. Riachi, R. Strudel, T. P. I. Sander Dieleman, Y. Ganin, and Z. Eaton-Rosen. Imagen 2. URL https://deepmind.google/technologies/imagen-2/ .

van den Oord et al. (2017) A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS’17, page 6309–6318, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.

Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998–6008, 2017.

Villegas et al. (2023) R. Villegas, M. Babaeizadeh, P.-J. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, J. Kunze, and D. Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations , 2023.

Walker et al. (2021) J. C. Walker, A. Razavi, and A. van den Oord. Predicting video with VQVAE, 2021.

Wang et al. (2023) Y. Wang, Y. He, Y. Li, K. Li, J. Yu, X. Ma, X. Chen, Y. Wang, P. Luo, Z. Liu, Y. Wang, L. Wang, and Y. Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation, 2023.

Wong et al. (2023) L. Wong, G. Grand, A. K. Lew, N. D. Goodman, V. K. Mansinghka, J. Andreas, and J. B. Tenenbaum. From word models to world models: Translating from natural language to the probabilistic language of thought, 2023.

Wu et al. (2022) C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang, and N. Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In European conference on computer vision , pages 720–736. Springer, 2022.

Xu et al. (2020) M. Xu, W. Dai, C. Liu, X. Gao, W. Lin, G.-J. Qi, and H. Xiong. Spatial-temporal transformer networks for traffic flow forecasting. arXiv preprint arXiv:2001.02908 , 2020.

Yan et al. (2021) W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas. Videogpt: Video generation using vq-vae and transformers, 2021.

Yan et al. (2023) W. Yan, D. Hafner, S. James, and P. Abbeel. Temporally consistent transformers for video generation. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 39062–39098. PMLR, 23–29 Jul 2023.

Yang et al. (2023) M. Yang, Y. Du, K. Ghasemipour, J. Tompson, D. Schuurmans, and P. Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114 , 2023.

Ye et al. (2022) W. Ye, Y. Zhang, P. Abbeel, and Y. Gao. Become a proficient player with limited data through watching pure videos. In The Eleventh International Conference on Learning Representations , 2022.

Yu et al. (2023) L. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M. Yang, Y. Hao, I. Essa, and L. Jiang. Magvit: Masked generative video transformer. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10459–10469, Los Alamitos, CA, USA, jun 2023. IEEE Computer Society. 10.1109/CVPR52729.2023.01008 .

Appendix A Additional Example Trajectories

附录 A 更多示例轨迹

Figure 15: More example trajectories: the model is prompted with either hand-drawn sketches, images generated from text-to-image generative models or realistic photos. Actions that drive the dynamics of the trajectory are provided by human input.

图 15：更多示例轨迹：模型可由手绘草图、文生图模型生成的图像或真实照片提示。驱动轨迹动态的动作由人工输入提供。

Figure 16: Controllable, consistent latent actions in Platformers: trajectories beginning from four different starting frames from our Platformers dataset. Each column shows the resulting frame from taking the same latent action five times. Despite training without action labels, not only are the same actions consistent across varied prompt frames, but also have semantic meaning: left, right, jump, and no-op.

图 16：Platformers 中可控且一致的潜在动作：从 Platformers 数据集四个不同起始帧出发的轨迹。每列展示连续执行同一潜在动作五次后的结果帧。尽管训练无动作标签，相同动作不仅在不同提示帧间一致，还具有语义：左、右、跳与空操作。

Appendix B Dataset

附录 B 数据集

B.1 Platformers Dataset

B.1 Platformers 数据集

Initial Dataset

We generated a dataset by filtering publicly available Internet videos, using the following criteria:

初始数据集

我们通过过滤公开互联网视频生成数据集，使用如下标准：

• • The title contains keywords relating to 2D platformer games.
• • The title or description must contain an action word, such as “speedrun” or “playthrough”.
• • The title must not contain negating words such as “movie” or “unboxing”.

• • 标题含与 2D 平台跳跃游戏相关的关键词。
• • 标题或描述须含动作词，如 “speedrun” 或 “playthrough”。
• • 标题不得含否定词，如 “movie” 或 “unboxing”。

We then split each video into 16s clips at 10 FPS, which corresponds to 160 frames per clip. Our resulting dataset contains 55M videos, which totals around 244k hours. When selecting keywords, we manually spot checked results to check that they typically produced 2D platformer gameplay videos which are not outnumbered by other sorts of videos which happen to share similar keywords.

随后将每段视频切成 16 秒、10 FPS 的片段，对应每段 160 帧。所得数据集含 5500 万段视频，合计约 24.4 万小时。选取关键词时我们人工抽查结果，确认通常得到 2D 平台跳跃游玩视频，且不会被碰巧共享相似关键词的其他视频淹没。

Filter Pipeline

We noticed that many of the videos in the dataset were of poor quality, impacting our model performance. We propose a scalable approach to systematically filter the data, using a learned classifier as in Baker et al. (2022). First, we define high quality videos as those that display clear gameplay and do not contain distractor items such as menu screen or streamer faces. We then filter this data as follows:

过滤流程

我们注意到数据集中许多视频质量较差，影响模型表现。我们提出可扩展的系统过滤方法，使用如 Baker et al. (2022) 中的学习分类器。首先，将高质量视频定义为清晰展示游玩、且不含菜单或主播脸等干扰物。随后按如下方式过滤：

1. Our team hand labelled 10k videos, with roughly ten hours of total human effort. The labels ranged from 5 (best) to 1 (worst) quality.
2. We trained a 11M parameter ResNet18 (He et al., 2016) with binary classification where we deleted all entries rated 2-4 and classified 5 as good and 1 as bad.
3. We then apply a decision rule based on model prediction and confidence to determine whether to keep the video.

1. 团队人工标注 1 万段视频，合计约十小时人力。标签从 5（最好）到 1（最差）。
2. 训练 1100 万参数的 ResNet18（He et al., 2016）做二分类：删除评分为 2–4 的条目，将 5 标为好、1 标为差。
3. 再根据模型预测与置信度应用决策规则，决定是否保留视频。

Consistent to findings in prior work Baker et al. (2022); Oquab et al. (2023), having high quality data outweighs the quantity of data – even though the curated datasaet is only just over 10% the size of the original dataset, the model trained on the curated dataset outperforms in terms of FVD, see Table 4. Our final dataset is 6.8M videos for a total of over 30k hours.

与先前工作 Baker et al. (2022); Oquab et al. (2023) 的发现一致：数据质量重于数量——尽管精选数据集规模仅略高于原始集的 10%，在其上训练的模型 FVD 更优，见表 4。最终数据集为 680 万段视频，合计超过 3 万小时。

	#Params	FVD ($\downarrow$)
Original dataset (55M videos)	580M	61.4
Curated dataset (6.8M videos)	580M	54.8

Table 4: Effect of dataset curation.

	#Params	FVD ($\downarrow$)
Original dataset (55M videos)	580M	61.4
Curated dataset (6.8M videos)	580M	54.8

表 4：数据集策展的效果。

Appendix C Training details

附录 C 训练细节

C.1 Latent Action Model Training

C.1 潜在动作模型训练

We found a benefit from increasing the number of codes (i.e. number of actions), at the cost of reduced playability for human and AI agents.

我们发现增加码数（即动作数）有益，但代价是人类与 AI 智能体的可玩性下降。

Component	Parameter	Value
Encoder	num_layers	20
	d_model	1024
	num_heads	16
Decoder	num_layers	20
	d_model	1024
	num_heads	16
Codebook	num_codes	8
	patch_size	16
	latent_dim	32

Table 5: Platformers action model hyperparameters

Component	Parameter	Value
Encoder	num_layers	20
	d_model	1024
	num_heads	16
Decoder	num_layers	20
	d_model	1024
	num_heads	16
Codebook	num_codes	8
	patch_size	16
	latent_dim	32

表 5：Platformers 动作模型超参数

Note that the model inputs are normalized between $0$ and $1$ and the final outputs of the decoder are placed through a sigmoid.

注意：模型输入归一化到 $0$ 与 $1$ 之间，解码器最终输出经 sigmoid。

C.2 Video Tokenizer Training

C.2 视频分词器训练

Here we describe our video tokenizer training. We found it more effective to scale our decoder than the encoder, and a marginal gain from increasing batch size (see Table 6).

下面描述视频分词器训练。我们发现扩大解码器比扩大编码器更有效，增大批大小也有小幅收益（见表 6）。

batch_size	training hardware	FLOPs	PSNR
64	64 TPUv2	$4.22\times 10^{20}$	35.7
384	64 TPUv3	$2.57\times 10^{21}$	36.5

Table 6: Tokenizer batch size scaling hyperparameters.

batch_size	training hardware	FLOPs	PSNR
64	64 TPUv2	$4.22\times 10^{20}$	35.7
384	64 TPUv3	$2.57\times 10^{21}$	36.5

表 6：分词器批大小缩放超参数。

Component	Parameter	Value
Encoder	num_layers	12
	d_model	512
	num_heads	8
	k/q_size	64
Decoder	num_layers	20
	d_model	1024
	num_heads	16
	k/q_size	64
Codebook	num_codes	1024
	patch_size	4
	latent_dim	32

Table 7: Platformers video tokenizer hyperparameters.

Component	Parameter	Value
Encoder	num_layers	12
	d_model	512
	num_heads	8
	k/q_size	64
Decoder	num_layers	20
	d_model	1024
	num_heads	16
	k/q_size	64
Codebook	num_codes	1024
	patch_size	4
	latent_dim	32

表 7：Platformers 视频分词器超参数。

We train our video tokenizer for 300k steps using the AdamW optimizer, with cosine decay, using the hyperparameters in Table 8.

我们用 AdamW 优化器、余弦衰减，按表 8 超参训练视频分词器 30 万步。

Parameter	Value
max_lr	3e-4
min_lr	3e-4
$\beta_{1}$	0.9
$\beta_{2}$	0.9
weight_decay	1e-4
warmup_steps	10k

Table 8: Video tokenizer optimizer hyperparameters

Parameter	Value
max_lr	3e-4
min_lr	3e-4
$\beta_{1}$	0.9
$\beta_{2}$	0.9
weight_decay	1e-4
warmup_steps	10k

表 8：视频分词器优化器超参数

C.3 Dynamics Model Training

C.3 动力学模型训练

Parameter	Value
max_lr	3e-5
min_lr	3e-6
$\beta_{1}$	0.9
$\beta_{2}$	0.9
weight_decay	1e-4
warmup_steps	5k

Table 9: Dynamics model optimizer hyperparameters

Parameter	Value
max_lr	3e-5
min_lr	3e-6
$\beta_{1}$	0.9
$\beta_{2}$	0.9
weight_decay	1e-4
warmup_steps	5k

表 9：动力学模型优化器超参数

Appendix D Scaling Experiments Details

附录 D 缩放实验细节

In this section we provide more details on the architecture as well as compute budget for the scaling experiments.

本节补充缩放实验的架构细节与算力预算。

Scaling model size

For all models we use a batch size of 256. We train all models for 200k steps, thus use a total of 750B training tokens for each run. All runs make use of batch parallelism and stage-3 ZeRO sharding (Rajbhandari et al., 2020), while our larger models also make use of tensor parallelism (Shoeybi et al., 2019). For this experiment we make use of TPUv2 and TPUv3 (Jouppi et al., 2020). See Table 10 for more details.

缩放模型规模

所有模型批大小为 256。各模型训练 20 万步，因而每次运行合计使用 750B 训练 token。全部运行使用批并行与 stage-3 ZeRO 分片（Rajbhandari et al., 2020）；更大模型还使用张量并行（Shoeybi et al., 2019）。本实验使用 TPUv2 与 TPUv3（Jouppi et al., 2020）。细节见表 10。

Parameters	num_layers	num_heads	d_model	k/q size	training hardware	training time	FLOPs
41M	18	8	512	64	64 TPUv2	3 days	$2.05\times 10^{20}$
96M	16	16	768	64	64 TPUv2	6 days	$3.58\times 10^{20}$
192M	20	18	1024	64	64 TPUv2	9 days	$6.4\times 10^{20}$
404M	21	12	1536	128	64 TPUv2	18 days	$1.2\times 10^{21}$
811M	20	20	2048	128	128 TPUv3	7 days	$2.2\times 10^{21}$
1.6B	28	22	2560	128	128 TPUv3	12 days	$4.04\times 10^{21}$
2.7B	36	22	3072	128	256 TPUv3	16 days	$6.91\times 10^{21}$

Table 10: Model size scaling architectures and compute usage. All models were trained for 200k steps with a batch size of 256, equating to 750B tokens.

Parameters	num_layers	num_heads	d_model	k/q size	training hardware	training time	FLOPs
41M	18	8	512	64	64 TPUv2	3 days	$2.05\times 10^{20}$
96M	16	16	768	64	64 TPUv2	6 days	$3.58\times 10^{20}$
192M	20	18	1024	64	64 TPUv2	9 days	$6.4\times 10^{20}$
404M	21	12	1536	128	64 TPUv2	18 days	$1.2\times 10^{21}$
811M	20	20	2048	128	128 TPUv3	7 days	$2.2\times 10^{21}$
1.6B	28	22	2560	128	128 TPUv3	12 days	$4.04\times 10^{21}$
2.7B	36	22	3072	128	256 TPUv3	16 days	$6.91\times 10^{21}$

表 10：模型规模缩放的架构与算力。所有模型训练 20 万步、批大小 256，合计 750B token。

Scaling batch size

All models use the same architecture with 2.3B parameters, as shown in Table 11, and train for 200k steps. The only difference between the three runs is hardware—the 128, 256 and 448 batch size models train on 64 TPUv3, 128 TPUv3 and 64 TPUv5p respectively.

缩放批大小

所有模型使用相同的 2.3B 参数架构，见表 11，并训练 20 万步。三次运行的唯一差别是硬件——批大小 128、256 与 448 的模型分别在 64 TPUv3、128 TPUv3 与 64 TPUv5p 上训练。

Parameters	num_layers	num_heads	d_model	k/q size
2.3B	34	20	2560	128

Table 11: Batch size scaling hyperparameters. All models use the following architecture for 200k steps, differing only in batch size.

Parameters	num_layers	num_heads	d_model	k/q size
2.3B	34	20	2560	128

表 11：批大小缩放超参数。所有模型按该架构训练 20 万步，仅批大小不同。

Genie Model

The parameter count, model architecture as well as compute usage of the dynamics model for the final Genie model is listed in Table 12. We train a 10.1B dynamics model with a batch size of 512, for a total of 125k steps using 256 TPUv5.

Genie 模型

最终 Genie 模型中动力学模型的参数量、架构与算力见表 12。我们用批大小 512、共 125k 步、256 块 TPUv5 训练 10.1B 动力学模型。

Parameters	num_layers	num_heads	d_model	k/q size	FLOPs
10.1B	48	36	5120	128	$6.6\times 10^{22}$

Table 12: Genie dynamics model hyperparameters.

Parameters	num_layers	num_heads	d_model	k/q size	FLOPs
10.1B	48	36	5120	128	$6.6\times 10^{22}$

表 12：Genie 动力学模型超参数。

Appendix E Behavioral Cloning Details

附录 E 行为克隆细节

In this section we provide more details about our behavioral cloning experiments. We train within the Procgen CoinRun environment (Cobbe et al., 2020) and evaluate in a held out test set. We assume we have a dataset of expert sequences in this environment from an agent trained with R2D2 (Kapturowski et al., 2018). We then train an agent to imitate from this data. Notably, the oracle agent has access to the corresponding ground-truth expert actions. We now discuss how we can utilize a pre-trained LAM to infer the actions taken.

本节补充行为克隆实验细节。我们在 Procgen CoinRun 环境（Cobbe et al., 2020）中训练，并在留出测试集上评估。假设有一份由 R2D2 训练智能体（Kapturowski et al., 2018）产生的该环境专家序列数据集，再训练智能体从这些数据模仿。值得注意的是，oracle 智能体可访问对应的真实专家动作。下面说明如何利用预训练 LAM 推断所采取的动作。

E.1 Genie LAM

In order to train an agent to imitate from unseen videos, we can use a frozen LAM from a Genie model trained on Internet videos. Given an expert sequence $\langle x_{t},x_{t+1}\rangle$ we extract the corresponding latent action label $a_{t}\leftarrow LAM(x_{t},x_{t+1})$. We then train a policy $\pi(a_{t}|x_{t})$ to predict the likelihood of the expert taking latent action $a_{t}$ given observation $x_{t}$. Note that this procedure is similar to prior works that learn from videos (Torabi et al., 2018; Baker et al., 2022). However, these approaches use ground-truth actions for labeling videos whereas we utilize latent actions learnt completely offline.

为从未见视频训练模仿智能体，可使用在互联网视频上训练的 Genie 模型中冻结的 LAM。给定专家序列 $\langle x_{t},x_{t+1}\rangle$，提取对应潜在动作标签 $a_{t}\leftarrow LAM(x_{t},x_{t+1})$。再训练策略 $\pi(a_{t}|x_{t})$，预测专家在观测 $x_{t}$ 下采取潜在动作 $a_{t}$ 的概率。该流程与从视频学习的先前工作类似（Torabi et al., 2018; Baker et al., 2022）。但那些方法用真实动作标注视频，而我们使用完全离线学到的潜在动作。

During inference, we must map latent actions emitted by the policy to real actions. To do this, we utilize a small set of action-labeled expert sequences. Given an expert sequence $\langle x_{t},u_{t},x_{t+1}\rangle$ (we denote $u_{t}$ for ground-truth actions to avoid confusion with predicted latent actions), we use the LAM to obtain a latent action $a_{t}$ and fill a dictionary $D$ consisting of mapped latents to a list of corresponding real actions. In summary, given an observation $x_{t}$ from the environment, we can obtain the most likely latent action as $a_{t}\sim\pi(s_{t})$, and then take the corresponding real action as $u_{t}\sim D[a_{t}]$.

推理时须把策略输出的潜在动作映射到真实动作。为此我们使用一小份带动作标注的专家序列。给定专家序列 $\langle x_{t},u_{t},x_{t+1}\rangle$（用 $u_{t}$ 表示真实动作，以免与预测潜在动作混淆），用 LAM 得到潜在动作 $a_{t}$，并填充字典 $D$，将各潜在动作映射到对应真实动作列表。概括而言：给定环境观测 $x_{t}$，取最可能潜在动作 $a_{t}\sim\pi(s_{t})$，再取对应真实动作 $u_{t}\sim D[a_{t}]$。

Note that other works have used data extracted from the agent’s policy to obtain a mapping from latent to real actions (Edwards et al., 2019; Ye et al., 2022), but we found using expert data enabled us to better evaluate the quality of the learnt policy. As shown in the main text, the agent was capable of adapting with as few as $200$ expert labels.

注意其他工作曾用智能体策略产生的数据建立潜在到真实动作的映射（Edwards et al., 2019; Ye et al., 2022），但我们发现使用专家数据能更好评估所学策略质量。如正文所示，智能体仅用约 $200$ 条专家标签即可适配。

E.2 Architecture

E.2 架构

We train a transformer as the policy for both the oracle and latent BC agents. We utilize our proposed ST-ViViT architecture for encoding the frames $\bm{x}_{1:t}=(x_{1},\cdots x_{t})$ . All previous actions are placed through a one-hot and then combined with the corresponding frame encoding as an additive embedding. We use a sequence length of $4$ during both training and inference and a batch size of $16$.

oracle 与潜在 BC 智能体均用 Transformer 作为策略。我们用所提出的 ST-ViViT 架构编码帧 $\bm{x}_{1:t}=(x_{1},\cdots x_{t})$。先前所有动作经 one-hot 后，作为加性嵌入与对应帧编码结合。训练与推理序列长度均为 $4$，批大小为 $16$。

Parameter	Value
max_lr	3e-5
min_lr	3e-6
$\beta_{1}$	0.9
$\beta_{2}$	0.96
weight_decay	1e-4
warmup_steps	5k

Table 13: BC model optimizer hyperparameters

Parameter	Value
max_lr	3e-5
min_lr	3e-6
$\beta_{1}$	0.9
$\beta_{2}$	0.96
weight_decay	1e-4
warmup_steps	5k

表 13：BC 模型优化器超参数

Component	Parameter	Value
Encoder	num_layers	12
	d_model	512
	patch_size	4
Policy	linear_layer	512

Table 14: BC policy hyperparameters

Component	Parameter	Value
Encoder	num_layers	12
	d_model	512
	patch_size	4
Policy	linear_layer	512

表 14：BC 策略超参数

Both the oracle and Genie LAM are trained with a cross-entropy loss where targets are either real or latent actions, respectively. During inference, we obtain the final prediction by sampling from the predicted logits. Note we found the oracle agent performed better when we randomly sampled actions $10\%$ of the time.

oracle 与 Genie LAM 均以交叉熵损失训练，目标分别为真实或潜在动作。推理时从预测 logits 采样得到最终预测。注意：我们发现 oracle 智能体在 $10\%$ 时间随机采样动作时表现更好。

Appendix F Reproducible Case Study

附录 F 可复现案例研究

In this section we describe a self-contained, fully reproducible case study that can be trained with a single mid range TPU/GPU in under a week.

本节描述一个自包含、可完全复现的案例，可在单块中端 TPU/GPU 上于一周内训练完成。

F.1 Data Collection

F.1 数据收集

First we need to collect the data to train our model. We use the CoinRun environment from the Procgen benchmark (Cobbe et al., 2020) since it has thousands of visually diverse levels with fairly simple platformer-like dynamics. Using the “hard” mode, we collect data using a random policy with no action repeats. We sample level seeds between zero and 10,000 and collect 1,000 timesteps for each level, for a total of 10M transitions.

首先需要收集训练数据。我们使用 Procgen 基准中的 CoinRun 环境（Cobbe et al., 2020）：它有数千个视觉多样的关卡，动力学接近简单平台跳跃。在 “hard” 模式下，用无动作重复的随机策略采集数据。关卡种子在 0 到 10000 间采样，每关采集 1000 个时间步，合计 1000 万次转移。

F.2 Video Tokenizer Training

F.2 视频分词器训练

Our video tokenizer for CoinRun follows the same setup as described in Section 2.1, trained with the optimizer configuration as in Section C.2. The primary difference in this example is we use smaller model sizes (see Table 15), and then use a batch size of 48 sequences, of length 16, for a total of 768 images per batch. This is sufficient to fit in a single TPU with 16G memory. The model is trained for three days using a single TPU which is sufficient to complete 300k steps.

CoinRun 的视频分词器设定与第 2.1 节相同，优化器配置同第 C.2 节。本示例主要差别是模型更小（见表 15），批大小为 48 个序列、每序列 16 帧，即每批共 768 张图像，足以放入 16G 内存的单块 TPU。模型用单块 TPU 训练三天，足以完成 30 万步。

Component	Parameter	Value
Encoder	num_layers	8
	d_model	512
	num_heads	8
Decoder	num_layers	8
	d_model	512
	num_heads	8
Codebook	num_codes	1024
	patch_size	4
	latent_dim	32

Table 15: CoinRun video tokenizer hyperparameters

Component	Parameter	Value
Encoder	num_layers	8
	d_model	512
	num_heads	8
Decoder	num_layers	8
	d_model	512
	num_heads	8
Codebook	num_codes	1024
	patch_size	4
	latent_dim	32

表 15：CoinRun 视频分词器超参数

F.3 Dynamics + Latent Action Model Training

F.3 动力学与潜在动作模型训练

Once we have trained the video tokenizer we can then jointly train the latent action and dynamics models. Once again we seek to fit our model training inside 16G memory, so we use a batch size of 36 sequences consisting of 16 frames each, for a total of 576 images. We train both the latent action model and dynamics model in parallel, using the setup described above (see: Section C.1 for the latent action model and Section C.3 for the dynamics model).

视频分词器训完后，可联合训练潜在动作与动力学模型。为使训练放入 16G 内存，批大小取 36 个序列、每序列 16 帧，合计 576 张图像。潜在动作模型与动力学模型并行训练，设定同上（潜在动作见第 C.1 节，动力学见第 C.3 节）。

We train both the latent action and dynamics models in parallel for 200k steps, using the optimizer hyperparameters in Table 9. We find this model generates consistent playable latent actions, resembling the original environment.

潜在动作与动力学模型并行训练 20 万步，优化器超参见表 9。我们发现该模型能生成一致、可玩的潜在动作，形态接近原始环境。

Component	Parameter	Value
Encoder	num_layers	8
	d_model	512
	num_heads	8
Decoder	num_layers	8
	d_model	512
	num_heads	8
Codebook	num_codes	6
	latent_dim	32

Table 16: CoinRun action model hyperparameters

Component	Parameter	Value
Encoder	num_layers	8
	d_model	512
	num_heads	8
Decoder	num_layers	8
	d_model	512
	num_heads	8
Codebook	num_codes	6
	latent_dim	32

表 16：CoinRun 动作模型超参数

Component	Parameter	Value
Architecture	num_layers	12
	d_model	512
	num_layers	8
Sampling	temperature	1.0
	maskgit_steps	25

Table 17: CoinRun dynamics model hyperparameters

Component	Parameter	Value
Architecture	num_layers	12
	d_model	512
	num_layers	8
Sampling	temperature	1.0
	maskgit_steps	25

表 17：CoinRun 动力学模型超参数