NitroGen: An Open Foundation Model for Generalist Gaming Agents

NitroGen：面向通才游戏智能体的开放基础模型

Loïc Magne¹*, Anas Awadalla¹ ²*, Guanzhi Wang¹ ³*†
Yinzhen Xu¹, Joshua Belofsky⁴, Fengyuan Hu¹, Joohwan Kim¹
Ludwig Schmidt², Georgia Gkioxari³, Jan Kautz¹
Yisong Yue³†, Yejin Choi¹ ²†, Yuke Zhu¹ ⁵†, Linxi “Jim” Fan¹†
¹ NVIDIA, ² Stanford, ³ Caltech, ⁴ UChicago, ⁵ UT Austin

• Co-lead, † Co-advise
  https://nitrogen.minedojo.org

Abstract

摘要

We introduce NitroGen, a vision-action foundation model for generalist gaming agents that is trained on 40,000 hours of gameplay videos across more than 1,000 games. We incorporate three key ingredients: 1) an internet-scale video-action dataset constructed by automatically extracting player actions from publicly available gameplay videos, 2) a multi-game benchmark environment that can measure cross-game generalization, and 3) a unified vision-action model trained with large-scale behavior cloning. NitroGen exhibits strong competence across diverse domains, including combat encounters in 3D action games, high-precision control in 2D platformers, and exploration in procedurally generated worlds. It transfers effectively to unseen games, achieving up to 52% relative improvement in task success rates over models trained from scratch. We release the dataset, evaluation suite, and model weights to advance research on generalist embodied agents.

我们介绍 NitroGen，一个面向通才游戏智能体的视觉–动作基础模型，它在超过 1,000 款游戏、共计 40,000 小时的玩法视频上训练。方法有三个要点：1）从公开玩法视频中自动提取玩家动作，构建互联网规模的视频–动作数据集；2）一套可衡量跨游戏泛化的多游戏基准环境；3）用大规模行为克隆训练的统一视觉–动作模型。NitroGen 在多种领域都表现出较强能力，包括三维动作游戏中的战斗、二维平台游戏中的高精度控制，以及程序化生成世界中的探索。它也能迁移到未见过的游戏，任务成功率相对从零训练的模型最高提升 52%。我们发布数据集、评测套件和模型权重，以推动通才具身智能体的研究。

1 Introduction

1 引言

Figure 1: NitroGen overview. NitroGen consists of three main components: (1) Multi-game foundation agent (center) - a generalist vision-action model that takes in game observations and generates gamepad actions, enabling zero-shot gameplay across multiple titles and serving as a foundation for fine-tuning on new games; (2) Universal simulator (left) - an environment wrapper that allows any commercial game to be controlled through a Gymnasium API; and (3) Internet-scale dataset (right) - the largest and most diverse open-source gaming dataset curated from 40,000 hours of publicly available gaming videos, spanning more than 1,000 games with extracted action labels.

图 1：NitroGen 总览。NitroGen 由三个主要部分组成：（1）多游戏基础智能体（中）——一个通才视觉–动作模型，根据游戏观测生成手柄动作，可在多个游戏上零样本游玩，也可作为在新游戏上微调的基础；（2）通用模拟器（左）——环境封装层，使任何商业游戏都可以通过 Gymnasium API 来控制；（3）互联网规模数据集（右）——目前规模最大、品类最多样的开源游戏数据集，从 40,000 小时公开游戏视频中整理而来，覆盖 1,000 款以上游戏，并带有提取出的动作标签。

Building generally capable embodied agents that can operate in unknown environments has long been considered a holy grail of AI research. While computer vision and large language models (LLMs) have achieved this generalization through large-scale pre-training on internet data (Brown et al., 2020; Devlin et al., 2019; Radford et al., 2021; Dosovitskiy et al., 2021), comparable progress in embodied AI has been impeded by the lack of large, diverse, and labeled action datasets. Video games present an ideal domain for advancing embodied AI since they offer visually rich interactive environments and tasks that span a wide range of complexities and temporal horizons. However, prior approaches face substantial limitations. LLM–based methods exploit either (1) hand-crafted programmatic APIs to expose internal game states and control agents (Wang et al., 2023; Volum et al., 2022; Wang et al., 2024) or (2) complicated perception modules for textual information extraction and object detection (Tan et al., 2024). They enable complex task-solving but require complicated domain-specific design and tuning. Reinforcement learning has achieved superhuman performance in individual games such as StarCraft II and Dota 2, but these agents are narrow, costly to train, and depend on specialized simulators rarely available for arbitrary games (Berner et al., 2019; Silver et al., 2016; Vinyals et al., 2019; Mnih et al., 2013; Mnih et al., 2015). Behavior-cloning approaches based on pixel observations have relied on expensive-to-collect demonstrations, constraining training to only a few game titles due to prohibitive data collection costs (Baker et al., 2022; Raad et al., 2024). To date, there has been little progress on developing open-source frameworks that can support the training and evaluation of generalist gaming agents, further hindering progress in this direction.

构建能够在未知环境中运作的通才具身智能体，长期以来被看作人工智能研究的终极目标之一。计算机视觉和大语言模型（LLM）已经借助互联网数据上的大规模预训练达到这种泛化（Brown et al., 2020; Devlin et al., 2019; Radford et al., 2021; Dosovitskiy et al., 2021），具身智能却受制于缺少规模大、种类多、且带动作标注的数据，进展相对缓慢。视频游戏环境视觉丰富、任务交互完整，复杂度和时间跨度也足够多样，是推进具身智能的理想领域。但已有方法各有明显局限。基于 LLM 的方法要么（1）用手工编写的程序接口暴露内部游戏状态并控制智能体（Wang et al., 2023; Volum et al., 2022; Wang et al., 2024），要么（2）依赖复杂的感知模块来抽取文本信息和做目标检测（Tan et al., 2024）。它们能完成较复杂的任务，却需要繁琐的领域专门设计和调参。强化学习曾在《星际争霸 II》《Dota 2》等单款游戏上达到超人类水平，但这些智能体很窄、训练代价高，还依赖专门模拟器，而这些模拟器很少能用在任意游戏上（Berner et al., 2019; Silver et al., 2016; Vinyals et al., 2019; Mnih et al., 2013; Mnih et al., 2015）。基于像素观测的行为克隆方法则依赖昂贵的示范采集，数据采集成本使训练往往只能覆盖少数几款游戏（Baker et al., 2022; Raad et al., 2024）。时至今日，仍缺少能够支撑通才游戏智能体训练与评测的开源框架，这进一步拖慢了这一方向的进展。

To address these limitations, we introduce NitroGen, an open foundation model for video game environments trained on 40,000 hours of publicly available internet videos covering more than 1,000 games. We make three major contributions (Figure 1):

为应对这些限制，我们介绍 NitroGen：一个面向电子游戏环境的开放基础模型，在覆盖 1,000 款以上游戏、共计 40,000 小时的公开互联网视频上训练。我们有三项主要贡献（图 1）：

1. Internet-scale dataset of action-labeled videos. We propose to use a new source of data from publicly available videos where content creators overlay their input commands in real time. We train an annotation model to extract frame-level actions with high accuracy, removing the need for costly manual data collection and capturing a wide spectrum of real player behaviors. Using this approach, we curate a dataset of 40,000 hours of video spanning more than 1,000 games, providing diverse demonstrations for large-scale training.

1. 带动作标注的互联网规模视频数据集。 我们提出利用一类新的公开视频数据：内容创作者会把输入指令实时叠加在画面上。我们训练标注模型，以较高精度提取逐帧动作，从而不必再做昂贵的人工采集，并能覆盖真实玩家行为的宽谱。据此我们整理出 40,000 小时、覆盖 1,000 款以上游戏的视频数据，为大规模训练提供多样化示范。

2. Multi-task multi-game evaluation suite. To assess generalization in realistic settings, we design benchmark environments that comprise 30 tasks of varied complexity from 10 commercial games, covering diverse challenges such as combat, navigation, decision-making, platforming, exploration, and puzzle-solving. This benchmark reflects the demands of modern game environments, where agents must learn to adapt across heterogeneous mechanics and objectives. We provide a universal Gymnasium API (Towers et al., 2024) for our evaluation suite that allows users to wrap any game to test diverse agent capabilities. This API is what we refer to as the universal simulator in Figure 1.

2. 多任务、多游戏评测套件。 为在真实设定中衡量泛化，我们设计了包含 10 款商业游戏、共 30 项不同复杂度任务的基准环境，覆盖战斗、导航、决策、平台跳跃、探索和解谜等挑战。这套基准反映了现代游戏环境的要求：智能体必须在机制和目标都不同的游戏之间适应。我们为评测套件提供通用的 Gymnasium API（Towers et al., 2024），使用者可以封装任意游戏来测试多种智能体能力。图 1 中所说的通用模拟器，指的就是这一 API。

3. Large-scale behavior-cloning pre-training. To demonstrate the feasibility and benefits of internet-scale pre-training, we train a vision-action transformer model on our dataset. We demonstrate strong results on our benchmark suite, validating our end-to-end pipeline and showing that it is possible to train a strong multi-game policy using only noisy internet data. We show the benefits of behavior-cloning pre-training by post-training our base model on games not seen during training. The model fine-tuned from the pre-trained NitroGen weights shows up to $52\%$ relative improvement in success rates over the model trained from scratch, given a fixed data and compute budget.

3. 大规模行为克隆预训练。 为说明互联网规模预训练是否可行、有何收益，我们在该数据集上训练了一个视觉–动作 Transformer 模型。它在我们的基准套件上取得了较强结果，验证了端到端流程，也表明仅用带噪声的互联网数据，也能训练出较强的多游戏策略。我们还把基座模型在训练中未见过的游戏上做后训练，以展示行为克隆预训练的好处。在数据和计算预算固定时，从预训练 NitroGen 权重微调得到的模型，成功率相对从零训练的模型最高提升 $52\%$。

We open-source the NitroGen dataset, simulator, and pre-trained weights. We envision NitroGen as a foundational resource that will enable the research community to accelerate progress toward building more generalist embodied agents, fostering new algorithms, model architectures, and applications in this emerging area.

我们开源 NitroGen 的数据集、模拟器和预训练权重。我们希望它成为一项基础资源，帮助研究社区更快推进通才具身智能体，并催生这一新兴方向上的新算法、新架构和新应用。

Figure 2: Video-action dataset pipeline overview. We extract actions from on-screen displays which show the gamepad actions of the player in real-time; called “input overlays”. (a) Dataset curation. We collect publicly available videos displaying a “gamepad overlay”. The diversity of these overlays presents significant challenges, as gamepads vary widely across content creators in controller types (e.g., Xbox, PlayStation, or others), transparency levels, and visual artifacts introduced by video compression. (b) Action extraction. For each collected video, we localize the gamepad by sampling 25 frames and running keypoint matching against a curated set of templates using SIFT and XFeat features. We use the template-matching results to localize and crop the gamepad region from each video. A hybrid classification–segmentation network is then trained to predict joystick positions and button states from the cropped controller images, enabling accurate reconstruction of player inputs.

图 2：视频–动作数据流水线概览。我们从实时显示玩家手柄动作的屏幕叠加（称为“输入叠加层”）中提取动作。（a）数据整理。我们收集画面上带有“手柄叠加层”的公开视频。这些叠加层差异很大，给处理带来明显挑战：不同创作者使用的手柄类型（如 Xbox、PlayStation 或其他）、透明度，以及视频压缩造成的视觉伪影都不相同。（b）动作提取。对每条收集到的视频，我们采样 25 帧，并用 SIFT 与 XFeat 特征，与一组整理好的模板做关键点匹配，以定位手柄。再利用模板匹配结果裁出手柄区域。随后训练一个分类–分割混合网络，从裁切后的手柄图像预测摇杆位置和按键状态，从而较准确地重建玩家输入。

2 Approach

2 方法

NitroGen consists of three novel components: (1) an internet-scale video dataset with action labels, (2) a multi-game benchmark with a Gymnasium environment wrapper, and (3) a vision–action model pre-trained through large-scale behavior cloning. In this section, we provide details of each component.

NitroGen 由三个新组件构成：（1）带动作标签的互联网规模视频数据集；（2）带 Gymnasium 环境封装的多游戏基准；（3）通过大规模行为克隆预训练的视觉–动作模型。本节分别说明各部分细节。

2.1 Internet-scale multi-game video-action dataset

2.1 互联网规模的多游戏视频–动作数据集

Annotation challenge. A central challenge in training policies from internet videos is recovering the corresponding actions, since most gameplay recordings typically do not include the player’s inputs. We address this limitation by using a novel source of publicly available videos in which such labels can be recovered. These videos feature input overlay software that displays a real-time visualization of the player’s actions, typically as a 2D image of a gamepad in a corner of the screen with pressed buttons highlighted (Figure 2(a)).

标注难点。 从互联网视频训练策略，核心困难之一是恢复对应的动作，因为大多数玩法录像并不包含玩家输入。我们的做法是利用一类可以恢复这些标签的公开视频。这类视频使用输入叠加软件，把玩家动作实时可视化，通常是在画面一角显示手柄的二维图像，并高亮已按下的按键（图 2(a)）。

Dataset curation. Although input overlays appear in only a fraction of online gameplay videos, they occur frequently enough to enable the construction of a large-scale dataset. We collect 71,000 hours of raw video containing gamepad overlays. While input overlay software was originally used primarily within the speedrunning community, its use has since expanded to many action games and among both expert and casual players. To avoid over-representation of any single title, we use a combination of keyword-based searches and curation guided by content diversity, ensuring coverage across games, genres, and skill levels. This approach balances casual and competitive play styles while maintaining broad genre representation. Figure 3 shows the distribution of gameplay hours by title and genre. The dataset covers more than 1,000 unique games, making it the largest labeled video–action dataset for video games to date. It contains 38,739 videos from 818 different content creators, with an average video duration of 1 hour and 50 minutes.

数据整理。 输入叠加只出现在一部分在线玩法视频中，但出现频率已经足以支撑大规模数据集的构建。我们收集了 71,000 小时带手柄叠加的原始视频。这类软件最初主要用在速通社区，后来扩展到许多动作游戏，使用者也覆盖高手和休闲玩家。为避免单款游戏占比过高，我们把关键词检索与注重内容多样性的整理结合起来，尽量覆盖不同游戏、类型和水平。这样既能平衡休闲与竞技玩法，也能保持较广的类型分布。图 3 给出了按游戏和类型统计的游戏时长分布。数据集覆盖 1,000 款以上不同游戏，是迄今规模最大的带标签视频–动作游戏数据集。其中包含来自 818 位不同创作者的 38,739 条视频，平均时长为 1 小时 50 分钟。

Figure 3: Distribution of the NitroGen dataset across games and genres. After filtering, the NitroGen dataset contains 40,000 hours of gameplay videos spanning more than 1,000 games. (a) Hours per game shows broad coverage, with 846 games having over one hour of data, 91 games with over 100 hours, and 15 games exceeding 1,000 hours each. (b) Genre distribution reveals Action-RPG games are most common (34.9% of total hours), followed by Platformer (18.4%) and Action-Adventure (9.2%) games, with the remainder distributed across seven genres.

图 3：NitroGen 数据集在游戏和类型上的分布。过滤之后，NitroGen 数据集包含 40,000 小时玩法视频，覆盖 1,000 款以上游戏。（a）各游戏时长分布较广：846 款游戏超过 1 小时，91 款超过 100 小时，15 款各自超过 1,000 小时。（b）类型分布上，动作角色扮演占比最高（占总时长的 34.9%），其次是平台游戏（18.4%）和动作冒险（9.2%），其余分布在另外七种类型中。

Action extraction. We extract player inputs from gameplay videos through a three-stage pipeline: (1) template matching to locate and crop the gamepad overlay, (2) gamepad action parsing using a fine-tuned segmentation model, and (3) quality filtering to ensure accurate and meaningful data.

动作提取。 我们用三阶段流水线从玩法视频中提取玩家输入：（1）用模板匹配定位并裁切手柄叠加层；（2）用微调后的分割模型解析手柄动作；（3）做质量过滤，以保证数据准确、有用。

Stage 1: Template matching. To locate gamepad overlays within gameplay videos, we apply template matching using a curated set of approximately 300 common controller templates. For each video, we sample 25 frames and perform feature matching with SIFT (Lowe, 2004) and XFeat (Potje et al., 2024) against all curated templates. We estimate an affine transformation from the paired keypoints and require at least 20 inliers for a match to be considered valid. We then extract the region with the highest matching score, which defines the gamepad location for subsequent processing. Figure 2(b) shows examples of successful match.

第一阶段：模板匹配。 为在玩法视频中定位手柄叠加层，我们用约 300 个常见手柄模板做模板匹配。对每条视频采样 25 帧，并用 SIFT（Lowe, 2004）和 XFeat（Potje et al., 2024）与全部整理好的模板做特征匹配。我们由配对关键点估计仿射变换，至少要有 20 个内点才算有效匹配。随后提取匹配分数最高的区域，作为后续处理中的手柄位置。图 2(b) 给出了成功匹配的例子。

Stage 2: Gamepad action parsing. We parse controller states using a fine-tuned SegFormer (Xie et al., 2021) segmentation model that processes pairs of consecutive frames. The model takes two consecutive frames as input (concatenated along the spatial dimension) to capture short-term temporal dynamics. It outputs a segmentation mask to localize joystick positions on a discrete $11\times 11$ grid, and binary button states (Figure 2(b)). Empirically, we find that estimating joystick positions via segmentation masks significantly outperforms direct regression of joystick coordinates.

第二阶段：手柄动作解析。 我们用微调后的 SegFormer（Xie et al., 2021）分割模型解析手柄状态，输入为连续两帧。模型把连续两帧沿空间维拼接，以捕捉短时动态，输出用于在离散 $11\times 11$ 网格上定位摇杆位置的分割掩码，以及按键的二值状态（图 2(b)）。经验上，用分割掩码估计摇杆位置，明显优于直接回归摇杆坐标。

We train the annotation model using synthetic data generated by sampling frames from the NitroGen training set and programmatically overlaying controller templates using the Open Joystick Display[^1], Input Overlay[^2], and GamePad Viewer[^3] software. For each template, we produce multiple frames with random button states and joystick positions, yielding 8M labeled frames. To simulate real-world visual artifacts, we vary overlay opacity, controller size, and video compression, generating multiple variants per frame. We train the action parsing SegFormer model using the AdamW optimizer (Kingma and Ba, 2017; Loshchilov and Hutter, 2017) with a learning rate of $0.0001$, linear learning rate decay, weight decay of $0.1$, and a batch size of $256$.

标注模型的训练数据是合成的：我们从 NitroGen 训练集中采样帧，再用 Open Joystick Display[^1]、Input Overlay[^2] 和 GamePad Viewer[^3] 在程序里叠上手柄模板。对每个模板，我们生成多帧、随机按键状态和摇杆位置，共得到 800 万张带标签的帧。为模拟真实视频中的视觉伪影，我们还改变叠加不透明度、手柄大小和视频压缩，为每帧生成多个变体。动作解析用的 SegFormer 以 AdamW 优化器训练（Kingma and Ba, 2017; Loshchilov and Hutter, 2017），学习率为 $0.0001$，学习率线性衰减，权重衰减为 $0.1$，批大小为 $256$。

At inference, we compute precise joystick positions by detecting contours for each joystick over the entire video. To estimate the center position of each joystick, we average positions from all frames where the joystick is classified as centered in the $11\times 11$ discrete grid. We then normalize the positions to the range $[-1.0,1.0]$ using the 99th percentile of absolute $x$ and $y$ values over the video to reduce the influence of outliers.

推理时，我们在整段视频上检测每个摇杆的轮廓，以计算精确摇杆位置。摇杆中心位置的估计方式是：把在 $11\times 11$ 离散网格中被判定为居中的所有帧的位置取平均。随后用整段视频中绝对 $x$、$y$ 值的第 99 百分位做归一化，把位置映射到 $[-1.0,1.0]$，以减弱离群点的影响。

Stage 3: Quality filtering. The final stage applies targeted filtering strategies to ensure high-quality data. During training, we observe that using the raw 71,000 hours of data leads to the model over-predicting the null action as noted in VPT Baker et al. (2022). To avoid that, we discard segments based on action density: we only keep chunks where at least 50% of the timesteps have non-zero button or joystick actions, resulting in 55% of the data being kept. For all gameplay videos, we mask the on-screen controller to prevent models from exploiting it as a shortcut.

第三阶段：质量过滤。 最后一阶段用有针对性的过滤来保证数据质量。训练中我们观察到，若直接使用全部 71,000 小时原始数据，模型会过度预测空动作，这与 VPT（Baker et al., 2022）中的现象一致。为避免这一点，我们按动作密度丢弃片段：只保留至少 50% 的时间步带有非零按键或摇杆动作的块，最终保留了 55% 的数据。对所有玩法视频，我们还会遮盖画面上的手柄，防止模型走捷径、直接“看”叠加层作答。

Figure 4: In-game rollouts. We show NitroGen performing tasks in diverse 2D and 3D environments. These tasks can take from a few seconds to a few minutes to perform. Some of them include memorization, while others are performed in procedurally generated worlds and require the model to adapt.

图 4：游戏内展开。我们展示 NitroGen 在多种二维和三维环境中完成任务。这些任务短则数秒，长则可到数分钟。有的需要记忆，有的发生在程序化生成的世界中，要求模型做出适应。

2.2 Evaluation suite

2.2 评测套件

Universal simulator for any game title. Many research environments provide a Gymnasium API (Towers et al., 2024) that enables programmatic control of the simulation. To bring this capability to commercial video games, which typically lack such an interface, we develop a universal simulator that can wrap any game title with a Gymnasium API for model development. The library intercepts the game engine’s system clock to control simulation time, enabling frame-by-frame interaction without modifying game code. This approach works with any title that uses the system clock for physics and interactions, which is a common practice in game development. We leave real-time or asynchronous deployment to future work. Frequent pausing and resuming during gameplay could potentially affect the game’s physics engine in unknown ways, we verify that this process does not alter games’ physics and behaviors (see Appendix B.1.).

可封装任意游戏的通用模拟器。 许多研究环境提供 Gymnasium API（Towers et al., 2024），以便对仿真做程序化控制。商业游戏通常没有这类接口。为把这一能力带到商业游戏上，我们开发了一个通用模拟器，可用 Gymnasium API 封装任意游戏，供模型开发使用。该库拦截游戏引擎的系统时钟以控制仿真时间，从而在不修改游戏代码的情况下实现逐帧交互。只要游戏用系统时钟驱动物理和交互——这在游戏开发中很常见——这一方法就适用。实时或异步部署留待后续工作。频繁暂停和恢复可能以未知方式影响物理引擎；我们验证了这一过程不会改变游戏的物理和行为（见附录 B.1）。

Unified observation and action space. Using this simulator, we introduce a multi-game, multi-task benchmark with a shared interface across all titles. Observations are single RGB frames. Actions consist of a standardized 16-dimensional binary vector for gamepad buttons (4 d-pad buttons, 4 face buttons, 2 shoulders, 2 triggers, 2 joystick thumb buttons, start, back) plus a 4-dimensional continuous vector for joystick positions. Unlike prior work that defines game or task-specific action spaces (Baker et al., 2022; Guss et al., 2019), this unified layout facilitates direct policy transfer across diverse games.

统一的观测与动作空间。 基于该模拟器，我们引入一套多游戏、多任务基准，所有游戏共用同一接口。观测是单帧 RGB 图像。动作由标准化的 16 维二值向量（手柄按键：4 个方向键、4 个面部键、2 个肩键、2 个扳机、2 个摇杆按下键，以及开始键和返回键）再加上 4 维连续向量（摇杆位置）组成。与先前为特定游戏或任务定义动作空间的工作不同（Baker et al., 2022; Guss et al., 2019），这种统一布局便于策略在不同游戏之间直接迁移。

Diverse evaluation tasks. The evaluation suite serves as a universal evaluation framework for multi-game policies, covering 10 games across diverse visual styles and genres with 30 tasks total. The suite includes five 2D games and five 3D games, each testing different skill combinations. The 2D games include three side-scrollers and two top-down roguelikes with procedurally generated levels. The 3D games consist of two open-world games, two combat-focused action-RPGs, and one sports game.

多样化的评测任务。 该评测套件是多游戏策略的通用评测框架，覆盖 10 款视觉风格和类型各异的游戏，共 30 项任务。套件包含五款二维游戏和五款三维游戏，各自考察不同的技能组合。二维游戏中有三款横版卷轴和两款关卡程序化生成的俯视类 roguelike；三维游戏则包括两款开放世界、两款偏战斗的动作角色扮演，以及一款体育游戏。

Tasks are distributed across three categories: 11 combat tasks (boss fights, enemy encounters), 10 navigation tasks (reaching specific locations, traversing environments), and 9 game-specific tasks (unique mechanics particular to individual games). Each task has clearly defined start and goal states, with attempts typically lasting a few minutes, though human players may require several hours of repeated attempts to succeed. We select tasks where the initial visual state provides sufficient context to elicit correct behavior, leaving language-conditioned specifications to future work. Success rates are measured through human evaluation.

任务分成三类：11 项战斗任务（首领战、遭遇战），10 项导航任务（到达指定位置、穿越环境），以及 9 项游戏特定任务（各游戏独有的机制）。每项任务都有明确的起始状态和目标状态，单次尝试通常持续数分钟，但人类玩家往往需要反复尝试数小时才能成功。我们选择的任务，其初始视觉状态已能提供足够上下文来引出正确行为；语言条件的任务描述留待后续工作。成功率由人工评测得到。

Figure 5: Gamepad parsing performance for different controller families. We verify the correctness of our action extraction pipeline by comparing performance across different controller families against ground-truth data. (a) shows joystick R2 correlation scores (averaged for both left and right joysticks) with an overall average of 0.84. (b) shows button frame accuracy with an overall average of 0.96.

图 5：不同手柄系列上的手柄解析表现。我们把动作提取流水线在不同手柄系列上的表现与真值对比，以检验其正确性。（a）为摇杆 $R^{2}$ 相关分数（左右摇杆取平均），总体平均为 0.84。（b）为按键的逐帧准确率，总体平均为 0.96。

2.3 NitroGen foundation model

2.3 NitroGen 基础模型

Architecture. Building on recent advances in generative modeling and robotics, NitroGen employs flow matching (Lipman et al., 2022) to generate chunks of future actions conditioned on visual observations. The architecture is adapted from GR00T N1 (Bjorck et al., 2025) with the language and state encoders removed, and a single action head. RGB inputs at $256\times 256$ resolution are encoded using a SigLIP 2 vision transformer (Tschannen et al., 2025), producing 256 image tokens per frame. Actions are generated with a diffusion transformer (DiT) (Peebles and Xie, 2023) that outputs multiple actions per forward pass. Noisy action chunks are first encoded by an MLP into one action token per timestep, then processed through several DiT blocks consisting of alternating self-attention and cross-attention layers. Cross-attention conditions action generation on the encoded frame tokens. The final action tokens are decoded into continuous action vectors using an MLP applied independently across the time dimension. Full mathematical details are provided in Appendix A.

架构。 借鉴生成建模和机器人领域的近期进展，NitroGen 采用流匹配（Lipman et al., 2022），在视觉观测的条件下生成未来动作块。架构改编自 GR00T N1（Bjorck et al., 2025），去掉了语言编码器和状态编码器，只保留一个动作头。$256\times 256$ 分辨率的 RGB 输入由 SigLIP 2 视觉 Transformer（Tschannen et al., 2025）编码，每帧得到 256 个图像词元。动作由扩散 Transformer（DiT）（Peebles and Xie, 2023）生成，一次前向传播输出多个动作。带噪声的动作块先由 MLP 编码为每个时间步一个动作词元，再经过若干由自注意力和交叉注意力交替组成的 DiT 块。交叉注意力把动作生成条件化在编码后的帧词元上。最后用一个沿时间维独立作用的 MLP，把动作词元解码为连续动作向量。完整数学细节见附录 A。

Design choices. Although the model can condition on multiple frames, we find no benefit from using more than one past frame, even with increased temporal gaps. This is likely because the initial state of these action games already provides sufficient context to elicit the appropriate behavior. We instead use a single context frame and generate 16-action chunks, which improves temporal consistency compared to single-action generation.

设计选择。 模型虽然可以条件于多帧，但我们发现即使用多于一帧的历史、并加大时间间隔，也没有额外收益。这很可能是因为这些动作游戏的初始状态本身已能提供足够上下文，引出合适行为。我们因此只用单帧上下文，并生成长度为 16 的动作块；与单步动作生成相比，这样能提高时间一致性。

Training and inference. We train NitroGen using the standard conditional flow-matching objective (Lipman et al., 2022; Black et al., 2024a), applied to 16-action chunks, with one $256\times 256$ frame of context. Inference follows the corresponding denoising process with $k=16$ steps.

训练与推理。 我们用标准的条件流匹配目标（Lipman et al., 2022; Black et al., 2024a）训练 NitroGen，作用于 16 步动作块，上下文为 1 帧 $256\times 256$ 图像。推理按相应的去噪过程进行，步数 $k=16$。

During training, we apply the following image augmentations: random brightness, contrast, saturation and hue, random rotation between $-5$ and $5$ degrees, and random crops. We train all models using AdamW (Kingma and Ba, 2017; Loshchilov and Hutter, 2017) optimizer with a weight decay of $0.001$. We use a warmup-stable-decay (WSD) schedule (Wen et al., 2024), which allows us to train for longer without a fixed training budget, with a constant learning rate phase of $0.0001$. Following Peebles and Xie (2023), we maintain an exponential moving average (EMA) of model weights during training with a decay of $0.9999$. All our results are obtained with the EMA weights, which we find consistently outperform the non-EMA weights.

训练时我们使用如下图像增强：随机亮度、对比度、饱和度和色相，$-5$ 到 $5$ 度的随机旋转，以及随机裁剪。全部模型用 AdamW（Kingma and Ba, 2017; Loshchilov and Hutter, 2017）训练，权重衰减为 $0.001$。我们采用 warmup-stable-decay（WSD）学习率日程（Wen et al., 2024），这样不必预先固定训练预算也能训得更久，其中恒定学习率阶段为 $0.0001$。按照 Peebles and Xie（2023），我们在训练中维护模型权重的指数滑动平均（EMA），衰减系数为 $0.9999$。本文所有结果都来自 EMA 权重；我们发现它始终优于非 EMA 权重。

Figure 6: NitroGen 500M pre-training results across different games. We evaluate NitroGen after behavior-cloning pre-training. The model is not fine-tuned for specific games. For each game, we measure the average task completion rate on 3 tasks with 5 rollouts per task. Despite being trained on a very noisy internet dataset, NitroGen is able to perform non-trivial tasks over games with different visual styles (3D, 2D top-down, 2D side-scrolling) and genres (platformer, action-RPG, roguelike, etc.).

图 6：5 亿参数 NitroGen 在不同游戏上的预训练结果。我们评估行为克隆预训练之后的 NitroGen，模型没有针对特定游戏再做微调。对每款游戏，我们在 3 项任务上各做 5 次展开，统计平均任务完成率。尽管训练数据来自噪声很大的互联网视频，NitroGen 仍能在不同视觉风格（三维、二维俯视、二维横版）和不同类型（平台、动作角色扮演、roguelike 等）的游戏中完成非平凡任务。

3 Experiments

3 实验

Performance of the gamepad action extraction model. To evaluate our action extraction pipeline, we construct a benchmark dataset by recording gameplay from six video games using OBS[^4], with randomized opacity, gamepad size, and gamepad type to mimic real-world conditions. We record ground-truth controller inputs at each frame and compare them with the extracted actions. We measure joystick accuracy with the $R^{2}$ score and button accuracy per frame. As shown in Figure 5, we achieve an average $R^{2}$ of 0.84 for joystick positions and an average button accuracy of 0.96 across the most popular controller families.

手柄动作提取模型的表现。 为评估动作提取流水线，我们用 OBS[^4] 录制六款游戏的玩法，并随机变化不透明度、手柄大小和手柄类型，以模拟真实条件。我们记录每一帧的真值手柄输入，并与提取出的动作对比。摇杆精度用 $R^{2}$ 分数衡量，按键精度按帧计算。如图 5 所示，在最常见的几类手柄上，摇杆位置的平均 $R^{2}$ 为 0.84，按键平均准确率为 0.96。

NitroGen demonstrates strong capabilities across a wide range of games. We train a single model on the entire dataset from Section 2.1. Without further fine-tuning, NitroGen achieves non-trivial success rates across many games and tasks. Figure 6 summarizes the main results. We observe that NitroGen performs well both on tasks that can be memorized and on tasks that require zero-shot generalization. For example, some games feature fixed layouts that the model may have partially encountered during training, while others employ procedural generation that ensures each playthrough is unique. We do not find significant differences in performance between these two categories, suggesting that NitroGen can both leverage memorization and adapt to unseen scenarios.

NitroGen 在多种游戏上都表现出较强能力。 我们在第 2.1 节的全部数据上训练单个模型。不再微调的情况下，NitroGen 已在许多游戏和任务上取得非平凡成功率。图 6 汇总了主要结果。我们看到，无论是可以靠记忆完成的任务，还是需要零样本泛化的任务，NitroGen 都表现不错。例如，有的游戏关卡布局固定，模型在训练中可能部分见过；有的则采用程序化生成，每一局都不同。这两类任务的表现没有明显差异，说明 NitroGen 既能利用记忆，也能适应未见过的情形。

This result validates that it is possible to train a robust policy using only noisy internet-scale data. The dataset includes several sources of noise that could hinder training: (a) actions are not strictly ground truth, since input overlay software introduces small delays, and parsing adds further inaccuracies; (b) video frames often contain creator-specific artifacts such as livestream chats, subscribe prompts, or progress trackers; and (c) controller configurations vary across players, differences in sensitivity settings or custom button mappings can change the semantic meaning of the same input. Despite these challenges, Figure 6 shows that large-scale pre-training yields a robust multi-game policy.

这一结果说明，仅用带噪声的互联网规模数据，也可以训练出稳健的策略。数据集中有几类可能干扰训练的噪声：（a）动作并非严格真值，输入叠加软件会引入小延迟，解析过程也会再叠加误差；（b）视频帧常常带有创作者特有的附加元素，例如直播聊天、订阅提示或进度条；（c）不同玩家的手柄配置不同，灵敏度或自定义按键映射会改变同一输入的语义。尽管有这些困难，图 6 表明大规模预训练仍能得到稳健的多游戏策略。

NitroGen pre-training improves downstream fine-tuning on unseen environments. We evaluate transfer learning by pre-training NitroGen on the full dataset except for a held-out game, then fine-tuning on this game with a limited amount of data. We compare this fine-tuned model with an identical architecture trained from scratch using the same data and compute budget. Results are shown in Figure 7. We study two representative games with different visual styles and genres: an isometric roguelike and a 3D action-RPG.

NitroGen 预训练能改善未见环境上的下游微调。 我们用迁移学习来评估：在完整数据集上预训练 NitroGen，但留出一款游戏，再在该游戏上用有限数据微调。对照是架构相同、数据和计算预算也相同、但从零训练的模型。结果见图 7。我们研究了两款视觉风格和类型都不同的代表性游戏：一款等距视角的 roguelike，以及一款三维动作角色扮演。

The effectiveness of pre-training varies by game type and task category. Across different data quantities, fine-tuning achieves an average relative improvement of 10% on the isometric roguelike, whereas the 3D action-RPG shows a 25% average relative improvement. This difference likely stems from better representation of 3D action-RPGs in the training distribution, while the isometric roguelike has gameplay mechanics and visual style that are less common in the training data.

预训练的效果随游戏类型和任务类别而变化。在不同数据量下，等距 roguelike 上微调的平均相对提升为 10%，三维动作角色扮演则为 25%。这一差异很可能是因为三维动作角色扮演在训练分布中更常见，而该 roguelike 的机制和视觉风格在训练数据中较少。

Furthermore, pre-training benefits are not uniform across task types. On the 3D action-RPG, generic tasks such as combat (52% relative improvement) and navigation (25% relative improvement) benefit substantially from pre-training, while game-specific tasks show only marginal gains (5% relative improvement). This suggests that NitroGen effectively learns transferable skills for common gameplay patterns, but game-specific mechanics still require targeted training on the new environment.

此外，预训练收益在不同类型的任务上也不均匀。在三维动作角色扮演中，战斗（相对提升 52%）和导航（相对提升 25%）这类通用任务从预训练中获益明显，游戏特定任务则只有很小的提升（相对提升 5%）。这说明 NitroGen 有效学到了常见玩法模式中可迁移的技能，但游戏特有机制仍需要在新环境上做有针对性的训练。

Figure 7: Post-training experiments: NitroGen pre-training improves downstream agents in unseen environments. We pre-train NitroGen on the dataset described in Section 2.1, holding out one game. We then fine-tune the pre-trained checkpoint on the held-out game and compare the results with a model trained from scratch using the same architecture, data and compute budget. (a) When varying data quantity, task-completion rate scales with dataset size, and fine-tuning achieves on average a 10% relative improvement in task-completion rate. (b) When varying task type in the low-data regime (30h), fine-tuning achieves up to 52% relative improvement in task-completion rate.

图 7：后训练实验：NitroGen 预训练能改善未见环境中的下游智能体。我们在第 2.1 节所述数据集上预训练 NitroGen，并留出一款游戏。随后在留出的游戏上微调该预训练检查点，并与架构、数据和计算预算相同、但从零训练的模型对比。（a）改变数据量时，任务完成率随数据规模上升，微调平均带来 10% 的相对提升。（b）在低数据设定（30 小时）下改变任务类型，微调的任务完成率相对提升最高可达 52%。

4 Limitations and future work

4 局限与未来工作

Design limitations. NitroGen is limited to being a fast-reacting system-1 sensory model. It cannot plan over long horizons or follow language instructions; the model only reacts to the short context it sees. We develop NitroGen aiming for it to serve as a foundation for future generalist agent development, where post-training for language-following and reinforcement learning can be applied to enhance planning capabilities and improve success rates.

设计上的局限。 NitroGen 目前只是一个反应迅速的系统 1 感知模型。它不能做长程规划，也不能遵循语言指令，而只对所见的短上下文做出反应。我们把它定位为未来通才智能体发展的基础，后续可以通过面向语言遵循和强化学习的后训练，增强规划能力并提高成功率。

Dataset bias. While diverse, our data collection method still restricts the types of games included in our dataset. The data distribution of the NitroGen dataset is biased toward action games (Figure 3), and games that are typically played with a gamepad. Keyboard-only games or those that involve complex manipulation are less represented in the dataset. This bias may limit the agent’s ability to generalize to genres like strategy or simulation games that rely more on planning and keyboard input.

数据偏差。 尽管已经比较多样，我们的采集方式仍限制了数据集中的游戏种类。NitroGen 数据分布偏向动作游戏（图 3），以及通常用手柄游玩的游戏。只用键盘的游戏，或涉及复杂操作的游戏，在数据中较少。这种偏差可能会限制智能体向策略、模拟等更依赖规划和键盘输入的类型泛化。

5 Related works

5 相关工作

Gaming agents. Video games have long been testbeds for AI, with approaches generally following three directions. Reinforcement learning achieved landmark successes from Atari with DQN (Mnih et al., 2013; Mnih et al., 2015) to AlphaGo (Silver et al., 2016), AlphaStar (Vinyals et al., 2019), and OpenAI Five (Berner et al., 2019), but these rely on engineered rewards, hand-crafted features, and specialized simulators. More recent vision-based methods like Dreamer 3 (Hafner et al., 2023) still require dedicated simulators and environment-specific training. A second line leverages large language models for high-level reasoning with structured APIs, as in Voyager (Wang et al., 2023) and Cradle (Tan et al., 2024), but these depend on hand-crafted interfaces. A third category learns directly from pixels or states via behavior cloning, including MineRL (Guss et al., 2019), VPT (Baker et al., 2022), SIMA (Raad et al., 2024), GATO (Reed et al., 2022), Dreamer 4 Hafner et al. (2025), Lumine Tan et al. (2025), and Farhang et al. Farhang et al. (2024), but they all rely on datasets bootstrapped from human demonstrations or RL-generated data. NitroGen advances this third direction by scaling behavior cloning to internet-scale, enabling training across hundreds of games without costly collection. Game-TARS Wang et al. (2025) is a concurrent work that also train a multi-game agent. They combine contractor data and multi-modal reasoning data to train on a total of 20,000 hours.

游戏智能体。 视频游戏长期是人工智能的试验场，方法大致沿三个方向展开。强化学习从 Atari 上的 DQN（Mnih et al., 2013; Mnih et al., 2015），到 AlphaGo（Silver et al., 2016）、AlphaStar（Vinyals et al., 2019）和 OpenAI Five（Berner et al., 2019），取得过标志性成功，但这些方法依赖设计好的奖励、手工特征和专门模拟器。更新的基于视觉的方法如 Dreamer 3（Hafner et al., 2023）仍然需要专用模拟器，并针对具体环境训练。第二条路线用大语言模型做高层推理，并配合结构化 API，如 Voyager（Wang et al., 2023）和 Cradle（Tan et al., 2024），但它们依赖手工接口。第三类则通过行为克隆直接从像素或状态学习，包括 MineRL（Guss et al., 2019）、VPT（Baker et al., 2022）、SIMA（Raad et al., 2024）、GATO（Reed et al., 2022）、Dreamer 4（Hafner et al., 2025）、Lumine（Tan et al., 2025）以及 Farhang 等人（Farhang et al., 2024），不过它们都依赖由人类示范或强化学习生成数据引导出的数据集。NitroGen 把这一第三方向推进到互联网规模的行为克隆，从而可以在数百款游戏上训练，而不必做昂贵的采集。Game-TARS（Wang et al., 2025）是同期工作，同样训练多游戏智能体；他们把承包商数据与多模态推理数据合在一起，总训练时长为 20,000 小时。

Embodied foundation models. Foundation models for embodied AI generally adopt either hierarchical reasoning or end-to-end learning. Hierarchical methods pair pre-trained LLMs or VLMs with low-level policies (Ahn et al., 2022; Driess et al., 2023; Huang et al., 2022; Liang et al., 2023; Singh et al., 2023) treating the foundation models as black-boxes. Vision-Language-Action (VLA) models (Bjorck et al., 2025; Kim et al., 2024; Black et al., 2024b; Brohan et al., 2022; Cheang et al., 2024; Wen et al., 2025; Team et al., 2024) instead train policies end-to-end on embodied data, though generalizing across tasks and embodiments remains challenging. NitroGen differs by discarding language conditioning and focusing purely on scalable vision-action mapping using diverse gameplay data.

具身基础模型。 具身智能的基础模型通常采用分层推理或端到端学习。分层方法把预训练的 LLM 或 VLM 与底层策略搭配（Ahn et al., 2022; Driess et al., 2023; Huang et al., 2022; Liang et al., 2023; Singh et al., 2023），把基础模型当黑盒使用。视觉–语言–动作（VLA）模型（Bjorck et al., 2025; Kim et al., 2024; Black et al., 2024b; Brohan et al., 2022; Cheang et al., 2024; Wen et al., 2025; Team et al., 2024）则在具身数据上端到端训练策略，但跨任务、跨形体的泛化仍然困难。NitroGen 的不同之处在于去掉语言条件，专注于用多样的玩法数据，规模化地学习视觉到动作的映射。

Large-scale action datasets. Progress in vision and NLP has been driven by large labeled datasets, but embodied AI lags behind due to the difficulty of collecting action-labeled data and defining standardized action spaces. Gaming datasets like MineRL (Guss et al., 2019) provide limited coverage, while MineDojo (Fan et al., 2022) scales video data without action labels. VPT (Baker et al., 2022) annotates 70,000 hours via inverse dynamics but is limited to Minecraft. Other work seeks to infer latent actions from videos (Edwards et al., 2019; Ye et al., 2024; Bruce et al., 2024; Parker-Holder et al., 2024), though scalability is unclear. In robotics, teleoperation has produced datasets such as Roboturk (Mandlekar et al., 2018; Mandlekar et al., 2019; Mandlekar et al., 2020), ALOHA (Aldaco et al., 2024), TeleMoMa (Dass et al., 2024), Open X-Embodiment (O’Neill et al., 2024), and AgiBot World (Bu et al., 2025), but these are costly, limited in scale, and lack organic diversity. NitroGen introduces a scalable alternative by leveraging input overlay software, which naturally provides action labels in publicly available gameplay videos.

大规模动作数据集。 视觉和自然语言处理的进展，很大程度上来自大规模带标签数据；具身智能则因为动作标注难采、动作空间难统一而落后。MineRL（Guss et al., 2019）这类游戏数据集覆盖有限，MineDojo（Fan et al., 2022）放大了视频规模却没有动作标签。VPT（Baker et al., 2022）用逆动力学为 70,000 小时视频标注，但只限于《我的世界》。另一些工作试图从视频推断潜在动作（Edwards et al., 2019; Ye et al., 2024; Bruce et al., 2024; Parker-Holder et al., 2024），可扩展性仍不明确。机器人领域通过遥操作得到了 Roboturk（Mandlekar et al., 2018; Mandlekar et al., 2019; Mandlekar et al., 2020）、ALOHA（Aldaco et al., 2024）、TeleMoMa（Dass et al., 2024）、Open X-Embodiment（O’Neill et al., 2024）和 AgiBot World（Bu et al., 2025）等数据集，但成本高、规模有限，也缺少自然产生的多样性。NitroGen 提供了一种可扩展的替代：利用输入叠加软件，在公开玩法视频中自然获得动作标签。

6 Conclusion

6 结论

In this work, we introduce NitroGen, an approach to scale up foundation pre-training for video game agents and demonstrate how internet pre-training can yield a generalist policy. We leverage a new source of publicly available data to build an internet-scale video-action dataset, and empirically demonstrate its effectiveness by successfully training a multi-game policy. NitroGen shows positive signs of generalization in fine-tuning experiments. By lowering the barrier to train agents on new environments, NitroGen serves as a starting point to develop more powerful and general-purpose agents.

本文介绍 NitroGen，一种把游戏智能体的基础预训练做大规模扩展的方法，并表明互联网预训练可以产生通才策略。我们利用一类新的公开数据，构建互联网规模的视频–动作数据集，并通过成功训练多游戏策略，从实验上说明其有效性。微调实验中，NitroGen 也显示出正向的泛化迹象。通过降低在新环境上训练智能体的门槛，NitroGen 可以作为发展更强、更通用智能体的起点。

References

参考文献

Ahn et al. (2022) M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. Do as i can, not as i say: grounding language in robotic affordances . arXiv preprint arXiv:2204.01691 . Cited by: §5 .

Aldaco et al. (2024) J. Aldaco, T. Armstrong, R. Baruch, J. Bingham, S. Chan, K. Draper, D. Dwibedi, C. Finn, P. Florence, S. Goodrich, et al. Aloha 2: an enhanced low-cost hardware for bimanual teleoperation . arXiv preprint arXiv:2405.02292 . Cited by: §5 .

Baker et al. (2022) B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune Video pretraining (vpt): learning to act by watching unlabeled online videos . Advances in Neural Information Processing Systems 35 , pp. 24639–24654 . Cited by: §1 , §2.1 , §2.2 , §5 , §5 .

Berner et al. (2019) C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning . arXiv preprint arXiv:1912.06680 . Cited by: §1 , §5 .

Bjorck et al. (2025) J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: an open foundation model for generalist humanoid robots . arXiv preprint arXiv:2503.14734 . Cited by: §A.1 , §2.3 , §5 .

Black et al. (2024a) K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky π 0 \pi_{0} : A vision-language-action flow model for general robot control . External Links: 2410.24164 , Link Cited by: §A.1 , §2.3 .

Black et al. (2024b) K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π 0 \pi_{0} : A vision-language-action flow model for general robot control . arXiv preprint arXiv:2410.24164 . Cited by: §5 .

Brohan et al. (2022) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: robotics transformer for real-world control at scale . arXiv preprint arXiv:2212.06817 . Cited by: §5 .

Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language models are few-shot learners . External Links: 2005.14165 , Link Cited by: §1 .

Bruce et al. (2024) J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. Genie: generative interactive environments . In Forty-first International Conference on Machine Learning , Cited by: §5 .

Bu et al. (2025) Q. Bu, J. Cai, L. Chen, X. Cui, Y. Ding, S. Feng, S. Gao, X. He, X. Hu, X. Huang, et al. Agibot world colosseo: a large-scale manipulation platform for scalable and intelligent embodied systems . arXiv preprint arXiv:2503.06669 . Cited by: §5 .

Cheang et al. (2024) C. Cheang, G. Chen, Y. Jing, T. Kong, H. Li, Y. Li, Y. Liu, H. Wu, J. Xu, Y. Yang, et al. Gr-2: a generative video-language-action model with web-scale knowledge for robot manipulation . arXiv preprint arXiv:2410.06158 . Cited by: §5 .

Dass et al. (2024) S. Dass, W. Ai, Y. Jiang, S. Singh, J. Hu, R. Zhang, P. Stone, B. Abbatematteo, and R. Martín-Martín Telemoma: a modular and versatile teleoperation system for mobile manipulation . arXiv preprint arXiv:2403.07869 . Cited by: §5 .

Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding . External Links: 1810.04805 , Link Cited by: §1 .

Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby An image is worth 16x16 words: transformers for image recognition at scale . External Links: 2010.11929 , Link Cited by: §1 .

Driess et al. (2023) D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence PaLM-e: an embodied multimodal language model . In arXiv preprint arXiv:2303.03378 , Cited by: §5 .

Edwards et al. (2019) A. Edwards, H. Sahni, Y. Schroecker, and C. Isbell Imitating latent policies from observation . In International conference on machine learning , pp. 1755–1763 . Cited by: §5 .

Fan et al. (2022) L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D. Huang, Y. Zhu, and A. Anandkumar Minedojo: building open-ended embodied agents with internet-scale knowledge . Advances in Neural Information Processing Systems 35 , pp. 18343–18362 . Cited by: §5 .

Farhang et al. (2024) A. R. Farhang, B. Mulcahy, D. Holden, I. Matthews, and Y. Yue Humanlike behavior in a third-person shooter with imitation learning . In 2024 IEEE Conference on Games (CoG) , pp. 1–4 . Cited by: §5 .

Guss et al. (2019) W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov Minerl: a large-scale dataset of minecraft demonstrations . arXiv preprint arXiv:1907.13440 . Cited by: §2.2 , §5 , §5 .

Hafner et al. (2023) D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap Mastering diverse domains through world models . arXiv preprint arXiv:2301.04104 . Cited by: §5 .

Hafner et al. (2025) D. Hafner, W. Yan, and T. Lillicrap Training agents inside of scalable world models . External Links: 2509.24527 , Link Cited by: §5 .

Huang et al. (2022) W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: embodied reasoning through planning with language models . arXiv preprint arXiv:2207.05608 . Cited by: §5 .

Kim et al. (2024) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: an open-source vision-language-action model . arXiv preprint arXiv:2406.09246 . Cited by: §5 .

Kingma and Ba (2017) D. P. Kingma and J. Ba Adam: a method for stochastic optimization . External Links: 1412.6980 , Link Cited by: §2.1 , §2.3 .

Liang et al. (2023) J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng Code as policies: language model programs for embodied control . In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pp. 9493–9500 . Cited by: §5 .

Lipman et al. (2022) Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling . arXiv preprint arXiv:2210.02747 . Cited by: §2.3 , §2.3 .

Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization . arXiv preprint arXiv:1711.05101 . Cited by: §2.1 , §2.3 .

Lowe (2004) D. G. Lowe Distinctive image features from scale-invariant keypoints . International journal of computer vision 60 , pp. 91–110 . Cited by: §2.1 .

Mandlekar et al. (2019) A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and L. Fei-Fei Scaling robot supervision to hundreds of hours with roboturk: robotic manipulation dataset through human reasoning and dexterity . In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 1048–1055 . Cited by: §5 .

Mandlekar et al. (2020) A. Mandlekar, D. Xu, R. Martín-Martín, Y. Zhu, L. Fei-Fei, and S. Savarese Human-in-the-loop imitation learning using remote teleoperation . arXiv preprint arXiv:2012.06733 . Cited by: §5 .

Mandlekar et al. (2018) A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, et al. Roboturk: a crowdsourcing platform for robotic skill learning through imitation . In Conference on Robot Learning , pp. 879–893 . Cited by: §5 .

Mnih et al. (2013) V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller Playing atari with deep reinforcement learning . arXiv preprint arXiv:1312.5602 . Cited by: §1 , §5 .

Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning . nature 518 ( 7540 ), pp. 529–533 . Cited by: §1 , §5 .

O’Neill et al. (2024) A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. Open x-embodiment: robotic learning datasets and rt-x models: open x-embodiment collaboration 0 . In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pp. 6892–6903 . Cited by: §5 .

Parker-Holder et al. (2024) J. Parker-Holder, P. Ball, J. Bruce, V. Dasagi, K. Holsheimer, C. Kaplanis, A. Moufarek, G. Scully, J. Shar, J. Shi, et al. Genie 2: a large-scale foundation world model . URL: https://deepmind. google/discover/blog/genie-2-a-large-scale-foundation-world-model . Cited by: §5 .

Peebles and Xie (2023) W. Peebles and S. Xie Scalable diffusion models with transformers . In Proceedings of the IEEE/CVF international conference on computer vision , pp. 4195–4205 . Cited by: §2.3 , §2.3 .

Potje et al. (2024) G. Potje, F. Cadar, A. Araujo, R. Martins, and E. R. Nascimento Xfeat: accelerated features for lightweight image matching . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2682–2691 . Cited by: §2.1 .

Raad et al. (2024) M. A. Raad, A. Ahuja, C. Barros, F. Besse, A. Bolt, A. Bolton, B. Brownfield, G. Buttimore, M. Cant, S. Chakera, et al. Scaling instructable agents across many simulated worlds . arXiv preprint arXiv:2404.10179 . Cited by: §1 , §5 .

Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision . External Links: 2103.00020 , Link Cited by: §1 .

Reed et al. (2022) S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas A generalist agent . External Links: 2205.06175 , Link Cited by: §5 .

Silver et al. (2016) D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search . nature 529 ( 7587 ), pp. 484–489 . Cited by: §1 , §5 .

Singh et al. (2023) I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg Progprompt: generating situated robot task plans using large language models . In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pp. 11523–11530 . Cited by: §5 .

Tan et al. (2025) W. Tan, X. Li, Y. Fang, H. Yao, S. Yan, H. Luo, T. Ao, H. Li, H. Ren, B. Yi, Y. Qin, B. An, L. Liu, and G. Shi Lumine: an open recipe for building generalist agents in 3d open worlds . External Links: 2511.08892 , Link Cited by: §5 .

Tan et al. (2024) W. Tan, W. Zhang, X. Xu, H. Xia, Z. Ding, B. Li, B. Zhou, J. Yue, J. Jiang, Y. Li, et al. Cradle: empowering foundation agents towards general computer control . arXiv preprint arXiv:2403.03186 . Cited by: §1 , §5 .

Team et al. (2024) O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: an open-source generalist robot policy . arXiv preprint arXiv:2405.12213 . Cited by: §5 .

Towers et al. (2024) M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goulão, A. Kallinteris, M. Krimmel, A. KG, et al. Gymnasium: a standard interface for reinforcement learning environments . arXiv preprint arXiv:2407.17032 . Cited by: §1 , §2.2 .

Tschannen et al. (2025) M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. Siglip 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features . arXiv preprint arXiv:2502.14786 . Cited by: §2.3 .

Vinyals et al. (2019) O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning . nature 575 ( 7782 ), pp. 350–354 . Cited by: §1 , §5 .

Volum et al. (2022) R. Volum, S. Rao, M. Xu, G. DesGarennes, C. Brockett, B. Van Durme, O. Deng, A. Malhotra, and B. Dolan Craft an iron sword: dynamically generating interactive game characters by prompting large language models tuned on code . In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022) , M. Côté, X. Yuan, and P. Ammanabrolu (Eds.) , Seattle, United States , pp. 25–43 . External Links: Link , Document Cited by: §1 .

Wang et al. (2023) G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar Voyager: an open-ended embodied agent with large language models . arXiv preprint arXiv:2305.16291 . Cited by: §1 , §5 .

Wang et al. (2024) Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, and Y. Liang Describe, explain, plan and select: interactive planning with large language models enables open-world multi-task agents . External Links: 2302.01560 , Link Cited by: §1 .

Wang et al. (2025) Z. Wang, X. Li, Y. Ye, J. Fang, H. Wang, L. Liu, S. Liang, J. Lu, Z. Wu, J. Feng, W. Zhong, Z. Li, Y. Wang, Y. Miao, B. Zhou, Y. Li, H. Wang, Z. Zhao, F. Wu, Z. Jiang, W. Tan, H. Yao, S. Yan, X. Li, Y. Liang, Y. Qin, and G. Shi Game-tars: pretrained foundation models for scalable generalist multimodal game agents . External Links: 2510.23691 , Link Cited by: §5 .

Wen et al. (2025) J. Wen, Y. Zhu, J. Li, M. Zhu, Z. Tang, K. Wu, Z. Xu, N. Liu, R. Cheng, C. Shen, et al. Tinyvla: towards fast, data-efficient vision-language-action models for robotic manipulation . IEEE Robotics and Automation Letters . Cited by: §5 .

Wen et al. (2024) K. Wen, Z. Li, J. Wang, D. Hall, P. Liang, and T. Ma Understanding warmup-stable-decay learning rates: a river valley loss landscape perspective . arXiv preprint arXiv:2410.05192 . Cited by: §2.3 .

Xie et al. (2021) E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Álvarez, and P. Luo SegFormer: simple and efficient design for semantic segmentation with transformers . In Neural Information Processing Systems , External Links: Link Cited by: §2.1 .

Ye et al. (2024) S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y. Chao, B. Y. Lin, et al. Latent action pretraining from videos . arXiv preprint arXiv:2410.11758 . Cited by: §5 .

Appendix A NitroGen model details

附录 A NitroGen 模型细节

A.1 Training objective

A.1 训练目标

Given a ground-truth action chunk $a\in\mathbb{R}^{16\times 24}$, an observation $o\in\mathbb{R}^{256\times 256}$, a flow-matching timestep $t\in[0,1]$, and Gaussian noise $\epsilon\sim\mathcal{N}(0,\mathcal{I})$, we construct the noisy action as

给定真值动作块 $a\in\mathbb{R}^{16\times 24}$、观测 $o\in\mathbb{R}^{256\times 256}$、流匹配时间步 $t\in[0,1]$，以及高斯噪声 $\epsilon\sim\mathcal{N}(0,\mathcal{I})$，我们把带噪声的动作构造为

$$a_{t}=(1-t)\cdot\epsilon+t\cdot a$$

and define the conditional velocity field as

并把条件速度场定义为

$$u^{\text{cond}}(x,t,a,\epsilon,o)=a-\epsilon.$$

The model is trained to predict the velocity field by minimizing the conditional flow-matching loss:

模型通过最小化条件流匹配损失来预测该速度场：

$$\mathcal{L}^{CFM}(\theta,\phi)=\mathbb{E}_{t,a,\epsilon}\left[\left\lVert\pi_{\theta}(a_{t},\psi_{\phi}(o),t)-(a-\epsilon)\right\rVert^{2}\right], \tag{1}$$

where $\pi_{\theta}$ is the DiT and $\psi_{\phi}$ is the image encoder. Following Bjorck et al. [2025], Black et al. [2024a], we sample $t$ from a shifted beta distribution that prioritizes small timesteps.

其中 $\pi_{\theta}$ 是 DiT，$\psi_{\phi}$ 是图像编码器。按照 Bjorck et al. [2025]、Black et al. [2024a]，我们从偏向较小时间步的平移 beta 分布中采样 $t$。

A.2 Inference

A.2 推理

At inference time, we initialize $a_{0}\sim\mathcal{N}(0,\mathcal{I})$ and iteratively denoise for $k$ steps using Euler integration:

推理时，我们初始化 $a_{0}\sim\mathcal{N}(0,\mathcal{I})$，并用欧拉积分迭代去噪 $k$ 步：

$$a_{t+1/k}=a_{t}+\frac{1}{k}\pi_{\theta}(a_{t},\psi_{\phi}(o),t). \tag{2}$$

We use $k=16$ denoising steps, as additional steps yield no measurable improvement.

我们使用 $k=16$ 步去噪，再增加步数没有可测到的改善。

Appendix B Evaluation

附录 B 评测

B.1 Synchronous inference

B.1 同步推理

As described in Section 2.2, we use a Gymnasium API that freezes the game while the model predicts the next action. Frequent pausing and resuming during gameplay could potentially affect the game’s physics engine in unknown ways. To rule out this possibility, we record videos and actions (ground truth) of humans playing several games for five minutes each, focusing on parts of the game that are expected to be deterministic (e.g., no enemy behavior randomness). We then replay the same actions from the same initial position: (a) in real time without pausing, and (b) while pausing and resuming the game with random pause durations at high frequency. We find that replayed sequences begin visually diverging after one minute for games with continuous actions and after about three minutes for games with only discrete actions. This result is the same for both (a) and (b), confirming the correctness of our approach: the divergence is simply due to error accumulation.

如第 2.2 节所述，我们使用在模型预测下一步动作时暂停游戏的 Gymnasium API。频繁暂停和恢复可能以未知方式影响游戏的物理引擎。为排除这一可能，我们录下人类在若干游戏中各游玩五分钟的视频和动作（真值），并尽量选择预期是确定性的片段（例如没有敌人行为随机性）。随后从同一初始位置重放相同动作：（a）实时、不暂停；（b）以较高频率暂停和恢复，暂停时长随机。我们发现，对于带连续动作的游戏，重放序列约在一分钟后开始出现视觉偏离；对于只有离散动作的游戏，约在三分钟后出现。这一结果在（a）和（b）中相同，从而确认了我们方法的正确性：偏离只是误差累积所致。

Footnotes

脚注

[^4]: Open Broadcaster Software: https://obsproject.com/; Input recording tool: https://github.com/loicmagne/input-rec