MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

MineDojo：用互联网规模知识构建开放式具身智能体

Linxi Fan Guanzhi Wang Yunfan Jiang Ajay Mandlekar Yuncong Yang Haoyi Zhu Affiliation: NVIDIA, Caltech, Stanford, Columbia, SJTU, UT Austin Andrew Tang De-An Huang Yuke Zhu Anima Anandkumar

Abstract

摘要

Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports a multitude of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MineDojo, a new framework built on the popular Minecraft game that features a simulation suite with thousands of diverse open-ended tasks and an internet-scale knowledge base with Minecraft videos, tutorials, wiki pages, and forum discussions. Using MineDojo’s data, we propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. We open-source the simulation suite, knowledge bases, algorithm implementation, and pretrained models (https://minedojo.org) to promote research towards the goal of generally capable embodied agents.

自主智能体在 Atari 游戏和围棋这类专项领域已经取得很大进展。可它们通常是在隔离环境里从零开始（ tabula rasa ）学习，目标既有限、又由人预先想好，因此很难在更宽的任务和能力谱上泛化。受人类在开放世界中持续学习、不断适应的启发，我们主张构建通用智能体需要三位一体的要素：1）能支撑大量任务与目标的环境，2）大规模多模态知识库，3）灵活且可扩展的智能体结构。我们提出 MineDojo，这是一套建立在广受欢迎的 Minecraft 游戏之上的新框架，既包含带有数千个多样开放式任务的仿真套件，也包含由 Minecraft 视频、教程、Wiki 页面和论坛讨论组成的互联网规模知识库。利用 MineDojo 的数据，我们提出一种新的智能体学习算法，把大规模预训练的视频-语言模型当作学到的奖励函数来用。我们的智能体能够完成用自由格式语言指定的多种开放式任务，而不需要任何人工设计的稠密塑形奖励。我们开源了仿真套件、知识库、算法实现和预训练模型（ https://minedojo.org），以推动面向通用具身智能体这一目标的研究。

1 Introduction

1 引言

Figure 1: MineDojo is a novel framework for developing open-ended, generally capable agents that can learn and adapt continually to new goals. MineDojo features a benchmarking suite with thousands of diverse open-ended tasks specified in natural language prompts, and also provides an internet-scale, multimodal knowledge base of YouTube videos, Wiki pages, and Reddit posts. The database captures the collective experience and wisdom of millions of Minecraft gamers for an AI agent to learn from. Best viewed zoomed in.

图 1：MineDojo 是一套用于发展开放式、具备通用能力的智能体的新框架，这类智能体能够持续学习并适应新目标。MineDojo 提供一个基准套件，其中有用自然语言提示指定的 数千个多样开放式任务 ，同时还提供由 YouTube 视频、Wiki 页面和 Reddit 帖子组成的 互联网规模多模态知识库 。该数据库捕捉了数百万 Minecraft 玩家的集体经验和智慧，供人工智能智能体学习。建议放大观看。

Developing autonomous embodied agents that can attain human-level performance across a wide spectrum of tasks has been a long-standing goal for AI research. There has been impressive progress towards this goal, most notably in games [80, 85, 126] and robotics [68, 99, 146, 134, 107]. These embodied agents are typically trained tabula rasa in isolated worlds with limited complexity and diversity. Although highly performant, they are specialist models that do not generalize beyond a narrow set of tasks. In contrast, humans inhabit an infinitely rich reality, continuously learn from and adapt to a wide variety of open-ended tasks, and are able to leverage large amount of prior knowledge from their own experiences as well as others.

打造能在广泛任务上达到人类水平的自主具身智能体，一直是人工智能研究的长期目标。朝着这个目标已经有了令人印象深刻的进展，尤其是在游戏 [80, 85, 126] 和机器人 [68, 99, 146, 134, 107] 中。这些具身智能体通常是在复杂度与多样性都有限的隔离世界里从零训练。它们虽然表现很强，却是专项模型，无法超出狭窄任务集合去泛化。相比之下，人类生活在无限丰富的现实里，持续从各种各样的开放式任务中学习并适应，还能大量利用自己的经验以及他人的先验知识。

We argue that three main pillars are necessary for generalist embodied agents to emerge. First, the environment in which the agent acts needs to enable an unlimited variety of open-ended goals [116, 71, 120, 117]. Natural evolution is able to nurture an ever-expanding tree of diverse life forms thanks to the infinitely varied ecological settings that the Earth supports [117, 129]. This process has not stagnated for billions of years. In contrast, today’s agent training algorithms cease to make new progress after convergence in narrow environments [80, 146]. Second, a large-scale database of prior knowledge is necessary to facilitate learning in open-ended settings. Just as humans frequently learn from the internet, agents should also be able to harvest practical knowledge encoded in large amounts of video demos [42, 77], multimedia tutorials [79], and forum discussions [127, 65, 54]. In a complex world, it would be extremely inefficient for an agent to learn everything from scratch through trial and error. Third, the agent’s architecture needs to be flexible enough to pursue any task in open-ended environments, and scalable enough to convert large-scale knowledge sources into actionable insights [19, 96]. This motivates the design of an agent that has a unified observation/action space, conditions on natural language task prompts, and adopts the Transformer pre-training paradigm [27, 91, 15] to internalize knowledge effectively.

我们认为，通用具身智能体的出现需要 三根主柱 。第一，智能体活动的环境必须 能够支撑无限多样的开放式目标 [116, 71, 120, 117]。自然演化之所以能培育出不断扩展的多样生命之树，靠的是地球所支持的无限多样的生态条件 [117, 129]。这个过程已经数十亿年没有停滞。相比之下，今天的智能体训练算法一旦在狭窄环境中收敛，就不会再有新的进展 [80, 146]。第二，开放式设定下的学习需要 大规模的先验知识库 。正如人类经常从互联网学习，智能体也应该能从大量视频演示 [42, 77]、多媒体教程 [79] 和论坛讨论 [127, 65, 54] 中收获实用知识。在一个复杂世界里，让智能体事事靠试错从零学起会极其低效。第三， 智能体的结构 需要足够灵活，才能在开放式环境中追求任意任务；也需要足够可扩展，才能把大规模知识来源转化成可执行的洞见 [19, 96]。这促使我们设计一种智能体：它有统一的观察/动作空间，以自然语言任务提示为条件，并采用 Transformer 预训练范式 [27, 91, 15] 来有效内化知识。

In light of these three pillars, we introduce MineDojo, a new framework to help the community develop open-ended, generally-capable agents. It is built on the popular Minecraft game, where a player explores a procedurally generated 3D world with diverse types of terrains to roam, materials to mine, tools to craft, structures to build, and wonders to discover. Unlike most other games [80, 85, 126], Minecraft defines no specific reward to maximize and no fixed storyline to follow, making it well suited for developing open-ended environments for embodied AI research. We make the following three major contributions:

基于这三根主柱，我们提出 MineDojo，帮助社区发展开放式、具备通用能力的智能体。它建立在广受欢迎的 Minecraft 游戏之上：玩家探索一个程序化生成的三维世界，其中有多样的地形可以漫游、材料可以开采、工具可以合成、结构可以建造，还有各种奇观可以发现。和多数其他游戏 [80, 85, 126] 不同，Minecraft 没有规定必须最大化的特定奖励，也没有必须遵循的固定剧情，因此很适合用来为具身人工智能研究发展开放式环境。我们做出以下三项主要贡献：

1. Simulation platform with thousands of diverse open-ended tasks. MineDojo provides convenient APIs on top of Minecraft that standardize task specification, world settings, and agent’s observation/action spaces. We introduce a benchmark suite that consists of thousands of natural language-prompted tasks, making it two orders of magnitude larger than prior Minecraft benchmarks like the MineRL Challenge [48, 62]. The suite includes long-horizon, open-ended tasks that cannot be easily evaluated through automated procedures, such as “ build an epic modern house with two floors and a swimming pool ”. Inspired by the Inception score [98] and FID score [55] that are commonly used to assess AI-generated image quality, we introduce a novel agent evaluation protocol using a large video-language model pre-trained on Minecraft YouTube videos. This complements human scoring [104] that is precise but more expensive. Our learned evaluation metric has good agreement with human judgment in a subset of the full task suite considered in the experiments.

1. 带有数千个多样开放式任务的仿真平台。 MineDojo 在 Minecraft 之上提供便捷的 API，用来标准化任务规格、世界设置以及智能体的观察/动作空间。我们引入一个由数千个自然语言提示任务组成的基准套件，规模比此前的 Minecraft 基准（如 MineRL Challenge [48, 62]）大约高出 两个数量级 。套件中包含难以用自动流程评估的长视野、开放式任务，例如“建一座带两层楼和游泳池的史诗级现代房子”。受常用于评估人工智能生成图像质量的 Inception 分数 [98] 和 FID 分数 [55] 启发，我们引入一种新的智能体评估协议，使用在 Minecraft YouTube 视频上预训练的大规模视频-语言模型。它补充了精确但更昂贵的人工评分 [104]。在实验所考虑的完整任务套件的一个子集上，我们学到的评估指标与人类判断吻合良好。

2. Internet-scale multimodal Minecraft knowledge base. Minecraft has more than 100 million active players [131], who have collectively generated an enormous wealth of data. They record tutorial videos, stream live play sessions, compile recipes, and discuss tips and tricks on forums. MineDojo features a massive collection of 730K+ YouTube videos with time-aligned transcripts, 6K+ free-form Wiki pages, and 340K+ Reddit posts with multimedia contents (Fig. 3). We hope that this enormous knowledge base can help the agent acquire diverse skills, develop complex strategies, discover interesting objectives, and learn actionable representations automatically.

2. 互联网规模的多模态 Minecraft 知识库。 Minecraft 拥有超过 1 亿活跃玩家 [131]，他们共同产生了海量数据。他们录制教程视频、直播游戏过程、整理配方，并在论坛上讨论技巧。MineDojo 收录了超过 73 万条带时间对齐字幕的 YouTube 视频、6 千多篇自由格式 Wiki 页面，以及 34 万多条带多媒体内容的 Reddit 帖子（图 3）。我们希望这座巨大的知识库能帮助智能体自动获得多样技能、发展复杂策略、发现有趣目标，并学到可执行的表征。

3. Novel algorithm for embodied agents with large-scale pre-training. We develop a new learning algorithm for embodied agents that makes use of the internet-scale domain knowledge we have collected from the web. Using the massive volume of YouTube videos from MineDojo, we train a video-text contrastive model in the spirit of CLIP [92], which associates natural language subtitles with their time-aligned video segments. We demonstrate that this learned correlation score can be used effectively as an open-vocabulary, massively multi-task reward function for RL training. Our agent solves the majority of 12 tasks in our experiment using the learned reward model (Fig. 2). It achieves competitive performance to agents trained with meticulously engineered dense-shaping rewards, and in some cases outperforms them, with up to 73% improvement in success rates. For open-ended tasks that do not have a simple success criterion, our agents also perform well without any special modifications.

3. 面向具身智能体、带大规模预训练的新算法。 我们为具身智能体设计了一种新的学习算法，利用我们从网上收集的互联网规模领域知识。借助 MineDojo 中海量的 YouTube 视频，我们按 CLIP [92] 的思路训练一个视频-文本对比模型，把自然语言字幕与时间对齐的视频片段联系起来。我们表明，学到的相关分数可以有效地用作强化学习训练中的 开放词汇、海量多任务奖励函数 。我们的智能体利用学到的奖励模型，完成了实验中 12 个任务的大多数（图 2）。它与用精心设计的稠密塑形奖励训练的智能体表现相当，在有些情况下还超过它们，成功率最高提升约 73%。对于没有简单成功标准的开放式任务，我们的智能体同样表现良好，不需要任何特殊改动。

In summary, this paper proposes an open-ended task suite, internet-scale domain knowledge, and agent learning with recent advances on large pre-trained models [13]. We have open-sourced MineDojo’s simulator, knowledge bases, algorithm implementations, pretrained model checkpoints, and task curation tools at https://minedojo.org/ . We hope that MineDojo will serve as an effective starter framework for the community to develop new algorithms and advance towards generally capable embodied agents.

总之，本文提出一套开放式任务套件、互联网规模的领域知识，以及结合大规模预训练模型近期进展的智能体学习方法 [13]。我们已在 https://minedojo.org/ 开源 MineDojo 的仿真器、知识库、算法实现、预训练模型检查点以及任务整理工具。希望 MineDojo 能成为社区发展新算法、迈向通用具身智能体的有效起步框架。

Figure 2: Visualization of our agent’s learned behaviors on four selected tasks. Leftmost texts are the task prompts used in training. Best viewed on a color display.

图 2：我们的智能体在四个选定任务上学到的行为可视化。最左侧文字是训练时使用的任务提示。建议在彩色显示器上观看。

2 MineDojo Simulator & Benchmark Suite

2 MineDojo 仿真器与基准套件

MineDojo offers a set of simulator APIs help researchers develop generally capable, open-ended agents in Minecraft. It builds upon the open-source MineRL codebase [48] and makes the following upgrades: 1) We provide unified observation and action spaces across all tasks, facilitating the development of multi-task and continually learning agents that can constantly adapt to new scenarios and novel tasks. This deviates from the MineRL Challenge design that tailors observation and action spaces to individual tasks; 2) Our simulation unlocks all three types of worlds in Minecraft, including the Overworld, the Nether, and the End, which substantially expands the possible task space, while MineRL only supports the Overworld natively; and 3) We provide convenient APIs to configure initial conditions and world settings to standardize our tasks.

MineDojo 提供一组仿真器 API，帮助研究者在 Minecraft 中发展具备通用能力的开放式智能体。它建立在开源 MineRL 代码库 [48] 之上，并做了如下升级：1）我们在所有任务上提供 统一的观察和动作空间 ，便于发展能够不断适应新场景和新任务的多任务、持续学习智能体。这不同于 MineRL Challenge 那种为单个任务定制观察和动作空间的设计；2）我们的仿真解锁了 Minecraft 中全部三类世界，包括 主世界 、 下界 和 末地 ，从而 大幅扩展了可能的任务空间 ，而 MineRL 原生只支持主世界；3）我们提供便捷的 API 来配置初始条件和世界设置，以标准化我们的任务。

With this MineDojo simulator, we define thousands of benchmarking tasks, which are divided into two categories: 1) Programmatic tasks that can be automatically assessed based on the ground-truth simulator states; and 2) Creative tasks that do not have well-defined or easily-automated success criteria, which motivates our novel evaluation protocol using a learned model (Sec. 4). To scale up the number of Creative tasks, we mine ideas from YouTube tutorials and use OpenAI’s GPT-3 [15] service to generate substantially more task definitions. Compared to Creative tasks, Programmatic tasks are simpler to get started, but tend to have restricted scope, limited language variations, and less open-endedness in general.

借助这套 MineDojo 仿真器，我们定义了数千个基准任务，分成两类：1） 程序化任务 ，可以根据仿真器的真实状态自动评估；2） 创意任务 ，没有定义清晰或易于自动化的成功标准，这促使我们使用学到的模型提出新的评估协议（第 4 节）。为了扩大创意任务的数量，我们从 YouTube 教程中挖掘想法，并使用 OpenAI 的 GPT-3 [15] 服务生成更多任务定义。与创意任务相比，程序化任务更容易上手，但范围通常更受限制，语言变化更少，总体上也不那么开放。

2.1 Task Suite I: Programmatic Tasks

2.1 任务套件一：程序化任务

T=(G,\mathcal{G},\mathcal{I},f_{\mathcal{S}},f_{\mathcal{R}})

T=(G,\mathcal{G},\mathcal{I},f_{\mathcal{S}},f_{\mathcal{R}})

MineDojo provides 4 categories of programmatic tasks with 1,581 template-generated natural language goals to evaluate the agent’s different capabilities systematically and comprehensively:

MineDojo 提供 4 类程序化任务，共有 1,581 个由模板生成的自然语言目标，用来系统而全面地评估智能体的不同能力：

Survival : surviving for a designated number of days.

生存 ：在指定天数内存活。

Harvest : finding, obtaining, cultivating, or manufacturing hundreds of materials and objects.

采集 ：寻找、获得、培育或制造数百种材料和物体。

Tech Tree : crafting and using a hierarchy of tools.

科技树 ：合成并使用一层层工具。

Combat : fighting various monsters and creatures that require fast reflex and martial skills.

战斗 ：与各种需要快速反应和战斗技巧的怪物和生物作战。

Each task template has a number of variations based on the terrain, initial inventory, quantity, etc., which form a flexible spectrum of difficulty. In comparison, the NeurIPS MineRL Diamond challenge [48] is a subset of our programmatic task suite, defined by the task goal “ obtain 1 diamond " in MineDojo.

每个任务模板会按地形、初始背包、数量等产生若干变体，形成灵活的难度谱。相比之下，NeurIPS 的 MineRL Diamond 挑战 [48] 是我们程序化任务套件的一个子集，在 MineDojo 中由任务目标“获得 1 颗钻石”定义。

2.2 Task Suite II: Creative Tasks

2.2 任务套件二：创意任务

T=(G,\mathcal{G},\mathcal{I})

T=(G,\mathcal{G},\mathcal{I})

We brainstorm and author 216 Creative tasks, such as “ build a haunted house with zombie inside ” and “ race by riding a pig ”. Nonetheless, such a manual approach is not scalable. Therefore, we develop two systematic approaches to extend the total number of task definitions to 1,560. This makes our Creative tasks 3 orders of magnitude larger than Minecraft BASALT challenge [104], which has 4 Creative tasks.

我们头脑风暴并亲手写了 216 个创意任务，例如“建一座里面有僵尸的鬼屋”和“骑猪赛跑”。不过这种手工方法难以扩展。于是我们发展了两种系统方法，把任务定义总数扩展到 1,560 个。这使我们的创意任务比只有 4 个创意任务的 Minecraft BASALT 挑战 [104] 大约高出 三个数量级 。

Approach 1. Task Mining from YouTube Tutorial Videos. We identify our YouTube dataset as a rich source of tasks, as many human players demonstrate and narrate creative missions in the tutorial playlists. To collect high-quality tasks and accompanying videos, we design a 3-stage pipeline that makes it easy to find and annotate interesting tasks (see Sec. C.2 for details). Through this pipeline, we extract 1,042 task ideas from the common wisdom of a huge number of veteran Minecraft gamers, such as “ make an automated mining machine ” and “ grow cactus up to the sky ”.

方法 1. 从 YouTube 教程视频中挖掘任务。 我们把 YouTube 数据集视为丰富的任务来源，因为许多人类玩家会在教程播放列表里演示并讲述有创意的任务。为了收集高质量任务及配套视频，我们设计了一个三阶段流水线，便于发现和标注有趣任务（细节见第 C.2 节）。通过这条流水线，我们从大量资深 Minecraft 玩家的共同智慧中提取出 1,042 个任务想法，例如“做一台自动采矿机”和“把仙人掌种到天上”。

Approach 2. Task Creation by GPT-3. We leverage GPT-3’s few-shot capability to generate new task ideas by seeding it with the tasks we manually author or mine from YouTube. The prompt template is: Here are some example creative tasks in Minecraft: {a few examples}. Let’s brainstorm more detailed while reasonable creative tasks in Minecraft. GPT-3 contributes 302 creative tasks after de-duplication, and demonstrates a surprisingly proficient understanding of Minecraft terminology.

方法 2. 由 GPT-3 创建任务。 我们利用 GPT-3 的少样本能力，把我们手工撰写或从 YouTube 挖掘出的任务作为种子，生成新的任务想法。提示模板是：Here are some example creative tasks in Minecraft: {若干例子}. Let’s brainstorm more detailed while reasonable creative tasks in Minecraft。去重之后，GPT-3 贡献了 302 个创意任务，并表现出对 Minecraft 术语出人意料的熟练理解。

2.3 Collection of Starter Tasks

2.3 入门任务合集

We curate a set of 64 core tasks for future researchers to get started more easily. If their agent works well on these tasks, they can more confidently scale to the full benchmark.

我们整理了一组 64 个核心任务，方便后续研究者更容易上手。如果他们的智能体在这些任务上表现良好，就可以更有信心地扩展到完整基准。

• 32 programmatic tasks : 16 “standard” and 16 “difficult”, spanning all 4 categories (survival, harvesting, combat, and tech tree). We rely on our Minecraft knowledge to decide the difficulty level. “Standard” tasks require fewer steps and lower resource dependencies to complete.

• 32 个程序化任务 ：16 个“标准”和 16 个“困难”，覆盖全部 4 类（生存、采集、战斗和科技树）。我们依据自己的 Minecraft 知识来判定难度。“标准”任务完成所需步数更少，对资源依赖也更低。

• 32 creative tasks : 16 “standard” and 16 “difficult”. Similarly, tasks labeled with “standard” are typically short-horizon tasks.

• 32 个创意任务 ：16 个“标准”和 16 个“困难”。同样，标为“标准”的任务通常是短视野任务。

We recommend that researchers run 100 evaluation episodes for each task and report the percentage success rate. The programmatic tasks have ground-truth success, while the creative tasks need our novel evaluation protocol (Sec. 5).

我们建议研究者对每个任务运行 100 个评估回合，并报告成功率百分比。程序化任务有真实成功判定，创意任务则需要我们的新评估协议（第 5 节）。

Figure 3: MineDojo’s internet-scale, multimodal knowledge base. Left, YouTube videos: Minecraft gamers showcase the impressive feats they are able to achieve. Clockwise order: an archery range, Hogwarts castle, Taj Mahal, a Nether homebase. Middle, Wiki: Wiki pages contain multimodal knowledge in structured layouts, such as comprehensive catalogs of creatures and recipes for crafting. More examples in Fig. A.4 and A.5. Right, Reddit: We create a word cloud from Reddit posts and comment threads. Gamers ask questions, share achievements, and discuss strategies extensively. Sample posts in Fig. A.7. Best viewed zoomed in.

图 3：MineDojo 的互联网规模多模态知识库。 左侧，YouTube 视频 ： Minecraft 玩家展示他们能够完成的惊人成就。按顺时针依次是：一座射箭场、霍格沃茨城堡、泰姬陵，以及一处下界基地。 中间，Wiki ： Wiki 页面以结构化版面提供多模态知识，例如生物的完备目录和合成配方。更多例子见图 A.4 和 A.5。 右侧，Reddit ： 我们根据 Reddit 帖子和评论线程制作了词云。玩家大量提问、分享成就并讨论策略。示例帖子见图 A.7。建议放大观看。

3 Internet-scale Knowledge Base

3 互联网规模知识库

Two commonly used approaches [112, 126, 85, 36] to train embodied agents include training agents from scratch using RL with well-tuned reward functions for each task, or using a large amount of human-demonstrations to bootstrap agent learning. However, crafting well-tuned reward functions is challenging or infeasible for our task suite (Sec. 2.2), and employing expert gamers to provide large amounts of demonstration data would also be costly and infeasible [126].

训练具身智能体常用的两种做法 [112, 126, 85, 36] 是：为每个任务调好奖励函数后用强化学习从零训练，或者用大量人类演示来启动智能体学习。可是，为我们的任务套件精心设计奖励函数很困难，甚至不可行（第 2.2 节）；雇用资深玩家提供大量演示数据同样成本高昂、难以实现 [126]。

Instead, we turn to the open web as an ever-growing, virtually unlimited source of learning material for embodied agents. The internet provides a vast amount of domain knowledge about Minecraft, which we harvest by extensive web scraping and filtering. We collect 33 years worth of YouTube videos, 6K+ Wiki pages, and millions of Reddit comment threads. Instead of hiring a handful of human demonstrators, we capture the collective wisdom of millions of Minecraft gamers around the world. Furthermore, language is a key and pervasive component of our database that takes the form of YouTube transcripts, textual descriptions in Wiki, and Reddit discussions. Language facilitates open-vocabulary understanding, provides grounding for image and video modalities, and unlocks the power of large language models [27, 109, 15] for embodied agents. To ensure socially responsible model development, we take special measures to filter out low-quality and toxic contents [13, 51] from our databases, detailed in the Appendix (Sec. D).

于是我们转向开放网络，把它当作具身智能体近乎无限、还在不断增长的学习材料来源。互联网提供了关于 Minecraft 的海量领域知识，我们通过广泛抓取和过滤来收集。我们收集了总时长相当于 33 年的 YouTube 视频、6 千多篇 Wiki 页面，以及数百万条 Reddit 评论线程。我们不是雇用少数几位人类演示者，而是捕捉全世界数百万 Minecraft 玩家的集体智慧。此外，语言是我们数据库中关键且无处不在的组成部分，表现为 YouTube 字幕、Wiki 中的文字描述和 Reddit 讨论。语言有助于开放词汇理解，为图像和视频模态提供锚定，并让大规模语言模型 [27, 109, 15] 的能力能为具身智能体所用。为确保模型发展对社会负责，我们采取专门措施过滤低质量和有害内容 [13, 51]，细节见附录（第 D 节）。

YouTube Videos and Transcripts. Minecraft is among the most streamed games on YouTube [41]. Human players have demonstrated a stunning range of creative activities and sophisticated missions that take hours to complete (examples in Fig. 3). We collect 730K+ narrated Minecraft videos, which add up to 33 years of duration and 2.2B words in English transcripts. In comparison, HowTo100M [77] is a large-scale human instructional video dataset that includes 15 years of experience in total - about half of our volume. The time-aligned transcripts enable the agent to ground free-form natural language in video pixels and learn the semantics of diverse activities without laborious human labeling. We operationalize this insight in our pre-trained video-language model (Sec. 4.1).

YouTube 视频与字幕。 Minecraft 是 YouTube 上被直播最多的游戏之一 [41]。人类玩家已经展示出令人惊叹的创造性活动和需要数小时才能完成的复杂任务（例子见图 3）。我们收集了超过 73 万条带旁白的 Minecraft 视频，总时长达 33 年，英文字幕共 22 亿词。相比之下，大规模人类教学视频数据集 HowTo100M [77] 的总经验大约是 15 年，约为我们体量的一半。时间对齐的字幕使智能体能够把自由格式自然语言锚定到视频像素上，并在不必费力人工标注的情况下学习多样活动的语义。我们在预训练的视频-语言模型中落实了这一想法（第 4.1 节）。

Minecraft Wiki. The Wiki pages cover almost every aspect of the game mechanics, and supply a rich source of unstructured knowledge in multimodal tables, recipes, illustrations, and step-by-step tutorials. We use Selenium [103] to scrape 6,735 pages that interleave text, images, tables, and diagrams. The pages are highly unstructured and do not share any common schema, as the Wiki is meant for human consumption rather than AI training. To preserve the layout information, we additionally save the screenshots of entire pages and extract 2.2M bounding boxes of the visual elements (visualization in Fig. A.4 and A.5). We do not use Wiki data in our current experiments. Since the Wiki contains detailed recipes for all crafted objects, they could be provided as input or training data for hierarchical planning methods and policy sketches [8]. Another promising future direction is to apply document understanding models such as LayoutLM [138, 137] and DocFormer [9] to learn actionable knowledge from these unstructured Wiki data.

Minecraft Wiki。 Wiki 页面几乎覆盖了游戏机制的每一个方面，并以多模态表格、配方、插图和逐步教程的形式提供丰富的非结构化知识。我们用 Selenium [103] 抓取了 6,735 个交错包含文本、图像、表格和示意图的页面。这些页面高度非结构化，没有共同模式，因为 Wiki 是给人看的，而不是为人工智能训练准备的。为了保留版面信息，我们额外保存了整页截图，并提取了 220 万个视觉元素的边界框（可视化见图 A.4 和 A.5）。当前实验没有使用 Wiki 数据。由于 Wiki 包含所有合成物品的详细配方，它们可以作为分层规划方法和策略草图 [8] 的输入或训练数据。另一个有前景的方向是应用 LayoutLM [138, 137] 和 DocFormer [9] 这类文档理解模型，从这些非结构化 Wiki 数据中学习可执行的知识。

Reddit. We scrape 340K+ posts along with 6.6M comments under the “r/Minecraft” subreddit. These posts ask questions on how to solve certain tasks, showcase cool architectures and achievements in image/video snippets, and discuss general tips and tricks for players of all expertise levels. We do not use Reddit data for training in Sec. 5, but a potential idea is to finetune large language models [27, 91] on our Reddit corpus to generate instructions and execution plans that are better grounded in the Minecraft domain. Concurrent works [3, 56, 143] have explored similar ideas and showed excellent results on robot learning, which is encouraging for more future research in MineDojo .

Reddit。 我们在 “r/Minecraft” 版块下抓取了超过 34 万条帖子以及 660 万条评论。这些帖子会询问如何完成某些任务，用图像/视频片段展示有趣的建筑和成就，并讨论面向各个水平玩家的一般技巧。第 5 节的训练没有使用 Reddit 数据，但一个可能的想法是在我们的 Reddit 语料上微调大规模语言模型 [27, 91]，以生成更好地锚定在 Minecraft 领域中的指令和执行计划。同期工作 [3, 56, 143] 探索了类似想法，并在机器人学习上展示了出色结果，这对 MineDojo 中的更多后续研究是一种鼓舞。

4 Agent Learning with Large-scale Pre-training

4 大规模预训练下的智能体学习

Figure 4: Algorithm design. MineCLIP is a contrastive video-language model pre-trained on MineDojo’s massive Youtube database. It computes the correlation between an open-vocabulary language goal string and a 16-frame video snippet. The correlation score can be used as a learned dense reward function to train a strong multi-task RL agent.

图 4：算法设计。MineCLIP 是在 MineDojo 大规模 YouTube 数据库上预训练的对比视频-语言模型。它计算开放词汇的语言目标字符串与 16 帧视频片段之间的相关性。该相关分数可以用作学到的稠密奖励函数，用来训练强大的多任务强化学习智能体。

One of the grand challenges of embodied AI is to build a single agent that can complete a wide range of open-world tasks. The MineDojo framework aims to facilitate new techniques towards this goal by providing an open-ended task suite (Sec. 2) and large-scale internet knowledge base (Sec. 3). Here we take an initial step towards this goal by developing a proof of concept that demonstrates how a single language-prompted agent can be trained in MineDojo to complete several complex Minecraft tasks. To this end, we propose a novel agent learning algorithm that takes advantage of the massive YouTube data offered by MineDojo. We note that this is only one of the numerous possible ways to use MineDojo’s internet database — the Wiki and Reddit corpus also hold great potential to drive new algorithm discoveries for the community in future works.

具身人工智能的一项重大挑战，是构建单个能够完成广泛开放世界任务的智能体。MineDojo 框架希望通过提供开放式任务套件（第 2 节）和大规模互联网知识库（第 3 节），推动迈向这一目标的新技术。这里我们先迈出一步，给出一个概念验证：展示如何在 MineDojo 中训练单个由语言提示的智能体，完成若干复杂的 Minecraft 任务。为此，我们提出一种新的智能体学习算法，利用 MineDojo 提供的海量 YouTube 数据。需要指出，这只是使用 MineDojo 互联网数据库的众多可能方式之一——Wiki 和 Reddit 语料同样有很大潜力，推动社区在未来发现新算法。

In this paper, we consider a multi-task reinforcement learning (RL) setting, where an agent is tasked with completing a collection of MineDojo tasks specified by language instructions (Sec. 2). Solving these tasks often requires the agent to interact with the Minecraft world in a prolonged fashion. Agents developed in popular RL benchmarks [119, 146] often rely on meticulously crafted dense and task-specific reward functions to guide random explorations. However, these rewards are hard or even infeasible to define for our diverse and open-ended tasks in MineDojo. To address this challenge, our key insight is to learn a dense, language-conditioned reward function from in-the-wild YouTube videos and their transcripts . Therefore, we introduce MineCLIP, a contrastive video-language model that learns to correlate video snippets and natural language descriptions (Fig. 4). MineCLIP is multi-task by design, as it is trained on open-vocabulary and diverse English transcripts.

本文考虑一种多任务强化学习设定：智能体需要完成一组由语言指令指定的 MineDojo 任务（第 2 节）。解决这些任务通常要求智能体与 Minecraft 世界进行长时间交互。在常见强化学习基准 [119, 146] 中发展出的智能体，往往依赖为每个任务精心设计的稠密、任务特定奖励函数来引导随机探索。可是，这些奖励对我们在 MineDojo 中多样而开放的任务来说很难定义，甚至无法定义。为应对这一挑战，我们的关键想法是从野外的 YouTube 视频及其字幕中学习 一个稠密的、以语言为条件的奖励函数 。于是我们引入 MineCLIP ，一个学习把视频片段与自然语言描述相对齐的对比视频-语言模型（图 4）。MineCLIP 在设计上就是多任务的，因为它在开放词汇、内容多样的英文字幕上训练。

During RL training, MineCLIP provides a high-quality reward signal without any domain adaptation techniques, despite the domain gap between noisy YouTube videos and clean simulator-rendered frames. MineCLIP eliminates the need to manually engineer reward functions for each and every MineDojo task. For Creative tasks that lack a simple success criterion (Sec. 2.2), MineCLIP also serves the dual purpose of an automatic evaluation metric that agrees well with human judgement on a subset of tasks we investigate (Sec. 4.2, Table 2). Because the learned reward model incurs a non-trivial computational overhead, we introduce several techniques to significantly improve RL training efficiency, making MineCLIP a practical module for open-ended agent learning in Minecraft (Sec. 4.2).

在强化学习训练期间，尽管嘈杂的 YouTube 视频和干净的仿真器渲染帧之间存在域差距，MineCLIP 仍能在 不做任何域适应 的情况下提供高质量奖励信号。MineCLIP 消除了为每一个 MineDojo 任务手工设计奖励函数的需要。对于缺少简单成功标准的创意任务（第 2.2 节），MineCLIP 还兼作 自动评估指标 ，在我们所考察的任务子集上与人类判断吻合良好（第 4.2 节，表 2）。由于学到的奖励模型会带来不可忽视的计算开销，我们引入若干技术来显著提高强化学习训练效率，使 MineCLIP 成为在 Minecraft 中进行开放式智能体学习的实用模块（第 4.2 节）。

6.5\pm 3.5

Tasks	Ours (Attn)	Ours (Avg)	Manual Reward	Sparse-only	CLIPOpenAI
Milk Cow	${\color[rgb]{0.09,0.45,0.27}\mathbf{64.5\pm 37.1}}$	$6.5\pm 3.5$	$62.8\pm 40.1$	$0.0\pm 0.0$	$0.0\pm 0.0$
Hunt Cow	${\color[rgb]{0.09,0.45,0.27}\mathbf{83.5\pm 7.1\hphantom{0}}}$	$0.0\pm 0.0$	$48.3\pm 35.9$	$0.3\pm 0.4$	$0.0\pm 0.0$
Shear Sheep	$12.1\pm 9.1\hphantom{0}$	$0.6\pm 0.2$	${\color[rgb]{0.09,0.45,0.27}\mathbf{52.3\pm 33.2}}$	$0.0\pm 0.0$	$0.0\pm 0.0$
Hunt Sheep	$8.1\pm 4.1$	$0.0\pm 0.0$	${\color[rgb]{0.09,0.45,0.27}\mathbf{41.9\pm 33.0}}$	$0.3\pm 0.4$	$0.0\pm 0.0$
Combat Spider	$80.5\pm 13.0$	$60.1\pm 42.5$	${\color[rgb]{0.09,0.45,0.27}\mathbf{87.5\pm 4.6\hphantom{0}}}$	$47.8\pm 33.8$	$0.0\pm 0.0$
Combat Zombie	$47.3\pm 10.6$	${\color[rgb]{0.09,0.45,0.27}\mathbf{72.3\pm 6.4\hphantom{0}}}$	$49.8\pm 26.9$	$\hphantom{0}8.8\pm 12.4$	$0.0\pm 0.0$
Combat Pigman	$1.6\pm 2.3$	$0.0\pm 0.0$	${\color[rgb]{0.09,0.45,0.27}\mathbf{13.6\pm 9.8\hphantom{0}}}$	$0.0\pm 0.0$	$0.0\pm 0.0$
Combat Enderman	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.3\pm 0.2$	$0.0\pm 0.0$	$0.0\pm 0.0$
Find Nether Portal	$37.4\pm 40.8$	${\color[rgb]{0.09,0.45,0.27}\mathbf{89.8\pm 5.7\hphantom{0}}}$	N/A	N/A	$26.3\pm 32.6$
Find Ocean	$33.4\pm 45.6$	${\color[rgb]{0.09,0.45,0.27}\mathbf{54.3\pm 40.7}}$	N/A	N/A	$\hphantom{0}9.9\pm 14.1$
Dig Hole	${\color[rgb]{0.09,0.45,0.27}\mathbf{91.6\pm 5.9\hphantom{0}}}$	$88.1\pm 13.3$	N/A	N/A	$0.0\pm 0.0$
Lay Carpet	$97.6\pm 1.9\hphantom{0}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{98.8\pm 1.0\hphantom{0}}}$	N/A	N/A	$0.0\pm 0.0$

6.5\pm 3.5

任务	本文（注意力）	本文（平均）	人工奖励	仅稀疏	CLIPOpenAI
挤牛奶	${\color[rgb]{0.09,0.45,0.27}\mathbf{64.5\pm 37.1}}$	$6.5\pm 3.5$	$62.8\pm 40.1$	$0.0\pm 0.0$	$0.0\pm 0.0$
猎牛	${\color[rgb]{0.09,0.45,0.27}\mathbf{83.5\pm 7.1\hphantom{0}}}$	$0.0\pm 0.0$	$48.3\pm 35.9$	$0.3\pm 0.4$	$0.0\pm 0.0$
剪羊毛	$12.1\pm 9.1\hphantom{0}$	$0.6\pm 0.2$	${\color[rgb]{0.09,0.45,0.27}\mathbf{52.3\pm 33.2}}$	$0.0\pm 0.0$	$0.0\pm 0.0$
猎羊	$8.1\pm 4.1$	$0.0\pm 0.0$	${\color[rgb]{0.09,0.45,0.27}\mathbf{41.9\pm 33.0}}$	$0.3\pm 0.4$	$0.0\pm 0.0$
对战蜘蛛	$80.5\pm 13.0$	$60.1\pm 42.5$	${\color[rgb]{0.09,0.45,0.27}\mathbf{87.5\pm 4.6\hphantom{0}}}$	$47.8\pm 33.8$	$0.0\pm 0.0$
对战僵尸	$47.3\pm 10.6$	${\color[rgb]{0.09,0.45,0.27}\mathbf{72.3\pm 6.4\hphantom{0}}}$	$49.8\pm 26.9$	$\hphantom{0}8.8\pm 12.4$	$0.0\pm 0.0$
对战猪灵	$1.6\pm 2.3$	$0.0\pm 0.0$	${\color[rgb]{0.09,0.45,0.27}\mathbf{13.6\pm 9.8\hphantom{0}}}$	$0.0\pm 0.0$	$0.0\pm 0.0$
对战末影人	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.3\pm 0.2$	$0.0\pm 0.0$	$0.0\pm 0.0$
寻找下界传送门	$37.4\pm 40.8$	${\color[rgb]{0.09,0.45,0.27}\mathbf{89.8\pm 5.7\hphantom{0}}}$	不适用	不适用	$26.3\pm 32.6$
寻找海洋	$33.4\pm 45.6$	${\color[rgb]{0.09,0.45,0.27}\mathbf{54.3\pm 40.7}}$	不适用	不适用	$\hphantom{0}9.9\pm 14.1$
挖洞	${\color[rgb]{0.09,0.45,0.27}\mathbf{91.6\pm 5.9\hphantom{0}}}$	$88.1\pm 13.3$	不适用	不适用	$0.0\pm 0.0$
铺地毯	$97.6\pm 1.9\hphantom{0}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{98.8\pm 1.0\hphantom{0}}}$	不适用	不适用	$0.0\pm 0.0$

4.1 Pre-Training MineCLIP on Large-scale Videos

4.1 在大规模视频上预训练 MineCLIP

\Phi_{\mathcal{R}}:(G,V)\rightarrow\mathbb{R}

\Phi_{\mathcal{R}}:(G,V)\rightarrow\mathbb{R}

\phi_{G}

\phi_{G}

\phi_{a}

\phi_{a}

4.2 RL with MineCLIP Reward

4.2 使用 MineCLIP 奖励做强化学习

We train a language-conditioned policy network that takes as input raw pixels and predicts discrete control. The policy is trained with PPO [102] on the MineCLIP rewards. In each episode, the agent is prompted with a language goal and takes a sequence of actions to fulfill this goal. When calculating the MineCLIP rewards, we concatenate the agent’s latest 16 egocentric RGB frames in a temporal window to form a video snippet. MineCLIP handles all task prompts zero-shot without any further finetuning. In our experiments (Sec. 5), we show that MineCLIP provides effective dense rewards out of the box, despite the domain shift between in-the-wild YouTube frames and simulator frames. Besides regular video data augmentation, we do not employ any special domain adaptation methods during pre-training. Our finding is consistent with CLIP’s strong zero-shot performances on robustness benchmarks in object recognition [92].

我们训练一个以语言为条件的策略网络，输入原始像素并预测离散控制。该策略用 PPO [102] 在 MineCLIP 奖励上训练。在每个回合中，智能体收到一个语言目标，并采取一系列动作来完成它。计算 MineCLIP 奖励时，我们把智能体最近 16 帧自我中心 RGB 画面在时间窗口中拼接成视频片段。MineCLIP 对所有任务提示都做 零样本 处理，不再额外微调。在实验中（第 5 节），我们表明 MineCLIP 开箱即用地提供了有效的稠密奖励，尽管野外 YouTube 帧和仿真器帧之间存在域偏移。除了常规的视频数据增强，我们在预训练期间没有采用任何特殊的域适应方法。这一发现与 CLIP 在物体识别稳健性基准上的强零样本表现一致 [92]。

Compared to hard-coded reward functions in popular benchmarks [146, 119, 34], the MineCLIP model has 150M parameters and is thus much more expensive to query. We make several design choices to greatly accelerate RL training with MineCLIP in the loop:

与常见基准 [146, 119, 34] 中硬编码的奖励函数相比，MineCLIP 模型有 1.5 亿参数，查询代价因此高得多。我们做了若干设计选择，以大幅加快循环调用 MineCLIP 的强化学习训练：

G

G

\phi_{I}

\phi_{I}

\phi_{V}

\phi_{V}

We leverage Self-Imitation Learning [84] to store the trajectories with high MineCLIP reward values in a buffer, and alternate between PPO and self-imitation gradient steps. It further improves sample efficiency as shown in the Appendix (Fig. A.8).

我们利用 自我模仿学习 [84]，把 MineCLIP 奖励值高的轨迹存进缓冲区，并在 PPO 与自我模仿梯度步之间交替。如附录所示（图 A.8），这进一步提升了样本效率。

98.7

Tasks	Find Nether Portal	Find Ocean	Dig Hole	Lay Carpet
Ours (Attn)	$98.7$	${\color[rgb]{0.09,0.45,0.27}\mathbf{100.0}}$	$99.4$	$97.4$
Ours (Avg)	${\color[rgb]{0.09,0.45,0.27}\mathbf{100.0}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{100.0}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{100.0}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{98.4}}$
CLIPOpenAI	$48.7$	$98.4$	$80.6$	$54.1$

98.7

任务	寻找下界传送门	寻找海洋	挖洞	铺地毯
本文（注意力）	$98.7$	${\color[rgb]{0.09,0.45,0.27}\mathbf{100.0}}$	$99.4$	$97.4$
本文（平均）	${\color[rgb]{0.09,0.45,0.27}\mathbf{100.0}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{100.0}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{100.0}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{98.4}}$
CLIPOpenAI	$48.7$	$98.4$	$80.6$	$54.1$

5 Experiments

5 实验

We evaluate our agent-learning approach (Section 4) on 8 Programmatic tasks and 4 Creative tasks from the MineDojo benchmarking suite. We select these 12 tasks due to the diversity of skills required to solve them (e.g., harvesting, combat, building, navigation) and domain-specific entities (e.g., animals, resources, monsters, terrains, and structures). We split the tasks into 3 groups and train one multi-task agent for each group: Animal-Zoo (4 Programmatic tasks on hunting or harvesting resource from animals), Mob-Combat (Programmatic, fight 4 types of hostile monsters), and Creative (4 tasks).

我们在 MineDojo 基准套件中的 8 个程序化任务和 4 个创意任务上评估智能体学习方法（第 4 节）。选择这 12 个任务，是因为完成它们所需的技能多样（例如采集、战斗、建造、导航），也因为它们涉及不同的领域实体（例如动物、资源、怪物、地形和结构）。我们把任务分成 3 组，并为每组训练一个多任务智能体：Animal-Zoo（4 个与猎取或从动物身上采集资源有关的程序化任务）、Mob-Combat（程序化任务，与 4 类敌对怪物战斗）以及 Creative（4 个任务）。

In the experiments, we empirically check the quality of MineCLIP against manually written reward functions, and quantify how different variants of our learned model affect the RL performance. Table 1 presents our main results, and Fig. 2 visualizes our learned agent behavior in 4 of the considered tasks. Policy networks of all methods share the same architecture and are trained by PPO + Self-Imitation (Sec. 4.2, training details in the Appendix, Sec. F). We compare the following methods:

在实验中，我们经验地检查 MineCLIP 相对于人工书写奖励函数的质量，并量化学到模型的不同变体如何影响强化学习表现。表 1 给出主要结果，图 2 可视化了我们的智能体在其中 4 个任务上学会的行为。所有方法的策略网络共享同一结构，并用 PPO + 自我模仿训练（第 4.2 节，训练细节见附录第 F 节）。我们比较以下方法：

• Ours (Attn) : our agent trained with the MineCLIP[attn] reward model. For Programmatic tasks, we also add the final success condition as a binary reward. For Creative tasks, MineCLIP is the only source of reward.

• 本文（注意力） ：用 MineCLIP[attn] 奖励模型训练的智能体。对程序化任务，我们还把最终成功条件加为二元奖励。对创意任务，MineCLIP 是唯一的奖励来源。

• Ours (Avg) : the average-pooling variant of our method.

• 本文（平均） ：我们方法的平均池化变体。

• Manual Reward : hand-engineered dense reward using ground-truth simulator states.

• 人工奖励 ：使用仿真器真实状态手工设计的稠密奖励。

• Sparse-only : the final binary success as a single sparse reward. Note that neither sparse-only nor manual reward is available for Creative tasks.

• 仅稀疏 ：把最终的二元成功当作单一稀疏奖励。注意，仅稀疏和人工奖励对创意任务都不可用。

• CLIPOpenAI : pre-trained OpenAI CLIP model that has not been finetuned on any MineDojo videos.

• CLIPOpenAI ：预训练的 OpenAI CLIP 模型， 没有 在任何 MineDojo 视频上微调。

0.3991\gg 0.05

0.3991\gg 0.05

64.5\pm 37.1

Tasks	Ours (Attn), train	Ours (Attn), unseen test	CLIPOpenAI, train	CLIPOpenAI, unseen test
Milk Cow	$64.5\pm 37.1$	${\color[rgb]{0.09,0.45,0.27}\mathbf{64.8\pm 31.3(+\hphantom{0}0.8\%)}}$	$90.0\pm 0.4$	$29.2\pm 3.7\hphantom{0}(-67.6\%)$
Hunt Cow	$83.5\pm 7.1\hphantom{0}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{55.9\pm 7.2\hphantom{0}(-32.9\%)}}$	$72.7\pm 3.5$	$16.7\pm 1.6\hphantom{0}(-77.0\%)$
Combat Spider	$80.5\pm 13.0$	${\color[rgb]{0.09,0.45,0.27}\mathbf{62.1\pm 29.7(-22.9\%)}}$	$79.5\pm 2.5$	$54.2\pm 9.6\hphantom{0}(-31.8\%)$
Combat Zombie	$47.3\pm 10.6$	${\color[rgb]{0.09,0.45,0.27}\mathbf{39.9\pm 25.3(-15.4\%)}}$	$50.2\pm 7.5$	$30.8\pm 14.4(-38.6\%)$

64.5\pm 37.1

任务	本文（注意力），训练	本文（注意力），未见测试	CLIPOpenAI，训练	CLIPOpenAI，未见测试
挤牛奶	$64.5\pm 37.1$	${\color[rgb]{0.09,0.45,0.27}\mathbf{64.8\pm 31.3(+\hphantom{0}0.8\%)}}$	$90.0\pm 0.4$	$29.2\pm 3.7\hphantom{0}(-67.6\%)$
猎牛	$83.5\pm 7.1\hphantom{0}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{55.9\pm 7.2\hphantom{0}(-32.9\%)}}$	$72.7\pm 3.5$	$16.7\pm 1.6\hphantom{0}(-77.0\%)$
对战蜘蛛	$80.5\pm 13.0$	${\color[rgb]{0.09,0.45,0.27}\mathbf{62.1\pm 29.7(-22.9\%)}}$	$79.5\pm 2.5$	$54.2\pm 9.6\hphantom{0}(-31.8\%)$
对战僵尸	$47.3\pm 10.6$	${\color[rgb]{0.09,0.45,0.27}\mathbf{39.9\pm 25.3(-15.4\%)}}$	$50.2\pm 7.5$	$30.8\pm 14.4(-38.6\%)$

MineCLIP provides automated evaluation. For Creative tasks (last 4 rows), there are no programmatic success criteria available. We convert MineCLIP into a binary success classifier by thresholding the reward value it outputs for an episode. To test the quality of MineCLIP as an automatic evaluation metric, we ask human judges to curate a dataset of 100 successful and 100 failed trajectories for each task. We then run both MineCLIP variants and CLIPOpenAI on the dataset and report the binary F1 score of their judgement against human ground-truth in Table 2. The results demonstrate that both MineCLIP[attn] and MineCLIP[avg] attain a very high degree of agreement with human evaluation results on this subset of the Creative task suite that we investigate. CLIPOpenAI baseline also achieves nontrivial agreement on Find Ocean and Dig Hole tasks, likely because real-world oceans and holes have similar texture. We use the attn variant as an automated success criterion to score the 4 Creative task results in Table 1. Our proposed method consistently learns better than CLIPOpenAI-guided agents. It shows that MineCLIP is an effective approach to solving open-ended tasks when no straightforward reward signal is available. We provide further analysis beyond these 4 tasks in the Appendix (Sec. G.4).

MineCLIP 提供自动评估。 对创意任务（后 4 行），没有可用的程序化成功标准。我们通过对一个回合输出的奖励值设阈值，把 MineCLIP 转换成二元成功分类器。为检验 MineCLIP 作为自动评估指标的质量，我们请人类评判者为每个任务整理出 100 条成功轨迹和 100 条失败轨迹。随后在该数据集上运行两种 MineCLIP 变体以及 CLIPOpenAI，并在表 2 中报告它们的判断相对于人类真实标签的二元 F1 分数。结果表明，在我们所考察的这部分创意任务套件上，MineCLIP[attn] 和 MineCLIP[avg] 都与人工评估达到很高的一致性。CLIPOpenAI 基线在“寻找海洋”和“挖洞”上也取得了不低的一致性，这很可能是因为现实中的海洋和坑洞有类似纹理。我们用注意力变体作为自动成功标准，为表 1 中 4 个创意任务的结果打分。我们所提出的方法始终比由 CLIPOpenAI 引导的智能体学得更好。这说明，当没有直接奖励信号可用时，MineCLIP 是解决开放式任务的有效途径。我们在附录（第 G.4 节）中提供了超出这 4 个任务的进一步分析。

MineCLIP shows good zero-shot generalization to significant visual distribution shift. We evaluate the learned policy without finetuning on a combination of unseen weather, lighting conditions, and terrains — 27 scenarios in total. For the baseline, we train agents with the original CLIPOpenAI image encoder (not trained on our YouTube videos) by imitation learning. The robustness against visual shift can be quantitatively measured by the relative performance degradation on novel test scenarios for each task. Table 3 shows that while all methods incur performance drops, agents with MineCLIP encoder is more robust to visual changes than the baseline across all tasks. Pre-training on diverse in-the-wild YouTube videos is important to boosting zero-shot visual generalization capability, a finding consistent with literature [92, 82].

MineCLIP 对显著的视觉分布偏移表现出良好的零样本泛化。 我们在不微调的情况下，于未见天气、光照条件和地形的组合上评估学到的策略——一共 27 种情形。作为基线，我们用原始 CLIPOpenAI 图像编码器（未在我们的 YouTube 视频上训练）通过模仿学习训练智能体。对视觉偏移的稳健性，可以用每个任务在新颖测试情形上的相对性能下降来定量衡量。表 3 表明，尽管所有方法都会下降，带 MineCLIP 编码器的智能体在全部任务上都比基线更能抵抗视觉变化。在多样的野外 YouTube 视频上预训练，对提升零样本视觉泛化能力很重要，这一发现与文献一致 [92, 82]。

Learning a Single Agent for All 12 Tasks We have also trained a single agent for all 12 tasks. To reduce the computational burden without loss of generality, the agent is trained by self-imitating from successful trajectories generated from the self-imitation learning pipeline (Section F.3). We summarize the result in Table 4. Similar to our main experiments, all numbers represent percentage success rates averaged over 3 training seeds, each tested for 200 episodes. Compared to the original agents, the new 12-multitask agent sees a performance boost in 6 tasks, degradation in 4 tasks, and roughly the same success rates in 2 tasks. This result suggests that there are both positive and negative task transfers happening. To improve the multi-task performance, more advanced algorithms [141, 133] can be employed to mitigate the negative transfer effects.

为全部 12 个任务学习单个智能体 我们也为全部 12 个任务训练了单个智能体。为在不失一般性的前提下减轻计算负担，该智能体通过自我模仿来学习，模仿对象是自我模仿学习流水线生成的成功轨迹（第 F.3 节）。结果汇总在表 4。与主要实验相同，所有数字都是 3 个训练种子的平均成功率（百分比），每个种子测试 200 个回合。与原先的智能体相比，这个新的 12 任务智能体在 6 个任务上有提升，在 4 个任务上有下降，在 2 个任务上成功率大致相同。这一结果表明，既有正迁移也有负迁移。为提升多任务表现，可以采用更先进的算法 [141, 133] 来减轻负迁移。

0.3720\gg 0.05

0.3720\gg 0.05

64.5\pm 37.1

Tasks	Single Agent on All Tasks	Original	Performance Change
Milk Cow	${\color[rgb]{0.09,0.45,0.27}\mathbf{91.5\pm 0.7}}$	$64.5\pm 37.1$	$\uparrow$
Hunt Cow	$46.8\pm 3.7$	${\color[rgb]{0.09,0.45,0.27}\mathbf{83.5\pm 7.1\hphantom{0}}}$	$\downarrow$
Shear Sheep	${\color[rgb]{0.09,0.45,0.27}\mathbf{73.5\pm 0.8}}$	$12.1\pm 9.1\hphantom{0}$	$\uparrow$
Hunt Sheep	${\color[rgb]{0.09,0.45,0.27}\mathbf{27.0\pm 1.0}}$	$8.1\pm 4.1$	$\uparrow$
Combat Spider	$72.1\pm 1.3$	${\color[rgb]{0.09,0.45,0.27}\mathbf{80.5\pm 13.0}}$	$\downarrow$
Combat Zombie	$27.1\pm 2.7$	${\color[rgb]{0.09,0.45,0.27}\mathbf{47.3\pm 10.6}}$	$\downarrow$
Combat Pigman	${\color[rgb]{0.09,0.45,0.27}\mathbf{6.5\pm 1.2}}$	$1.6\pm 2.3$	$\uparrow$
Combat Enderman	$0.0\pm 0.0$	$0.0\pm 0.0$	$=$
Find Nether Portal	${\color[rgb]{0.09,0.45,0.27}\mathbf{99.1\pm 0.4}}$	$37.4\pm 40.8$	$\uparrow$
Find Ocean	${\color[rgb]{0.09,0.45,0.27}\mathbf{95.1\pm 1.5}}$	$33.4\pm 45.6$	$\uparrow$
Dig Hole	$85.8\pm 1.2$	${\color[rgb]{0.09,0.45,0.27}\mathbf{91.6\pm 5.9\hphantom{0}}}$	$\downarrow$
Lay Carpet	$96.5\pm 0.8$	${\color[rgb]{0.09,0.45,0.27}\mathbf{97.6\pm 1.9}}\hphantom{0}$	$=$

64.5\pm 37.1

任务	全部任务上的单智能体	原先	性能变化
挤牛奶	${\color[rgb]{0.09,0.45,0.27}\mathbf{91.5\pm 0.7}}$	$64.5\pm 37.1$	$\uparrow$
猎牛	$46.8\pm 3.7$	${\color[rgb]{0.09,0.45,0.27}\mathbf{83.5\pm 7.1\hphantom{0}}}$	$\downarrow$
剪羊毛	${\color[rgb]{0.09,0.45,0.27}\mathbf{73.5\pm 0.8}}$	$12.1\pm 9.1\hphantom{0}$	$\uparrow$
猎羊	${\color[rgb]{0.09,0.45,0.27}\mathbf{27.0\pm 1.0}}$	$8.1\pm 4.1$	$\uparrow$
对战蜘蛛	$72.1\pm 1.3$	${\color[rgb]{0.09,0.45,0.27}\mathbf{80.5\pm 13.0}}$	$\downarrow$
对战僵尸	$27.1\pm 2.7$	${\color[rgb]{0.09,0.45,0.27}\mathbf{47.3\pm 10.6}}$	$\downarrow$
对战猪灵	${\color[rgb]{0.09,0.45,0.27}\mathbf{6.5\pm 1.2}}$	$1.6\pm 2.3$	$\uparrow$
对战末影人	$0.0\pm 0.0$	$0.0\pm 0.0$	$=$
寻找下界传送门	${\color[rgb]{0.09,0.45,0.27}\mathbf{99.1\pm 0.4}}$	$37.4\pm 40.8$	$\uparrow$
寻找海洋	${\color[rgb]{0.09,0.45,0.27}\mathbf{95.1\pm 1.5}}$	$33.4\pm 45.6$	$\uparrow$
挖洞	$85.8\pm 1.2$	${\color[rgb]{0.09,0.45,0.27}\mathbf{91.6\pm 5.9\hphantom{0}}}$	$\downarrow$
铺地毯	$96.5\pm 0.8$	${\color[rgb]{0.09,0.45,0.27}\mathbf{97.6\pm 1.9}}\hphantom{0}$	$=$

1.3\pm 0.6

	Tasks	Ours (zero-shot)	Ours (after RL finetune)	Baseline (RL from scratch)
	Hunt Pig	$1.3\pm 0.6$	${\color[rgb]{0.09,0.45,0.27}\mathbf{46.0\pm 15.3}}$	$0.0\pm 0.0$
	Harvest Spider String	$1.6\pm 1.3$	${\color[rgb]{0.09,0.45,0.27}\mathbf{36.5\pm 16.9}}$	$12.5\pm 12.7$

1.3\pm 0.6

	任务	本文（零样本）	本文（强化学习微调后）	基线（从零开始的强化学习）
	猎猪	$1.3\pm 0.6$	${\color[rgb]{0.09,0.45,0.27}\mathbf{46.0\pm 15.3}}$	$0.0\pm 0.0$
	采集蜘蛛丝	$1.6\pm 1.3$	${\color[rgb]{0.09,0.45,0.27}\mathbf{36.5\pm 16.9}}$	$12.5\pm 12.7$

Generalize to Novel Tasks To test the ability to generalize to new open-vocabulary commands, we evaluate the agent on two novel tasks: “harvest spider string” and “hunt pig”. Table 5 shows that the agent struggles in the zero-shot setting because it has not interacted with pigs or spider strings during training, and thus does not know how to interact with them effectively. However, the performance improves substantially by finetuning with the MineCLIP reward. Here the baseline methods are trained from scratch using RL with the MineCLIP encoders and reward. Therefore, the only difference is whether the policy has been pre-trained on the 12 tasks or not. Given the same environment sampling budget (only around 5% of total samples), it significantly outperforms baselines. It suggests that the multitask agent has learned transferable knowledge on hunting and resource collection, which enables it to quickly adapt to novel tasks.

泛化到新任务 为检验对新型开放词汇指令的泛化能力，我们在两个新任务上评估智能体：“采集蜘蛛丝”和“猎猪”。表 5 表明，智能体在零样本设定下表现吃力，因为它在训练中没有与猪或蜘蛛丝交互过，因而不知道如何有效地与它们互动。不过，用 MineCLIP 奖励微调后，表现有明显提升。这里的基线方法是用带 MineCLIP 编码器和奖励的强化学习从零训练。因此，唯一差别在于策略是否已在这 12 个任务上预训练。在相同的环境采样预算下（大约只占总样本的 5%），它显著超过基线。这表明多任务智能体已经学到关于狩猎和资源收集的可迁移知识，从而能够迅速适应新任务。

6 Related work

6 相关工作

Open-ended Environments for Decision-making Agents. There are many environments developed with the goal of open-ended agent learning. Prior works include maze-style worlds [121, 129, 61], purely text-based game [69], grid worlds [21, 16], browser/GUI-based environments [108, 124], and indoor simulators for robotics [1, 107, 114, 34, 110, 99, 89]. Minecraft offers an exciting alternative for open-ended agent learning. It is a 3D visual world with procedurally generated landscapes and extremely flexible game mechanics that support an enormous variety of activities. Prior methods in open-ended agent learning [30, 57, 130, 63, 26] do not make use of external knowledge, but our approach leverages internet-scale database to learn open-vocabulary reward models, thanks to Minecraft’s abundance of gameplay data online.

面向决策智能体的开放式环境。 已经有许多环境以开放式智能体学习为目标而发展出来。先前工作包括迷宫风格世界 [121, 129, 61]、纯文本游戏 [69]、格子世界 [21, 16]、基于浏览器/图形界面的环境 [108, 124]，以及面向机器人的室内仿真器 [1, 107, 114, 34, 110, 99, 89]。Minecraft 为开放式智能体学习提供了一个令人兴奋的替代选择。它是一个三维视觉世界，景观由程序化生成，游戏机制又极为灵活，能够支持极其多样的活动。先前的开放式智能体学习方法 [30, 57, 130, 63, 26] 没有利用外部知识，而我们的方法借助互联网规模数据库来学习开放词汇奖励模型，这得益于 Minecraft 在网上拥有大量游戏数据。

Minecraft for AI Research. The Malmo platform [60] is the first comprehensive release of a Gym-style agent API [14] for Minecraft. Based on Malmo, MineRL [48] provides a codebase and human play trajectories for the annual Diamond Challenge at NeurIPS [47, 49, 62]. MineDojo’s simulator builds upon the pioneering work of MineRL, but greatly expands the API and benchmarking task suite. Other Minecraft benchmarks exist with different focuses. For example, CraftAssist [44] and IGLU [66] study agents with interactive dialogues. BASALT [104] applies human evaluation to 4 open-ended tasks. EvoCraft [45] is designed for structure building, and Crafter [50] optimizes for fast experimentation. Unlike prior works, MineDojo’s core mission is to facilitate the development of generally capable embodied agents using internet-scale knowledge. We include a feature comparison table of different Minecraft platforms for AI research in Table A.1.

面向人工智能研究的 Minecraft。 Malmo 平台 [60] 是第一个全面发布的、面向 Minecraft 的 Gym 风格智能体 API [14]。基于 Malmo，MineRL [48] 为 NeurIPS 上一年一度的 Diamond Challenge [47, 49, 62] 提供了代码库和人类游玩轨迹。MineDojo 的仿真器建立在 MineRL 的开创性工作之上，但大幅扩展了 API 和基准任务套件。其他 Minecraft 基准各有侧重。例如，CraftAssist [44] 和 IGLU [66] 研究带交互式对话的智能体。BASALT [104] 对 4 个开放式任务采用人工评估。EvoCraft [45] 面向结构建造，Crafter [50] 则针对快速实验做了优化。与先前工作不同，MineDojo 的核心使命是利用互联网规模知识，推动通用具身智能体的发展。我们在表 A.1 中给出了不同 Minecraft 人工智能研究平台的特性对照表。

Internet-scale Multimodal Knowledge Bases. Big dataset such as Common Crawl [24], the Pile [37], LAION [100], YouTube-8M [2] and HowTo100M [77] have been fueling the success of large pre-trained language models [27, 91, 15] and multimodal models [118, 6, 78, 145, 7, 4, 136]. While generally useful for learning representations, these datasets are not specifically targeted at embodied agents. To provide agent-centric training data, RoboNet [25] collects video frames from 7 robot platforms, and Ego4D [43] recruits volunteers to record egocentric videos of household activities. In comparison, MineDojo’s knowledge base is constructed without human curation efforts, much larger in volume, more diverse in data modalities, and comprehensively covers all aspects of the Minecraft environment.

互联网规模的多模态知识库。 Common Crawl [24]、the Pile [37]、LAION [100]、YouTube-8M [2] 和 HowTo100M [77] 这类大型数据集，一直在推动大规模预训练语言模型 [27, 91, 15] 和多模态模型 [118, 6, 78, 145, 7, 4, 136] 的成功。这些数据集对学习表征通常很有用，但并不是专门面向具身智能体。为了提供以智能体为中心的训练数据，RoboNet [25] 从 7 个机器人平台收集视频帧，Ego4D [43] 则招募志愿者记录家务活动的自我中心视频。相比之下，MineDojo 的知识库无需人工整理，体量更大，数据模态更多样，并且全面覆盖 Minecraft 环境的各个方面。

带大规模预训练的具身智能体。 受自然语言处理成功的启发，具身智能体研究 [29, 11, 94, 23] 中采用大规模预训练范式的做法明显增多。近期进展大致可分为 4 类。1） 新的智能体结构 ：Decision Transformer [19, 58, 144] 把强大的自注意力模型应用到序列决策。GATO [95] 和 Unified-IO [74] 学习单个模型，以不同的控制接口解决各种决策任务。VIMA [59] 用多模态提示统一了广泛的机器人操作任务。2） 预训练以获得更好表征 ：R3M [82] 在 Ego4D 视频 [43] 上为机器人感知训练通用视觉编码器。CLIPort [111] 利用预训练 CLIP 模型 [92]，使机器人操作能够接受自由格式语言指令。3） 预训练以获得更好策略 ：AlphaStar [126] 通过模仿大量人类演示，在《星际争霸》上达到冠军水平。SayCan [3] 利用大规模语言模型把价值函数锚定到物理世界中。[72] 和 [96] 则直接把预训练语言模型复用为策略主干。VPT [10] 是一项同期工作，它从人类承包者那里学习逆动力学模型，为 YouTube 视频打伪标签再做行为克隆。VPT 与我们的方法互补，可以用我们学到的奖励模型微调，以解决以语言为条件的开放式任务。4） 数据驱动的奖励函数 ：Concept2Robot [105] 和 DVD [18] 学习一个二元分类器，从野外视频 [42] 为行为打分。LOReL [81] 众包人类标签，为离线强化学习训练以语言为条件的奖励函数。AVID [113] 和 XIRL [142] 通过循环一致性提取奖励信号。MineDojo 的任务基准和互联网知识库，对上述各类新算法的发展通常都有帮助。在第 4 节中，我们也利用 MineDojo 的 YouTube 视频提出了一种开放词汇、多任务的奖励模型。

7 Conclusion

7 结论

In this work, we introduce the MineDojo framework for developing generally capable embodied agents. MineDojo features a benchmarking suite of thousands of Programmatic and Creative tasks, and an internet-scale multimodal knowledge base of videos, wiki, and forum discussions. As an example of the novel research possibilities enabled by MineDojo, we propose MineCLIP as an effective language-conditioned reward function trained with in-the-wild YouTube videos. MineCLIP achieves strong performance empirically and agrees well with human evaluation results, making it a good automatic metric for Creative tasks. We look forward to seeing how MineDojo empowers the community to make progress on the important challenge of open-ended agent learning.

在这项工作中，我们引入 MineDojo 框架，用于发展具备通用能力的具身智能体。MineDojo 的特色是一个包含数千个程序化任务和创意任务的基准套件，以及由视频、Wiki 和论坛讨论组成的互联网规模多模态知识库。作为 MineDojo 所开启的新型研究可能的一个例子，我们提出 MineCLIP，它是用野外 YouTube 视频训练出的有效的、以语言为条件的奖励函数。MineCLIP 在经验上取得了很强的表现，并与人工评估结果吻合良好，因而成为创意任务的一个不错的自动指标。我们期待看到 MineDojo 如何帮助社区在开放式智能体学习这一重要挑战上取得进展。

8 Acknowledgement

8 致谢

We are extremely grateful to Anssi Kanervisto, Shikun Liu, Zhiding Yu, Chaowei Xiao, Weili Nie, Jean Kossaifi, Jonathan Raiman, Neel Kant, Saad Godil, Jaakko Haapasalo, Bryan Catanzaro, John Spitzer, Zhiyuan “Jerry” Lin, Yingqi Zheng, Chen Tessler, Dieter Fox, Oli Wright, Jeff Clune, Jack Parker-Holder, and many other colleagues and friends for their helpful feedback and insightful discussions. We also thank the anonymous reviewers for offering us highly constructive advice and kind encouragement during the review and rebuttal period. NVIDIA provides the necessary computing resource and infrastructure for this project. Guanzhi Wang is supported by the Kortschak fellowship in Computing and Mathematical Sciences at Caltech.

我们非常感谢 Anssi Kanervisto、Shikun Liu、Zhiding Yu、Chaowei Xiao、Weili Nie、Jean Kossaifi、Jonathan Raiman、Neel Kant、Saad Godil、Jaakko Haapasalo、Bryan Catanzaro、John Spitzer、Zhiyuan “Jerry” Lin、Yingqi Zheng、Chen Tessler、Dieter Fox、Oli Wright、Jeff Clune、Jack Parker-Holder，以及许多其他同事和朋友提供的有益反馈和富有洞察的讨论。我们也感谢匿名审稿人在审稿与回应期间给予的建设性意见和鼓励。NVIDIA 为这个项目提供了必要的计算资源和基础设施。Guanzhi Wang 受到加州理工学院计算与数学科学 Kortschak fellowship 的资助。

References

参考文献

Yu et al. [2020] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv: Arxiv-2001.06782 , 2020. Zakka et al. [2021] Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. arXiv preprint arXiv: Arxiv-2106.03911 , 2021. Zeng et al. [2022] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv: Arxiv-2204.00598 , 2022. Zheng et al. [2022] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint arXiv: Arxiv-2202.05607 , 2022. Zhu and Yang [2020] Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. arXiv preprint arXiv: Arxiv-2011.07231 , 2020. Zhu et al. [2020] Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martín. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv: Arxiv-2009.12293 , 2020.

Appendix A Minecraft Framework Comparison

附录 A Minecraft 框架比较

3,000+

Environment	Simulator	Task Suite	Knowledge Base
Features	Real Minecraft	Number of Tasks	Language- grounded	Features	Data Scale
MineDojo	Unified observation and action space; unlocks all three types of world (the Overworld, the Nether, and the End)	✓	$3,000+$	✓	Automatically scraped from the Internet; multimodal data (videos, images, texts, tables and diagrams)	740K YouTube videos; 7K Wiki pages; 350K Reddit posts
MineRL v0.4 [48]	Built on top of Malmo; actively maintained	✓	11		Annotated state-action pairs of human demonstrations	60M frames of recorded human player data
MineRL v1.0 (VPT) [10]	Mouse and keyboard control	✓	5		Labeled contractor data; unlabeled videos scraped from the Internet	2K hours of contractor data; 270K hours of unlabeled videos
MarLÖ [87]	Cooperative and competitive multiagent tasks; parameterizable environments	✓	14
Malmo [60]	First comprehensive release of a Gym-style agent API for Minecraft	✓	N/A
CraftAssist [44]	Bot assistant; dialogue interactions	✓	N/A	✓	Interactive dialogues; crowd-sourced house building dataset	800K dialogue-action dictionary pairs; 2.6K houses with atomic building actions
IGLU [66]	Interactive dialogues with humans; aimed at building structures described by natural language		157	✓
EvoCraft [45]	Aimed at generating creative artifacts; allows for direction manipulation of blocks		N/A
Crafter [50]	2D clone of Minecraft; fast experimentation		22		Human experts dataset	100 episodes

3,000+

环境	仿真器	任务套件	知识库
特性	真实 Minecraft	任务数量	语言锚定	特性	数据规模
MineDojo	统一的观察和动作空间；解锁全部三类世界（主世界、下界和末地）	✓	$3,000+$	✓	从互联网自动抓取；多模态数据（视频、图像、文本、表格和示意图）	74 万条 YouTube 视频；7 千篇 Wiki 页面；35 万条 Reddit 帖子
MineRL v0.4 [48]	建立在 Malmo 之上；持续维护	✓	11		带标注的人类演示状态-动作对	6,000 万帧录制的人类玩家数据
MineRL v1.0 (VPT) [10]	鼠标和键盘控制	✓	5		带标签的承包者数据；从互联网抓取的未标注视频	2 千小时承包者数据；27 万小时未标注视频
MarLÖ [87]	合作与竞争的多智能体任务；可参数化环境	✓	14
Malmo [60]	第一个全面发布的、面向 Minecraft 的 Gym 风格智能体 API	✓	不适用
CraftAssist [44]	机器人助手；对话交互	✓	不适用	✓	交互式对话；众包房屋建造数据集	80 万组对话-动作词典对；2,600 座带原子建造动作的房屋
IGLU [66]	与人类的交互式对话；目标是建造自然语言描述的结构		157	✓
EvoCraft [45]	目标是生成有创造性的制品；允许直接操作方块		不适用
Crafter [50]	Minecraft 的二维克隆；便于快速实验		22		人类专家数据集	100 个回合

Appendix B MineDojo Simulator

附录 B MineDojo 仿真器

We design unified observation and action spaces across all tasks to facilitate the development of multi-tasking and continually learning agents that can adapt to novel tasks and scenarios. The codebase is open sourced at github.com/MineDojo/MineDojo.

我们在所有任务上设计统一的观察和动作空间，以便发展能够适应新任务和新场景的多任务、持续学习智能体。代码库开源在 github.com/MineDojo/MineDojo。

B.1 Observation Space

B.1 观察空间

Our observation space contains multiple modalities. The agent perceives the world mainly through the RGB screen. To provide the same information as human players receive, we also supplement the agent with observations about its inventory, location, health, surrounding blocks, etc. The full observation space is shown below. We refer readers to see our code documentation for technical details such as data type for each observable item.

我们的观察空间包含多种模态。智能体主要通过 RGB 画面感知世界。为了提供与人类玩家相同的信息，我们还补充关于背包、位置、生命值、周围方块等的观察。完整观察空间如下所示。关于每个可观察项的数据类型等技术细节，请参阅我们的代码文档。

We also support a LIDAR sensor that returns the groundtruth type of the blocks that the agent sees, however this is considered privileged information and does not go into the benchmarking specification. However, it is still useful for hand engineering the dense reward function, which we use in our experiments (Sec. 5). Amounts and directions of LIDAR rays can be arbitrarily configured at the cost of a lower simulation throughput.

我们还支持一种激光雷达传感器，它返回智能体所看到方块的真实类型；不过这被视为特权信息，不进入基准规格。尽管如此，它对于手工设计稠密奖励函数仍然有用，我们也在实验中使用了它（第 5 节）。激光雷达射线的数量和方向可以任意配置，代价是仿真吞吐会降低。

B.2 Action Space

B.2 动作空间

We design a compound action space. At each step the agent chooses one movement action (forward, backward, camera actions, etc.) and one optional functional action as listed in the table below. Some functional actions such as craft take one argument, while others like attack does not take any argument. This compound action space can be modelled in an autoregressive manner [126]. We refer readers to our code documentation for example usages of our action space.

我们设计了一种复合动作空间。每一步，智能体选择一个移动动作（前进、后退、相机动作等）以及下表所列的一个可选功能动作。有些功能动作（例如 craft）接受一个参数，另一些（例如 attack）则不接受任何参数。这种复合动作空间可以按自回归方式建模 [126]。关于动作空间的用法示例，请参阅我们的代码文档。

B.3 Customizing the Environment

B.3 定制环境

Environments in MineCLIP simulator can be easily and flexibly customized. Through our simulator API, users can control terrain, weather, day-night condition (different lighting), the spawn rate and range of specified entities and materials, etc. We support a wide range of terrains, such as desert, jungle, taiga, and iced plain, and special in-game structures, such as ocean monument, desert temple, and End city. Please visit our website for video demonstrations.

MineCLIP 仿真器中的环境可以方便而灵活地定制。通过我们的仿真器 API，用户可以控制地形、天气、昼夜条件（不同光照），以及指定实体和材料的生成率和范围等。我们支持广泛的地形，例如沙漠、丛林、针叶林和冰原，以及游戏中的特殊结构，例如海底神殿、沙漠神殿和末地城。视频演示请访问我们的网站。

Appendix C MineDojo Task Suite

附录 C MineDojo 任务套件

In this section, we explain how we collect the Programmatic (Sec. 2.1) and Creative tasks (Sec. 2.2).

在这一节中，我们说明如何收集程序化任务（第 2.1 节）和创意任务（第 2.2 节）。

1 survival_sword_food: 2 category: survival 3 prompt: survive as long as possible given a sword and some food 4 5 harvest_wool_with_shears_and_sheep: 6 category: harvest 7 prompt: harvest wool from a sheep with shears and a sheep nearby 8 9 techtree_from_barehand_to_wooden_sword: 10 category: tech-tree 11 prompt: find material and craft a wooden sword 12 13 combat_zombie_pigman_nether_diamond_armors_diamond_sword_shield: 14 category: combat 15 prompt: combat a zombie pigman in nether with a diamond sword, 16 shield, and a full suite of diamond armors Figure A.1: Example specifications.

1 survival_sword_food: 2 category: survival 3 prompt: 带着一把剑和一些食物尽可能长时间地存活 4 5 harvest_wool_with_shears_and_sheep: 6 category: harvest 7 prompt: 用剪刀从附近的羊身上采集羊毛 8 9 techtree_from_barehand_to_wooden_sword: 10 category: tech-tree 11 prompt: 寻找材料并合成一把木剑 12 13 combat_zombie_pigman_nether_diamond_armors_diamond_sword_shield: 14 category: combat 15 prompt: 在下界用钻石剑、盾牌和全套钻石盔甲 16 与僵尸猪灵战斗 图 A.1：规格示例。

C.1 Programmatic Tasks

C.1 程序化任务

Programmatic tasks are constructed by filling manually written templates for four categories of tasks, namely “Survival”, “Harvest”, “Tech Tree”, and “Combat”. The task specifications are included in our codebase. Please refer to Fig. A.1 for a few samples. We briefly explain each task category:

程序化任务通过填写手工编写的模板来构造，覆盖四类任务，即“生存”“采集”“科技树”和“战斗”。任务规格包含在我们的代码库中。若干样例请见图 A.1。我们简要说明每一类任务：

Survival. This task group tests the ability to stay alive in the game. It is nontrivial to survive in Minecraft, because the agent grows hungry as time passes and the health bar drops gradually. Hostile mobs like zombie and skeleton spawn at night, which are very dangerous if the agent does not have the appropriate armor to protect itself or weapons to fight back. We provide two tasks with different levels of difficulty for Survival. One is to start from scratch without any assistance. The other is to start with initial weapons and food.

生存。 这一任务组检验在游戏中活下去的能力。在 Minecraft 中生存并非易事，因为智能体会随着时间推移感到饥饿，生命值也会逐渐下降。僵尸和骷髅这类敌对生物会在夜晚生成；如果智能体没有合适的盔甲保护自己，也没有武器还击，就会非常危险。我们为生存提供了两个不同难度的任务。一个是在没有任何协助的情况下从零开始。另一个则带着初始武器和食物开始。

Harvest. This task group tests the agent’s ability to collect useful resources such as minerals (iron, diamond, obsidian), food (beef, pumpkin, carrots, milk), and other useful items (wool, oak wood, coal). We construct these tasks by enumerating the Cartesian product between target items to collect, initial inventory, and world conditions (terrain, weather, lighting, etc.) so that they cover a spectrum of difficulty. For instance, if the task is to harvest wool, then it is relatively easy if the agent has a shear in its initial inventory with a sheep nearby, but more difficult if the agent has to craft the shear from raw material and explore extensively to find a sheep. We filter out combinations that are impossible (such as farming certain plants in the desert) from the Cartesian product.

采集。 这一任务组检验智能体收集有用资源的能力，例如矿物（铁、钻石、黑曜石）、食物（牛肉、南瓜、胡萝卜、牛奶）以及其他有用物品（羊毛、橡木、煤炭）。我们通过对要收集的目标物品、初始背包和世界条件（地形、天气、光照等）做笛卡尔积来构造这些任务，使它们覆盖一个难度谱。例如，如果任务是采集羊毛，那么当智能体初始背包里有剪刀且附近有羊时相对容易；如果智能体必须用原材料合成剪刀，并做大量探索才能找到羊，就会更难。我们会从笛卡尔积中过滤掉不可能的组合（例如在沙漠里种植某些植物）。

\rightarrow

\rightarrow

Combat. We test agent’s reflex and martial skills to fight against various monsters and creatures. Similar to how we develop the Harvest task group, we generate these tasks by enumerating the Cartesian product between the target entity to combat with, initial inventory, and world conditions to cover a spectrum of difficulty.

战斗。 我们检验智能体与各种怪物和生物作战时的反应和战斗技巧。与采集任务组的构造方式类似，我们通过对要作战的目标实体、初始背包和世界条件做笛卡尔积来生成这些任务，以覆盖一个难度谱。

C.2 Creative Tasks

C.2 创意任务

We construct Creative tasks using three approaches: 1) manual brainstorming, 2) mining from YouTube tutorial videos, and 3) generate by querying GPT-3 API. We elaborate the second and third approaches below.

我们用三种方法构造创意任务：1）人工头脑风暴，2）从 YouTube 教程视频中挖掘，3）通过查询 GPT-3 API 来生成。下面详细说明第二和第三种方法。

Task Mining from YouTube Tutorial Videos. Figure A.2: Labeling UI to mine tasks from YouTube. A human annotator can choose to reject the video (Invalid), adjust the timestamps, select the title, or edit and expand the original description to be the new task goal.

从 YouTube 教程视频中挖掘任务。 图 A.2：用来从 YouTube 挖掘任务的标注界面。人类标注者可以选择拒绝该视频（ 无效 ），调整时间戳，选择标题，或者编辑并扩展原描述，使其成为新的任务目标。

Our YouTube dataset serves the dual purpose of a rich task source, as many human players demonstrate and narrate creative missions in the tutorial playlists. To collect high-quality tasks and accompanying videos, we design a 3-stage pipeline that makes it easy to find and annotate interesting tasks.

我们的 YouTube 数据集同时是丰富的任务来源，因为许多人类玩家会在教程播放列表里演示并讲述有创意的任务。为了收集高质量任务及配套视频，我们设计了一个三阶段流水线，便于发现和标注有趣任务。

Stage 1: We search for YouTube playlists with the key phrases, “Minecraft Tutorial” and “Minecraft Guide”. Then we apply heuristic rules (see Sec. D.1) to filter out low-quality videos;

第 1 阶段：我们用关键词短语 “Minecraft Tutorial” 和 “Minecraft Guide” 搜索 YouTube 播放列表。然后应用启发式规则（见第 D.1 节）过滤掉低质量视频；

Stage 2: We only show the title of the video to a human annotator through a command-line interface, who makes a binary decision to accept or reject it as a potential task. This step is typically very fast, taking a few seconds on average;

第 2 阶段：我们通过命令行界面只向人类标注者展示视频标题，由标注者做二元决定，接受或拒绝它作为潜在任务。这一步通常非常快，平均只需几秒钟；

Stage 3: For the accepted tasks in stage 2, we design a labeling UI using Label Studio [70] that displays the full video and YouTube description. A human annotator can choose to reject the video, adjust the timestamps, select the title, or refine the description to be the task goal (Fig. A.2). Through this pipeline, we extract 1,042 task ideas from the common wisdom of a huge number of veteran Minecraft gamers. Some examples are “ make an automated mining machine ” and “ grow cactus up to the sky ”.

第 3 阶段：对第 2 阶段接受的任务，我们用 Label Studio [70] 设计了一个标注界面，展示完整视频和 YouTube 描述。人类标注者可以选择拒绝该视频、调整时间戳、选择标题，或润色描述使其成为任务目标（图 A.2）。通过这条流水线，我们从大量资深 Minecraft 玩家的共同智慧中提取出 1,042 个任务想法。一些例子是“做一台自动采矿机”和“把仙人掌种到天上”。

C.3 GPT-3 Guidance

C.3 GPT-3 指引

We leverage OpenAI’s GPT-3-davinci API to automatically generate detailed guidance for a subset of the tasks. Inspired by [67], we adopt the following template to prompt GPT-3: How to {task goal} in Minecraft? Let’s think step by step. Here are some examples:

我们利用 OpenAI 的 GPT-3-davinci API，为一部分任务自动生成详细指引。受 [67] 启发，我们采用如下模板来提示 GPT-3：How to {task goal} in Minecraft? Let’s think step by step. Here are some examples:

The guidance for the task “ find material and craft a gold pickaxe ” is 1) Find a place with a lot of trees; 2) Cut down the trees and gather the wood; 3) Find a place with a lot of stone; 4) Mine the stone and gather the cobblestone; 5) Find a place with a lot of iron; 6) Mine the iron and gather the iron ingots; 7) Find a place with a lot of gold; 8) Mine the gold and gather the gold ingots; 9) Craft a gold pickaxe.

任务“寻找材料并合成一把金镐”的指引是：1）找到一个树木很多的地方；2）砍树并收集木头；3）找到一个石头很多的地方；4）开采石头并收集圆石；5）找到一个铁矿很多的地方；6）开采铁矿并收集铁锭；7）找到一个金矿很多的地方；8）开采金矿并收集金锭；9）合成一把金镐。

The guidance for the task “ sail on boat with a sheep ” is 1) Find a boat; 2) Place the sheep in the boat; 3) Right-click on the boat with an empty hand to get in; 4) Use the WASD keys to move the boat. The sheep should stay in the boat.

任务“乘船带着一只羊航行”的指引是：1）找到一艘船；2）把羊放进船里；3）空手右键点击船以登船；4）用 WASD 键移动船。羊应当留在船里。

C.4 Playthrough: Defeat the Ender Dragon

C.4 通关：击败末影龙

Our benchmarking suite includes a special task called “Playthrough”. The agent is initialized bare-handed in a freshly created world and aims to defeat the Ender dragon, which is considered the final boss of Minecraft. This task holds a unique position in our benchmark because killing the dragon means “beating the game” in the traditional sense of the phrase, and is considered the most significant achievement for a new player. This boss is optional and plenty of people choose to skip it without affecting their open-ended game experience.

我们的基准套件包含一个名为“通关”（Playthrough）的特殊任务。智能体在一个全新创建的世界里空手初始化，目标是击败被视为 Minecraft 最终首领的 末影龙 。该任务在我们的基准中地位独特，因为杀掉这条龙意味着传统意义上的“通关游戏”，也被视为新玩家最重要的成就。这个首领是可选的，许多人选择跳过它，而这并不影响他们的开放式游戏体验。

“Playthrough” is technically a programmatic task, because we can check the simulator state for the defeat of the Ender dragon. However, we decide to create its own category due to the uniqueness as well as the sheer difficulty of the task. The mission requires lots of preparation, exploration, agility, and trial-and-error, which may take a regular human player many days to complete. It would be extremely long horizon (hundreds of thousands of steps) and difficult for an agent to tackle. We consider this one of the moonshot goals in MineDojo.

“通关”从技术上说是一个程序化任务，因为我们可以检查仿真器状态来确认末影龙是否被击败。不过，由于该任务的独特性以及极高的难度，我们决定单独为它设立一类。这项任务需要大量准备、探索、灵敏和反复试错，普通人类玩家可能要许多天才能完成。对智能体而言，它会是极长视野的（数十万步）且非常困难。我们把它视为 MineDojo 中的登月目标之一。

Appendix D Internet-Scale Database

附录 D 互联网规模数据库

We upload our databases to zenodo.org, which is an open repository platform operated by CERN. The data DOIs, URLs, and licenses are listed below. In this section, we describe our database properties and data collection process in details.

我们把数据库上传到由 CERN 运营的开放仓储平台 zenodo.org。数据的 DOI、网址和许可证列在下面。在这一节中，我们详细描述数据库的性质和采集过程。

D.1 YouTube Videos and Transcripts

D.1 YouTube 视频与字幕

Figure A.3: Distribution of YouTube video duration. The histogram is trimmed by the 85th percentile to hide much longer videos that can run for many hours.

图 A.3：YouTube 视频时长的分布。直方图按第 85 百分位截断，以隐藏那些可以长达许多小时的更长视频。

Minecraft is among the most streamed games on YouTube [41]. Human players have demonstrated a stunning range of creative activities and sophisticated missions that take hours to complete. We collect 33 years worth of video and 2.2B words in the accompanying English transcripts. The distribution of video duration is shown in Fig. A.3. The time-aligned transcripts enable the agent to ground free-form natural language in video pixels and learn the semantics of diverse activities without laborious human labeling.

Minecraft 是 YouTube 上被直播最多的游戏之一 [41]。人类玩家已经展示出令人惊叹的创造性活动和需要数小时才能完成的复杂任务。我们收集了总时长相当于 33 年的视频，以及配套英文字幕中的 22 亿词。视频时长分布见图 A.3。时间对齐的字幕使智能体能够把自由格式自然语言锚定到视频像素上，并在不必费力人工标注的情况下学习多样活动的语义。

We use the official YouTube Data API [140] to collect our database, following the procedure below:

我们使用官方 YouTube Data API [140] 按如下流程收集数据库：

a) Search for channels that contain Minecraft videos using a list of keywords, e.g., “Minecraft”, “Minecraft Guide”, “Minecraft Walkthrough”, “Minecraft Beginner”. We do not directly search for videos at this step because there is a limit of total results returned by the API;

a）用一组关键词搜索包含 Minecraft 视频的 频道 ，例如 “Minecraft”“Minecraft Guide”“Minecraft Walkthrough”“Minecraft Beginner”。我们在这一步不直接搜索视频，因为 API 返回的总结果数量有上限；

b) Search for all the video IDs uploaded by each channel that we obtain at the previous step. There are many false positives at this step because some channels (like gaming news channel) may cover a range of topics other than Minecraft;

b）搜索上一步得到的每个频道上传的全部视频 ID。这一步会有许多误报，因为有些频道（例如游戏资讯频道）可能覆盖 Minecraft 以外的一系列主题；

c) To remove the false positives, we rely on the video category chosen by the user when the video was uploaded and filter out all the videos that do not belong to the Minecraft category;

c）为去除误报，我们依据用户上传视频时所选的视频类别，过滤掉所有不属于 Minecraft 类别的视频；

d) To curate a language-grounded dataset, we favor videos that have English transcripts, which can be manually uploaded by the user, automatically transcribed from audio, or automatically translated from another language by the YouTube engine. For each video, we filter it out if 1) the view count is less than 100; or 2) the aspect ratio is less 1; or 3) the duration is less than 1 minute long; or 4) marked as age-restricted.

d）为整理出一个有语言锚定的数据集，我们优先选择带英文字幕的视频。这些字幕可以由用户手工上传、由音频自动转写，或由 YouTube 引擎从另一种语言自动翻译而来。对每个视频，若出现以下情形则过滤掉：1）观看次数少于 100；或 2）宽高比小于 1；或 3）时长不到 1 分钟；或 4）被标记为限制级。

e) To further clean the dataset and remove potentially harmful contents, we employ the Detoxify [51] tool to process each video title and description. Detoxify is trained on Wikipedia comments to predict multiple types of toxicity like threats, obscenity, insults, and identity-based hate speech. We delete a video if the toxicity probability in any category is above 0.5.

e）为进一步清洗数据集并去除可能有害的内容，我们用 Detoxify [51] 工具处理每个视频的标题和描述。Detoxify 在维基百科评论上训练，用来预测多种类型的有害性，例如威胁、猥亵、侮辱以及基于身份的仇恨言论。如果任一类别的有害概率高于 0.5，我们就删除该视频。

We release all the video IDs along with metadata such as video titles, view counts, like counts, duration, and FPS. In line with prior practice [64], we do not release the actual MP4 files and transcripts due to legal concerns.

我们发布全部视频 ID 以及视频标题、观看次数、点赞数、时长和帧率等元数据。与先前做法一致 [64]，出于法律方面的考虑，我们不发布实际的 MP4 文件和字幕。

D.2 Minecraft Wiki

Figure A.4: Wiki dataset examples. Closewise order: Villager trade table, mineral ingredient descriptions, monster gallery, and terrain explanation.

图 A.4：Wiki 数据集示例。按顺时针依次是：村民交易表、矿物成分说明、怪物图鉴和地形解释。

Figure A.5: More Wiki database examples with bounding boxes (annotated in red). Left: wood block introduction; right: first day tutorial.

图 A.5：更多带边界框（用红色标注）的 Wiki 数据库示例。左：木方块介绍；右：第一天教程。

The Wiki pages cover almost every aspect of the game mechanics, and supply a rich source of unstructured knowledge in multimodal tables, recipes, illustrations, and step-by-step tutorials (example screenshots in Fig. A.4 and Fig. A.5). We use Selenium [103] to scrape 6,735 pages that interleave text, images, tables, and diagrams. We elaborate the details of each web element scraped by Selenium:

Wiki 页面几乎覆盖了游戏机制的每一个方面，并以多模态表格、配方、插图和逐步教程的形式提供丰富的非结构化知识（示例截图见图 A.4 和图 A.5）。我们用 Selenium [103] 抓取了 6,735 个交错包含文本、图像、表格和示意图的页面。下面详细说明 Selenium 抓取的每类网页元素：

a) Screenshot. Using Selenium’s built-in function, we take a full screenshot of the rendered Wiki page in order to preserve the human-readable visual formatting. We also record the bounding boxes of each salient web element on the page.

a） 截图 。 我们使用 Selenium 的内置功能，对渲染后的 Wiki 页面做整页截图，以保留人类可读的视觉版式。我们还记录页面上每个显著网页元素的边界框。

b) Text. We hand-select several HTML tags that likely contain meaningful text data, such as p, h1, h2, ul, dl.

b） 文本 。 我们手工挑选若干很可能包含有意义文本数据的 HTML 标签，例如 p、h1、h2、ul、dl。

c) Images and Animations. We download the raw source file of each image element (JPG, PNG, GIF, etc.), as well as the corresponding caption if available. There are also animation effects enabled by JavaScript on the Wiki. We save all image frames in the animation.

c） 图像与动画 。 我们下载每个图像元素的原始源文件（JPG、PNG、GIF 等），以及对应图注（如有）。Wiki 上还有由 JavaScript 实现的动画效果。我们保存动画中的全部图像帧。

d) Sprites. Sprite elements are micro-sized image icons that are typically embedded in text to create multimodal tutorials and explanations. We save all the sprites and locate their bounding boxes within the text too.

d） 精灵图 。 精灵图元素是微小型图像图标，通常嵌在文本中以制作多模态教程和说明。我们保存全部精灵图，并定位它们在文本中的边界框。

e) Tables. We save the text content and bounding box of each cell that a table element contains. We store the header cells separately as they carry the semantic meaning of each column. A table can be easily reconstructed with the stored text strings and bounding boxes.

e） 表格 。 我们保存表格元素中每个单元格的文本内容和边界框。表头单元格单独存储，因为它们承载每一列的语义。利用存下的文本字符串和边界框，可以很容易重建表格。

D.3 Reddit

Figure A.6: Distribution of Reddit post types.

图 A.6：Reddit 帖子类型的分布。

Figure A.7: Examples of posts and comment threads from the Reddit database.

图 A.7：来自 Reddit 数据库的帖子和评论线程示例。

There are more than 1M subreddits (i.e., Reddit topics) where people can discuss a wide range of themes and subjects. Prior works use Reddit data for conversational response selection [5, 139, 54] and abstractive summarization [127, 65]. The r/Minecraft subreddit contains free-form discussions of game strategies and images/videos showcases of Minecraft builds and creations (examples in Fig. A.7). The distribution of post types is shown in Fig. A.6.

Reddit 上有超过 100 万个版块（即讨论主题），人们可以讨论广泛的主题和内容。先前工作用 Reddit 数据做对话回复选择 [5, 139, 54] 和抽象摘要 [127, 65]。r/Minecraft 版块包含关于游戏策略的自由讨论，以及 Minecraft 建筑与创作的图像/视频展示（例子见图 A.7）。帖子类型分布见图 A.6。

To scrape the Reddit contents, we use PRAW [88], a Python wrapper on top of the official Reddit API. Our procedure is as follows:

为抓取 Reddit 内容，我们使用建立在官方 Reddit API 之上的 Python 封装 PRAW [88]。流程如下：

\geq 5

\geq 5

b) Determine each post’s type. There are 4 native post types - text, image/video, link, and poll. We group text and poll posts together as text posts, and store their body text. For image/video and link posts, we store the source file URLs on external media hosting sites like Imgur and Gfycat. Based on the URL of each link post, we classify it as an image post, a video post or a general link post.

b）判定每条帖子的类型。原生帖子类型有 4 种：文本、图像/视频、链接和投票。我们把文本帖和投票帖归为一类文本帖，并存储其正文。对图像/视频帖和链接帖，我们存储 Imgur 和 Gfycat 这类外部媒体托管站点上的源文件网址。根据每条链接帖的网址，我们把它分类为图像帖、视频帖或一般链接帖。

c) Scrape the comments and store the parent ID of each comment so that we can reconstruct the threaded discussion.

c）抓取评论，并存储每条评论的父 ID，以便重建线程式讨论。

d) Similarly to our YouTube database (Sec. D.1), we run Detoxify [51] on the scraped Reddit contents to filter out potentially toxic and harmful posts.

d）与 YouTube 数据库类似（第 D.1 节），我们在抓取到的 Reddit 内容上运行 Detoxify [51]，以过滤可能有害的帖子。

We release all post IDs and their corresponding metadata. We also provide a Python function based on PRAW for researchers to download the post contents after obtaining a license key for the official Reddit API.

我们发布全部帖子 ID 及其对应元数据。在研究者获得官方 Reddit API 的许可密钥后，我们还提供一个基于 PRAW 的 Python 函数，供他们下载帖子内容。

Appendix E MineCLIP Algorithm Details

附录 E MineCLIP 算法细节

We implement all our neural networks in PyTorch v1.11 [86]. Training MineCLIP uses the PyTorch-Lightning framework [32], pre-trained models hosted on HuggingFace [132], and the x-transformers library for Transformer variants [128].

我们全部神经网络都在 PyTorch v1.11 [86] 中实现。训练 MineCLIP 使用 PyTorch-Lightning 框架 [32]、托管在 HuggingFace [132] 上的预训练模型，以及用于 Transformer 变体的 x-transformers 库 [128]。

E.1 Video-Text Pair Extraction

E.1 视频-文本对提取

Similar to VideoCLIP [136], we sample 640K pairs of 16-second video snippets and time-aligned English transcripts by the following procedure:

与 VideoCLIP [136] 类似，我们按如下流程采样 64 万对 16 秒视频片段与时间对齐的英文字幕：

Collect a list of keywords corresponding to the supported entities, blocks, and items in Minecraft;

1）收集一份与 Minecraft 中受支持的实体、方块和物品相对应的关键词列表；

Perform string matching over our YouTube video transcripts to obtain 640K text segments;

2）在我们的 YouTube 视频字幕上做字符串匹配，得到 64 万个文本片段；

\sim

\sim

Randomly sample a timestamp within the start and end time of the matched transcript as the center for the video clip;

4）在匹配字幕的起止时间之内随机采样一个时间戳，作为视频片段的中心；

\sim

\sim

E.2 Architecture

E.2 网络结构

MineCLIP architecture is composed of three parts:

MineCLIP 的结构由三部分组成：

\phi_{I}

\phi_{I}

\phi_{a}

\phi_{a}

Average pooling (MineCLIP[avg]): a simple, parameter-free operator. It is fast to execute but loses the temporal information, because average pooling is permutation-invariant.

平均池化（MineCLIP[avg]）：一种简单、无参数的算子。执行很快，但会丢失时间信息，因为平均池化对排列不变。

Self-Attention (MineCLIP[attn]): a 2-layer transformer encoder with 512 embedding size, 8 attention heads, and Gated Linear Unit variant with Swish activation [106, 22]. The transformer sequence encoder is relatively slower, but captures more temporal information and achieves better performance in our experiments (Table 1).

自注意力（MineCLIP[attn]）：一个 2 层 Transformer 编码器，嵌入维度为 512，有 8 个注意力头，并采用带 Swish 激活的门控线性单元变体 [106, 22]。该 Transformer 序列编码器相对较慢，但能捕获更多时间信息，并在我们的实验中取得更好表现（表 1）。

0.5\times

Hyperparameter	Value
LR schedule	Cosine with warmup [73]
Warmup steps	500
Peak LR	1.5e-4
Final LR	1e-5
Weight decay	0.2
Layerwise LR decay	0.65
Pre-trained layers LR multiplier	$0.5\times$
Batch size per GPU	64
Parallel GPUs	8
Video resolution	$160\times 256$
Number of frames	16
Image encoder	ViT-B/16 [28]

0.5\times

超参数	取值
学习率日程	带预热的余弦 [73]
预热步数	500
峰值学习率	1.5e-4
最终学习率	1e-5
权重衰减	0.2
分层学习率衰减	0.65
预训练层学习率乘数	$0.5\times$
每块 GPU 的批量大小	64
并行 GPU 数	8
视频分辨率	$160\times 256$
帧数	16
图像编码器	ViT-B/16 [28]

\phi_{G}

\phi_{G}

E.3 Training

E.3 训练

\times 0.5

\times 0.5

Appendix F Policy Learning Details

附录 F 策略学习细节

\pi_{\theta}

\pi_{\theta}

In this section, we elaborate how a trained MineCLIP can be adapted as a reward function with two different formulations. We then discuss the algorithm for policy learning. Finally, we demonstrate how we combine self imitation learning and on-policy learning to further improve sample efficiency.

在这一节中，我们详细说明如何用两种不同形式把训练好的 MineCLIP 改写成奖励函数。接着讨论策略学习算法。最后展示我们如何结合自我模仿学习和同策略学习，以进一步提升样本效率。

F.1 Adapt MineCLIP as Reward Function

F.1 把 MineCLIP 改写成奖励函数

We investigate two ways to convert MineCLIP output to scalar reward, dubbed Direct and Delta. The ablation results for Animal-Zoo task group are presented in Table A.3.

我们考察了两种把 MineCLIP 输出转换成标量奖励的方式，分别称为 Direct 和 Delta。Animal-Zoo 任务组上的消融结果见表 A.3。

T

G

t

t

8.6\pm 1.2

Tasks	Direct	Direct-Naive	Delta
Milk Cow	${\color[rgb]{0.09,0.45,0.27}\mathbf{64.5\pm 37.1}}$	$8.6\pm 1.2$	$7.6\pm 5.2$
Hunt Cow	${\color[rgb]{0.09,0.45,0.27}\mathbf{83.5\pm 7.1\hphantom{0}}}$	$0.0\pm 0.0$	$0.0\pm 0.0$
Shear Sheep	${\color[rgb]{0.09,0.45,0.27}\mathbf{12.1\pm 9.1\hphantom{0}}}$	$0.8\pm 0.6$	$1.8\pm 1.5$
Hunt Sheep	${\color[rgb]{0.09,0.45,0.27}\mathbf{8.1\pm 4.1}}$	$0.1\pm 0.2$	$0.0\pm 0.0$

8.6\pm 1.2

任务	Direct	Direct-Naive	Delta
挤牛奶	${\color[rgb]{0.09,0.45,0.27}\mathbf{64.5\pm 37.1}}$	$8.6\pm 1.2$	$7.6\pm 5.2$
猎牛	${\color[rgb]{0.09,0.45,0.27}\mathbf{83.5\pm 7.1\hphantom{0}}}$	$0.0\pm 0.0$	$0.0\pm 0.0$
剪羊毛	${\color[rgb]{0.09,0.45,0.27}\mathbf{12.1\pm 9.1\hphantom{0}}}$	$0.8\pm 0.6$	$1.8\pm 1.5$
猎羊	${\color[rgb]{0.09,0.45,0.27}\mathbf{8.1\pm 4.1}}$	$0.1\pm 0.2$	$0.0\pm 0.0$

F.2 Policy Network Architecture

F.2 策略网络结构

Our policy architecture consists of three parts: an input feature encoder, a policy head, and a value function. To handle multimodal observations (Sec. B.1), the feature extractor contains several modality-specific components:

我们的策略结构由三部分组成：输入特征编码器、策略头和价值函数。为处理多模态观察（第 B.1 节），特征提取器包含若干针对特定模态的组件：

\phi_{I}

\phi_{I}

\phi_{G}

\phi_{G}

\sin(\cdot)

\sin(\cdot)

• GPS: normalize and featurize via MLP.

• GPS：归一化后通过 MLP 提取特征。

3\times 3\times 3

3\times 3\times 3

• Past action: our agent is conditioned on its immediate past action, which is embedded and featurized by MLP.

• 过去动作：我们的智能体以刚刚过去的动作为条件，该动作经嵌入后由 MLP 提取特征。

Features from all modalities are concatenated, passed through another fusion MLP, and finally fed into the policy head and value function head. We use an MLP to model the policy head that maps from the input feature vectors to the action probability distribution. We use another MLP to estimate the value function, conditioned on the same input features.

所有模态的特征被拼接起来，再经过另一个融合 MLP，最后送入策略头和价值函数头。我们用一个 MLP 作为策略头，把输入特征向量映射为动作概率分布。我们用另一个 MLP 估计价值函数，它以相同的输入特征为条件。

F.3 RL Training

F.3 强化学习训练

PPO. We use the popular PPO algorithm [102] (Proximal Policy Optimization) as our RL training backbone. PPO is an on-policy method that optimizes for a surrogate objective while ensuring that the deviation from the previous policy is relatively small. PPO updates the policy network by

PPO。 我们使用流行的 PPO 算法 [102]（近端策略优化）作为强化学习训练的主干。PPO 是一种同策略方法，它优化一个代理目标，同时确保与先前策略的偏离相对较小。PPO 通过下式更新策略网络

\underset{\theta}{\text{maximize}}\;\mathbb{E}_{s,a\sim\pi_{\theta_{\text{old}}}}L(s,a,\theta_{\text{old}},\theta), \tag{1}

where

其中

L(s,a,\theta_{\text{old}},\theta)=\min\left(\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}A^{\pi_{\theta_{\text{old}}}}(s,a),\text{clip}\left(\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{\text{old}}}(a|s)},1-\epsilon,1+\epsilon\right)A^{\pi_{\theta_{\text{old}}}}(s,a)\right). \tag{2}

A

A

\mathcal{D}_{SI}

\mathcal{D}_{SI}

\mathcal{D}_{SI}

\mathcal{D}_{SI}

In Fig. A.8, we demonstrate that adding self-imitation dramatically improves the stability, performance, and sample efficiency of RL training in MineDojo.

在图 A.8 中，我们表明加入自我模仿会显著提升 MineDojo 中强化学习训练的稳定性、表现和样本效率。

(a) “Milk Cow”

（a）“挤牛奶”

Appendix G Experiment Details

附录 G 实验细节

G.1 Task Details

G.1 任务细节

We experiment with three task groups with four tasks per group. We train one multi-task agent for each group. In this section, we describe each task goals, initial setup, and the manual dense-shaping reward function.

我们实验了三个任务组，每组四个任务。我们为每一组训练一个多任务智能体。在这一节中，我们描述每个任务的目标、初始设置以及人工稠密塑形奖励函数。

Animal Zoo: 4 Programmatic tasks on hunting or harvesting resource from animals. We spawn various animal types (pig, sheep, and cow) in the same environment to serve as distractors. It is considered a failure if the agent does not take action on the correct animal specified by the prompt.

Animal-Zoo： 4 个与猎取或从动物身上采集资源有关的程序化任务。我们在同一环境中生成多种动物（猪、羊和牛）作为干扰项。如果智能体没有对提示所指定的正确动物采取动作，则视为失败。

r_{t}=\lambda_{\text{nav}}\max(d_{\text{min},t-1}-d_{\text{min},t},0)+\lambda_{\text{success}}\mathbbm{1}(\text{milk collected})

r_{t}=\lambda_{\text{nav}}\max(d_{\text{min},t-1}-d_{\text{min},t},0)+\lambda_{\text{success}}\mathbbm{1}(\text{milk collected})

r_{t}=\lambda_{\text{attack}}\mathbbm{1}(\text{valid attack})+\lambda_{\text{nav}}\max(d_{\text{min},t-1}-d_{\text{min},t},0)+\lambda_{\text{success}}\mathbbm{1}(\text{cow hunted})

r_{t}=\lambda_{\text{attack}}\mathbbm{1}(\text{valid attack})+\lambda_{\text{nav}}\max(d_{\text{min},t-1}-d_{\text{min},t},0)+\lambda_{\text{success}}\mathbbm{1}(\text{cow hunted})

• Shear Sheep : find and approach a sheep, then collect wool from the sheep with a shear. The prompt is shear a sheep. We initialize the agent with a shear. The manual dense reward shaping is a navigation reward based on geodesic distance obtained from the privileged LIDAR sensor, similar to “Milk Cow”.

• 剪羊毛 ：找到并靠近一只羊，然后用剪刀从羊身上采集羊毛。提示是 shear a sheep。我们给智能体初始化一把剪刀。人工稠密奖励塑形是一种基于特权激光雷达传感器得到的测地距离的导航奖励，与“挤牛奶”类似。

• Hunt Sheep : find and approach a sheep, then hunt with a sword. The sheep will run away so the agent needs to chase after it. An episode will terminate once any entity is hunted. The prompt is hunt a sheep. We initialize the agent with a diamond sword. The manual dense reward shaping consists of two parts, a valid attack reward and a navigation reward based on geodesic distance obtained from the privileged LIDAR sensor, similar to “Hunt Cow”.

• 猎羊 ：找到并靠近一只羊，然后用剑猎杀。羊会逃跑，因此智能体需要追赶它。一旦任何实体被猎杀，该回合就会终止。提示是 hunt a sheep。我们给智能体初始化一把钻石剑。人工稠密奖励塑形由两部分组成：有效攻击奖励，以及基于特权激光雷达传感器得到的测地距离的导航奖励，与“猎牛”类似。

r_{t}=\lambda_{\text{attack}}\mathbbm{1}(\text{valid attack})+\lambda_{\text{success}}\mathbbm{1}(\texttt{\lx@text@lbrace monster\lx@text@rbrace}\text{ hunted})

r_{t}=\lambda_{\text{attack}}\mathbbm{1}(\text{valid attack})+\lambda_{\text{success}}\mathbbm{1}(\texttt{\lx@text@lbrace monster\lx@text@rbrace}\text{ hunted})

Creative: 4 tasks that do not have manual dense reward shaping or code-defined success criterion.

创意任务： 4 个既没有人工稠密奖励塑形、也没有用代码定义成功标准的任务。

• Find Nether Portal : find and move close to a Nether Portal, then enter the Nether world through the portal. The prompt is find a nether portal.

• 寻找下界传送门 ：找到并靠近一座下界传送门，然后穿过传送门进入 下界 。提示是 find a nether portal。

• Find Ocean : find and move close to an ocean. The prompt is find an ocean.

• 寻找海洋 ：找到并靠近一片海洋。提示是 find an ocean。

• Dig Hole : dig holes in the ground. The prompt is dig a hole. We initialize the agent with an iron shovel.

• 挖洞 ：在地上挖洞。提示是 dig a hole。我们给智能体初始化一把铁铲。

• Lay Carpet : lay down carpets to cover the wooden floor inside a house. The prompt is put carpets on the floor. We initialize the agent with a number of carpets in its inventory.

• 铺地毯 ：在一座房子里铺上地毯，覆盖木地板。提示是 put carpets on the floor。我们在智能体的背包里初始化若干地毯。

Note that we categorize “Find Nether Portal” and “Find Ocean” as Creative tasks even though they seem similar to object navigation [12]. While finding terrains and other structures is semantically well defined, it is not easy to define a function to evaluate success automatically because the simulator does not have the exact location information of these structures given a randomly generated world. In principle, we can make a sweep by querying each chunk of voxels in the world to recognize the terrains, but that would be prohibitively expensive. Therefore, we opt to use MineCLIP as the reward signal and treat these tasks as Creative.

需要注意，尽管“寻找下界传送门”和“寻找海洋”看起来类似于物体导航 [12]，我们仍把它们归为创意任务。寻找地形和其他结构在语义上定义清楚，但由于仿真器在随机生成的世界中没有这些结构的精确位置信息，不容易定义一个函数来自动评估成功。原则上，我们可以通过查询世界中每一块体素来识别地形，但那会极其昂贵。因此我们选择用 MineCLIP 作为奖励信号，并把这些任务当作创意任务。

10^{-4}

NN Architecture	Training
Hyperparameter	Animal-Zoo	Mob-Combat	Creative
RGB Feature Size	512	Learning Rate	$10^{-4}$	$10^{-4}$	$10^{-4}$
Task Prompt Feature Size	512	Cosine Decay Minimal LR	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$
{state} MLP Hidden Size	128	$\gamma$	$0.99$	$0.99$	$0.99$
{state} MLP Output Size	128	Entropy Weight (Stage 1)	$5\times 10^{-3}$	$5\times 10^{-3}$	$5\times 10^{-3}$
{state} MLP Hidden Depth	2	Entropy Weight (Stage 2)	$10^{-2}$	N/A	$10^{-2}$
Embed Dim	8	PPO Optimizer	Adam	Adam	Adam
Num Feature Fusion Layers	1	SI Learning Rate	$10^{-4}$	$10^{-4}$	$10^{-4}$
Feature Fusion Output Size	512	SI Cosine Decay Minimal LR	$10^{-6}$	$10^{-6}$	$10^{-6}$
Prev Action Conditioning	True	SI Epoch	10	10	10
Policy Head Hidden Size	256	SI Frequency (Env Steps)	100K	100K	100K
Policy Head Hidden Depth	3	SI Optimizer	Adam	Adam	Adam
VF Hidden Size	256	SI Buffer Threshold	$2\sigma$	$2\sigma$	$0.5\sigma$
VF Hidden Depth	3	PPO Buffer Size	100K	100K	100K
		Frame Stack	1	1	1
		VF Loss Weight	0.5	0.5	0.5
		GAE $\lambda$	0.95	0.95	0.95
		Gradient Clip	10	10	10
		PPO $\epsilon$	0.2	0.2	0.2
		Action Smooth Weight	$10^{-7}$	$10^{-7}$	$10^{-7}$
		Action Smooth Window Size	3	3	3
		MineCLIP Reward Formulation	Direct	Direct	Delta

10^{-4}

神经网络结构	训练
超参数	Animal-Zoo	Mob-Combat	Creative
RGB 特征维度	512	学习率	$10^{-4}$	$10^{-4}$	$10^{-4}$
任务提示特征维度	512	余弦衰减最小学习率	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$
{state} MLP 隐藏层大小	128	$\gamma$	$0.99$	$0.99$	$0.99$
{state} MLP 输出大小	128	熵权重（阶段 1）	$5\times 10^{-3}$	$5\times 10^{-3}$	$5\times 10^{-3}$
{state} MLP 隐藏深度	2	熵权重（阶段 2）	$10^{-2}$	不适用	$10^{-2}$
嵌入维度	8	PPO 优化器	Adam	Adam	Adam
特征融合层数	1	自我模仿学习率	$10^{-4}$	$10^{-4}$	$10^{-4}$
特征融合输出大小	512	自我模仿余弦衰减最小学习率	$10^{-6}$	$10^{-6}$	$10^{-6}$
以先前动作为条件	是	自我模仿周期数	10	10	10
策略头隐藏层大小	256	自我模仿频率（环境步）	100K	100K	100K
策略头隐藏深度	3	自我模仿优化器	Adam	Adam	Adam
价值函数隐藏层大小	256	自我模仿缓冲区阈值	$2\sigma$	$2\sigma$	$0.5\sigma$
价值函数隐藏深度	3	PPO 缓冲区大小	100K	100K	100K
		帧堆叠	1	1	1
		价值函数损失权重	0.5	0.5	0.5
		GAE $\lambda$	0.95	0.95	0.95
		梯度裁剪	10	10	10
		PPO $\epsilon$	0.2	0.2	0.2
		动作平滑权重	$10^{-7}$	$10^{-7}$	$10^{-7}$
		动作平滑窗口大小	3	3	3
		MineCLIP 奖励形式	Direct	Direct	Delta

G.2 Observation and Action Space

G.2 观察与动作空间

We use a subset of the full observation and action space listed in Sec. B.1 and B.2, because the tasks in our current experiments do not involve actions like crafting or inventory management. Our observation space consists of RGB frame, compass, GPS, and Voxels.

我们使用第 B.1 节和第 B.2 节所列完整观察和动作空间的一个子集，因为当前实验中的任务不涉及合成或背包管理这类动作。我们的观察空间由 RGB 帧、罗盘、GPS 和体素组成。

9\times 9

9\times 9

G.3 RL Training

G.3 强化学习训练

All hyperparameters used in our RL experiment are listed in Table A.4. We visualize the learned behaviors of 4 tasks in Figure 2. Demos of more tasks can be found on our website https://minedojo.org .

我们强化学习实验中使用的全部超参数列在表 A.4。我们在图 2 中可视化了 4 个任务上学到的行为。更多任务的演示可以在我们的网站 https://minedojo.org 上找到。

Action smoothing. Due to the stochastic nature of PPO, we observe a lot of action jittering in the agent’s behavior during training. This leads to two negative effects that degrade the learning performance: 1) exploration difficulty due to inconsistent action sequence. For example, the agent may be required to take multiple consecutive attack actions in order to complete certain tasks; and 2) rapidly switching different movement and camera motions result in videos that are highly non-smooth and disorienting. This causes a domain gap from the training data of MineCLIP, which are typically smooth human gameplay videos. Therefore, the reward signal quality deteriorates significantly.

动作平滑。 由于 PPO 的随机性，我们观察到训练期间智能体行为中有大量动作抖动。这会带来两个损害学习表现的负面效果：1）动作序列不一致导致探索困难。例如，智能体可能需要连续执行多次攻击动作才能完成某些任务；2）在不同移动和相机动作之间快速切换，会产生非常不平滑、令人迷失方向的视频。这会与 MineCLIP 的训练数据形成域差距，那些数据通常是平滑的人类游戏视频。因此，奖励信号的质量会明显下降。

\mathcal{W}

|\mathcal{W}|

\mathcal{L}_{\text{smooth}}=\frac{1}{|\mathcal{W}|}\sum_{i=1}^{|\mathcal{W}|-1}KL(\pi_{t}\|\pi_{t-|\mathcal{W}|+i}), \tag{3}

KL(\cdot)

KL(\cdot)

Multi-stage training for multi-task RL. Due to hardware limitations, we are not able to run a large number of parallel simulators for all tasks in a task group. Therefore, we adopt a multi-stage strategy to split the tasks and train them sequentially with a single policy network. For the task groups Animal-Zoo and Creative, we split the four tasks into two stages of two parallel training tasks each. We carry over the self-imitation buffers when switching to the next stage. We also follow the recommended practice in [83] and reset the policy head at the beginning of stage 2 to encourage exploration and reduce overfitting. We adopt a similar replay buffer balancing strategy as [46] to prevent any task from dominating the training.

多任务强化学习的多阶段训练。 受硬件限制，我们无法为一组中的全部任务运行大量并行仿真器。因此我们采用多阶段策略，把任务拆开，用单个策略网络依次训练。对 Animal-Zoo 和 Creative 这两组，我们把四个任务拆成两个阶段，每阶段并行训练两个任务。切换到下一阶段时，我们沿用自我模仿缓冲区。我们还遵循 [83] 中建议的做法，在第 2 阶段开始时重置策略头，以鼓励探索并减少过拟合。我们采用与 [46] 类似的回放缓冲区平衡策略，防止任一任务主导训练。

G.4 Evaluation

G.4 评估

K=2

K=2

To further investigate MineCLIP’s evaluation on more complex Creative tasks, we annotate 50 YouTube video segments each for 5 more tasks that are much more semantically complex: “build a farm”, “ build a fence”, “build a house”, “ride a minecart”, and “build a swimming pool”. We then run MineCLIP evaluation on these videos against a negative set. As shown in Table A.5, though not perfect, MineCLIP generally has a positive agreement with human judgment. We note that the current MineCLIP is a proof-of-concept step in leveraging internet data for automated evaluation, and further scaling on more training data and parameters may lead to more improvements. Meanwhile, human judgment remains a useful and important alternative [93, 97].

为进一步考察 MineCLIP 在更复杂创意任务上的评估，我们又为 5 个语义上复杂得多的任务各标注了 50 段 YouTube 视频片段：“建一座农场”“建一道围栏”“建一座房子”“乘坐矿车”以及“建一座游泳池”。随后我们针对一个负例集合，在这些视频上运行 MineCLIP 评估。如表 A.5 所示，尽管并不完美，MineCLIP 总体上与人类判断呈正向一致。我们指出，当前的 MineCLIP 只是利用互联网数据做自动评估的一个概念验证步骤，在更多训练数据和参数上继续扩大规模可能会带来更多改进。与此同时，人类判断仍然是有用且重要的替代选择 [93, 97]。

95.9

Tasks	Build a Farm	Build a Fence	Build a House	Ride a Minecart	Build a Swimming Pool
Ours (Attn)	${\color[rgb]{0.09,0.45,0.27}\mathbf{78.7}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{91.4}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{63.7}}$	$95.9$	$85.0$
Ours (Avg)	$73.4$	$83.1$	$37.4$	${\color[rgb]{0.09,0.45,0.27}\mathbf{96.9}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{94.7}}$
CLIPOpenAI	$62.5$	$24.5$	$52.9$	$70.0$	$71.7$

95.9

任务	建农场	建围栏	建房子	乘坐矿车	建游泳池
本文（注意力）	${\color[rgb]{0.09,0.45,0.27}\mathbf{78.7}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{91.4}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{63.7}}$	$95.9$	$85.0$
本文（平均）	$73.4$	$83.1$	$37.4$	${\color[rgb]{0.09,0.45,0.27}\mathbf{96.9}}$	${\color[rgb]{0.09,0.45,0.27}\mathbf{94.7}}$
CLIPOpenAI	$62.5$	$24.5$	$52.9$	$70.0$	$71.7$

Appendix H Limitations and Potential Societal Impact

附录 H 局限与潜在社会影响

Unlike human demonstrations [126] or offline RL datasets [35], our YouTube dataset contains only the video screen observations but not the actual control actions. This allows us to scale up the dataset tremendously, but at the same time poses a challenge to imitation learning algorithms that require observation-action pairs to learn. Our proposed algorithm, MineCLIP, side-steps this problem by learning a reward model, but we believe that directly inferring the human expert policy from YouTube is another important direction complementary to our approach. There are promising techniques that can potentially overcome this limitation, such as the Learning-from-Observation (LfO) family of algorithms [123, 122, 115, 31].

与人类演示 [126] 或离线强化学习数据集 [35] 不同，我们的 YouTube 数据集只包含视频屏幕观察，而不包含实际控制动作。这使我们能够极大扩展数据集规模，但同时也给需要观察-动作对才能学习的模仿学习算法带来挑战。我们提出的算法 MineCLIP 通过学习奖励模型绕开了这个问题，但我们相信，直接从 YouTube 推断人类专家策略是与我们的方法互补的另一个重要方向。有一些有前景的技术有可能克服这一局限，例如观察学习（Learning-from-Observation，LfO）这一族算法 [123, 122, 115, 31]。

Our database is scraped from the internet, which inevitably contains offensive YouTube videos or toxic Reddit posts. While we have made our best effort to filter out these harmful contents (Sec. D.1), there can still be undesirable biases and toxicity that elude our automatic filters. Furthermore, we advocate the use of large pre-trained language models in our main paper, and MineCLIP is finetuned from the pre-trained weights of OpenAI CLIP [92]. These foundation models are known to contain harmful stereotypes and generate hateful commentary [15, 13, 40]. We ask the researchers who will use our code and database to exercise their best judgment during new model development to avoid any negative social impact.

我们的数据库从互联网抓取而来，其中难免包含冒犯性的 YouTube 视频或有害的 Reddit 帖子。尽管我们已尽最大努力过滤这些有害内容（第 D.1 节），仍可能有不良偏见和毒性逃过自动过滤器。此外，我们在正文中主张使用大规模预训练语言模型，而 MineCLIP 是从 OpenAI CLIP [92] 的预训练权重微调而来。众所周知，这些基础模型包含有害刻板印象，也可能生成仇恨性评论 [15, 13, 40]。我们请将使用我们的代码和数据库的研究者，在发展新模型时做出审慎判断，以避免任何负面社会影响。

Appendix I Datasheet

附录 I 数据说明书

We present a Datasheet [39] for documentation and responsible usage of our internet knowledge databases.

我们提供一份数据说明书（Datasheet）[39]，用于记录我们的互联网知识数据库并促进负责任的使用。

I.1 Motivation

I.1 动机

For what purpose was the dataset created? We create this internet-scale multimodal knowledge base to facilitate research towards open-ended, generally capable embodied agents.

创建该数据集是为了什么目的？ 我们创建这座互联网规模的多模态知识库，是为了推动面向开放式、具备通用能力的具身智能体的研究。

Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? This knowledge base was created by Linxi Fan (Nvidia), Guanzhi Wang (Caltech), Yunfan Jiang (Stanford), Ajay Mandlekar (Nvidia), Yuncong Yang (Columbia), Haoyi Zhu (SJTU), Andrew Tang (Columbia), De-An Huang (Nvidia), Yuke Zhu (Nvidia and UT Austin), and Anima Anandkumar (Nvidia and Caltech).

谁创建了该数据集（例如哪个团队、研究组），又是代表哪个实体（例如公司、机构、组织）？ 这座知识库由 Linxi Fan（Nvidia）、Guanzhi Wang（Caltech）、Yunfan Jiang（Stanford）、Ajay Mandlekar（Nvidia）、Yuncong Yang（Columbia）、Haoyi Zhu（SJTU）、Andrew Tang（Columbia）、De-An Huang（Nvidia）、Yuke Zhu（Nvidia and UT Austin）和 Anima Anandkumar（Nvidia and Caltech）创建。

I.2 Distribution

I.2 分发

Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? Yes, the dataset is publicly available on the internet.

该数据集是否会分发给创建它所代表的实体（例如公司、机构、组织）之外的第三方？ 是的，该数据集在互联网上公开可用。

How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? All datasets can be downloaded from https://zenodo.org/ . Please refer to this table of URL, DOI, and licensing:

该数据集将如何分发（例如网站上的打包文件、API、GitHub）？ 全部数据集均可从 https://zenodo.org/ 下载。请参阅这张包含网址、DOI 和许可证的表格：

Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.

是否有第三方对实例相关数据施加了基于知识产权或其他方面的限制？ 没有。

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? No.

该数据集或其中的个别实例是否受到出口管制或其他监管限制？ 没有。

I.3 Maintenance

I.3 维护

Who will be supporting/hosting/maintaining the dataset? The authors will be supporting, hosting, and maintaining the dataset.

谁将支持/托管/维护该数据集？ 作者将支持、托管并维护该数据集。

How can the owner/curator/manager of the dataset be contacted (e.g., email address)? Please contact Linxi Fan (linxif@nvidia.com), Guanzhi Wang (guanzhi@caltech.edu), and Yunfan Jiang (yunfanj@cs.stanford.edu).

Is there an erratum? No. We will make announcements if there is any.

是否有勘误表？ 没有。如果有任何勘误，我们会发布公告。

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? Yes. New updates will be posted on https://minedojo.org .

该数据集是否会被更新（例如纠正标注错误、增加新实例、删除实例）？ 会。新的更新将发布在 https://minedojo.org。

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? N/A.

如果数据集与人有关，对与实例相关数据的保留是否有适用限制（例如，相关个人是否被告知其数据会保留固定期限然后删除）？ 不适用。

Will older versions of the dataset continue to be supported/hosted/maintained? Yes, old versions will be permanently accessible on zenodo.org.

该数据集的旧版本是否会继续得到支持/托管/维护？ 会，旧版本将永久可在 zenodo.org 上访问。

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? Yes, please refer to https://minedojo.org .

如果其他人想扩展/增强/基于该数据集继续建设/做出贡献，是否有相应机制？ 有，请参阅 https://minedojo.org。

I.4 Composition

I.4 构成

What do the instances that comprise the dataset represent? For YouTube videos, our data is in JSON format with video URLs and metadata. We do not provide the raw MP4 files for legal concerns. For Wiki, we provide the text, images, tables, and diagrams embedded on the web pages. For Reddit, our data is in JSON format with post IDs and metadata, similar to YouTube. Users can reconstruct the Reddit dataset by running our script after obtaining an official Reddit API license key.

构成该数据集的实例代表什么？ 对 YouTube 视频，我们的数据是带有视频网址和元数据的 JSON 格式。出于法律方面的考虑，我们不提供原始 MP4 文件。对 Wiki，我们提供网页中嵌入的文本、图像、表格和示意图。对 Reddit，我们的数据与 YouTube 类似，是带有帖子 ID 和元数据的 JSON 格式。用户在获得官方 Reddit API 许可密钥后，可以通过运行我们的脚本重建 Reddit 数据集。

How many instances are there in total (of each type, if appropriate)? There are more than 730K YouTube videos with 2.2B words of transcripts, 6,735 Wiki pages with 2.2M bounding boxes of visual elements, and more than 340K Reddit posts with 6.6M comments.

总共有多少个实例（如果适用，每一类各有多少）？ 有超过 73 万条 YouTube 视频及 22 亿词字幕、6,735 篇带有 220 万个视觉元素边界框的 Wiki 页面，以及超过 34 万条带有 660 万条评论的 Reddit 帖子。

Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? We provide all instances in our Zenodo data repositories.

该数据集是否包含所有可能的实例，还是来自更大集合的一个样本（不一定是随机样本）？ 我们在 Zenodo 数据仓库中提供全部实例。

Is there a label or target associated with each instance? No.

每个实例是否带有标签或目标？ 没有。

Is any information missing from individual instances? No.

个别实例是否缺失任何信息？ 没有。

Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? We provide metadata for each YouTube video link and Reddit post ID.

个别实例之间的关系是否被明确给出（例如用户的电影评分、社交网络链接）？ 我们为每个 YouTube 视频链接和 Reddit 帖子 ID 提供元数据。

Are there recommended data splits (e.g., training, development/validation, testing)? No. The entire database is intended for pre-training.

是否有推荐的数据划分（例如训练、开发/验证、测试）？ 没有。整个数据库旨在用于预训练。

Are there any errors, sources of noise, or redundancies in the dataset? Please refer to Sec. D

该数据集中是否存在任何错误、噪声来源或冗余？ 请参阅第 D 节。

Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? We follow prior works [64] and only release the video URLs of YouTube videos due to legal concerns. Researchers need to acquire the MP4 and transcript files separately. Similarly, we only release the post IDs for the Minecraft Reddit database, but we also provide a script that can reconstruct the full Reddit dataset given a free official license key.

该数据集是自包含的，还是链接到或依赖于外部资源（例如网站、推文、其他数据集）？ 我们遵循先前工作 [64]，出于法律方面的考虑只发布 YouTube 视频的网址。研究者需要另行获取 MP4 文件和字幕。类似地，我们只发布 Minecraft Reddit 数据库的帖子 ID，但也提供一个脚本，在获得免费的官方许可密钥后可以重建完整的 Reddit 数据集。

Does the dataset contain data that might be considered confidential? No.

该数据集是否包含可能被视为机密的数据？ 没有。

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? We have made our best efforts to detoxify the contents via an automated procedure. Please refer to Sec. D.

该数据集是否包含若直接观看可能冒犯、侮辱、威胁，或可能引发焦虑的数据？ 我们已尽最大努力通过自动流程净化这些内容。请参阅第 D 节。

I.5 Collection Process

I.5 采集过程

The collection procedure, preprocessing, and cleaning are explained in details in Sec. D.

采集流程、预处理和清洗在第 D 节中有详细说明。

Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? All data collection, curation, and filtering are done by MineDojo coauthors.

谁参与了数据采集过程（例如学生、众包工作者、承包者），他们如何获得报酬（例如众包工作者的酬劳是多少）？ 全部数据采集、整理和过滤都由 MineDojo 的共同作者完成。

Over what timeframe was the data collected? The data was collected between Dec. 2021 and May 2022.

数据是在什么时间范围内采集的？ 数据采集于 2021 年 12 月至 2022 年 5 月之间。

I.6 Uses

I.6 用途

Has the dataset been used for any tasks already? Yes, we have used the MineDojo YouTube database for agent pre-training. Please refer to Sec. 5 and Sec. G for algorithmic and training details.

该数据集是否已经用于任何任务？ 是的，我们已将 MineDojo 的 YouTube 数据库用于智能体预训练。算法和训练细节请参阅第 5 节和第 G 节。

What (other) tasks could the dataset be used for? Our knowledge base is primarily intended to facilitate research in open-ended, generally capable embodied agents. However, it can also be broadly applicable to research in video understanding, document understanding, language modeling, multimodal learning, and so on.

该数据集还可以用于哪些（其他）任务？ 我们的知识库主要用于推动开放式、具备通用能力的具身智能体研究。不过，它也可以广泛适用于视频理解、文档理解、语言建模、多模态学习等方面的研究。

Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? No.

该数据集的构成，或其采集和预处理/清洗/标注方式，是否有任何可能影响未来用途的地方？ 没有。

Are there tasks for which the dataset should not be used? We strongly oppose any research that intentionally generates harmful or toxic contents using our YouTube, Wiki, and Reddit data.

是否有该数据集不应当被用于的任务？ 我们强烈反对任何有意使用我们的 YouTube、Wiki 和 Reddit 数据生成有害内容的研究。

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

MineDojo：用互联网规模知识构建开放式具身智能体

Abstract

摘要

1 Introduction

1 引言

1. Simulation platform with thousands of diverse open-ended tasks.

1. 带有数千个多样开放式任务的仿真平台。

2. Internet-scale multimodal Minecraft knowledge base.

2. 互联网规模的多模态 Minecraft 知识库。

3. Novel algorithm for embodied agents with large-scale pre-training.

3. 面向具身智能体、带大规模预训练的新算法。

2 MineDojo Simulator & Benchmark Suite

2 MineDojo 仿真器与基准套件

2.1 Task Suite I: Programmatic Tasks

2.1 任务套件一：程序化任务

2.2 Task Suite II: Creative Tasks

2.2 任务套件二：创意任务

Approach 1. Task Mining from YouTube Tutorial Videos.

方法 1. 从 YouTube 教程视频中挖掘任务。

Approach 2. Task Creation by GPT-3.

方法 2. 由 GPT-3 创建任务。

2.3 Collection of Starter Tasks

2.3 入门任务合集

3 Internet-scale Knowledge Base

3 互联网规模知识库

YouTube Videos and Transcripts.

YouTube 视频与字幕。

Minecraft Wiki.

Minecraft Wiki。

Reddit.

Reddit。

4 Agent Learning with Large-scale Pre-training

4 大规模预训练下的智能体学习

4.1 Pre-Training MineCLIP on Large-scale Videos

4.1 在大规模视频上预训练 MineCLIP

4.2 RL with MineCLIP Reward

4.2 使用 MineCLIP 奖励做强化学习

5 Experiments

5 实验

MineCLIP is competitive with manual reward.

MineCLIP 可与人工奖励相媲美。

MineCLIP provides automated evaluation.

MineCLIP 提供自动评估。

MineCLIP shows good zero-shot generalization to significant visual distribution shift.

MineCLIP 对显著的视觉分布偏移表现出良好的零样本泛化。

Learning a Single Agent for All 12 Tasks

为全部 12 个任务学习单个智能体

Generalize to Novel Tasks

泛化到新任务

6 Related work

6 相关工作

Open-ended Environments for Decision-making Agents.

面向决策智能体的开放式环境。

Minecraft for AI Research.

面向人工智能研究的 Minecraft。

Internet-scale Multimodal Knowledge Bases.

互联网规模的多模态知识库。

Embodied Agents with Large-scale Pre-training.

带大规模预训练的具身智能体。

7 Conclusion

7 结论

8 Acknowledgement

8 致谢

References

参考文献

Appendix A Minecraft Framework Comparison

附录 A Minecraft 框架比较

Appendix B MineDojo Simulator

附录 B MineDojo 仿真器

B.1 Observation Space

B.1 观察空间

B.2 Action Space

B.2 动作空间

B.3 Customizing the Environment

B.3 定制环境

Appendix C MineDojo Task Suite

附录 C MineDojo 任务套件

C.1 Programmatic Tasks

C.1 程序化任务

Frame-wise image encoder $\phi_{I}$

逐帧图像编码器 $\phi_{I}$

Temporal aggregator $\phi_{a}$

时间聚合器 $\phi_{a}$

Text encoder $\phi_{G}$

文本编码器 $\phi_{G}$