ProcTHOR: Large-Scale Embodied AI Using Procedural Generation

ProcTHOR: Large-Scale Embodied AI Using Procedural Generation

ProcTHOR：用程序化生成实现大规模具身人工智能

Matt Deitke Eli VanderBilt Alvaro Herrasti Luca Weihs Jordi Salvador Kiana Ehsani Winson Han Eric Kolve Ali Farhadi Affiliation: PRIOR @ Allen Institute for AI, University of Washington, Seattle procthor.allenai.org Aniruddha Kembhavi Roozbeh Mottaghi

Abstract

摘要

Massive datasets and high-capacity models have driven many recent advancements in computer vision and natural language understanding. This work presents a platform to enable similar success stories in Embodied AI. We propose ProcTHOR, a framework for procedural generation of Embodied AI environments. ProcTHOR enables us to sample arbitrarily large datasets of diverse, interactive, customizable, and performant virtual environments to train and evaluate embodied agents across navigation, interaction, and manipulation tasks. We demonstrate the power and potential of ProcTHOR via a sample of 10,000 generated houses and a simple neural model. Models trained using only RGB images on ProcTHOR, with no explicit mapping and no human task supervision produce state-of-the-art results across 6 embodied AI benchmarks for navigation, rearrangement, and arm manipulation, including the presently running Habitat 2022, AI2-THOR Rearrangement 2022, and RoboTHOR challenges. We also demonstrate strong 0-shot results on these benchmarks, via pre-training on ProcTHOR with no fine-tuning on the downstream benchmark, often beating previous state-of-the-art systems that access the downstream training data.

大规模数据集和大容量模型推动了计算机视觉和自然语言理解的许多近期进展。这项工作给出一个平台，希望在具身人工智能中促成类似的成功。我们提出 ProcTHOR，一套用于程序化生成具身人工智能环境的框架。ProcTHOR 使我们能够采样任意大规模的多样、可交互、可定制且性能足够好的虚拟环境，以在导航、交互和操作任务上训练并评估具身智能体。我们通过采样生成的 1 万套房子和一个简单神经网络，展示了 ProcTHOR 的能力和潜力。仅用 RGB 图像、在 ProcTHOR 上训练、不做显式建图、也没有人工任务监督的模型，在导航、重排和机械臂操作共 6 个具身人工智能基准上取得了当时最优结果，包括当时正在进行的 Habitat 2022、AI2-THOR Rearrangement 2022 和 RoboTHOR 挑战。我们还表明，只在 ProcTHOR 上预训练、不对下游基准微调的零样本结果也很强，常常超过此前使用过下游训练数据的最优系统。

1 Introduction

1 引言

Computer vision and natural language processing models have become increasingly powerful through the use of large-scale training data. Recent models such as CLIP [93], DALL-E [95], GPT-3 [10], and Flamingo [3] use massive amounts of task agnostic data to pre-train large neural architectures that perform remarkably well at downstream tasks, including in zero and few-shot settings. In comparison, the Embodied AI (E-AI) research community predominantly trains agents in simulators with far fewer scenes [94, 63, 27]. Due to the complexity of tasks and the need for long planning horizons, the best performing E-AI models continue to overfit on the limited training scenes and thus generalize poorly to unseen environments.

借助大规模训练数据，计算机视觉和自然语言处理模型正变得越来越强。CLIP [93]、DALL-E [95]、GPT-3 [10] 和 Flamingo [3] 等近期模型用海量与任务无关的数据预训练大型神经结构，它们在下游任务上表现出色，也包括零样本和少样本设定。相比之下，具身人工智能（E-AI）研究社区主要在场景少得多的仿真器中训练智能体 [94, 63, 27]。由于任务复杂，而且需要很长的规划视野，表现最好的具身模型仍会在有限的训练场景上过拟合，因而对未见环境泛化很差。

In recent years, E-AI simulators have become increasingly more powerful with support for physics, manipulators, object states, deformable objects, fluids, and real-sim counterparts [63, 101, 104, 37, 124], but scaling them up to tens of thousands of scenes has remained challenging. Existing E-AI environments are either designed manually [63, 37] or obtained via 3D scans of real structures [101, 94]. The former approach requires 3D artists to spend a significant amount of time designing 3D assets, arranging them in sensible configurations within large spaces, and carefully configuring the right textures and lighting in these environments. The latter involves moving specialized cameras through many real-world environments and then stitching the resulting images together to form 3D reconstructions of the scenes. These approaches are not scalable, and expanding existing scene repositories multiple orders of magnitude is not practical.

近年来，具身仿真器变得越来越强，已经支持物理、机械臂、物体状态、可变形物体、流体以及真实-仿真对应物 [63, 101, 104, 37, 124]，但要把它们扩展到数万个场景仍然很难。现有的具身环境要么手工设计 [63, 37]，要么通过对真实建筑做三维扫描得到 [101, 94]。前一种做法需要 3D 美术花大量时间设计三维资产、在大空间里把它们摆成合理配置，并仔细设置纹理和光照。后一种做法则要带着专用相机穿过许多真实环境，再把得到的图像拼接成场景的三维重建。这些做法都难以扩展，要把现有场景库再扩大几个数量级并不实际。

我们提出 ProcTHOR，一套建立在 AI2-THOR [63] 之上的框架，用来为具身人工智能研究程序化生成完全可交互、带物理的环境。给定房间规格（例如一座有 3 间卧室、3 间卫生间和 1 间厨房的房子），ProcTHOR 可以生成大量满足这些要求的多样平面图（图 1）。一个包含 108 类物体、1633 个完全可交互实例的大型资产库被用来自动填充每张平面图，并确保物体摆放在物理上合理、自然且接近真实。人们还可以改变各场景中照明元件（包括人工照明和模拟天空盒）的强度和颜色，以模拟室内光照和一天中不同时刻的变化。资产（例如家具和水果）以及墙壁和门这类更大的结构，都可以被赋予多种颜色和纹理，从每一资产类别的一组合理颜色和材质中采样。布局、资产、摆放和光照上的多样性合在一起，带来任意大规模的环境集合——使 ProcTHOR 能够比当下仿真器所支持的场景数量再高出几个数量级。此外，ProcTHOR 支持动态材质随机化，每次把环境载入内存进行训练时，都可以随机化单个资产的颜色和材质。重要的是，与用三维扫描得到的环境不同，ProcTHOR 生成的场景中的物体既支持多种不同状态（ 例如 打开、关闭、损坏 等 ），又完全可交互，因此可以被带机械臂的智能体物理操作。我们还提出 ArchitecTHOR，这是由 3D 美术设计的 10 套高质量、完全可交互的房子，专门用作家庭环境研究的仅测试环境。与 AI2-iTHOR（单房间）和 RoboTHOR（视觉多样性较弱）相比，ArchitecTHOR 包含更大、更多样、也更接近真实住宅的房子。

Figure 1: We propose ProcTHOR, a framework to procedurally generate a large variety of diverse, interactable, and customizable houses.

图 1：我们提出 ProcTHOR，一套用于程序化生成大量多样、可交互且可定制房屋的框架。

{>}3

{>}3

In summary, our contributions are (1) ProcTHOR, a framework that allows for the performant procedural generation of an unbounded number of diverse, fully-interactive, simulated environments, (2) ArchitecTHOR, a new, 3D artist-designed set of houses for E-AI evaluation, and (3) SoTA results across six E-AI benchmarks covering manipulation and navigation tasks, including strong 0-shot results. ProcTHOR will be open-sourced and the code used in this work will be released.

总之，我们的贡献是：(1) ProcTHOR，一套能够高性能地程序化生成任意数量多样、完全可交互仿真环境的框架；(2) ArchitecTHOR，一套新的、由 3D 美术设计的房子，用于具身人工智能评估；(3) 在覆盖操作和导航任务的六个具身基准上的最优结果，包括很强的零样本结果。ProcTHOR 将会开源，本工作中使用的代码也将发布。

2 Related Work

2 相关工作

Embodied AI platforms. Various Embodied AI platforms have been developed over the past several years [63, 101, 104, 124, 37, 121]. These platforms target different design goals. AI2-THOR [63] and its variants (ManipulaTHOR [33] and RoboTHOR [27]) are built in the Unity game engine and focus on agent-object interactions, object state changes, and accurate physics simulation. Unlike AI2-THOR, Habitat [101] provides scenes constructed from 3D scans of houses, however, objects and scenes are not interactable. A more recent version, Habitat 2.0 [109], introduces object interactions at the expense of being limited to one floorplan and synthetic scenes. iGibson [104] includes photo-realistic scenes, but with limited interactions such as pushing. iGibson 2.0 [69] extends iGibson by focusing on household tasks and object state changes in synthetic scenes and includes a virtual reality interface. ThreeDWorld [37] targets high-fidelity physics simulation such as liquid and deformable object simulation. VirtualHome [92] is designed for simulating human activities via programs. RLBench [53], RoboSuite [133] and Sapien [124] target fine-grained manipulation. The main advantage of ProcTHOR is that we can generate a diverse set of interactive scenes procedurally, enabling studies of data augmentation and large-scale training in the context of Embodied AI.

具身人工智能平台 。 过去几年里发展出了多种具身人工智能平台 [63, 101, 104, 124, 37, 121]。这些平台瞄准不同的设计目标。AI2-THOR [63] 及其变体（ManipulaTHOR [33] 和 RoboTHOR [27]）建立在 Unity 游戏引擎上，关注智能体与物体的交互、物体状态变化以及精确的物理仿真。与 AI2-THOR 不同，Habitat [101] 提供由房屋三维扫描构造的场景，但物体和场景不可交互。更新的 Habitat 2.0 [109] 引入了物体交互，代价是局限于一种平面图和合成场景。iGibson [104] 包含照片级真实感场景，但交互有限，例如推动。iGibson 2.0 [69] 在合成场景中把重点放在家务任务和物体状态变化上，并包含虚拟现实界面。ThreeDWorld [37] 面向高保真物理仿真，例如液体和可变形物体仿真。VirtualHome [92] 设计用来通过程序模拟人类活动。RLBench [53]、RoboSuite [133] 和 Sapien [124] 面向细粒度操作。ProcTHOR 的主要优势是我们可以程序化生成多样的 可交互 场景，从而能够在具身人工智能的语境中研究数据增强和大规模训练。

Large-scale datasets. Large-scale datasets have resulted in major breakthroughs in different domains such as image classification [28, 67], vision and language [18, 111], 3D understanding [14, 125], autonomous driving [11, 107], and robotic object manipulation [91, 82]. However, there are not many interactive large-scale datasets for Embodied AI research. ProcTHOR includes interactive houses generated procedurally. Hence, there are an arbitrarily large number of scenes in the framework. The closest works to ours are [94, 90, 72]. HM3D [94] is a recent framework that includes 1,000 scenes generated using 3D scans of real environments. ProcTHOR has a number of key distinctions: (1) unlike HM3D which includes static scenes, the scenes in ProcTHOR are interactive i.e., objects can move and change state, the lighting and texture of objects can change, and a physics engine determines the future states of the scenes; (2) it is challenging to scale up HM3D as it requires scanning a house and cleaning up the data, while we can procedurally generate more houses; (3) HM3D can be used only for navigation tasks (as there is no physics simulation and object interaction), while ProcTHOR can be used for tasks other than navigation. OpenRooms [72] is similar to HM3D in terms of the source of the data (3D scans) and dataset size. However, OpenRooms is interactive. OpenRooms is also confined to the set of scanned houses, and it takes a significant amount of time to annotate a new scene (e.g., labeling materials for one object takes 1 minute), while ProcTHOR does not suffer from these issues. Megaverse [90] is another large-scale Embodied AI platform that includes procedurally generated environments. Although it is impressive in terms of simulation speed, it includes only game-like environments with a simplified appearance. In contrast, ProcTHOR mimics real-world houses in terms of the complexity of appearance, physics, and object interactions.

大规模数据集 。 大规模数据集已经在图像分类 [28, 67]、视觉与语言 [18, 111]、三维理解 [14, 125]、自动驾驶 [11, 107] 和机器人物体操作 [91, 82] 等不同领域带来重大突破。不过，面向具身人工智能研究的可交互大规模数据集并不多。ProcTHOR 包含程序化生成的可交互房屋，因此框架中的场景数量可以任意大。与我们最接近的工作是 [94, 90, 72]。HM3D [94] 是一个近期框架，包含用真实环境三维扫描生成的 1,000 个场景。ProcTHOR 有几处关键不同：(1) 与包含静态场景的 HM3D 不同，ProcTHOR 中的场景是可交互的，也就是说物体可以移动并改变状态，物体的光照和纹理可以改变，并由物理引擎决定场景的未来状态；(2) 扩大 HM3D 很有挑战，因为它需要扫描房屋并清理数据，而我们可以程序化生成更多房子；(3) HM3D 只能用于导航任务（因为没有物理仿真和物体交互），而 ProcTHOR 还可以用于导航之外的任务。OpenRooms [72] 在数据来源（三维扫描）和数据集规模上与 HM3D 类似，不过 OpenRooms 是可交互的。OpenRooms 同样局限于已扫描的房屋集合，标注一个新场景要花大量时间（例如为一个物体标注材质需要 1 分钟），而 ProcTHOR 没有这些问题。Megaverse [90] 是另一个包含程序化生成环境的大规模具身人工智能平台。尽管它在仿真速度上令人印象深刻，却只包含外观简化的游戏式环境。相比之下，ProcTHOR 在外观复杂度、物理和物体交互上模仿真实世界的房屋。

Scene generation. Indoor scene synthesis has been studied extensively in computer vision and graphics communities. [16, 17, 13] address generating 3D scenes from text descriptions. [120, 47, 85] learn to generate house floorplans. [98, 129, 20, 113, 56, 70] use generative models for indoor scene generation. [131, 100] propose potential objects for a query location in an indoor scene. Others have used procedural generation [57, 31] and unsupervised learning [29] to synthesize grid-world environments for AI. ProcTHOR is specifically designed for Embodied AI research in the sense that (1) all scenes are interactive and physics-enabled, and the placement of objects respects the physics of the world (e.g., there are no two objects that clip through each other), (2) there are various forms of scene augmentation such as randomization of object placements while following certain commonsense rules, variation in the appearance of objects and structures, and variation in lighting.

场景生成 。 室内场景合成已在计算机视觉和图形学社区中得到广泛研究。[16, 17, 13] 研究从文本描述生成三维场景。[120, 47, 85] 学习生成房屋平面图。[98, 129, 20, 113, 56, 70] 用生成模型做室内场景生成。[131, 100] 为室内场景中的查询位置提出可能的物体。也有人用程序化生成 [57, 31] 和无监督学习 [29] 为人工智能合成格子世界环境。ProcTHOR 专门为具身人工智能研究设计，体现在：(1) 所有场景都可交互并带有物理，物体摆放尊重世界的物理（例如不会有两个物体相互穿模）；(2) 有多种形式的场景增强，例如在遵循某些常识规则的同时随机化物体摆放、改变物体和结构的外观，以及改变光照。

3 ProcTHOR

ProcTHOR is a framework to procedurally generate E-AI environments. It extends AI2-THOR and, thereby, inherits AI2-THOR’s large asset library, robotic agents, and accurate physics simulation. Just as in scenes painstakingly created by designers in AI2-THOR, environments in ProcTHOR are fully interactive and support navigation, object manipulation, and multi-agent interaction.

ProcTHOR 是一套程序化生成具身人工智能环境的框架。它扩展了 AI2-THOR，因而继承了 AI2-THOR 的大型资产库、机器人智能体和精确物理仿真。正如 AI2-THOR 中由设计师精心搭建的场景一样，ProcTHOR 中的环境完全可交互，并支持导航、物体操作和多智能体交互。

Figure 2: Procedurally generating a house using ProcTHOR.

图 2：用 ProcTHOR 程序化生成一座房子。

Fig. 2 shows a high-level schematic of the procedure used by ProcTHOR to generate a scene. Given a room specification (e.g. house with 1 bedroom + 1 bathroom), ProcTHOR uses multi-stage conditional sampling to, iteratively, generate a floor plan, create an external wall structure, sample lighting, and doors, then sample assets including large, small and wall objects, pick colors and textures, and determine appropriate placements for assets within the scene. We refer the reader to the appendix for details regarding our procedural generation and sampling mechanism, but highlight five key characteristics of ProcTHOR: Diversity, Interactivity, Customizability, Scale, and Efficiency .

图 2 给出了 ProcTHOR 用来生成场景的流程的高层次示意。给定房间规格（ 例如 一座带 1 间卧室和 1 间卫生间的房子），ProcTHOR 使用多阶段条件采样，依次生成平面图、创建外墙结构、采样光照和门，再采样包括大件、小件和墙面物体在内的资产，选择颜色和纹理，并确定资产在场景中的合适摆放。关于程序化生成和采样机制的细节请参阅附录，这里我们强调 ProcTHOR 的五个关键性质： 多样性 、 交互性 、 可定制性 、 规模 和 效率 。

Diversity. ProcTHOR enables the creation of rich and diverse environments. Mirroring the success of pre-training models with diverse data in the vision and NLP domains, we demonstrate the utility of this diversity on several E-AI tasks. Scenes in ProcTHOR exhibit diversity across several facets:

多样性 。 ProcTHOR 能够创建丰富而多样的环境。呼应视觉和自然语言处理领域用多样数据做预训练的成功，我们在若干具身任务上展示了这种多样性的用处。ProcTHOR 中的场景在多个方面展现多样性：

Diversity of floor plans. Given a room specification, we first employ iterative boundary cutting to obtain an external scene layout (that can range from a simple rectangle to a complex polygon). The recursive layout generation algorithm by Lopes et al. [73] is then used to divide the scene into the desired rooms. Finally, we determine connectivity between rooms using a set of user-defined constraints. These procedures result in natural room layouts (e.g., bedrooms are often connected to adjoining bathrooms via a door, bathrooms more often have a single entrance, etc). As exemplified in Fig. 3, ProcTHOR generates hugely diverse floor plans using this procedure.

平面图的多样性。 给定房间规格，我们首先用迭代边界切割得到外部场景轮廓（可以从简单矩形到复杂多边形）。随后用 Lopes 等人 [73] 的递归布局生成算法把场景划分成所需房间。最后，我们用一组用户定义的约束确定房间之间的连通。这些流程会得到自然的房间布局（例如卧室常常通过一扇门与相邻卫生间相连，卫生间更常只有一个入口，等等）。如图 3 所示，ProcTHOR 用这套流程生成极为多样的平面图。

Figure 3: Floorplan diversity. Examples showing the diversity of the generated floorplans. Rooms in the house are colored by Bedroom, Bathroom, Kitchen, and Living Room.

图 3： 平面图多样性。展示所生成平面图多样性的若干例子。房屋中的房间按颜色区分：卧室、卫生间、厨房和客厅 。

Diversity of assets. ProcTHOR populates scenes with small and large assets from its database of 1633 household assets across 108 categories (examples in Fig. 4). While many assets are inherited from AI2-THOR, we also introduce new assets such as windows, doors, and countertops, hand-designed by 3D graphic designers. Asset instances are split into train/val/test subsets and are interactable, i.e. objects can be picked and placed within the scenes, some objects have multiple states (e.g. a light can be on or off) and several objects consists of parts with rigid body motions (e.g. door on a microwave).

资产的多样性。 ProcTHOR 从包含 108 类、共 1633 个家居资产的数据库中取出大件和小件资产来填充场景（例子见图 4）。许多资产继承自 AI2-THOR，我们也引入了由 3D 图形设计师手工设计的新资产，例如窗户、门和台面。资产实例被划分到训练/验证/测试子集，并且可交互，也就是说物体可以在场景中被拿起和放下，有些物体有多种状态（ 例如 一盏灯可以开或关），若干物体还包含带刚体运动的部件（ 例如 微波炉的门）。

\cdots

Figure 4: Object diversity. A subset of instances for four object categories.

图 4： 物体多样性。四个物体类别中的一部分实例 。

Diversity of materials. Walls can have two kinds of materials - one of 40 solid (and popular) colors or one of 122 wall textures such as brick and tile. We also provide 55 floor materials. The ceiling material for the entire house is sampled from the set of wall materials. ProcTHOR also provides the ability to randomize materials of objects. Materials are only randomized within categories, which ensures objects still look and behave like the class they represent.

材质的多样性。 墙壁可以有两类材质：40 种纯色（且常见）颜色之一，或砖块和瓷砖等 122 种墙壁纹理之一。我们还提供 55 种地面材质。整座房子的天花板材质从墙壁材质集合中采样。ProcTHOR 也提供随机化物体材质的能力。材质只在类别内部随机化，这确保物体看起来仍然像、也仍然表现为它所代表的类别。

Figure 5: Material augmentation. Different materials for objects and structural elements like walls and floors.

图 5： 材质增强。物体以及墙壁和地面这类结构元素的不同材质 。

Diversity of object placements. Asset categories have several soft annotations that help place them realistically within a house. These include room assignments (e.g. couch in a living room but not a bathroom) and location assignments (e.g. fridge along a wall, TV not on the floor). We also develop the notion of a Semantic Asset Group (SAG) - groups of assets that typically co-occur (e.g. dining table with four chairs) and thus must be sampled and placed using dependent sampling. Given a layout, individual assets and SAGs that lie on the floor are sampled and placed iteratively, ensuring that rooms continue to have adequate floor space for agents to navigate and manipulate objects. Then wall objects such as windows and paintings get placed, and finally, surface objects (ones found on top of other assets) are placed (e.g. cups on the kitchen counter). This sampling allows for a large and diverse set of object choices and placements within any layout. Fig. 6 shows such variations.

物体摆放的多样性。 资产类别带有若干软标注，帮助把它们真实地摆进房子里。这些标注包括房间分配（ 例如 沙发在客厅而不在卫生间）和位置分配（ 例如 冰箱靠墙，电视不在地上）。我们还提出语义资产组（Semantic Asset Group，SAG）的概念——通常一起出现的资产组（ 例如 一张餐桌配四把椅子），因此必须用相关采样来采样和摆放。给定布局后，位于地面上的单个资产和语义资产组被迭代采样并摆放，同时确保房间继续保有足够的地面空间，供智能体导航和操作物体。随后摆放窗户和画作等墙面物体，最后摆放表面物体（出现在其他资产之上的物体， 例如 厨房台面上的杯子）。这种采样允许在任何布局中做出大量多样的物体选择和摆放。图 6 展示了这类变化。

Figure 6: Object placement. Four examples of object placement within the same room layout.

图 6： 物体摆放。同一房间布局下的四种摆放例子 。

Diversity of lighting. ProcTHOR supports a single directional light (analogous to the sun) and several point lights (analogous to lightbulbs). Varying the color, intensity, and placement of these sources allows us to simulate different artificial lighting, typically observed in houses, and also at different times of the day. Lighting has a significant effect on the rendered images as seen in Fig. 7.

光照的多样性。 ProcTHOR 支持一盏方向光（类似于太阳）和若干点光源（类似于灯泡）。改变这些光源的颜色、强度和位置，使我们能够模拟房屋中常见的不同人工照明，以及一天中的不同时刻。如图 7 所示，光照对渲染图像有显著影响。

Figure 7: Lighting variation. Morning, dusk, and night lighting for an example scene.

图 7： 光照变化。同一示例场景在早晨、黄昏和夜晚的光照 。

Interactivity. A key property of ProcTHOR is the ability to interact with objects to change their location or state (Fig. 8). This capability is fundamental to many Embodied AI tasks. Datasets like HM3D [94] that are created from static 3D scans do not possess this capability. ProcTHOR supports agents with arms capable of manipulating objects and interacting with each other.

交互性 。 ProcTHOR 的一个关键性质，是能够与物体交互以改变它们的位置或状态（图 8）。这种能力对许多具身人工智能任务至关重要。像 HM3D [94] 这样由静态三维扫描创建的数据集不具备这种能力。ProcTHOR 支持带手臂的智能体，它们能够操作物体并彼此交互。

Figure 8: Interactivity. Object states can change (e.g., the laptop or the lamp in the left panel), and the agents can interact with objects and other agents (middle and right panels).

图 8： 交互性。物体状态可以改变（例如左图中的笔记本或台灯），智能体也可以与物体以及其他智能体交互（中图和右图） 。

Customizability. ProcTHOR supports many room, asset, material, and lighting specifications. With a few simple lines of specification, one can easily generate customized environments of interest. Fig. 9 shows examples of such varied scenes (classroom, library, and office).

可定制性 。 ProcTHOR 支持许多房间、资产、材质和光照规格。只需几行简单的规格，就可以很容易地生成感兴趣的定制环境。图 9 展示了这类变化场景的例子（教室、图书馆和办公室）。

Figure 9: Customizability. ProcTHOR can be used to construct custom scene types such as classrooms, libraries, and offices.

图 9： 可定制性。ProcTHOR 可以用来构造教室、图书馆和办公室这类定制场景类型 。

Scale and Efficiency. ProcTHOR currently uses 16 different scene specifications to seed the scene generation process. These can result in over 100 billion layouts. ProcTHOR uses 18 different Semantic Asset groups and 1633 assets. These can result in roughly 20 million unique asset groups. Each of these assets can be placed in numerous locations. In addition, each house gets scaled and uses a variety of lighting. This diversity of layouts, assets, materials, placements, and lighting enables the generation of arbitrarily large sets of houses - either statically generated and stored as a dataset or dynamically generated at each iteration of training. Scenes are efficiently represented in a JSON specification and are loaded into AI2-THOR at runtime, making the memory overhead of storing houses incredibly efficient. Moreover, the scene generation process is fully automatic and fast and ProcTHOR provides high framerates for training E-AI models (see Sec. 4 for details).

规模与效率 。 ProcTHOR 目前使用 16 种不同的场景规格来播种场景生成过程。它们可以产生超过 1000 亿种布局。ProcTHOR 使用 18 种不同的语义资产组和 1633 个资产。它们大约可以产生 2000 万个独特的资产组。这些资产中的每一个又可以放在许多位置。此外，每座房子还会被缩放，并使用多种光照。布局、资产、材质、摆放和光照上的这种多样性，使生成 任意大规模 的房屋集合成为可能——既可以静态生成并存成数据集，也可以在每次训练迭代时动态生成。场景被高效地表示成 JSON 规格，并在运行时载入 AI2-THOR，使得存储房屋的内存开销非常小。而且，场景生成过程完全自动且很快，ProcTHOR 为训练具身模型提供了很高的帧率（细节见第 4 节）。

4 ProcTHOR -10K

Figure 10: ProcTHOR -10K statistics. Left: distribution of the number of objects in each room; Middle: distribution of the area of each house, bucketed into small, medium, and large houses; Right: bar plot showing the distribution over the number of rooms that make up each house.

图 10：**ProcTHOR-10K 统计。*左：*每个房间中物体数量的分布；*中：*每座房子面积的分布，划分为小、中、大三类房屋； 右： 组成每座房子的房间数分布柱状图 。

Figure 11: Example scenes in ProcTHOR-10K with top-down and an egocentric view.

图 11： ProcTHOR-10K 中的示例场景，包含俯视图和自我中心视角 。

We demonstrate the power and potential of ProcTHOR using a sampled set of 10,000 fully interactive houses obtained by the procedural generation process described in Section 3 - which we label ProcTHOR-10K. An additional set of 1,000 validation and 1,000 testing houses are available for evaluation. Asset splits across train/val/test are detailed in the Appendix. All houses are fully navigable, allowing an agent to traverse through each room without any interaction. In terms of scale, ProcTHOR-10K is one of the largest sets of interactive home environments for Embodied AI - as a comparison, AI2-iTHOR [63] includes 120 scenes, RoboTHOR [27] has 89 scenes, iGibson [104] has 15 scenes, Habitat Matterport 3D [94] has 1,000 static (non-interactive) scenes, and Habitat 2.0 [109] has 105 scene layouts. Scaling beyond 10K houses is straightforward and inexpensive. This set of 10K houses was generated in 1 hour on a local workstation with 4 NVIDIA RTX A5000 GPUs. Fig. 11 shows examples of ego-centric and top-down views of houses present in ProcTHOR-10K.

我们用第 3 节所述程序化生成过程采样出的 1 万套完全可交互房屋，来展示 ProcTHOR 的能力和潜力——我们把它称为 ProcTHOR-10K。另外还有 1,000 套验证房屋和 1,000 套测试房屋可用于评估。训练/验证/测试上的资产划分详见附录。所有房屋都完全可导航，允许智能体在不做任何交互的情况下穿过每一个房间。就规模而言，ProcTHOR-10K 是面向具身人工智能的最大可交互家居环境集合之一——作为对照，AI2-iTHOR [63] 包含 120 个场景，RoboTHOR [27] 有 89 个场景，iGibson [104] 有 15 个场景，Habitat Matterport 3D [94] 有 1,000 个静态（不可交互）场景，Habitat 2.0 [109] 有 105 种场景布局。把规模扩大到超过 1 万套房子既直接又便宜。这 1 万套房子是在一台配有 4 块 NVIDIA RTX A5000 GPU 的本地工作站上用 1 小时生成的。图 11 展示了 ProcTHOR-10K 中房屋的自我中心视角和俯视例子。

Scene statistics. Houses in ProcTHOR-10K are generated using 16 different room specifications. An example room spec is: A house with 1 bedroom connected to 1 bathroom, 1 kitchen, and 1 living room and is visualized in Fig. 2. Houses in this dataset have as few as 1 room and as many as 10. Fig. 10 shows the distribution of areas (middle) and the number of rooms (right) of these generated houses. Our use of room specifications enables us to change the distribution of the size and complexity of houses fairly easily. ProcTHOR-10K encompasses a wider spectrum of scenes than AI2-iTHOR [63] and RoboTHOR [27] (biased towards room-sized scenes) and Gibson [123] and HM3D [94] (biased towards large houses).

场景统计 。 ProcTHOR-10K 中的房屋用 16 种不同的房间规格生成。一个房间规格的例子是： 一座房子有 1 间与 1 间卫生间相连的卧室、1 间厨房和 1 间客厅 ，可视化见图 2。该数据集中的房屋少则 1 个房间，多则 10 个。图 10 展示了这些生成房屋的面积分布（中）和房间数分布（右）。使用房间规格使我们能够相当容易地改变房屋大小和复杂度的分布。与偏向房间尺度场景的 AI2-iTHOR [63] 和 RoboTHOR [27]，以及偏向大房子的 Gibson [123] 和 HM3D [94] 相比，ProcTHOR-10K 覆盖了更宽的场景谱。

Rooms in each of these houses contain objects from 95 different categories including common household objects such as fridges, countertops, beds, toilets, and house plants, and structure objects such as doorways and windows. Fig. 10 (left) shows the distribution of the number of objects per room per house, which shows that houses in ProcTHOR-10K are well populated. They also contain objects sampled via 18 different Semantic Asset groups. Examples of Semantic asset groups (SAG) are a Dining Table with 4 Chairs or Bed with 2 Pillows . Given our large asset library and SAGs, we can create 19.3 million combinations of group instantiations.

这些房子的各个房间包含来自 95 个不同类别的物体，既有冰箱、台面、床、马桶和室内植物这类常见家居物体，也有门洞和窗户这类结构物体。图 10（左）展示了每座房子每个房间的物体数量分布，说明 ProcTHOR-10K 中的房子被填充得很充分。它们还包含通过 18 种不同语义资产组采样得到的物体。语义资产组（SAG）的例子有 一张餐桌配四把椅子 或 一张床配两个枕头 。凭借我们的大型资产库和语义资产组，我们可以创建 1930 万种组合实例。

\pm

	Navigation FPS			Environment Query FPS
Compute	Small	Large	Small	Large	Small	Large
8 GPUs	8,599 $\pm$ 359	3,208 $\pm$ 127	6,488 $\pm$ 250	2,861 $\pm$ 107	480,205 $\pm$ 19,684	433,587 $\pm$ 18,729
1 GPU	1,427 $\pm$ 74	6,280 $\pm$ 40	1,265 $\pm$ 71	597 $\pm$ 37	160,622 $\pm$ 2,846	157,567 $\pm$ 2,689
1 Process	240 $\pm$ 69	115 $\pm$ 19	180 $\pm$ 42	93 $\pm$ 15	14,825 $\pm$ 199	14,916 $\pm$ 186

\pm

	导航 FPS			环境查询 FPS
计算资源	Small	Large	Small	Large	Small	Large
8 块 GPU	8,599 $\pm$ 359	3,208 $\pm$ 127	6,488 $\pm$ 250	2,861 $\pm$ 107	480,205 $\pm$ 19,684	433,587 $\pm$ 18,729
1 块 GPU	1,427 $\pm$ 74	6,280 $\pm$ 40	1,265 $\pm$ 71	597 $\pm$ 37	160,622 $\pm$ 2,846	157,567 $\pm$ 2,689
1 个进程	240 $\pm$ 69	115 $\pm$ 19	180 $\pm$ 42	93 $\pm$ 15	14,825 $\pm$ 199	14,916 $\pm$ 186

Rendering speed. A crucial requirement for large-scale training is high rendering speed since the training algorithms require millions of iterations to converge. Table 1 shows these statistics. Experiments were run on a server with 8 NVIDIA Quadro RTX 8000 GPUs. For the 1 GPU experiments, we use 15 processes and for the 8 GPU experiments, we use 120 processes, evenly distributed across the GPUs. ProcTHOR provides framerates comparable to iTHOR and RoboTHOR environments in spite of having larger houses (See Appendix for details), rendering it fast enough for training large models for hundreds of millions of steps in a reasonable amount of time.

渲染速度 。 大规模训练的一个关键要求是高渲染速度，因为训练算法需要数百万次迭代才能收敛。表 1 给出了这些统计。实验在一台配有 8 块 NVIDIA Quadro RTX 8000 GPU 的服务器上运行。1 块 GPU 的实验使用 15 个进程，8 块 GPU 的实验使用 120 个进程，均匀分配到各 GPU 上。尽管房屋更大，ProcTHOR 仍提供与 iTHOR 和 RoboTHOR 环境相当的帧率（细节见附录），因此足够快，可以在合理时间内把大模型训练数亿步。

5 Experiments

5 实验

Tasks. We now present results for models pre-trained on ProcTHOR-10K on several navigation and manipulation benchmarks to demonstrate the benefits of large-scale training. We consider ObjectNav (navigation towards a specific object category) in ProcTHOR, ArchitecTHOR, RoboTHOR [27], HM3D [94], and AI2-iTHOR [63]. We also consider two manipulation-based tasks: ArmPointNav [33] and 1-phase Room Rearrangement [115]. In ArmPointNav, the agent moves an object using a robotic arm from a source location to a destination location specified in the 3D coordinate frame. In Room Rearrangement, the goal is to move objects or change their state to reach a target scene state.

任务 。 我们现在给出在 ProcTHOR-10K 上预训练的模型在若干导航和操作基准上的结果，以展示大规模训练的好处。我们考虑在 ProcTHOR、ArchitecTHOR、RoboTHOR [27]、HM3D [94] 和 AI2-iTHOR [63] 上的 ObjectNav（朝特定物体类别导航）。我们也考虑两个基于操作的任务：ArmPointNav [33] 和单阶段房间重排 [115]。在 ArmPointNav 中，智能体用机械臂把一个物体从源位置移到在三维坐标系中指定的目标位置。在房间重排中，目标是移动物体或改变它们的状态，以到达目标场景状态。

Models. Our models for all tasks consist of a CNN to encode visual information and a GRU to capture temporal information. We deliberately use a simple architecture across all tasks to show the benefits of large-scale training. Our ObjectNav and Rearrangement models use the CLIP-based architectures of [58]. Our ArmPointNav model uses a simpler visual encoder with 3 convolutional layers; we found this more effective than the CLIP encoder. All models are trained with the AllenAct [116] framework, see the Appendix for training details.

模型 。 我们所有任务的模型都由一个用来编码视觉信息的 CNN 和一个用来捕捉时间信息的 GRU 组成。我们有意在全部任务上使用简单结构，以展示大规模训练的好处。我们的 ObjectNav 和重排模型使用 [58] 中基于 CLIP 的结构。我们的 ArmPointNav 模型使用带 3 个卷积层的更简单视觉编码器；我们发现这比 CLIP 编码器更有效。所有模型都用 AllenAct [116] 框架训练，训练细节见附录。

Results. We present results in two settings: zero-shot and after fine-tuning on the training scenes provided by the downstream benchmark. Zero-shot experiments show us how well models trained on ProcTHOR generalize to new environments, whereas fine-tuning experiments tell us if representations learned from ProcTHOR can serve as a good initialization for quick tuning. For all experiments, we use only RGB images (no depth and other modalities is used).

结果 。 我们在两种设定下给出结果：零样本，以及在下游基准提供的训练场景上微调之后。零样本实验告诉我们在 ProcTHOR 上训练的模型对新环境泛化得有多好，而微调实验则告诉我们从 ProcTHOR 学到的表征能否作为快速微调的良好初始化。对所有实验，我们只使用 RGB 图像（不使用深度和其他模态）。

Zero-shot is particularly challenging since other environments have different appearance statistics, layouts, and object distributions compared to ProcTHOR. ArchitecTHOR and AI2-iTHOR [63] are high-fidelity artist-designed scenes with high-quality shadows and lighting. HM3D is constructed from 3D scans of houses which can differ quite a bit from synthetic environments. RoboTHOR [27] houses use wall panels and floors with very specific textures.

零样本尤其具有挑战性，因为其他环境在外观统计、布局和物体分布上与 ProcTHOR 不同。ArchitecTHOR 和 AI2-iTHOR [63] 是高保真的美术设计场景，带有高质量阴影和光照。HM3D 由房屋的三维扫描构造而成，可能与合成环境有相当大的差别。RoboTHOR [27] 的房屋使用非常特定纹理的墙板和地面。

Zero-shot transfer results. Models trained only on ProcTHOR and evaluated zero-shot outperform previous SoTA models on 3 benchmarks (refer to zero-shot rows of Table 2). These are very strong results since the models generalize to not only unseen objects and scenes, but also different appearance and layout statistics.

零样本迁移结果。 只在 ProcTHOR 上训练并做零样本评估的模型，在 3 个基准上超过了此前的最优模型（见表 2 中的 零样本 各行）。这些结果非常强，因为模型不仅泛化到未见物体和场景，还泛化到不同的外观和布局统计。

Fine-tuning results. Further fine-tuning of the model using each benchmark’s training data, achieves state-of-the-art results on all benchmarks (refer to fine-tune rows of Table 2). Notably, our model is ranked first on three public leaderboards as of 10am PT, June 14th 2022: Habitat 2022 ObjectNav challenge, AI2-THOR Rearrangement 2022 challenge, and RoboTHOR ObjectNav challenge. It should be noted that our model achieves these results using a very simple architecture and only RGB images. Other techniques typically use more complex architectures that include mapping or visual odometry modules and use additional perception sensors such as depth images.

微调结果。 再用每个基准自己的训练数据进一步微调模型，在全部基准上都取得了最优结果（见表 2 中的 微调 各行）。值得注意的是，截至 2022 年 6 月 14 日上午 10 点（太平洋时间），我们的模型在三个公开排行榜上排名第一：Habitat 2022 ObjectNav 挑战、AI2-THOR Rearrangement 2022 挑战，以及 RoboTHOR ObjectNav 挑战。应当指出，我们的模型是用非常简单的结构、并且只使用 RGB 图像取得这些结果的。其他方法通常使用包含建图或视觉里程计模块的更复杂结构，并使用深度图像等额外感知传感器。

Scale ablation. To evaluate the effect of scale we train the models on 10, 100, 1,000, and 10,000 houses. For this experiment, we do not use any material augmentations. As shown in Table 3, the performance improves as we use more houses for training, demonstrating the benefits of large-scale data for Embodied AI tasks.

规模消融 。 为评估规模的影响，我们在 10、100、1,000 和 10,000 套房子上训练模型。该实验不使用任何材质增强。如表 3 所示，用于训练的房子越多，表现越好，说明了大规模数据对具身人工智能任务的好处。

Task Benchmark Method Metrics Success SPL ObjectNav RoboTHOR Challenge EmbCLIP [58] a 47.0% 0.200 ProcTHOR 0-shot 55.0% 0.237 ProcTHOR + fine-tune 65.2 % 0.288 Success SPL MLNLC c 52.0% 0.280 Habitat Challenge FusionNav (AIRI) c 54.0% 0.270 ObjectNav (2022) ProcTHOR 0-shot 9.00% 0.055 HM3D-Semantics ProcTHOR + fine-tune 53.0% 0.270 ProcTHOR + Large d + 0-shot 13.2% 0.077 ProcTHOR + Large d + fine-tune 54.4 % 0.318 Success SPL ObjectNav AI2-iTHOR EmbCLIP [58] b 68.4% 0.516 ProcTHOR 0-shot 75.7% 0.644 ProcTHOR + fine-tune 77.5 % 0.621 Success SPL ObjectNav ArchitecTHOR EmbCLIP [58] b 18.5% 0.118 ProcTHOR 31.4 % 0.195 Rearrangement Success % Fixed Strict AI2-THOR Challenge EmbCLIP [58] 7.10% 0.190 1-phase (2022) ProcTHOR 0-shot 3.80% 0.156 ProcTHOR + fine-tune 7.40 % 0.245 Success % PickUp SR ArmPointNav ManipulaTHOR iTHOR-SimpleConv [33] e 29.2% 73.4 ProcTHOR 0-shot 37.9% 74.8 Table 2: Results for models trained on ProcTHOR and evaluated 0-shot and with fine-tuning on several E-AI benchmarks. For each benchmark we also compare to the relevant baselines (previous SoTA or leaderboard submissions where applicable). a EmbCLIP [58] trained on RoboTHOR, b EmbCLIP [58] trained on AI2-iTHOR, c submission on the Habitat 2022 ObjectNav leaderboard [79]. d For HM3D we present results when pretraining using the standard EmbCLIP architecture (which uses a CLIP-pretrained ResNet50 backbone) as well as with a “Large” model which uses a larger CLIP backbone CNN as well as a wider RNN, see supplement for details. e uses the model from [33] but retrains on the complete iTHOR data with RGB inputs. 0-shot results, whereby models are pre-trained on ProcTHOR-10K and do not use any training data from the benchmark that they are evaluated on.

Task	Benchmark	Method	Metrics
			Success	SPL
ObjectNav	RoboTHOR Challenge	EmbCLIP [58]a	47.0%	0.200
		ProcTHOR 0-shot	55.0%	0.237
		ProcTHOR + fine-tune	65.2%	0.288
			Success	SPL
		MLNLCc	52.0%	0.280
	Habitat Challenge	FusionNav (AIRI)c	54.0%	0.270
ObjectNav	(2022)	ProcTHOR 0-shot	9.00%	0.055
	HM3D-Semantics	ProcTHOR + fine-tune	53.0%	0.270
		ProcTHOR + Larged + 0-shot	13.2%	0.077
		ProcTHOR + Larged + fine-tune	54.4%	0.318
			Success	SPL
ObjectNav	AI2-iTHOR	EmbCLIP [58]b	68.4%	0.516
		ProcTHOR 0-shot	75.7%	0.644
		ProcTHOR + fine-tune	77.5%	0.621
			Success	SPL
ObjectNav	ArchitecTHOR	EmbCLIP [58]b	18.5%	0.118
		ProcTHOR	31.4%	0.195
Rearrangement			Success	% Fixed Strict
AI2-THOR Challenge	EmbCLIP [58]	7.10%	0.190
1-phase (2022)	ProcTHOR 0-shot	3.80%	0.156
	ProcTHOR + fine-tune	7.40%	0.245
			Success	% PickUp SR
ArmPointNav	ManipulaTHOR	iTHOR-SimpleConv [33]e	29.2%	73.4
		ProcTHOR 0-shot	37.9%	74.8

任务 基准 方法 指标 成功 SPL ObjectNav RoboTHOR Challenge EmbCLIP [58] a 47.0% 0.200 ProcTHOR 零样本 55.0% 0.237 ProcTHOR + 微调 65.2 % 0.288 成功 SPL MLNLC c 52.0% 0.280 Habitat Challenge FusionNav (AIRI) c 54.0% 0.270 ObjectNav (2022) ProcTHOR 零样本 9.00% 0.055 HM3D-Semantics ProcTHOR + 微调 53.0% 0.270 ProcTHOR + Large d + 零样本 13.2% 0.077 ProcTHOR + Large d + 微调 54.4 % 0.318 成功 SPL ObjectNav AI2-iTHOR EmbCLIP [58] b 68.4% 0.516 ProcTHOR 零样本 75.7% 0.644 ProcTHOR + 微调 77.5 % 0.621 成功 SPL ObjectNav ArchitecTHOR EmbCLIP [58] b 18.5% 0.118 ProcTHOR 31.4 % 0.195 重排 成功 % Fixed Strict AI2-THOR Challenge EmbCLIP [58] 7.10% 0.190 单阶段 (2022) ProcTHOR 零样本 3.80% 0.156 ProcTHOR + 微调 7.40 % 0.245 成功 % 抓取成功率 ArmPointNav ManipulaTHOR iTHOR-SimpleConv [33] e 29.2% 73.4 ProcTHOR 零样本 37.9% 74.8 表 2：在 ProcTHOR 上训练、并在若干具身基准上做零样本评估和微调评估的模型结果。对每个基准，我们还与相关基线比较（此前最优，或在适用时与排行榜提交比较）。 a 在 RoboTHOR 上训练的 EmbCLIP [58]， b 在 AI2-iTHOR 上训练的 EmbCLIP [58]， c Habitat 2022 ObjectNav 排行榜上的提交 [79]。 d 对 HM3D，我们同时给出使用标准 EmbCLIP 结构（使用 CLIP 预训练的 ResNet50 主干）预训练的结果，以及使用“Large”模型的结果，后者使用更大的 CLIP 主干 CNN 以及更宽的 RNN，细节见补充材料。 e 使用 [33] 中的模型，但在完整 iTHOR 数据上用 RGB 输入重新训练。 零样本结果指模型在 ProcTHOR-10K 上预训练，且不使用它们所评估的基准中的任何训练数据。

任务	基准	方法	指标
			成功	SPL
ObjectNav	RoboTHOR Challenge	EmbCLIP [58]a	47.0%	0.200
		ProcTHOR 零样本	55.0%	0.237
		ProcTHOR + 微调	65.2%	0.288
			成功	SPL
		MLNLCc	52.0%	0.280
	Habitat Challenge	FusionNav (AIRI)c	54.0%	0.270
ObjectNav	(2022)	ProcTHOR 零样本	9.00%	0.055
	HM3D-Semantics	ProcTHOR + 微调	53.0%	0.270
		ProcTHOR + Larged + 零样本	13.2%	0.077
		ProcTHOR + Larged + 微调	54.4%	0.318
			成功	SPL
ObjectNav	AI2-iTHOR	EmbCLIP [58]b	68.4%	0.516
		ProcTHOR 零样本	75.7%	0.644
		ProcTHOR + 微调	77.5%	0.621
			成功	SPL
ObjectNav	ArchitecTHOR	EmbCLIP [58]b	18.5%	0.118
		ProcTHOR	31.4%	0.195
重排			成功	% Fixed Strict
AI2-THOR Challenge	EmbCLIP [58]	7.10%	0.190
单阶段 (2022)	ProcTHOR 零样本	3.80%	0.156
	ProcTHOR + 微调	7.40%	0.245
			成功	% 抓取成功率
ArmPointNav	ManipulaTHOR	iTHOR-SimpleConv [33]e	29.2%	73.4
		ProcTHOR 零样本	37.9%	74.8

ArchitecTHOR Test RoboTHOR Test (0-Shot) HM3D Valid (0-Shot) AI2-iTHOR Test (0-Shot) # Houses SPL SR SPL SR SPL SR SPL SR 10 Houses 0.077 11.3% 0.040 8.53% 0.007 1.60% 0.249 28.7% 100 Houses 0.102 18.6% 0.076 20.9% 0.050 10.4% 0.352 42.0% 1,000 Houses 0.122 17.2% 0.157 33.1% 0.027 4.65% 0.456 53.0% 10,000 Houses 0.185 27.0% 0.210 44.5% 0.060 9.70% 0.554 64.9% ArchitecTHOR Test RoboTHOR Test (0-Shot) HM3D Valid (0-Shot) AI2-iTHOR Test (0-Shot) # Houses SPL SR SPL SR SPL SR SPL SR 10 Houses 0.077 11.3% 0.040 8.53% 0.007 1.60% 0.249 28.7% 100 Houses 0.102 18.6% 0.076 20.9% 0.050 10.4% 0.352 42.0% 1,000 Houses 0.122 17.2% 0.157 33.1% 0.027 4.65% 0.456 53.0% 10,000 Houses 0.185 27.0% 0.210 44.5% 0.060 9.70% 0.554 64.9% ArchitecTHOR Test RoboTHOR Test (0-Shot) HM3D Valid (0-Shot) AI2-iTHOR Test (0-Shot) Table 3: Ablation study to evaluate the effect of the number of training houses. Each model is trained to 80% success during training. Test performance increases with the number of training houses.

ArchitecTHOR 测试 RoboTHOR 测试（零样本） HM3D 验证（零样本） AI2-iTHOR 测试（零样本） 房屋数 SPL SR SPL SR SPL SR SPL SR 10 套房子 0.077 11.3% 0.040 8.53% 0.007 1.60% 0.249 28.7% 100 套房子 0.102 18.6% 0.076 20.9% 0.050 10.4% 0.352 42.0% 1,000 套房子 0.122 17.2% 0.157 33.1% 0.027 4.65% 0.456 53.0% 10,000 套房子 0.185 27.0% 0.210 44.5% 0.060 9.70% 0.554 64.9% 表 3：评估训练房屋数量影响的消融研究。每个模型都训练到训练成功率为 80%。测试表现随着训练房屋数量增加而提升。

6 Conclusion

6 结论

We propose ProcTHOR, a framework to procedurally generate arbitrarily large sets of interactive, physics-enabled houses for Embodied AI research. We pre-train simple models on 10,000 generated houses and show state-of-the-art results across 6 embodied navigation and manipulation benchmarks with strong 0-shot results, even outperforming prior state-of-the-art on 3 of these benchmarks.

我们提出 ProcTHOR，一套为具身人工智能研究程序化生成 任意大规模 可交互、带物理房屋集合的框架。我们在生成的 1 万套房子上预训练简单模型，并在 6 个具身导航和操作基准上展示最优结果，零样本结果也很强，甚至在其中 3 个基准上超过此前的最优水平。

Acknowledgements

致谢

We would like to thank the teams behind the open-source packages used in this project, including AI2-THOR [63], AllenAct [116], Habitat [101], Datasets [68], NumPy [45], PyTorch [88], Pandas [78], Wandb [8], Shapely [41], Hydra [126], SciPy [112], UMAP [77], NetworkX [44], EvalAI [127], TensorFlow [1], OpenAI Gym [9], Seaborn [114], PySAT [50], and Matplotlib [49].

我们感谢本项目中所用开源软件包背后的团队，包括 AI2-THOR [63]、AllenAct [116]、Habitat [101]、Datasets [68]、NumPy [45]、PyTorch [88]、Pandas [78]、Wandb [8]、Shapely [41]、Hydra [126]、SciPy [112]、UMAP [77]、NetworkX [44]、EvalAI [127]、TensorFlow [1]、OpenAI Gym [9]、Seaborn [114]、PySAT [50] 和 Matplotlib [49]。

References

参考文献

[11] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , 2020. [12] Carnegie Mellon University. Locobot: an open source low cost robot. http://www.locobot.org/ . [13] Angel Chang, Manolis Savva, and Christopher D Manning. Interactive learning of spatial knowledge for text to 3d scene generation. In ACL Workshop on Interactive Language Learning, Visualization, and Interfaces , 2014. [14] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. arXiv , 2015. [15] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. arXiv , 2015. [16] Angel X. Chang, Will Monroe, Manolis Savva, Christopher Potts, and Christopher D. Manning. Text to 3d scene generation with rich lexical grounding. In ACL , 2015. [17] Angel X. Chang, Manolis Savva, and Christopher D. Manning. Learning spatial knowledge for text to 3d scene generation. In EMNLP , 2014. [18] Soravit Changpinyo, Piyush Kumar Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , 2021. [19] Jorge L. Charco, Angel Domingo Sappa, Boris Xavier Vintimilla, and Henry O. Velesaca. Camera pose estimation in multi-view environments: From virtual scenarios to the real world. Image Vis. Comput. , 2021. [20] Siddhartha Chaudhuri, Daniel Ritchie, Kai Xu, and Hao Zhang. Learning generative models of 3d structures. In Eurographics , 2019.

[51] Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander Schwing. A cordial sync: Going beyond marginal policies for multi-agent embodied tasks. In ECCV , 2020. [52] Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexander G Schwing, and Aniruddha Kembhavi. Two body problem: Collaborative visual task completion. In CVPR , 2019. [53] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters , 2020. [54] Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters , 2020. [55] Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang. Learning adaptive language interfaces through decomposition. arXiv , 2020. [56] Mohammad Keshavarzi, Aakash Parikh, Xiyu Zhai, Melody Mao, Luisa Caldas, and Allen Y Yang. Scenegen: Generative contextual scene augmentation using scene graph priors. arXiv , 2020. [57] Ahmed Khalifa, Philip Bontrager, Sam Earle, and Julian Togelius. Pcgrl: Procedural content generation via reinforcement learning. In AIIDE , 2020. [58] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. In CVPR , 2021. [59] Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to simulate dynamic environments with gamegan. In CVPR , 2020. [60] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv , 2014.

[71] Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In CoRL , 2022. [72] Zhengqin Li, Ting Yu, Shen Sang, Sarah Wang, Mengcheng Song, Yuhan Liu, Yu-Ying Yeh, Rui Zhu, Nitesh B. Gundavarapu, Jia Shi, Sai Bi, Hong-Xing Yu, Zexiang Xu, Kalyan Sunkavalli, Milos Hasan, Ravi Ramamoorthi, and Manmohan Chandraker. Openrooms: An open framework for photorealistic indoor scene datasets. In CVPR , 2021. [73] Ricardo Lopes, Tim Tutenel, Ruben M Smelik, Klaas Jan De Kraker, and Rafael Bidarra. A constrained growth method for procedural floor plan generation. In Game-ON , 2010. [74] Haokuan Luo, Albert Yue, Zhang-Wei Hong, and Pulkit Agrawal. Stubborn: A strong baseline for indoor object navigation. arXiv , 2022. [75] Fernando Marson and Soraia Raupp Musse. Automatic real-time generation of floor plans based on squarified treemaps algorithm. International Journal of Computer Games Technology , 2010. [76] Cristina Mata, Nick Locascio, Mohammed Azeem Sheikh, Kenny Kihara, and Dan Fischetti. Standardsim: A synthetic dataset for retail environments. In ICIAP , 2022. [77] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software , 2018. [78] Wes McKinney et al. pandas: a foundational python library for data analysis and statistics. Python for high performance and scientific computing , 2011. [79] Meta AI. Habitat ObjectNav Challenge 2022. https://aihabitat.org/challenge/2022 . [80] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

[91] Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In ICRA , 2016. [92] Xavier Puig, Kevin Kyunghwan Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In CVPR , 2018. [93] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021. [94] Santhosh K. Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel Xuan Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv , 2021. [95] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML , 2021. [96] Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. In CVPR , 2022. [97] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In ICCV , 2021. [98] Daniel Ritchie, Kai Wang, and Yu-An Lin. Fast and flexible indoor scene synthesis via deep convolutional generative models. In CVPR , 2019. [99] Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS , 2011. [100] Manolis Savva, Angel X. Chang, and Maneesh Agrawala. Scenesuggest: Context-driven 3d scene design. arXiv , 2017.

[111] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl S. Ni, Douglas N. Poland, Damian Borth, and Li-Jia Li. Yfcc100m: the new data in multimedia research. Comm. of the ACM , 2016. [112] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods , 2020. [113] Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with transformers. In 3DV , 2021. [114] Michael L Waskom. Seaborn: statistical data visualization. Journal of Open Source Software , 2021. [115] Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In CVPR , 2021. [116] Luca Weihs, Jordi Salvador, Klemen Kotar, Unnat Jain, Kuo-Hao Zeng, Roozbeh Mottaghi, and Aniruddha Kembhavi. AllenAct: A framework for embodied AI research. arXiv , 2020. [117] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. In ICLR , 2019. [118] Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. In CVPR , 2019. [119] Qi Wu, Cheng-Ju Wu, Yixin Zhu, and Jungseock Joo. Communicative learning with natural gestures for embodied navigation agents with human-in-the-scene. In IROS , 2021. [120] Wenming Wu, Xiao-Ming Fu, Rui Tang, Yuhan Wang, Yu-Hao Qi, and Ligang Liu. Data-driven interior plan generation for residential buildings. ACM Trans. on Graphics , 2019.

[121] Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv , 2018. [122] Fei Xia, Chengshu Li, Roberto Mart’in-Mart’in, Or Litany, Alexander Toshev, and Silvio Savarese. Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation. In ICRA , 2021. [123] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In CVPR , 2018. [124] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X.Chang, Leonidas Guibas, and Hao Su. SAPIEN: A SimulAted Part-based Interactive ENvironment. In CVPR , 2020. [125] Yu Xiang, Wonhui Kim, Wei Chen, Jingwei Ji, Christopher Bongsoo Choy, Hao Su, Roozbeh Mottaghi, Leonidas J. Guibas, and Silvio Savarese. Objectnet3d: A large scale database for 3d object recognition. In ECCV , 2016. [126] Omry Yadan. Hydra - a framework for elegantly configuring complex applications. Github, 2019. [127] Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvijit Chattopadhyay, Taranjeet Singh, Akash Jain, Shiv Baran Singh, Stefan Lee, and Dhruv Batra. Evalai: Towards better evaluation systems for ai agents. arXiv , 2019. [128] Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Clip on wheels: Zero-shot object navigation as object localization and exploration. arXiv , 2022. [129] Zaiwei Zhang, Zhenpei Yang, Chongyang Ma, Linjie Luo, Alexander G. Huth, Etienne Vouga, and Qixing Huang. Deep generative modeling for scene synthesis via hybrid representations. ACM Trans. on Graphics , 2020. [130] Kaiyu Zheng, Rohan Chitnis, Yoonchang Sung, George Konidaris, and Stefanie Tellex. Towards Optimal Correlational Object Search. In ICRA , 2022.

[131] Yang Zhou, Zachary While, and Evangelos Kalogerakis. Scenegraphnet: Neural message passing for 3d indoor scene augmentation. In ICCV , 2019. [132] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA , 2017. [133] Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martín. robosuite: A modular simulation framework and benchmark for robot learning. arXiv , 2020.

Contributions

贡献

Matt Deitke designed and implemented the procedure to generate houses, implemented ObjectNav pre-training experiments and fine-tuning experiments, built the website, advised and implemented parts of the Unity backend, built the platform to visualize assets and create semantic asset groups, contributed to visuals, and wrote the paper.

Matt Deitke 设计并实现了生成房屋的流程，实现了 ObjectNav 预训练实验和微调实验，搭建了网站，为 Unity 后端的若干部分提供建议并参与实现，搭建了可视化资产并创建语义资产组的平台，为视觉呈现做出贡献，并撰写了论文。

Kiana Ehsani implemented ArmPointNav experiments and wrote parts of the paper.

Kiana Ehsani 实现了 ArmPointNav 实验，并撰写了论文的部分内容。

Ali Farhadi advised on the research direction.

Ali Farhadi 为研究方向提供建议。

Alvaro Herrasti led the Unity backend development that creates a house from a JSON specification.

Alvaro Herrasti 领导了根据 JSON 规格创建房屋的 Unity 后端开发。

Aniruddha Kembhavi advised on research direction, the ArchitecTHOR development, and the house generation process and wrote the paper.

Aniruddha Kembhavi 为研究方向、ArchitecTHOR 开发以及房屋生成流程提供建议，并撰写了论文。

Eric Kolve advised on the Unity backend development.

Eric Kolve 为 Unity 后端开发提供建议。

Roozbeh Mottaghi advised on the research direction, the Unity backend, the ArchitecTHOR development, and the house generation process and wrote the paper.

Roozbeh Mottaghi 为研究方向、Unity 后端、ArchitecTHOR 开发以及房屋生成流程提供建议，并撰写了论文。

Jordi Salvador implemented rearrangement experiments, advised on multi-node training experiments, and wrote parts of the paper.

Jordi Salvador 实现了重排实验，为多节点训练实验提供建议，并撰写了论文的部分内容。

Eli VanderBilt standardized AI2-THOR’s asset and material database to make it usable with ProcTHOR, led the development of ArchitecTHOR, implemented parts of the Unity backend, created new 3D assets and skyboxes, advised on lighting the houses, and contributed to visuals.

Eli VanderBilt 标准化了 AI2-THOR 的资产和材质数据库以便与 ProcTHOR 配合使用，领导了 ArchitecTHOR 的开发，实现了 Unity 后端的若干部分，创建了新的三维资产和天空盒，为房屋光照提供建议，并为视觉呈现做出贡献。

Winson Han implemented parts of ArchitecTHOR, implemented parts of the Unity backend, and contributed to visuals.

Winson Han 实现了 ArchitecTHOR 的若干部分、Unity 后端的若干部分，并为视觉呈现做出贡献。

Luca Weihs advised the work on experiments, assisted with rearragement experiments, implemented ObjectNav fine-tuning on HM3D-Semantics, and wrote parts of the paper.

Luca Weihs 为实验工作提供建议，协助重排实验，实现了 HM3D-Semantics 上的 ObjectNav 微调，并撰写了论文的部分内容。

Appendix

附录

Appendix A ProcTHOR Assets

附录 A ProcTHOR 资产

Fridge_1

Appendix B House Generation

附录 B 房屋生成

This section gives more details about the process we developed to procedurally sample houses.

这一节更详细地说明我们为程序化采样房屋所设计的流程。

B.1 Examples

B.1 例子

B.1.1 3-Room Houses Figure 14: Examples of 3-room houses generated in ProcTHOR-10K.

B.1.1 三房间房屋 图 14：ProcTHOR-10K 中生成的三房间房屋例子。

B.1.2 4-Room Houses Figure 15: Examples of 4-room houses generated in ProcTHOR-10K.

B.1.2 四房间房屋 图 15：ProcTHOR-10K 中生成的四房间房屋例子。

B.1.3 5-Room Houses Figure 16: Examples of 5-room houses generated in ProcTHOR-10K.

B.1.3 五房间房屋 图 16：ProcTHOR-10K 中生成的五房间房屋例子。

B.1.4 6-Room Houses Figure 17: Examples of 6-room houses generated in ProcTHOR-10K.

B.1.4 六房间房屋 图 17：ProcTHOR-10K 中生成的六房间房屋例子。

B.1.5 7+ Room Houses Figure 18: Examples of 7+ room houses generated in ProcTHOR-10K.

B.1.5 七个及以上房间的房屋 图 18：ProcTHOR-10K 中生成的七个及以上房间房屋的例子。

B.2 Room Specs

B.2 房间规格

(a) 4-Room House

(a) 四房间房屋

Room specs provide the ability to specify the rooms that appear in a house, the relative size of each room, and how the rooms are connected with doors. Their idea was first proposed in [75]. A room spec is manually specified with a tree data structure.

房间规格用来指定一座房子里会出现哪些房间、每个房间的相对大小，以及房间之间如何用门连接。这一想法最早在 [75] 中提出。房间规格用树形数据结构手工指定。

\mathcal{Z}_{bb}=\{

\mathcal{Z}_{bb}=\{

\mathcal{Z}_{bb}

\mathcal{Z}_{bb}

Room specs allow us to flexibly choose the distribution of houses we sample, allowing us to specify massive mansions, studio apartments, and anything in-between. Moreover, just a few room specs can go a long way. To generate our houses, we use 16 room specs, which each uses between 1 to 10 rooms. To generate the houses dataset, we assign a sampling weight to each of our room specs, and then use weighted sampling to sample a room spec for each house.

房间规格使我们能够灵活选择所采样房屋的分布，既可以指定巨大的宅邸，也可以指定单间公寓，以及介于两者之间的任何情况。而且，只需少数几种房间规格就能走得很远。为生成我们的房屋，我们使用 16 种房间规格，每种使用 1 到 10 个房间。为生成房屋数据集，我们给每种房间规格分配一个采样权重，然后用加权采样为每座房子采样一种房间规格。

B.3 Sampling Floor Plans

B.3 采样平面图

(x_{s},z_{s})

(x_{s},z_{s})

x

x

n_{c}

(x_{s}

(x_{s},z_{s})

(x_{s},z_{s})

(c_{x}

(c_{x}

(x_{s},z_{s})

(x_{s},z_{s})

Figure 22: An example of the recursive floor plan generation algorithm, given an interior boundary and the room spec in Figure 19(a). Here, we first divide the room into a “bedroom & bathroom” and a “kitchen & living room” zone. Then, within the “bedroom & bathroom” zone we place both the bedroom and bathroom, and within the “kitchen & living Room” zone, we place both the kitchen and living room.

图 22：递归平面图生成算法的一个例子，给定一个内部边界以及图 19(a) 中的房间规格。这里我们先把房间划分成“卧室和卫生间”以及“厨房和客厅”两个区域。然后，在“卧室和卫生间”区域里放置卧室和卫生间，在“厨房和客厅”区域里放置厨房和客厅。

Given a room spec and an interior boundary, we use the algorithm proposed in [73] to divide the interior boundary into rooms. The algorithm recursively subdivides the interior boundary for each subtree in the room spec. Figure 22 shows an example using Figure 19(a)’s room spec. The algorithm first divides the interior boundary into two zones, the “bedroom & bathroom” zone and the “kitchen & living room” zone. The “bedroom & bathroom” zone then subdivides into two rooms, the bedroom and bathroom. Similarly, the “kitchen & living room” zone is also subdivided into two rooms, the kitchen and living room. The growth weight is used to approximate the size of each subdivision. By recursively subdividing the zones of each subtree, we satisfy the constraint that we can traverse between child nodes of the same parent in the room spec.

给定房间规格和内部边界，我们使用 [73] 中提出的算法把内部边界划分成房间。该算法对房间规格中的每棵子树递归地细分内部边界。图 22 展示了使用图 19(a) 房间规格的一个例子。算法先把内部边界分成两个区域：“卧室和卫生间”区域以及“厨房和客厅”区域。“卧室和卫生间”区域再细分成两个房间，即卧室和卫生间。类似地，“厨房和客厅”区域也被细分成厨房和客厅两个房间。生长权重用来近似每次细分的大小。通过递归细分每棵子树的区域，我们满足了可以在房间规格中同一父节点的子节点之间穿行的约束。

s\sim U(1.6,2.2)

s\sim U(1.6,2.2)

B.4 Connecting Rooms

B.4 连接房间

(a) Doorway Connection

(a) 门洞连接

Figure 23 shows the 3 types of ways adjacent rooms may be connected. Specifically, rooms may be connected using 3 different types of connections: doorways, door frames, or open room connections. We determine which rooms should have doors between them based on the constraints in the room spec. Amongst adjacent rooms that may have doors between them, subject to the constraints in the room spec, we randomly sample which rooms have doors. We also impose the constraint that neighboring rooms in the room spec may have at most 1 room connected to it.

图 23 展示了相邻房间可能被连接的 3 种方式。具体而言，房间可以用 3 种不同类型的连接相连：门洞、门框或开放房间连接。我们根据房间规格中的约束，确定哪些房间之间应当有门。在房间规格约束允许有门的相邻房间中，我们随机采样哪些房间有门。我们还施加一条约束：房间规格中的相邻房间最多只能与其中一个房间相连。

To choose the type of connection, we consider the rooms we are connecting. Specifically, we only allow open room connections and door frame connections between kitchen and living room room types. We impose this constraint because it would be unrealistic for a room like a bathroom to be fully visible from another room. For connecting room types that do support open room connections or door frames, we annotate the probability of sampling a doorway, door frame, and open room connection. Between a kitchen and living room the probability is 0.375 for sampling both an open room connection and a door frame connection, and 0.25 for sampling a doorway connection.

为选择连接类型，我们考虑正在连接的房间。具体而言，我们只允许在厨房和客厅这两种房间类型之间使用开放房间连接和门框连接。我们施加这一约束，是因为让卫生间这类房间从另一个房间完全可见会不真实。对于支持开放房间连接或门框的房间类型组合，我们标注采样门洞、门框和开放房间连接的概率。在厨房和客厅之间，采样开放房间连接和门框连接的概率均为 0.375，采样门洞连接的概率为 0.25。

If a doorway or door frame is sampled, we filter to use a valid asset that is smaller than the wall connecting the rooms. For our generation, the minimum wall size is always greater than a single door size, but occasionally the filter might remove double doors from valid doors that can be sampled as they would be too big. The placement of the door is then uniformly sampled from anywhere along the wall. For doorways, the open direction is uniformly sampled. Finally, if the open state from any 2 doorways collides, we also use rejection sampling to potentially change the open direction and modify the placement of doorways.

如果采样到门洞或门框，我们会过滤，只使用比连接这两个房间的墙壁更小的有效资产。在我们的生成中，最小墙壁尺寸总是大于单扇门的尺寸，但过滤偶尔会把双开门从可采样的有效门中去掉，因为它们会太大。门的摆放随后沿墙壁均匀采样。对门洞，打开方向均匀采样。最后，如果任意两扇门洞的打开状态发生碰撞，我们还使用拒绝采样，以可能改变打开方向并修改门洞的摆放。

Each house also has a permanently closed exterior door connecting to the outside. We prioritize placing this door in kitchen and living room room types, as it is unnatural to have to go through a bathroom or bedroom to go outside. However, in the case where the room spec does not include a kitchen or living room (e.g. if the room is a standalone bathroom), we randomly place a door to the outside in one of the remaining rooms.

每座房子还有一扇永久关闭的外门通向室外。我们优先把这扇门放在厨房和客厅类型的房间里，因为必须穿过卫生间或卧室才能出门会不自然。不过，当房间规格不包含厨房或客厅时（ 例如 该房间是独立卫生间），我们会在剩余房间之一中随机放置一扇通向室外的门。

B.5 Structure Materials

B.5 结构材质

Wall materials. To choose the materials that make up the walls, we consider 2 families of wall materials: solid colors and texture-based materials. Our solid color materials consist of 40 unique colors of popular paint colors found in houses. We constrain ourselves to only using popular paint colors, so we do not randomize the walls to unrealistic colors such as bright green or yellow. For the texture-based materials, we annotate 122 different AI2-THOR materials to be suitable as wall materials. These include materials for brick textures, drywall textures, and tiling textures, amongst others.

墙壁材质 。 为选择构成墙壁的材质，我们考虑两类墙壁材质：纯色和基于纹理的材质。我们的纯色材质包含 40 种在房屋中常见的流行涂料颜色。我们限制自己只使用流行涂料颜色，因此不会把墙壁随机成鲜绿或明黄这类不真实的颜色。对基于纹理的材质，我们标注了 122 种适合作为墙壁材质的 AI2-THOR 材质。其中包括砖块纹理、石膏板纹理和瓷砖纹理等。

w_{\textit{solid}}\sim\text{Bernoulli}(0.5)

w_{\textit{solid}}\sim\text{Bernoulli}(0.5)

w_{\textit{same}}

w_{\textit{same}}

f_{\textit{same}}\sim\text{Bernoulli}(0.15)

f_{\textit{same}}\sim\text{Bernoulli}(0.15)

B.6 Ceiling Height

B.6 层高

Figure 24: The distribution of the ceiling height of each house, in meters.

图 24：每座房子层高的分布，单位是米。

c_{h}\sim h_{\min}+(h_{\max}-h_{\min})\cdot\text{Beta}(\alpha_{h},\beta_{h})

c_{h}\sim h_{\min}+(h_{\max}-h_{\min})\cdot\text{Beta}(\alpha_{h},\beta_{h})

\beta_{h}

\beta_{h}

B.7 Lighting

B.7 光照

Lighting Placement. Each procedural house places two types of lights: a directional light and point lights. The directional light is analogous to the sun in the scene, where only 1 is placed in each scene. Light from point lights are analogous to the light emitted from lightbulbs. We place a point light in each room near the ceiling, centered at the centroid of the room’s floor polygon. Using the centroid ensures that the light is always placed inside of the room, even for L-shaped rooms. Additionally, desk lamp and floor lamp objects have a point light associated with them.

光照摆放 。 每座程序化房屋放置两类灯光：平行光和点光源。平行光相当于场景中的太阳，每个场景只放一盏。点光源发出的光相当于灯泡发出的光。我们在每个房间靠近天花板处放一盏点光源，位置取房间地面多边形的质心。用质心可以保证即使是 L 形房间，灯光也总是落在房间内部。此外，台灯和落地灯这类物体各自还关联一盏点光源。

(a) Midday Skybox

(a) 正午天空盒

Effects by the time of day . Skyboxes may appear at 3 different times of day: midday, golden hour, and blue hour. The time of day determines the intensity, hue, and direction of the ambient outdoor lighting. For each time of day, there exist multiple skyboxes, which dictate the lighting of the environment. Figure 25 shows examples of how the time of day visually affects the scene. At this time, there are 16 midday skyboxes, 5 golden hour skyboxes, and 1 blue hour skybox, based on full 360-degree photos taken in Seattle and San Francisco.

一天中不同时段的效果 。 天空盒可以出现在一天中的 3 个时段：正午、黄金时段和蓝调时段。时段决定室外环境光的强度、色调和方向。每个时段都有多种 天空盒 ，用来规定环境的光照。图 25 展示了一天中不同时段如何在视觉上影响场景。目前有 16 个正午天空盒、5 个黄金时段天空盒和 1 个蓝调时段天空盒，均基于在西雅图和旧金山拍摄的完整 360 度照片。

B.8 Object Placement

B.8 物体摆放

In this section, we discuss how objects are placed realistically in the house. We hypothesize reasonable object placement is necessary in order to train efficient agents. For instance, if a toilet could appear anywhere in the house, the agent would have a much harder search problem, leading to longer episodes, than if the toilet was always in the bathroom. Moreover, we do not want objects to appear in unnatural positions, such as a fridge facing the wall, as it would make it unnatural, and even unusable, for interaction.

这一节讨论如何把物体真实地摆进房子。我们假设，合理的物体摆放对训练高效智能体是必要的。例如，如果马桶可以出现在房子里的任何地方，智能体的搜索问题就会困难得多，回合也会更长；如果马桶总是在卫生间里，情况会好很多。我们也不希望物体出现在不自然的位置，例如冰箱正对墙壁，那样既不真实，也无法交互。

Finally, we do not always want objects to spawn independently. For instance, we might want a table to be surrounded by chairs. We achieve dependant sampling by developing SAGs, which are described in the section that follows.

最后，我们并不总是希望物体独立生成。例如，我们可能希望一张桌子周围摆着椅子。我们通过开发语义资产组（SAG）来实现这种依赖采样，详见下一小节。

B.8.1 Assets The ProcTHOR asset database consists of 1,633 interactive household assets across 108 object types (see Appendix A for more details). The majority of assets come from AI2-THOR. Windows, doors, and counter tops are built into the exterior of rooms in AI2-THOR, which prevents us from spawning them in as standalone assets. Thus, we have also hand-built 21 windows, 20 doors, and 33 counter tops.

B.8.1 资产 ProcTHOR 资产库包含 108 种物体类型、共 1,633 件可交互家居资产（详见附录 A）。其中大多数资产来自 AI2-THOR。窗户、门和台面在 AI2-THOR 中是嵌在房间外墙上的，无法作为独立资产生成。因此我们还手工制作了 21 扇窗户、20 扇门和 33 个台面。

Asset Annotations. Our assets include several annotations that help us place them realistically in a house. Figure 26 shows an example of the asset annotations used to place an arm chair. For an individual asset, we annotate its object type, computationally obtain its 3D bounding box, and partition assets of object types into training, validation, and testing splits. Then, we annotate how each object type might be spawned into the house. Annotating the 108 object types, as opposed to annotating the 1,633 individual assets, allows us to scale up the number of unique assets dramatically. Moreover, it does not require any new annotation to add an asset that can be grouped with an existing object type.

资产标注 。 我们的资产包含若干标注，帮助我们把它们真实地摆进房子。图 26 展示了摆放一把扶手椅时所使用的资产标注例子。对单个资产，我们标注其物体类型，计算得到它的三维包围盒，并把同一物体类型的资产划分到训练、验证和测试划分。然后，我们标注每种物体类型如何生成到房子里。对 108 种物体类型做标注，而不是对 1,633 件单独资产逐一标注，使我们能够大幅扩大独特资产的数量。而且，新增一件可以归入已有物体类型的资产时，不需要任何新标注。

Figure 26: An example of the asset annotations used to place an arm chair asset. This particular instance is annotated with its object type, bounding box, and split. Annotations about how it is placed in the house are done at an object type level, applying to all instances of that type.

图 26：摆放一件扶手椅资产时所使用的资产标注例子。这个实例标注了物体类型、包围盒和数据划分。关于它如何摆进房子的标注是在物体类型层面完成的，适用于该类型的全部实例。

If instances of an object type cannot be placed independently on the floor, the rest of its annotations are not considered. For instance, we do not allow television object types to be placed alone on the floor, rather they are often placed on top of a television stand or mounted on the wall, which is discussed later in this section. Similarly, we also annotate small objects, like a fork, pen, and mug to not be placed independently on the floor. However, typical large object types, such as counter top, arm chair, or fridge object types can be placed independently on the floor.

如果某类物体的实例不能独立摆在地面上，就不会再考虑它的其余标注。例如，我们不允许电视机类型单独摆在地面上，它们通常放在电视柜上或挂在墙上，后文会讨论。类似地，我们也把叉子、笔和马克杯这类小物体标注为不能独立摆在地面上。不过，典型的大物体类型，例如台面、扶手椅或冰箱，可以独立摆在地面上。

r_{w}\in\{0,1,2,3\}

r_{w}\in\{0,1,2,3\}

\nicefrac{{2}}{{3}}

\nicefrac{{2}}{{3}}

B.8.2 Semantic Asset Groups (SAGs) (a) An interface for viewing SAGs showing child asset anchoring and rotational randomness.

B.8.2 语义资产组（SAG） (a) 用于查看 SAG 的界面，展示子资产锚定和旋转随机性。

A Semantic Asset Group (SAG) provides a flexible and diverse way to encode which objects may appear near each other. The power of SAGs comes in their ability to support randomized asset and rotational sampling. SAGs can be created and exported in seconds with our user-friendly drag-and-drop web interface.

语义资产组 （SAG）以灵活多样的方式编码哪些物体可以彼此靠近出现。SAG 的力量来自它们对随机资产采样和旋转采样的支持。借助我们友好的拖放式网页界面，可以在几秒钟内创建并导出 SAG。

Figure 27 shows an example of how we might construct a SAG that has two chairs pushed into the side of a dining table. The SAG includes two chair samplers and a dining table sampler. Asset samplers contain a set of unique 3D modeled asset instances that may be sampled. When the SAG is instantiated, each asset sampler randomly chooses one of its instances. Asset samplers can also be linked, where multiple samplers sample the same asset instance each time. Here, linking may allow for multiple instances of the same chair to be placed at a dining table, instead of independently sampling a different chair for each sampler.

图 27 展示了我们如何构造一个把两把椅子推进餐桌侧面的 SAG。该 SAG 包含两个椅子采样器和一个餐桌采样器。资产采样器包含一组可能被采样的独特三维建模资产实例。实例化 SAG 时，每个资产采样器随机选择其中一个实例。资产采样器也可以被链接，使多个采样器每次都采样同一个资产实例。在这里，链接可以让同一把椅子的多个实例摆在餐桌旁，而不是为每个采样器独立采样一把不同的椅子。

800

800

\mathcal{V}=\{\textsc{Top},\textsc{Center},\textsc{Bottom}\}

\mathcal{V}=\{\textsc{Top},\textsc{Center},\textsc{Bottom}\}

(a) Center Right Anchor Center Left Pivot

(a) 中右锚点，中左枢轴

The motivation for the relative positioning of asset samplers is to prevent the meshes from clipping into each other. For instance, with the same SAG in Figure 27, consider what would happen if the dining table sampler samples a table that is double the size of the current table. Instead of the chairs being stuck in a fixed global position, and effectively colliding with the new dining table, the chairs will reactively move back, and be re-positioned to remain slightly tucked under the larger table. Moreover, consider that the size of instances that are sampled from an asset sampler are often quite different. For instance, one table might be square-ish, while another is elongated. If we only used a Center Center pivot and an offset, one would not be able to reliably place asset samplers, containing differently sized objects, directly beside each other without it resulting in clipping.

对资产采样器做相对定位的动机，是防止网格彼此穿插。例如，仍用图 27 中的同一个 SAG，考虑如果餐桌采样器采样到一张尺寸翻倍的桌子会发生什么。椅子不会卡在固定的全局位置上、实际上与新餐桌发生碰撞，而会响应式地后退，重新定位，仍然略微收进更大的桌子下面。此外，从资产采样器中采样出的实例尺寸往往差异很大。例如，一张桌子可能接近正方形，另一张则是细长的。如果只使用中心—中心枢轴再加一个偏移，就无法可靠地把包含不同尺寸物体的资产采样器直接并排放在一起，而不导致穿插。

(a) Rejected

(a) 被拒绝

While setting anchoring and pivot positions solves many mesh clipping issues, some cases may still arise. Figure 29 shows an example, where if our dining table sampler samples a short dining table, it may clip into certain chairs. Such issues are rare in practice, but object clipping would lead to less realistic and interactive houses. To solve the clipping issue, we use rejection sampling to resample the assets of a SAG until none of the 3D meshes of the sampled assets are clipping.

设置锚定和枢轴位置可以解决许多网格穿插问题，但有些情况仍会出现。图 29 给出了一个例子：如果我们的餐桌采样器采样到一张较矮的餐桌，它可能穿进某些椅子。这类问题在实践中很少见，但物体穿插会让房屋更不真实、更难交互。为解决穿插问题，我们使用拒绝采样，反复对 SAG 的资产重新采样，直到所采样资产的三维网格不再穿插。

In ProcTHOR-10K, we construct 18 SAGs, which can be instantiated with over 20 million unique combinations of assets. These include semantic asset groups for chairs around tables, pillows on top of beds, sofas and arm chairs looking at a television on top of a TV stand, faucets on top of sinks, and a desk with a chair, amongst others.

在 ProcTHOR-10K 中，我们构造了 18 个 SAG，可以用超过 2000 万种独特的资产组合来实例化。其中包括桌子周围的椅子、床上的枕头、沙发和扶手椅对着电视柜上的电视机、水槽上的水龙头，以及配椅子的书桌等。

B.8.3 Floor Object Placement Figure 30: Diagram detailing how floor objects are placed in a room. First, we rectangularize the top-down view of the room’s open floor plan by drawing horizontal and vertical dividers from each corner point. Then, we construct all possible rectangles that are formed within the dividers. We then sample one of those rectangles and place the object within that rectangle. The sampled object’s top-down bounding box (with margin) is shown in blue. The bounding box is then subtracted from the open floor plan before repeating the process again.

B.8.3 地面物体摆放 图 30：说明如何在房间里摆放地面物体的示意图。首先，我们从每个角点画出水平和垂直分隔线，把房间开放平面图的俯视图矩形化。然后，我们构造分隔线内形成的全部可能矩形。接着采样其中一个矩形，并把物体放进该矩形。所采样物体的俯视包围盒（含边距）用蓝色标出。随后从开放平面图中减去该包围盒，再重复这一过程。

We start object placement by first placing objects on the floor of the house. Objects are independently placed on a room-by-room basis, where we may first place objects in the bedroom and then place objects in the bathroom, without either affecting each other.

我们先从把物体摆到房子地面上开始。物体在各个房间中独立摆放：我们可能先在卧室里摆物体，再在卫生间里摆物体，两者互不影响。

r_{w}>0

r_{w}>0

(o_{w},o_{h})

(o_{w},o_{h})

r_{\ell}\sim\text{Bernoulli}(0.8)

r_{\ell}\sim\text{Bernoulli}(0.8)

(r_{w},r_{h})

(r_{w},r_{h})

If any of the rectangle’s corners is in a corner of the room, then we will place an object in that corner of the room. If multiple of the rectangle’s corners are in a corner of the room, then we uniformly sample a corner amongst one of those corners.

如果矩形的任一角落在房间的一个角落里，我们就会把一个物体放在房间的那个角落。如果矩形有多个角落落在房间的角落里，我们就在这些角落中均匀采样一个。

Now, we will filter down objects and asset groups to only consider:

接下来，我们把物体和资产组过滤为只考虑：

Those that are annotated specifying that they can be placed in the corner of the room. For example, we might annotate a fridge to be placed in the corner of the room, but we might not annotate a SAG consisting of a dining table to be placed in the corner of the room.

那些被标注为可以放在房间角落的。例如，我们可能标注冰箱可以放在房间角落，但可能不会标注由餐桌组成的 SAG 放在房间角落。

The annotated split of the asset instance matches the current split of the generated house. See Appendix B.8.1 which talks about asset splits to create train/val/test homes.

资产实例的标注划分与当前所生成房屋的划分相匹配。参见附录 B.8.1，其中讨论了用于创建训练/验证/测试房屋的资产划分。

(o_{h}+w_{\textit{pad}}\leq r_{w}\text{ and }o_{w}+w_{\textit{pad}}\leq r_{h})\text{ or }(o_{h}+w_{\textit{pad}}\leq r_{h}\text{ and }o_{w}+w_{\textit{pad}}\leq r_{w})

(o_{h}+w_{\textit{pad}}\leq r_{w}\text{ 且 }o_{w}+w_{\textit{pad}}\leq r_{h})\text{ 或 }(o_{h}+w_{\textit{pad}}\leq r_{h}\text{ 且 }o_{w}+w_{\textit{pad}}\leq r_{w})

(a) Edge Rotations

(a) 边缘旋转

We sample an object or asset group that satisfies all of the previous conditions. If there are no objects or asset groups that satisfy all conditions, we continue to the next iteration and remove the selected rectangle from consideration. We slightly prioritize placing asset groups over standalone assets when possible. Once we have chosen an object or asset group, the bounding box with margin is then anchored to the corner of the rectangle, and hence to the corner of the room. We then subtract the object’s bounding box, with margin, from the open polygon representing the space remaining in the room before doing the same process again.

我们采样一个满足前述全部条件的物体或资产组。如果没有任何物体或资产组满足全部条件，我们就进入下一次迭代，并把所选矩形从候选中去掉。在可能的情况下，我们略微优先摆放资产组，而不是独立资产。一旦选定物体或资产组，带边距的包围盒就会锚定到矩形角落，从而锚定到房间角落。然后我们从表示房间剩余空间的开放多边形中减去该物体带边距的包围盒，再重复同一过程。

r_{\textit{edge}}\sim\text{Bernoulli}(0.7)

r_{\textit{edge}}\sim\text{Bernoulli}(0.7)

r_{i}

r_{i}

r_{i}\sim\begin{cases}1&p=\nicefrac{{1}}{{200}}\4&p=\nicefrac{{2}}{{200}}\5&p=\nicefrac{{4}}{{200}}\6&p=\nicefrac{{20}}{{200}}\7&p=\nicefrac{{173}}{{200}}\end{cases}. \tag{1}

r_{i}

r_{i}

By iteratively choosing the largest, or near largest, rectangle in the room’s open polygon, placing an object in it, and subtracting the object’s bounding box with margin from the open room polygon, we enable great coverage across the entirety of the room, and hence the entirety of the house.

通过迭代选择房间开放多边形中最大或接近最大的矩形、把物体放进去、再从开放房间多边形中减去物体带边距的包围盒，我们能够在整个房间、进而整座房子里获得很好的覆盖。

B.8.4 Wall Object Placement (a) Window

B.8.4 墙面物体摆放 (a) 窗户

After placing objects on floors, we then place objects on walls. We currently place window, painting, and television objects on the walls. Figure 32 shows some examples. Window and television objects may appear in kitchen, living room, and bedroom room types. Paintings may appear in any room type.

在地面上摆好物体之后，我们再把物体摆到墙上。目前我们会把窗户、画作和电视机摆到墙上。图 32 给出了一些例子。窗户和电视机可以出现在厨房、客厅和卧室这几类房间中。画作可以出现在任何房间类型中。

Windows. Window objects are the first objects we place on the walls of the house. We only consider placing a window on walls that are connected to the outside of the house, such that we do not place a window between two indoor rooms. For each kitchen, living room, and bedroom in the house, we sample

窗户 。 窗户是我们最先摆到房屋墙面上的物体。我们只考虑把窗户放在与室外相连的墙上，因此不会在两个室内房间之间开窗。对房子里的每个厨房、客厅和卧室，我们采样

n_{w}\sim\begin{cases}0&p=0.125\1&p=0.375\2&p=0.5\end{cases} \tag{2}

maximum window objects to be placed.

作为最多要摆放的窗户物体数量。

w_{\max}=\min(3,c_{h})

w_{\max}=\min(3,c_{h})

Paintings. Painting objects are placed on the walls after window objects. They may be placed in any room. The maximum number of painting objects that are attempted to be placed in each room is sampled from

画作 。 画作在窗户之后摆到墙上。它们可以放在任何房间里。每个房间尝试摆放的画作数量上限从下式采样

n_{p}\sim\begin{cases}0&p=0.05\1&p=0.1\2&p=0.5\3&p=0.25\4&p=0.1\end{cases}. \tag{3}

The placement of painting objects is similar to the placement of window objects. However, multiple painting objects may be placed along the same wall, so instead of removing the entire wall segment after an object is placed on it, we subtract the width of the painting from the segment. Moreover, we also allow painting objects to be placed above edge floor objects if the height of the edge object is less than 1.15 meters. Here, this allows for a painting to be above an object like a counter top, but not behind a taller object like a fridge.

画作的摆放与窗户类似。不过，同一面墙上可以放多幅画作，因此在墙上放好一件物体后，我们不是去掉整段墙线段，而是从线段中减去画作的宽度。此外，如果边缘地面物体的高度小于 1.15 米，我们也允许把画作放在这些边缘地面物体上方。这样可以把画作放在台面这类物体上方，但不会放在冰箱这类更高物体的后面。

o_{y}\sim w_{\min}+(w_{\max}-w_{\min})\cdot\text{Beta}(12,12)

o_{y}\sim w_{\min}+(w_{\max}-w_{\min})\cdot\text{Beta}(12,12)

\text{Bernoulli}(0.8)

\text{Bernoulli}(0.8)

b_{\textit{house}}

b_{\textit{house}}

After placing objects on the floor and wall of the house, we focus on placing objects on the surface of the floor objects just placed. For example, we may place objects like a coffee machine, plate, or knife on of a receptacle like a counter top.

在房子的地面和墙壁上摆好物体之后，我们转而把物体摆到刚刚摆好的地面物体表面上。例如，我们可能把咖啡机、盘子或刀这类物体放到台面这类容器上。

For each receptacle object, we approximate the probability that each object type appears on its surface. We use the hand-modeled AI2-iTHOR or RoboTHOR rooms to obtain these approximations. Here, we compute the total number of times each object type is on the receptacle type and divide it by the total number of times the receptacle type appears across the scenes.

对每个容器物体，我们近似每种物体类型出现在其表面上的概率。我们用手工建模的 AI2-iTHOR 或 RoboTHOR 房间来得到这些近似。在这里，我们计算每种物体类型出现在该容器类型上的总次数，再除以该容器类型在各场景中出现的总次数。

p_{\textit{spawn}}

p_{\textit{spawn}}

b_{\textit{house}}

b_{\textit{house}}

b_{\textit{receptacle}}

b_{\textit{receptacle}}

b_{\textit{object}}

b_{\textit{object}}

p_{\textit{spawn}}+b_{\textit{house}}+b_{\textit{receptacle}}+b_{\textit{object}}

p_{\textit{spawn}}+b_{\textit{house}}+b_{\textit{receptacle}}+b_{\textit{object}}

n_{\textit{pa}}=5

n_{\textit{pa}}=5

n_{\textit{or}}\sim\min(s_{\max},\text{Geom}(p_{\textit{spawn}})-1)-1

n_{\textit{or}}\sim\min(s_{\max},\text{Geom}(p_{\textit{spawn}})-1)-1

B.9 Material and Color Randomization

B.9 材质与颜色随机化

(a) Examples of color randomization for a vase object. The original color is shown on the left. Notice that the vase still looks realistic with many possible colors.

(a) 花瓶物体颜色随机化的例子。左侧是原始颜色。可以看到，花瓶在许多可能颜色下仍然看起来真实。

r_{c}\sim\text{Bernoulli}(0.8)

r_{c}\sim\text{Bernoulli}(0.8)

r_{m}\sim\text{Bernoulli}(0.8)

r_{m}\sim\text{Bernoulli}(0.8)

B.10 Object States

B.10 物体状态

(a) Openness state randomness example with a laptop.

(a) 用笔记本电脑展示开合状态随机性的例子。

We randomize object states to expose the agent to more diverse objects during training. For instance, instead of always having an open laptop or a clean bed, we randomize the openness of each laptop and if each bed is clean or dirty. Figure 35 shows some examples. Our current set of state randomizations include:

我们随机化物体状态，以便在训练中让智能体接触更多样的物体。例如，我们不会总是让笔记本电脑开着或床铺干净，而是随机化每台笔记本电脑的开合程度，以及每张床是干净还是脏的。图 35 给出了一些例子。我们目前的状态随机化包括：

• Toggling objects. Floor lamp and desk lamp object types have their state toggled on or off.

• 开关物体 。 落地灯和台灯类型的状态会被打开或关闭。

• Cleaning or dirtying objects. Bed object types may appear as either clean or dirty.

• 清洁或弄脏物体 。 床类型可以表现为干净或脏。

• Opening or closing objects. Box and laptop object types may

• 打开或关闭物体 。 盒子和笔记本电脑类型可以

toggling objects on or off (for floor lamp and desk lamp object types), setting objects to clean or dirty (for bed object types), and openness randomizations (for box and laptop object types).

打开或关闭物体（针对落地灯和台灯类型）、把物体设为干净或脏（针对床类型），以及开合随机化（针对盒子和笔记本电脑类型）。

B.11 Validator

B.11 校验器

0.25\times 0.25

0.25\times 0.25

B.12 Limitations and Future Work

B.12 局限与未来工作

ProcTHOR-10K only uses 1-floor houses. We plan to support multi-floor houses in ProcTHOR-v2.0. This will allow us to capture a wider range of houses and provide better fine-tuning results. Additionally, we plan to scale up our asset databases by leveraging many open-source 3D asset databases, such as ABO [26], PartNet [81], ShapeNet [15], Google Scanned Objects [30], and CO3D [97], among others.

ProcTHOR-10K 只使用单层房屋。我们计划在 ProcTHOR-v2.0 中支持多层房屋。这将使我们能够覆盖更广范围的房屋，并提供更好的微调结果。此外，我们计划借助许多开源三维资产库来扩大我们的资产库，例如 ABO [26]、PartNet [81]、ShapeNet [15]、Google Scanned Objects [30] 和 CO3D [97] 等。

Appendix C ProcTHOR Datasheet

附录 C ProcTHOR 数据说明书

Motivation
For what purpose was the dataset created?	The dataset was created to enable the training of simulated embodied agents in substantially more diverse environments.
Who created and funded the dataset?	This work was created and funded by the PRIOR team at Allen Institute for AI. See the contributions section for specific details.
Composition
What do the instances that comprise the dataset represent?	Each house is specified as a JSON file, which specifies how to populate a 3D Unity scene in AI2-THOR.
How many instances are there in total (of each type, if appropriate)?	There are 10K houses released in the dataset, along with the code to sample substantially more. Section 4 shows the distribution of houses in ProcTHOR-10K.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?	We make 10K houses available, but more houses can easily be sampled with the procedural generation scripts.
What data does each instance consist of?	Each house is specified as a JSON file, which precisely describes how our AI2-THOR build should create the house. The procedurally generated JSON files are typically several thousand lines long.
Is there a label or target associated with each instance?	No.
Is any information missing from individual instances?	No.
Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?	Each house is generated independently, meaning there are no relationships between the houses.
Are there recommended data splits?	Yes. See Appendix B.8.1.
Are there any errors, sources of noise, or redundancies in the dataset?	No.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?	The dataset is self-contained.
Does the dataset contain data that might be considered confidential?	No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?	No.
Collection Process
How was the data associated with each instance acquired?	Each house was procedurally generated. See Appendix A.
If the dataset is a sample from a larger set, what was the sampling strategy?	The dataset consists of 1 million houses sampled from the procedural generation scripts.
Who was involved in the data collection process?	The authors were the only people involved in constructing the dataset.
Over what timeframe was the data collected?	Data was collected between the end of 2021 and the beginning of 2022.
Were any ethical review processes conducted?	No.
Preprocessing/Cleaning/Labeling
Was any preprocessing/cleaning/labeling of the data done?	Section B.8 describes the labeling that was done to make the assets spawn in realistic places. We have also gone through every asset in the asset database to make sure the pivots for each asset are facing a consistent direction.
Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data?	There is no raw data associated with the house JSON files.
Is the software that was used to preprocess/clean/label the data available?	The code to generate the houses is made available.
Uses
Has the dataset been used for any tasks already?	Yes. See Section 5 of the paper.
What (other) tasks could the dataset be used for?	The houses can be used in a wide variety of interactive tasks in embodied AI and computer vision. Any task that can be performed in AI2-THOR can be performed in ProcTHOR. For instance, in embodied AI, the houses may be used for navigation [58, 89, 118, 132, 117, 128, 74, 130], multi-agent interaction [51, 52, 2], rearrangement and interaction [115, 36, 39, 23, 106], manipulation [33, 86, 32, 122], Sim2Real transfer [27, 54, 66], embodied vision-and-language [105, 87, 48, 65, 42, 55], audio-visual navigation [22, 38, 21], and virtual reality interaction [119, 83, 46], among others. In the broader field of computer vision, the dataset may be used to study object detection [64]; NeRFs [80, 110, 43, 71]; segmentation, depth, and optimal flow estimation [35, 43]; generative modeling [59, 62, 61]; occlusion reasoning [34]; and pose estimation [19], among others. Our framework for loading in procedurally generated houses from a JSON spec also enables the study of scene clutter generation, building more realistic procedurally generated homes, and the development of synthetically generated spaces to train embodied agents in factories [84], offices, grocery stores [76], and full procedurally generated cities.
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?	No.
Are there tasks for which the dataset should not be used?	Our dataset may be used for both commercial and non-commercial purposes.
Distribution
Will the dataset be distributed to third parties outside of the entity on behalf of which the dataset was created?	Yes. We plan to make the entirety of the work open-source, including the code used to generate and load houses, the initial static dataset of 1 million procedurally generated house JSON files, and the asset and material databases.
How will the dataset be distributed?	The static house JSON files will be distributed with a custom Python package. The code, asset, and material databases will be distributed on GitHub.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?	The house dataset, 3D asset database, and generation code will be released under the Apache 2.0 license.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances?	No.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?	No.
Maintenance
Who will be supporting/hosting/maintaining the dataset?	The authors will be providing support, hosting, and maintaining the dataset.
How can the owner/curator/manager of the dataset be contacted?	For inquiries, email mattd@allenai.org.
Is there an erratum?	We will use GitHub issues to track issues with the dataset.
Will the dataset be updated?	We expect to continue adding support for new features to continue to make procedurally generated houses even more diverse and realistic. We also intend to support new tasks in the future.
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)?	The dataset does not relate to people.
Will older versions of the dataset continue to be supported/hosted/maintained?	Yes. Revision history will be available for older versions of the dataset.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?	Yes. The work will be open-sourced and we intend to provide support to help others use and build upon the dataset.

Appendix D ArchitecTHOR

附录 D ArchitecTHOR

Figure 36: Top-down images of the 5 custom-built interactive validation houses in ArchitecTHOR. The goal of these houses is to evaluate interactive agents in more realistic and larger home environments.

图 36：ArchitecTHOR 中 5 套定制搭建的可交互验证房屋的俯视图。这些房子的目标是在更真实、也更大的家居环境中评估可交互智能体。

D.1 Datasheet

D.1 数据说明书

Motivation
For what purpose was the dataset created?	ArchitecTHOR was created to enable the evaluation of embodied agents in large, realistic, and interactive household environments.
Who created and funded the dataset?	This work was created and funded by the PRIOR team at Allen Institute for AI. See the contributions section for specific details.
Composition
What do the instances that comprise the dataset represent?	Instances of the dataset comprise interactive 3D houses that were built in Unity and can be used with our custom build of the AI2-THOR API.
How many instances are there in total (of each type, if appropriate)?	There are 10 total houses, comprising 5 validation houses and 5 testing houses.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?	The dataset is self-contained.
What data does each instance consist of?	Each instance of a house is a Unity scene, which includes data such as the placement of objects, lighting, and texturing.
Is there a label or target associated with each instance?	No.
Is any information missing from individual instances?	No.
Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?	Each house was independently created.
Are there recommended data splits?	Yes. The houses themselves are partitioned as 5 validation houses and 5 testing houses. The assets placed in the house follow the same train/val/test splits used in ProcTHOR-10K.
Are there any errors, sources of noise, or redundancies in the dataset?	No.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?	The dataset is self-contained.
Does the dataset contain data that might be considered confidential?	No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?	No.
Collection Process
How was the data associated with each instance acquired?	Each house was professionally hand-modeled by 3D artists. Most objects placed in the hosues come from the ProcTHOR asset database. However, countertops, showers, and many cabinets were custom built.
If the dataset is a sample from a larger set, what was the sampling strategy?	The dataset consists of 1 million houses sampled from the procedural generation scripts.
Over what timeframe was the data collected?	The houses were built towards the beginning of 2022.
Were any ethical review processes conducted?	No.
Preprocessing/Cleaning/Labeling
Was any preprocessing/cleaning/labeling of the data done?	No.
Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data?	There is no raw data associated with the ArchitecTHOR houses.
Is the software that was used to preprocess/clean/label the data available?	Yes. We will open-source the ArchitecTHOR houses and they can be opened and viewed in Unity.
Uses
Has the dataset been used for any tasks already?	Yes. Please see Section 5 of the paper.
What (other) tasks could the dataset be used for?	The tasks can be used for any type of navigation and interaction tasks in embodied AI. The houses are built into our build of AI2-THOR, meaning ArchitecTHOR can work with any task that can be performed in AI2-THOR. We especially think ArchitecTHOR will be useful as an evaluation suite for evaluating different sets of ProcTHOR tasks and evaluating agents trained on different sets of procedurally generated houses.
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?	No.
Are there tasks for which the dataset should not be used?	Our dataset may be used for both commercial and non-commercial purposes.
Distribution
Will the dataset be distributed to third parties outside of the entity on behalf of which the dataset was created?	Yes. All houses in ArchitecTHOR will be released to the open-source community and available through our build of the AI2-THOR Python API.
How will the dataset be distributed?	The houses will be distributed on GitHub and available to open as Unity scenes.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?	ArchitecTHOR will be released under the Apache 2.0 license.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances?	No.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?	No.
Maintenance
Who will be supporting/hosting/maintaining the dataset?	The authors will be providing support, hosting, and maintaining the dataset.
How can the owner/curator/manager of the dataset be contacted?	Omitted for anonymous review.
Is there an erratum?	We will use GitHub issues to track issues with the dataset once it is published.
Will the dataset be updated?	ArchitecTHOR is currently in maintenance mode and we do not expect it to update much from its current state. However, we plan to actively support future AI2-THOR functionalities in ArchitecTHOR, such as support for new robots, more advanced interaction capabilities, and bug fixes.
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)?	The dataset does not relate to people.
Will older versions of the dataset continue to be supported/hosted/maintained?	Yes. Revision history will be available in the GitHub repository.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?	Yes. The work will be open-sourced and we intend to provide support to help others use and build upon the dataset.

动机
创建该数据集是为了什么目的？	创建 ArchitecTHOR 是为了能够在大型、真实且可交互的家居环境中评估具身智能体。
谁创建并资助了该数据集？	这项工作由 Allen Institute for AI 的 PRIOR 团队创建并资助。具体细节见贡献一节。
构成
构成该数据集的实例代表什么？	数据集中的实例是在 Unity 中搭建的可交互三维房屋，可以与我们定制构建的 AI2-THOR API 一起使用。
总共有多少个实例（如果适用，每一类各有多少）？	一共有 10 套房子，包括 5 套验证房屋和 5 套测试房屋。
该数据集是否包含所有可能的实例，还是来自更大集合的一个样本（不一定是随机样本）？	该数据集是自包含的。
每个实例由什么数据组成？	每套房子的实例都是一个 Unity 场景，其中包括物体摆放、光照和纹理等数据。
每个实例是否带有标签或目标？	没有。
个别实例是否缺失任何信息？	没有。
个别实例之间的关系是否被明确给出（例如用户的电影评分、社交网络链接）？	每套房子都是独立创建的。
是否有推荐的数据划分？	有。房屋本身被划分为 5 套验证房屋和 5 套测试房屋。房子里摆放的资产遵循与 ProcTHOR-10K 相同的训练/验证/测试划分。
该数据集中是否存在任何错误、噪声来源或冗余？	没有。
该数据集是自包含的，还是链接到或依赖于外部资源（例如网站、推文、其他数据集）？	该数据集是自包含的。
该数据集是否包含可能被视为机密的数据？	没有。
该数据集是否包含若直接观看可能冒犯、侮辱、威胁，或可能引发焦虑的数据？	没有。
采集过程
与每个实例相关的数据是如何获得的？	每套房子都由专业 3D 美术手工建模。房子里摆放的大多数物体来自 ProcTHOR 资产数据库。不过，台面、淋浴间和许多橱柜是定制搭建的。
如果该数据集是更大集合中的一个样本，采样策略是什么？	该数据集由从程序化生成脚本中采样的 100 万套房子组成。
数据是在什么时间范围内采集的？	这些房子建于 2022 年初。
是否进行过任何伦理审查流程？	没有。
预处理/清洗/标注
是否对数据做过任何预处理/清洗/标注？	没有。
除了经过预处理/清洗/标注的数据外，是否还保存了“原始”数据？	ArchitecTHOR 房屋没有与之关联的原始数据。
用来预处理/清洗/标注数据的软件是否可用？	是的。我们将开源 ArchitecTHOR 房屋，它们可以在 Unity 中打开和查看。
用途
该数据集是否已经用于任何任务？	是的。请参阅论文第 5 节。
该数据集还可以用于哪些（其他）任务？	这些任务可用于具身人工智能中的任何类型的导航和交互任务。这些房子被做进我们构建的 AI2-THOR 中，意味着 ArchitecTHOR 可以配合任何能在 AI2-THOR 中执行的任务使用。我们尤其认为，ArchitecTHOR 作为评估套件会很有用，既可以评估不同组的 ProcTHOR 任务，也可以评估在不同组程序化生成房屋上训练的智能体。
该数据集的构成，或其采集和预处理/清洗/标注方式，是否有任何可能影响未来用途的地方？	没有。
是否有该数据集不应当被用于的任务？	我们的数据集既可用于商业用途，也可用于非商业用途。
分发
该数据集是否会分发给创建它所代表的实体之外的第三方？	会。ArchitecTHOR 中的全部房屋都将向开源社区发布，并通过我们构建的 AI2-THOR Python API 提供。
该数据集将如何分发？	这些房子将在 GitHub 上分发，并可以作为 Unity 场景打开。
该数据集是否会在版权或其他知识产权许可下分发，以及/或者在适用的使用条款下分发？	ArchitecTHOR 将以 Apache 2.0 许可证发布。
是否有第三方对实例相关数据施加了基于知识产权或其他方面的限制？	没有。
该数据集或其中的个别实例是否受到出口管制或其他监管限制？	没有。
维护
谁将支持/托管/维护该数据集？	作者将提供支持、托管并维护该数据集。
如何联系该数据集的所有者/管理者/维护者？	为匿名审稿而省略。
是否有勘误表？	数据集发布后，我们将使用 GitHub issues 来跟踪相关问题。
该数据集是否会被更新？	ArchitecTHOR 目前处于维护模式，我们预期它不会相对当前状态有太多更新。不过，我们计划在 ArchitecTHOR 中积极支持未来的 AI2-THOR 功能，例如对新机器人的支持、更高级的交互能力，以及缺陷修复。
如果数据集与人有关，对与实例相关数据的保留是否有适用限制（例如，相关个人是否被告知其数据会保留固定期限然后删除）？	该数据集与人无关。
该数据集的旧版本是否会继续得到支持/托管/维护？	会。修订历史将可在 GitHub 仓库中查看。
如果其他人想扩展/增强/基于该数据集继续建设/做出贡献，是否有相应机制？	有。这项工作将开源，我们也打算提供支持，帮助其他人使用并基于该数据集继续建设。

D.2 Analysis

D.2 分析

ArchitecTHOR consists of 10 remarkably high-quality large interactive 3D houses. Figure 36 shows top-down images of the 5 validation houses. Figure 37 shows some examples of images taken inside of 2 kitchens and a bedroom from ArchitecTHOR validation.

ArchitecTHOR 由 10 套质量非常高的大型可交互三维房屋组成。图 36 展示了 5 套验证房屋的俯视图。图 37 展示了从 ArchitecTHOR 验证集中 2 间厨房和 1 间卧室内部拍摄的一些例子。

Figure 37: Examples of images inside of 2 hand-modeled kitchens and 1 hand-modeled bathroom from ArchitecTHOR validation.

图 37：ArchitecTHOR 验证集中 2 间手工建模厨房和 1 间手工建模卫生间内部的图像例子。

ArchitecTHOR was built to be much larger than AI2-iTHOR and RoboTHOR. Figure 38 shows the size comparisons between comparable hand-built scene datasets in AI2-iTHOR and RoboTHOR, measured in navigable area. Notice that the navigable area in ArchitecTHOR is substantially larger than in those. The figure also shows the navigable areas in ProcTHOR-10K span the spectrum of navigable areas between AI2-iTHOR, RoboTHOR, and ArchitecTHOR.

ArchitecTHOR 被建得比 AI2-iTHOR 和 RoboTHOR 大得多。图 38 以可导航面积衡量，展示了它与 AI2-iTHOR 和 RoboTHOR 中可比的手工搭建场景数据集的尺寸比较。可以看到，ArchitecTHOR 中的可导航面积明显更大。该图还表明，ProcTHOR-10K 中的可导航面积覆盖了 AI2-iTHOR、RoboTHOR 和 ArchitecTHOR 之间的谱。

Figure 38: Box plots of the navigable areas for ArchitecTHOR compared to AI2-iTHOR, RoboTHOR, and ProcTHOR-10K. Validation scenes were used to calculate the data for ArchitecTHOR, and training scenes were used to calculate the data for AI2-iTHOR, RoboTHOR, and ProcTHOR-10K.

图 38：ArchitecTHOR 与 AI2-iTHOR、RoboTHOR 和 ProcTHOR-10K 可导航面积的箱线图。ArchitecTHOR 使用验证场景计算数据，AI2-iTHOR、RoboTHOR 和 ProcTHOR-10K 使用训练场景计算数据。

In total, the creation of the 10 houses in ArchitecTHOR took approximately 320 hours of cumulative work by professional 3D artists. Figure 39 shows the time breakdown of which parts of the process took the longest. In particular, the creation of custom assets for the kitchen, such as modeling each of the countertops and cabinets, took the longest amount of time, followed by modeling the 3D structure of house.

总计，ArchitecTHOR 中 10 套房子的创建大约花了专业 3D 美术 320 个小时的累计工作。图 39 展示了这个过程中哪些部分花的时间最长。特别是，为厨房创建定制资产，例如为每个台面和橱柜建模，花的时间最长，其次是为房屋的三维结构建模。

Figure 39: Cumulative time breakdown of the development of ArchitecTHOR across 3D artists.

图 39：ArchitecTHOR 开发过程中 3D 美术累计用时的分解。

Appendix E Input Modalities

附录 E 输入模态

(a) RGB

Appendix F Experiment details

附录 F 实验细节

This section discusses the training details used for our experiments. We discuss baselines, ProcTHOR pre-training, and environment-specific fine-tuning details for the tasks of ObjectNav, ArmPointNav, and rearrangement.

这一节讨论我们实验中使用的训练细节。我们分别说明基线、ProcTHOR 预训练，以及物体导航、ArmPointNav 和重排任务的环境特定微调细节。

F.1 ObjectNav experiments

F.1 物体导航实验

For ObjectNav experiments, agents are given a target object type (e.g. a bed) and are tasked with finding a path in the environment that navigates to that target object type. The task setup matches what is commonly used in embodied AI [27, 7, 58, 96], although we only utilize forward-facing egocentric RGB images at each time step. All ObjectNav experiments are trained with a simulated LoCoBot (Low Cost Robot) agent [12]. The task and training details are described below.

在物体导航实验中，智能体会得到一个目标物体类型（ 例如 一张床），任务是在环境中找到一条导航到该目标物体类型的路径。任务设置与具身人工智能中的常见做法一致 [27, 7, 58, 96]，不过我们在每个时间步只使用朝前的自我中心 RGB 图像。全部物体导航实验都用模拟的 LoCoBot（低成本机器人）智能体训练 [12]。任务与训练细节如下。

Evaluation. Following [6], an ObjectNav task is considered successful if all of the following conditions are met:

评估。 遵循 [6]，当且仅当同时满足以下全部条件时，一次物体导航任务才算成功：

The agent terminates the episode by issuing the Done action.

智能体发出 Done 动作以结束该回合。

The target object type is within a distance of 1 meter from the agent’s camera.

目标物体类型与智能体相机的距离在 1 米以内。

The object is visible in the final frame from the agent’s camera. For instance, if (1) and (2) are satisfied, and the agent is looking in the direction of the object, but the target object is occluded behind a wall, then the task is unsuccessful. Similarly, if the target object type is located in the opposite direction of where the agent is looking, then the task will be unsuccessful.

该物体在智能体相机的最后一帧中可见。例如，如果 (1) 和 (2) 都满足，智能体也朝向该物体，但目标物体被墙挡住，则任务不成功。类似地，如果目标物体类型位于智能体朝向的相反方向，任务也不成功。

We also use SPL to evaluate the efficiency of the agent’s trajectory to the target object. SPL is defined and discussed in [6, 7]. A house may have multiple instances of objects for a given type that the agent can successfully reach. For instance, a house may have multiple bedrooms, where each bedroom includes a bed. Here, if the agent navigates to any of the beds, the episode is successful. To calculate SPL in these scenarios, the shortest path length for the task is the minimum shortest path length from the starting position of the agent to any of the reachable target objects of the given type, regardless of which instance the agent navigates towards.

我们还用 SPL 评估智能体到达目标物体的轨迹效率。SPL 的定义与讨论见 [6, 7]。一座房子里可能有同一类型的多个物体实例是智能体可以成功到达的。例如，一座房子可能有多间卧室，每间卧室都有一张床。这时，只要智能体导航到其中任意一张床，该回合就算成功。在这些情形下计算 SPL 时，任务的最短路径长度是：从智能体起始位置到该类型任一可达目标物体的最短路径长度的最小值，而不论智能体实际走向哪一个实例。

Actions. For each of the trained models, we use a discrete action space consisting of 6 actions, which is shown in Table 6. Following common practice [27, 54], we use stochastic actuation to better simulate noise in the real world.

动作。 对每个训练出的模型，我们使用由 6 个动作组成的离散动作空间，如表 6 所示。遵循常见做法 [27, 54]，我们使用随机执行，以便更好地模拟真实世界中的噪声。

\delta_{m}\sim\mathcal{N}(\mu=0.25,\sigma=0.01)

Action	Description
MoveAhead	Attempts to move the agent forward by $\delta_{m}\sim\mathcal{N}(\mu=0.25,\sigma=0.01)$ meters from its current facing direction. If moving the agent forward by $\delta_{m}$ meters results in a collision in the scene (e.g. there is a wall directly in-front of the agent within $\delta_{m}$ meters), the action fails and the agent’s position remains unchanged.

RotateRight RotateLeft	Rotates the agent rightwards or leftwards from its current forward facing direction by $\delta_{r}\sim\mathcal{N}(\mu=30,\sigma=0.5)$ degrees.

LookUp LookDown	Tilts the agent’s camera up or down by 30 degrees.

Done	A signal from the agent to terminate the episode and evaluate the trajectory from its current state. Discussed in [6].

\delta_{m}\sim\mathcal{N}(\mu=0.25,\sigma=0.01)

动作	说明
MoveAhead	尝试让智能体沿当前朝向前进 $\delta_{m}\sim\mathcal{N}(\mu=0.25,\sigma=0.01)$ 米。如果前进 $\delta_{m}$ 米会在场景中发生碰撞（例如智能体正前方 $\delta_{m}$ 米内有一面墙），则该动作失败，智能体位置保持不变。

RotateRight RotateLeft	使智能体从当前朝向向右或向左旋转 $\delta_{r}\sim\mathcal{N}(\mu=30,\sigma=0.5)$ 度。

LookUp LookDown	将智能体相机向上或向下倾斜 30 度。

Done	智能体发出的结束回合信号，并从其当前状态评估该轨迹。讨论见 [6]。

3\times 224\times 224

3\times 224\times 224

\{0,1,\ldots,T\}

\{0,1,\ldots,T\}

\mathbf{V}^{\prime}_{t}

\mathbf{V}^{\prime}_{t}

\mathbf{b}_{t}

\mathbf{b}_{t}

512{\times}384

512{\times}384

\gamma

Hyperparameter	Value
Discount factor ( $\gamma$ )	0.99
GAE parameter ( $\lambda$ )	0.95
Value loss coefficient	0.5
Entropy loss coefficient	0.01
Clip parameter ( $\epsilon$ )	0.1
Rollout timesteps	20
Rollouts per minibatch	1
Learning rate	3e-4
Optimizer	Adam [60]
Gradient clip norm	0.5

\gamma

超参数	取值
折扣因子 ( $\gamma$ )	0.99
GAE 参数 ( $\lambda$ )	0.95
价值损失系数	0.5
熵损失系数	0.01
裁剪参数 ( $\epsilon$ )	0.1
每次 rollout 时间步	20
每个 minibatch 的 rollout 数	1
学习率	3e-4
优化器	Adam [60]
梯度裁剪范数	0.5

\epsilon=0.1

\epsilon=0.1

r_{t}=\max(0,\min\Delta_{0:t-1}-\Delta_{t})+s_{t}-\rho

r_{t}=\max(0,\min\Delta_{0:t-1}-\Delta_{t})+s_{t}-\rho

\min\Delta_{0:t-1}

\min\Delta_{0:t-1}

\Delta_{t}

\Delta_{t}

s_{t}

s_{t}

\rho

\rho

T=16

T=16

Object Type RoboTHOR HM3D-Semantics AI2-iTHOR ArchitecTHOR Alarm Clock Apple Baseball Bat Basketball Bed Bowl Chair Garbage Can House Plant Laptop Mug Sofa Spray Bottle Television Toilet Vase Table 8: The target objects that are used for each ObjectNav task.

物体类型 RoboTHOR HM3D-Semantics AI2-iTHOR ArchitecTHOR Alarm Clock Apple Baseball Bat Basketball Bed Bowl Chair Garbage Can House Plant Laptop Mug Sofa Spray Bottle Television Toilet Vase 表 8：各物体导航任务所使用的目标物体。

Sampling target object types. To sample the target object type for a given episode, we restrict ourselves to only sampling target object types that have a possibility of leading to a successful episode. For instance, even if there is an object like an apple in the scene, it might be located in the fridge, and so if it was used as a target object, the agent would never succeed because the object would never appear visible in the frame (without any manipulation actions). Therefore, we impose a constraint that the target object must be visible without any form of manipulation.

采样目标物体类型。 为某个回合采样目标物体类型时，我们只采样有可能导向成功回合的目标物体类型。例如，即使场景中有苹果这类物体，它也可能位于冰箱里；如果把它用作目标物体，智能体永远不会成功，因为该物体永远不会在画面中可见（在没有任何操作动作的情况下）。因此我们施加一条约束：目标物体必须在不做任何形式操作时就可见。

0.25\times 0.25

0.25\times 0.25

\epsilon

\epsilon

T

T

HM3D-Semantics. We evaluate on HM3D-Semantics in both a 0-shot and fine-tuned setting using the “ProcTHOR” and “ProcTHOR+Large” architectures described above, these two architectures have slightly different pretraining strategies.

HM3D-Semantics。 我们在零样本和微调两种设置下，用上文描述的“ProcTHOR”和“ProcTHOR+Large”结构在 HM3D-Semantics 上评估；这两种结构的预训练策略略有不同。

T

T

“ProcTHOR+Large” model. We pre-train this model using ProcTHORLarge-10K a variant of ProcTHOR-10K with houses sampled to better align to the distribution of houses in HM3D. In particular, ProcTHORLarge-10K contains 10K procedurally generated houses each of which contains between 4 and 10 rooms (houses in ProcTHORLarge-10K thus tend to be much larger than houses in ProcTHOR-10K). Moreover, during pretraining we only train our agent to navigate to the 6 object categories used in HM3D-Semantics. Fine-tuning is done identically as above. We use an early checkpoint from ProcTHOR pre-training, specifically from after 125 million steps. We performed fine-tuning on a machine with 8 NVIDIA RTX A6000 GPUs for approximately 185M steps taking 85 hours to complete.

“ProcTHOR+Large”模型。 我们用 ProcTHORLarge-10K 预训练该模型，这是 ProcTHOR-10K 的一个变体，房屋采样更接近 HM3D 中的房屋分布。具体来说，ProcTHORLarge-10K 包含 1 万座程序化生成的房屋，每座有 4 到 10 个房间（因此 ProcTHORLarge-10K 中的房屋往往比 ProcTHOR-10K 中的大得多）。此外，预训练时我们只训练智能体导航到 HM3D-Semantics 使用的 6 个物体类别。微调方式与上文相同。我们使用 ProcTHOR 预训练的一个较早检查点，具体是 1.25 亿步之后的检查点。我们在配有 8 块 NVIDIA RTX A6000 GPU 的机器上微调大约 1.85 亿步，耗时 85 小时完成。

T

T

ArchitecTHOR. Since ArchitecTHOR does not include any training scenes, we only use it for evaluation of the ProcTHOR pre-trained model. As shown in Table 8, ArchitecTHOR evaluation uses the full-set of target object types that are used during ProcTHOR pre-training.

ArchitecTHOR。 由于 ArchitecTHOR 不包含任何训练场景，我们只把它用于评估 ProcTHOR 预训练模型。如表 8 所示，ArchitecTHOR 评估使用 ProcTHOR 预训练时的全部目标物体类型。

F.2 ArmPointNav experiments

F.2 ArmPointNav 实验

512

512

\gamma

Hyperparameter	Value
Learning rate	3e-4
Gradient steps	128
Discount factor ( $\gamma$ )	0.99
GAE parameter ( $\lambda$ )	0.95
Gradient clip norm	0.5
Rotation Degrees	45
Step penalty	-0.01
Number of RNN Layers	1
Rollouts per minibatch	1
Optimizer	Adam [60]

\gamma

超参数	取值
学习率	3e-4
梯度步数	128
折扣因子 ( $\gamma$ )	0.99
GAE 参数 ( $\lambda$ )	0.95
梯度裁剪范数	0.5
旋转度数	45
步惩罚	-0.01
RNN 层数	1
每个 minibatch 的 rollout 数	1
优化器	Adam [60]

ProcTHOR pre-training. We pre-train our model on a subset of 7000 houses, on 58 object categories. For each episode, we move the agent to a random location, randomly choose an object in the room that is pickupable, and randomly select a target location. We train our model for 100M frames, running on 4 AWS g4dn.12xlarge machines. Running on a total of 16 GPUs and 192 CPU cores took 3 days of training. Table 9 shows the hyperparameters used for pre-training.

ProcTHOR 预训练。 我们在 7000 座房屋的一个子集上、针对 58 个物体类别预训练模型。对每个回合，我们把智能体移到一个随机位置，随机选择房间里一件可拾取的物体，并随机选择一个目标位置。我们把模型训练 1 亿帧，运行在 4 台 AWS g4dn.12xlarge 机器上。总共使用 16 块 GPU 和 192 个 CPU 核心，训练耗时 3 天。表 9 给出了预训练使用的超参数。

AI2-iTHOR evaluation. We evaluate our model on 20 test rooms of AI2-THOR (5 kitchens, 5 living rooms, 5 bedrooms, 5 bathrooms), on a subset of 28 object categories for a total of 528 tasks. We attempted to perform fine-tuning on AI2-iTHOR, but none of the fine-tuning models performed better than the zero-shot model trained with ProcTHOR pre-training.

AI2-iTHOR 评估。 我们在 AI2-THOR 的 20 个测试房间（5 个厨房、5 个客厅、5 个卧室、5 个卫生间）上评估模型，物体类别子集为 28 个，共 528 个任务。我们尝试过在 AI2-iTHOR 上微调，但没有任何微调模型的表现优于用 ProcTHOR 预训练得到的零样本模型。

F.3 Rearrangement experiments

F.3 重排实验

Following [115, 58], we use imitation learning (IL) to train all models for the 1-phase modality of the task. We divide the full training of the final model into two stages: pre-training in ProcTHOR and fine-tuning in AI2-iTHOR.

遵循 [115, 58]，我们对任务的单阶段模态用模仿学习（IL）训练全部模型。我们把最终模型的完整训练分成两个阶段：在 ProcTHOR 中预训练，以及在 AI2-iTHOR 中微调。

7.4\cdot 10^{-4}

Hyperparameter	Value
Rollout timesteps	64
Batch size	7,680
Learning rate	$7.4\cdot 10^{-4}$
Optimizer	Adam [60]
Gradient clip norm	0.5
BC ${}^{\text{tf}=1}$ steps	200,000
DAgger steps	2,000,000

7.4\cdot 10^{-4}

超参数	取值
每次 rollout 时间步	64
批大小	7,680
学习率	$7.4\cdot 10^{-4}$
优化器	Adam [60]
梯度裁剪范数	0.5
BC ${}^{\text{tf}=1}$ 步数	200,000
DAgger 步数	2,000,000

2\cdot 10^{5}

2\cdot 10^{5}

Running on 6 AWS g4dn.12xlarge (totaling 24 GPUs and 288 virtual CPU cores), pre-training with 240 parallel simulations took 4 days. Table 10 shows the hyperparameters used during pre-training.

在 6 台 AWS g4dn.12xlarge（共计 24 块 GPU 和 288 个虚拟 CPU 核心）上运行，用 240 个并行仿真做预训练耗时 4 天。表 10 给出了预训练期间使用的超参数。

AI2-iTHOR fine-tuning. We use the training dataset provided by [4] (4,000 episodes over 80 single-room scenes), and a small subset of 200 episodes from the also provided full validation set to perform model selection. We fine-tune for 3 million steps with 64-step long rollouts, 6 additional million steps with 96-step long rollouts, and another 6 million steps with 128-step long rollouts.

AI2-iTHOR 微调。 我们使用 [4] 提供的训练数据集（80 个单房间场景上的 4,000 个回合），以及同样提供的完整验证集中的一小批 200 个回合来做模型选择。我们先用 64 步长的 rollout 微调 300 万步，再用 96 步长的 rollout 微调额外 600 万步，最后用 128 步长的 rollout 再微调 600 万步。

Running on 8 Titan X GPUs and 56 virtual CPU cores, fine-tuning with 40 parallel simulations took 16 hours.

在 8 块 Titan X GPU 和 56 个虚拟 CPU 核心上运行，用 40 个并行仿真做微调耗时 16 小时。

Appendix G Performance Benchmark

附录 G 性能基准

3\times 224\times 224

3\times 224\times 224

\pm

	Navigation FPS			Environment Query FPS
Compute	AI2-iTHOR	RoboTHOR	AI2-iTHOR	RoboTHOR	AI2-iTHOR	RoboTHOR
8 GPUs	5,779 $\pm$ 189	9,195 $\pm$ 294	5,411 $\pm$ 190	6,331 $\pm$ 137	463,446 $\pm$ 18,577	412,550 $\pm$ 21,806
1 GPU	1,316 $\pm$ 19	1,648 $\pm$ 11	1,451 $\pm$ 72	1,539 $\pm$ 5	169,092 $\pm$ 4,232	163,660 $\pm$ 3,336
1 Process	180 $\pm$ 9	340 $\pm$ 26	141 $\pm$ 2	217 $\pm$ 1	15,584 $\pm$ 156	15,578 $\pm$ 164
	ProcTHOR-S	ProcTHOR-L	ProcTHOR-S	ProcTHOR-L	ProcTHOR-S	ProcTHOR-L
8 GPUs	8,599 $\pm$ 359	3,208 $\pm$ 127	6,488 $\pm$ 250	2,861 $\pm$ 107	480,205 $\pm$ 19,684	433,587 $\pm$ 18,729
1 GPU	1,427 $\pm$ 74	6,280 $\pm$ 40	1,265 $\pm$ 71	597 $\pm$ 37	160,622 $\pm$ 2,846	157,567 $\pm$ 2,689
1 Process	240 $\pm$ 69	115 $\pm$ 19	180 $\pm$ 42	93 $\pm$ 15	14,825 $\pm$ 199	14,916 $\pm$ 186

\pm

	导航 FPS			环境查询 FPS
计算资源	AI2-iTHOR	RoboTHOR	AI2-iTHOR	RoboTHOR	AI2-iTHOR	RoboTHOR
8 块 GPU	5,779 $\pm$ 189	9,195 $\pm$ 294	5,411 $\pm$ 190	6,331 $\pm$ 137	463,446 $\pm$ 18,577	412,550 $\pm$ 21,806
1 块 GPU	1,316 $\pm$ 19	1,648 $\pm$ 11	1,451 $\pm$ 72	1,539 $\pm$ 5	169,092 $\pm$ 4,232	163,660 $\pm$ 3,336
1 个进程	180 $\pm$ 9	340 $\pm$ 26	141 $\pm$ 2	217 $\pm$ 1	15,584 $\pm$ 156	15,578 $\pm$ 164
	ProcTHOR-S	ProcTHOR-L	ProcTHOR-S	ProcTHOR-L	ProcTHOR-S	ProcTHOR-L
8 块 GPU	8,599 $\pm$ 359	3,208 $\pm$ 127	6,488 $\pm$ 250	2,861 $\pm$ 107	480,205 $\pm$ 19,684	433,587 $\pm$ 18,729
1 块 GPU	1,427 $\pm$ 74	6,280 $\pm$ 40	1,265 $\pm$ 71	597 $\pm$ 37	160,622 $\pm$ 2,846	157,567 $\pm$ 2,689
1 个进程	240 $\pm$ 69	115 $\pm$ 19	180 $\pm$ 42	93 $\pm$ 15	14,825 $\pm$ 199	14,916 $\pm$ 186

Appendix H Broader Impact

附录 H 更广泛的影响

This work focuses on increasing the generalization abilities of robotic agents on various tasks. We specifically focus on robots that operate in household environments. More capable robotic agents can help improve the lives of many by assisting with cooking, cleaning, and providing social interaction. Furthermore, robots can provide a wide range of health benefits. For example, they could give domestic assistance to individuals with physical and mental disabilities and the elderly. They could provide social and emotional support to children, adolescents, and adults, such as delivering personalized educational content, reducing loneliness, and counseling in times of crisis. We can also use home-assisted robots to monitor and provide feedback on people’s physical activity, sleep, and diet.

这项工作关注提升机器人智能体在各种任务上的泛化能力。我们特别关注在家居环境中工作的机器人。能力更强的机器人智能体可以通过协助烹饪、清洁和提供社交互动，帮助改善许多人的生活。此外，机器人可以带来广泛的健康益处。例如，它们可以为有身体和心理障碍的人以及老年人提供居家协助。它们也可以为儿童、青少年和成人提供社交和情感支持，例如提供个性化教育内容、减轻孤独，并在危机时刻给予辅导。我们还可以用居家辅助机器人来监测人们的身体活动、睡眠和饮食，并给出反馈。

However, the adoption of home-assisted robots could have several undesirable social consequences. One is that home-assisted social robots may lead individuals to become more dependant on robots for companionship and care, leading to increased social isolation and loneliness. Another concern is that they may exacerbate existing inequities, as those who can afford to buy and maintain robots will have access to care and assistance that those who cannot will not. Furthermore, because robots would have access to sensitive information about people’s daily lives, they could threaten privacy and security. Finally, robots have the potential to be exploited for malicious intent, such as for mass surveillance or being used for autonomous warfare. As a community, we need to work to reduce the risks of social robots while maximizing the benefits for the common good.

不过，采用居家辅助机器人也可能带来若干不受欢迎的社会后果。其一是，居家辅助社交机器人可能让人们在陪伴和照料上更加依赖机器人，从而导致社交隔离和孤独加重。另一个担忧是，它们可能加剧既有的不平等，因为买得起并维护得起机器人的人将获得其他人无法获得的照料和协助。此外，由于机器人会接触到人们日常生活中的敏感信息，它们可能威胁隐私和安全。最后，机器人有可能被用于恶意目的，例如大规模监控或被用于自主作战。作为一个社区，我们需要努力降低社交机器人的风险，同时尽可能放大它们对公共利益的好处。

Object Type	RoboTHOR	HM3D-Semantics	AI2-iTHOR	ArchitecTHOR
Alarm Clock
Apple
Baseball Bat
Basketball
Bed
Bowl
Chair
Garbage Can
House Plant
Laptop
Mug
Sofa
Spray Bottle
Television
Toilet
Vase

ProcTHOR: Large-Scale Embodied AI Using Procedural Generation

ProcTHOR：用程序化生成实现大规模具身人工智能

Abstract

摘要

1 Introduction

1 引言

2 Related Work

2 相关工作

3 ProcTHOR

4 ProcTHOR-10K

5 Experiments

5 实验

6 Conclusion

6 结论

Acknowledgements

致谢

References

参考文献

Contributions

贡献

Appendix

附录

Appendix A ProcTHOR Assets

附录 A ProcTHOR 资产

Appendix B House Generation

附录 B 房屋生成

B.1 Examples

B.1 例子

B.1.1 3-Room Houses

B.1.1 三房间房屋

B.1.2 4-Room Houses

B.1.2 四房间房屋

B.1.3 5-Room Houses

B.1.3 五房间房屋

B.1.4 6-Room Houses

B.1.4 六房间房屋

B.1.5 7+ Room Houses

B.1.5 七个及以上房间的房屋

B.2 Room Specs

B.2 房间规格

B.3 Sampling Floor Plans

B.3 采样平面图

B.4 Connecting Rooms

B.4 连接房间

B.5 Structure Materials

B.5 结构材质

B.6 Ceiling Height

B.6 层高

B.7 Lighting

B.7 光照

B.8 Object Placement

B.8 物体摆放

B.8.1 Assets

B.8.1 资产

B.8.2 Semantic Asset Groups (SAGs)

B.8.2 语义资产组（SAG）

B.8.3 Floor Object Placement

B.8.3 地面物体摆放

B.8.4 Wall Object Placement

B.8.4 墙面物体摆放

B.8.5 Surface Object Placement

B.8.5 表面物体摆放

B.9 Material and Color Randomization

B.9 材质与颜色随机化

B.10 Object States

B.10 物体状态

B.11 Validator

B.11 校验器

B.12 Limitations and Future Work

B.12 局限与未来工作

Appendix C ProcTHOR Datasheet

附录 C ProcTHOR 数据说明书

Appendix D ArchitecTHOR

附录 D ArchitecTHOR

D.1 Datasheet

D.1 数据说明书

D.2 Analysis

D.2 分析

Appendix E Input Modalities

附录 E 输入模态