PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators

PoliFormer：用 Transformer 扩展同策略强化学习，造就精湛导航智能体

Kuo-Hao Zeng Zichen Zhang Kiana Ehsani Rose Hendrix Jordi Salvador Affiliation:  Alvaro Herrasti Ross Girshick Aniruddha Kembhavi Luca Weihs Affiliation:  PRIOR @ @ Allen Institute for AI Affiliation:  poliformer.allen.ai

Abstract

摘要

We present PoliFormer (Policy Transformer), an RGB-only indoor navigation agent trained end-to-end with reinforcement learning at scale that generalizes to the real-world without adaptation despite being trained purely in simulation. PoliFormer uses a foundational vision transformer encoder with a causal transformer decoder enabling long-term memory and reasoning. It is trained for hundreds of millions of interactions across diverse environments, leveraging parallelized, multi-machine rollouts for efficient training with high throughput. PoliFormer is a masterful navigator, producing state-of-the-art results across two distinct embodiments, the LoCoBot and Stretch RE-1 robots, and four navigation benchmarks. It breaks through the plateaus of previous work, achieving an unprecedented $85.5$% success rate in object goal navigation on the Chores-$\mathbb{S}$ benchmark, a $28.5$% absolute improvement. PoliFormer can also be trivially extended to a variety of downstream applications such as object tracking, multi-object navigation, and open-vocabulary navigation with no finetuning.

我们提出 PoliFormer（Policy Transformer）：一种仅用 RGB 的室内导航智能体，以大规模强化学习端到端训练；尽管纯在仿真中训练，无需适配即可泛化到真实世界。PoliFormer 采用基础视觉 Transformer 编码器，配合因果 Transformer 解码器，以支持长期记忆与推理。它在多样环境中接受数亿次交互训练，借助并行化、多机 rollout 实现高吞吐高效训练。PoliFormer 是精湛导航者：在 LoCoBot 与 Stretch RE-1 两种不同本体、四个导航基准上均给出最先进结果。它突破以往工作的平台期，在 Chores-$\mathbb{S}$ 基准的物体目标导航上达到前所未有的 $85.5\%$ 成功率，绝对提升 $28.5\%$。PoliFormer 还可无需微调地平凡扩展到物体跟踪、多物体导航、开放词汇导航等多种下游应用。

Keywords: Embodied Navigation, On-Policy RL, Transformer Policy

关键词： 具身导航、同策略强化学习、Transformer 策略

1 Introduction

1 引言

Reinforcement Learning (RL) has been used extensively to train embodied robotic agents to complete a variety of indoor navigation tasks. Large-scale, on-policy, end-to-end RL training with DD-PPO [1] enables near-perfect PointNav[^1] performance when using a shallow GRU-based [2] architecture. However, this approach fails to result in the same breakthroughs for harder navigation problems like Object Goal Navigation (ObjectNav [3]) where an agent must explore its environment to locate and navigate to an object of the requested type. RL approaches for ObjectNav have generally not advanced beyond shallow GRU architectures due to challenges presented by training instability and unreasonably long training times with wider and deeper models, such as scaled-up transformers [4].

强化学习（RL）已被广泛用于训练具身机器人智能体完成各类室内导航任务。大规模同策略端到端 RL 训练配合 DD-PPO [1]，在浅层 GRU 架构 [2] 下能使 PointNav[^1] 接近完美。但同一路径未能在更难的导航问题上带来同样突破——例如物体目标导航（ObjectNav [3]）：智能体须探索环境以定位并导航到所请求类型的物体。ObjectNav 上的 RL 方法一般仍停在浅层 GRU，因为更宽更深的模型（如放大的 Transformer [4]）会带来训练不稳与训练时间不合理过长等挑战。

In a departure from on-policy RL, which is sample inefficient and often uses complex reward shaping and auxiliary losses [5], Imitation Learning (IL) has recently shown promise for ObjectNav. Ehsani et al. (2023) [6] demonstrated that the transformer-based SPOC agent, when trained to imitate heuristic shortest-path planners, can be trained stably, is sample efficient, and is significantly more effective than prior RL approaches on their benchmark. SPOC, however, ultimately falls short of mastery, plateauing at a success rate of $\sim$57%. Critically, as we show in our experiments, the performance of SPOC does not seem to improve significantly when further scaling up data and model depth; we suspect this is a consequence of insufficient state-space exploration as expert trajectory datasets frequently contain few examples of error recovery, which can lead to sub-optimal performance due to compounding errors [7] or non-trivial domain shifts during inference.

不同于样本效率低、常需复杂奖励塑形与辅助损失的同策略 RL [5]，模仿学习（IL）近来在 ObjectNav 上显现潜力。Ehsani 等人（2023） [6] 表明，基于 Transformer 的 SPOC 智能体在模仿启发式最短路径规划器时，可稳定训练、样本效率高，并显著优于其基准上先前的 RL 方法。但 SPOC 最终未能达到精通，成功率平台约在 $\sim$57%。关键的是，如我们实验所示，进一步放大数据与模型深度似乎不能显著提升 SPOC；我们怀疑这是状态空间探索不足的后果——专家轨迹数据集往往很少包含纠错示例，推理时可能因误差累积 [7] 或非平凡域偏移导致次优表现。

Figure 1: PoliFormer, a transformer-based policy trained using RL at scale in simulation, achieves significant performance improvements in simulation (bottom-left) and the real world (bottom-right), across two embodiments. SR denotes Success Rate. We scale on-policy RL training across multiple dimensions: (top-left) we observe continual performance improvement with scaling RL training; (top-middle) we leverage hundreds of parallel rollouts for higher throughput; (top-right) we develop a transformer-based policy scaling model parameters to hundreds of millions.

图 1：PoliFormer 是在仿真中大规模用 RL 训练的基于 Transformer 的策略，在仿真（左下）与真实世界（右下）上、跨两种本体均取得显著性能提升。SR 表示成功率。我们从多个维度扩展同策略 RL 训练：（左上）随 RL 训练扩展观察到持续性能提升；（中上）利用数百条并行 rollout 提高吞吐；（右上）开发基于 Transformer 的策略，把模型参数扩展到数亿。

In contrast to IL, RL requires that agents learn via interactive trial-and-error, allowing for deep exploration of the state space. This exploration has the potential to produce agents that can learn behaviors that are superior to those in the expert demonstrations. This raises the question: can we bring together the modeling insights from SPOC-like architectures and the exploratory power of RL to train a masterful navigation agent? Unfortunately, this cannot be done naively due to RL’s sample inefficiency and complexities in using deep (transformer) architectures with RL algorithms. In this work, we develop an effective method for training large transformer-based architectures with RL, breaking through the plateaus of past work, and achieving SoTA results across four benchmarks.

与 IL 相对，RL 要求智能体通过交互试错学习，从而可对状态空间做深度探索。这种探索有潜力产生优于专家示范的行为。于是问题变成：能否把 SPOC 类架构的建模洞见与 RL 的探索能力结合起来，训练出精湛导航智能体？遗憾的是，因 RL 样本效率低、以及深度（Transformer）架构与 RL 算法结合的复杂性，不能天真地这样做。本文我们提出用 RL 有效训练大型 Transformer 架构的方法，突破以往平台期，并在四个基准上达到 SoTA。

Our method, PoliFormer, is a transformer-based model trained with on-policy RL in the AI2-THOR [8] simulator at scale, that can effectively navigate in the real world without any adaptation. We highlight three primary design decisions that make this result possible. (i) Scale in Architecture: Building on the SPOC architecture, we develop a fully transformer-based policy model that uses a powerful visual foundation model (DINOv2 [9], a vision transformer), incorporates a transformer state encoder for improved state summarization, and employs a transformer decoder for explicit temporal memory modeling (Fig. 1, top-right). Importantly, our transformer decoder is causal and uses a KV-cache [10], which allows us to avoid huge computational costs during rollout collection and makes RL training affordable. (ii) Scale in Rollouts: We leverage hundreds of parallel rollouts and large batch sizes, which leads to high training throughput and allows us to train using a huge number of environment interactions (Fig. 1, top-middle). (iii) Scale in Diverse Environment Interactions: Training PoliFormer with RL at scale in $150$k procedurally generated ProcTHOR houses [11] using optimized Objaverse [12, 13, 14] assets results in steady validation set gains (Fig. 1, top-left).

我们的方法 PoliFormer 是基于 Transformer 的模型，在 AI2-THOR [8] 仿真器中大规模用同策略 RL 训练，无需任何适配即可在真实世界有效导航。我们强调使该结果成为可能的三项主要设计决策。(i) 架构规模：在 SPOC 架构之上，我们开发全 Transformer 策略模型——使用强视觉基础模型（DINOv2 [9]，一种视觉 Transformer）、引入 Transformer 状态编码器以改善状态摘要，并用 Transformer 解码器做显式时序记忆建模（图 1，右上）。重要的是，我们的 Transformer 解码器是因果的，并使用 KV-cache [10]，从而在收集 rollout 时避免巨大计算开销，使 RL 训练可负担。(ii) Rollout 规模：我们利用数百条并行 rollout 与大批大小，带来高训练吞吐，并允许用海量环境交互训练（图 1，中上）。(iii) 多样环境交互规模：在 $150$k 个程序生成的 ProcTHOR 房屋 [11] 中，用优化过的 Objaverse [12, 13, 14] 资产大规模 RL 训练 PoliFormer，验证集持续提升（图 1，左上）。

PoliFormer achieves excellent results across multiple navigation benchmarks in simulation. On Chores-$\mathbb{S}$, it achieves an impressive $85.5\%$ Success Rate, a higher than $+28.5\%$ absolute improvement over the previous SoTA model [6]. Similarly, it also obtains SoTA Success Rates on ProcTHOR ( $+8.7\%$), ArchitecTHOR ($+10.0\%$) and AI2-iTHOR ($+6.9\%$); see Fig. 1, left. These results hold across two embodiments, LoCoBot [15] and Stretch RE-1 [16], with distinct action spaces. In the real world (Fig. 1, right), it outperforms ObjectNav baselines in the sim-to-real zero-shot transfer setting using LoCoBot ($+13.3\%$) and Stretch RE-1 ($+33.3\%$).[^2]

PoliFormer 在仿真中多个导航基准上取得优异结果。在 Chores-$\mathbb{S}$ 上成功率达 $85.5\%$，相对先前 SoTA 模型 [6] 绝对提升超过 $+28.5\%$。在 ProcTHOR（$+8.7\%$）、ArchitecTHOR（$+10.0\%$）与 AI2-iTHOR（$+6.9\%$）上同样取得 SoTA 成功率；见图 1 左。这些结果在动作空间不同的两种本体 LoCoBot [15] 与 Stretch RE-1 [16] 上均成立。真实世界中（图 1 右），在仿真到真实零样本迁移设定下，用 LoCoBot（$+13.3\%$）与 Stretch RE-1（$+33.3\%$）均优于 ObjectNav 基线。[^2]

We further train PoliFormer-BoxNav that accepts a bounding box (e.g., from an off-the-shelf open-vocab object detector [18, 19] or VLMs [20, 21]) as its goal specification in place of a given category. This abstraction makes PoliFormer-BoxNav a general purpose navigator that can be “prompted” by an external model akin to the design of Segment Anything [22]. It is extremely effective at exploring its environment and, once it observes a bounding box, beelines towards it. PoliFormer-BoxNav is a leap towards training a foundation model for navigation; with no further training, this general navigator can be used in the real world for multiple downstream tasks such as open vocabulary ObjectNav, multi-target ObjectNav, human following, and object tracking.

我们进一步训练 PoliFormer-BoxNav：用边界框（例如来自现成开放词汇物体检测器 [18, 19] 或 VLM [20, 21]）替代给定类别作为目标规格。这一抽象使 PoliFormer-BoxNav 成为通用导航器，可被外部模型「提示」，类似 Segment Anything [22] 的设计。它极善于探索环境，一旦观察到边界框便径直朝其前进。PoliFormer-BoxNav 是迈向导航基础模型的一步；无需进一步训练，这一通用导航器即可在真实世界用于开放词汇 ObjectNav、多目标 ObjectNav、跟随人、物体跟踪等多种下游任务。

In summary, our contributions include: (i) PoliFormer, a transformer-based policy trained by RL at scale in simulation that achieves SoTA results over four benchmarks in simulation and in the real world across two different embodiments. (ii) A training recipe that enables effective end-to-end policy learning with large-scale neural models via on-policy RL. (iii) A general purpose navigator, PoliFormer-BoxNav, that can be used zero-shot for multiple downstream navigation tasks.

总结，我们的贡献包括：(i) PoliFormer——在仿真中大规模用 RL 训练的基于 Transformer 的策略，在仿真四个基准与真实世界、跨两种不同本体上达到 SoTA；(ii) 一套使大规模神经模型能通过同策略 RL 有效端到端策略学习的训练配方；(iii) 通用导航器 PoliFormer-BoxNav，可零样本用于多种下游导航任务。

2 Related Work

2 相关工作

IL and RL on Embodied Navigation. Recent advancements include Point Goal Navigation [1], Object Goal Navigation [3, 23, 24, 25], Exploration [26, 27, 28], and Social Navigation [29], as well as successful sim-to-real transfer [6] and high performance in various downstream applications [16, 30, 31, 32, 33, 34, 35]. The prevalent approaches to building capable navigation agents include both end-to-end training [1, 6] and modular methods that leverage mapping [36, 37, 38] or off-the-shelf foundation models [39, 40, 41, 42, 43]. In these frameworks, the policy models are typically optimized via Imitation Learning (IL) using expert trajectories [44, 45, 46, 47, 48, 49, 50] or Reinforcement Learning (RL) within interactive environments [29, 51, 52, 53, 54, 55, 56, 57, 58, 59] and with carefully tuned reward shaping and auxiliary losses [5, 60, 61, 62, 63].

具身导航上的 IL 与 RL。 近期进展包括点目标导航 [1]、物体目标导航 [3, 23, 24, 25]、探索 [26, 27, 28] 与社交导航 [29]，以及成功的仿真到真实迁移 [6] 与多种下游应用中的高表现 [16, 30, 31, 32, 33, 34, 35]。构建有能力导航智能体的通行做法包括端到端训练 [1, 6]，以及利用建图 [36, 37, 38] 或现成基础模型 [39, 40, 41, 42, 43] 的模块化方法。这些框架中，策略模型通常经模仿学习、用专家轨迹优化 [44, 45, 46, 47, 48, 49, 50]，或在交互环境中经强化学习优化 [29, 51, 52, 53, 54, 55, 56, 57, 58, 59]，并配合精心调校的奖励塑形与辅助损失 [5, 60, 61, 62, 63]。

Transformer-based Policies for Embodied Agents. Recent works use transformer-based architectures for embodied agents. Decision Transformer (DT [64]) learns a policy offline, conditioning on previous states, actions, and future returns, providing a path for using transformers as sequential decision models. ODT [65] builds on DT and proposes to blend offline pretraining and online finetuning, showing competitive results in D4RL [66]. More recently, MADT [67] shows few-shot online RL finetuning on the offline trained DT in multi-agent game environments. The Skill Transformer [68] learns a policy via IL for long-horizon mobile manipulation tasks. While the focus of these works is control towards specific tasks (relying on IL pre-training), we train from scratch using pure RL targeting a navigation policy that can be used zero-shot for many downstream tasks.

具身智能体的基于 Transformer 的策略。 近期工作为具身智能体使用基于 Transformer 的架构。Decision Transformer（DT [64]）离线学习策略，以前一状态、动作与未来回报为条件，为把 Transformer 用作序列决策模型提供路径。ODT [65] 在 DT 之上提出混合离线预训练与在线微调，在 D4RL [66] 上表现有竞争力。近来 MADT [67] 在多智能体游戏环境中展示对离线训练的 DT 做少样本在线 RL 微调。Skill Transformer [68] 经 IL 学习长时程移动操作任务的策略。这些工作的焦点是面向特定任务的控制（依赖 IL 预训练）；我们则用纯 RL 从零训练，目标是可零样本用于许多下游任务的导航策略。

There has been a huge effort on improving the scale and training stability of policies. GTrXL [69] proposes to augment causal transformer policies with gating layers towards stable RL training in [70]. PDiT [71] proposes to interleave perception and decision transformer-based blocks showing its effectiveness in fully-observed environments. GATO [72] learns a large-scale transformer-based policy on a diverse set of tasks, including locomotion and manipulation. Performer-MPC [73] proposes learnable Model Predictive Control policies with low-rank attention transformers (Performer [74]) as learnable cost functions. SLAP [75] learns a policy for mobile manipulation based on a hybrid design. Radosavovic et al. [76] learn a causal transformer using RL and IL for real-world humanoid locomotion. Their input is proprioceptive state, without visual observations, and their learned policy is intended to be stationary (walking steadily in the real-world). In contrast to existing approaches, we obtain efficient large-scale transformer-based on-policy RL training for a partially-observed, long-horizon, navigation task using RGB sensor inputs and show that our policy can be seamlessly deployed in the real world. We accomplish this without complex inductive biases or multi-task dataset aggregation due to the sheer amount of procedural scene layouts and visual variety of objects in simulation.

已有大量努力改进策略的规模与训练稳定性。GTrXL [69] 提出用门控层增强因果 Transformer 策略，以在 [70] 中稳定 RL 训练。PDiT [71] 提出交错感知与决策 Transformer 块，并在完全可观测环境中展示有效性。GATO [72] 在含运动与操作的多样任务上学习大规模基于 Transformer 的策略。Performer-MPC [73] 提出可学习的模型预测控制策略，用低秩注意力 Transformer（Performer [74]）作可学习代价函数。SLAP [75] 基于混合设计学习移动操作策略。Radosavovic 等人 [76] 用 RL 与 IL 学习因果 Transformer，用于真实世界人形运动；其输入是本体感受状态、无视觉观测，学到的策略意在保持定常（真实世界中稳定行走）。相对现有方法，我们为部分可观测、长时程、以 RGB 传感器为输入的导航任务，获得高效的大规模基于 Transformer 的同策略 RL 训练，并展示策略可无缝部署到真实世界。由于仿真中程序场景布局与物体视觉多样性极大，我们无需复杂归纳偏置或多任务数据聚合即可做到这一点。

Toward Navigation Foundation Models. Many recent works attempt to produce general-purpose “foundational” navigation models. The causal transformer for humanoid outdoor locomotion [76], relying on proprioceptive state, shows emergent behaviors like arm swinging and in-context adaptation. GNM [77] trains image goal-conditioned policies (i.e., conditioning on an image at the desired goal location) by IL across various embodiments, which allows them to scale up the size of their training data and manages to generalize to controlling unseen embodiments. NOMAD [78], which extends ViNT [79] uses a diffusion policy and also uses image goal conditioning. Unlike these works, we navigate from rich RGB image inputs and specify goals with natural-language text (or with bounding boxes), as such goals are far more easily available in real-world settings.

迈向导航基础模型。 许多近期工作试图产出通用「基础」导航模型。面向人形户外运动的因果 Transformer [76] 依赖本体感受状态，展现摆臂、上下文适应等涌现行为。GNM [77] 用 IL 跨多种本体训练图像目标条件策略（即以期望目标位置的图像为条件），从而放大训练数据规模，并泛化到控制未见本体。扩展 ViNT [79] 的 NOMAD [78] 使用扩散策略，同样用图像目标条件。与这些工作不同，我们从丰富的 RGB 图像输入导航，并用自然语言文本（或边界框）指定目标——这类目标在真实世界设定中远更易得。

3 Method

3 方法

Figure 2: PoliFormer is a fully transformer-based policy model. At each timestep $t$, it takes an ego-centric RGB observation $i^{t}$, extracts visual representations $r^{t}$ using a vision transformer model, further encodes state features $s^{t}$ using the visual representations and goal features $g$ (and optional detected bounding box goal features $g_{b}^{t}$), models state belief $b^{t}$ over time, employing a causal transformer decoder, and, finally, predicts action logits $a^{t}$ and a value estimation $e^{t}$ via linear actor and critic heads, respectively. For rollout collection and inference, we leverage the KV-cache [10] as our temporal cache strategy to prevent recomputing the forward pass for all prior timesteps at each new timestep, saving memory and speeding up both training and inference.

图 2：PoliFormer 是全 Transformer 策略模型。每个时间步 $t$，它接收以自我为中心的 RGB 观测 $i^{t}$，用视觉 Transformer 提取视觉表征 $r^{t}$，再结合目标特征 $g$（以及可选的检测到的边界框目标特征 $g_{b}^{t}$）编码状态特征 $s^{t}$，用因果 Transformer 解码器对时间建模状态信念 $b^{t}$，最后经线性 actor 与 critic 头分别预测动作 logits $a^{t}$ 与价值估计 $e^{t}$。收集 rollout 与推理时，我们用 KV-cache [10] 作为时序缓存策略，避免每新时间步对所有先前时间步重算前向，从而节省显存并加速训练与推理。

RL, while effective and intuitive for training policies, has not yet been scaled in embodied AI to the same degree as in other domains. RL has primarily been applied to short-horizon tasks [80, 81], smaller environments [82, 83], or tasks utilizing privileged knowledge [1]. We scale learning across three domains; (i) Network Capacity, via PoliFormer’s deep, high-capacity, model architecture (Sec. 3.1); (ii) Parallelism and Batch size via our highly-parallelized training methodology that leverages large batch sizes for efficiency (Sec. 3.2), and (iii) Training Environments via our optimization of the simulation environment to support high-speed training (Sec. 3.3).

RL 虽对训练策略有效且直观，在具身 AI 中尚未扩展到与其他领域同等程度。RL 主要被用于短时程任务 [80, 81]、较小环境 [82, 83]，或利用特权知识的任务 [1]。我们在三个维度扩展学习：(i) 网络容量——经 PoliFormer 深而高容量的模型架构（第 3.1 节）；(ii) 并行与批大小——经高度并行化、利用大批大小以提高效率的训练方法（第 3.2 节）；(iii) 训练环境——经优化仿真环境以支持高速训练（第 3.3 节）。

3.1 The PoliFormer Architecture

3.1 PoliFormer 架构

We will now detail PoliFormer’s transformer architecture (see Fig. 2), which is inspired by SPOC [6]. While we make several subtle, but important, changes to the SPOC architecture (detailed below), our largest architectural difference is in the transformer decoder: we replace the standard transformer decoder block [4, 84] with the implementation used in the Llama 2 LLM [85]. This change has dramatic implications for training and inference speed, see Sec. 3.1.

下面详述 PoliFormer 的 Transformer 架构（见图 2），其灵感来自 SPOC [6]。虽对 SPOC 架构做了若干微妙但重要的改动（见下），最大的架构差异在 Transformer 解码器：我们用 Llama 2 LLM [85] 中的实现替换标准 Transformer 解码器块 [4, 84]。这一改动对训练与推理速度有巨大影响，见第 3.1 节。

At each timestep $t$, PoliFormer takes an ego-centric RGB observation $i^{t}$ as input and employs a frozen vision transformer model (see Sec. 3.1) to extract a visual representation $r^{t}$. This representation, along with an embedding $g$ of the goal (see Sec. 3.1), are summarized by a transformer state encoder to produce the state representation $s^{t}$. The causal transformer decoder then encodes the state feature $s^{t}$ (along with $s^{0},\ldots,s^{t-1}$) into a belief $b^{t}$. Finally, the linear actor and critic heads project $b^{t}$ to predict action logits $a^{t}$ and a value estimate $e^{t}$, respectively.

每个时间步 $t$，PoliFormer 以自我为中心的 RGB 观测 $i^{t}$ 为输入，用冻结的视觉 Transformer 模型（见第 3.1 节）提取视觉表征 $r^{t}$。该表征与目标的嵌入 $g$（见第 3.1 节）由 Transformer 状态编码器摘要为状态表征 $s^{t}$。因果 Transformer 解码器再把状态特征 $s^{t}$（连同 $s^{0},\ldots,s^{t-1}$）编码为信念 $b^{t}$。最后，线性 actor 与 critic 头把 $b^{t}$ 投影为动作 logits $a^{t}$ 与价值估计 $e^{t}$。

Vision Transformer Model. Inspired by prior work [6, 86], we choose DINOv2 [9] as our visual foundation backbone, because of its remarkable dense prediction and sim-to-real transfer abilities. The visual backbone takes an ego-centric RGB observation $i\in\mathbb{R}^{H\times W\times 3}$ as input and produces a patch-wise representation $r\in\mathbb{R}^{\frac{H}{14}\times\frac{W}{14}\times h}$, where $H$ and $W$ are the observation height and width, and $h$ is the hidden dimension of the visual representation. We reshape the visual representation into a $\ell\times h$ matrix, $\ell=H\cdot W/196$, and project the representation to produce $v\in\mathbb{R}^{\ell\times d}$, where $d$ is the input dimension to the transformer state encoder. For effective sim-to-real transfer, it is important to keep the vision transformer model frozen when training in simulation.

视觉 Transformer 模型。 受先前工作 [6, 86] 启发，我们选择 DINOv2 [9] 作为视觉基础骨干，因其在稠密预测与仿真到真实迁移上表现突出。视觉骨干以自我为中心的 RGB 观测 $i\in\mathbb{R}^{H\times W\times 3}$ 为输入，产出逐块表征 $r\in\mathbb{R}^{\frac{H}{14}\times\frac{W}{14}\times h}$，其中 $H$、$W$ 为观测高宽，$h$ 为视觉表征隐藏维。我们把视觉表征重塑为 $\ell\times h$ 矩阵（$\ell=H\cdot W/196$），再投影得到 $v\in\mathbb{R}^{\ell\times d}$，$d$ 为 Transformer 状态编码器的输入维。为有效仿真到真实迁移，在仿真中训练时保持视觉 Transformer 冻结很重要。

Goal Encoder. For fair comparison, when training agents on ObjectNav benchmarks using the LoCoBot, we follow EmbCLIP [39] and use a one-hot embedding layer to encode the target object category. On benchmarks using the Stretch RE-1 robot, we follow SPOC [6] and use a FLAN-T5 small [87, 88] model to encode the given natural language goal and use the last hidden state from the T5 model as the goal embedding. Before passing the goal embedding to the transformer state encoder, we always project the embedding to the desired dimension $d$, resulting in $g\in\mathbb{R}^{1\times d}$.

目标编码器。 为公平比较，在 LoCoBot 上训练 ObjectNav 基准智能体时，我们沿用 EmbCLIP [39]，用 one-hot 嵌入层编码目标物体类别。在 Stretch RE-1 机器人基准上，沿用 SPOC [6]，用 FLAN-T5 small [87, 88] 编码给定自然语言目标，并以 T5 最后隐藏状态为目标嵌入。在把目标嵌入送入 Transformer 状态编码器前，始终投影到期望维 $d$，得到 $g\in\mathbb{R}^{1\times d}$。

In select experiments (detailed in Sec. 4), we specify the goal object via a bounding box (b-box), either in addition to or as an alternative to text. In this case, the goal encoder processes the b-box using both the sinusoidal positional encoding and the coordinate-type embeddings to embed the top-left, bottom-right b-box coordinates and its area (5 values in total), followed by an MLP to project the hidden feature to the desired dimension $d$, resulting in $g_{b}\in\mathbb{R}^{5\times d}$.

在部分实验中（第 4 节详述），我们用边界框（b-box）指定目标物体，可与文本并用或替代文本。此时目标编码器对 b-box 同时使用正弦位置编码与坐标类型嵌入，嵌入左上、右下 b-box 坐标及其面积（共 5 个值），再经 MLP 投影到期望维 $d$，得到 $g_{b}\in\mathbb{R}^{5\times d}$。

Transformer State Encoder. This module summarizes the state at each timestep as a vector $s\in\mathbb{R}^{d}$. The input to this encoder includes the visual representation $v$, the goal feature $g$, and an embedding $f$ of a STATE token. We concatenate these features together and feed them to a non-causal transformer encoder. This encoder then returns the output corresponding to the STATE token as the state feature vector. Since the transformer state encoder digests both visual and goal features, the produced state feature vector can also be seen as a goal-conditioned visual state representation.

Transformer 状态编码器。 该模块把每个时间步的状态摘要为向量 $s\in\mathbb{R}^{d}$。输入包括视觉表征 $v$、目标特征 $g$，以及 STATE token 的嵌入 $f$。我们将这些特征拼接后送入非因果 Transformer 编码器，再返回对应 STATE token 的输出作为状态特征向量。由于该编码器同时消化视觉与目标特征，产出的状态特征向量也可看成目标条件的视觉状态表征。

Causal Transformer Decoder. We use a causal transformer decoder to perform explicit memory modeling over time. This can enable both long-horizon (e.g., exhaustive exploration with backtracking) and short-horizon (e.g., navigating around an object) planning. Concretely, the causal transformer decoder constructs its state belief $b^{t}$ using the sequence of state features $\textbf{s}=\{s^{j}|_{j=0}^{j=t}\}$ within the same trajectories.

因果 Transformer 解码器。 我们用因果 Transformer 解码器做显式时序记忆建模。这既能支持长时程（例如带回溯的穷尽探索），也能支持短时程（例如绕过物体）规划。具体地，因果 Transformer 解码器用同一轨迹内的状态特征序列 $\textbf{s}=\{s^{j}|_{j=0}^{j=t}\}$ 构造状态信念 $b^{t}$。

Unlike RNN-based causal decoders [89] which, during rollout collection, require only constant time to compute the representation at each timestep, standard implementations of causal transformers require $t^{2}$ time to compute the representation at timestep $t$. This substantial computational cost makes large-scale on-policy RL with causal transformer models slow. To overcome this challenge, we leverage the KV-cache technique [10] to keep past feed-forward results in two cache matrices, one for Keys and one for Values. With a KV-cache, our causal transformer decoder only performs feedforward computations with the most current state feature which results in computation time growing only linearly in $t$ rather than quadratically. While this is still mathematically slower than the constant time required for RNN-based decoders, empirically, we found only a small difference between KV-cache-equipped causal transformers and RNNs in overall training FPS for our setting. Comparing to other temporal cache strategies, we find that KV-Cache offers significant speed improvements. Please see App. B.3 and Fig. 5 for the discussion about the impacts on training speed resulting from different cache strategies.

基于 RNN 的因果解码器 [89] 在收集 rollout 时每时间步只需常数时间计算表征；而标准因果 Transformer 实现在时间步 $t$ 需要 $t^{2}$ 时间。这一巨大计算开销使大规模同策略 RL 配合因果 Transformer 变慢。为克服该挑战，我们利用 KV-cache 技术 [10]，把过去前馈结果保存在两个缓存矩阵中，分别对应 Key 与 Value。有了 KV-cache，因果 Transformer 解码器只需对最新状态特征做前馈，计算时间仅随 $t$ 线性增长而非二次。虽在数学上仍慢于 RNN 解码器所需的常数时间，经验上我们发现：在本设定中，配备 KV-cache 的因果 Transformer 与 RNN 在整体训练 FPS 上只有很小差异。相对其他时序缓存策略，KV-Cache 带来显著加速。关于不同缓存策略对训练速度的影响，见附录 B.3 与图 5。

3.2 Scalable RL Training Recipe

3.2 可扩展的 RL 训练配方

While KV-cache accelerates the causal transformer, this alone is not sufficient to enable efficient and fast RL training. In this section, we describe the methodology we use to achieve faster training. Although each of these individual findings may have been discussed in other works, we emphasize the critical importance of these hyperparameters for efficient training and, consequently, stellar results.

KV-cache 虽加速因果 Transformer，单靠它仍不足以实现高效快速的 RL 训练。本节描述我们用于更快训练的方法。虽这些个别发现可能已在其他工作中讨论过，我们强调这些超参对高效训练——进而对出色结果——的关键重要性。

We parallelize the training process using multi-node training, increasing the number of parallel rollouts by a factor of 4 compared to previous approaches [30, 59, 86] (we use $192$ parallel rollouts for Stretch RE-1 agents and $384$ for LoCoBot agents). We employ the DD-PPO [1] learning algorithm across 4 nodes, utilizing a total of 32 A6000 GPUs and 512 CPU cores. This scaling accelerates the training speed by approximately $3.4\times$, a near-linear gain compared to single-node training; we train for $\sim$4.5 days with 4 nodes, while this would have required 15.3 days with a single node.

我们用多节点训练并行化过程，把并行 rollout 数相对先前方法 [30, 59, 86] 提高 4 倍（Stretch RE-1 智能体用 $192$ 条并行 rollout，LoCoBot 用 $384$ 条）。我们在 4 个节点上采用 DD-PPO [1] 学习算法，共使用 32 张 A6000 GPU 与 512 个 CPU 核心。这一扩展把训练速度提高约 $3.4\times$，相对单节点近线性增益；4 节点约训 $\sim$4.5 天，单节点则需 15.3 天。

Our batch size during training, in number of frames, is equal to the number of parallel rollouts ($R$) multiplied by the length of these rollouts ($T$) and thus, by increasing $R$ as above, we multiplicatively increase the total batch size. We follow SPOC [6] and use a small constant learning rate of $2\cdot 10^{-4}$ throughout the experiments. Instead of annealing the learning rate during training, we instead follow ProcTHOR [11] and increase the batch size by changing the rollout length from $T{=}32$, to $T{=}64$, and finally to $T{=}128$ (resulting in a final batch size of 49,152 frames for LoCoBot agents). We make these increases every 10M steps until we reach $T{=}128$.

训练时批大小（以帧计）等于并行 rollout 数（$R$）乘以 rollout 长度（$T$）；因此如上增大 $R$ 会成倍增大总批大小。我们沿用 SPOC [6]，全程使用较小常数学习率 $2\cdot 10^{-4}$。训练中不退火学习率，而是沿用 ProcTHOR [11]，通过把 rollout 长度从 $T{=}32$ 增到 $T{=}64$、最终 $T{=}128$（LoCoBot 智能体最终批大小为 49,152 帧）来增大批大小。每 10M 步做一次增大，直至 $T{=}128$。

3.3 Scaling Environment Interactions

3.3 扩展环境交互

For the experiments on LoCoBot, we use the original ProcTHOR-10k houses for training for a fair comparison to baselines. With the KV-cache technique, our training steps per second is ${\sim}2.3$k for training PoliFormer, using $384$ rollouts on $32$ GPUs. For the experiments on Stretch, following [6], we use the ProcTHOR-150k houses with ${\sim}40$k annotated Objaverse 3D assets. As on-policy RL requires the agent to interact with the environment on-the-fly, we found the continuous physics simulation used by the Stretch RE-1 agent in AI2-THOR to be too slow to efficiently train at scale. To overcome this, we discretely approximate the agent’s continuous movement via a teleportation-with-collision-checks approach. In particular, to move the agent, we perform a physics cast using a capsule collider representing the agent along the desired movement direction. If the cast does not hit any object (suggesting no potential collisions), we teleport the agent to the target location, otherwise we let AI2-THOR simulate continuous movement as usual. For rotations, we first teleport the agent to the target rotation pose. If the agent’s capsule collides with any object, we teleport the agent back and let the AI2-THOR simulate the continuous rotation. For our tasks, these approximations increase simulation speed by ${\sim}40\%$. We also found that AI2-THOR scene loading and resetting was a significant bottleneck when using Objaverse assets. We streamline AI2-THOR’s Objaverse 3D asset loading pipeline by saving objects in the Msgpack format (natively usable with Unity’s C# backend) as opposed to Python-only Pickle files. This change dramatically decreases the loading time of a new scene from ${\sim}25$s to ${\sim}8$s. With all these changes, our training steps per second increases from ${\sim}550$ to ${\sim}950$ using $192$ rollouts on $32$ GPUs for training Stretch RE-1 agent, reducing the training time by ${\sim}42\%$.

LoCoBot 实验中，为与基线公平比较，训练使用原始 ProcTHOR-10k 房屋。借助 KV-cache，用 $384$ 条 rollout、$32$ 张 GPU 训练 PoliFormer 时，每秒约 ${\sim}2.3$k 训练步。Stretch 实验中，沿用 [6]，使用填入约 ${\sim}40$k 个标注 Objaverse 3D 资产的 ProcTHOR-150k 房屋。因同策略 RL 要求智能体即时与环境交互，我们发现 AI2-THOR 中 Stretch RE-1 智能体所用的连续物理仿真过慢，难以高效大规模训练。为克服这一点，我们用「带碰撞检查的传送」离散近似智能体的连续运动：移动时，用代表智能体的胶囊碰撞体沿期望运动方向做物理投射；若投射未击中任何物体（暗示无潜在碰撞），则把智能体传送到目标位置，否则仍让 AI2-THOR 按常模拟连续运动。旋转时，先把智能体传送到目标旋转姿态；若胶囊与任何物体碰撞，则传回并让 AI2-THOR 模拟连续旋转。对我们的任务，这些近似把仿真速度提高约 ${\sim}40\%$。我们还发现加载与重置 AI2-THOR 场景在使用 Objaverse 资产时是显著瓶颈。我们把 AI2-THOR 的 Objaverse 3D 资产加载管线改流：用 Msgpack 格式保存物体（可被 Unity 的 C# 后端原生使用），而非仅 Python 可用的 Pickle 文件。这把新场景加载时间从约 ${\sim}25$s 大幅降到约 ${\sim}8$s。综合这些改动，用 $192$ 条 rollout、$32$ 张 GPU 训练 Stretch RE-1 智能体时，每秒训练步从约 ${\sim}550$ 升到约 ${\sim}950$，训练时间减少约 ${\sim}42\%$。

4 Results

4 结果

We now present our experimental results. In Sec. 4.1 we demonstrate that scaling RL with PoliFormer produces SoTA results on four simulation benchmarks across two embodiments (LoCoBot and Stretch RE-1). Then, in Sec. 4.2, we show that PoliFormer, despite being trained purely in simulation, transfers very effectively to the real world, outperforming previous work; we again show these results on the above two embodiments. In Sec. 4.3, we provide ablations for various design choices. Finally, we qualitatively show that PoliFormer-BoxNav can be extended to a variety of downstream applications in a zero-shot manner in Sec. 4.4.

下面给出实验结果。第 4.1 节展示：用 PoliFormer 扩展 RL 在两种本体（LoCoBot 与 Stretch RE-1）的四个仿真基准上产出 SoTA。第 4.2 节展示：尽管纯在仿真中训练，PoliFormer 能非常有效地迁移到真实世界，优于先前工作；同样在上述两种本体上展示。第 4.3 节给出各设计选择的消融。最后，第 4.4 节定性展示 PoliFormer-BoxNav 可零样本扩展到多种下游应用。

Baselines. For our baselines, we chose a set of prior works in both imitation learning and reinforcement learning. SPOC [6] is a supervised imitation learning baseline trained on shortest path expert trajectories in AI2THOR. SPOC* is similar to SPOC, but is trained on more expert trajectories (2.3M vs. 100k). We build this baseline to verify if SPOC easily scales with more data. EmbSigLIP [6] is an RL baseline also used by [6], but trained with comparable GPU hours with SPOC. The baselines [11, 63, 86] for LoCoBot-embodiment are all trained by RL using the same training steps used by PoliFormer. We have three experiment configurations for our studies, specifying the goal as (a) natural language instruction text, (b) a b-box for the target object, and (c) b-box & text. Following [6], b-boxes in simulation are the ground-truth b-boxes, and so such models are not comparable fairly with RGB-only models. In the real world we replace GT b-boxes with outputs from the open-vocabulary object detector Detic [18], which uses only RGB images as input, and so all comparisons are fair.

基线。 基线选自模仿学习与强化学习两方面的先前工作。SPOC [6] 是在 AI2THOR 上用最短路径专家轨迹训练的监督模仿学习基线。SPOC* 与 SPOC 类似，但用更多专家轨迹训练（2.3M 对 100k）。我们建该基线以验证 SPOC 是否易随更多数据扩展。EmbSigLIP [6] 也是 [6] 使用的 RL 基线，但 GPU 小时与 SPOC 相当。LoCoBot 本体的基线 [11, 63, 86] 均用与 PoliFormer 相同的训练步数以 RL 训练。研究有三种实验配置，目标分别指定为：(a) 自然语言指令文本；(b) 目标物体的 b-box；(c) b-box 与文本。沿用 [6]，仿真中的 b-box 是真实 b-box，因此此类模型不能与仅 RGB 模型公平比较。真实世界中我们用开放词汇物体检测器 Detic [18] 的输出替换 GT b-box，后者仅以 RGB 图像为输入，故所有比较公平。

Implementation Details. For LoCoBot benchmarks, we follow [86] to train our model and baselines on $10$k training scenes in ProcTHOR houses for $435$M training steps. The evaluation sets consist of $800$ tasks in $20$ scenes for AI2-iTHOR, $1200$ tasks in $5$ scenes for ArchitecTHOR, and $1500$ tasks in $150$ scenes for ProcTHOR-10k. For Stretch-RE1, we train our model and baselines on $150$k ProcTHOR houses populated with Objaverse assets processed by ObjaTHOR. Then, we evaluate on $200$ tasks in $200$ scenes as in [6]. Training takes $4.5$ days on $32$ GPUs. Please see App. B for more details about training, inference, and environment setups.

实现细节。 LoCoBot 基准上，我们按 [86] 在 ProcTHOR 房屋的 $10$k 训练场景上训练模型与基线，共 $435$M 训练步。评估集：AI2-iTHOR 为 $20$ 个场景中的 $800$ 个任务，ArchitecTHOR 为 $5$ 个场景中的 $1200$ 个任务，ProcTHOR-10k 为 $150$ 个场景中的 $1500$ 个任务。Stretch-RE1 上，在填入经 ObjaTHOR 处理的 Objaverse 资产的 $150$k 个 ProcTHOR 房屋上训练，再按 [6] 在 $200$ 个场景的 $200$ 个任务上评估。训练在 $32$ 张 GPU 上耗时 $4.5$ 天。训练、推理与环境设置更多细节见附录 B。

4.1 PoliFormer Achieves SoTA on four Benchmarks

4.1 PoliFormer 在四个基准上达到 SoTA

Table 1(a) shows that, on the Chores-$\mathbb{S}$ benchmark, PoliFormer achieves 28.5% absolute gain in success rate over the previous best model. EmbSigLIP baseline results are taken from [6]; we suspect that its poor performance is, in part, due to its training budget being limited to match the training budget of SPOC.

表 1(a) 显示：在 Chores-$\mathbb{S}$ 基准上，PoliFormer 相对先前最佳模型在成功率上绝对提升 28.5%。EmbSigLIP 基线结果取自 [6]；我们怀疑其表现不佳部分是因为训练预算被限制为与 SPOC 匹配。

Inputs	Model	Loss	Chores-$\mathbb{S}$ ObjectNav
Success (SEL)
RGB+text	SPOC [6]	IL	57.0 (46.2)
SPOC∗	IL	60.0 (30.5)
EmbSigLIP [6]	RL	36.5 (24.5)
PoliFormer	RL	85.5 (61.2)
RGB	SPOC	IL	85.0 (61.4)
+text+b-box	PoliFormer	RL	95.5 (71.4)
RGB+b-box	PoliFormer	RL	92.0 (73.9)

Inputs	Model	ProcTHOR-10k	ArchitecTHOR	AI2-iTHOR
Success (SPL)
RGB+text	ProcTHOR [11][^3]	67.7 (49.0)	55.8 (38.3)	70.0 (57.1)
SGC [63]	70.8 (48.6)	53.8 (34.8)	71.4 (59.3)
EmbCodebook [86]	73.7 (48.4)	58.3 (35.6)	78.4 (23.7)
PoliFormer	82.4 (58.5)	68.3 (45.1)	85.3 (72.7)
RGB	PoliFormer	90.4 (66.6)	81.9 (55.6)	94.9 (83.5)
+text+b-box
RGB+b-box	PoliFormer	87.4 (56.2)	85.7 (47.6)	92.1 (78.6)

(a) Stretch RE-1 on Chores-$\mathbb{S}$

输入	模型	损失	Chores-$\mathbb{S}$ ObjectNav
成功率 (SEL)
RGB+文本	SPOC [6]	IL	57.0 (46.2)
SPOC∗	IL	60.0 (30.5)
EmbSigLIP [6]	RL	36.5 (24.5)
PoliFormer	RL	85.5 (61.2)
RGB	SPOC	IL	85.0 (61.4)
+文本+b-box	PoliFormer	RL	95.5 (71.4)
RGB+b-box	PoliFormer	RL	92.0 (73.9)

输入	模型	ProcTHOR-10k	ArchitecTHOR	AI2-iTHOR
成功率 (SPL)
RGB+文本	ProcTHOR [11][^3]	67.7 (49.0)	55.8 (38.3)	70.0 (57.1)
SGC [63]	70.8 (48.6)	53.8 (34.8)	71.4 (59.3)
EmbCodebook [86]	73.7 (48.4)	58.3 (35.6)	78.4 (23.7)
PoliFormer	82.4 (58.5)	68.3 (45.1)	85.3 (72.7)
RGB	PoliFormer	90.4 (66.6)	81.9 (55.6)	94.9 (83.5)
+文本+b-box
RGB+b-box	PoliFormer	87.4 (56.2)	85.7 (47.6)	92.1 (78.6)

(a) Stretch RE-1 在 Chores-$\mathbb{S}$ 上

PoliFormer works across embodiments. Table 1(b) shows that the remarkable performance of PoliFormer is not limited to one embodiment. The LoCoBot and Stretch-RE1 robots have different body sizes, action spaces, and camera configurations. Nevertheless, PoliFormer is able to achieve 8.7%, 10.0%, and 6.9% absolute gain over the best baseline on ProcTHOR-10k (val), ArchitecTHOR, and AI2-iTHOR.

PoliFormer 跨本体有效。 表 1(b) 显示 PoliFormer 的出色表现不限于一种本体。LoCoBot 与 Stretch-RE1 机器人有不同体型、动作空间与相机配置。尽管如此，PoliFormer 仍能在 ProcTHOR-10k（验证集）、ArchitecTHOR 与 AI2-iTHOR 上相对最佳基线绝对提升 8.7%、10.0% 与 6.9%。

4.2 PoliFormer Generalizes to the Real World

4.2 PoliFormer 泛化到真实世界

We evaluated PoliFormer on two real-world benchmarks for two different embodiments to show its sim-to-real transfer capabilities. We used the same evaluation set of $15$ tasks for LoCoBot ($3$ different starting poses with $5$ different targets), and $18$ tasks for Stretch-RE1 ($3$ different starting poses with $6$ different goal specifications), similar to [17] and [6], respectively. We use no real-world finetuning and the testing environments are unseen houses. In the text-only setting, PoliFormer achieves a remarkable improvement of 33.3% and 13.3% over the baselines for Stretch-RE1 and LoCoBot respectively (Tab. 2(b)). PoliFormer even outperforms Phone2Proc [17], a baseline that has access to privileged information about the layout of the environment.

我们在两种不同本体的两个真实世界基准上评估 PoliFormer，以展示其仿真到真实迁移能力。LoCoBot 用与 [17] 相同的 $15$ 个任务评估集（$3$ 个不同起始姿态 × $5$ 个不同目标），Stretch-RE1 用与 [6] 相同的 $18$ 个任务（$3$ 个起始姿态 × $6$ 个目标规格）。无真实世界微调，测试环境为未见房屋。仅文本设定下，PoliFormer 对 Stretch-RE1 与 LoCoBot 分别相对基线显著提升 33.3% 与 13.3%（表 2(b)）。PoliFormer 甚至优于拥有环境布局特权信息的基线 Phone2Proc [17]。

Model	ProcTHOR-10k (val)	ArchitecTHOR
Vision Backbone	Encoder	Decoder	Success
CLIP (ResNet50)	1x CNN	1x GRU	67.7	55.8
DINOv2 (ViTs)	1x CNN	1x GRU	73.1	60.8
DINOv2 (ViTs)	3x Tx	3x GRU	73.6	59.8
DINOv2 (ViTs)	3x Tx	1x Tx	77.2	59.1
DINOv2 (ViTs)	3x Tx	3x Tx	80.4	63.9
DINOv2 (ViTb)	3x Tx	3x Tx	82.4	68.3

Model	Stretch RE-1	LoCoBot
ProcTHOR [11]	-	26.7
Phone2Proc [17]	-	66.7
SPOC [6]	50.0	-
PoliFormer (ours)	83.3	80.0
SPOC+Detic [6]	83.3	-
PoliFormer +Detic (ours)	88.9	-

(a) Ablations on design choices for scaling model capacity

模型	ProcTHOR-10k（验证）	ArchitecTHOR
视觉骨干	编码器	解码器	成功率
CLIP (ResNet50)	1x CNN	1x GRU	67.7	55.8
DINOv2 (ViTs)	1x CNN	1x GRU	73.1	60.8
DINOv2 (ViTs)	3x Tx	3x GRU	73.6	59.8
DINOv2 (ViTs)	3x Tx	1x Tx	77.2	59.1
DINOv2 (ViTs)	3x Tx	3x Tx	80.4	63.9
DINOv2 (ViTb)	3x Tx	3x Tx	82.4	68.3

模型	Stretch RE-1	LoCoBot
ProcTHOR [11]	-	26.7
Phone2Proc [17]	-	66.7
SPOC [6]	50.0	-
PoliFormer（本文）	83.3	80.0
SPOC+Detic [6]	83.3	-
PoliFormer +Detic（本文）	88.9	-

(a) 扩展模型容量设计选择的消融

4.3 Ablation Studies

4.3 消融研究

The recipe for building the effective RL navigation model is the end product of several design decisions guided by experimentation, the combination resulting in our SoTA performance. Table 2(a), shows the process of scaling PoliFormer. Row 1 vs. Row 2: Moving from the CLIP ResNet-50 visual encoder to the ViT-small DINOv2 improves the success rate by an average of 5.2%. Row 3 vs. Row 5: Using a 3-layer transformer decoder instead of 3-layer GRU increases the performance by 5.5%. Row 4 vs. Row 5: Increasing the number of decoder layers from 1 to 3 increases the performance by 4%. Row 5 vs. Row 6: Using a larger capacity visual encoder (ViT-b instead of ViT-s), results in a 3.2% gain. In all, these changes result in a 13.6% absolute avg. improvement.

构建有效 RL 导航模型的配方，是多轮实验引导下若干设计决策的最终产物，其组合带来我们的 SoTA 表现。表 2(a) 展示扩展 PoliFormer 的过程。第 1 行 对 第 2 行：从 CLIP ResNet-50 视觉编码器换到 ViT-small DINOv2，成功率平均提升 5.2%。第 3 行 对 第 5 行：用 3 层 Transformer 解码器替代 3 层 GRU，性能提升 5.5%。第 4 行 对 第 5 行：解码器层数从 1 增到 3，性能提升 4%。第 5 行 对 第 6 行：用更大容量视觉编码器（ViT-b 替代 ViT-s），再获 3.2%。合计这些改动带来 13.6% 的绝对平均提升。

Has PoliFormer performance saturated? Figure 1 (top-left) shows Chores-$\mathbb{S}$ ObjectNav validation success rate as we train PoliFormer for ${\sim}$700M environment steps. As that plot shows, PoliFormer’s performance does not appear to have converged, and we expect that performance will continue to improve with more compute. This suggests that achieving near-perfect ObjectNav performance may only require further scaling our training approach.

PoliFormer 性能是否已饱和？ 图 1（左上）显示我们训练 PoliFormer 约 ${\sim}$700M 环境步时，Chores-$\mathbb{S}$ ObjectNav 验证成功率。如该图所示，PoliFormer 的表现似乎尚未收敛，我们预期更多算力下性能会继续提升。这暗示接近完美的 ObjectNav 表现或许只需进一步扩展我们的训练方法。

Figure 3: We use PoliFormer-BoxNav zero-shot to find a book with a particular title, navigate to a kitchen, navigate to multiple objects sequentially, and follow a toy car around an office building.

图 3：我们零样本使用 PoliFormer-BoxNav 寻找特定书名的书、导航到厨房、依次导航到多个物体，以及在办公楼中跟随遥控玩具车。

4.4 Scaling PoliFormer to Everyday Tasks

4.4 把 PoliFormer 扩展到日常任务

By specifying PoliFormer’s goal purely using b-boxes, we produce PoliFormer-BoxNav. While PoliFormer-BoxNav lacks the ability to leverage some helpful priors about where objects of certain types should reside (comparing rows 6 & 7 in Table 1(a) we see PoliFormer-BoxNav performs slightly worse than a model having both b-box and text goal specifications), this design decision makes PoliFormer-BoxNav a fully general-purpose, promptable, navigation system that can navigate to any goal specifiable using a b-box. Figure 3 shows four qualitative examples where, by using an open-vocabulary object detector (Detic [18]) and a VLM (GPT-4o [90]), PoliFormer-BoxNav is able to, zero-shot, navigate to (1) a book with a particular title, (2) a given room type, (3) multiple objects sequentially, and (4) a toy car as the car is driven around an office building. Simply by specifying goals as b-boxes and leveraging off-the-shelf systems, we obtain complex navigation behaviors that would otherwise need to be trained for individually: a painful process requiring new reward functions or data collection.

仅用 b-box 指定 PoliFormer 的目标，即得到 PoliFormer-BoxNav。虽 PoliFormer-BoxNav 无法利用「某类物体通常在何处」这类有帮助的先验（比较表 1(a) 第 6、7 行可见：同时有 b-box 与文本目标时略优于仅 b-box），这一设计决策使它成为完全通用、可提示的导航系统，可导航到任何能用 b-box 指定的目标。图 3 给出四个定性例子：借助开放词汇物体检测器（Detic [18]）与 VLM（GPT-4o [90]），PoliFormer-BoxNav 能零样本导航到 (1) 特定书名的书，(2) 给定房间类型，(3) 依次多个物体，以及 (4) 在办公楼中被遥控行驶的玩具车。只需把目标指定为 b-box 并借助现成系统，即可获得原本需分别训练（痛苦地设计新奖励或采集数据）的复杂导航行为。

5 Discussion

5 讨论

Limitations: Training RL agents for long-horizon tasks with a large search space requires extensive compute and demands careful reward shaping. While we believe PoliFormer is capable of scaling to other tasks, it requires crafting new reward models for novel tasks such as manipulation. More discussion on limitations in App. E. Conclusion: In this paper we provide a recipe for scaling RL for long-horizon navigation tasks. Our model, PoliFormer, achieves SoTA results on four simulation benchmarks and two real-world benchmarks across two different embodiments. We also show that PoliFormer has remarkable potential for use in downstream everyday tasks.

局限： 为搜索空间大的长时程任务训练 RL 智能体需要大量算力，并要求仔细的奖励塑形。我们相信 PoliFormer 有能力扩展到其他任务，但对操作等新任务需要打造新奖励模型。更多局限讨论见附录 E。结论： 本文给出扩展长时程导航任务 RL 的配方。我们的模型 PoliFormer 在四个仿真基准与两个真实世界基准、跨两种不同本体上达到 SoTA。我们也展示它在下游日常任务中有显著潜力。

References

参考文献

[1] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. DD-PPO: learning near-perfect pointgoal navigators from 2.5 billion frames. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=H1gX8C4YPr .

[2] K. Cho, B. van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In A. Moschitti, B. Pang, and W. Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL , pages 1724–1734. ACL, 2014. doi: 10.3115/V1/D14-1179 . URL https://doi.org/10.3115/v1/d14-1179 .

[3] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans. ObjectNav revisited: On evaluation of embodied agents navigating to objects. CoRR , abs/2006.13171, 2020. URL https://arxiv.org/abs/2006.13171 .

[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998–6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html .

[5] J. Ye, D. Batra, A. Das, and E. Wijmans. Auxiliary tasks and exploration enable objectnav. CoRR , abs/2104.04112, 2021. URL https://arxiv.org/abs/2104.04112 .

[6] K. Ehsani, T. Gupta, R. Hendrix, J. Salvador, L. Weihs, K.-H. Zeng, K. P. Singh, Y. Kim, W. Han, A. Herrasti, et al. Imitating shortest paths in simulation enables effective navigation and manipulation in the real world. In CVPR , 2024.

[7] N. Rajaraman, Y. Han, L. F. Yang, K. Ramchandran, and J. Jiao. Provably breaking the quadratic error compounding barrier in imitation learning, optimally. ArXiv , abs/2102.12948, 2021. URL https://api.semanticscholar.org/CorpusID:232046264 .

[8] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, A. Kembhavi, A. K. Gupta, and A. Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. ArXiv , abs/1712.05474, 2017. URL https://api.semanticscholar.org/CorpusID:28328610 .

[9] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision, 2023.

[10] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean. Efficiently scaling transformer inference. In MLSys , 2023.

[11] M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, K. Ehsani, J. Salvador, W. Han, E. Kolve, A. Kembhavi, and R. Mottaghi. ProcTHOR: Large-scale embodied AI using procedural generation. In NeurIPS , 2022a. URL http://papers.nips.cc/paper_files/paper/2022/hash/27c546ab1e4f1d7d638e6a8dfbad9a07-Abstract-Conference.html .

[12] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A Universe of Annotated 3D Objects. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13142–13153, 2022b. URL https://api.semanticscholar.org/CorpusID:254685588 .

[13] Y. Yang, F. Sun, L. Weihs, E. VanderBilt, A. Herrasti, W. Han, J. Wu, N. Haber, R. Krishna, L. Liu, C. Callison-Burch, M. Yatskar, A. Kembhavi, and C. Clark. Holodeck: Language guided generation of 3d embodied AI environments. CoRR , abs/2312.09067, 2023. doi: 10.48550/ARXIV.2312.09067 . URL https://doi.org/10.48550/arXiv.2312.09067 .

[14] A. I. for AI. ObjaTHOR: Python package for importing and loading external assets into ai2thor. https://github.com/allenai/objathor , 2024.

[15] A. Gupta, A. Murali, D. P. Gandhi, and L. Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. In NeurIPS , 2018.

[16] C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich. The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environments. In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022 , pages 3150–3157. IEEE, 2022. doi: 10.1109/ICRA46639.2022.9811922 . URL https://doi.org/10.1109/ICRA46639.2022.9811922 .

[17] M. Deitke, R. Hendrix, L. Weihs, A. Farhadi, K. Ehsani, and A. Kembhavi. Phone2Proc: Bringing robust robots into our chaotic world, 2022.

[18] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty-thousand classes using image-level supervision. In S. Avidan, G. J. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX , volume 13669 of Lecture Notes in Computer Science , pages 350–368. Springer, 2022. doi: 10.1007/978-3-031-20077-9_21 . URL https://doi.org/10.1007/978-3-031-20077-9_21 .

[19] M. Minderer, A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection. In NeurIPS , 2023.

[20] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.

[21] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In NeurIPS , 2023.

[22] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollár, and R. B. Girshick. Segment anything. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 3992–4003. IEEE, 2023. doi: 10.1109/ICCV51070.2023.00371 . URL https://doi.org/10.1109/ICCV51070.2023.00371 .

[23] A. Majumdar, G. Aggarwal, B. Devnani, J. Hoffman, and D. Batra. ZSON: zero-shot object-goal navigation using multimodal goal embeddings. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/d0b8f0c8f79d3a621af945cafb669f4b-Abstract-Conference.html .

[24] S. Wani, S. Patel, U. Jain, A. X. Chang, and M. Savva. MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/6e01383fd96a17ae51cc3e15447e7533-Abstract.html .

[25] R. Ramrakhya, E. Undersander, D. Batra, and A. Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 5163–5173. IEEE, 2022. doi: 10.1109/CVPR52688.2022.00511 . URL https://doi.org/10.1109/CVPR52688.2022.00511 .

[26] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning to explore using active neural slam. ICLR , 2020.

[27] B. Yamauchi. A frontier-based approach for autonomous exploration. In Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA’97.’Towards New Computational Principles for Robotics and Automation’ , pages 146–151. IEEE, 1997.

[28] J. Ye, D. Batra, A. Das, and E. Wijmans. Auxiliary tasks and exploration enable objectnav. CoRR , abs/2104.04112, 2021. URL https://arxiv.org/abs/2104.04112 .

[29] X. Puig, E. Undersander, A. Szot, M. D. Cote, T. Yang, R. Partsey, R. Desai, A. W. Clegg, M. Hlavac, S. Y. Min, V. Vondrus, T. Gervet, V. Berges, J. M. Turner, O. Maksymets, Z. Kira, M. Kalakrishnan, J. Malik, D. S. Chaplot, U. Jain, D. Batra, A. Rai, and R. Mottaghi. Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots. CoRR , abs/2310.13724, 2023. doi: 10.48550/ARXIV.2310.13724 . URL https://doi.org/10.48550/arXiv.2310.13724 .

[30] S. Yenamandra, A. Ramachandran, K. Yadav, A. Wang, M. Khanna, T. Gervet, T. Yang, V. Jain, A. W. Clegg, J. M. Turner, Z. Kira, M. Savva, A. X. Chang, D. S. Chaplot, D. Batra, R. Mottaghi, Y. Bisk, and C. Paxton. Homerobot: Open-vocabulary mobile manipulation. CoRR , abs/2306.11565, 2023. doi: 10.48550/ARXIV.2306.11565 . URL https://doi.org/10.48550/arXiv.2306.11565 .

[31] K. Ehsani, A. Farhadi, A. Kembhavi, and R. Mottaghi. Object manipulation via visual target localization. In S. Avidan, G. J. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX , volume 13699 of Lecture Notes in Computer Science , pages 321–337. Springer, 2022. doi: 10.1007/978-3-031-19842-7_19 . URL https://doi.org/10.1007/978-3-031-19842-7_19 .

[32] N. Yokoyama, A. Clegg, E. Undersander, S. Ha, D. Batra, and A. Rai. Adaptive skill coordination for robotic mobile manipulation. ArXiv , abs/2304.00410, 2023. URL https://api.semanticscholar.org/CorpusID:261886226 .

[33] A. Goyal, J. Xu, Y. Guo, V. Blukis, Y. Chao, and D. Fox. RVT: Robotic View Transformer for 3D Object Manipulation. CoRR , abs/2306.14896, 2023. doi: 10.48550/ARXIV.2306.14896 . URL https://doi.org/10.48550/arXiv.2306.14896 .

[34] J. Gu, D. S. Chaplot, H. Su, and J. Malik. Multi-skill Mobile Manipulation for Object Rearrangement. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/pdf?id=Z3IClM_bzvP .

[35] J. Yang, C. Glossop, A. Bhorkar, D. Shah, Q. Vuong, C. Finn, D. Sadigh, and S. Levine. Pushing the limits of cross-embodiment learning for manipulation and navigation. In arXiv preprint arXiv:2402.19432 , 2024.

[36] D. S. Chaplot, R. Salakhutdinov, A. Gupta, and S. Gupta. Neural topological slam for visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020.

[37] M. Chang, T. Gervet, M. Khanna, S. Yenamandra, D. Shah, S. Y. Min, K. Shah, C. Paxton, S. Gupta, D. Batra, et al. GOAT: Go to any thing. arXiv preprint arXiv:2311.06430 , 2023.

[38] T. Gervet, S. Chintala, D. Batra, J. Malik, and D. S. Chaplot. Navigating to objects in the real world. Science Robotics , 2023.

[39] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: CLIP embeddings for embodied AI. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 14809–14818. IEEE, 2022. doi: 10.1109/CVPR52688.2022.01441 . URL https://doi.org/10.1109/CVPR52688.2022.01441 .

[40] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and where pathways for robotic manipulation. In CoRL , 2022.

[41] A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V. Berges, P. Abbeel, J. Malik, D. Batra, Y. Lin, O. Maksymets, A. Rajeswaran, and F. Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? CoRR , abs/2303.18240, 2023. doi: 10.48550/ARXIV.2303.18240 . URL https://doi.org/10.48550/arXiv.2303.18240 .

[42] B. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K. Lee, Y. Kuang, S. Jesmonth, N. J. Joshi, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu. Do as I can, not as I say: Grounding language in robotic affordances. In K. Liu, D. Kulic, and J. Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand , volume 205 of Proceedings of Machine Learning Research , pages 287–318. PMLR, 2022. URL https://proceedings.mlr.press/v205/ichter23a.html .

[43] S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, et al. PIVOT: Iterative visual prompting elicits actionable knowledge for vlms. In arXiv preprint arXiv:2402.07872 , 2024.

[44] S. Srivastava, C. Li, M. Lingelbach, R. Mart’in-Mart’in, F. Xia, K. Vainio, Z. Lian, C. Gokmen, S. Buch, C. K. Liu, S. Savarese, H. Gweon, J. Wu, and L. Fei-Fei. BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments. In Conference on Robot Learning , 2021. URL https://api.semanticscholar.org/CorpusID:236957374 .

[45] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, M. Anvari, M. Hwang, M. Sharma, A. Aydin, D. Bansal, S. Hunter, K. Kim, A. Lou, C. R. Matthews, I. Villa-Renteria, J. H. Tang, C. Tang, F. Xia, S. Savarese, H. Gweon, K. Liu, J. Wu, and L. Fei-Fei. BEHAVIOR-1K: A Benchmark for Embodied AI with 1, 000 Everyday Activities and Realistic Simulation. In K. Liu, D. Kulic, and J. Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand , volume 205 of Proceedings of Machine Learning Research , pages 80–93. PMLR, 2022. URL https://proceedings.mlr.press/v205/li23a.html .

[46] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. S. Ryoo, G. Salazar, P. R. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. T. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-1: robotics transformer for real-world control at scale. In K. E. Bekris, K. Hauser, S. L. Herbert, and J. Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023 , 2023a. doi: 10.15607/RSS.2023.XIX.025 . URL https://doi.org/10.15607/RSS.2023.XIX.025 .

[47] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. S. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. T. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-2: vision-language-action models transfer web knowledge to robotic control. CoRR , abs/2307.15818, 2023b. doi: 10.48550/ARXIV.2307.15818 . URL https://doi.org/10.48550/arXiv.2307.15818 .

[48] O. X. Collaboration, A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Raj, A. Singh, A. Brohan, A. Raffin, A. Wahid, B. Burgess-Limerick, B. Kim, B. Schölkopf, B. Ichter, C. Lu, C. Xu, C. Finn, C. Xu, C. Chi, C. Huang, C. Chan, C. Pan, C. Fu, C. Devin, D. Driess, D. Pathak, D. Shah, D. Büchler, D. Kalashnikov, D. Sadigh, E. Johns, F. Ceola, F. Xia, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Schiavi, G. Kahn, H. Su, H. Fang, H. Shi, H. B. Amor, H. I. Christensen, H. Furuta, H. Walke, H. Fang, I. Mordatch, I. Radosavovic, and et al. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. CoRR , abs/2310.08864, 2023. doi: 10.48550/ARXIV.2310.08864 . URL https://doi.org/10.48550/arXiv.2310.08864 .

[49] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. DROID: A large-scale in-the-wild robot manipulation dataset. In arXiv preprint arXiv:2403.12945 , 2024.

[50] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In RSS , 2024.

[51] J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao, X. Yuan, P. Xie, Z. Huang, R. Chen, and H. Su. ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/pdf?id=b_CQDy9vrD1 .

[52] C. Li, F. Xia, R. Martín-Martín, M. Lingelbach, S. Srivastava, B. Shen, K. E. Vainio, C. Gokmen, G. Dharan, T. Jain, A. Kurenkov, C. K. Liu, H. Gweon, J. Wu, L. Fei-Fei, and S. Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. In A. Faust, D. Hsu, and G. Neumann, editors, Conference on Robot Learning, 8-11 November 2021, London, UK , volume 164 of Proceedings of Machine Learning Research , pages 455–465. PMLR, 2021. URL https://proceedings.mlr.press/v164/li22b.html .

[53] B. Shen, F. Xia, C. Li, R. Martín-Martín, L. Fan, G. Wang, C. Pérez-D’Arpino, S. Buch, S. Srivastava, L. Tchapmi, M. Tchapmi, K. Vainio, J. Wong, L. Fei-Fei, and S. Savarese. iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021, Prague, Czech Republic, September 27 - Oct. 1, 2021 , pages 7520–7527. IEEE, 2021. doi: 10.1109/IROS51168.2021.9636667 . URL https://doi.org/10.1109/IROS51168.2021.9636667 .

[54] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. M. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. X. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training Home Assistants to Rearrange their Habitat. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 251–266, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/021bbc7ee20b71134d53e20206bd6feb-Abstract.html .

[55] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A SimulAted Part-based Interactive ENvironment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

[56] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba. VirtualHome: Simulating Household Activities via Programs. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 8494–8502. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00886 . URL http://openaccess.thecvf.com/content_cvpr_2018/html/Puig_VirtualHome_Simulating_Household_CVPR_2018_paper.html .

[57] C. Gan, J. Schwartz, S. Alter, D. Mrowca, M. Schrimpf, J. Traer, J. D. Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, K. Kim, E. Wang, M. Lingelbach, A. Curtis, K. T. Feigelis, D. Bear, D. Gutfreund, D. D. Cox, A. Torralba, J. J. DiCarlo, J. Tenenbaum, J. H. McDermott, and D. Yamins. ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual , 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/735b90b4568125ed6c3f678819b6e058-Abstract-round1.html .

[58] Z. Zhang and L. Weihs. When learning is out of reach, reset: Generalization in autonomous visuomotor reinforcement learning. arXiv preprint arXiv: Arxiv-2303.17600 , 2023.

[59] M. Khanna, Y. Mao, H. Jiang, S. Haresh, B. Schacklett, D. Batra, A. Clegg, E. Undersander, A. X. Chang, and M. Savva. Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. In CVPR , 2024.

[60] Z. D. Guo, M. G. Azar, B. Piot, B. A. Pires, and R. Munos. Neural predictive belief representations. ICLR , 2019.

[61] Z. D. Guo, B. A. Pires, B. Piot, J.-B. Grill, F. Altché, R. Munos, and M. G. Azar. Bootstrap latent-predictive representations for multitask reinforcement learning. In International Conference on Machine Learning , 2020.

[62] K.-H. Zeng, L. Weihs, A. Farhadi, and R. Mottaghi. Pushing it out of the way: Interactive visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

[63] K. P. Singh, J. Salvador, L. Weihs, and A. Kembhavi. Scene Graph Contrastive Learning for Embodied Navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 10884–10894, October 2023.

[64] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 15084–15097, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/7f489f642a0ddb10272b5c31057f0663-Abstract.html .

[65] Q. Zheng, A. Zhang, and A. Grover. Online decision transformer. In ICML , 2022.

[66] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: Datasets for deep data-driven reinforcement learning. ArXiv , abs/2004.07219, 2020. URL https://api.semanticscholar.org/CorpusID:215827910 .

[67] L. Meng, M. Wen, C. Le, X. Li, D. Xing, W. Zhang, Y. Wen, H. Zhang, J. Wang, Y. Yang, et al. Offline pre-trained multi-agent decision transformer. Machine Intelligence Research , 2023.

[68] X. Huang, D. Batra, A. Rai, and A. Szot. Skill transformer: A monolithic policy for mobile manipulation. In CVPR , 2023.

[69] E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury, et al. Stabilizing transformers for reinforcement learning. In ICML , 2020.

[70] C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, J. Schrittwieser, K. Anderson, S. York, M. Cant, A. Cain, A. Bolton, S. Gaffney, H. King, D. Hassabis, S. Legg, and S. Petersen. Deepmind lab. ArXiv , abs/1612.03801, 2016. URL https://api.semanticscholar.org/CorpusID:3221395 .

[71] H. Mao, R. Zhao, Z. Li, Z. Xu, H. Chen, Y. Chen, B. Zhang, Z. Xiao, J. Zhang, and J. Yin. PDiT: Interleaving perception and decision-making transformers for deep reinforcement learning. In AAMAS , 2024.

[72] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. In TMLR , 2022.

[73] X. Xiao, T. Zhang, K. Choromanski, E. Lee, A. Francis, J. Varley, S. Tu, S. Singh, P. Xu, F. Xia, et al. Learning model predictive controllers with real-time attention for real-world navigation. In CoRL , 2023.

[74] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, et al. Rethinking attention with performers. In ICLR , 2021.

[75] P. Parashar, V. Jain, X. Zhang, J. Vakil, S. Powers, Y. Bisk, and C. Paxton. Spatial-language attention policies for efficient robot learning. In CoRL , 2023.

[76] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics , 2024.

[77] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. GNM: A general navigation model to drive any robot. In ICRA , 2023.

[78] A. Sridhar, D. Shah, C. Glossop, and S. Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In ICRA , 2023.

[79] D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. Vint: A foundation model for visual navigation. In CoRL , 2023.

[80] T. Haarnoja, A. Zhou, S. Ha, J. Tan, G. Tucker, and S. Levine. Learning to walk via deep reinforcement learning. ArXiv , abs/1812.11103, 2018. URL https://api.semanticscholar.org/CorpusID:57189150 .

[81] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. J. Mach. Learn. Res. , 17:39:1–39:40, 2015. URL https://api.semanticscholar.org/CorpusID:7242892 .

[82] W. Yang, X. Wang, A. Farhadi, A. K. Gupta, and R. Mottaghi. Visual semantic navigation using scene priors. ArXiv , abs/1810.06543, 2018. URL https://api.semanticscholar.org/CorpusID:53116049 .

[83] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv , abs/1312.5602, 2013. URL https://api.semanticscholar.org/CorpusID:15238391 .

[84] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , 2019.

[85] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR , abs/2307.09288, 2023. doi: 10.48550/ARXIV.2307.09288 . URL https://doi.org/10.48550/arXiv.2307.09288 .

[86] A. Eftekhar, K.-H. Zeng, J. Duan, A. Farhadi, A. Kembhavi, and R. Krishna. Selective visual representations improve convergence and generalization for embodied ai. arXiv preprint arXiv:2311.04193 , 2023.

[87] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR , 2020.

[88] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling instruction-finetuned language models, 2022. URL https://arxiv.org/abs/2210.11416 .

[89] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In arXiv preprint arXiv:1412.3555 , 2014.

[90] OpenAI. GPT-4 technical report. CoRR , abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774 . URL https://doi.org/10.48550/arXiv.2303.08774 .

[91] L. Weihs, J. Salvador, K. Kotar, U. Jain, K.-H. Zeng, R. Mottaghi, and A. Kembhavi. AllenAct: A framework for embodied ai research. arXiv preprint arXiv:2008.12760 , 2020.

Appendices for PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators

PoliFormer：用 Transformer 扩展同策略强化学习，造就精湛导航智能体 附录

These appendices contain additional information about our:

这些附录包含关于以下内容的补充信息：

• • Zero-shot real-world applications (App. A),
• • Training procedure (App. B),
• • Environment, benchmarks, and quantitative real-world experiments (App. C),
• • Simulation evaluations (App. D), and
• • Limitations (App. E).

• • 零样本真实世界应用（附录 A），
• • 训练流程（附录 B），
• • 环境、基准与定量真实世界实验（附录 C），
• • 仿真评估（附录 D），以及
• • 局限（附录 E）。

Please find our project website (see the poliformer.allen.ai) that contains

请访问项目网站（见 poliformer.allen.ai），其中包含

• • Six real-world qualitative videos where PoliFormer performs the everyday tasks of Section 4.4 (recall also Figure 3), and
• • Four qualitative videos in simulation showing our PoliFormer’s behavior in the four benchmark environments (CHORES, ProcTHOR, AI2-iTHOR, and ArchitecTHOR).

• • 六段真实世界定性视频，展示 PoliFormer 完成第 4.4 节的日常任务（亦见图 3），以及
• • 四段仿真定性视频，展示 PoliFormer 在四个基准环境（CHORES、ProcTHOR、AI2-iTHOR、ArchitecTHOR）中的行为。

Appendix A Details about Zero-shot Real-world Downstream Applications using an Open-Vocab Object Detector and VLM

附录 A 用开放词汇物体检测器与 VLM 做零样本真实世界下游应用的细节

By specifying PoliFormer’s goal purely using b-boxes, we produce PoliFormer-BoxNav. PoliFormer-BoxNav is extremely effective at exploring its environment and, once it observes a bounding box, takes a direct and efficient path towards it. We now describe how we utilize this behavior to apply PoliFormer-BoxNav zero-shot to a variety of downstream applications by leveraging an open vocabulary object detector (Detic [18]) and a VLM (GPT-4o [90]).

仅用 b-box 指定 PoliFormer 的目标，即得到 PoliFormer-BoxNav。它极善于探索环境，一旦观察到边界框便走直接高效的路径朝其前进。下面描述如何借助开放词汇物体检测器（Detic [18]）与 VLM（GPT-4o [90]），把这一行为零样本用于多种下游应用。

Open Vocabulary ObjectNav. To perform open vocabulary object navigation (i.e., where one must navigate to any given object type), we simply prompt the Detic object detector with the novel object type, for example, Bicycle. As PoliFormer-BoxNav relies on the b-box as its goal specification, it finds a bicycle in the scene smoothly.

开放词汇 ObjectNav。 要执行开放词汇物体导航（即导航到任意给定物体类型），只需用新物体类型（例如 Bicycle）提示 Detic 检测器。因 PoliFormer-BoxNav 依赖 b-box 作为目标规格，它能顺利在场景中找到自行车。

Multi-target ObjectNav. To enable multi-target object goal navigation, we make a few simple modifications to the inputs and output of the Detic detector. On the input side, we query with multiple prompts simultaneously (one for each object type); for instance, HousePlant, Toilet, and Sofa, as shown in Fig. 3 (bottom-left). We then, on the output side, only return the b-box with the highest confidence score. Since the returned b-box also contains the predicted object type, we know what the target object the agent finds is when issuing a Done action. Therefore, we remove the found target from the list of target types, and reset the PoliFormer’s KV-cache. If the agent issues a Done action without a detected b-box, we terminate the episode and consider it a failure. As a result, the agent is required to find all the targets from the list of target types to succeed in an episode.

多目标 ObjectNav。 为实现多目标物体导航，我们对 Detic 检测器的输入与输出做少量简单修改。输入侧同时用多个提示查询（每种物体类型一个）；例如图 3（左下）中的 HousePlant、Toilet 与 Sofa。输出侧只返回置信度最高的 b-box。因返回的 b-box 也含预测物体类型，发出 Done 动作时可知智能体找到了何种目标。于是从目标类型列表中移除已找到目标，并重置 PoliFormer 的 KV-cache。若智能体在无检测到 b-box 时发出 Done，则终止回合并判失败。因此，智能体须从目标类型列表中找到全部目标才算成功。

Human Following. We change the Detic prompt to Person. Once a b-box is detected, PoliFormer drives the agent to approach it. Our experiment participant continues to walk away, so the agent keeps approaching them to minimize the distance.

跟随人。 把 Detic 提示改为 Person。一旦检测到 b-box，PoliFormer 驱动智能体靠近。实验参与者持续走开，智能体便持续靠近以缩短距离。

Object Tracking. In this example, we control a remote control car that moves in the environment, and prompt the agent to find the car. Similar to Human Following, we change the prompt to Toy Truck in this example. As a result, the agent keeps trying to move closer to the detected b-box of the RC car, while avoiding collisions with objects in the dynamic scene.

物体跟踪。 本例中我们控制环境中移动的遥控车，并提示智能体寻找该车。与跟随人类似，本例把提示改为 Toy Truck。于是智能体持续试图靠近检测到的遥控车 b-box，同时避免与动态场景中的物体碰撞。

Room Navigation. In this example, shown in Fig. 3 (middle-left), we provide no detections to the agent. As the agent sees no detections, it continuously explores the scene. As the agent explores, we query GPT4-o every 5 timesteps with the prompt Am I in a Kitchen? Please return Yes or No. with the most recent visual observation. Once GPT-4o returns Yes, the agent issues a Done action to end the episode.

房间导航。 本例见图 3（中左）：不向智能体提供任何检测。因看不到检测，智能体持续探索场景。探索时，我们每 5 个时间步用最近视觉观测向 GPT-4o 查询提示「Am I in a Kitchen? Please return Yes or No.」。一旦 GPT-4o 返回 Yes，智能体发出 Done 结束回合。

Instance Description Navigation. In this example, shown in Fig. 3 (upper-left), the agent is prompted to find a specific book titled “Humans”. Detic can generate open-vocabulary bounding boxes using instance-level descriptions but we found that doing this alone leads to high false-positive rates. To reduce these errors, we use GPT4-o to filter positive detections from Detic. In particular, a sample filtering prompt is “Is there a book titled “Humans” in this image? Please return Yes or No.”. We find this combination works well in practice. The agent, not GPT-4o, remains responsible for deciding when it has successfully completed its task, and in the Fig. 3 example sees many books in its search but perseveres and eventually finds the correct one.

实例描述导航。 本例见图 3（左上）：提示智能体寻找书名为 “Humans” 的特定书。Detic 可用实例级描述生成开放词汇边界框，但我们发现仅靠它假阳性率很高。为降低这些错误，我们用 GPT-4o 过滤 Detic 的正检测。例如过滤提示为「Is there a book titled “Humans” in this image? Please return Yes or No.」。实践中该组合效果良好。决定任务是否成功完成仍由智能体而非 GPT-4o 负责；图 3 例子中智能体搜索时见到许多书，但仍坚持并最终找到正确的那本。

Figure 4: Attention Masks for training with block lower triangular structure.

图 4：训练时块下三角结构的注意力掩码。

Appendix B Additional Training Details

附录 B 补充训练细节

B.1 Reward Shaping

B.1 奖励塑形

For reward shaping, we follow EmbCodebook [86] and ProcTHOR [11] and use the implementation in AllenAct [91]: $\mathcal{R}_{penalty}+\mathcal{R}_{success}+\mathcal{R}_{distance}$, where $\mathcal{R}_{penalty}=-0.01$ encourages an efficient navigation, $\mathcal{R}_{success}=10$ when the agent successfully completes the task ($=0$ otherwise), and $\mathcal{R}_{distance}$ is the change of L2 distances from target between two consecutive steps. Note that we only provide a nonzero $\mathcal{R}_{distance}$ if the new distance is less than previously seen in the episode. We do not enforce a negative reward for increasing distance. This formulation encourages exploration.

奖励塑形上，我们沿用 EmbCodebook [86] 与 ProcTHOR [11]，并使用 AllenAct [91] 中的实现：$\mathcal{R}_{penalty}+\mathcal{R}_{success}+\mathcal{R}_{distance}$，其中 $\mathcal{R}_{penalty}=-0.01$ 鼓励高效导航，$\mathcal{R}_{success}=10$ 在智能体成功完成任务时给出（否则 $=0$），$\mathcal{R}_{distance}$ 为连续两步相对目标的 L2 距离变化。注意：仅当新距离小于本回合先前所见距离时，才给出非零 $\mathcal{R}_{distance}$。我们不对距离增大施加负奖励。该形式鼓励探索。

B.2 Episodic Attention Mask

B.2 回合注意力掩码

During training, to ensure that the causal transformer decoder cannot access observations or states across different episodes, we construct the episodic attention mask to only allow the past experiences within the same episode to be attended. In Fig. 4, we show a couple of possible rollouts collected during training. With the episodic attention mask, observations and states in an episode can only attend to previous ones within the same episode, in contrast with a naive causal mask where they could also potentially attend to observations and states in previous episodes.

训练时，为保证因果 Transformer 解码器无法跨不同回合访问观测或状态，我们构造回合注意力掩码，只允许同一回合内的过去经验被注意。图 4 展示训练时收集到的若干可能 rollout。有了回合注意力掩码，一回合内的观测与状态只能注意同一回合内先前的那些；相对朴素因果掩码，后者还可能注意先前回合的观测与状态。

Figure 5: Different temporal cache strategies and their impact on the training speed. We ablate four different cache strategies, including (i) No-Cache, (ii) Feature-Cache, (iii) State-Cache, and (iv) KV-Cache, shown at top. The bottom chart shows the training Step per Second (SPS) achieved by different strategies, on both LoCoBot and Stretch RE-1 agents.

图 5：不同时序缓存策略及其对训练速度的影响。我们消融四种缓存策略：(i) No-Cache，(ii) Feature-Cache，(iii) State-Cache，(iv) KV-Cache，见上方。下方图显示不同策略在 LoCoBot 与 Stretch RE-1 智能体上达到的每秒训练步（SPS）。

B.3 Different Temporal Cache Strategies

B.3 不同时序缓存策略

Besides the KV-Cache, in this subsection, we ablate four different temporal cache strategies (shown in Fig. 5 top):

除 KV-Cache 外，本小节消融四种不同时序缓存策略（图 5 上方）：

(i) No-Cache, we cache the raw frames (visual observation $i^{t}$) to provide past experiences for the causal transformer decoder. Therefore, PoliFormer has to rerun the feedforward across all modules with the cached frames and the latest visual observation at each timestep.
(ii) Feature-Cache, we cache the features, including visual representation $v^{t}$ and goal embedding $g^{t}$ (as well as $g_{b}^{t}$). In this case, PoliFormer needs to recompute the transformer state encoder and causal transformer decoder with the cached features and the latest feature at each timestep.
(iii) State-Cache, we cache the state feature vector $s^{t}$. Since it is right before the causal transformer decoder, PoliFormer only requires to pass the cache state features and the latest state feature vector to the decoder, without recomputing the visual transformer model, goal encoder, and transformer state encoder.
(iv) KV-Cache, as described in Sec. 3.1, we cache the Keys and Values inside the causal transformer decoder, further reducing the required computation time for the transformer decoder from $t^{2}$ to $t$ theoretically. In addition, since this strategy operates at very end, all the recomputations required by modules preceding causal transformer decoder can be saved.

(i) No-Cache：缓存原始帧（视觉观测 $i^{t}$）为因果 Transformer 解码器提供过去经验。因此 PoliFormer 每个时间步须用缓存帧与最新视觉观测对所有模块重跑前馈。
(ii) Feature-Cache：缓存特征，包括视觉表征 $v^{t}$ 与目标嵌入 $g^{t}$（以及 $g_{b}^{t}$）。此时 PoliFormer 每个时间步须用缓存特征与最新特征重算 Transformer 状态编码器与因果 Transformer 解码器。
(iii) State-Cache：缓存状态特征向量 $s^{t}$。因它恰在因果 Transformer 解码器之前，PoliFormer 只需把缓存状态特征与最新状态特征向量送入解码器，无需重算视觉 Transformer、目标编码器与 Transformer 状态编码器。
(iv) KV-Cache：如第 3.1 节所述，在因果 Transformer 解码器内缓存 Key 与 Value，理论上进一步把解码器所需计算时间从 $t^{2}$ 降到 $t$。此外，因该策略作用在最末端，因果 Transformer 解码器之前各模块所需的重算均可省去。

The speed profile results are shown in Fig. 5 bottom. It clearly shows that placing the cache closer to the causal transformer decoder improves the training efficiency significantly.

速度剖面结果见图 5 下方。清楚显示：把缓存放得越靠近因果 Transformer 解码器，训练效率提升越显著。

Training and Model Details
Parameter	Value
Allowed Steps	600 (Stretch RE-1), 500 (LoCoBot)
Total Rollouts	192 (Stretch RE-1), 384 (LoCoBot)
Learing Rate	0.002
Mini Batch per Update	1
Update Repeats	4
Max Gradient Norm	0.5
Discount Value Factor $\gamma$	0.99
GAE $\lambda$	0.95
PPO Surrogate Objective Clipping	0.1
Value Loss Weight	0.5
Entropy Loss Weight	0.01
Training Stages	3
Steps for PPO Update Stage 1	32
Steps for PPO Update Stage 2	64
Steps for PPO Update Stage 3	128
Transformer State Encoder Layers	3
Transformer State Encoder Hidden Dims	512
Transformer State Encoder Heads	8
Causal Transformer Deocder Layers	3
Causal Transformer Deocder Hidden Dims	512
Causal Transformer Deocder Heads	8

Table 3: Hyperparameters for training and model architecture.

训练与模型细节
参数	取值
允许步数	600（Stretch RE-1），500（LoCoBot）
总 Rollout 数	192（Stretch RE-1），384（LoCoBot）
学习率	0.002
每次更新的 Mini Batch	1
更新重复次数	4
最大梯度范数	0.5
折扣因子 $\gamma$	0.99
GAE $\lambda$	0.95
PPO 代理目标裁剪	0.1
价值损失权重	0.5
熵损失权重	0.01
训练阶段数	3
PPO 更新阶段 1 步数	32
PPO 更新阶段 2 步数	64
PPO 更新阶段 3 步数	128
Transformer 状态编码器层数	3
Transformer 状态编码器隐藏维	512
Transformer 状态编码器头数	8
因果 Transformer 解码器层数	3
因果 Transformer 解码器隐藏维	512
因果 Transformer 解码器头数	8

表 3：训练与模型架构超参。

B.4 Hyperparameters for Training

B.4 训练超参

Tab. 3 lists the hyperparameters used in our training and model architecture design. Please find more details such as scene texture randomization, visual observation augmentations, and goal specification randomization when using text instruction in our codebase.

表 3 列出训练与模型架构设计所用超参。场景纹理随机化、视觉观测增强、以及使用文本指令时的目标规格随机化等更多细节，见我们的代码库。

Appendix C Additional Details about Environment, Benchmarks, and Real-World Experiments

附录 C 环境、基准与真实世界实验的补充细节

Action Space. Following prior work using AI2-THOR, we discretize the action space for both LoCoBot and Stretch RE-1. For LoCoBot, we discretize the action space into 6 actions, including {MoveAhead, RotateRight, RotateLeft, LookUp, LookDown, Done}, where MoveAhead moves the agent forward by 0.2 meters, RotateRight rotates the agent clockwise by 30∘ around the yaw-axis, RotateLeft rotates the agent counter-clockwise by 30∘ around the yaw-axis, LookUp rotates agent’s camera clockwise by 30∘ around the roll-axis, LookDown rotates agent’s camera counter-clockwise by 30∘ around the roll-axis, and Done indicates that the agent found the target and ends an episode. We follow previous works [11, 17, 86] to use the same action space for LoCoBot for a fair comparison. For Stretch RE-1, we remove the LookUp and LookDown camera actions, and add MoveBack, RotateRightSmall, and RotateLeftSmall to the action space, where MoveBack moves the agent backward by 0.2 meters, RotateRightSmall rotates the agent clockwise by 6∘ around the yaw-axis, and RotateLeftSmall rotates the agent counter-clockwise by 6∘ around the yaw-axis. Again, this action space is identical to the one used in prior work [6] for fair comparison.

动作空间。 沿用使用 AI2-THOR 的先前工作，我们对 LoCoBot 与 Stretch RE-1 均离散化动作空间。LoCoBot 离散为 6 个动作：{MoveAhead, RotateRight, RotateLeft, LookUp, LookDown, Done}，其中 MoveAhead 使智能体前进 0.2 米，RotateRight / RotateLeft 绕偏航轴顺/逆时针旋转 30∘，LookUp / LookDown 使相机绕横滚轴顺/逆时针旋转 30∘，Done 表示找到目标并结束回合。为公平比较，我们按先前工作 [11, 17, 86] 对 LoCoBot 使用相同动作空间。Stretch RE-1 去掉 LookUp、LookDown 相机动作，并向动作空间加入 MoveBack、RotateRightSmall、RotateLeftSmall：MoveBack 后退 0.2 米，RotateRightSmall / RotateLeftSmall 绕偏航轴顺/逆时针旋转 6∘。该动作空间同样与先前工作 [6] 相同，以利公平比较。

Success Criteria. We follow the definition of Object Goal Navigation defined in [3], where an agent must explore its environment to locate and navigate to an object of interest within an allowed number of steps $n$. The agent has to issue the Done action to indicate it found the target. The environment will then judge if the agent is within a distance $d$ from the target and if the target can be seen in the agent’s view. An episode is also classified as failed if the agent runs more than $n$ steps without issuing any Done action. Across different benchmarks, $n$ and $d$ vary depending on the scenes size and complexity and agent’s capabilities. We follow ProcTHOR [11] to use $n=500$ and $d=1$ meter for LoCoBot, and follow Chores-$\mathbb{S}$ [6] to use $n=600$ and $d=2$ meters for Stretch RE-1.

成功判据。 我们沿用 [3] 中物体目标导航的定义：智能体须在允许步数 $n$ 内探索环境以定位并导航到感兴趣物体。智能体须发出 Done 动作表明找到目标；环境随后判断智能体是否距目标在距离 $d$ 内、以及目标是否出现在智能体视野中。若智能体超过 $n$ 步仍未发出任何 Done，回合亦判失败。跨不同基准，$n$ 与 $d$ 随场景大小复杂度与智能体能力而变。我们按 ProcTHOR [11] 对 LoCoBot 取 $n=500$、$d=1$ 米；按 Chores-$\mathbb{S}$ [6] 对 Stretch RE-1 取 $n=600$、$d=2$ 米。

SPL and SEL. Success Weighted by Path Length (SPL) and Success Weighted by Episode Legnth (SEL) are two popular evaluation metrics to evaluate how efficient an agent is to find the target. SPL is defined as $\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{l_{i}}{max(l_{i},p_{i})}$, where $N$ is the total number of episodes, $S_{i}$ is a binary indicator of success for episode $i$, $l_{i}$ is the shortest travel distance to the target, and $p_{i}$ is the actual travel distance. SEL is defined similarly: $\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{w_{i}}{max(w_{i},e_{i})}$, where $w_{i}$ is the shortest number of steps to find the target, and $e_{i}$ is the actual number of steps used by the agent. By definition, SPL focuses on how far the agent has traveled, while SEL focuses on how many steps the agent has used (which also penalizes excessive in-place rotation). SPL can be derived by computing the geodesic distance between the agent’s starting location and the target’s location, while SEL needs a planner with privileged environment information to calculate the number of steps of expert trajectories. Therefore, we follow ProcTHOR [11] to report SPL to evaluate the LoCoBot agent, since those benchmarks do not provide planner, while we follow Chores-$\mathbb{S}$ [6] to report SEL, since expert trajectories are available.

SPL 与 SEL。 路径长度加权成功率（SPL）与回合长度加权成功率（SEL）是评估智能体找目标效率的两种常用指标。SPL 定义为 $\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{l_{i}}{max(l_{i},p_{i})}$，其中 $N$ 为总回合数，$S_{i}$ 为回合 $i$ 成功与否的二值指示，$l_{i}$ 为到目标的最短旅行距离，$p_{i}$ 为实际旅行距离。SEL 类似： $\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{w_{i}}{max(w_{i},e_{i})}$，其中 $w_{i}$ 为找到目标的最短步数，$e_{i}$ 为智能体实际使用步数。按定义，SPL 关注智能体走了多远，SEL 关注用了多少步（也惩罚过多原地旋转）。SPL 可由智能体起点与目标位置间的测地距离导出；SEL 则需带特权环境信息的规划器来计算专家轨迹步数。因此我们按 ProcTHOR [11] 报告 SPL 评估 LoCoBot（那些基准不提供规划器），按 Chores-$\mathbb{S}$ [6] 报告 SEL（专家轨迹可用）。

Figure 6: Starting Poses of (a) LoCoBot and (b) Stretch RE-1 used in the real world experiments. The arrow direction indicates where the agent faces with.

图 6：真实世界实验中 (a) LoCoBot 与 (b) Stretch RE-1 的起始姿态。箭头方向表示智能体朝向。

Real-world Experiment Setup. For the experiments using LoCoBot, we follow Phone2Proc [17] to use the same five target object categories, including Apple, Bed, Sofa, Television, and Vase, and the three starting poses, shown in Fig. 6 (a). Among those target categories, Apple can be found in the Living room and Kitchen, Bed can only be found in the Bedroom, Sofa and Television can only be found in the Living room, and Vase can be found in the Livingroom, Corridor, Office, and Kitchen. For the experiments using Stretch RE-1, we follow SPOC [6] to use the same six target object categories, including Apple, Bed, Chair, HousePlant, Sofa, and Vase, and the three starting poses, shown in Fig. 6 (b). Among the categories not mentioned above, Chair can be found in the Living room, Office, and Kitchen, and HousePlant can be found in the Living room, Office, Bathroom, and Kitchen.

真实世界实验设置。 LoCoBot 实验中，我们按 Phone2Proc [17] 使用相同的五个目标物体类别：Apple、Bed、Sofa、Television、Vase，以及图 6(a) 所示三个起始姿态。这些目标类别中，Apple 可在客厅与厨房找到，Bed 仅在卧室，Sofa 与 Television 仅在客厅，Vase 可在客厅、走廊、办公室与厨房找到。Stretch RE-1 实验中，按 SPOC [6] 使用相同的六个目标类别：Apple、Bed、Chair、HousePlant、Sofa、Vase，以及图 6(b) 所示三个起始姿态。上述未提及的类别中，Chair 可在客厅、办公室与厨房找到，HousePlant 可在客厅、办公室、浴室与厨房找到。

Appendix D More Simulation Evaluations

附录 D 更多仿真评估

Performance Variance. On Chores-$\mathbb{S}$, since we follow SPOC [6] to apply test-time data augmentation and non-deterministic action sampling, we found that performance varies even using the same checkpoint, especially given that we are only evaluating on $200$ episodes. As a result, we re-evaluate our PoliFormer and SPOC*[^4] 16 times and report mean success rate (mSR) and standard deviation (std). PoliFormer achieves $82.5\%$ mSR with $1.897$ std, while SPOC* achieves $56.7\%$ mSR with $2.697$ std. This result indicates that PoliFormer not only achieves a higher mSR than SPOC*, but also exhibits more reliably consistent behavior, i.e. a lower std, when run on the same episodes multiple times.

性能方差。 在 Chores-$\mathbb{S}$ 上，因我们按 SPOC [6] 应用测试时数据增强与非确定性动作采样，发现即便同一检查点性能也会波动，尤其评估仅 $200$ 个回合时更明显。因此我们对 PoliFormer 与 SPOC*[^4] 各重评 16 次，报告平均成功率（mSR）与标准差（std）。PoliFormer 达到 $82.5\%$ mSR、$1.897$ std；SPOC* 为 $56.7\%$ mSR、$2.697$ std。这表明 PoliFormer 不仅 mSR 更高，在相同回合上多次运行时行为也更可靠一致，即 std 更低。

Inputs	Model	Loss	EasyObjectNav	RegularObjectNav	HardObjectNav
Success (SEL)	Success (SEL)	Success (SEL)
RGB+text	SPOC [6]	IL	62.9 (40.5)	48.2 (38.9)	34.1 (27.4)
SPOC∗	IL	69.7 (43.3)	53.5 (34.3)	31.0 (19.6)
PoliFormer	RL	89.0 (62.1)	82.6 (71.8)	72.3 (62.8)
RGB	SPOC	IL	90.3 (67.7)	78.7 (62.6)	70.6 (52.5)
+text+b-box	PoliFormer	RL	98.1 (86.5)	90.4 (79.6)	86.0 (75.0)
RGB+b-box	PoliFormer	RL	97.1 (83.2)	91.9 (79.8)	87.6 (75.0)

(a) Stretch RE-1 on Chores-$\mathbb{S}$

输入	模型	损失	EasyObjectNav	RegularObjectNav	HardObjectNav
指标：成功率 (SEL)
RGB+文本	SPOC [6]	IL	62.9 (40.5)	48.2 (38.9)	34.1 (27.4)
SPOC∗	IL	69.7 (43.3)	53.5 (34.3)	31.0 (19.6)
PoliFormer	RL	89.0 (62.1)	82.6 (71.8)	72.3 (62.8)
RGB	SPOC	IL	90.3 (67.7)	78.7 (62.6)	70.6 (52.5)
+文本+b-box	PoliFormer	RL	98.1 (86.5)	90.4 (79.6)	86.0 (75.0)
RGB+b-box	PoliFormer	RL	97.1 (83.2)	91.9 (79.8)	87.6 (75.0)

(a) Stretch RE-1 在 Chores-$\mathbb{S}$ 上

Larger Scale Simulation Benchmark using Stretch RE-1. To further analyze PoliFormer’s performance through different difficulty settings, we construct 3 different levels of Object Goal Navigation benchmarks, EasyObjectNav, RegularObjectNav, and HardObjectNav, where each level contains 2k episodes, using Stretch RE-1. We construct these differentiated tasks by ensuring the oracle expert path length between the agent and target is 1 to 3 meters long for EasyObjectNav, greater than 3 meters for RegularObjectNav, and larger than 10 meters for HardObjectNav. The results are shown in Tab. 4. We observe that every model performs better as the agent is closer to the target at the episode start. In addition, on EasyObjectNav the agent barely needs exploration to find the target. Thereby, we find that PoliFormer lagging behind PoliFormer-BoxNav by ${\sim}9\%$ could result from a Recognition Issue. Moreover, the gap on HardObjectNav is widened to ${\sim}13.7\%$, and it could result from an additional Exploration Issue. The performance gap between HardObjectNav and EasyObjectNav could also support that an Exploration Issue exists, but not just the Recognition Issue.

用 Stretch RE-1 的更大规模仿真基准。 为进一步按不同难度分析 PoliFormer 表现，我们构建三个难度的物体目标导航基准：EasyObjectNav、RegularObjectNav、HardObjectNav，每级含 2k 个回合，使用 Stretch RE-1。区分方式是保证智能体与目标之间的 oracle 专家路径长度：EasyObjectNav 为 1 到 3 米，RegularObjectNav 大于 3 米，HardObjectNav 大于 10 米。结果见表 4。我们观察到：回合开始时智能体离目标越近，各模型表现越好。此外，EasyObjectNav 上智能体几乎不需探索即可找到目标；因此 PoliFormer 落后 PoliFormer-BoxNav 约 ${\sim}9\%$，可能来自识别问题。HardObjectNav 上差距扩大到约 ${\sim}13.7\%$，可能另有探索问题。HardObjectNav 与 EasyObjectNav 之间的性能差距也可支持存在探索问题，而非仅识别问题。

Appendix E Additional Discussion on Limitations

附录 E 局限的补充讨论

Depth Sensor. It is important to note that PoliFormer is not equipped with a depth sensor (which has been proven to be effective for manipulation). While the lack of the depth sensor does not affect our agent’s performance on navigation, we acknowledge that integrating the depth sensor into our visual representation is an interesting direction for future work, especially when considering mobile-manipulation extensions.

深度传感器。 须指出 PoliFormer 未配备深度传感器（深度对操作已被证明有效）。虽缺少深度传感器不影响我们智能体的导航表现，我们承认把深度传感器纳入视觉表征是未来工作的有趣方向，尤其在考虑移动操作扩展时。

Discretized Action Space. To have a fair comparison with baselines, we use the same discretized action space in this work (see Sec. C). The discretized action space might not be efficient and realistic in many real-world scenarios where the agent must act in a timely manner.

离散动作空间。 为与基线公平比较，本文使用相同的离散动作空间（见第 C 节）。在许多要求智能体及时行动的真实世界场景中，离散动作空间可能不够高效与现实。

Cross-embodiment. In this paper, we demonstrate that we can train PoliFormer using LoCoBot and Stretch RE-1. However, we have not yet explored training a single PoliFormer for both embodiments. We leave this interesting research direction as future work.

跨本体。 本文展示可用 LoCoBot 与 Stretch RE-1 分别训练 PoliFormer。但我们尚未探索为两种本体训练单一 PoliFormer。将该有趣研究方向留作未来工作。

Further Scaling. Our training and validation curves strongly suggest that even further scaling of model parameters and training time may lead to even more masterful models than those we have trained in this work. This perspective is exciting and we hope to enable further scaling with more computation resources and better visual foundation models in the near future.

进一步扩展。 我们的训练与验证曲线强烈暗示：进一步扩展模型参数与训练时间，可能得到比本文所训更精湛的模型。这一前景令人振奋；我们希望在不久的将来以更多计算资源与更好的视觉基础模型实现进一步扩展。

Failure Analysis. The main mode of failure for PoliFormer is the agent’s limited memory. PoliFormer clearly demonstrates memorization capabilities and is able to perform long-horizon tasks by exploring large indoor scenes without access to explicit mapping. However, as the trajectories get longer (specifically after visiting more than 4 rooms in an environment), the agent’s recollection of the rooms it has explored deteriorates and the robot might re-visit rooms that it has explored previously.

失败分析。 PoliFormer 的主要失败模式是智能体记忆有限。它清楚展现记忆能力，并能在无显式建图的情况下探索大型室内场景、完成长时程任务。但随轨迹变长（尤其访问环境中超过 4 个房间后），智能体对已探索房间的回忆会退化，机器人可能重访先前已探索的房间。