NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration

NoMaD：面向导航与探索的目标掩码扩散策略

Ajay Sridhar    Dhruv Shah    Catherine Glossop    Sergey Levine Affiliation:  UC Berkeley

Abstract

摘要

Robotic learning for navigation in unfamiliar environments needs to provide policies for both task-oriented navigation (i.e., reaching a goal that the robot has located), and task-agnostic exploration (i.e., searching for a goal in a novel setting). Typically, these roles are handled by separate models, for example by using subgoal proposals, planning, or separate navigation strategies. In this paper, we describe how we can train a single unified diffusion policy to handle both goal-directed navigation and goal-agnostic exploration, with the latter providing the ability to search novel environments, and the former providing the ability to reach a user-specified goal once it has been located. We show that this unified policy results in better overall performance when navigating to visually indicated goals in novel environments, as compared to approaches that use subgoal proposals from generative models, or prior methods based on latent variable models. We instantiate our method by using a large-scale Transformer-based policy trained on data from multiple ground robots, with a diffusion model decoder to flexibly handle both goal-conditioned and goal-agnostic navigation. Our experiments, conducted on a real-world mobile robot platform, show effective navigation in unseen environments in comparison with five alternative methods, and demonstrate significant improvements in performance and lower collision rates, despite utilizing smaller models than state-of-the-art approaches.

陌生环境中的机器人导航学习，既要给出任务导向的导航策略（到达机器人已定位到的目标），也要给出与任务无关的探索策略（在新场景中搜寻目标）。这两类角色通常由不同模型分别承担，例如借助子目标提议、规划，或各自独立的导航策略。本文说明如何训练一个统一的扩散策略，同时处理有目标导航与无目标探索：后者负责在新环境中搜索，前者在定位到用户指定目标后负责到达。我们表明，在陌生环境中导航到视觉指示目标时，这一统一策略的整体表现优于依赖生成模型做子目标提议的方法，以及基于潜变量模型的先前方法。我们用大规模 Transformer 策略实例化该方法，在多台地面机器人数据上训练，并以扩散模型解码器灵活处理有条件与无条件导航。在真实移动机器人平台上的实验表明，相对五种对照方法，NoMaD 在未见环境中导航更有效，性能显著提升且碰撞率更低，所用模型也小于当前最先进方案。

Fig. 1: NoMaD is the first flexibly conditioned diffusion model of robot actions that can perform both goal-conditioned navigation and undirected exploration in previously unseen environments. It uses goal masking to condition on an optional goal image, and a diffusion policy to model complex, multimodal action distributions in challenging real-world environments.

图 1：NoMaD 是首个可灵活条件化的机器人动作扩散模型，能在先前未见过的环境中同时完成有目标导航与无向探索。它用目标掩码对可选目标图像做条件化，并用扩散策略在具有挑战性的真实环境中建模复杂、多模态的动作分布。

I Introduction

I 引言

Robotic learning provides us with a powerful tool for acquiring multi-task policies that, when conditioned on a goal or another task specification, can perform a wide variety of different behaviors. Such policies are appealing not only because of their flexibility, but because they can leverage data from a variety of tasks and domains and, by sharing knowledge across these settings, acquire policies that are more performant and more generalizable. However, in practical settings, we might encounter situations where the robot doesn’t know which task to perform because the environment is unfamiliar, the task requires exploration, or the direction provided by the user is incomplete. In this work, we study a particularly important instance of this problem in the domain of robotic navigation, where the user might specify a destination visually (i.e., via a picture), and the robot must locate this destination by searching its environment. In such settings, standard multi-task policies trained to perform the user-specified task are not enough by themselves: we also need some way for the robot to explore, potentially trying different tasks (e.g., different possible destinations for searching the environment), before figuring out how to perform the desired task (i.e., locating the object of interest). Prior works have often addressed this challenge by training a separate high-level policy or goal proposal system that generates suitable exploratory tasks, for example using high-level planning [1], hierarchical reinforcement learning [2], and generative models [3]. However, this introduces additional complexity and often necessitates task-specific mechanisms. Can we instead train a single highly expressive policy that can represent both task-specific and task-agnostic behavior, utilizing the task-agnostic behavior for exploration and switching to task-specific behavior as needed to solve the task?

机器人学习是获取多任务策略的有力工具：在目标或其他任务规格的条件下，同一策略可展现多种多样的行为。这类策略不仅灵活，还能汇聚来自不同任务与领域的数据；通过跨设定共享知识，得到更强、更可泛化的策略。但在实际中，环境陌生、任务需要探索，或用户指示不完整时，机器人往往并不清楚该执行哪一项任务。本文研究该问题在机器人导航中的一个重要实例：用户以视觉方式指定目的地（例如一张图片），机器人必须通过搜索环境来定位该目的地。此类设定下，仅训练「执行用户指定任务」的标准多任务策略不够：机器人还需要某种探索方式，可能先尝试不同任务（例如不同的候选目的地），再弄清如何完成期望任务（定位感兴趣对象）。先前工作常通过训练独立的高层策略或目标提议系统来应对，例如高层规划 [1]、分层强化学习 [2] 与生成模型 [3]。但这引入额外复杂度，且往往依赖任务专用机制。我们能否反过来，训练一个表达力足够强的单一策略，同时表示任务特定与任务无关行为，用后者做探索、在需要时切到前者来完成任务？

In this paper, we present a design for such a policy by combining a Transformer backbone for encoding the high-dimensional stream of visual observations with diffusion models for modeling a sequence of future actions and instantiate it for the particular problem of visual exploration and goal-seeking in novel environments. Our main insight is that such an architecture is uniquely suited for modeling task-specific and task-agnostic pathways since it provides high capacity (both for modeling perception and control) and the ability to represent complex, multimodal distributions.

本文给出这样一种策略的设计：用 Transformer 骨干编码高维视觉观测流，用扩散模型建模未来动作序列，并针对新颖环境中的视觉探索与寻的问题加以实例化。我们的核心洞见是：该架构特别适合同时建模任务特定与任务无关通路——既有足够容量（感知与控制），又能表示复杂、多模态分布。

The main contribution of our work is Navigation with Goal Masked Diffusion, or NoMaD, a novel architecture for robotic navigation in previously unseen environments that uses a unified diffusion policy to jointly represent exploratory task-agnostic behavior and goal-directed task-specific behavior in a framework that combines graph search, frontier exploration, and highly expressive policies. We evaluate the performance of NoMaD on both undirected and goal-conditioned experiments across challenging indoor and outdoor environments, and report improvements over the state-of-the-art, while also being $15\times$ more computationally efficient. To the best of our knowledge, NoMaD is the first successful instantiation of a goal-conditioned action diffusion model, as well as a unified model for both task-agnostic and task-oriented behavior, deployed on a physical robot.

本文的主要贡献是 Navigation with Goal Masked Diffusion，即 NoMaD：一种面向先前未见环境的机器人导航新架构。它用统一的扩散策略联合表示探索性的任务无关行为与目标导向的任务特定行为，并在结合图搜索、前沿探索与高表达力策略的框架中落地。我们在具有挑战性的室内外环境中评估 NoMaD 的无向与有目标实验表现，相对当前最先进方法有提升，同时计算效率高约 $15\times$。据我们所知，NoMaD 是首个成功实例化的有目标动作扩散模型，也是首个在实体机器人上部署的、同时覆盖任务无关与任务导向行为的统一模型。

II Related Work

II 相关工作

Exploring a new environment is often framed as the problem of efficient mapping, posed in terms of information maximization to guide the robot to new regions. Some prior exploration methods use local strategies for generating control actions for the robots [4, 5, 6, 7], while others use global strategies based on the frontier method  [8, 9, 10]. However, building high-fidelity geometric maps can be hard without reliable depth information. Inspired by prior work [1, 11, 12], we factorize the exploration problem into (i) learned control policies that can take diverse, short-horizon actions, and (ii) a high-level planner based on a topological graph that uses the policy for long-horizon goal-seeking.

探索新环境常被表述为高效建图问题，以信息最大化引导机器人进入新区域。部分先前探索方法用局部策略生成控制动作 [4, 5, 6, 7]，另一些则基于前沿方法的全局策略 [8, 9, 10]。但在缺少可靠深度信息时，构建高保真几何地图可能很困难。受先前工作 [1, 11, 12] 启发，我们将探索问题分解为：(i) 能采取多样短时程动作的学习控制策略；(ii) 基于拓扑图的高层规划器，利用该策略做长时程寻的。

Several prior works have proposed learning-based approaches for robotic exploration by leveraging privileged information in simulation [13, 14, 15, 16] or learn directly from real-world experience [17]. These policies have been trained with reinforcement learning to maximize coverage, predicting semantically rich regions, intrinsic rewards [13, 18, 14, 19], or by using planning in conjunction with latent variable and affordance models [20, 17, 21]. However, policies trained in simulation tend to transfer poorly to real-world environments [16, 22], and our experiments reveal that even the best exploration policies trained on real-world data perform poorly in complex indoor and outdoor environments.

若干先前工作提出基于学习的机器人探索方法：在仿真中利用特权信息 [13, 14, 15, 16]，或直接从真实世界经验学习 [17]。这些策略用强化学习训练以最大化覆盖、预测语义丰富区域、使用内在奖励 [13, 18, 14, 19]，或将规划与潜变量及可供性模型结合 [20, 17, 21]。但仿真训练的策略往往难以迁移到真实环境 [16, 22]；我们的实验也表明，即便是在真实数据上训练的最佳探索策略，在复杂室内外环境中表现仍然不佳。

The closest related work to NoMaD is ViNT, which uses a goal-conditioned navigation policy in conjunction with a separate high-capacity subgoal proposal model [3]. The subgoal proposal model is instantiated as a 300M parameter image diffusion model [23], generating candidate subgoal images conditioned on the robot’s current observation. NoMaD uses diffusion models differently: rather than generating subgoal images with diffusion and conditioning on these generations, we directly model actions conditioned on the robot’s observation with diffusion. Empirically, we find that NoMaD outperforms the ViNT system by over 25% in undirected exploration. Furthermore, since NoMaD does not generate high-dimensional images, it requires over $15\times$ fewer parameters, providing a more compact and efficient approach that can run directly on the less powerful onboard computers (e.g., NVIDIA Jetson Orin).

与 NoMaD 最接近的相关工作是 ViNT：它将有目标导航策略与一个独立的高容量子目标提议模型配合使用 [3]。该子目标提议模型实例化为约 3 亿参数的图像扩散模型 [23]，以机器人当前观测为条件生成候选子目标图像。NoMaD 对扩散的用法不同：不是用扩散生成子目标图像再以其为条件，而是直接以机器人观测为条件对动作建模。经验上，NoMaD 在无向探索上相对 ViNT 系统提升超过 25%。此外，由于不生成高维图像，参数量少约 $15\times$，更紧凑高效，可直接跑在算力较弱的机载计算机上（例如 NVIDIA Jetson Orin）。

A key challenge with predicting sequences of robot actions for exploration is the difficulty in modeling multimodal action distributions. Prior work has addressed this by exploring different action representations, such as autoregressive predictions of quantized actions [24, 25, 26, 27], using latent variable models [17, 21], switching to an implicit policy representation [28], and, most recently, using conditional diffusion models for planning and control [29, 30, 31, 32, 33, 34]. State- or observation-conditioned diffusion models of action [30, 31] are particularly powerful, since they enable modeling complex action distributions without the cost and added complexity of inferring future states/observations. NoMaD extends this formulation by additionally conditioning the action distribution on both the robot’s observations and the optional goal information, giving the first instantiation of a “diffusion policy” that can work in both goal-conditioned and undirected modes.

为探索预测机器人动作序列的一个关键挑战，是难以建模多模态动作分布。先前工作通过不同动作表示应对：对量化动作的自回归预测 [24, 25, 26, 27]、使用潜变量模型 [17, 21]、改用隐式策略表示 [28]，以及最近用条件扩散模型做规划与控制 [29, 30, 31, 32, 33, 34]。以状态或观测为条件的动作扩散模型 [30, 31] 尤其有力：能在无需推断未来状态/观测的额外代价与复杂度下，建模复杂动作分布。NoMaD 将该形式进一步扩展——动作分布同时以机器人观测与可选目标信息为条件——给出首个既能在有目标模式也能在无向模式下工作的「扩散策略」实例。

III Preliminaries

III 预备知识

Fig. 2: Model Architecture. NoMaD uses two EfficientNet encoders $\psi,\phi$ to generate input tokens to a Transformer decoder. We use goal masking to jointly reason about task-agnostic and task-oriented behaviors through the observation context $c_{t}$. We use action diffusion conditioned on the context $c_{t}$ to obtain a highly expressive policy that can be used in both a goal-conditioned and undirected manner.

图 2：模型架构。 NoMaD 用两个 EfficientNet 编码器 $\psi,\phi$ 生成送入 Transformer 解码器的输入 token。我们用目标掩码，通过观测上下文 $c_{t}$ 联合推断任务无关与任务导向行为。以 $c_{t}$ 为条件的动作扩散给出高表达力策略，可同时用于有目标与无向方式。

Our objective is to design a control policy $\pi$ for visual navigation that takes the robot’s current and past RGB observations as input ${\mathbf{o}}_{t}:=o_{t-P:t}$ and outputs a distribution over future actions ${\mathbf{a}}_{t}:=a_{t:t+H}$. The policy may additionally have access to an RGB image of a goal $o_{g}$, which can be used to specify the navigation task. When a goal $o_{g}$ is provided, $\pi$ must take actions that make progress towards the goal, and eventually reach it. In an unseen environment, the goal image $o_{g}$ may not be available, and $\pi$ must explore the environment by taking safe and reasonable navigation actions (e.g., avoiding obstacles, following hallways etc.), while providing sufficient coverage of the valid behaviors in the environment. To facilitate long-horizon exploration and goal-seeking, we follow the setup of ViKiNG [35] and pair $\pi({\mathbf{o}}_{t})$ with a topological memory of the environment ${\mathcal{M}}$, and a high-level planner that encourages the robot to explore the environment by navigating to unexplored regions.

我们的目标是设计视觉导航控制策略 $\pi$：以机器人当前与过去的 RGB 观测 ${\mathbf{o}}_{t}:=o_{t-P:t}$ 为输入，输出未来动作上的分布 ${\mathbf{a}}_{t}:=a_{t:t+H}$。策略还可额外获得目标的 RGB 图像 $o_{g}$，用以指定导航任务。当提供目标 $o_{g}$ 时，$\pi$ 须采取朝目标推进并最终到达的动作。在未见环境中，目标图像 $o_{g}$ 可能不可用，$\pi$ 须通过采取安全合理的导航动作（例如避障、沿走廊前进等）来探索环境，同时对有效行为提供足够覆盖。为便于长时程探索与寻的，我们沿用 ViKiNG [35] 的设定，将 $\pi({\mathbf{o}}_{t})$ 与环境拓扑记忆 ${\mathcal{M}}$ 配对，并由高层规划器鼓励机器人通过前往未探索区域来探索环境。

Visual goal-conditioned policies: To train goal-conditioned policies for visual inputs, we follow a large body of prior work on training high-capacity policies based on the Transformer architecture [36, 37, 38, 3]. Specifically, we use the Visual Navigation Transformer (ViNT) [3] policy as the backbone for processing the robot’s visual observations ${\mathbf{o}}_{t}$ and goal $o_{g}$. ViNT uses an EfficientNet-B0 encoder [39] $\psi(o_{i})$ to process each observation image $i\in\{t-P,\ldots,t\}$ independently, as well as a goal fusion encoder $\phi(o_{t},o_{g})$ to tokenize the inputs. These tokens are processed using multi-headed attention layers $f(\psi(o_{i}),\phi(o_{t},o_{g}))$ to obtain a sequence of context vectors that are concatenated to obtain the final context vector $c_{t}$. The context vector is then used to predict future actions ${\mathbf{a}}_{t}=f_{a}(c_{t})$ and temporal distance between the observation and the goal $d(o_{t},o_{g})=f_{d}(c_{t})$, where $f_{a},f_{c}$ are fully-connected layers. The policy is trained using supervised learning using a maximum likelihood objective, corresponding to regression to the ground-truth actions and temporal distance. While ViNT shows state-of-the-art performance in goal-conditioned navigation, it cannot perform undirected exploration and requires an external subgoal proposal mechanism. NoMaD extends ViNT to enable both goal-conditioned and undirected navigation.

视觉有目标策略： 为训练视觉输入上的有目标策略，我们沿用基于 Transformer 架构训练高容量策略的大量先前工作 [36, 37, 38, 3]。具体地，以 Visual Navigation Transformer（ViNT） [3] 策略为骨干，处理机器人视觉观测 ${\mathbf{o}}_{t}$ 与目标 $o_{g}$。ViNT 用 EfficientNet-B0 编码器 [39] $\psi(o_{i})$ 独立处理每张观测图像 $i\in\{t-P,\ldots,t\}$，并用目标融合编码器 $\phi(o_{t},o_{g})$ 将输入 token 化。这些 token 经多头注意力层 $f(\psi(o_{i}),\phi(o_{t},o_{g}))$ 得到上下文向量序列，拼接成最终上下文向量 $c_{t}$。随后用 $c_{t}$ 预测未来动作 ${\mathbf{a}}_{t}=f_{a}(c_{t})$ 以及观测与目标之间的时间距离 $d(o_{t},o_{g})=f_{d}(c_{t})$，其中 $f_{a},f_{d}$ 为全连接层。策略以监督学习、最大似然目标训练，对应对真实动作与时间距离的回归。ViNT 在有目标导航上表现先进，但无法做无向探索，并需要外部子目标提议机制。NoMaD 扩展 ViNT，使有目标与无向导航同时成为可能。

Exploration with topological maps: While goal-conditioned policies can exhibit useful affordances and collision-avoidance behavior, they may be insufficient for navigation in large environments that require reasoning over long time horizons. To facilitate long-horizon exploration and goal-seeking in large environments, we follow the setup of ViKiNG [35] and integrate the policy with episodic memory ${\mathcal{M}}$ in the form of a topological graph of the robot’s experience in the environment. ${\mathcal{M}}$ is represented by a graph structure with nodes corresponding to the robot’s visual observations in the environment, and edges corresponding to navigable paths between two nodes, as determined by the policy’s goal-conditioned distance predictions. When navigating large environments, the robot’s visual observations ${\mathbf{o}}_{t}$ may not be sufficient to plan long-horizon trajectories to the goal. Instead, the robot can use the topological map ${\mathcal{M}}$ to plan a sequence of subgoals that guide the robot to the goal. When exploring previously unseen environments, we construct ${\mathcal{M}}$ online while the robot searches the environment for a goal. Beyond undirected coverage exploration, this graph-based framework also supports the ability to reach high-level goals $G$, which may be arbitrarily far away and specified as GPS positions, locations on a map, language instructions, etc. In this work, we focus on frontier-based exploration, which tests the ability of NoMaD to propose diverse subgoals and search unseen environments. We largely follow the setup of prior work [35], swapping the learned policy with NoMaD.

基于拓扑地图的探索： 有目标策略可以展现有用的可供性与避障行为，但在需要长时程推理的大环境导航中可能不够。为便于大环境中的长时程探索与寻的，我们沿用 ViKiNG [35] 的设定，将策略与情节记忆 ${\mathcal{M}}$ 结合——以机器人在环境中经验构成的拓扑图形式。${\mathcal{M}}$ 以图结构表示：节点对应机器人在环境中的视觉观测，边对应两节点间可导航路径，由策略的有目标距离预测判定。在大环境导航时，仅凭视觉观测 ${\mathbf{o}}_{t}$ 往往不足以规划到目标的长时程轨迹；机器人可改用拓扑地图 ${\mathcal{M}}$ 规划一串引导其到达目标的子目标。在探索先前未见环境时，我们在机器人搜寻目标的过程中在线构建 ${\mathcal{M}}$。除无向覆盖探索外，该基于图的框架还支持到达高层目标 $G$——目标可任意远，并以 GPS 位置、地图上的点、语言指令等形式给出。本文聚焦基于前沿的探索，以检验 NoMaD 提出多样子目标并搜索未见环境的能力。我们大体沿用先前工作 [35] 的设定，将学习策略替换为 NoMaD。

IV Method

IV 方法

Unlike prior work that uses separate policies for goal-conditioned navigation and open-ended exploration [3], we posit that learning a single model for both behaviors is more efficient and generalizable. Training a shared policy across both behaviors allows the model to learn a more expressive prior over actions ${\mathbf{a}}_{t}$, which can be used for both conditional and unconditional inference. In this section, we describe our proposed NoMaD architecture, which is a goal-conditioned diffusion policy that can be used for both goal-reaching and undirected exploration. The NoMaD architecture has two key components: (i) attention-based goal-masking, which provides a flexible mechanism for conditioning the policy on (or masking out) an optional goal image $o_{g}$; and (ii) a diffusion policy, which provides an expressive prior over collision-free actions that the robot can take. Figure 2 shows an overview of the NoMaD architecture, and we describe each component in detail below.

与用独立策略分别做有目标导航与开放式探索的先前工作 [3] 不同，我们认为用单一模型学习两种行为更高效、也更可泛化。跨两种行为训练共享策略，使模型学到对动作 ${\mathbf{a}}_{t}$ 更具表达力的先验，可用于条件与无条件推断。本节描述我们提出的 NoMaD 架构：一种可用于目标到达与无向探索的有目标扩散策略。NoMaD 有两个关键组件：(i) 基于注意力的目标掩码，为对可选目标图像 $o_{g}$ 做条件化（或掩蔽）提供灵活机制；(ii) 扩散策略，为机器人可采取的无碰撞动作提供表达力强的先验。图 2 给出 NoMaD 架构概览，下文分别详述各组件。

IV-A Goal Masking

IV-A 目标掩码

In order to train a shared policy for goal-reaching and undirected exploration, we modify the ViNT architecture described in Section III by introducing a binary “goal mask” $m$, such that $c_{t}=f(\psi(o_{i}),\phi(o_{t},o_{g}),m)$. $m$ can be used to mask out the goal token $\phi(o_{t},o_{g})$, thus blocking the goal-conditioned pathway of the policy. We implement masked attention by setting the goal mask $m=1$, such that the downstream computation of $c_{t}$ does not attend to the goal token. We implement unmasked attention by setting $m=0$, such that the goal token is used alongside observation tokens in the downstream computation of $c_{t}$. During training, the goal mask $m$ is sampled from a Bernoulli distribution with probability $p_{m}$. We use a fixed $p_{m}=0.5$ during training, corresponding to equal number of training samples corresponding to goal-reaching and undirected exploration. At test time, we set $m$ corresponding to the desired behavior: $m=1$ for undirected exploration, and $m=0$ for reaching user-specified goal images. We find that this simple masking strategy is very effective for training a single policy for both goal-reaching and undirected exploration.

为训练同时服务目标到达与无向探索的共享策略，我们修改第 III 节所述 ViNT 架构，引入二值「目标掩码」$m$，使得 $c_{t}=f(\psi(o_{i}),\phi(o_{t},o_{g}),m)$。$m$ 可用于掩蔽目标 token $\phi(o_{t},o_{g})$，从而阻断策略的有目标通路。我们通过设 $m=1$ 实现掩码注意力，使下游对 $c_{t}$ 的计算不去关注目标 token；设 $m=0$ 实现非掩码注意力，使目标 token 与观测 token 一并参与下游计算。训练时，$m$ 从概率为 $p_{m}$ 的伯努利分布采样。我们固定 $p_{m}=0.5$，对应目标到达与无向探索样本数量相等。测试时按期望行为设定 $m$：$m=1$ 做无向探索，$m=0$ 到达用户指定的目标图像。我们发现这一简单掩码策略对训练单一策略同时覆盖两种行为非常有效。

IV-B Diffusion Policy

IV-B 扩散策略

While goal masking allows for a convenient way to condition the policy on a goal image, the distribution over actions that results from this, particularly when a goal is not provided, can be very complex. For example, at a junction, the policy might need to assign high probabilities to left and right turns, but low probability to any action that might result in a collision. Training a single policy to model such complex, multimodal distributions over action sequences is challenging. To effectively model such complex distributions, we use a diffusion model [23] to approximate the conditional distribution $p({\mathbf{a}}_{t}|c_{t})$, where $c_{t}$ is the observation context obtained after goal masking.

目标掩码为以目标图像条件化策略提供了便利途径，但由此得到的动作分布——尤其是未提供目标时——可能非常复杂。例如在路口，策略可能需要对左右转弯赋予高概率，而对任何可能导致碰撞的动作赋予低概率。训练单一策略去建模动作序列上如此复杂的多模态分布颇具挑战。为有效建模，我们用扩散模型 [23] 近似条件分布 $p({\mathbf{a}}_{t}|c_{t})$，其中 $c_{t}$ 是目标掩码之后得到的观测上下文。

We sample a sequence of future actions ${\mathbf{a}}_{t}^{K}$ from a Gaussian distribution and perform $K$ iterations of denoising to produce a series of intermediate action sequences with decreasing levels of noise $\{{\mathbf{a}}_{t}^{K},{\mathbf{a}}_{t}^{K-1},\ldots,{\mathbf{a}}_{t}^{0}\}$, until the desired noise-free output ${\mathbf{a}}_{t}^{0}$ is formed. The iterative denoising process follows the equation

我们从高斯分布采样未来动作序列 ${\mathbf{a}}_{t}^{K}$，并执行 $K$ 次去噪迭代，产生噪声水平递减的一系列中间动作序列 $\{{\mathbf{a}}_{t}^{K},{\mathbf{a}}_{t}^{K-1},\ldots,{\mathbf{a}}_{t}^{0}\}$，直至形成期望的无噪声输出 ${\mathbf{a}}_{t}^{0}$。迭代去噪过程遵循方程

$${\mathbf{a}}^{k-1}_{t}=\alpha\cdot(a^{k}_{t}-\gamma\epsilon_{\theta}(c_{t},{\mathbf{a}}^{k}_{t},k)+\mathcal{N}(0,\sigma^{2}I)) \tag{1}$$

where $k$ is the number of denoising steps, $\epsilon_{\theta}$ is a noise prediction network parameterized by $\theta$, and $\alpha,\gamma$ and $\sigma$ are functions of the noise schedule.

其中 $k$ 为去噪步数，$\epsilon_{\theta}$ 为参数 $\theta$ 的噪声预测网络，$\alpha,\gamma$ 与 $\sigma$ 是噪声日程的函数。

The noise prediction network $\varepsilon_{\theta}$ is conditioned on the observation context $c_{t}$, which may or may not include goal information, as determined by the mask $m$. Note that we model the conditional (and not joint) action distribution, excluding $c_{t}$ from the output of the denoising process, which enables real-time control and end-to-end training of the diffusion process and visual encoder. During training, we train $\varepsilon_{\theta}$ by adding noise to ground truth action sequences. The predicted noise is compared to the actual noise through the mean squared error (MSE) loss.

噪声预测网络 $\varepsilon_{\theta}$ 以观测上下文 $c_{t}$ 为条件；$c_{t}$ 是否包含目标信息由掩码 $m$ 决定。注意我们建模的是条件（而非联合）动作分布，去噪过程的输出中排除 $c_{t}$，从而支持实时控制以及对扩散过程与视觉编码器的端到端训练。训练时，我们向真实动作序列加噪来训练 $\varepsilon_{\theta}$，以均方误差（MSE）损失比较预测噪声与实际噪声。

Fig. 3: Visualizing the task-agnostic (yellow) and goal-directed pathways for two goal images (green, blue) learned by NoMaD. NoMaD predicts a bimodal distribution of collision-free actions in the absence of a goal, and snaps to a narrower distribution after conditioning on two different goal images.

图 3：可视化 NoMaD 学到的任务无关通路（黄）以及两条目标图像（绿、蓝）下的目标导向通路。无目标时，NoMaD 预测无碰撞动作的双峰分布；以两张不同目标图像为条件后，分布收束为更窄的分布。

IV-C Training Details

IV-C 训练细节

The NoMaD model architecture is illustrated in Figure 2. We train NoMaD on a combination of GNM and SACSoN datasets, large heterogeneous datasets collected across a diverse set of environments and robotic platforms, including pedestrian-rich environments, spanning over 100 hours of real-world trajectories [40, 41]. NoMaD is trained end-to-end with supervised learning using the following loss function:

NoMaD 模型架构见图 2。我们在 GNM 与 SACSoN 数据集的组合上训练 NoMaD——这是跨多样环境与机器人平台采集的大规模异构数据，包含行人密集场景，真实轨迹超过 100 小时 [40, 41]。NoMaD 用监督学习端到端训练，损失函数如下：

$$\begin{split}\mathcal{L}{\text{NoMaD }}(\phi,\psi,f,\theta,f{d})=&,MSE(\varepsilon^{k},\varepsilon_{\theta}(c_{t},{\mathbf{a}}{t}^{0}+\varepsilon^{k},k))\
&+\lambda\cdot MSE(d({\mathbf{o}}{t},o_{g}),f_{d}(c_{t}))\end{split} \tag{2}$$

where $\psi,\phi$ correspond to visual encoders for the observation and goal images, $f$ corresponds to the Transformer layers, $\theta$ corresponds to the parameters of the diffusion process, and $f_{d}$ corresponds to the temporal distance predictor. $\lambda=10^{-4}$ is a hyperparameter that controls the relative weight of the temporal distance loss. During training, we use a goal masking probability $p_{m}=0.5$, corresponding to an equal number of goal-reaching and undirected exploration samples. The diffusion policy is trained with the Square Cosine Noise Scheduler [42] and $K=10$ denoising steps. We uniformly sample a denoising iteration $k$, and we also sample a corresponding noise $\epsilon^{k}$ with the variance defined at iteration $k$. The noise prediction network, $\epsilon_{\theta}$, consists of a 1D conditional U-Net [29, 31] with 15 convolutional layers.

其中 $\psi,\phi$ 对应观测与目标图像的视觉编码器，$f$ 对应 Transformer 层，$\theta$ 对应扩散过程参数，$f_{d}$ 对应时间距离预测器。$\lambda=10^{-4}$ 是控制时间距离损失相对权重的超参数。训练时目标掩码概率 $p_{m}=0.5$，对应目标到达与无向探索样本数量相等。扩散策略使用 Square Cosine Noise Scheduler [42] 与 $K=10$ 去噪步。我们均匀采样去噪迭代 $k$，并按该迭代处定义的方差采样对应噪声 $\epsilon^{k}$。噪声预测网络 $\epsilon_{\theta}$ 是含 15 个卷积层的一维条件 U-Net [29, 31]。

We use the AdamW optimizer [43] with a learning rate of $10^{-4}$ and train NoMaD for 30 epochs with a batch size of $256$. We use cosine scheduling and warmup to stabilize the training process and follow other hyperparameters from ViNT [3]. For the ViNT observation encoder, we use EfficientNet-B0 [39] to tokenize observations and goals into 256-dimensional embeddings, followed by a Transformer decoder with 4 layers and 4 heads.

我们使用 AdamW 优化器 [43]，学习率 $10^{-4}$，批大小 $256$，训练 NoMaD 共 30 个 epoch。采用余弦调度与 warmup 稳定训练，其余超参数沿用 ViNT [3]。对 ViNT 观测编码器，我们用 EfficientNet-B0 [39] 将观测与目标 token 化为 256 维嵌入，后接 4 层、4 头的 Transformer 解码器。

V Evaluation

V 评估

We evaluate NoMaD in 6 distinct indoor and outdoor environments, and formulate our experiments to answer the following questions:

我们在 6 个不同的室内外环境中评估 NoMaD，并围绕以下问题组织实验：

Q1. How does NoMaD compare to prior work for visual exploration and goal-reaching in real-world environments?
Q2. How does a joint task-agnostic and task-specific policy compare to the individual behavior policies?
Q3. How important is the choice of visual encoder and goal masking to the performance of NoMaD?

Q1. 在真实环境中的视觉探索与寻的上，NoMaD 相对先前工作表现如何？
Q2. 联合的任务无关与任务特定策略，相对各自独立的行为策略如何？
Q3. 视觉编码器与目标掩码的选择对 NoMaD 表现有多重要？

V-A Benchmarking Performance

V-A 性能基准

Fig. 4: Visualizing rollouts of NoMaD deployed in challenging indoor (top) and outdoor (bottom) environments on the LoCoBot platform, showcasing successful exploration trajectories. Future action samples from the undirected mode are shown in yellow, and the action selected by the high-level planner is shown in blue. The predicted actions follow implicit navigational affordances, such as following hallways, and become multimodal at decision points, such as intersections in the hallway.

图 4：在 LoCoBot 平台上部署 NoMaD，于具有挑战性的室内（上）与室外（下）环境中可视化 rollout，展示成功探索轨迹。无向模式下的未来动作样本以黄色标出，高层规划器选定的动作以蓝色标出。预测动作遵循隐式导航可供性（例如沿走廊前进），并在决策点（例如走廊交叉口）呈现多模态。

Towards understanding Q1, we compare NoMaD to six performant baselines for exploration and navigation in 6 challenging real-world environments. We follow the experimental setup of ViNT [3] and evaluate the methods on their ability to (i) explore a novel environment effectively in search of a goal position, or (ii) reach a goal indicated by an image in a previously explored environment, where the robot uses the policy to create a topological graph as episodic memory. All baselines are trained on a combination of GNM and SACSoN datasets for 20 epochs, and we perform minimal hyperparameter tuning to ensure stable training for each baseline. We report the mean success rate for each baseline, as well as the mean number of collisions per experiment.

为回答 Q1，我们在 6 个具有挑战性的真实环境中，将 NoMaD 与六种表现良好的探索与导航基线比较。我们沿用 ViNT [3] 的实验设定，评估各方法在以下能力上的表现：(i) 在新颖环境中有效探索以搜寻目标位置；或 (ii) 在先前已探索环境中到达图像指示的目标——此时机器人用策略构建作为情节记忆的拓扑图。所有基线均在 GNM 与 SACSoN 组合数据上训练 20 个 epoch，我们仅做最少超参数调优以保证各基线训练稳定。报告每种基线的平均成功率，以及每次实验的平均碰撞次数。

VIB: We use the authors’ implementation of a latent goal model for exploration [17], which uses a variational information bottleneck (VIB) to model a distribution of actions conditioned on observations.

VIB： 我们使用作者实现的潜目标探索模型 [17]，以变分信息瓶颈（VIB）建模以观测为条件的动作分布。

Masked ViNT: We integrate our goal masking with the ViNT policy [3] to flexibly condition on the observation context $c_{t}$. This baseline predicts point estimates of future actions conditioned on $c_{t}$, rather than modeling the distribution.

Masked ViNT： 将我们的目标掩码与 ViNT 策略 [3] 结合，以灵活地在观测上下文 $c_{t}$ 上条件化。该基线预测以 $c_{t}$ 为条件的未来动作点估计，而非对分布建模。

Autoregressive: This baseline uses autoregressive predictions over a discretized action space to better represent multimodal action distributions. Our implementation uses a categorical representation of the action distribution, goal masking, and the same visual encoder design.

Autoregressive： 该基线在离散化动作空间上做自回归预测，以更好表示多模态动作分布。我们的实现使用动作分布的类别表示、目标掩码，以及相同的视觉编码器设计。

Subgoal Diffusion: We use the authors’ implementation of the ViNT system [3] that pairs a goal-conditioned policy with an image diffusion model for generating candidate subgoal images, which are used by the policy to predict exploration actions. This is the best published baseline we compare against, but uses a $15\times$ larger model than NoMaD.

Subgoal Diffusion： 使用作者实现的 ViNT 系统 [3]：将有目标策略与图像扩散模型配对，生成候选子目标图像，再由策略据此预测探索动作。这是我们比较的已发表最佳基线，但模型规模约为 NoMaD 的 $15\times$。

Random Subgoals: A variation of the above ViNT system which replaces subgoal diffusion with randomly sampling the training data for a candidate subgoal, which is passed to the goal-conditioned policy to predict exploration actions. This baseline does not use image diffusion, and has comparable parameter-count to NoMaD.

Random Subgoals： 上述 ViNT 系统的变体，以从训练数据中随机采样候选子目标取代子目标扩散，再交给有目标策略预测探索动作。该基线不使用图像扩散，参数量与 NoMaD 相当。

		Exploration	Navigation
Method	Params	Success	Coll.	Success
Masked ViNTm	15M	50%	1.0	30%
VIB [17]	6M	30%	4.0	15%
Autoregressivem	19M	90%	2.0	60%
Random Subgoals [3]	30M	70%	2.7	90%
Subgoal Diffusion [3]	335M	77%	1.7	90%
NoMaD	19M	98%	0.2	90%

TABLE I: NoMaD paired with a topological graph consistently outperforms all baselines for the task of exploration in previously unseen environments, and navigation in known environments. Most notably, NoMaD outperforms the state-of-the-art (Subgoal Diffusion) by 25%, while also avoiding collisions and requiring $15\times$ fewer parameters. mThese baselines that use goal masking.

		探索	导航
方法	参数量	成功率	碰撞	成功率
Masked ViNTm	15M	50%	1.0	30%
VIB [17]	6M	30%	4.0	15%
Autoregressivem	19M	90%	2.0	60%
Random Subgoals [3]	30M	70%	2.7	90%
Subgoal Diffusion [3]	335M	77%	1.7	90%
NoMaD	19M	98%	0.2	90%

表 I：与拓扑图配对的 NoMaD 在先前未见环境中的探索，以及已知环境中的导航上，持续优于所有基线。尤其值得注意的是，NoMaD 相对当前最先进（Subgoal Diffusion）提升 25%，同时避免碰撞，且参数量少约 $15\times$。m 表示使用目标掩码的基线。

Table I summarizes the results of our experiments in 5 challenging indoor and outdoor environments. VIB and Masked ViNT struggle in all the environments we tested and frequently end in collisions, likely due to challenges with effectively modeling multimodal action distributions. The Autoregressive baseline uses a more expressive policy class and outperforms these baselines, but struggles in complex environments. Furthermore, the deployed policy tends to be jerky and slow to respond to dynamic obstacles in the environment, likely due to the discretized action space (see supplemental video for experiments). NoMaD consistently outperforms all baselines and results in smooth, reactive policies. For exploratory goal discovery, NoMaD outperforms the best published baseline (Subgoal Diffusion) by over 25% in terms of both efficiency and collision avoidance, and succeeds in all but the hardest environment. For navigation in known environments, using a topological graph, NoMaD matches the performance of the best published baseline, while also requiring a $15\times$ smaller model and running entirely on-the-edge. Figure 4 shows example rollouts of the NoMaD policy exploring unknown indoor and outdoor environments in search for the goal.

表 I 汇总了我们在 5 个具有挑战性的室内外环境中的实验结果。VIB 与 Masked ViNT 在所有测试环境中都表现吃力，常以碰撞告终，很可能源于难以有效建模多模态动作分布。Autoregressive 基线使用更具表达力的策略类，优于上述基线，但在复杂环境中仍吃力。此外，部署后的策略往往抖动、对动态障碍响应偏慢，可能与离散化动作空间有关（见补充视频中的实验）。NoMaD 持续优于所有基线，并得到平滑、反应迅速的策略。在探索式目标发现上，NoMaD 相对已发表最佳基线（Subgoal Diffusion）在效率与避撞上均提升超过 25%，除最难环境外全部成功。在已知环境导航上，配合拓扑图，NoMaD 与已发表最佳基线表现持平，同时模型小约 $15\times$，并完全在边缘侧运行。图 4 展示 NoMaD 策略在未知室内外环境中搜寻目标的示例 rollout。

Analyzing the policy predictions across baselines (see Figure 5), we find that while the Autoregressive policy representation can (in principle) express multimodal distributions, the predictions are largely unimodal, equivalent to the policy learning the average action distribution. The Subgoal Diffusion baseline tends to represent the multiple modes well, but is not very robust. NoMaD consistently captures the multimodal distribution, and also makes accurate predictions when conditioned on a goal image.

分析各基线的策略预测（见图 5），我们发现：尽管 Autoregressive 策略表示（原则上）可表达多模态分布，预测却大体是单峰的，等价于策略在学习平均动作分布。Subgoal Diffusion 基线往往能较好表示多个模态，但不够稳健。NoMaD 持续捕捉多模态分布，并在以目标图像为条件时也能做出准确预测。

V-B Unified v/s Dedicated Policies

V-B 统一策略对比专用策略

Fig. 5: Examples of action predictions from NoMaD and baselines in undirected mode (yellow) and goal-directed mode with two different goal images (blue towards left, green towards right). Only NoMaD can consistently represent multimodal undirected predictions while avoiding collisions with pillars or walls, as well as correctly predicting the goal-conditioned action predictions for the two goals. †Note that Subgoal Diffusion and Random Subgoals baselines only represent point estimates when conditioned on a goal image.

图 5：NoMaD 与各基线在无向模式（黄）以及两种不同目标图像的目标导向模式（蓝向左、绿向右）下的动作预测示例。只有 NoMaD 能持续表示多模态的无向预测并避免与柱子或墙壁碰撞，同时正确预测两个目标下的有条件动作。†注意 Subgoal Diffusion 与 Random Subgoals 基线在以目标图像为条件时仅表示点估计。

Method	Params	Undirected	Goal-Conditioned
Diffusion Policy [31]	15M	98%	✗
ViNT Policy [3]	16M	✗	92%
NoMaD	19M	98%	92%

TABLE II: Despite having comparable model capacities, NoMaD matches the performance of the best individual behavior policies for undireced exploration adn goal-conditioned navigation.

方法	参数量	无向	有目标
Diffusion Policy [31]	15M	98%	✗
ViNT Policy [3]	16M	✗	92%
NoMaD	19M	98%	92%

表 II：尽管模型容量相当，NoMaD 在无向探索与有目标导航上均匹配各自最佳的独立行为策略表现。

With the flexibility that a policy with task-specific and task-agnostic capabilities offers, Q2 aims to understand the impact of goal masking on the individual behaviors learned by the policy. Specifically, we compare the performance of the jointly trained NoMaD model to the best-performing goal-conditioned and undirected models. We report the mean success rate for each baseline.

在兼具任务特定与任务无关能力的策略所提供的灵活性下，Q2 旨在理解目标掩码对策略所学各行为的影响。具体地，我们比较联合训练的 NoMaD 模型与表现最佳的有目标、无向模型，并报告各基线的平均成功率。

Diffusion Policy: We train a diffusion policy [31] with the same visual encoder as NoMaD and $m$ = 0. This is the best exploration baseline, outperforming both VIB and IBC.

Diffusion Policy： 我们用与 NoMaD 相同的视觉编码器训练扩散策略 [31]，且 $m$ = 0。这是最佳探索基线，优于 VIB 与 IBC。

ViNT Policy: We use the authors’ published checkpoint of the ViNT navigation policy [3], which predicts point estimates of future actions conditioned on observations and a goal. This is the best navigation baseline.

ViNT Policy： 我们使用作者公开发布的 ViNT 导航策略检查点 [3]，以观测与目标为条件预测未来动作的点估计。这是最佳导航基线。

Comparing the unified NoMaD policy to the above, we find that despite having comparable model capacities, the unified policy trained with goal masking matches the performance of ViNT policy for goal-conditioned navigation and DP for undirected exploration. This suggests that training for these two behaviors involves learning shared representations and affordances, and a single policy can indeed excel at both task-agnostic and task-oriented behaviors simultaneously.

将统一的 NoMaD 策略与上述对比，我们发现：尽管模型容量相当，经目标掩码训练的统一策略在有目标导航上匹配 ViNT 策略，在无向探索上匹配 DP。这表明训练这两种行为涉及学习共享表示与可供性，单一策略确实可以同时在任务无关与任务导向行为上表现优异。

V-C Visual Encoder and Goal Masking

V-C 视觉编码器与目标掩码

Visual Encoder	Success	# Collisions
Late Fusion CNN	52%	3.2
Early Fusion CNN	68%	1.5
ViT	32%	2.5
NoMaD	98%	0.2

TABLE III: The performance of NoMaD depends on the choice of visual encoder and goal masking strategy. The ViNT encoder with attention-based goal masking outperforms all alternatives.

视觉编码器	成功率	碰撞次数
Late Fusion CNN	52%	3.2
Early Fusion CNN	68%	1.5
ViT	32%	2.5
NoMaD	98%	0.2

表 III：NoMaD 的表现取决于视觉编码器与目标掩码策略的选择。带基于注意力目标掩码的 ViNT 编码器优于所有替代方案。

We explore variations of the visual encoder and goal masking architectures to understand Q3. We consider two alternative visual encoder designs based on CNN and ViT backbones, and implement goal masking in different ways. We report the mean success rate for each baseline, as well as the mean number of collisions per experiment.

我们探索视觉编码器与目标掩码架构的变体，以理解 Q3。我们考虑基于 CNN 与 ViT 骨干的两种替代视觉编码器设计，并以不同方式实现目标掩码。报告各基线的平均成功率，以及每次实验的平均碰撞次数。

Early/Late Fusion CNN: We use convolutional encoders followed by an MLP to encode the observation and goal images, and use dropout on the goal embeddings followed by another MLP block to flexibly condition the observation context $c_{t}$ on the goal. $c_{t}$ obtained after dropout is used for conditioning the diffusion model in the same manner as NoMaD. We use a straight-through estimator [44] for propagating gradients to the observation and goal encoders during training. The goal can be combined with the observations either before or after the final MLP layers.

Early/Late Fusion CNN： 我们用卷积编码器后接 MLP 编码观测与目标图像，对目标嵌入做 dropout，再经另一 MLP 块，使观测上下文 $c_{t}$ 可灵活地以目标为条件。dropout 后得到的 $c_{t}$ 以与 NoMaD 相同的方式条件化扩散模型。训练时用直通估计器 [44] 将梯度传播到观测与目标编码器。目标可在最终 MLP 层之前或之后与观测结合。

ViT: We divide the observation and goal images into $6\times 6$ patches, and encode them using a Vision Transformer [45] into observation context $c_{t}$. We use attention masks to block the goal patches from propagating information downstream.

ViT： 我们将观测与目标图像划分为 $6\times 6$ 的 patch，用 Vision Transformer [45] 编码为观测上下文 $c_{t}$。用注意力掩码阻断目标 patch 向下游传播信息。

We find the choice of visual encoder to be crucial for training diffusion policies, as summarized in Table III. NoMaD outperforms both the ViT- and CNN-based architectures, successfully reaching the goal while avoiding collisions. CNN with early fusion outperforms late fusion, confirming similar analysis in prior work [38, 3], but struggles to effectively condition on goal information. Despite it’s high capacity, the ViT encoder struggled learn a good policy, likely due to optimization challenges in training end-to-end with diffusion.

我们发现视觉编码器的选择对训练扩散策略至关重要，如表 III 所汇总。NoMaD 优于基于 ViT 与 CNN 的架构，成功到达目标并避免碰撞。早期融合的 CNN 优于晚期融合，印证了先前工作中的类似分析 [38, 3]，但难以有效以目标信息为条件。尽管容量高，ViT 编码器仍难以学到好策略，可能源于与扩散端到端训练时的优化挑战。

VI Discussion

VI 讨论

We presented NoMaD, the first instantiation of a goal-conditioned diffusion policy, that can perform both task-agnostic exploration and task-oriented navigation. Our unified navigation policy uses a high-capacity Transformer encoder with masked attention approach to flexibly condition on the task, such as goal images for navigation, and models the actions conditioned on observations using a diffusion model. We study the performance of this unified model in the context of long-horizon exploration and navigation in previously unseen indoor and outdoor environments, demonstrating over 25% improvements in performance over the state-of-the-art in previously unseen settings, while also requiring $15\times$ fewer computational resources.

我们提出了 NoMaD——首个有目标扩散策略的实例化，可同时完成任务无关探索与任务导向导航。统一导航策略用高容量 Transformer 编码器配合掩码注意力，灵活地以任务（例如用于导航的目标图像）为条件，并用扩散模型在以观测为条件的动作上建模。我们在先前未见的室内外环境中，于长时程探索与导航背景下研究该统一模型的表现：相对当前最先进方法，在先前未见设定上性能提升超过 25%，同时所需计算资源少约 $15\times$。

While our experiments provide a proof of concept that unified policies can provide more effective navigation in new environments, our system has a number of limitations that could be addressed in future work. The navigation tasks are specified via goal images which, though quite general, are sometimes not the most natural modalities for users to employ. Extending our approach into a complete navigation system that can accommodate a variety of goal modalities, including language and spatial coordinates, would make our approach more broadly applicable. Additionally, our exploration method uses a standard frontier-based exploration strategy for high-level planning, leveraging our policy to explore at the frontier. Intelligently selecting which regions to explore, such as strategies based on semantics and prior knowledge, could further improve performance. We hope that these directions would lead to even more practical and capable systems, enabled by our policy representation.

尽管实验提供了统一策略能在新环境中提供更有效导航的概念验证，我们的系统仍有若干局限，可供未来工作改进。导航任务以目标图像指定——虽相当通用，有时并非用户最自然的模态。将我们的方法扩展为能容纳多种目标模态（包括语言与空间坐标）的完整导航系统，会使其适用面更广。此外，我们的探索方法对高层规划使用标准的基于前沿的策略，借助我们的策略在前沿处探索。智能地选择探索哪些区域——例如基于语义与先验知识的策略——可能进一步提升表现。我们希望这些方向能导向更实用、更有能力的系统，而我们的策略表示正是其基础。

Acknowledgments

致谢

This research was partly supported by ARL DCIST CRA W911NF-17-2-0181, ARO W911NF-21-1-0097, IIS-2150826, and compute from Google TPU Research Cloud, NSF CloudBank, and the Berkeley Research Computing program. The authors would like to thank Kyle Stachowicz for useful discussions in early stages of the project, and Cheng Chi for help with reproducing the diffusion policy. The authors would also like to thank Michael Equi, Nathaniel Simon, and Anirudha Majumdar, for testing and providing feedback on preliminary versions of the NoMaD model.

本研究部分得到 ARL DCIST CRA W911NF-17-2-0181、ARO W911NF-21-1-0097、IIS-2150826 以及 Google TPU Research Cloud、NSF CloudBank 与 Berkeley Research Computing 项目算力的支持。作者感谢 Kyle Stachowicz 在项目早期阶段的有益讨论，以及 Cheng Chi 在复现扩散策略上的帮助。作者亦感谢 Michael Equi、Nathaniel Simon 与 Anirudha Majumdar 对 NoMaD 模型初步版本的测试与反馈。

References

参考文献

[1] A. Faust, K. Oslund, O. Ramirez, A. Francis, L. Tapia, M. Fiser, and J. Davidson, “Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning,” in 2018 International Conference on Robotics and Automation (ICRA) , 2018.

[2] C. Li, F. Xia, R. Martín-Martín, and S. Savarese, “HRL4IN: hierarchical reinforcement learning for interactive navigation with mobile manipulators,” in 3rd Annual Conference on Robot Learning (CoRL) , 2019.

[3] D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine, “ViNT: A Foundation Model for Visual Navigation,” in 7th Annual Conference on Robot Learning (CoRL) , 2023.

[4] B. Kuipers and Y.-T. Byun, “A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations,” Robotics and Autonomous Systems , 1991, special Issue Toward Learning Robots.

[5] F. Bourgault, A. A. Makarenko, S. B. Williams, B. Grocholsky, and H. F. Durrant-Whyte, “Information based adaptive robotic exploration,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2002.

[6] T. Kollar and N. Roy, “Efficient optimization of information-theoretic exploration in slam,” in AAAI Conference on Artificial Intelligence (AAAI) , 2008.

[7] W. Tabib, K. Goel, J. Yao, M. Dabhi, C. Boirum, and N. Michael, “Real-time information-theoretic exploration with gaussian mixture model maps.” in Robotics: Science and Systems , 2019.

[8] B. Yamauchi, “A frontier-based approach for autonomous exploration,” in IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA) , 1997.

[9] B. Charrow, S. Liu, V. Kumar, and N. Michael, “Information-theoretic mapping using cauchy-schwarz quadratic mutual information,” in International Conference on Robotics and Automation (ICRA) , 2015.

[10] D. Holz, N. Basilico, F. Amigoni, and S. Behnke, “A comparative evaluation of exploration strategies and heuristics to improve them,” 2011.

[11] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine, “ViNG: Learning Open-World Navigation with Visual Goals,” in International Conference on Robotics and Automation (ICRA) , 2021.

[12] X. Meng, N. Ratliff, Y. Xiang, and D. Fox, “Scaling Local Control to Large-Scale Topological Navigation,” in International Conference on Robotics and Automation (ICRA) , 2020.

[13] D. S. Chaplot, H. Jiang, S. Gupta, and A. Gupta, “Semantic curiosity for active visual learning,” in ECCV , 2020.

[14] T. Chen, S. Gupta, and A. Gupta, “Learning exploration policies for navigation,” in International Conference on Learning Representations (ICLR) , 2019.

[15] H. Tan, L. Yu, and M. Bansal, “Learning to navigate unseen environments: Back translation with environmental dropout,” in 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)) , 2019.

[16] A. Kadian, J. Truong, A. Gokaslan, A. Clegg, E. Wijmans, S. Lee, M. Savva, S. Chernova, and D. Batra, “Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?” IEEE Robotics and Automation Letters , 2020.

[17] D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine, “Rapid exploration for open-world navigation with latent goal models,” in Conference on Robot Learning (CoRL) , 2021.

[18] D. S. Chaplot, R. Salakhutdinov, A. Gupta, and S. Gupta, “Neural topological slam for visual navigation,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

[19] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self-supervised prediction,” in International Conference on Machine Learning , 2017.

[20] A. Khazatsky, A. Nair, D. Jing, and S. Levine, “What can i do here? learning new skills by imagining visual affordances,” in IEEE International Conference on Robotics and Automation (ICRA) , 2021.

[21] K. Fang, P. Yin, A. Nair, H. R. Walke, G. Yan, and S. Levine, “Generalization with lossy affordances: Leveraging broad offline data for learning visuomotor tasks,” in 6th Annual Conference on Robot Learning , 2022.

[22] T. Gervet, S. Chintala, D. Batra, J. Malik, and D. S. Chaplot, “Navigating to objects in the real world,” Science Robotics , 2023.

[23] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Neural Information Processing Systems , 2020.

[24] L. Metz, J. Ibarz, N. Jaitly, and J. Davidson, “Discrete sequential prediction of continuous actions for deep rl,” 2019.

[25] R. Dadashi, L. Hussenot, D. Vincent, S. Girgin, A. Raichuk, M. Geist, and O. Pietquin, “Continuous control with action quantization from demonstrations,” in 39th International Conference on Machine Learning (ICML) , 2022.

[26] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto, “Behavior transformers: Cloning $k$ modes with one stone,” in Advances in Neural Information Processing Systems (NeurIPS) , A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022.

[27] Y. Chebotar et al. , “Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions,” in 7th Annual Conference on Robot Learning (CoRL) , 2023.

[28] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson, “Implicit behavioral cloning,” in 5th Annual Conference on Robot Learning (CoRL) , 2021.

[29] M. Janner, Y. Du, J. Tenenbaum, and S. Levine, “Planning with diffusion for flexible behavior synthesis,” in International Conference on Machine Learning (ICML) , 2022.

[30] Z. Wang, J. J. Hunt, and M. Zhou, “Diffusion policies as an expressive policy class for offline reinforcement learning,” in The Eleventh International Conference on Learning Representations (ICLR) , 2023.

[31] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” in Robotics: Science and Systems (RSS) , 2023.

[32] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, and S. Devlin, “Imitating human behaviour with diffusion models,” in The Eleventh International Conference on Learning Representations (ICLR) , 2023.

[33] M. Reuss, M. Li, X. Jia, and R. Lioutikov, “Goal conditioned imitation learning using score-based diffusion policies,” in Robotics: Science and Systems (RSS) , 2023.

[34] P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine, “Idql: Implicit q-learning as an actor-critic method with diffusion policies,” 2023.

[35] D. Shah and S. Levine, “ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints,” in Robotics: Science and Systems (RSS) , 2022.

[36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017.

[37] A. Brohan et al. , “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint , 2022.

[38] N. Nayakanti, R. Al-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp, “Wayformer: Motion forecasting via simple and efficient attention networks,” 2022.

[39] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” 2020.

[40] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine, “GNM: A General Navigation Model to Drive Any Robot,” in International Conference on Robotics and Automation (ICRA) , 2023.

[41] N. Hirose, D. Shah, A. Sridhar, and S. Levine, “Sacson: Scalable autonomous data collection for social navigation,” arXiv , 2023.

[42] A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in International Conference on Machine Learning (ICML) , 2021.

[43] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations (ICLR) , 2019.

[44] Y. Bengio, N. Léonard, and A. C. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” ArXiv , 2013.

[45] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations (ICLR) , 2021.