Planning-oriented Autonomous Driving

Planning-oriented Autonomous Driving

面向规划的自动驾驶

Yihan Hu Jiazhi Yang Li Chen Keyu Li Chonghao Sima Xizhou Zhu Affiliation: SenseTime Research * Equal contribution † Project lead https://github.com/OpenDriveLab/UniAD Siqi Chai Affiliation: Wuhan University Senyao Du Affiliation: Wuhan University Tianwei Lin Affiliation: Wuhan University Wenhai Wang Lewei Lu Affiliation: SenseTime Research * Equal contribution † Project lead https://github.com/OpenDriveLab/UniAD Xiaosong Jia Qiang Liu Affiliation: Wuhan University Jifeng Dai Yu Qiao Hongyang Li [2mm] OpenDriveLab OpenGVLab Shanghai AI Laboratory

Abstract

摘要

Modern autonomous driving system is characterized as modular tasks in sequential order, i.e., perception, prediction, and planning. In order to perform a wide diversity of tasks and achieve advanced-level intelligence, contemporary approaches either deploy standalone models for individual tasks, or design a multi-task paradigm with separate heads. However, they might suffer from accumulative errors or deficient task coordination. Instead, we argue that a favorable framework should be devised and optimized in pursuit of the ultimate goal, i.e., planning of the self-driving car. Oriented at this, we revisit the key components within perception and prediction, and prioritize the tasks such that all these tasks contribute to planning. We introduce Unified Autonomous Driving (UniAD), a comprehensive framework up-to-date that incorporates full-stack driving tasks in one network. It is exquisitely devised to leverage advantages of each module, and provide complementary feature abstractions for agent interaction from a global perspective. Tasks are communicated with unified query interfaces to facilitate each other toward planning. We instantiate UniAD on the challenging nuScenes benchmark. With extensive ablations, the effectiveness of using such a philosophy is proven by substantially outperforming previous state-of-the-arts in all aspects. Code and models are public.

现代自动驾驶系统常按感知、预测、规划等模块顺序组织。为覆盖多样任务并迈向更高智能，现有做法要么为各任务部署独立模型，要么用多任务范式配多头。它们容易累积误差，或任务协同不足。我们认为，良好框架应以最终目标——自车规划——来设计与优化。据此，我们重审感知与预测中的关键组件，并按「服务规划」重新排定任务优先级。我们提出 Unified Autonomous Driving（UniAD）：迄今将全栈驾驶任务纳入同一网络的综合框架。它精心利用各模块优势，从全局视角为智能体交互提供互补特征抽象；各任务经统一 query 接口互通，共同指向规划。我们在具有挑战的 nuScenes 基准上实例化 UniAD。大量消融表明，这一理念在各方面显著超过此前最优方法。代码与模型已公开。

1 Introduction

1 引言

With the successful development of deep learning, autonomous driving algorithms are assembled with a series of tasks[^1], including detection, tracking, mapping in perception; and motion and occupancy forecast in prediction. As depicted in Fig. 1(a), most industry solutions deploy standalone models for each task independently [68, 71], as long as the resource bandwidth of the onboard chip allows. Although such a design simplifies the R&D difficulty across teams, it bares the risk of information loss across modules, error accumulation and feature misalignment due to the isolation of optimization targets [66, 57, 82].

随着深度学习成功发展，自动驾驶算法被组装成一系列任务[^1]，包括感知中的检测、跟踪、建图，以及预测中的运动与占据预报。如图 1(a) 所示，多数工业方案在车载芯片带宽允许时，为各任务独立部署模型 [68, 71]。该设计虽简化跨团队研发难度，却因优化目标隔离而带来模块间信息损失、误差累积与特征错位的风险 [66, 57, 82]。

Figure 1: Comparison on the various designs of autonomous driving framework. (a) Most industrial solutions deploy separate models for different tasks. (b) The multi-task learning scheme shares a backbone with divided task heads. (c) The end-to-end paradigm unites modules in perception and prediction. Previous attempts either adopt a direct optimization on planning in (c.1) or devise the system with partial components in (c.2). Instead, we argue in (c.3) that a desirable system should be planning-oriented as well as properly organize preceding tasks to facilitate planning.

图 1： 自动驾驶框架多种设计的比较 。 (a) 多数工业方案为不同任务部署独立模型。 (b) 多任务学习共享骨干、分任务头。 (c) 端到端范式统一感知与预测模块。先前尝试或如 (c.1) 直接优化规划，或如 (c.2) 仅纳入部分组件。我们在 (c.3) 主张：理想系统应以规划为导向，并妥善组织上游任务以服务规划。

A more elegant design is to incorporate a wide span of tasks into a multi-task learning (MTL) paradigm, by plugging several task-specific heads into a shared feature extractor as shown in Fig. 1(b). This is a popular practice in many domains, including general vision [92, 108, 79], autonomous driving[^2] [105, 60, 101, 15], such as Transfuser [20], BEVerse [105], and industrialized products, e.g ., Mobileye [68], Tesla [87], Nvidia [71], etc . In MTL, the co-training strategy across tasks could leverage feature abstraction; it could effortlessly extend to additional tasks, and save computation cost for onboard chips. However, such a scheme may cause undesirable “negative transfer” [23, 64].

更优雅的设计是把广泛任务纳入多任务学习（MTL）范式：如图 1(b)，在共享特征提取器上插入若干任务头。这在通用视觉 [92, 108, 79]、自动驾驶[^2] [105, 60, 101, 15]（如 Transfuser [20]、BEVerse [105]）以及 Mobileye [68]、Tesla [87]、Nvidia [71] 等工业产品中都很常见。MTL 中跨任务共训可利用特征抽象，易扩展新任务，并节省车载算力。但该方案也可能导致不良的「负迁移」 [23, 64]。

By contrast, the emergence of end-to-end autonomous driving [19, 97, 15, 11, 38] unites all nodes from perception, prediction and planning as a whole . The choice and priority of preceding tasks should be determined in favor of planning. The system should be planning-oriented, exquisitely designed with certain components involved, such that there are few accumulative error as in the standalone option or negative transfer as in the MTL scheme. Table 1 describes the task taxonomy of different framework designs.

相比之下，端到端自动驾驶 [19, 97, 15, 11, 38] 把感知、预测与规划各节点连成 整体 。上游任务的取舍与优先级应由规划决定。系统应以规划为导向，精心纳入特定组件，从而减少独立方案中的累积误差与 MTL 中的负迁移。表 1 描述了不同框架设计的任务分类。

Following the end-to-end paradigm, one “tabula-rasa” practice is to directly predict the planned trajectory, without any explicit supervision of perception and prediction as shown in Fig. 1(c.1). Pioneering works [14, 21, 22, 16, 106, 78, 97, 95] verified this vanilla design in the closed-loop simulation [26]. While such a direction deserves further exploration, it is inadequate in safety guarantee and interpretability, especially for highly dynamic urban scenarios. In this paper, we lean toward another perspective and ask the following question: Toward a reliable and planning-oriented autonomous driving system, how to design the pipeline in favor of planning? which preceding tasks are requisite?

沿端到端范式，一种「白板」做法是直接预测规划轨迹，不对感知与预测做显式监督，如图 1(c.1)。开创性工作 [14, 21, 22, 16, 106, 78, 97, 95] 在闭环仿真 [26] 中验证了这一朴素设计。该方向值得继续探索，但对高度动态城市场景，安全保证与可解释性仍不足。本文转向另一视角，并提出： 面向可靠且以规划为导向的自动驾驶系统，应如何设计有利于规划的管线？哪些上游任务是必需的？

{}^{\text{\textdagger}}

Design	Approach	Perception	Prediction	Plan
Det.	Track	Map	Motion	Occ.
(b)	NMP [101]	✓			✓		✓
NEAT [19]			✓			✓
BEVerse [105]	✓		✓		✓
(c.1)	[14, 16, 78, 97]						✓
(c.2)	PnPNet ${}^{\text{\textdagger}}$ [57]	✓	✓		✓
ViP3D ${}^{\text{\textdagger}}$ [30]	✓	✓		✓
P3 [82]					✓	✓
MP3 [11]			✓		✓	✓
ST-P3 [38]			✓		✓	✓
LAV [15]	✓		✓	✓		✓
(c.3)	UniAD (ours)	✓	✓	✓	✓	✓	✓

{}^{\text{\textdagger}}

设计	方法	感知	预测	规划
检测	跟踪	地图	运动	占据
(b)	NMP [101]	✓			✓		✓
NEAT [19]			✓			✓
BEVerse [105]	✓		✓		✓
(c.1)	[14, 16, 78, 97]						✓
(c.2)	PnPNet ${}^{\text{\textdagger}}$ [57]	✓	✓		✓
ViP3D ${}^{\text{\textdagger}}$ [30]	✓	✓		✓
P3 [82]					✓	✓
MP3 [11]			✓		✓	✓
ST-P3 [38]			✓		✓	✓
LAV [15]	✓		✓	✓		✓
(c.3)	UniAD（本文）	✓	✓	✓	✓	✓	✓

An intuitive resolution would be to perceive surrounding objects, predict future behaviors and plan a safe maneuver explicitly, as illustrated in Fig. 1(c.2). Contemporary approaches [57, 30, 82, 11, 38] provide good insights and achieve impressive performance. However, we argue that the devil lies in the details; previous works more or less fail to consider certain components (see block (c.2) in Table 1), being reminiscent of the planning-oriented spirit. We elaborate on the detailed definition and terminology, the necessity of these modules in the Supplementary.

直观做法是显式感知周围物体、预测未来行为并规划安全动作，如图 1(c.2)。当代方法 [57, 30, 82, 11, 38] 很有启发且表现亮眼。但我们认为细节决定成败；先前工作或多或少遗漏某些组件（见表 1 中 (c.2) 块），未能充分体现规划导向。详细定义、术语与这些模块的必要性见补充材料。

To this end, we introduce UniAD, a Unified Autonomous Driving algorithm framework to leverage five essential tasks toward a safe and robust system as depicted in Fig. 1(c.3) and Table 1(c.3). UniAD is designed in a planning-oriented spirit. We argue that this is not a simple stack of tasks with mere engineering effort. A key component is the query-based design to connect all nodes. Compared to the classic bounding box representation, queries benefit from a larger receptive field to soften the compounding error from upstream predictions. Moreover, queries are flexible to model and encode a variety of interactions, e.g ., relations among multiple agents. To the best of our knowledge, UniAD is the first work to comprehensively investigate the joint cooperation of such a variety of tasks including perception, prediction and planning in the field of autonomous driving.

为此，我们提出 UniAD ：统一自动驾驶算法框架，纳入五项关键任务以迈向安全稳健系统，见图 1(c.3) 与表 1(c.3)。UniAD 以规划为导向设计。我们认为这 并非 简单堆叠任务的工程拼凑。关键是用基于 query 的设计连接所有节点。相对经典边界框表示，query 感受野更大，可软化上游预测的复合误差；query 也灵活建模多种交互，如多智能体关系。据我们所知，UniAD 是自动驾驶领域首次全面研究感知、预测与规划如此多任务联合协同的工作。

The contributions are summarized as follows. (a) we embrace a new outlook of autonomous driving framework following a planning-oriented philosophy, and demonstrate the necessity of effective task coordination, rather than standalone design or simple multi-task learning. (b) we present UniAD, a comprehensive end-to-end system that leverages a wide span of tasks. The key component to hit the ground running is the query design as interfaces connecting all nodes. As such, UniAD enjoys flexible intermediate representations and exchanging multi-task knowledge toward planning. (c) we instantiate UniAD on the challenging benchmark for realistic scenarios. Through extensive ablations, we verify the superiority of our method over previous state-of-the-arts in all aspects.

贡献总结如下。 (a) 我们以规划导向理念重新审视自动驾驶框架，并论证有效任务协同的必要性，而非独立设计或简单多任务学习。 (b) 我们提出 UniAD，覆盖广泛任务的端到端综合系统。落地关键是用 query 作为连接各节点的接口，使 UniAD 享有灵活中间表示，并交换多任务知识以服务规划。 (c) 我们在面向真实场景的挑战性基准上实例化 UniAD。经大量消融，验证方法在各方面优于此前最优。

We hope this work could shed some light on the target-driven design for the autonomous driving system, providing a starting point for coordinating various driving tasks.

希望本工作能为自动驾驶系统的目标驱动设计提供一些启发，并为协调多样驾驶任务提供一个起点。

Figure 2: Pipeline of Unified Autonomous Driving (UniAD). It is exquisitely devised following planning-oriented philosophy. Instead of a simple stack of tasks, we investigate the effect of each module in perception and prediction, leveraging the benefits of joint optimization from preceding nodes to final planning in the driving scene. All perception and prediction modules are designed in a transformer decoder structure, with task queries as interfaces connecting each node. A simple attention-based planner is in the end to predict future waypoints of the ego-vehicle considering the knowledge extracted from preceding nodes. The map over occupancy is for visual purpose only.

图 2： Unified Autonomous Driving（UniAD）管线 。按规划导向理念精心设计。我们并非简单堆叠任务，而是考察感知与预测中各模块的作用，利用从上游节点到最终规划的联合优化收益。所有感知与预测模块采用 Transformer 解码器结构，以任务 query 为接口连接各节点。末端是简单的基于注意力的规划器，结合上游提取的知识预测自车未来航点。占据上叠加的地图仅供可视化。

2 Methodology

2 方法

Q

Q

2.1 Perception: Tracking and Mapping

2.1 感知：跟踪与建图

B

B

N

N

2.2 Prediction: Motion Forecasting

2.2 预测：运动预报

Q_{A}

Q_{A}

N

N

Q_{a/m}=\texttt{MHCA}(\texttt{MHSA}(Q),Q_{A}/Q_{M}), \tag{1}

where MHCA, MHSA denote multi-head cross-attention and multi-head self-attention [91] respectively. As it is also important to focus on the intended position, i.e ., goal point, to refine the predicted trajectory, we devise an agent-goal point attention via deformable attention [109] as follows:

其中 MHCA、MHSA 分别表示多头交叉注意力与多头自注意力 [91]。关注意图位置（即目标点）以 refinement 预测轨迹同样重要，我们用可变形注意力 [109] 设计智能体—目标点注意力：

Q_{g}=\texttt{DeformAttn}(Q,\hat{\mathbf{x}}_{T}^{l-1},B), \tag{2}

\hat{\mathbf{x}}_{T}^{l-1}

\hat{\mathbf{x}}_{T}^{l-1}

Q_{\text{ctx}}

Q_{\text{ctx}}

\displaystyle Q_{\text{pos}}= \\ \displaystyle\text{MLP}(\text{PE}(I^{s}))+\text{MLP}(\text{PE}(I^{a})) \\ \displaystyle+ \\ \displaystyle\text{MLP}(\text{PE}(\hat{\mathbf{x}}_{0}))+\text{MLP}(\text{PE}(\hat{\mathbf{x}}_{T}^{l-1})).

\text{PE}(\cdot)

\text{PE}(\cdot)

Non-linear Optimization. Different from conventional motion forecasting works which have direct access to ground truth perceptual results, i.e ., agents’ location and corresponding tracks, we consider the prediction uncertainty from the prior module in our end-to-end paradigm. Brutally regressing the ground-truth waypoints from an imperfect detection position or heading angle may lead to unrealistic trajectory predictions with large curvature and acceleration. To tackle this, we adopt a non-linear smoother [7] to adjust the target trajectories and make them physically feasible given an imprecise starting point predicted by the upstream module. The process is:

非线性优化。 与可直接使用真实感知结果（智能体位置与对应轨迹）的常规运动预报不同，端到端范式需考虑上游模块的预测不确定性。从有偏差的检测位置或航向角硬回归真实航点，可能导致曲率与加速度过大的不现实轨迹。为此，我们采用非线性平滑器 [7]，在上游模块给出不精确起点时调整目标轨迹使其物理可行。过程为：

\tilde{\mathbf{x}}^{*}=\arg\min_{\mathbf{x}}c(\mathbf{x},\tilde{\mathbf{x}}), \tag{4}

\tilde{\mathbf{x}}

\tilde{\mathbf{x}}

c(\mathbf{x},\tilde{\mathbf{x}})=\lambda_{\text{xy}}\lVert\mathbf{x},\tilde{\mathbf{x}}\rVert_{2}+\lambda_{\text{goal}}\lVert\mathbf{x}_{T},\tilde{\mathbf{x}}_{T}\rVert_{2}+\sum_{\phi\in\Phi}\phi(\mathbf{x}), \tag{5}

\lambda_{\text{xy}}

\lambda_{\text{xy}}

2.3 Prediction: Occupancy Prediction

2.3 预测：占据预测

Occupancy grid map is a discretized BEV representation where each cell holds a belief indicating whether it is occupied, and the occupancy prediction task is to discover how the grid map changes in the future. Previous approaches utilize RNN structure for temporally expanding future predictions from observed BEV features [35, 38, 105]. However, they rely on highly hand-crafted clustering post-processing to generate per-agent occupancy maps, as they are mostly agent-agnostic by compressing BEV features as a whole into RNN hidden states. Due to the deficient usage of agent-wise knowledge, it is challenging for them to predict the behaviors of all agents globally, which is essential to understand how the scene evolves. To address this, we present OccFormer to incorporate both scene-level and agent-level semantics in two aspects: (1) a dense scene feature acquires agent-level features via an exquisitely designed attention module when unrolling to future horizons; (2) we produce instance-wise occupancy easily by a matrix multiplication between agent-level features and dense scene features without heavy post-processing.

占据栅格图是离散 BEV 表示，每个格子保存是否被占据的信念；占据预测任务是发现栅格图在未来如何变化。先前方法用 RNN 从观测 BEV 特征时序展开未来预测 [35, 38, 105]。但它们多把 BEV 特征整体压入 RNN 隐状态，依赖高度手工的聚类后处理生成逐智能体占据图，对智能体知识利用不足，难以全局预测所有智能体行为——而这正是理解场景演化所必需的。为此，我们提出 OccFormer，从两方面融合场景级与智能体级语义：(1) 稠密场景特征在向未来时域展开时，经精心设计的注意力模块获取智能体级特征；(2) 用智能体级特征与稠密场景特征的矩阵乘法，无需繁重后处理即可得到实例级占据。

T_{o}

T_{o}

G^{t}=\text{MLP}_{t}([Q_{A},P_{A},Q_{X}]),\ t={1,\dots,T_{o}}, \tag{6}

[\cdot]

[\cdot]

F_{\text{ds}}^{t}

F_{\text{ds}}^{t}

D_{\text{ds}}^{t}=\texttt{MHCA}(\texttt{MHSA}(F_{\text{ds}}^{t}),G^{t},\text{attn\_mask}=O_{m}^{t}). \tag{7}

O_{m}^{t}

O_{m}^{t}

H\!\times\!W

B

\hat{O}_{A}^{t}=U^{t}\cdot F_{\text{dec}}^{t}. \tag{8}

2.4 Planning

2.4 规划

B

B

\hat{\tau}

\hat{\tau}

\tau^{*}=\arg\min_{\tau}f(\tau,\hat{\tau},\hat{O}), \tag{9}

\hat{\tau}

\hat{\tau}

f(\tau,\hat{\tau},\hat{O})=\lambda_{\text{coord}}\lVert\tau,\hat{\tau}\rVert_{2}+\lambda_{\text{obs}}\sum_{t}\mathcal{D}(\tau_{t},\hat{O}^{t}), \tag{10}

\mathcal{D}(\tau_{t},\hat{O}^{t})=\sum_{(x,y)\in\mathcal{S}}\frac{1}{\sigma\sqrt{2\pi}}\text{exp}(-\frac{\lVert\tau_{t}-(x,y)\rVert_{2}^{2}}{2\sigma^{2}}). \tag{11}

\lambda_{\text{coord}}

\lambda_{\text{coord}}

\uparrow

ID	Modules	Tracking	Mapping	Motion Forecasting	Occupancy Prediction	Planning
Track	Map	Motion	Occ.	Plan	AMOTA $\uparrow$	AMOTP $\downarrow$	IDS $\downarrow$	IoU-lane $\uparrow$	IoU-road $\uparrow$	minADE $\downarrow$	minFDE $\downarrow$	MR $\downarrow$	IoU-n. $\uparrow$	IoU-f. $\uparrow$	VPQ-n. $\uparrow$	VPQ-f. $\uparrow$	avg.L2 $\downarrow$	avg.Col. $\downarrow$
0∗	✓	✓	✓	✓	✓	0.356	1.328	893	0.302	0.675	0.858	1.270	0.186	55.9	34.6	47.8	26.4	1.154	0.941
1	✓					0.348	1.333	791	-	-	-	-	-	-	-	-	-	-	-
2		✓				-	-	-	0.305	0.674	-	-	-	-	-	-	-	-	-
3	✓	✓				0.355	1.336	785	0.301	0.671	-	-	-	-	-	-	-	-	-
4			✓			-	-	-	-	-	0.815	1.224	0.182	-	-	-	-	-	-
5	✓		✓			0.360	1.350	919	-	-	0.751	1.109	0.162	-	-	-	-	-	-
6	✓	✓	✓			0.354	1.339	820	0.303	0.672	0.736(-9.7%)	1.066(-12.9%)	0.158	-	-	-	-	-	-
7				✓		-	-	-	-	-	-	-	-	60.5	37.0	52.4	29.8	-	-
8	✓			✓		0.360	1.322	809	-	-	-	-	-	62.1	38.4	52.2	32.1	-	-
9	✓	✓	✓	✓		0.359	1.359	1057	0.304	0.675	0.710(-3.5%)	1.005(-5.8%)	0.146	62.3	39.4	53.1	32.2	-	-
10					✓		-	-	-	-	-	-	-	-	-	-	-	1.131	0.773
11	✓	✓	✓		✓	0.366	1.337	889	0.303	0.672	0.741	1.077	0.157	-	-	-	-	1.014	0.717
12	✓	✓	✓	✓	✓	0.358	1.334	641	0.302	0.672	0.728	1.054	0.154	62.3	39.5	52.8	32.3	1.004	0.430

\uparrow

ID	模块	跟踪	建图	运动预报	占据预测	规划
Track	Map	Motion	Occ.	Plan	AMOTA $\uparrow$	AMOTP $\downarrow$	IDS $\downarrow$	IoU-lane $\uparrow$	IoU-road $\uparrow$	minADE $\downarrow$	minFDE $\downarrow$	MR $\downarrow$	IoU-n. $\uparrow$	IoU-f. $\uparrow$	VPQ-n. $\uparrow$	VPQ-f. $\uparrow$	avg.L2 $\downarrow$	avg.Col. $\downarrow$
0∗	✓	✓	✓	✓	✓	0.356	1.328	893	0.302	0.675	0.858	1.270	0.186	55.9	34.6	47.8	26.4	1.154	0.941
1	✓					0.348	1.333	791	-	-	-	-	-	-	-	-	-	-	-
2		✓				-	-	-	0.305	0.674	-	-	-	-	-	-	-	-	-
3	✓	✓				0.355	1.336	785	0.301	0.671	-	-	-	-	-	-	-	-	-
4			✓			-	-	-	-	-	0.815	1.224	0.182	-	-	-	-	-	-
5	✓		✓			0.360	1.350	919	-	-	0.751	1.109	0.162	-	-	-	-	-	-
6	✓	✓	✓			0.354	1.339	820	0.303	0.672	0.736(-9.7%)	1.066(-12.9%)	0.158	-	-	-	-	-	-
7				✓		-	-	-	-	-	-	-	-	60.5	37.0	52.4	29.8	-	-
8	✓			✓		0.360	1.322	809	-	-	-	-	-	62.1	38.4	52.2	32.1	-	-
9	✓	✓	✓	✓		0.359	1.359	1057	0.304	0.675	0.710(-3.5%)	1.005(-5.8%)	0.146	62.3	39.4	53.1	32.2	-	-
10					✓		-	-	-	-	-	-	-	-	-	-	-	1.131	0.773
11	✓	✓	✓		✓	0.366	1.337	889	0.303	0.672	0.741	1.077	0.157	-	-	-	-	1.014	0.717
12	✓	✓	✓	✓	✓	0.358	1.334	641	0.302	0.672	0.728	1.054	0.154	62.3	39.5	52.8	32.3	1.004	0.430

2.5 Learning

2.5 学习

UniAD is trained in two stages. We first jointly train perception parts, i.e ., the tracking and mapping modules, for a few epochs (6 in our experiments), and then train the model end-to-end for 20 epochs with all perception, prediction and planning modules. The two-stage training is found more stable empirically. We refer the audience to the Supplementary for details of each loss.

UniAD 分两阶段训练。我们先联合训练感知部分（即跟踪与建图模块）若干 epoch（实验中为 6），再端到端训练全部感知、预测与规划模块 20 个 epoch。经验上两阶段训练更稳定。各损失细节见补充材料。

Shared matching. Since UniAD involves instance-wise modeling, pairing predictions to the ground truth set is required in perception and prediction tasks. Similar to DETR [8, 56], the bipartite matching algorithm is adopted in the tracking and online mapping stage. As for tracking, candidates from detection queries are paired with newborn ground truth objects, and predictions from track queries inherit the assignment from previous frames. The matching results in the tracking module are reused in motion and occupancy nodes to consistently model agents from historical tracks to future motions in the end-to-end framework.

共享匹配。 由于 UniAD 涉及实例级建模，感知与预测任务需要把预测与真实集合配对。类似 DETR [8, 56]，跟踪与在线建图阶段采用二分图匹配。对跟踪，detection query 候选与新生真实目标配对，track query 的预测继承先前帧的分配。跟踪模块中的匹配结果在运动与占据节点中复用，以便在端到端框架中从历史轨迹到未来运动一致地建模智能体。

3 Experiments

3 实验

We conduct experiments on the challenging nuScenes dataset [6]. In this section, we validate the effectiveness of our design in three aspects: joint results revealing the advantage of task coordination and its effect on planning, modular results of each task compared with previous methods, and ablations on the design space for specific modules. Due to space limit, the full suite of protocols, some ablations and visualizations are provided in the Supplementary.

我们在具有挑战的 nuScenes 数据集 [6] 上实验。本节从三方面验证设计有效性：揭示任务协同优势及其对规划影响的联合结果、各任务相对先前方法的模块结果，以及对特定模块设计空间的消融。因篇幅限制，完整协议、部分消融与可视化见补充材料。

3.1 Joint Results

3.1 联合结果

We conduct extensive ablations as shown in Table 2 to prove the effectiveness and necessity of preceding tasks in the end-to-end pipeline. Each row of this table shows the model performance when incorporating task modules listed in the second Modules column. The first row (ID-0) serves as a vanilla multi-task baseline with separate task heads for comparison. The best result of each metric is marked in bold, and the runner-up result is underlined in each column.

我们做了大量消融（表 2）以证明端到端管线中上游任务的有效性与必要性。表中每行对应第二列「模块」所列任务组合下的模型表现。第一行（ID-0）为带独立任务头的朴素多任务基线。各列最优指标加粗，次优加下划线。

m

-

3.2 Modular Results

3.2 模块结果

Following the sequential order of perception-prediction-planning, we report the performance of each task module in comparison to prior state-of-the-arts on the nuScenes validation set. Note that UniAD jointly performs all these tasks with a single trained network. The main metric for each task is marked with gray background in tables.

按感知—预测—规划的顺序，我们在 nuScenes 验证集上报告各任务模块相对先前最优方法的表现。注意 UniAD 用单一训练网络联合完成所有这些任务。各任务主指标在表中以灰色背景标出。

Perception results. As for multi-object tracking in Table 3, UniAD yields a significant improvement of +6.5 and +14.2 AMOTA(%) compared to MUTR3D [104] and ViP3D [30] respectively. Moreover, UniAD achieves the lowest ID switch score, showing its temporal consistency for each tracklet. For online mapping in Table 4, UniAD performs well on segmenting lanes (+7.4 IoU(%) compared to BEVFormer), which is crucial for downstream agent-road interaction in the motion module. As our tracking module follows an end-to-end paradigm, it is still inferior to tracking-by-detection methods with complex associations such as Immortal Tracker [93], and our mapping results trail previous perception-oriented methods on specific classes. We argue that UniAD is to benefit final planning with perceived information rather than optimizing perception with full model capacity.

感知结果。 表 3 多目标跟踪上，UniAD 相对 MUTR3D [104] 与 ViP3D [30] 分别取得 +6.5 与 +14.2 AMOTA(%) 的显著提升，且 ID switch 最低，显示每条轨迹的时序一致性。表 4 在线建图上，UniAD 在车道分割上表现良好（相对 BEVFormer +7.4 IoU(%)），这对运动模块中下游智能体—道路交互至关重要。我们的跟踪与建图模块面向规划所需信息，而非以全部模型容量去优化感知本身。

Prediction results. Motion forecasting results are shown in Table 5, where UniAD remarkably outperforms previous vision-based end-to-end methods. It reduces prediction errors by 38.3% and 65.4% on minADE compared to PnPNet-vision [57] and ViP3D [30] respectively. In terms of occupancy prediction reported in Table 6, UniAD gets notable advances in nearby areas, yielding +4.0 and +2.0 on IoU-near(%) compared to FIERY [35] and BEVerse [105] with heavy augmentations, respectively.

预测结果。 运动预报见表 5：UniAD 显著超过先前基于视觉的端到端方法，相对 PnPNet-vision [57] 与 ViP3D [30]，minADE 误差分别降低 38.3% 与 65.4% 。表 6 占据预测上，UniAD 在近距区域取得明显进展，相对带强增强的 FIERY [35] 与 BEVerse [105]，IoU-near(%) 分别 +4.0 与 +2.0 。

Planning results. Benefiting from rich spatial-temporal information in both the ego-vehicle query and occupancy, UniAD reduces planning L2 error and collision rate by 51.2% and 56.3% compared to ST-P3 [38], in terms of the average value for the planning horizon. Moreover, it notably outperforms several LiDAR-based counterparts, which is often deemed challenging for perception tasks.

规划结果。 得益于自车 query 与占据中的丰富时空信息，按规划时域平均值计，UniAD 相对 ST-P3 [38] 将规划 L2 误差与碰撞率分别降低 51.2% 与 56.3% 。此外，它明显超过若干基于 LiDAR 的对照，而这在感知任务上通常被认为颇具挑战。

\uparrow

Method	AMOTA $\uparrow$	AMOTP $\downarrow$	Recall $\uparrow$	IDS $\downarrow$
Immortal Tracker ${}^{\text{\textdagger}}$ [93]	0.378	1.119	0.478	936
ViP3D [30]	0.217	1.625	0.363	-
QD3DT [36]	0.242	1.518	0.399	-
MUTR3D [104]	0.294	1.498	0.427	3822
UniAD	0.359	1.320	0.467	906

\uparrow

Method	AMOTA $\uparrow$	AMOTP $\downarrow$	Recall $\uparrow$	IDS $\downarrow$
Immortal Tracker ${}^{\text{\textdagger}}$ [93]	0.378	1.119	0.478	936
ViP3D [30]	0.217	1.625	0.363	-
QD3DT [36]	0.242	1.518	0.399	-
MUTR3D [104]	0.294	1.498	0.427	3822
UniAD	0.359	1.320	0.467	906

\uparrow

Method	Lanes $\uparrow$	Drivable $\uparrow$	Divider $\uparrow$	Crossing $\uparrow$
VPN [72]	18.0	76.0	-	-
LSS [76]	18.3	73.9	-	-
BEVFormer [55]	23.9	77.5	-	-
BEVerse ${}^{\text{\textdagger}}$ [105]	-	-	30.6	17.2
UniAD	31.3	69.1	25.7	13.8

\uparrow

Method	Lanes $\uparrow$	Drivable $\uparrow$	Divider $\uparrow$	Crossing $\uparrow$
VPN [72]	18.0	76.0	-	-
LSS [76]	18.3	73.9	-	-
BEVFormer [55]	23.9	77.5	-	-
BEVerse ${}^{\text{\textdagger}}$ [105]	-	-	30.6	17.2
UniAD	31.3	69.1	25.7	13.8

m

Method	minADE( $m$ ) $\downarrow$	minFDE( $m$ ) $\downarrow$	MR $\downarrow$	EPA $\uparrow$
PnPNet ${}^{\text{\textdagger}}$ [57]	1.15	1.95	0.226	0.222
ViP3D [30]	2.05	2.84	0.246	0.226
Constant Pos.	5.80	10.27	0.347	-
Constant Vel.	2.13	4.01	0.318	-
UniAD	0.71	1.02	0.151	0.456

m

Method	minADE( $m$ ) $\downarrow$	minFDE( $m$ ) $\downarrow$	MR $\downarrow$	EPA $\uparrow$
PnPNet ${}^{\text{\textdagger}}$ [57]	1.15	1.95	0.226	0.222
ViP3D [30]	2.05	2.84	0.246	0.226
Constant Pos.	5.80	10.27	0.347	-
Constant Vel.	2.13	4.01	0.318	-
UniAD	0.71	1.02	0.151	0.456

\uparrow

Method	IoU-n. $\uparrow$	IoU-f. $\uparrow$	VPQ-n. $\uparrow$	VPQ-f. $\uparrow$
FIERY [35]	59.4	36.7	50.2	29.9
StretchBEV [1]	55.5	37.1	46.0	29.0
ST-P3 [38]	-	38.9	-	32.1
BEVerse ${}^{\text{\textdagger}}$ [105]	61.4	40.9	54.3	36.1
UniAD	63.4	40.2	54.7	33.5

\uparrow

Method	IoU-n. $\uparrow$	IoU-f. $\uparrow$	VPQ-n. $\uparrow$	VPQ-f. $\uparrow$
FIERY [35]	59.4	36.7	50.2	29.9
StretchBEV [1]	55.5	37.1	46.0	29.0
ST-P3 [38]	-	38.9	-	32.1
BEVerse ${}^{\text{\textdagger}}$ [105]	61.4	40.9	54.3	36.1
UniAD	63.4	40.2	54.7	33.5

m

Method	L2( $m$ ) $\downarrow$	Col. Rate(%) $\downarrow$
1 $s$	2 $s$	3 $s$	Avg.	1 $s$	2 $s$	3 $s$	Avg.
NMP ${}^{\text{\textdagger}}$ [101]	-	-	2.31	-	-	-	1.92	-
SA-NMP ${}^{\text{\textdagger}}$ [101]	-	-	2.05	-	-	-	1.59	-
FF ${}^{\text{\textdagger}}$ [37]	0.55	1.20	2.54	1.43	0.06	0.17	1.07	0.43
EO ${}^{\text{\textdagger}}$ [47]	0.67	1.36	2.78	1.60	0.04	0.09	0.88	0.33
ST-P3 [38]	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71
UniAD	0.48	0.96	1.65	1.03	0.05	0.17	0.71	0.31

m

Method	L2( $m$ ) $\downarrow$	Col. Rate(%) $\downarrow$
1 $s$	2 $s$	3 $s$	Avg.	1 $s$	2 $s$	3 $s$	Avg.
NMP ${}^{\text{\textdagger}}$ [101]	-	-	2.31	-	-	-	1.92	-
SA-NMP ${}^{\text{\textdagger}}$ [101]	-	-	2.05	-	-	-	1.59	-
FF ${}^{\text{\textdagger}}$ [37]	0.55	1.20	2.54	1.43	0.06	0.17	1.07	0.43
EO ${}^{\text{\textdagger}}$ [47]	0.67	1.36	2.78	1.60	0.04	0.09	0.88	0.33
ST-P3 [38]	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71
UniAD	0.48	0.96	1.65	1.03	0.05	0.17	0.71	0.31

3.3 Qualitative Results

3.3 定性结果

Fig. 3 visualizes the results of all tasks for one complex scene. The ego vehicle drives with notice to the potential movement of a front vehicle and lane. In the Supplementary, we show more visualizations of challenging scenarios and one promising case for the planning-oriented design, that inaccurate results occur in prior modules while the later tasks could still recover, e.g ., the planned trajectory remains reasonable though objects have a large heading angle deviation or fail to be detected in tracking results. Besides, we analyze that failure cases of UniAD are mainly under some long-tail scenarios such as large trucks and trailers, shown in the Supplementary as well.

图 3 可视化一个复杂场景中全部任务的结果。自车行驶时注意到前方车辆与车道的潜在运动。补充材料中展示更多挑战场景可视化，以及规划导向设计的一个有希望案例：上游模块结果不准确时，后续任务仍可恢复，例如目标航向偏差很大或漏检时，规划轨迹仍保持合理。我们分析 UniAD 的失败案例主要出现在大卡车、拖车等长尾场景，亦见补充材料。

Q

Scene-l.
Anch.

Goal
Inter.

minFDE
-mAP∗

Q

Scene-l.
Anch.

Goal
Inter.

minFDE
-mAP∗

\uparrow

Cross.
Attn.

Attn.
Mask

Mask
Feat.

\uparrow

Cross.
Attn.

Attn.
Mask

Mask
Feat.

\downarrow

ID	BEV	Col.	Occ.	L2 $\downarrow$	Col. Rate $\downarrow$
Att.	Loss	Optim.	1 $s$	2 $s$	3 $s$	1 $s$	2 $s$	3 $s$
1				0.44	0.99	1.71	0.56	0.88	1.64
2	✓			0.44	1.04	1.81	0.35	0.71	1.58
3	✓	✓		0.44	1.02	1.76	0.30	0.51	1.39
4	✓	✓	✓	0.54	1.09	1.81	0.13	0.42	1.05

\downarrow

ID	BEV	Col.	Occ.	L2 $\downarrow$	Col. Rate $\downarrow$
Att.	Loss	Optim.	1 $s$	2 $s$	3 $s$	1 $s$	2 $s$	3 $s$
1				0.44	0.99	1.71	0.56	0.88	1.64
2	✓			0.44	1.04	1.81	0.35	0.71	1.58
3	✓	✓		0.44	1.02	1.76	0.30	0.51	1.39
4	✓	✓	✓	0.54	1.09	1.81	0.13	0.42	1.05

Figure 3: Visualization results. We show results for all tasks in surround-view images and BEV. Predictions from motion and occupancy modules are consistent, and the ego vehicle is yielding to the front black car in this case. Each agent is illustrated with a unique color. Only top-1 and top-3 trajectories from motion forecasting are selected for visualization on image-view and BEV respectively.

图 3： 可视化结果。我们在环视图像与 BEV 中展示全部任务结果。运动与占据模块的预测一致，本例中自车对前方黑色车辆让行。每个智能体用独特颜色表示。图像视图与 BEV 分别只选取运动预报的 top-1 与 top-3 轨迹可视化 。

3.4 Ablation Study

3.4 消融研究

Effect of designs in MotionFormer. Table 8 shows that all of our proposed components described in Sec. 2.2 contribute to final performance regarding minADE, minFDE, Miss Rate and minFDE-mAP metrics. Notably, the rotated scene-level anchor shows a significant performance boost (-15.8% minADE, -11.2% minFDE, +1.9 minFDE-mAP(%)), indicating that it is essential to do motion forecasting in the scene-centric manner. The agent-goal point interaction enhances the motion query with the planning-oriented visual feature, and surrounding agents can further benefit from considering the ego vehicle’s intention. Moreover, the non-linear optimization strategy improves the performance (-5.0% minADE, -8.4% minFDE, -1.0 MR(%), +0.7 minFDE-mAP(%)) by taking perceptual uncertainty into account in the end-to-end scenario.

-

M^{t}

M^{t}

Effect of designs in Planner. We provide ablations on the proposed designs in planner in Table 10, i.e ., attending BEV features, training with the collision loss and the optimization strategy with occupancy. Similar to previous research [37, 38], a lower collision rate is preferred for safety over naive trajectory mimicking (L2 metric), and is reduced with all parts applied in UniAD.

规划器设计的影响。 表 10 给出规划器中所提设计的消融，即关注 BEV 特征、用碰撞损失训练以及基于占据的优化策略。与先前研究 [37, 38] 类似，出于安全考虑，更低碰撞率优于朴素轨迹模仿（L2 指标）；在 UniAD 中应用全部组件后碰撞率下降。

4 Conclusion and Future Work

4 结论与未来工作

We discuss the system-level design for the autonomous driving algorithm framework. A planning-oriented pipeline is proposed toward the ultimate pursuit for planning, namely UniAD. We provide detailed analyses on the necessity of each module within perception and prediction. To unify tasks, a query-based design is proposed to connect all nodes in UniAD, benefiting from richer representations for agent interaction in the environment. Extensive experiments verify the proposed method in all aspects.

我们讨论自动驾驶算法框架的系统级设计，提出以规划为终极追求的规划导向管线 UniAD，并详细分析感知与预测中各模块的必要性。为统一任务，采用基于 query 的接口连接各节点。受益于环境中智能体交互的更丰富表示，UniAD 表现优异。大量实验从各方面验证了所提方法。

Limitations and future work. Coordinating such a comprehensive system with multiple tasks is non-trivial and needs extensive computational power, especially trained with temporal history. How to devise and curate the system for a lightweight deployment deserves future exploration. Moreover, whether or not to incorporate more tasks such as depth estimation, behavior prediction, and how to embed them into the system, are worthy future directions as well.

局限与未来工作 。协调含多任务的综合系统并不轻松，尤其在带时序历史训练时需要大量算力。如何设计与精简系统以便轻量部署值得未来探索。此外，是否纳入深度估计、行为预测等更多任务，以及如何嵌入系统，也是有价值的方向。

Acknowledgements. This work is partially supported by National Key R&D Program of China (2022ZD0160100), and in part by Shanghai Committee of Science and Technology (21DZ1100100), and NSFC (62206172).

致谢 。本工作部分得到国家重点研发计划（2022ZD0160100）、上海市科委（21DZ1100100）与国家自然科学基金（62206172）资助。

References

参考文献

Appendix

附录

[1] Adil Kaan Akan and Fatma Güney. StretchBEV: Stretching future instance prediction spatially and temporally. In ECCV , 2022. [2] Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079 , 2018. [3] Hans Georg Bock and Karl-Josef Plitt. A multiple shooting algorithm for direct solution of optimal control problems. IFAC Proceedings Volumes , 1984. [4] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Zieba Karol. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 , 2016. [5] Thibault Buhet, Émilie Wirbel, and Xavier Perrotton. PLOP: Probabilistic polynomial objects trajectory planning for autonomous driving. In CoRL , 2020. [6] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , 2020. [7] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810 , 2021. [8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020. [9] Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urtasun. Spagnn: Spatially-aware graph neural networks for relational behavior forecasting from sensor data. In ICRA , 2020. [10] Sergio Casas, Wenjie Luo, and Raquel Urtasun. Intentnet: Learning to predict intention from raw sensor data. In CoRL , 2018.

[11] Sergio Casas, Abbas Sadat, and Raquel Urtasun. Mp3: A unified model to map, perceive, predict and plan. In CVPR , 2021. [12] Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In CoRL , 2020. [13] Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, and Fabien Moutarde. GRI: General reinforced imitation and its application to vision-based autonomous driving. arXiv preprint 2111.08575 , 2021. [14] Dian Chen, Vladlen Koltun, and Philipp Krähenbühl. Learning to drive from a world on rails. In ICCV , 2021. [15] Dian Chen and Philipp Krähenbühl. Learning from all vehicles. In CVPR , 2022. [16] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In CoRL , 2020. [17] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR , 2022. [18] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS , 2021. [19] Kashyap Chitta, Aditya Prakash, and Andreas Geiger. NEAT: Neural attention fields for end-to-end autonomous driving. In ICCV , 2021. [20] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE TPAMI , 2022.

[31] Chunrui Han, Jianjian Sun, Zheng Ge, Jinrong Yang, Runpei Dong, Hongyu Zhou, Weixin Mao, Yuang Peng, and Xiangyu Zhang. Exploring recurrent long-term temporal fusion for multi-view 3d perception. arXiv preprint arXiv:2303.05970 , 2023. [32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016. [33] Noureldin Hendy, Cooper Sloan, Feng Tian, Pengfei Duan, Nick Charchut, Yuesong Xie, Chuang Wang, and James Philbin. Fishing net: Future inference of semantic heatmaps in grids. arXiv preprint arXiv:2006.09917 , 2020. [34] Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. In NeurIPS , 2022. [35] Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. FIERY: Future instance prediction in bird’s-eye view from surround monocular cameras. In ICCV , 2021. [36] Hou-Ning Hu, Yung-Hsu Yang, Tobias Fischer, Trevor Darrell, Fisher Yu, and Min Sun. Monocular quasi-dense 3d object tracking. IEEE TPAMI , 2022. [37] Peiyun Hu, Aaron Huang, John Dolan, David Held, and Deva Ramanan. Safe local motion planning with self-supervised freespace forecasting. In CVPR , 2021. [38] Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. ST-P3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In ECCV , 2022. [39] Zhiyu Huang, Haochen Liu, Jingda Wu, and Chen Lv. Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving. arXiv preprint arXiv:2207.10422 , 2022. [40] Boris Ivanovic, Amine Elhafsi, Guy Rosman, Adrien Gaidon, and Marco Pavone. MATS: An interpretable trajectory forecasting representation for planning and control. In CoRL , 2021.

[41] Xiaosong Jia, Li Chen, Penghao Wu, Jia Zeng, Junchi Yan, Hongyang Li, and Yu Qiao. Towards capturing the temporal dynamics for trajectory prediction: a coarse-to-fine approach. In CoRL , 2022. [42] Xiaosong Jia, Liting Sun, Masayoshi Tomizuka, and Wei Zhan. Ide-net: Interactive driving event and pattern extraction from human data. IEEE RA-L , 2021. [43] Xiaosong Jia, Liting Sun, Hang Zhao, Masayoshi Tomizuka, and Wei Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. In CoRL , 2021. [44] Xiaosong Jia, Penghao Wu, Li Chen, Hongyang Li, Yu Liu, and Junchi Yan. HDGT: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. arXiv preprint arXiv:2205.09753 , 2022. [45] Alexey Kamenev, Lirui Wang, Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nistér, and Nikolai Smolyanskiy. Predictionnet: Real-time joint probabilistic traffic prediction for planning, control, and simulation. In ICRA , 2022. [46] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In ICRA , 2019. [47] Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Differentiable raycasting for self-supervised occupancy forecasting. In ECCV , 2022. [48] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Video panoptic segmentation. In CVPR , 2020. [49] Jinkyu Kim, Reza Mahjourian, Scott Ettinger, Mayank Bansal, Brandyn White, Ben Sapp, and Dragomir Anguelov. Stopnet: Scalable trajectory and occupancy prediction for urban autonomous driving. arXiv preprint arXiv:2206.00991 , 2022. [50] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR , 2019.

[51] Youngwan Lee, Joong-won Hwang, Sangrok Lee, Yuseok Bae, and Jongyoul Park. An energy and gpu-computation efficient backbone network for real-time object detection. In CVPR Workshop , 2019. [52] Feng Li, Hao Zhang, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeung Shum. Mask dino: Towards a unified transformer-based framework for object detection and segmentation. In CVPR , 2023. [53] Lingyun Luke Li, Bin Yang, Ming Liang, Wenyuan Zeng, Mengye Ren, Sean Segal, and Raquel Urtasun. End-to-end contextual perception and prediction with interaction transformer. In IROS , 2020. [54] Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representation with transformer for 3d object detection. In NeurIPS , 2022. [55] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. BEVFormer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV , 2022. [56] Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In CVPR , 2022. [57] Ming Liang, Bin Yang, Wenyuan Zeng, Yun Chen, Rui Hu, Sergio Casas, and Raquel Urtasun. Pnpnet: End-to-end perception and prediction with tracking in the loop. In CVPR , 2020. [58] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. BEVFusion: A simple and robust lidar-camera fusion framework. In NeurIPS , 2022. [59] Xiaodan Liang, Tairui Wang, Luona Yang, and Eric Xing. Cirl: Controllable imitative reinforcement learning for vision-based self-driving. In ECCV , 2018. [60] Xiwen Liang, Yangxin Wu, Jianhua Han, Hang Xu, Chunjing Xu, and Xiaodan Liang. Effective adaptation in multi-task co-training for unified autonomous driving. In NeurIPS , 2022.

[61] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV , 2017. [62] Jerry Liu, Wenyuan Zeng, Raquel Urtasun, and Ersin Yumer. Deep structured reactive planning. In ICRA , 2021. [63] Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers. In CVPR , 2021. [64] Zhijian Liu, Haotian Tang, Alexander Amini, Xingyu Yang, Huizi Mao, Daniela Rus, and Song Han. BEVFusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation. In ICRA , 2023. [65] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2018. [66] Wenjie Luo, Bin Yang, and Raquel Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In CVPR , 2018. [67] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV , 2016. [68] Mobileye. Mobileye under the hood. https://www.mobileye.com/ces-2022/, 2022. [69] Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. arXiv preprint arXiv:2207.05844 , 2022. [70] Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David Weiss, Ben Sapp, Zhifeng Chen, and Jonathon Shlens. Scene transformer: A unified multi-task model for behavior prediction and planning. In ICLR , 2022.

[71] Nvidia. NVIDIA DRIVE End-to-End Solutions for Autonomous Vehicles. https://developer.nvidia.com/drive, 2022. [72] Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolei Zhou. Cross-view semantic segmentation for sensing surroundings. IEEE RA-L , 2020. [73] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In ICCV , 2021. [74] Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, and Wei Zhan. Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection. arXiv preprint arXiv:2210.02443 , 2022. [75] Neehar Peri, Jonathon Luiten, Mengtian Li, Aljoša Ošep, Laura Leal-Taixé, and Deva Ramanan. Forecasting from lidar via future object detection. In CVPR , 2022. [76] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV , 2020. [77] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In NeurIPS , 1988. [78] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In CVPR , 2021. [79] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022. [80] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR , 2019.

[81] Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. PRECOG: Prediction conditioned on goals in visual multi-agent settings. In ICCV , 2019. [82] Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In ECCV , 2020. [83] Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. In CoRL , 2022. [84] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In NeurIPS , 2022. [85] Yining Shi, Jingyan Shen, Yifan Sun, Yunlong Wang, Jiaxin Li, Shiqi Sun, Kun Jiang, and Diange Yang. Srcn3d: Sparse r-cnn 3d surround-view camera object detection and tracking for autonomous driving. arXiv preprint arXiv:2206.14451 , 2022. [86] Haoran Song, Wenchao Ding, Yuxuan Chen, Shaojie Shen, Michael Yu Wang, and Qifeng Chen. Pip: Planning-informed trajectory prediction for autonomous driving. In ECCV , 2020. [87] Tesla. Tesla AI Day. https://www.youtube.com/watch?v=ODSJsviD_SU, 2022. [88] Sebastian Thrun and Arno Bücken. Integrating grid-based and topological maps for mobile robot navigation. In AAAI , 1996. [89] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In CVPR , 2020. [90] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, and Benjamin Sapp. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. arXiv preprint arXiv:2111.14973 , 2021.

[91] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017. [92] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML , 2022. [93] Qitai Wang, Yuntao Chen, Ziqi Pang, Naiyan Wang, and Zhaoxiang Zhang. Immortal tracker: Tracklet never dies. arXiv preprint arXiv:2111.13672 , 2021. [94] Bob Wei, Mengye Ren, Wenyuan Zeng, Ming Liang, Bin Yang, and Raquel Urtasun. Perceive, attend, and drive: Learning spatial attention for safe self-driving. In ICRA , 2021. [95] Penghao Wu, Li Chen, Hongyang Li, Xiaosong Jia, Junchi Yan, and Yu Qiao. Policy pre-training for autonomous driving via self-supervised geometric modeling. In ICLR , 2023. [96] Pengxiang Wu, Siheng Chen, and Dimitris N Metaxas. Motionnet: Joint perception and motion prediction for autonomous driving based on bird’s eye view maps. In CVPR , 2020. [97] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline. In NeurIPS , 2022. [98] Jinrong Yang, En Yu, Zeming Li, Xiaoping Li, and Wenbing Tao. Quality matters: Embracing quality clues for robust 3d multi-object tracking. arXiv preprint arXiv:2208.10976 , 2022. [99] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting. In ICCV , 2021. [100] Fangao Zeng, Bin Dong, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. In ECCV , 2021.

[101] Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end interpretable neural motion planner. In CVPR , 2019. [102] Wenyuan Zeng, Shenlong Wang, Renjie Liao, Yun Chen, Bin Yang, and Raquel Urtasun. Dsdnet: Deep structured self-driving network. In ECCV , 2020. [103] Jimuyang Zhang and Eshed Ohn-Bar. Learning by watching. In CVPR , 2021. [104] Tianyuan Zhang, Xuanyao Chen, Yue Wang, Yilun Wang, and Hang Zhao. MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries. In CVPR Workshop , 2022. [105] Yunpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang, Guan Huang, Jie Zhou, and Jiwen Lu. BEVerse: Unified perception and prediction in birds-eye-view for vision-centric autonomous driving. arXiv preprint arXiv:2205.09743 , 2022. [106] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In ICCV , 2021. [107] Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, Congcong Li, and Dragomir Anguelov. TNT: Target-driven trajectory prediction. In CoRL , 2020. [108] Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-perceiver-moe: Learning sparse generalist models with conditional moes. In NeurIPS , 2022. [109] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR , 2020.

Appendix A Task Definition

附录 A 任务定义

Q_{A}

检测与跟踪。 检测与跟踪是自动驾驶两项关键感知任务，我们侧重在三维空间表示它们以便下游使用。三维检测负责在每一时刻定位周围物体（坐标、长宽高等）；跟踪旨在跨时刻找到不同物体对应关系并做时序关联（即为每个智能体分配一致的 track ID）。文中有时用多目标跟踪统称检测与跟踪过程。最终输出为每帧关联的三维框序列及其对应特征。

Q_{A}

Q_{A}

Q_{A}

Q_{A}

T_{o}

T_{o}

B

B

Appendix B The Necessity of Each Task

附录 B 各任务的必要性

In terms of perception, tracking in the loop as does in PnPNet [57] and ViP3D [30] is proven to complement spatial-temporal features and provide history tracks for occluded agents, refraining from catastrophic decisions for downstream planning. With the aid of HD maps [101, 82, 57, 30] and motion forecasting, planning becomes more accurate toward higher-level intelligence. However, such information is expensive to construct and prone to be outdated, raising the demand for online mapping without HD maps. As for prediction, motion forecasting [10, 42, 41, 29, 107] generates long-term future behaviors and preserves agent identity in form of sparse waypoint outputs. Nonetheless, there exists the challenge to integrate non-differentiable box representation into subsequent planning module [30, 57]. Some recent literature investigates another type of prediction task named occupancy [88] prediction to assist end-to-end planning, in form of cost maps. However, the lack of agent identity and dynamics in occupancy makes it impractical to model social interactions for safe planning. The large computational consumption of modeling multi-step dense features also leads to a much shorter temporal horizon compared to motion forecasting. Therefore, to benefit from the two complementary types of prediction tasks for safe planning, we incorporate both agent-centric motion and whole-scene occupancy in UniAD.

在感知方面，如 PnPNet [57] 与 ViP3D [30] 那样把 跟踪 纳入闭环，已被证明可互补时空特征，并为被遮挡智能体提供历史轨迹，避免下游规划灾难性决策。借助 HD 地图 [101, 82, 57, 30] 与运动预报，规划可更准确、迈向更高智能。但这类信息构建昂贵且易过时，因而需要无 HD 地图的在线 建图 。在预测方面， 运动 预报 [10, 42, 41, 29, 107] 生成长期未来行为，并以稀疏航点形式保留智能体身份。然而，它在遮挡与形状不规则物体上仍有局限。 占据 预测以稠密场景级表示互补，便于碰撞检查。规划作为终极目标，需要这些上游任务的协同支持，而非孤立优化。

Appendix C Related Work

附录 C 相关工作

C.1 Joint perception and prediction

C.1 联合感知与预测

Joint learning of perception and prediction is proposed to avoid the cascading error in traditional modular-independence pipelines. Similar to the motion forecasting task alone, it usually has two types of output representations: agent-level bounding boxes and scene-level occupancy grid maps. Pioneering work FaF [66] predicts boxes in the future and aggregates past information to produce tracklets. IntentNet [10] extends it to reason about intentions and [25, 28] further predict future states in a refinement fashion. Some exploit detection first and utilize agent features in the second prediction stage [9, 53, 75]. Noticing that history information is ignored, PnPNet [57] enriches it by estimating tracking association scores to avert the non-differentiable optimization process as adopted by the tracking-by-detection paradigm [54, 85, 64, 98]. Yet, all these methods rely on non-maximum suppression (NMS) in detection which still leads to information loss. ViP3D [30] which is closely related to our work, employs agent queries in [104] to forecast, taking HD map as another input. We follow the philosophy of [104, 30] in agent track queries, but also develop non-linear optimization on target trajectories to alleviate the potential inaccurate perception problem. Moreover, we introduce an ego-vehicle query for better capturing the ego behaviors in the dynamic environment, and incorporate online mapping to prevent the localization risk or high construction cost with HD map.

联合学习感知与预测旨在避免传统模块独立管线中的级联误差。与单独的运动预报类似，其输出表示通常有两类：智能体级边界框与场景级占据栅格图。开创性工作 FaF [66] 预测未来框并聚合过去信息以产生轨迹片段。IntentNet [10] 扩展到意图推理；[25, 28] 以 refinement 方式进一步预测未来状态。一些工作先做检测，再在第二阶段预测中利用智能体特征 [9, 53, 75]。注意到历史信息被忽略后，PnPNet [57] 通过估计轨迹将其丰富。ViP3D [30] 等继续推进视觉端到端设定下的联合感知—预测。

The alternative representation, namely the occupancy grid map, discretizes the BEV map into grid cells which holds a belief indicating if it is occupied. Wu et al . [96] estimate a dense motion field, while it could not capture multimodal behaviors. Fishing Net [33] also predicts deterministic future BEV semantic segmentation with multiple sensors. To address this, P3 [82] proposes non-parametric distribution of future semantic occupancy and FIERY [35] devises the first paradigm for multi-view cameras. A few methods improve the performance of FIERY with more sophisticated uncertainty modeling [38, 1, 105]. Notably, this representation could easily extend to motion planning for collision avoidance [82, 11, 38], while it loses the agent identity characteristic and takes a heavy burden to computation which may constrain the prediction horizon. In contrast, we leverage agent-level information for occupancy prediction and ensure accurate and safe planning by unifying these two modes.

另一表示——占据栅格图——把 BEV 图离散为网格，每格保存是否占据的信念。Wu et al . [96] 估计稠密运动场，但难以捕获多模态行为。Fishing Net [33] 也用多传感器预测确定性未来 BEV 语义分割。为此，P3 [82] 提出未来语义占据的非参数分布，FIERY [35] 设计了首个面向多视角相机的范式。若干方法以更精细的不确定性建模改进 FIERY [38, 1, 105]。值得注意的是，该表示易于扩展到运动规划以做碰撞避免。

C.2 Joint prediction and planning

C.2 联合预测与规划

PRECOG [81] proposes a recurrent model that conditions forecasting on the goal position of the ego vehicle, while PiP [86] generates agents’ motion considering complete presumed planning trajectories. However, producing a rough future trajectory is still challenging in the real world, toward which [62] presents a deep structured model to derive both prediction and planning from the same set of learnable costs. [40, 39] couple the prediction model with classic optimization methods. Meanwhile, some motion forecasting methods implicitly include the planning task by producing their future trajectories simultaneously [12, 70, 45]. Similarly, we encode possible behaviors of the ego vehicle in the scene-centric motion forecasting module, but the interpretable occupancy map is utilized to further optimize the plan to stay safe.

PRECOG [81] 提出以自车目标位置为条件的循环预报模型；PiP [86] 则在完整假定规划轨迹条件下生成智能体运动。然而在真实世界产生粗略未来轨迹仍具挑战，为此 [62] 提出深度结构化模型，从同一组可学习代价同时导出预测与规划。[40, 39] 将预测模型与经典优化耦合。同时，一些运动预报方法通过同时产出自车未来轨迹隐式包含规划任务 [12, 70, 45]。类似地，我们在运动模块中编码自车可能行为。

C.3 End-to-end motion planning

C.3 端到端运动规划

自 Pomerleau [77] 用单一神经网络直接预测控制信号以来，端到端运动规划一直是活跃研究方向。后续研究在闭环仿真中以更深网络 [4]、多模态输入 [2, 21, 78]、多任务学习 [97, 20]、强化学习 [59, 46, 89, 13, 14] 以及从特权知识蒸馏 [16, 106, 103] 取得重要进展。然而，对此类从传感器数据直接生成控制输出的方法，从合成环境迁移到真实应用时，稳健性与安全保障仍成问题 [22, 38]。因此研究者转向显式设计中间任务，以提升可解释性与安全性；UniAD 沿这一路线，用规划导向的全栈任务协同组织系统。

Appendix D Notations

附录 D 符号

We provide a lookup table of notations and their shapes mentioned in this paper in Table 11 for reference.

我们在表 11 中提供本文出现的符号及其形状的查阅表，供参考。

Q_{o}

Notation	Shape & Params.	Description
$Q_{o}$	900	number of initial object queries
$D$	256	embed dimensions
$B$	$200\times 200\times 256$	BEV feature encoded by a multi-view framework
$N$	6	number of transformer decoder layers for TrackFormer
$N$	6	number of transformer decoder layers for MapFormer
$N$	4	number of mask decoder layers for MapFormer
$N$	3	number of transformer decoder layers for MotionFormer
$N$	5	number of transformer decoder layers for OccFormer
$N$	3	number of transformer decoder layers for Planner
$N_{a}$	dynamic	number of agents from TrackFormer
$N_{m}$	300	number of map queries from MapFormer
$Q_{A}$	$N_{a}\times 256$	agent features from TrackFormer
$P_{A}$	$N_{a}\times 256$	agent positions from TrackFormer
$Q_{M}$	$N_{m}\times 256$	map features from MapFormer
$\mathcal{K}$	6	number of forecasting modality in MotionFormer
$\tilde{\mathbf{x}}$	$T\times 2$	ground truth for one agent’s motion forecasting
$\hat{\mathbf{x}}$	$N_{a}\times T\times 2$	prediction of motion forecasting
$T$	12	length of prediction timestamps in MotionFormer
$Q_{\text{pos}}$	$N_{a}\times\mathcal{K}\times 256$	query position in MotionFormer
$Q_{\text{ctx}}$	$N_{a}\times\mathcal{K}\times 256$	query context in MotionFormer
$Q_{a}$	$N_{a}\times\mathcal{K}\times 256$	motion query after agent-agent interaction in MotionFormer
$Q_{m}$	$N_{a}\times\mathcal{K}\times 256$	motion query after agent-map interaction in MotionFormer
$Q_{g}$	$N_{a}\times\mathcal{K}\times 256$	motion query after agent-goal point interaction in MotionFormer
$l$	-	index of decoder layer
$PE$	-	sinusoidal position encoding function
$I^{s}$	$\mathcal{K}\times T\times 2$	scene-level anchor position in MotionFormer
$I^{a}$	$\mathcal{K}\times T\times 2$	agent-level anchor position in MotionFormer
$\Phi$	-	kinematic cost function set
$T_{o}$	5	length of prediction timestamps in OccFormer
$G^{t}$	$N_{a}\times 256$	agent feature input
$F^{t}$	$200\times 200\times 256$	future state output
$Q_{X}$	$N_{a}\times 256$	motion query (max-pooled on modality level) from the last layer of MotionFormer
$F_{\text{ds}}^{t}$	$25\times 25\times 256$	downscaled dense feature
$F_{\text{dec}}^{t}$	$200\times 200\times 256$	decoded dense feature after convolutional decoder
$D_{\text{ds}}^{t}$	$25\times 25\times 256$	agent-aware dense feature after pixel-agent interaction
$\hat{O}_{A}^{t}$	$N_{a}\times 200\times 200$	instance-level probability map
$\hat{O}^{t}$	$200\times 200$	classical instance-agnostic occupancy map merged from $\hat{O}_{A}^{t}$ for planning
$O_{m}^{t}$	$200\times 200$	attention mask for pixel-agent interaction
$M^{t}$	$N_{a}\times 256$	mask feature
$U^{t}$	$N_{a}\times 256$	occupancy feature
$T_{p}$	6	length of planning timestamps in Planner
$\hat{\tau}$	$T_{p}\times 2$	planned trajectory before the optimization with occupancy prediction
$\tau^{*}$	$T_{p}\times 2$	ultimate plan output
$\lambda$	-	hyperparameters in cost functions, target functions, etc.

Q_{o}

符号	形状与参数	说明
$Q_{o}$	900	初始目标 query 数量
$D$	256	嵌入维
$B$	$200\times 200\times 256$	多视角框架编码的 BEV 特征
$N$	6	TrackFormer 的 Transformer 解码器层数
$N$	6	MapFormer 的 Transformer 解码器层数
$N$	4	MapFormer 的掩码解码器层数
$N$	3	MotionFormer 的 Transformer 解码器层数
$N$	5	OccFormer 的 Transformer 解码器层数
$N$	3	Planner 的 Transformer 解码器层数
$N_{a}$	动态	TrackFormer 得到的智能体数
$N_{m}$	300	地图 query 数量 from MapFormer
$Q_{A}$	$N_{a}\times 256$	agent features from TrackFormer
$P_{A}$	$N_{a}\times 256$	agent positions from TrackFormer
$Q_{M}$	$N_{m}\times 256$	map features from MapFormer
$\mathcal{K}$	6	number of forecasting modality in MotionFormer
$\tilde{\mathbf{x}}$	$T\times 2$	ground truth for one agent’s motion forecasting
$\hat{\mathbf{x}}$	$N_{a}\times T\times 2$	prediction of motion forecasting
$T$	12	length of prediction timestamps in MotionFormer
$Q_{\text{pos}}$	$N_{a}\times\mathcal{K}\times 256$	query position in MotionFormer
$Q_{\text{ctx}}$	$N_{a}\times\mathcal{K}\times 256$	query context in MotionFormer
$Q_{a}$	$N_{a}\times\mathcal{K}\times 256$	motion query after agent-agent interaction in MotionFormer
$Q_{m}$	$N_{a}\times\mathcal{K}\times 256$	motion query after agent-map interaction in MotionFormer
$Q_{g}$	$N_{a}\times\mathcal{K}\times 256$	motion query after agent-goal point interaction in MotionFormer
$l$	-	index of decoder layer
$PE$	-	sinusoidal position encoding function
$I^{s}$	$\mathcal{K}\times T\times 2$	scene-level anchor position in MotionFormer
$I^{a}$	$\mathcal{K}\times T\times 2$	agent-level anchor position in MotionFormer
$\Phi$	-	kinematic cost function set
$T_{o}$	5	length of prediction timestamps in OccFormer
$G^{t}$	$N_{a}\times 256$	agent feature input
$F^{t}$	$200\times 200\times 256$	future state output
$Q_{X}$	$N_{a}\times 256$	motion query (max-pooled on modality level) from the last layer of MotionFormer
$F_{\text{ds}}^{t}$	$25\times 25\times 256$	downscaled dense feature
$F_{\text{dec}}^{t}$	$200\times 200\times 256$	decoded dense feature after convolutional decoder
$D_{\text{ds}}^{t}$	$25\times 25\times 256$	agent-aware dense feature after pixel-agent interaction
$\hat{O}_{A}^{t}$	$N_{a}\times 200\times 200$	instance-level probability map
$\hat{O}^{t}$	$200\times 200$	classical instance-agnostic occupancy map merged from $\hat{O}_{A}^{t}$ for planning
$O_{m}^{t}$	$200\times 200$	attention mask for pixel-agent interaction
$M^{t}$	$N_{a}\times 256$	mask feature
$U^{t}$	$N_{a}\times 256$	occupancy feature
$T_{p}$	6	length of planning timestamps in Planner
$\hat{\tau}$	$T_{p}\times 2$	planned trajectory before the optimization with occupancy prediction
$\tau^{*}$	$T_{p}\times 2$	ultimate plan output
$\lambda$	-	hyperparameters in cost functions, target functions, etc.

Appendix E Implementation Details

附录 E 实现细节

E.1 Detection and Tracking

E.1 检测与跟踪

B

B

Training stage: At the beginning (i.e ., first frame) of each training sequence, all queries are considered detection queries and predict all newborn objects, which is actually the same as BEVFormer. Detection queries are matched to the ground truth by the Hungarian algorithm [8]. They will be stored and updated via the query interaction module (QIM) for the next timestamp serving as track queries following MOTR [100]. In the next timestamp, track queries will be directly matched with a part of ground-truth objects according to the corresponding track ID, and detection queries will be matched with the remaining ground-truth objects (newborn objects). To stabilize training, we adopt the 3D IoU metric to filter the matched queries. Only those predictions having the 3D IoU with ground-truth boxes larger than a certain threshold (0.5 in practice) will be stored and updated.

训练阶段 ：每个训练序列开始（即第一帧）时，全部 query 视为 detection query，预测所有新生目标，实际与 BEVFormer 相同。Detection query 用匈牙利算法 [8] 与真实目标匹配，再经 query 交互模块（QIM）存储并更新，供下一时间戳作为 track query，遵循 MOTR [100]。下一时间戳中，track query 按对应 track ID 直接与部分真实目标匹配，detection query 与剩余真实目标（新生目标）匹配。为稳定训练，我们采用适当的匹配与更新策略。

s

s

E.2 Online Mapping

E.2 在线建图

Q_{M}

Q_{M}

N

N

Figure 5: Illustration of agent-goal interaction Module. The BEV visual feature is sampled near each agent’s goal points with deformable cross-attention.

图 5： 智能体—目标交互模块示意。用可变形交叉注意力在每个智能体目标点附近采样 BEV 视觉特征 。

E.3 Motion Forecasting

E.3 运动预报

I^{a}_{T}

I^{a}_{T}

I_{i,T}^{s}=R_{i}I_{T}^{a}+T_{i}, \tag{12}

i

i

Q_{A}

Q_{A}

E.4 Occupancy Prediction

E.4 占据预测

T_{o}

T_{o}

T_{o}

T_{o}

E.5 Planning

E.5 规划

N

N

Q_{A}^{\text{ego}}

Q_{A}^{\text{ego}}

E.6 Training Details

E.6 训练细节

Joint learning. UniAD is trained in two stages which we find more stable. In stage one, we pre-train perception tasks including tracking and online mapping to stabilize perception predictions. Specifically, we load corresponding pre-trained BEVFormer [55] weights to UniAD for fast convergence including image backbone, FPN, BEV encoder and detection decoder except for object query embeddings (due to the additional ego-vehicle query). We stop the gradient back-propagation in the image backbone to reduce memory cost and train UniAD for 6 epochs with tracking and online mapping losses as follows:

联合学习。 UniAD 分两阶段训练，我们发现更稳定。 阶段一 预训跟踪与在线建图等感知任务以稳定感知预测。具体地，加载对应预训练 BEVFormer [55] 权重以加速收敛，包括图像骨干、FPN、BEV 编码器与检测解码器，但不含目标 query 嵌入（因额外引入 自车 query ）。为降低显存，在图像骨干停止梯度回传，用跟踪与在线建图损失训练 UniAD 6 个 epoch：

L_{1}=L_{\text{track}}+L_{\text{map}}. \tag{13}

In stage two, we keep the image backbone frozen as well, and additionally freeze BEV encoder, which is used for view transformation from image view to BEV, to further reduce memory consumption with more downstream modules. UniAD now is trained with all task losses including tracking, mapping, motion forecasting, occupancy prediction, and planning for 20 epochs (for various ablation studies in main paper, it’s trained for 8 epochs for efficiency):

阶段二 同样冻结图像骨干，并额外冻结用于图像视图到 BEV 变换的 BEV 编码器，以在更多下游模块下进一步降低显存。此时 UniAD 用全部任务损失——跟踪、建图、运动预报、占据预测与规划——训练 20 个 epoch（正文各类消融为效率训练 8 个 epoch）：

L_{2}=L_{\text{track}}+L_{\text{map}}+L_{\text{motion}}+L_{\text{occ}}+L_{\text{plan}}. \tag{14}

L_{1}

L_{1}

l_{1}

l_{1}

L_{\text{map}}=\lambda_{\text{focal}}L_{\text{focal}}+\lambda_{l_{1}}L_{l_{1}}+\lambda_{\text{giou}}L_{\text{giou}}+\lambda_{\text{dice}}L_{\text{dice}}

L_{\text{map}}=\lambda_{\text{focal}}L_{\text{focal}}+\lambda_{l_{1}}L_{l_{1}}+\lambda_{\text{giou}}L_{\text{giou}}+\lambda_{\text{dice}}L_{\text{dice}}

L_{\text{cls}}

L_{\text{cls}}

L_{\text{occ}}=\lambda_{\text{bce}}L_{\text{bce}}+\lambda_{\text{dice}}L_{\text{dice}}

L_{\text{occ}}=\lambda_{\text{bce}}L_{\text{bce}}+\lambda_{\text{dice}}L_{\text{dice}}

l_{2}

l_{2}

L_{\text{col}}(\hat{\tau},\delta)=\sum_{i,t}\texttt{IoU}(box(\hat{\tau}_{t},w+\delta,l+\delta),b_{i,t})), \tag{15}

L_{\text{plan}}=\lambda_{\text{imi}}|\hat{\tau},\tilde{\tau}|_{2}+\lambda_{\text{col}}\sum_{(\omega,\delta)}\omega L_{\text{col}}(\hat{\tau},\delta), \tag{16}

\lambda_{\text{imi}}\!=\!1

\lambda_{\text{imi}}\!=\!1

Appendix F Experiments

附录 F 实验

F.1 Protocols

F.1 协议

2\!\times\!10^{-4}

2\!\times\!10^{-4}

F.2 Metrics

F.2 指标

Multi-object tracking. Following the standard evaluation protocols, we use AMOTA (Average Multi-object Tracking Accuracy), AMOTP (Average Multi-object Tracking Precision), Recall, and IDS (Identity Switches) to evaluate the 3D tracking performance of UniAD on nuScenes dataset. AMOTA and AMOTP are computed by integrating MOTA (Multi-object Tracking Accuracy) and MOTP (Multi-object Tracking Precision) values over all recalls:

多目标跟踪。 遵循标准评估协议，我们使用 AMOTA、AMOTP、召回率与 IDS（身份切换）等指标。AMOTA 越高越好，AMOTP 与 IDS 越低越好。

\text{AMOTA}=\frac{1}{n-1}\sum_{r\in\{\frac{1}{n-1},\frac{2}{n-1},...,1\}}\text{MOTA}_{r}, \tag{17}

\text{MOTA}_{r}=\mathop{\max}(0,1-\frac{\text{FP}_{r}+\text{FN}_{r}+\text{IDS}_{r}-(1-r)\text{GT}}{r\text{GT}}), \tag{18}

\text{FP}_{r}

\text{FP}_{r}

\text{AMOTP}=\frac{1}{n-1}\sum_{r\in\{\frac{1}{n-1},\frac{2}{n-1},...,1\}}\frac{\sum_{i,t}d_{i,t}}{\text{TP}_{r}}, \tag{19}

d_{i,t}

d_{i,t}

Online mapping. We have four categories for the online mapping task, i.e ., lanes, boundaries, pedestrian crossings and drivable area. We calculate the intersection-over-union (IoU) metric for each class between the network outputs and ground truth maps.

在线建图。 在线建图任务有四类：车道、边界、人行横道与可行驶区域。我们对网络输出与真实地图计算各类别交并比（ IoU ）。

m

m

\uparrow

Methods	Encoder	Tracking	Mapping	Motion Forecasting	Occupancy Prediction	Planning
AMOTA $\uparrow$	AMOTP $\downarrow$	IDS $\downarrow$	IoU-lane $\uparrow$	IoU-road $\uparrow$	minADE $\downarrow$	minFDE $\downarrow$	MR $\downarrow$	EPA $\uparrow$	IoU-n. $\uparrow$	IoU-f. $\uparrow$	VPQ-n. $\uparrow$	VPQ-f. $\uparrow$	avg.L2 $\downarrow$	avg.Col. $\downarrow$
UniAD-S	R50	0.241	1.488	958	0.315	0.689	0.788	1.126	0.156	0.381	59.4	35.6	49.2	28.9	1.04	0.32
UniAD-B	R101	0.359	1.320	906	0.313	0.691	0.708	1.025	0.151	0.456	63.4	40.2	54.7	33.5	1.03	0.31
UniAD-L	V2-99∗	0.409	1.259	1583	0.323	0.709	0.723	1.067	0.158	0.508	64.1	42.6	55.8	36.9	1.03	0.29

\uparrow

Methods	Encoder	Tracking	Mapping	Motion Forecasting	Occupancy Prediction	Planning
AMOTA $\uparrow$	AMOTP $\downarrow$	IDS $\downarrow$	IoU-lane $\uparrow$	IoU-road $\uparrow$	minADE $\downarrow$	minFDE $\downarrow$	MR $\downarrow$	EPA $\uparrow$	IoU-n. $\uparrow$	IoU-f. $\uparrow$	VPQ-n. $\uparrow$	VPQ-f. $\uparrow$	avg.L2 $\downarrow$	avg.Col. $\downarrow$
UniAD-S	R50	0.241	1.488	958	0.315	0.689	0.788	1.126	0.156	0.381	59.4	35.6	49.2	28.9	1.04	0.32
UniAD-B	R101	0.359	1.320	906	0.313	0.691	0.708	1.025	0.151	0.456	63.4	40.2	54.7	33.5	1.03	0.31
UniAD-L	V2-99∗	0.409	1.259	1583	0.323	0.709	0.723	1.067	0.158	0.508	64.1	42.6	55.8	36.9	1.03	0.29

Occupancy prediction. We evaluate the quality of predicted occupancy in both whole-scene level and instance-level following [35, 105]. Specifically, The IoU measures the whole-scene categorical segmentations which is instance-agnostic, while the Video Panoptic Quality (VPQ) [48] takes into account each instance’s presence and consistency over time. The VPQ metric is calculated as follows:

占据预测。 我们遵循 [35, 105] 在整场景级与实例级评估预测占据质量。具体地， IoU 衡量与实例无关的整场景类别分割，而视频全景质量（ VPQ ） [48] 考虑每个实例随时间的出现与一致性。VPQ 计算如下：

\text{VPQ}=\sum_{t=0}^{H}\frac{\sum_{(p_{t},q_{t})\in\text{TP}_{t}}\texttt{IoU}(p_{t},q_{t})}{|\text{TP}_{t}|+\frac{1}{2}|\text{FP}_{t}|+\frac{1}{2}|\text{FN}_{t}|}, \tag{20}

H

H

ID Det. Track Map Motion Occ. Plan #Params FLOPs FPS 0[105] ✓ ✓ ✓ 102.5M 1921G - 1 ✓ 65.9M 1324G 4.2 2 ✓ ✓ 68.2M 1326G 2.7 3 ✓ ✓ ✓ 95.8M 1520G 2.2 4 ✓ ✓ ✓ ✓ 108.6M 1535G 2.1 5 ✓ ✓ ✓ ✓ ✓ 122.5M 1701G 2.0 6 ✓ ✓ ✓ ✓ ✓ ✓ 125.0M 1709G 1.8 Table 13: Computational complexity and runtime with different modules incorporated. ID.1 is similar to original BEVFormer [55], and ID. 0 (BEVerse-Tiny) [105] is an MTL framework.

ID	Det.	Track	Map	Motion	Occ.	Plan	#Params	FLOPs	FPS
0[105]	✓		✓		✓		102.5M	1921G	-
1	✓						65.9M	1324G	4.2
2	✓	✓					68.2M	1326G	2.7
3	✓	✓	✓				95.8M	1520G	2.2
4	✓	✓	✓	✓			108.6M	1535G	2.1
5	✓	✓	✓	✓	✓		122.5M	1701G	2.0
6	✓	✓	✓	✓	✓	✓	125.0M	1709G	1.8

ID Det. Track Map Motion Occ. Plan #Params FLOPs FPS 0[105] ✓ ✓ ✓ 102.5M 1921G - 1 ✓ 65.9M 1324G 4.2 2 ✓ ✓ 68.2M 1326G 2.7 3 ✓ ✓ ✓ 95.8M 1520G 2.2 4 ✓ ✓ ✓ ✓ 108.6M 1535G 2.1 5 ✓ ✓ ✓ ✓ ✓ 122.5M 1701G 2.0 6 ✓ ✓ ✓ ✓ ✓ ✓ 125.0M 1709G 1.8 表 13： 不同模块组合下的计算复杂度与运行时间。 ID.1 类似原版 BEVFormer [55], and ID. 0 (BEVerse-Tiny) [105] 为多任务学习框架 。

ID	Det.	Track	Map	Motion	Occ.	Plan	#Params	FLOPs	FPS
0[105]	✓		✓		✓		102.5M	1921G	-
1	✓						65.9M	1324G	4.2
2	✓	✓					68.2M	1326G	2.7
3	✓	✓	✓				95.8M	1520G	2.2
4	✓	✓	✓	✓			108.6M	1535G	2.1
5	✓	✓	✓	✓	✓		122.5M	1701G	2.0
6	✓	✓	✓	✓	✓	✓	125.0M	1709G	1.8

Planning. We adopt the same metrics as in ST-P3 [38], i.e ., L2 error and collision rate at various timestamps.

规划。 我们采用与 ST-P3 [38] 相同的指标，即不同时刻的 L2 误差 与 碰撞率 。

F.3 Model complexity and Computational cost

F.3 模型复杂度与计算代价

We measure the complexity of UniAD and runtime on an Nvidia Tesla A100 GPU, as depicted in Table 13. Though the decoder part of tasks brings a certain amount of parameters, the computational complexity mainly comes from the encoder part, compared to the original BEVFormer detector (ID. 1). We also provide a comparison with the recent BEVerse [105]. UniAD owns more tasks, achieves superior performance, and has lower FLOPs - indicating affordable budget to additional computation cost.

我们在 Nvidia Tesla A100 GPU 上测量 UniAD 的复杂度与运行时间，见表 13。尽管任务解码器带来一定参数量，相对原版 BEVFormer 检测器（ID. 1），计算复杂度主要来自编码器部分。我们也与近期 BEVerse [105] 比较：UniAD 任务更多、表现更优，且 FLOPs 更低 ——表明额外计算代价在可承受预算内。

F.4 Model scale

F.4 模型规模

We provide three variations of UniAD under different model scales as shown in Table 12. The chosen image backbones for image-view feature extraction are ResNet-50[32], ResNet-101 and VoVNet 2-99[51] for UniAD-S, UniAD-B and UniAD-L respectively. Since the model scale (image encoder) mainly influences the BEV feature quality, we could observe that the perceptual scores improve with a larger backbone, which further could lead to better prediction and planning performance.

我们在表 12 给出三种不同规模的 UniAD 变体。图像视图特征提取所用骨干分别为 UniAD-S 的 ResNet-50[32]、UniAD-B 的 ResNet-101 与 UniAD-L 的 VoVNet 2-99[51]。由于模型规模（图像编码器）主要影响 BEV 特征质量，可观察到更大骨干带来更高感知分数，并进一步改善预测与规划表现。

F.5 Qualitative results

F.5 定性结果

Attention mask visualization. To investigate the internal mechanism and show its explainability, we visualize the attention mask of the cross-attention module in the planner. As shown in Fig. 8, the predicted tracking bounding boxes, planned trajectory, and the ground truth HD Map are rendered for reference, and the attention mask is overlayered on top. From left to right, we show two consecutive frames in a time sequence but with different navigation commands. We can observe that the planned trajectory varies largely according to the command. Also, much attention is paid to the goal lane as well as the critical agents that are yielding to our ego vehicle.

注意力掩码可视化。 为考察内部机制并展示可解释性，我们可视化规划器中交叉注意力模块的注意力掩码。如图 8，渲染预测跟踪边界框、规划轨迹与真实 HD 地图作参考，并叠加注意力掩码。从左到右展示同一时序中连续两帧但导航指令不同的情况。可见规划轨迹随指令大幅变化；大量注意力落在目标车道以及对自车让行的关键智能体上。

Visualization of different scenarios. We provide visualizations for more scenarios, including cruising around the urban areas (Fig. 9), critical cases (Fig. 10), and obstacle avoidance scenarios (Fig. 11). One promising evidence for our planning-oriented design is shown in Fig. 12, where inaccurate results occur in prior modules while the later tasks could still recover. Similarly, we show results for all tasks in surround-view images, BEV, as well as the attention mask from the planner. A demo video[^3] is also provided for reference.

不同场景可视化。 我们提供更多场景可视化，包括城区巡航（图 9）、关键案例（图 10）与避障场景（图 11）。图 12 展示规划导向设计的一个有希望证据：上游模块结果不准确时，后续任务仍可恢复。同样，我们在环视图像、BEV 以及规划器注意力掩码中展示全部任务结果。亦提供演示视频[^3]供参考。

Failure cases are essential for an autonomous driving algorithm to understand its weakness and guide future work, and here we present some failure cases of UniAD. The failure cases of UniAD are mainly under some long-tail scenarios where all modules are affected, as depicted in Fig. 13 and Fig. 14.

失败案例 对自动驾驶算法理解自身弱点并指导未来工作至关重要，此处展示 UniAD 的一些失败案例。如图 13 与图 14，失败主要出现在所有模块均受影响的长尾场景。

Figure 8: Effectiveness of navigation command and attention mask visualization. Here we demonstrate how attention is paid in accordance with the navigation command. We render the attention mask from the BEV interaction module in the planning module, the predicted tracking bounding boxes as well as the planned trajectory. The navigation command is printed on the bottom left, and the HD Map is rendered only for reference. From left to right, we show two consecutive frames in a time sequence but with different navigation commands. We can observe that the planned trajectory varies largely according to the command. Also, much attention is paid to the goal lane as well as the critical agents that are yielding to our ego vehicle.

图 8： 导航指令效果与注意力掩码可视化。展示注意力如何随导航指令变化。我们渲染规划模块 BEV 交互的注意力掩码、预测跟踪边界框以及规划轨迹。导航指令印在左下，HD 地图仅供参考。从左到右为同一时序连续两帧但指令不同：规划轨迹随指令大幅变化；大量注意力落在目标车道以及对自车让行的关键智能体上 。

Figure 9: Visualization for cruising around the urban areas. UniAD can generate high-quality interpretable perceptual and prediction results, and make a safe maneuver. The first three columns show six camera views, and the last two columns are the predicted results and the attention mask from the planning module respectively. Each agent is illustrated with a unique color. Only top-1 and top-3 trajectories from motion forecasting are selected for visualization on images-view and BEV respectively.

图 9： 城区巡航可视化。UniAD 可生成高质量、可解释的感知与预测结果，并做出安全动作。前三列为六路相机视图，后两列分别为预测结果与规划模块注意力掩码。每个智能体用独特颜色表示。图像视图与 BEV 分别只选取运动预报的 top-1 与 top-3 轨迹可视化 。

Figure 10: Critical case visualization. Here we demonstrate two critical cases. The first scenario (top) shows that the ego vehicle is yielding to two pedestrians crossing the street, and the second scenario (down) shows that the ego vehicle is yielding to a fast-moving car and waiting to go straight without protection near an intersection. We can observe that much attention is paid to the most critical agents, i.e ., pedestrians and fast-moving vehicles, as well as the intended goal location.

图 10： 关键案例可视化。展示两个关键案例。上：自车对两名过马路行人让行；下：自车对快速驶来车辆让行，并在无保护路口等待直行。可见大量注意力落在最关键智能体（行人与快速车辆）以及意图目标位置上 。

Figure 11: Obstacles avoidance visualization. In these two scenarios, the ego vehicle is changing lanes attentively to avoid the obstacle vehicle. From the attention mask, we can observe that our method focuses on the obstacles as well as the road in the front and back.

图 11： 避障可视化。两例中自车注意变道以避开障碍车辆。从注意力掩码可见，方法关注障碍物以及前后方道路 。

Figure 12: Visualization for planning recovering from perception failures. We show an interesting case where inaccurate results occur in prior modules while the later tasks could still recover. The top row and the down row represent two consecutive frames from the same scenario. The vehicle in the red circle is moving from a far distance toward the intersection at a high speed. It is observed that the tracking module misses it at first, then captures it at the latter frame. The blue circles show a stationary car yielding to the traffic, and it is missed in both frames. Interestingly, both vehicles show strong reactions to the attention masks of the planner, even though they are missed in the prior modules. It means that our planner still pays attention to those critical though missed agents, which is intractable in previous fragmented and non-unified driving systems, and demonstrates the robustness of UniAD.

图 12： 规划从感知失败中恢复的可视化。展示上游模块结果不准确而后续任务仍可恢复的有趣案例。上下两行表示同一场景连续两帧。红圈中车辆正高速从远处驶向路口：跟踪模块起初漏检，后一帧捕获。蓝圈为对交通让行的静止车，两帧均漏检。有趣的是，尽管跟踪漏检，两车仍在规划器注意力掩码中有强烈响应 。

Figure 13: Failure cases 1. Here we present a long-tail scenario, where a large trailer with a white container occupies the entire road. We can observe that our tracking module fails to detect the accurate size of the front trailer and heading angles of vehicles beside the road.

图 13： 失败案例 1。展示长尾场景：带白色集装箱的大型拖车占据整条道路。可见跟踪模块未能准确检测前方拖车尺寸以及路边车辆航向角 。

Figure 14: Failure cases 2. In this case, the planner is over-cautious about the incoming vehicle in the narrow street. The dark environment is one critical type of long-tail scenarios in autonomous driving. Applying smaller collision loss weight and more regulation regarding the boundary might mitigate the problem.

图 14： 失败案例 2。本例中规划器对窄巷迎面来车过于谨慎。黑暗环境是自动驾驶长尾场景的一类关键类型。减小碰撞损失权重并加强边界相关正则或可缓解该问题 。

Planning-oriented Autonomous Driving

面向规划的自动驾驶

Abstract

摘要

1 Introduction

1 引言

2 Methodology

2 方法

Overview.

概览。

2.1 Perception: Tracking and Mapping

2.1 感知：跟踪与建图

TrackFormer.

TrackFormer。

MapFormer.

MapFormer。

2.2 Prediction: Motion Forecasting

2.2 预测：运动预报

MotionFormer.

MotionFormer。

Motion queries.

运动 query。

Non-linear Optimization.

非线性优化。

2.3 Prediction: Occupancy Prediction

2.3 预测：占据预测

Pixel-agent interaction

像素—智能体交互

Instance-level occupancy.

实例级占据。

2.4 Planning

2.4 规划

2.5 Learning

2.5 学习

Shared matching.

共享匹配。

3 Experiments

3 实验

3.1 Joint Results

3.1 联合结果

Roadmap toward safe planning.

通向安全规划的路线图。

3.2 Modular Results

3.2 模块结果

Perception results.

感知结果。

Prediction results.

预测结果。

Planning results.

规划结果。

3.3 Qualitative Results

3.3 定性结果

3.4 Ablation Study

3.4 消融研究

Effect of designs in MotionFormer.

MotionFormer 设计的影响。

Effect of designs in OccFormer.

OccFormer 设计的影响。

Effect of designs in Planner.

规划器设计的影响。

4 Conclusion and Future Work

4 结论与未来工作

References

参考文献

Appendix A Task Definition

附录 A 任务定义

Detection and tracking.

检测与跟踪。

Online mapping.

在线建图。

Motion forecasting.

运动预报。

Occupancy prediction.

占据预测。

Planning.

规划。

Appendix B The Necessity of Each Task

附录 B 各任务的必要性

Appendix C Related Work

附录 C 相关工作