Real-Time Anomaly Detection and Reactive Planning with Large Language Models

Rohan Sinha Affiliation:  Dept. of Aeronautics and Astronautics, Stanford University.    Amine Elhafsi Affiliation:  Dept. of Aeronautics and Astronautics, Stanford University.    Christopher Agia Affiliation:  Dept. of Computer Science, Stanford University.    Matthew Foutter Affiliation:  Dept. of Mechanical Engineering, Stanford University.    Edward Schmerling Affiliation:  NVIDIA. Contact: {rhnsinha, amine, cagia, mfoutter, pavone}@stanford.edu , eschmerling@nvidia.com .    Marco Pavone Affiliation:  Dept. of Aeronautics and Astronautics, Stanford University. Affiliation:  NVIDIA. Contact: {rhnsinha, amine, cagia, mfoutter, pavone}@stanford.edu , eschmerling@nvidia.com .

Abstract

Foundation models, e.g., large language models (LLMs), trained on internet-scale data possess zero-shot generalization capabilities that make them a promising technology towards detecting and mitigating out-of-distribution failure modes of robotic systems. Fully realizing this promise, however, poses two challenges: (i) mitigating the considerable computational expense of these models such that they may be applied online, and (ii) incorporating their judgement regarding potential anomalies into a safe control framework. In this work, we present a two-stage reasoning framework: First is a fast binary anomaly classifier that analyzes observations in an LLM embedding space, which may trigger a slower fallback selection stage that utilizes the reasoning capabilities of generative LLMs. These stages correspond to branch points in a model predictive control strategy that maintains the joint feasibility of continuing along various fallback plans to account for the slow reasoner’s latency as soon as an anomaly is detected, thus ensuring safety. We show that our fast anomaly classifier outperforms autoregressive reasoning with state-of-the-art GPT models, even when instantiated with relatively small language models. This enables our runtime monitor to improve the trustworthiness of dynamic robotic systems, such as quadrotors or autonomous vehicles, under resource and time constraints. Videos illustrating our approach in both simulation and real-world experiments are available on our project page: https://sites.google.com/view/aesop-llm.

在互联网规模数据上训练的基础模型（如大语言模型 LLM）具备零样本泛化能力，为检测并缓解机器人系统的分布外失效模式提供了有前景的技术路径。但充分兑现这一潜力面临两项挑战：(i) 缓解这些模型的高昂计算开销以便在线使用；(ii) 将其关于潜在异常的判断纳入安全控制框架。本文提出两阶段推理框架：首先是在 LLM 嵌入空间中分析观测的快速二分类异常分类器；一旦触发，再进入利用生成式 LLM 推理能力的较慢回退选择阶段。这两阶段对应模型预测控制策略中的分支点：一旦检测到异常，即维持沿各回退计划继续前进的联合可行性，以覆盖慢推理器的时延，从而保证安全。我们表明，即便用相对较小的语言模型实例化，快速异常分类器也优于用最先进 GPT 模型做自回归推理。这使得运行时监视器能在资源与时间约束下提升四旋翼或自动驾驶汽车等动态机器人系统的可信度。仿真与真机实验视频见项目页：https://sites.google.com/view/aesop-llm。

I Introduction

I 引言

Autonomous robotic systems are rapidly advancing in capabilities, seemingly on the cusp of widespread deployment in the real world. However, a persistent challenge is that the finite datasets used to develop these systems are unlikely to capture the limitless variety of the real world, leading to unexpected failure modes when conditions deviate from training data, or when the robot encounters rare situations that were not well-represented at design time. To mitigate the resulting safety implications, we require methods that can 1) assess the reliability of a machine learning (ML) enabled system at runtime and 2) judiciously enact safety-preserving interventions if necessary.

自主机器人系统能力快速提升，似乎已接近大规模真实部署。然而持久挑战是：用于开发这些系统的有限数据难以覆盖真实世界的无限多样性，当条件偏离训练数据、或遭遇设计时代表性不足的罕见情形时，会出现意外失效模式。为缓解随之而来的安全影响，我们需要能在运行时 1) 评估机器学习赋能系统的可靠性，并 2) 必要时审慎实施保全安全的干预。

In this work, we investigate the utility of foundation models (FMs), specifically, large language models (LLMs), towards these two objectives by employing LLMs as runtime monitors tasked with 1) detecting anomalous conditions and 2) reasoning about the appropriate safety-preserving course of action. We do so because recent work has shown that the internet-scale pretraining data provides FMs with strong zero-shot reasoning capabilities, which has enabled robots to perform complex tasks [5], identify and correct failures [18], and reason about potential safety hazards in their surroundings [12] without explicit training to do so.

本文研究基础模型（尤其是 LLM）对上述两项目标的效用：将 LLM 用作运行时监视器，负责 1) 检测异常条件，2) 推理恰当的保全安全行动。近期工作表明，互联网规模预训练使基础模型具备强零样本推理能力，使机器人无需专门训练即可执行复杂任务 [5]、识别并纠正失败 [18]、推理周围潜在安全危害 [12]。

However, the adoption of FMs in-the-loop of safety-critical robotic systems is immediately met with two challenges. First, the ever growing scale of FMs poses a major obstacle towards enabling real-time, reactive reasoning about unexpected safety-critical events, especially on agile robotic systems with limited compute. Hence, existing work that applies FMs to robotics has focused on quasi-static (e.g., manipulation) or offline settings that afford large times delays while the LLM completes its reasoning. Second, the application of FMs as runtime monitors requires that they are grounded with respect to the task and capabilities of the system. However, the community has not converged on rigorous methods for grounding FMs without compromising on their generalist zero-shot reasoning abilities (e.g., fine-tuning [24] or linear probing [54] often underperform OOD); prompt design remains a standard practice.

但在安全关键机器人系统的闭环中采用基础模型，立即面临两项挑战。其一，基础模型规模持续增长，严重阻碍对突发安全关键事件做实时、反应式推理，尤其在算力有限的敏捷机器人上；因此现有将基础模型用于机器人的工作多聚焦准静态（如操作）或可承受大时延等待 LLM 完成推理的离线设定。其二，把基础模型用作运行时监视器要求其相对任务与系统能力被* grounding*；但社区尚未就在不牺牲通才零样本推理的前提下 grounding 基础模型形成严格方法（例如微调 [24] 或线性探测 [54] 在 OOD 上常表现不佳）；提示设计仍是标准做法。

Fig. 1: We present an embedding-based runtime monitoring scheme using fast and slow language model reasoners in concert. During nominal operation, the fast reasoner differentiates between nominal and anomalous robot observations. If an anomaly is flagged, the system enters a fallback-safe state while the slow reasoner determines the anomaly’s hazard. In this fallback-safe state, we guarantee access to a set of safe recovery plans (if the anomaly is consequential) and access to continued nominal operation (if the anomaly is inconsequential).

图 1：我们提出结合快慢语言模型推理器的基于嵌入的运行时监视方案。标称运行时，快推理器区分标称与异常机器人观测。一旦标记异常，系统进入回退安全状态，同时慢推理器判断异常危害。在该回退安全状态中，我们保证：若异常后果严重则可访问一组安全恢复计划；若异常无关紧要则可继续标称运行。

To address these challenges, we present AESOP[^1], an anomaly detection and reactive planning framework that aims to derive maximum utility of an LLM’s zero-shot reasoning capabilities while taking LLM inference latencies into account within the control design. As shown in Fig. 1, AESOP splits the monitoring task into two separate stages: The first is rapid, real-time detection of anomalies---conditions that deviate from the nominal conditions where the robot performs reliably---by querying similarity with previously recorded observations within the contextual embedding space of an LLM. The second stage is slower, methodical generative reasoning on how to respond to an anomalous scenario once it has been detected. We combine the resultant monitoring pipeline with a model predictive control strategy that maintains multiple trajectory plans, each corresponding to a safety-preserving intervention, in a way that ensures their joint feasibility for an upper bound on the time it takes the slower, generative reasoning to complete[^2]. As such, our contributions are threefold:

为应对这些挑战，我们提出 AESOP[^1]：在控制设计中显式考虑 LLM 推理时延的同时，尽量发挥 LLM 零样本推理能力的异常检测与反应规划框架。如图 1，AESOP 将监视任务分为两阶段：第一阶段在 LLM 上下文嵌入空间中查询与既往观测的相似度，快速实时检测偏离机器人可靠工作标称条件的异常；第二阶段在检测到异常后，用较慢、更审慎的生成式推理决定如何应对。我们将该监视流水线与模型预测控制策略结合：维护多条轨迹计划（各对应一种保全安全干预），并保证在慢生成推理完成时间的上界内它们联合可行。[^2]因此本文贡献有三：

1. Fast reasoning with embeddings: We propose a real-time anomaly detection method that, using relatively small FMs (e.g., $120$M parameters) and the robot’s previous nominal experiences, surpasses generative chain-of-thought (CoT) reasoning with high-capacity LLMs such as GPT-4. Our method runs at $20\text{Hz}$ on an Nvidia Jetson AGX ORIN, a $357$x speed up over cloud querying GPT-4. To our knowledge, this is the first application of FM embeddings to the task of runtime monitoring, enabling safe and real-time control of an agile robotic system.

1. 用嵌入做快推理： 我们提出实时异常检测方法，用相对较小的基础模型（如 $120$M 参数）与机器人既往标称经验，超越用 GPT-4 等高容量 LLM 做生成式思维链（CoT）推理。方法在 Nvidia Jetson AGX ORIN 上以 $20\text{Hz}$ 运行，相对云端查询 GPT-4 加速 $357$ 倍。据我们所知，这是首次将基础模型嵌入用于运行时监视，使敏捷机器人系统可安全实时控制。

2. Slow reasoning through autoregressive generation: While the faster anomaly detector merely detects deviations from prior experiences, we show that autoregressive generation of longer output sequences allows the LLM-based monitor to methodically reason about the safety consequences of out-of-distribution scenarios and decide whether intervention is necessary in a zero-shot fashion; i.e., not all anomalies lead to system-level failures.

2. 用自回归生成做慢推理： 较快的异常检测器仅检测相对既往经验的偏离；我们表明，通过自回归生成更长输出序列，基于 LLM 的监视器可审慎推理分布外场景的安全后果，并以零样本方式决定是否需要干预——并非所有异常都会导致系统级失败。

3. Hierarchical multi-contingency planning: Facilitated by our fast anomaly detector, we introduce a predictive control framework to integrate both FM-based reasoners in a lower-level reactive control loop by maintaining multiple feasible trajectories, each corresponding to a high-level intervention strategy. This allows the robot to 1) react to sudden semantic changes in the robot’s environment, 2) maintain closed-loop safety while waiting for a slow reasoner to return a decision, and 3) exhibit dynamic, agile behaviors within the range of scenarios where the nominal autonomy stack is trustworthy.

3. 分层多应急规划： 借助快速异常检测器，我们引入预测控制框架，在下层反应控制环中同时接入两种基础模型推理器：维护多条可行轨迹，各对应一种高层干预策略。这使得机器人能够 1) 对环境中突发语义变化做出反应，2) 在等待慢推理器返回决策时保持闭环安全，3) 在标称自主栈可信的场景范围内展现动态敏捷行为。

We demonstrate these facts across several commonplace LLMs, ranging from $10^{8}-10^{12}$ parameters, as well as conventional OOD detection techniques on 1) an extensive suite of synthetic text-based domains, 2) simulated and real-world closed-loop quadrotor experiments resembling a drone delivery service, and 3) careful recreations of recent real-world failure modes of autonomous vehicles in the CARLA simulator [11]. We conclude that the use of FMs not only presents a promising direction to significantly improve the robustness of autonomous robotic systems to out-of-distribution scenarios, but also that their real-time integration within dynamic, agile robotic systems is already practically feasible.

我们在参数量 $10^{8}-10^{12}$ 的多种常见 LLM 以及传统 OOD 检测技术上验证上述结论，覆盖：1) 大量合成文本域；2) 类似无人机配送的仿真与真机闭环四旋翼实验；3) 在 CARLA 仿真器 [11] 中仔细复现自动驾驶汽车近期真实失效模式。结论是：使用基础模型不仅有望显著提升自主机器人系统对分布外场景的鲁棒性，且其实时接入动态敏捷机器人系统在实践上已可行。

Organization: We first discuss related work in Section II and formalize the problem setup in Section III. Then, we present our approach in Section IV and evaluate our method in Section V. Finally, we conclude and provide a future outlook in Section VI. In addition, we include a full overview of the notation and conventions used in this paper in Section -A.

组织结构： 第 II 节讨论相关工作，第 III 节形式化问题设定，第 IV 节给出方法，第 V 节评估，第 VI 节结论与展望。此外第 -A 节给出全文符号与约定总览。

II Related Work

II 相关工作

Out-of-Distribution Robustness: The fact that learning-based systems often behave unreliably on data that is dissimilar from their training data has been extensively documented in both the machine learning and robotics literature [14, 37, 33, 45]. Approaches to address the subsequent challenges broadly fall into two categories [45]: First are methods that strengthen a model’s performance in the face of distributional shift. For example, through robust training (e.g., [41]) or by adapting the model to changing conditions (e.g., [16, 8]). Second are so called out-of-distribution detection algorithms [42, 40], that aim to detect when a given model is unreliable, e.g., by computing the variance of an ensemble [25] or computing energy scores [29]. Recent work has shown the merits of generalist FMs like LLMs in both domains: Studies have shown that zero-shot application of a FM (e.g., in [54], the authors apply CLIP zero-shot on ImageNet), vastly improves OOD generalization over previous approaches, like distributionally robust training [31, 54, 6]. In addition, existing OOD detection methodologies and their application within robot autonomy stacks are tailored to detect conditions that compromise the reliability of individual components of an autonomy stack, like whether a perception system’s detections are correct [39, 13, 36, 46]. Instead, recent work showed that LLMs may provide a more general mechanism to detect context dependent safety hazards, especially those that are hard to measure with predefined performance metrics [12]. For example, an autonomous EVTOL may monitor the quality of the vision system’s landing pad location estimate, but even if the EVTOL has high confidence that it can land successfully, the outcome of landing on a building that is on fire can have profound negative consequences. However, despite the attractive properties of LLMs, these works do not propose practical strategies to integrate them in closed-loop. Therefore, we propose a closed-loop control framework that can both use the LLM to identify unseen anomalies and strengthen performance in the presence of rare failure modes.

分布外鲁棒性： 学习系统在与训练数据不相似的数据上常表现不可靠，已在机器学习与机器人文献中广泛记载 [14, 37, 33, 45]。应对思路大致两类 [45]：一类强化模型在分布偏移下的表现，例如鲁棒训练 [41] 或适应变化条件 [16, 8]；另一类是分布外检测算法 [42, 40]，旨在发现给定模型不可靠，例如计算集成方差 [25] 或能量分数 [29]。近期工作显示通才基础模型在这两方面都有优势：零样本应用基础模型（如 [54] 在 ImageNet 上零样本用 CLIP）相对分布鲁棒训练等先前方法大幅改善 OOD 泛化 [31, 54, 6]。此外，现有 OOD 检测及其在机器人自主栈中的应用多针对损害栈内单组件可靠性的条件，例如感知检测是否正确 [39, 13, 36, 46]。近期工作则表明 LLM 可提供更一般的机制来检测依赖上下文、尤其难以用预定义性能指标度量的安全危害 [12]——例如自主 eVTOL 可监视视觉系统对着陆垫位置估计的质量，但即便有高置信度成功着陆，落在着火建筑物上仍可能带来严重负面后果。尽管 LLM 性质诱人，这些工作未提出将其接入闭环的实用策略。因此我们提出既能用 LLM 识别未见异常、又能在罕见失效模式下强化表现的闭环控制框架。

Foundation Models in Robotics: The integration of large lanugage models (LLMs) and, more broadly, foundation models (FMs) into robotics has sparked considerable interest due to their proficiency in managing complex, unstructured tasks that demand sophisticated reasoning skills. These models have been instrumental in bridging the gap between natural language instructions and the execution of physical actions in the real world. Various approaches utilizing these models have been developed for online use in applications in areas such as manipulation [19], navigation [43], drone flight [9], and long-horizon planning [5, 28]. FMs have also been used to define reinforcement learning reward functions [58], generate robot policy code [27], or create additional training data [56, 57, 2].

机器人中的基础模型： 将 LLM 乃至更广的基础模型接入机器人引发广泛兴趣，因其擅长管理需要复杂推理的非结构化任务，有助于弥合自然语言指令与真实世界物理动作执行之间的鸿沟。已有方法用于操作 [19]、导航 [43]、无人机飞行 [9]、长时程规划 [5, 28] 等在线应用；也被用于定义强化学习奖励 [58]、生成机器人策略代码 [27] 或创造额外训练数据 [56, 57, 2]。

However, the issue of response time associated with FMs has not been a focal point in these studies. The aforementioned online methods predominantly rely on a quasi-static assumption, implying that the timing of the robot’s actions is not critical. This assumption allows the system the luxury of time to consult the LLMs and await their responses without urgency. Conversely, the latter methods either operate offline or utilize LLMs in manners that are similarly insensitive to response time.

但这些研究未把基础模型的响应时间作为焦点。上述在线方法大多依赖准静态假设，即机器人动作时机不关键，系统有充裕时间咨询 LLM 并等待回复。后一类方法则离线运行，或以同样对响应时间不敏感的方式使用 LLM。

As such, existing work demonstrates limited dynamic reactivity of the policy, which is essential for fast-moving, agile robots like quadrotors. These robots can quickly find themselves in situations where a delayed response can result in an unavoidable crash. To mitigate this issue, our approach specifically considers the delays introduced by the reasoning process. We enhance reactivity by implementing a more rapid anomaly detection system, thereby reducing the risk of crashes by allowing for timely corrective actions.

因此现有工作对策略的动态反应性展示有限，而对四旋翼等快速敏捷机器人而言，反应性至关重要——延迟响应可能导致不可避免的坠毁。为缓解这一问题，我们的方法显式考虑推理过程引入的时延，并通过更快的异常检测提升反应性，从而及时纠正动作、降低坠毁风险。

Accelerating Inference: It is well-recognized that increasing FM capabilities are accompanied with increasing computational cost and inference latency. As such, substantial effort is being dedicated to the acceleration of these models, of which several popular strategies have emerged such as model distillation [15, 17], quantization [20, 55], and parameter sparsification [49, 34]. Ultimately, these approaches improve the cost of the forward pass through a transformer model, but do not address the fact that LLMs typically need to generate long sequences of outputs to reason towards the correct decision [53], a process unlikely to run in real-time for time-sensitive tasks. Querying remotely hosted models on large-scale hardware (e.g., GPT-4 [1]) is a potential solution if computation constraints become too stringent for a system to perform onboard FM inference, yet network conditions may incur inconsistent and potentially significant delays, and connectivity may be unreliable for in-the-wild deployments.

加速推理： 基础模型能力提升伴随计算成本与推理时延上升已是共识。大量工作致力于加速，如模型蒸馏 [15, 17]、量化 [20, 55]、参数稀疏化 [49, 34]。这些方法改善 Transformer 前向传播成本，但未解决 LLM 通常需生成长输出序列才能推理到正确决策 [53] 的事实——对时间敏感任务这很难实时。若机载推理算力过紧，在大规模硬件上远程查询模型（如 GPT-4 [1]）或是出路，但网络条件可能带来不一致且可能显著的时延，野外部署连通性也不可靠。

III Problem Formulation

III 问题形式化

In this work, we consider a robot with discrete time dynamics

本文考虑具有离散时间动力学的机器人

$$\bm{x}_{t+1}=\bm{f}(\bm{x}_{t},\bm{u}_{t}), \tag{1}$$

where $\bm{x}_{t}\in\mathbb{R}^{n}$ represents the robot’s state, and $\bm{u}_{t}\in\mathbb{R}^{m}$ is the control input. Nominally, we aim to minimize some control objective $C$ that depends on the states and inputs, subject to safety constraints on the state $\bm{x}_{t}\in\mathcal{X}\subseteq\mathbb{R}^{n}$ and input $\bm{u}_{t}\in\mathcal{U}\subseteq\mathbb{R}^{m}$. For example, a quadrotor’s state consists of its pose and velocity (estimated from e.g., GPS, visual-SLAM, and IMUs), and its objective may be to minimize distance to a landing zone subject to collision avoidance constraints.

其中 $\bm{x}_{t}\in\mathbb{R}^{n}$ 为机器人状态，$\bm{u}_{t}\in\mathbb{R}^{m}$ 为控制输入。标称上我们最小化依赖状态与输入的控制目标 $C$，并受状态安全约束 $\bm{x}_{t}\in\mathcal{X}\subseteq\mathbb{R}^{n}$ 与输入约束 $\bm{u}_{t}\in\mathcal{U}\subseteq\mathbb{R}^{m}$。例如四旋翼状态含位姿与速度（可由 GPS、视觉 SLAM、IMU 等估计），目标可以是在避碰约束下最小化到着陆区的距离。

In addition to the state variables tracked by the nominal control loop, the robot receives an observation $\bm{o}_{t}\in\mathcal{O}$ at each timestep, which provides further contextual information about the robot’s environment. Our goal is to design a runtime monitor that interferes with the nominal system to avoid system-level safety hazards, which may depend on environmental factors not represented in the robot’s state $\bm{x}_{t}$. For example, a quadrotor cannot safely land on a landing zone covered in burning debris even if the nominal control stack has the ability to do so. In the spirit of [12], we refer to such events as semantic failure modes, as they do not necessarily constitute violation of precise state constraints $\mathcal{X}$, but instead depend on the qualitative context of the robot’s task.

除标称控制环跟踪的状态变量外，机器人每步还收到观测 $\bm{o}_{t}\in\mathcal{O}$，提供关于环境的更多上下文。我们的目标是设计运行时监视器，在标称系统可能因未体现在状态 $\bm{x}_{t}$ 中的环境因素而面临系统级安全危害时进行干预。例如，即便标称控制栈有能力着陆，着陆区若覆盖燃烧残骸，四旋翼仍不能安全着陆。沿 [12] 的精神，我们称此类事件为语义失效模式：它们未必违反精确状态约束 $\mathcal{X}$，而取决于任务的定性上下文。

Further, we assume that we have access to a dataset $\mathcal{D}_{\mathrm{nom}}=\{\bm{o}_{i}\}_{i=1}^{N}$ of nominal observations wherein the robot was safe and reliable. Conceptually, $\mathcal{D}_{\mathrm{nom}}$ corresponds to the operational data of a notionally mature system, and may consist of data used to train the system or of previously collected deployment data. This data will overwhelmingly contain mundane scenarios where the robot performs well; our monitoring framework is targeted instead at the challenging, extremely rare corner cases that are unlikely to have been recorded before and threaten the robot’s reliability.

进一步假设我们可访问标称观测数据集 $\mathcal{D}_{\mathrm{nom}}=\{\bm{o}_{i}\}_{i=1}^{N}$，其中机器人安全可靠。概念上 $\mathcal{D}_{\mathrm{nom}}$ 对应名义上成熟系统的运行数据，可含训练数据或既往部署数据。这些数据绝大多数是机器人表现良好的平常场景；我们的监视框架针对的是极难事先记录、却威胁可靠性的极罕见边角情形。

In the event that a failure mode of the nominal autonomy stack is imminent, we must select and engage a safety-preserving intervention. For example, we may choose to land the quadrotor in another open landing zone like a grassy field. To this end, we follow [46] and we assume that we are given a number of recovery regions $\mathcal{X}_{R}^{1},\mathcal{X}_{R}^{2},\dots,\mathcal{X}_{R}^{d}\subseteq\mathcal{X}$, control invariant subsets of the state space that correspond to high-level safety interventions. For example, $\mathcal{X}_{R}^{i}$ may represent the alternate landing zone. By planning a trajectory to the appropriate recovery set, potential safety hazards can be avoided. As shown in [46], such sets can both be hand defined up-front and identified using reachability analysis.

当标称自主栈的失效模式迫近时，必须选择并启用保全安全的干预。例如可选择在草地等另一开放着陆区降落。为此我们遵循 [46]，假设给定若干恢复区域 $\mathcal{X}_{R}^{1},\mathcal{X}_{R}^{2},\dots,\mathcal{X}_{R}^{d}\subseteq\mathcal{X}$，它们是状态空间中对应高层安全干预的控制不变子集；例如 $\mathcal{X}_{R}^{i}$ 可表示备选着陆区。规划到恰当恢复集的轨迹即可规避潜在安全危害。如 [46] 所示，这些集合既可事先手工定义，也可用可达性分析识别。

IV Proposed Approach

IV 提出的方法

It is virtually impossible to account for all the corner-cases and semantic failure modes that a system may experience through a standard engineering pipeline. Even if we train e.g., classifiers to detect obstructions on landing zones, there may always remain a class of semantic failure modes that we have not accounted for. Instead, we propose to leverage generalist foundation models to detect and reason holistically about a robot’s environment. We first present our FM-based monitoring approach, after which we construct a planning algorithm that accounts for the latency that FM-based reasoning may induce.

通过标准工程流程几乎不可能枚举系统可能遭遇的全部边角情形与语义失效模式。即便训练例如检测着陆区障碍的分类器，也总会留下一类未计入的语义失效模式。相反，我们提议借助通才基础模型对机器人环境做整体检测与推理。我们先给出基于基础模型的监视方法，再构造考虑基础模型推理可能引入时延的规划算法。

IV-A Runtime Monitor: Fast and Slow Reasoning

IV-A 运行时监视器：快慢推理

To detect and avoid semantic failure modes, we propose a two-stage pipeline. The first is the detection of anomalies, simply defined as conditions that deviate from the mundane, nominal experiences where we know that our notionally mature system is reliable. The second is slower reasoning about the downstream consequence of an anomaly, if detected, towards a high-level decision on whether a safety-preserving intervention should be executed. We refer to Section -B for a brief introduction to anomaly detection used hereafter.

为检测并规避语义失效模式，我们提出两阶段流水线。第一阶段检测异常——简单定义为偏离我们知道名义成熟系统可靠的平常标称经验的条件。第二阶段在检测到异常后，较慢地推理其下游后果，以做出是否执行保全安全干预的高层决策。下文所用异常检测的简要介绍见第 -B 节。

Fast Anomaly Detection: To detect anomalies, we need to inform a FM of the context within which the autonomous system is known to be trustworthy. The prior, nominal experiences of the robot serve as such grounding. We construct an anomaly score function $\bm{s}(\bm{o}_{t},\mathcal{D}_{\mathrm{nom}})\in\mathbb{R}$ to query whether a current observation $\bm{o}_{t}$ differs from the previous experiences in $\mathcal{D}_{\mathrm{nom}}$. We do not require any particular methodology to generate the score, we just require that scoring an observation is computationally feasible in real-time; that is, within a single time step.

快速异常检测： 为检测异常，需告知基础模型自主系统被认为可信的上下文；机器人既往标称经验即提供此种 grounding。我们构造异常分数函数 $\bm{s}(\bm{o}_{t},\mathcal{D}_{\mathrm{nom}})\in\mathbb{R}$，查询当前观测 $\bm{o}_{t}$ 是否不同于 $\mathcal{D}_{\mathrm{nom}}$ 中的既往经验。不要求特定打分方法，只要求对观测打分在计算上可实时完成，即在单个时间步内完成。

This work emphasizes the value of computing anomaly scores using language-based representations, which we show capture the semantics of the observation within the context of the robot’s task in Section V. To do so, we first create a cache of embedding vectors $\mathcal{D}_{e}=\{\bm{e}_{i}\}_{i=1}^{N}$ where $\bm{e}_{i}=\phi(\bm{o}_{i})\in\mathbb{R}^{e}$ for each $\bm{o}_{i}\in\mathcal{D}_{\mathrm{nom}}$ by embedding the robot’s prior experiences offline using an embedding FM $\phi$. Then, at runtime, we observe $\bm{o}_{t}$, compute its corresponding embedding $\bm{e}_{t}$, and compute an anomaly score $\bm{s}(\bm{e}_{t};\mathcal{D}_{e})$ using the vector cache. We investigate several simple score functions (see Section -D3 for a full list), each of which roughly measures a heuristic notion of difference with respect to $\mathcal{D}_{\mathrm{nom}}$. For example, the simplest metric uses the maximum cosine similarity with respect to samples in the prior experience cache,

本文强调用基于语言的表示计算异常分数的价值；第 V 节表明它们能在机器人任务上下文中刻画观测语义。为此，我们先用嵌入基础模型 $\phi$ 离线嵌入机器人既往经验，建立嵌入向量缓存 $\mathcal{D}_{e}=\{\bm{e}_{i}\}_{i=1}^{N}$，其中 $\bm{e}_{i}=\phi(\bm{o}_{i})\in\mathbb{R}^{e}$，$\bm{o}_{i}\in\mathcal{D}_{\mathrm{nom}}$。运行时观测 $\bm{o}_{t}$，计算对应嵌入 $\bm{e}_{t}$，再用向量缓存计算异常分数 $\bm{s}(\bm{e}_{t};\mathcal{D}_{e})$。我们考察若干简单分数函数（完整列表见第 -D3 节），各自大致度量相对 $\mathcal{D}_{\mathrm{nom}}$ 的启发式差异。例如最简单度量使用相对既往经验缓存样本的最大余弦相似度，

$$\bm{s}(\bm{e}_{t};\mathcal{D}_{e}):=-\max_{\bm{e}_{i}\in\mathcal{D}_{e}}{\frac{\bm{e}_{i}^{T}\bm{e}_{t}}{\|\bm{e}_{i}\|\|\bm{e}_{t}\|}},$$

which, in effect, retrieves the most similar prior experience from $\mathcal{D}_{e}$ to construct the score. Intuitively, this approach measures whether anything similar to the current observation has been seen before.

这实际上从 $\mathcal{D}_{e}$ 检索最相似的既往经验来构造分数。直观上，该方法度量当前观测是否与此前见过的内容相似。

Finally, to classify whether an observation should be treated as nominal or anomalous, we can calibrate a threshold $\tau\in\mathbb{R}$ as the $\alpha\in(0,1)$ quantile of the nominal prior experiences,

最后，为将观测分类为标称或异常，可将阈值 $\tau\in\mathbb{R}$ 校准为标称既往经验的 $\alpha\in(0,1)$ 分位数，

$$\tau=\inf\bigg\{q\in\mathbb{R}\ :\ \frac{|{\bm{e}_{i}\in\mathcal{D}_{e}:\bm{s}(\bm{e}_{i};\mathcal{D}_{e}\setminus\{\bm{e}_{i}\})\leq q}|}{N}\geq\alpha\bigg\}, \tag{2}$$

i.e., the smallest value of $q$ that upper bounds at least $\alpha N$ nominal samples. Note that for nominal embeddings, we must compute the anomaly score $\bm{s}$ in a leave-one-out fashion, since $\bm{s}(\bm{e}_{i};\mathcal{D}_{e})=-1$ for $\bm{e}_{i}\in\mathcal{D}_{e}$. Determining the threshold $\tau$ using empirical quantiles as in Eq. 2 is a standard approach [39], but could be extended in future work to make precise guarantees on false positive or negative rates using recent results in conformal prediction [3, 30].

即至少上界 $\alpha N$ 个标称样本的最小 $q$。注意对标称嵌入须以留一法计算异常分数 $\bm{s}$，因为对 $\bm{e}_{i}\in\mathcal{D}_{e}$ 有 $\bm{s}(\bm{e}_{i};\mathcal{D}_{e})=-1$。用经验分位数如式 2 确定阈值 $\tau$ 是标准做法 [39]，未来可用共形预测近期结果 [3, 30] 扩展以给出假阳性/假阴性率的精确保证。

Slow Generative Reasoning: Once we detect an anomaly, we trigger the autoregressive generation of an LLM to generate a zero-shot assessment of whether we need to engage any of the interventions associated with the recovery sets $\mathcal{X}_{R}^{1},\dots,\mathcal{X}_{R}^{d}$ (Section III) to maintain the safety of the system. The value of this approach is that the LLM’s internet-scale pretraining data allows it to generate outputs that resemble the generalist common sense reasoning that a human operator is likely to suggest, as a result, making superior decisions on OOD examples, on which existing task-specific learning algorithms are notoriously unreliable.

慢生成推理： 一旦检测到异常，我们触发 LLM 自回归生成，对是否需要启用与恢复集 $\mathcal{X}_{R}^{1},\dots,\mathcal{X}_{R}^{d}$（第 III 节）相关的干预以维持系统安全做零样本评估。该方法的价值在于：LLM 的互联网规模预训练使其输出类似人类操作员可能给出的通才常识推理，从而在现有任务专用学习算法往往不可靠的 OOD 样本上做出更优决策。

To do so, we follow [12] in using a VLM to convert the robot’s current visual observation into a text description of the environment. We simply encode this scene description into a prompt that provides context on the monitoring task, as illustrated in Fig. 3. We then parse the resulting output string to yield a classification $y\in\{0,1,\dots d\}$ on whether the anomaly does not present a hazard and the system can continue it’s nominal operation ($y=0$), or whether we should engage intervention $y\in\{1,\dots,d\}$ and steer the state into recovery set $\mathcal{X}_{R}^{y}$. As we illustrate in our experiments, the recovery sets naturally correspond to high-level behaviors (e.g., landing in a field), which facilitates prompt design. We use the shorthand $\bm{w}(\bm{o}_{t},\mathcal{Y})$ to denote the output of the slow reasoner when given observation $\bm{o}_{t}$ and a (sub)set of intervention strategies $\mathcal{Y}\subseteq\{1,\dots,d\}$.

为此我们遵循 [12]，用视觉语言模型把机器人当前视觉观测转为环境文本描述，再编码进提供监视任务上下文的提示（如图 3）。随后解析输出字符串，得到分类 $y\in\{0,1,\dots d\}$：异常不构成危害、系统可继续标称运行（$y=0$），或应启用干预 $y\in\{1,\dots,d\}$ 并将状态导入恢复集 $\mathcal{X}_{R}^{y}$。如实验所示，恢复集自然对应高层行为（如在田野着陆），便于提示设计。记 $\bm{w}(\bm{o}_{t},\mathcal{Y})$ 为慢推理器在给定观测 $\bm{o}_{t}$ 与干预策略（子）集 $\mathcal{Y}\subseteq\{1,\dots,d\}$ 时的输出。

Whether inference is run onboard or the model is queried remotely over unreliable networks in the cloud, we must account for the latency that autoregressive reasoning introduces. For example, a fast moving vehicle may collide with an anomalous obstacle if it’s reaction time is too slow. Therefore, we account for the LLM’s compute latency by assuming that it takes at most $K\in\mathbb{N}_{>0}$ timesteps to receive the output string from the slow reasoner. It is usually straightforward to identify the value of $K$ in practice, since we prompt the model to adhere to a strict output template that tends to stabilize the length of the output generations. Alternatively, as we describe in Section V-C and Section -I, a simple field-test can be sufficient to identify an upper bound on typical network latency.

无论推理在机载运行还是经不可靠网络远程查询云端模型，都必须考虑自回归推理引入的时延——例如反应过慢时，快速移动车辆可能撞上异常障碍。因此我们假设最多 $K\in\mathbb{N}_{>0}$ 个时间步可收到慢推理器输出字符串。实践中识别 $K$ 通常直接：我们提示模型遵守严格输出模板，往往使输出长度稳定；或者如第 V-C 节与第 -I 节所述，简单现场测试即可识别典型网络时延上界。

IV-B Planning a Tree of Recovery Trajectories

IV-B 规划恢复轨迹树

We control the robot’s dynamics Eq. 1 in state-feedback using a receding horizon control strategy that 1) minimizes the nominal control objective along a horizon of $T>K$ timesteps, while 2) maintaining a set of $d$ recovery trajectories that each reach one of the respective recovery sets $\mathcal{X}_{R}^{i}$ within the horizon $T$. The goal of this approach is to ensure that the high-level safety interventions provided to the slow reasoner can be executed. Additionally, it is essential that these options remain feasible throughout the $K$ time steps it takes the monitor to decide on the most appropriate choice. Otherwise, a fast moving robot may, for example, no longer be able to stop in time to avoid a collision. To this end, we solve the following finite-time optimal control problem online, which maintains a consensus between the recovery trajectories for $K$ timesteps:

我们用滚动时域控制以状态反馈控制式 1 的动力学：1) 在 $T>K$ 步时域上最小化标称控制目标；2) 维护 $d$ 条恢复轨迹，各自在时域 $T$ 内到达对应恢复集 $\mathcal{X}_{R}^{i}$。目标是确保提供给慢推理器的高层安全干预可被执行，且在监视器决策的 $K$ 步内保持可行——否则快速机器人可能来不及停下避碰。为此我们在线求解如下有限时域最优控制问题，使恢复轨迹在 $K$ 步内保持一致：

$$\displaystyle J_{t}(\mathcal{Y},K,T)= \ \displaystyle\begin{aligned} \minimize_{{\bm{x}{t:t+T+1|t}^{i},\bm{u}{t:t+T|t}^{i}}{i\in\mathcal{Y}\cup{0}}}&\kern 5.0ptC(\bm{x}{t:t+T+1|t}^{0},\bm{u}{t:t+T|t}^{0})\
\subjectto\kern 5.0pt&\bm{x}{t+k+1|t}^{i}=\bm{f}(\bm{x}{t+k|t}^{i},\bm{u}{t+k|t}^{i})\
&\bm{u}{t+k|t}^{i}\in\mathcal{U}\quad\bm{x}{t+k|t}^{i}\in\mathcal{X}\
&\bm{x}{t|t}^{i}=\bm{x}{t}\
&\bm{x}{t+T+1|t}^{i}\in\mathcal{X}{R}^{i}\quad\forall i\in\mathcal{Y}\
&\bm{u}{t|t}^{i}=\bm{u}{t|t}^{0}\quad\forall i\in\mathcal{Y}\
&\bm{u}{t:t+K|t}^{i}=\bm{u}{t:t+K|t}^{j}\ \forall i,j\in\mathcal{Y}\end{aligned} \tag{3}$$

Here, the notation $\bm{x}^{i}_{t+k|t}$ indicates the predicted value of variable $\bm{x}$ at time $t+k$ computed at time $t$ for each trajectory $i\in\mathcal{Y}\cup\{0\}$. The MPC in Section IV-B optimizes a set of $|\mathcal{Y}|+1$ trajectories. The first corresponds to a nominal trajectory $\bm{x}_{t:t+T+1|t}^{0}$ plan that minimizes the control objective and a set of $|\mathcal{Y}|$ recovery trajectories that each reach their respective recovery set $\mathcal{X}_{R}^{i}$ within $T$ timesteps. In addition, the MPC problem Section IV-B includes two consensus constraints, one associated with the fast anomaly detector and the other with the slow reasoner. First, by fixing consensus along the first input of the nominal trajectory and all the recovery trajectories, we ensure that the set of feasible interventions is non-empty during nominal operation. The second fixes consensus for $K$ timesteps along the set of recovery trajectories, in effect generating a branching tree of recovery trajectories. If we then use the fast anomaly detector to both trigger execution of the first $K$ actions of the recovery trajectories and the slower reasoning, we ensure that the options we provide to the slow reasoner are still available when it returns its output.

记号 $\bm{x}^{i}_{t+k|t}$ 表示在时刻 $t$ 对轨迹 $i\in\mathcal{Y}\cup\{0\}$ 预测的 $t+k$ 时刻变量 $\bm{x}$。第 IV-B 节 MPC 优化 $|\mathcal{Y}|+1$ 条轨迹：第一条为最小化控制目标的标称轨迹 $\bm{x}_{t:t+T+1|t}^{0}$，另有 $|\mathcal{Y}|$ 条在 $T$ 步内到达各自恢复集 $\mathcal{X}_{R}^{i}$ 的恢复轨迹。此外含两条一致性约束，分别关联快异常检测器与慢推理器：其一固定标称轨迹与全部恢复轨迹的首个输入一致，确保标称运行时可行干预集非空；其二固定恢复轨迹集在 $K$ 步内一致，从而生成分支的恢复轨迹树。若再用快异常检测器同时触发恢复轨迹前 $K$ 步执行与慢推理，即可保证提供给慢推理器的选项在其返回输出时仍可用。

We summarize this methodology in Algorithm 1, which guarantees that we reach the recovery set chosen by the slow reasoner within at most $T+1$ timesteps after detecting an anomaly:

我们将该方法总结为算法 1，保证检测到异常后至多 $T+1$ 步内到达慢推理器所选恢复集：

Input: State $\bm{x}_{0}$ such that Section IV-B is feasible, fast anomaly detector $\bm{h}$, slow reasoner $\bm{w}$ with latency $\leq K$.
$t_{\mathrm{anom}}\leftarrow\emptyset$ 1
for $t=0,1,2,\dots$ do 2
Observe $\bm{x}_{t},\bm{o}_{t}$ 3
if $t_{\mathrm{anom}}=\emptyset$ or $\bm{w}(\bm{o}_{t_{\mathrm{anom}}},\mathcal{Y}_{t_{\mathrm{anom}}})=0$ then 4
5
Choose $\mathcal{Y}_{t}\subset\{1,\dots d\}$ s.t. Section IV-B is feasible with arguments $\mathcal{Y}_{t}$, consensus horizon $K$ 6
Solve Section IV-B with $J_{t}(\mathcal{Y}_{t},K,T)$ 7
if $\bm{h}(\bm{o}_{t})=\mathrm{True}$ then 8
$t_{\mathrm{anom}}\leftarrow t$ 9
$\bm{w}(\bm{o}_{t},\mathcal{Y}_{t})\mathrm{.start()}$ end if 10
11
Apply optimal control input $\bm{u}^{\star,0}_{t|t}$ end if 12
else if $t_{\mathrm{anom}}\neq\emptyset$ and not $\bm{w}(\bm{o}_{t_{\mathrm{anom}}},\mathcal{Y}_{t_{\mathrm{anom}}}){\mathrm{.done()}}$ then 13
14
Set $k\leftarrow t-t_{\mathrm{anom}}$ 15
Solve Section IV-B with $J_{t}(\mathcal{Y}_{t_{\mathrm{anom}}},K-k,T-k)$ 16
Apply optimal control input $\bm{u}^{\star,0}_{t|t}$ 17
end if 18
else 19
Set $k\leftarrow t-t_{\mathrm{anom}}$ 20
Solve Section IV-B with $J_{t}(\{\bm{w}(\bm{o}_{t_{\mathrm{anom}}},\mathcal{Y}_{t_{\mathrm{anom}}})\},0,\max\{0,T-k\})$ 21
Apply optimal control input $\bm{u}^{\star,y}_{t|t}$, where $y=\bm{w}(\bm{o}_{t_{\mathrm{anom}}},\mathcal{Y}_{t_{\mathrm{anom}}})$ end if 22
end for 23

Algorithm 1 AESOP

输入： 使第 IV-B 节可行的状态 $\bm{x}_{0}$，快异常检测器 $\bm{h}$，时延 $\leq K$ 的慢推理器 $\bm{w}$。
$t_{\mathrm{anom}}\leftarrow\emptyset$ 1
for $t=0,1,2,\dots$ do 2
观测 $\bm{x}_{t},\bm{o}_{t}$ 3
if $t_{\mathrm{anom}}=\emptyset$ 或 $\bm{w}(\bm{o}_{t_{\mathrm{anom}}},\mathcal{Y}_{t_{\mathrm{anom}}})=0$ then 4
选择使第 IV-B 节在参数 $\mathcal{Y}_{t}$、一致时域 $K$ 下可行的 $\mathcal{Y}_{t}\subset\{1,\dots d\}$ 6
用 $J_{t}(\mathcal{Y}_{t},K,T)$ 求解第 IV-B 节 7
if $\bm{h}(\bm{o}_{t})=\mathrm{True}$ then 8
$t_{\mathrm{anom}}\leftarrow t$ 9
$\bm{w}(\bm{o}_{t},\mathcal{Y}_{t})\mathrm{.start()}$ 10
施加最优控制 $\bm{u}^{\star,0}_{t|t}$ 12
else if $t_{\mathrm{anom}}\neq\emptyset$ 且 $\bm{w}$ 未完成 then 13
设 $k\leftarrow t-t_{\mathrm{anom}}$ 15
用 $J_{t}(\mathcal{Y}_{t_{\mathrm{anom}}},K-k,T-k)$ 求解 16
施加 $\bm{u}^{\star,0}_{t|t}$ 17
else 19
设 $k\leftarrow t-t_{\mathrm{anom}}$ 20
用 $J_{t}(\{\bm{w}(\bm{o}_{t_{\mathrm{anom}}},\mathcal{Y}_{t_{\mathrm{anom}}})\},0,\max\{0,T-k\})$ 求解 21
施加 $\bm{u}^{\star,y}_{t|t}$，其中 $y=\bm{w}(\bm{o}_{t_{\mathrm{anom}}},\mathcal{Y}_{t_{\mathrm{anom}}})$ 22
end for 23

算法 1 AESOP

Theorem 1.

Suppose that at $t=0$, the MPC in Section IV-B is feasible for some set of recovery strategies $\mathcal{Y}\subset\{1,\dots,d\}$, i.e., that $J_{0}(\mathcal{Y},K,T)<\infty$. Then, the closed-loop system formed by Eq. 1 and Algorithm 1 ensures the following: 1) We satisfy state and input constraints $\bm{x}_{t}\in\mathcal{X}$, $\bm{u}_{t}\in\mathcal{U}$ for all $t\geq 0$. 2) At any time $t\geq 0$, there always exists at least one safety intervention $y\in\{1,\dots,d\}$ for which the MPC Section IV-B is feasible. 3) If the slow reasoner $\bm{w}$, triggered at some time $t_{\mathrm{anom}}>0$, chooses an intervention $y\in\{1,\dots,d\}$, then for all $t\geq t_{\mathrm{anom}}+T+1$ it holds that $\bm{x}_{t}\in\mathcal{X}_{R}^{y}$.

定理 1。

设 $t=0$ 时，对某恢复策略集 $\mathcal{Y}\subset\{1,\dots,d\}$，第 IV-B 节 MPC 可行，即 $J_{0}(\mathcal{Y},K,T)<\infty$。则式 1 与算法 1 构成的闭环系统保证：1) 对所有 $t\geq 0$ 满足状态与输入约束 $\bm{x}_{t}\in\mathcal{X}$、$\bm{u}_{t}\in\mathcal{U}$；2) 任意 $t\geq 0$ 至少存在一种安全干预 $y\in\{1,\dots,d\}$ 使第 IV-B 节 MPC 可行；3) 若在某 $t_{\mathrm{anom}}>0$ 触发的慢推理器 $\bm{w}$ 选择干预 $y\in\{1,\dots,d\}$，则对所有 $t\geq t_{\mathrm{anom}}+T+1$ 有 $\bm{x}_{t}\in\mathcal{X}_{R}^{y}$。

Proof.

See Section -H. ∎

证明。

见第 -H 节。∎

The emergent behavior of Algorithm 1 is that once the fast anomaly detector issues a warning regarding an unusual observation, the robot will balance progress along the nominal trajectory and jointly maintaining dynamic feasibility of the fallback options available at $t_{\mathrm{anom}}$. This generally leads the robot to slow down to preserve its options, thereby providing the slow reasoner with time to think. Upon reaching a decision from the slow reasoner, the robot either transitions back to nominal operations, if the observation is not hazardous, or engages the selected safety intervention and commits the robot to the associated recovery set. While this scheme does not explicitly ensure that multiple strategies remain available to the robot throughout nominal operation (though at least one will remain so), we find in a practical setting (see Section V-B, Section V-C) that multiple simple hand-designed intervention strategies remain persistently feasible. Still, methods for dynamically identifying and selecting recovery regions present an exciting avenue for future work.

算法 1 的涌现行为是：一旦快异常检测器对异常观测发出警告，机器人会在沿标称轨迹推进与联合维持 $t_{\mathrm{anom}}$ 时可用回退选项的动态可行性之间取得平衡；这通常使机器人减速以保留选项，从而给慢推理器思考时间。慢推理器做出决策后，若观测无危害则回到标称运行，否则启用所选安全干预并承诺对应恢复集。该方案未显式保证标称运行全程有多种策略可用（但至少一种始终可用）；实践中（第 V-B、V-C 节）多种简单手工干预策略可持久可行。动态识别与选择恢复区域仍是有吸引力的未来方向。

V Experiments

V 实验

Having outlined our approach, we conduct a series of experiments to test the following five hypothesis:

在概述方法后，我们设计一系列实验检验以下五个假设：

By quantifying semantic differences of observations with respect to the prior experience of a system, our fast embedding-based anomaly detector performs favorably to generative reasoning-based approaches.

通过量化观测相对系统既往经验的语义差异，我们的快速嵌入异常检测器相对基于生成推理的方法表现更优。

Embedding-based anomaly detection does not necessitate the use of high-capacity generative models; small models incurring marginal costs can be used.

基于嵌入的异常检测不必使用高容量生成模型；可用成本边际的小模型。

Once an anomaly is detected, generative reasoning approaches can effectively deduce whether the anomaly warrants enacting safety-preserving interventions.

一旦检测到异常，生成推理方法能有效推断该异常是否值得实施保全安全干预。

Our full approach, which unifies embedding-based anomaly detection and generative reasoning-based anomaly assessment, can be integrated in a broader robotics stack for real-time control of an agile system.

将基于嵌入的异常检测与基于生成推理的异常评估统一的完整方法，可接入更广的机器人栈，对敏捷系统做实时控制。

Additional forms of embeddings, including those from vision and multi-modal models, offer a promising future avenue for end-to-end anomaly detection.

包括视觉与多模态模型在内的其他嵌入形式，为端到端异常检测提供有前景的未来方向。

Experiment Rationale: We run four main experiments. The first experiment (Section V-A) tests the performance of our fast anomaly detector in three synthetic (i.e., text-based) robotic environments. We then evaluate the slow generative reasoner for the assessment of detected anomalies on two of these environments. The second experiment (Section V-B) evaluates our full approach (integrating the runtime monitor with the MPC fallback planner) in a simulation of real-time control of an agile drone system. The third is a full-stack experiment on real quadrotor hardware, including a timing breakdown for each component in our approach running on a Jetson AGX Orin module, thereby demonstrating viability for hardware deployment. The fourth experiment evaluates whether our runtime monitor transfers to a realistic, semantically rich self-driving environment, where we investigate the use of both language and multi-modal embeddings for anomaly detection.

实验理据： 我们做四项主实验。第一项（第 V-A 节）在三个合成（文本）机器人环境中测试快异常检测器，并在其中两个环境评估慢生成推理器对已检测异常的评估。第二项（第 V-B 节）在敏捷无人机实时控制仿真中评估完整方法（运行时监视器与 MPC 回退规划器集成）。第三项为真机四旋翼全栈实验，含在 Jetson AGX Orin 上各组件时序分解，展示硬件部署可行性。第四项评估运行时监视器能否迁移到语义丰富的真实感自动驾驶环境，并考察语言与多模态嵌入用于异常检测。

All code used in our experiments, including scripts to generate the synthetic datasets and prompt templates, can be found through our project page at https://sites.google.com/view/aesop-llm. In addition, we provide a brief description of our prompting strategy in Section -G.

实验全部代码（含合成数据生成脚本与提示模板）见项目页 https://sites.google.com/view/aesop-llm。提示策略简述见第 -G 节。

V-A Synthetics—Manipulation, Autonomous Vehicles, VTOL

V-A 合成实验——操作、自动驾驶、VTOL

We construct three synthetic domains to support our analysis: a Warehouse Manipulator domain, an Autonomous Vehicle domain, and a Vertical Take-off and Landing (VTOL) Aircraft domain. Each domain consists of scenarios in which a notionally mature autonomous robot may (or may not) encounter a safety concerning observation during its typical operations. The robots’ observations take the form of a collection of concepts, which we define as one or more objects and their semantic relationships. For example, in the VTOL domain, both “ice” and “helipad” represent concepts of a single object class, whereas “icy helipad” represents a third concept formed by their conjunction. We follow definition of anomalies given in Section IV-A: observations that, in the context of a given task, deviate from the robots’ nominal experiences. Thus, anomalies may not pose safety risks but must still be identified for further analysis.

我们构造三个合成域：仓储操作臂、自动驾驶汽车、垂直起降（VTOL）飞行器。各域包含名义成熟自主机器人在典型运行中可能（或不）遭遇安全相关观测的场景。机器人观测由一组概念组成，即一个或多个物体及其语义关系——例如 VTOL 域中“冰”“直升机坪”各为单类物体概念，“结冰直升机坪”为其合取形成的第三概念。我们沿用第 IV-A 节的异常定义：在给定任务上下文中偏离机器人标称经验的观测；异常未必构成安全风险，但仍须识别以供进一步分析。

Fig. 2: Embedding-based (fast) anomaly detection results for the manipulation, autonomous vehicle, and VTOL domains. The top row of figures plot anomaly detection accuracy as a function of experiences sampled IID from the respective domain datasets. The bottom row of figures plot accuracy as a function of the concepts sampled from the respective domain datasets. We use top-5 scoring with the anomaly detection threshold set at the $95$-th quantile Eq. 2 of the scores in the sampled data.

图 2：操作、自动驾驶与 VTOL 域上基于嵌入（快）异常检测结果。上行：异常检测准确率相对从各域数据集 IID 采样的经验数；下行：准确率相对从各域采样的概念数。使用 top-5 打分，异常检测阈值设为采样数据分数的 $95$ 分位数（式 2）。

We briefly describe the synthetic domains below:

合成域简述如下：

• • Warehouse Manipulator (WM): A mobile WM robot performs the task of “sorting objects on a conveyor belt.” Observations contain concepts sampled from a predefined set of conveyor belt (e.g., a package) and surrounding environment (e.g., a storage shelf) objects. Anomalies consist of hazardous objects on the conveyor belt (e.g., a leaking bleach bottle) or object combinations in the surrounding environment (e.g., two forklifts in collision). The dataset contains $1551$ scenarios.

• • 仓储操作臂（WM）： 移动 WM 机器人执行“在传送带上分拣物体”。观测含从预定义传送带物体（如包裹）与周围环境物体（如货架）采样的概念。异常含传送带上的危险物体（如泄漏漂白剂瓶）或周围环境中的物体组合（如两辆叉车相撞）。数据集含 $1551$ 场景。

• • Autonomous Vehicle (AV): An AV operating as a taxi service performs the task of “driving to a set destination.” Observations contain concepts sampled from a predefined set of task-relevant (e.g., a car, bus, or traffic light) and task-irrelevant (e.g., an airplane in the sky) objects. Anomalies consist of unusual task-relevant objects (e.g., a blank speed limit sign) or combinations (e.g., a traffic light on a truck). The dataset contains $840$ scenarios.

• • 自动驾驶汽车（AV）： 作为出租车运行的 AV 执行“驶向设定目的地”。观测含任务相关（如汽车、公交、红绿灯）与任务无关（如空中飞机）物体概念。异常含异常任务相关物体（如空白限速牌）或组合（如卡车上的红绿灯）。数据集含 $840$ 场景。

• • Vertical Take-off and Landing (VTOL): A VTOL aircraft operating as an urban air taxi performs one of two tasks: “flying toward a set destination” or “landing on a designated building.” Observations contain concepts sampled from a predefined set of flying objects, landing zones, and ground regions. Anomalies consist of unanticipated flying objects (e.g., a large swarming flock of birds) and/or landing zones (e.g., a building rooftop on fire). The dataset contains $18400$ scenarios.

• • 垂直起降（VTOL）： 作为城市空中出租车的 VTOL 执行“飞向设定目的地”或“降落在指定建筑”之一。观测含飞行物体、着陆区与地面区域概念。异常含未预期飞行物体（如大群鸟群）和/或着陆区（如着火屋顶）。数据集含 $18400$ 场景。

The synthetic domains vary in terms of size and complexity, with WM being the simplest, and VTOL being the most challenging. Complexity is determined by extent to which anomalous concepts differ from the nominal experiences of the robot [^3].

合成域在规模与复杂度上不同：WM 最简单，VTOL 最具挑战。复杂度由异常概念相对机器人标称经验的差异程度决定。[^3]

V-A1 Fast Reasoning for Anomaly Detection (H1, H2)

The role of the anomaly detector is to analyze the robot’s observations and identify whether, due to the presence of an atypical object or concept, an observation qualifies as an anomalous.

V-A1 用于异常检测的快推理（H1，H2）

异常检测器的角色是分析机器人观测，识别因非典型物体或概念而使观测构成异常。

Methods: We evaluate our fast anomaly detector with nine language models, varying in size and function: BERT-base (110M) and BERT-large (336M) uncased [10], Sentence Transformer MPNet (110M; BERT-base architecture trained for embeddings) [47, 38], completion and instruction-tuned Llama 2 models (7B) [51], Mistral (7.11B) [21, 52], and three OpenAI embedding models (parameters not disclosed). The choice of score function $\bm{s}(\bm{e}_{t};\mathcal{D}_{e})$ (e.g., cosine similarity, top-$k$ scoring, Mahalanobis distance) did not yield significant performance variation. Thus, we only report results for top-$5$ scoring and refer to Section -D for extended results.

方法： 我们用九种规模与功能不同的语言模型评估快异常检测器：BERT-base（110M）与 BERT-large（336M）uncased [10]、Sentence Transformer MPNet（110M；面向嵌入训练的 BERT-base 结构） [47, 38]、补全与指令微调的 Llama 2（7B） [51]、Mistral（7.11B） [21, 52]，以及三种 OpenAI 嵌入模型（参数未公开）。分数函数 $\bm{s}(\bm{e}_{t};\mathcal{D}_{e})$ 的选择（余弦相似度、top-$k$、马氏距离等）未带来显著性能差异，故仅报告 top-$5$ 打分，扩展结果见第 -D 节。

Baselines: Our baselines consist of generative reasoning with GPT-4, queried to classify the robot’s observation as “nominal” or “anomalous.” We consider a two variants: single-token (ST) prediction and chain-of-thought reasoning (CoT). For CoT, GPT-4 first reasons over each concept in the observation before outputting an overall anomaly classification.

基线： 基线为用 GPT-4 做生成推理，查询以将机器人观测分类为“标称”或“异常”。考虑两种变体：单 token（ST）预测与思维链推理（CoT）。CoT 下 GPT-4 先对观测中每个概念推理，再输出整体异常分类。

We use prompts with identical prefixes across all methods, making minor modifications to, e.g., elicit CoT reasoning. For metrics we report accuracy, setting the detection threshold at the $95$-th quantile (Eq. 2) of the scores in the nominal dataset.

各方法使用相同前缀提示，仅做如引出 CoT 等小幅修改。指标报告准确率，检测阈值设为标称数据集分数的 $95$ 分位数（式 2）。

Results & Analysis: The results are shown in Fig. 2. The first row shows the accuracy of our fast anomaly detectors with increasing sample size (i.e., the size $N$ of the embedding cache $\mathcal{D}_{e}=\{\bm{e}_{i}\}_{i=1}^{N}$) drawn IID from the full dataset of nominal observations $\mathcal{D}_{\mathrm{nom}}$. The second row is identical to the first, except, showing accuracy with increasing percent of concept coverage; that is, the percent of nominal concepts contained in the embedding cache $\mathcal{D}_{e}$ used to construct our detector.

结果与分析： 结果见图 2。第一行展示快异常检测器准确率随样本规模（即从完整标称观测集 $\mathcal{D}_{\mathrm{nom}}$ IID 抽取的嵌入缓存 $\mathcal{D}_{e}$ 大小 $N$）增加的变化；第二行相同，但横轴为概念覆盖百分比，即用于构造检测器的嵌入缓存所含标称概念占比。

Comparing the top and bottom rows for each domain, we observe that performance increases logarithmically with relative sample size, but only linearly with increasing concept coverage. This indicates that our fast anomaly detector scales favorably with the diversity of nominal concepts represented in the robot’s experience as opposed to the shear scale of experience. Provided with sufficient concept coverage, our anomaly detector clearly outperforms the single-token and CoT generative reasoning baselines with GPT-4, validating our first hypothesis H1.

比较各域上下行可见：性能随相对样本规模对数增长，但随概念覆盖近似线性增长。这表明快异常检测器更受益于机器人经验中标称概念的多样性，而非经验体量本身。在概念覆盖充足时，我们的异常检测器明显优于 GPT-4 的单 token 与 CoT 生成推理基线，验证假设 H1。

Comparing among language models, we find that the performance of models does not strictly correlate with their size, but rather, depends on their training data and strategy. For example, Sentence Transformer MPNet, a 110M parameter model trained for embeddings, often outperforms BERT Large (336M) and the OpenAI embedding models, and even performs comparatively to the Llama 2 (7B) and Mistral (7.11B) models. This validates our second hypothesis H2.

比较各语言模型可见，性能并不严格随规模上升，而取决于训练数据与策略。例如面向嵌入训练的 110M Sentence Transformer MPNet 常优于 BERT Large（336M）与 OpenAI 嵌入模型，甚至可与 Llama 2（7B）、Mistral（7.11B）媲美。这验证假设 H2。

The key advantage of our anomaly detectors is that they ground the analysis of observations in the embedding space of previously observed concepts. It is unreasonable to attempt to ground the generative baselines in such a way due to their limited context windows, among other challenges (e.g., recency bias [59]). Moreover, we see that GPT-4’s performance gradually decreases as the complexity of the environments increase (e.g., from AV to VTOL), and as such, we may expect further performance drops in real-world settings.

异常检测器的关键优势是在既往观测概念的嵌入空间中 grounding 观测分析。因上下文窗口等限制（如近因偏差 [59]），难以对生成基线做同样 grounding。此外 GPT-4 性能随环境复杂度升高（如从 AV 到 VTOL）逐渐下降，真实场景中可能进一步下降。

V-A2 Slow Reasoning for Anomaly Assessment (H3)

Once an anomaly has been identified, it is the role of the slow reasoner to assess whether or not the anomaly warrants the enactment of safety-preserving interventions. Recall, we use LLMs for their generalist knowledge—acquired through internet-scale pretraining—to infer the need for a fallback on an observation identified as dissimilar from the robot’s previous experiences.

V-A2 用于异常评估的慢推理（H3）

一旦识别异常，慢推理器负责评估是否值得实施保全安全干预。我们用 LLM，因其经互联网规模预训练获得的通才知识，可在观测被判定为不同于既往经验时推断是否需要回退。

Method	TPR	FPR	Accuracy
Llama 2 (7B)	0.52	0.46	0.52
GPT-3.5 Turbo	0.97	0.54	0.73
GPT-3.5 Turbo CoT	0.82	0.28	0.77
GPT-4	0.65	0.06	0.79
GPT-4 CoT	0.89	0.10	0.90

TABLE I: Slow Generative Reasoning for Anomaly Assessment in VTOL. Best scores are bolded; second best are underlined.

Method	TPR	FPR	Accuracy
Llama 2 (7B)	0.52	0.46	0.52
GPT-3.5 Turbo	0.97	0.54	0.73
GPT-3.5 Turbo CoT	0.82	0.28	0.77
GPT-4	0.65	0.06	0.79
GPT-4 CoT	0.89	0.10	0.90

表 I：VTOL 中慢生成推理的异常评估。最优加粗，次优下划线。

Fig. 3: Closed-loop trajectory of a quadrotor using the AESOP algorithm. The figure represents a snapshot of the quadrotor at $t=2.5\mathrm{s}$: The trajectory until time $t$ is in black. The nominal trajectory plan is shown in blue, with a blue dot denoting the first consensus constraint in Section IV-B. The overlapping recovery trajectory plans, up to the consensus horizon corresponding to the LLM latency $K$, are in orange. The recovery trajectory plans deviate after $K$, shown in red, and they each reach their respective recovery region (in green). The blue text callout shows how the fast anomaly detector issues a warning and triggers the slow reasoner at $t=2.5\mathrm{s}$. The red callout shows the response from the slow reasoner, which the LLM returns within the $K$ consensus timesteps in the recovery plans.

图 3：使用 AESOP 算法的四旋翼闭环轨迹快照（$t=2.5\mathrm{s}$）。至时刻 $t$ 的轨迹为黑；标称轨迹计划为蓝，蓝点为第 IV-B 节首个一致性约束。与 LLM 时延 $K$ 对应的一致时域内重叠恢复轨迹为橙；之后恢复轨迹分叉为红，各自到达绿色恢复区域。蓝字标注显示快异常检测器在 $t=2.5\mathrm{s}$ 发出警告并触发慢推理器；红字标注为慢推理器响应，LLM 在恢复计划的 $K$ 个一致时间步内返回。

We evaluate the ability of LLMs to assess anomalies in the VTOL domain and predict one of two options: whether the VTOL should 1) “continue” it’s nominal operation or 2) “fallback,” enacting one of several listed safety-preserving interventions. To do so effectively, the LLM must differentiate between safety-redundant (e.g., a plane on a known flight path) and safety-concerning (e.g., a fighter jet) anomalies. For this experiment, we evaluate four LLMs: Llama 2 (7B) single-token prediction, GPT-3.5 Turbo CoT, GPT-4 single-token prediction, and GPT-4 CoT. As before, the CoT approach must first assess the safety risk each concept contained in the observation before outputting a fallback classification.

我们在 VTOL 域评估 LLM 评估异常并预测二选一的能力：1)“继续”标称运行，或 2)“回退”并启用所列保全安全干预之一。要有效做到，LLM 须区分安全冗余异常（如已知航路上的飞机）与安全相关异常（如战斗机）。本实验评估四种 LLM：Llama 2（7B）单 token、GPT-3.5 Turbo CoT、GPT-4 单 token、GPT-4 CoT。同前，CoT 须先评估观测中各概念的安全风险再输出回退分类。

We report true positive rate (TPR), false positive rate (FPR), and accuracy. A true positive corresponds to the LLM correctly engaging a fallback intervention on a safety-concerning anomaly, while a true negative corresponds to correctly dismissing a safety-redundant anomaly.

我们报告真阳性率（TPR）、假阳性率（FPR）与准确率。真阳性对应 LLM 对安全相关异常正确启用回退；真阴性对应正确驳回安全冗余异常。

The results are shown in Table I. We observe that in these out-of-distribution scenarios, model capacity is an important consideration, with the GPT variants clearly outperforming the Llama 2 (7B) baseline. As expected, we also find that CoT reasoning yields a notable improvement in overall classification accuracy (e.g., $11$% for GPT-4) and reduces the number of false positives. These findings are corroborated in the manipulation domain (Table VII). This validates our third hypothesis H3.

结果见表 I。在这些分布外场景中，模型容量很重要，GPT 变体明显优于 Llama 2（7B）基线。如预期，CoT 推理显著提升整体分类准确率（如 GPT-4 提升 $11$%），并减少假阳性。操作域结果（表 VII）也印证这一点。这验证假设 H3。

V-B Full Stack—Quadrotor Simulation(H4)

V-B 全栈——四旋翼仿真（H4）

	Naive MPC	FS-MPC [46]	AESOP
Successful Recovery Rate	15%	23%	100%

TABLE II: Percentage of trajectories where the quadrotor successfully recovered to the LLM’s choice of recovery region.

	Naive MPC	FS-MPC [46]	AESOP
Successful Recovery Rate	15%	23%	100%

表 II：四旋翼成功恢复到 LLM 所选恢复区域的轨迹百分比。

Here, we demonstrate the efficacy of our framework, from anomaly detection and LLM reasoning to closed-loop control with Algorithm 1, on an example of a quadrotor delivering a package. In this simulation, the quadrotor’s task is to fly toward and subsequently land at a target location. To simulate a variety of safety hazards and instantiate the fast anomaly detector, we recycle the VTOL synthetic observations and show the quadrotor anomalous observations at random time intervals. In this simulation, we encode the location of two landing regions, a parking lot and an open field, as polytopic state constraints representing $\mathcal{X}_{R}^{1,2}$. To instantiate the MPC in Section IV-B, we use a fixed dynamics model discretized at a timestep of $\mathrm{dt}=.1\mathrm{s}$, with a planning horizon of $T=4\mathrm{s}$. We assume that the slow reasoning LLM may take up to $K=1.5\mathrm{s}$ to return an output, a number consistent with the latency of cloud querying GPT-4 reported in [50] using a conversational chat-based prompting approach.

这里在四旋翼投递包裹示例上展示从异常检测、LLM 推理到算法 1 闭环控制的框架效力。仿真中四旋翼任务是飞向目标位置并着陆。为模拟多种安全危害并实例化快异常检测器，我们复用 VTOL 合成观测，在随机时间间隔向四旋翼展示异常观测。编码停车场与开阔田野两个着陆区为多面体状态约束 $\mathcal{X}_{R}^{1,2}$。第 IV-B 节 MPC 使用 $\mathrm{dt}=.1\mathrm{s}$ 离散化固定动力学，规划时域 $T=4\mathrm{s}$。假设慢推理 LLM 最多 $K=1.5\mathrm{s}$ 返回输出，与 [50] 报告的云端查询 GPT-4（对话式提示）时延一致。

Fig. 4: Annotated depiction of our quadrotor hardware experiment. The quadrotor’s goal is to land on the red box. In the event of an anomaly, it can either recover by landing on the blue box, or by hovering within the designated holding zone.

图 4：四旋翼硬件实验注释示意。目标是降落在红盒上；异常时可恢复到蓝盒着陆，或在指定悬停区内悬停。

In Fig. 3, we show a snapshot of the closed-loop trajectory of the quadrotor as it flies towards the landing zone. Fig. 3 shows that the AESOP MPC Section IV-B plans two recovery trajectories that safely abort the control task and land the drone in their respective recovery regions, while still allowing the quadrotor to make progress towards its nominal objective. Furthermore, the recovery trajectories are aligned for the first $K$ timesteps, budgeting time for the LLM to output which, if any, of the recovery trajectories should be executed. The trajectories in Fig. 3 show that the drone descends and slows down during the first $K$ timesteps of the recovery trajectories, thereby explicitly budgeting time for the LLM to reason.

图 3 展示四旋翼飞向着陆区时的闭环轨迹快照：AESOP MPC 规划两条恢复轨迹，可安全中止控制任务并在各自恢复区着陆，同时仍允许朝标称目标推进。恢复轨迹在前 $K$ 步对齐，为 LLM 输出是否执行某条恢复轨迹预留时间；轨迹显示无人机在恢复轨迹前 $K$ 步下降并减速，显式为 LLM 推理预留时间。

While we only show a single example in Fig. 3 to illustrate the qualitative behavior of our method, we include additional plots in Section -E and videos of closed-loop trajectories on the project page to more extensively demonstrate our approach. Furthermore, Table II shows the results of a quantitative ablation over a set of $500$ scenarios and compares AESOP with 1) the fallback-planning method in [45] (which ignores the runtime monitor’s latency), and 2) a naive planner that only tries to compute a recovery plan post-hoc after detecting a dangerous event. We refer to Section -E for a detailed description of these experiments and baselines. While the FSMPC algorithm mildly improves over the naive baseline, Table II showcases the impact of accounting for the slow reasoner’s latency within the control design. Moreover, throughout the closed-loop trajectory in Fig. 3, the average speed of the quadrotor is around $2.5\mathrm{m/s}$, demonstrating that our framework allows for dynamic control of the robot while leveraging the slower LLM to improve safety in a reactive, real-time manner. This supports our fourth hypothesis H4.

图 3 仅展示定性行为一例；更多图见第 -E 节，闭环轨迹视频见项目页。表 II 在 $500$ 个场景上定量消融，比较 AESOP 与 1) [45] 的回退规划（忽略运行时监视器时延），2) 仅在检测到危险后事后计算恢复计划的朴素规划器。实验与基线详述见第 -E 节。FSMPC 相对朴素基线略有提升，表 II 凸显在控制设计中计入慢推理器时延的影响。图 3 闭环轨迹中四旋翼平均速度约 $2.5\mathrm{m/s}$，表明框架可在利用较慢 LLM 以反应式实时方式提升安全的同时做动态控制。支持假设 H4。

V-C Quadrotor Hardware Demonstration & Timing(H4)

V-C 四旋翼硬件演示与时序（H4）

Furthermore, we conduct hardware experiments with a physical quadrotor equipped with a downward facing camera (Intel Realsense D435). The quadrotor’s nominal goal is to land on a designated red box amidst a cluttered environment. As shown in Fig. 4, the ground is scattered with various objects, such as a bicycle tire, a soccer ball, and a drill, which are inconsequential to the landing task. Fig. 4 also shows the quadrotor’s two recovery strategies to avoid failures when e.g., another quadrotor has landed on the red box: 1) landing at an alternative landing site and 2) to hover in a designated holding zone.

我们还用配备下视相机（Intel Realsense D435）的物理四旋翼做硬件实验。标称目标是在杂乱环境中降落在指定红盒上。如图 4，地面散落自行车轮胎、足球、电钻等与着陆任务无关的物体。图 4 也展示异常时（如另一四旋翼已落在红盒上）的两种恢复策略：1) 备选点着陆；2) 在指定悬停区悬停。

In order to construct the embedding cache for the fast detector, we first record images by flying the drone in a circular pattern above the operational area with a nominal clutter of objects on the ground. We use the open vocabulary object detector OWL-ViT[32] to extract context-aware descriptions of visible objects (e.g.,“on the red box” or “on the ground”) from the image observations, which we then use to construct prompts for the embedding model (MPNet) and generative reasoner (GPT-3.5-Turbo).

为构造快检测器嵌入缓存，先让无人机在作业区上空圆周飞行，地面有标称杂物时记录图像。用开放词汇目标检测器 OWL-ViT[32] 从图像提取上下文感知的可见物体描述（如“在红盒上”“在地面上”），再构造嵌入模型（MPNet）与生成推理器（GPT-3.5-Turbo）的提示。

We evaluate our system on the following three scenarios. For more details on these scenarios, see Section -I and the videos on the project page.

我们在以下三种场景评估系统。细节见第 -I 节与项目页视频。

1. Nominal Operation: There are no obstructions on the red box, so the quadrotor lands normally despite the clutter.

1. 标称运行： 红盒无遮挡，尽管地面杂乱，四旋翼仍正常着陆。

2. Consequential Anomaly: We consider two variants of this scenario. In the first, another quadrotor has already landed on the red box, necessitating a diversion to the blue box for landing. In the second, other quadrotors occupy both red and blue boxes, necessitating a diversion to the holding zone.

2. 有后果异常： 两种变体。其一，另一四旋翼已落在红盒，须改降蓝盒；其二，红蓝盒均被占用，须改去悬停区。

3. Inconsequential Anomaly: A previously unseen object (specifically, a keyboard) on the ground triggers the fast anomaly detector, after which the LLM correctly decides to proceed with landing at the nominal site.

3. 无后果异常： 地面未见过的物体（键盘）触发快异常检测器，随后 LLM 正确决定继续在标称点着陆。

Component	Mean (s)	Standard Deviation (s)
MPC solve of Section IV-B	0.023	0.019
OWL-ViT	0.025	0.002
MPNet	0.028	0.005
Mistral	0.32	0.08
GPT-3-Turbo CoT	3.10	0.85
GPT-4 CoT	18.88	3.923

TABLE III: Inference times for the OWL-ViT object detector, LLM embedding models (MPNet, Mistral), and cloud-querying GPT-3/4 on a Jetson AGX Orin module.

Component	Mean (s)	Standard Deviation (s)
MPC solve of Section IV-B	0.023	0.019
OWL-ViT	0.025	0.002
MPNet	0.028	0.005
Mistral	0.32	0.08
GPT-3-Turbo CoT	3.10	0.85
GPT-4 CoT	18.88	3.923

表 III：Jetson AGX Orin 上 OWL-ViT、LLM 嵌入模型（MPNet、Mistral）与云端查询 GPT-3/4 的推理时间。

In addition, we evaluate the computational cost of our pipeline on our hardware platform, an Nvidia Jetson AGX Orin, which is designed for embedded systems like a quadrotor. Table III shows that the OWL-ViT detection parsing and MPNet embedding computation can jointly run at $18.8\mathrm{Hz}$. This ensures that AESOP (Algorithm 1) can comfortably operate at approximately $10\mathrm{Hz}$[^4]. We cloud query GPT-3.5-turbo with chain-of-thought prompting for the slow reasoning process, which, as shown in Table III, has a non-negligible latency. Together with the simulations in Section V-B, these hardware results demonstrate that our approach effectively leverages LLMs to improve robot reliability despite their inference costs, thereby validating our hypothesis (H4). We include further hardware results, detailing 1) an additional evaluation of the fast anomaly detector, 2) a analysis of LLM query latencies informing our choice of $K$, 3) implementation details of the MPC solver in Section -I.

我们还在面向嵌入式（如四旋翼）的 Nvidia Jetson AGX Orin 上评估流水线计算成本。表 III 显示 OWL-ViT 检测解析与 MPNet 嵌入合计可跑到 $18.8\mathrm{Hz}$，确保 AESOP（算法 1）可舒适运行在约 $10\mathrm{Hz}$。[^4]慢推理用思维链提示云端查询 GPT-3.5-turbo，表 III 显示时延不可忽略。结合第 V-B 节仿真，这些硬件结果说明方法能在 LLM 推理成本下有效提升机器人可靠性，验证假设（H4）。更多硬件结果（快异常检测器额外评估、指导 $K$ 选择的 LLM 查询时延分析、MPC 求解器实现细节）见第 -I 节。

V-D Ablation—Autonomous Vehicles Simulation

V-D 消融——自动驾驶仿真

We run ablations on self-driving scenarios curated in [12]. Here, CARLA was used to generate anomalous observations inspired by documented failure modes of self-driving perception systems[^5]. All anomalies in this domain require safety-preserving intervention on the nominal system’s operation. Thus, this experiment resembles the synthetic fast reasoning experiments (Section V-A1), but additionally considers high-dimensional RGB observations as inputs to the runtime monitors.

我们在 [12] 整理的自动驾驶场景上做消融。此处用 CARLA 生成受自动驾驶感知系统已记录失效模式启发的异常观测。[^5]该域全部异常都要求对标称系统运行做保全安全干预。因而本实验类似合成快推理实验（第 V-A1 节），但额外以高维 RGB 观测作为运行时监视器输入。

V-D1 End-to-End Reasoning for Anomaly Detection (H5)

Given the release of multi-modal models such as GPT-4V, we ablate performance differences between a two-step pipeline and a single-step pipeline. The two-step pipeline constructs a prompt consisting of detections from the open vocabulary OWL-ViT object detector [32] on CARLA images, whereas the single-step pipeline directly queries GPT-4V to output an anomaly classification based on the image observation. For each of these approaches, we consider both single-token and CoT reasoning.

V-D1 端到端异常检测推理（H5）

鉴于 GPT-4V 等多模态模型发布，我们消融两步与单步流水线的性能差异。两步流水线用开放词汇 OWL-ViT [32] 在 CARLA 图像上的检测构造提示；单步流水线直接查询 GPT-4V 基于图像观测输出异常分类。两种方法均考虑单 token 与 CoT 推理。

The results are presented in Table IV. In both text- and vision-based anomaly detection, we corroborate the notion that CoT reasoning facilitates more accurate responses to a complex task compared to single-token reasoning. Furthermore, the results of GPT-4V (relative to GPT-3/4) suggest that the anomaly detection task could be performed end-to-end.

结果见表 IV。在基于文本与视觉的异常检测中，均印证相对单 token，CoT 推理对复杂任务能给出更准确响应。GPT-4V（相对 GPT-3/4）结果也提示异常检测可端到端完成。

	Method	TPR	FPR	Bal. Accuracy
Text	GPT-4	0.74	0.19	0.78
GPT-3 CoT [12]	0.89	0.26	0.82
MPNet (Ours)	0.69	0.05	0.82
Mistral (Ours)	0.95	0.05	0.95
Vision	SCOD	0.40	0.06	0.67
Mahal.	0.40	0.13	0.64
GPT-4V	0.97	0.27	0.85
GPT-4V CoT	0.89	0.10	0.90
CLIP (Ours)	0.86	0.05	0.90
CLIP (Ours) Abl.	0.99	0.57	0.71

TABLE IV: CARLA Evaluation. Text and Vision-based Anomaly Detection.

	Method	TPR	FPR	Bal. Accuracy
Text	GPT-4	0.74	0.19	0.78
GPT-3 CoT [12]	0.89	0.26	0.82
MPNet (Ours)	0.69	0.05	0.82
Mistral (Ours)	0.95	0.05	0.95
Vision	SCOD	0.40	0.06	0.67
Mahal.	0.40	0.13	0.64
GPT-4V	0.97	0.27	0.85
GPT-4V CoT	0.89	0.10	0.90
CLIP (Ours)	0.86	0.05	0.90
CLIP (Ours) Abl.	0.99	0.57	0.71

表 IV：CARLA Evaluation. Text and Vision-based Anomaly Detection.

V-D2 End-to-end Embedding-Based Anomaly Detection (H5)

For the following embedding evaluations, we use the top performing language embedding models, MPNet (110M) and Mistral (7B), operating on the existing prompts which list detected objects parsed from the output of OWL-ViT. Additionally, we evaluate image embeddings from a ViT, specifically CLIP [35], that provides direct visual grounding for the task. As in Section V-A1, we use top-$5$ scoring across all methods.

V-D2 端到端基于嵌入的异常检测（H5）

后续嵌入评估使用表现最好的语言嵌入模型 MPNet（110M）与 Mistral（7B），作用于由 OWL-ViT 输出解析得到的检测物体列表提示。另评估来自 ViT——具体为 CLIP [35]——的图像嵌入，为任务提供直接视觉 grounding。同第 V-A1 节，各方法均用 top-$5$ 打分。

To ensure we fairly compare the expressiveness of purely language-based embeddings with vision-language embeddings, we report results using ground-truth object detections to construct the prompts for the text-based embedding models in Table IV, in the spirit of a more comprehensively engineered system with reliable object detection. We do so because, as noted in [12], the vision model suffers from a real-to-sim domain shift, and sometimes tends to, e.g., characterize nominal observations of stop signs as “images of stop signs”. We include an ablation showing the impact of vision errors in Section -F.

为公平比较纯语言嵌入与视觉–语言嵌入的表达力，表 IV 中文本嵌入模型的提示用真值物体检测构造，以贴近具备可靠物体检测的更完整工程系统。原因如 [12] 所述：视觉模型存在仿真到现实域偏移，有时会把标称停车标志观测描述成“停车标志的图像”。视觉误差影响的消融见第 -F 节。

First, we observe that language embeddings from the relatively small MPNet model are less capable at discerning anomalies among many nominal observations than Mistral, independent of whether OWL-ViT returns a correct scene description (see Section -F).

首先，相对较小的 MPNet 语言嵌入在大量标称观测中辨别异常的能力弱于 Mistral，且与 OWL-ViT 是否返回正确场景描述无关（见第 -F 节）。

Second, we surpass GPT-4 and GPT-4V with Mistral (roughly $64$x larger than MPNet). Besides suggesting that a smaller model’s language embeddings are less semantically rich and therefore less capable at capturing the presence of more subtle anomalies, we also show in Section -F that MPNet’s accuracy degrades as the number of objects in an observation increases, whereas Mistral’s accuracy remains consistent when the number of detections in an image varies.

其次，Mistral（约比 MPNet 大约 $64$ 倍）超过 GPT-4 与 GPT-4V。这不仅说明更小模型的语言嵌入语义不够丰富、更难捕捉细微异常，第 -F 节还显示：随观测中物体数增加，MPNet 准确率下降，而 Mistral 在检测数变化时准确率保持稳定。

Third, we note that the use of CLIP embeddings derived directly from the vehicle’s RGB observations also achieves high performance, comparable to GPT-4V CoT. This suggests that compute-intensive, CoT reasoning is not necessary to discern the anomalies directly from vision in semantically rich environments. However, we nuance this finding by noting that the observations in the CARLA dataset were constructed by driving a car along simulated routes and placing object assets that trick the vehicle into making unsafe decision along those routes [12]. This means that the routes appear both with and without anomalous objects and that episodes wherein the vehicle takes unsafe actions include both nominal and anomalous observations. When we change the calibration strategy to only construct the embedding cache with nominal observations from routes that never pass by anomalous objects (denoted by “Abl.” in Table IV), we find that CLIP’s FPR increases significantly. This is because CLIP embeddings contain a mix of visual and semantic features [23], and therefore the slight visual novelties in an unseen route are flagged as anomalous even though they are semantically uninteresting. As such, CLIP’s limitations are related to the SCOD [44] and Mahalanobis distance [26] baseline OOD detectors (taken from [12]) that wrap the AV’s base object detector (DETR [7]). In contrast, as we further ablate in Section -F, the two-stage detection approach is unaffected by differences in visual appearance. Despite this, and noting that further work should examine how to disentangle semantic and visual features to increase robustness, we argue that our preliminary findings on using multi-modal embeddings directly offers significant promise for streamlining the implementation of our framework. This validates our fifth and final hypothesis H5.

第三，直接从车辆 RGB 观测得到的 CLIP 嵌入也取得高表现，可与 GPT-4V CoT 媲美。这表明在语义丰富环境中，不必用计算密集的 CoT 推理即可直接从视觉辨别异常。但需注意：CARLA 数据集观测是沿仿真路线行驶并放置诱使车辆不安全决策的物体资产构成的 [12]，因而路线既有无异常物体版本，不安全动作的 episode 也同时含标称与异常观测。若将校准改为仅用从未经过异常物体的路线上的标称观测构造嵌入缓存（表 IV 中记为 “Abl.”），CLIP 的 FPR 显著上升——因 CLIP 嵌入混合视觉与语义特征 [23]，未见过路线上的轻微视觉新奇也会被标为异常，即便语义上无趣。因此 CLIP 的局限与包裹 AV 基础物体检测器（DETR [7]）的 SCOD [44]、马氏距离 [26] 等基线 OOD 检测器相关（取自 [12]）。相反，第 -F 节进一步消融显示两阶段检测不受视觉外观差异影响。尽管如此，并指出未来应研究如何解耦语义与视觉特征以提升鲁棒性，我们仍认为直接使用多模态嵌入的初步结果为简化框架实现提供了重要前景。这验证第五个也是最后一个假设 H5。

VI Conclusion and Outlook

VI 结论与展望

In this paper, we presented a runtime monitoring framework utilizing generalist foundation models to facilitate safe and real-time control of agile robotic systems faced with real-world anomalies. This is enabled through a reasoning hierarchy: a fast anomaly classifier querying similarity with the robot’s prior experiences in an LLM embedding space, and a slow generative reasoner assessing the safety implications of detected anomalies and selecting the appropriate mitigation strategy. These reasoners are interfaced with a new model predictive control strategy that maintains the feasibility of multiple safe recovery plans.

本文提出利用通才基础模型的运行时监视框架，以促进敏捷机器人系统在面对真实世界异常时的安全实时控制。该框架将监视拆为快慢两阶段：快阶段在嵌入空间检测相对标称经验的偏离；慢阶段用生成式 LLM 评估安全后果；并以计入慢推理时延的多应急 MPC 保证回退选项在决策返回前仍可行。

In extensive experiments, we demonstrate that a) embedding-based anomaly detection performs favorably to zero-shot generative reasoning with high-capacity LLMs, thanks in part to the grounding afforded by the prior embedding experience of the robot; b) embedding-based anomaly detection attains strong performance even when instantiated with small language models, allowing our method to run onboard computationally constrained robotic systems; c) dual-stage reasoning enables LLMs to operate in the real-time reactive control loop of an agile robot; d) alternative forms of embeddings, such as those obtained from vision-based foundation models, can be used to efficiently detect anomalies in high-dimensional observation spaces.

大量实验表明：a) 得益于既往经验提供的 grounding，基于嵌入的异常检测优于用高容量 LLM 做零样本生成推理；b) 小嵌入模型即可胜任；c) 生成推理能有效评估已检测异常是否值得干预；d) 完整方法可接入敏捷系统实时控制栈；e) 包括视觉与多模态模型在内的其他嵌入为端到端检测提供有前景方向。

As such, our work highlights the potential of LLMs and, more broadly, foundation models toward significant increases in the robustness of autonomous robots with respect to unpredictable and unusual out-of-distribution scenarios or tail events. Improving the performance and generality of our framework presents several promising avenues for future research. For example, the impact of LLM inference latencies could be reduced by devising methods to constrain generative reasoning to a fixed word budget, or by using intermediate generations to inform decision-making during the generation process. Further analysis is required on the correctness of fallback plans selected by the LLM, and whether fallbacks can be programmatically determined upon latency timeout. Finally, continual learning based on the delayed anomaly assessment of the generative reasoner could be used to avoid triggering the slow reasoner on non-safety-critical anomalies a second time.

因此，本工作凸显 LLM 乃至更广基础模型在显著提升自主机器人对不可预测、异常分布外场景鲁棒性方面的潜力，并表明其实时接入动态敏捷机器人系统在实践上已可行。未来工作包括解耦语义与视觉特征、动态选择恢复区域，以及进一步降低慢推理时延。

Acknowledgments

The authors would like to thank Brian Ichter and Fei Xia for insightful discussions and feedback throughout the project. In addition, the authors are indebted to Jun En Low, Keiko Nagami, and Alvin Sun for their assistance in setting up the hardware experiments. The NASA University Leadership initiative (grant #80NSSC20M0163) and the Toyota Research Institute (TRI) provided funds to assist the authors with their research, but this article solely reflects the opinions and conclusions of its authors and not any NASA or TRI entity.

作者感谢 Brian Ichter 与 Fei Xia 在整个项目中的深入讨论与反馈。此外，作者感谢 Jun En Low、Keiko Nagami 与 Alvin Sun 在硬件实验方面的协助，以及项目过程中其他同事的支持。

References

These appendices contain further details on the experiments in the main body of the paper, additional results and ablations supporting the main hypotheses of the paper, analysis of these supplementary results, and proofs of theoretical results. Besides the results within this document, we also refer the reader to videos of the quadrotor experiments in the quad_videos/ directory of the supplemental material. Furthermore, we include videos describing our approach, prompt templates, and further results on our project page: https://sites.google.com/view/aesop-llm. These appendices are organized as follows:

这些附录包含正文实验的更多细节、支持主要假设的额外结果与消融，以及定理 1 的证明。

Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Ahn et al. [2024] Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, et al. Autort: Embodied foundation models for large scale orchestration of robotic agents. arXiv preprint arXiv:2401.12963 , 2024.

Angelopoulos and Bates [2022] Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification, 2022.

Borrelli et al. [2017] F. Borrelli, A. Bemporad, and M. Morari. Predictive control for linear and hybrid systems . 2017.

Brohan et al. [2023] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning , pages 287–318. PMLR, 2023.

Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Arxiv eprint arXiv:2005.14165 , 2020.

Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213–229. Springer, 2020.

Chen et al. [2023a] Annie S. Chen, Govind Chada, Laura Smith, Archit Sharma, Zipeng Fu, Sergey Levine, and Chelsea Finn. Adapt on-the-go: Behavior modulation for single-life robot deployment, 2023a.

Chen et al. [2023b] Guojun Chen, Xiaojing Yu, and Lin Zhong. Typefly: Flying drones with large language model. arXiv preprint arXiv:2312.14950 , 2023b.

Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Dosovitskiy et al. [2017] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning , pages 1–16. PMLR, 2017.

Elhafsi et al. [2023] Amine Elhafsi, Rohan Sinha, Christopher Agia, Edward Schmerling, Issa A. D. Nesnas, and Marco Pavone. Semantic anomaly detection with large language models. Autonomous Robots , 47(8):1035–1055, Dec 2023. ISSN 1573-7527. doi: 10.1007/s10514-023-10132-6 . URL https://doi.org/10.1007/s10514-023-10132-6 .

Filos et al. [2020] Angelos Filos, Panagiotis Tigas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In ICML , ICML’20. JMLR.org, 2020.

Geirhos et al. [2020] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665–673, Nov 2020. ISSN 2522-5839. doi: 10.1038/s42256-020-00257-z . URL https://doi.org/10.1038/s42256-020-00257-z .

Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Hoffman et al. [2018] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1989–1998. PMLR, 10–15 Jul 2018.

Hsieh et al. [2023] Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301 , 2023.

Huang et al. [2022] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 , 2022.

Huang et al. [2023] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning , pages 540–562. PMLR, 2023.

Jacob et al. [2018] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2704–2713, 2018.

Jiang et al. [2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

Kahneman [2011] Daniel Kahneman. Thinking, fast and slow . macmillan, 2011.

Karamcheti et al. [2023] Siddharth Karamcheti, Suraj Nair, Annie S. Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. In Robotics: Science and Systems (RSS) , 2023.

Kumar et al. [2022] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=UYneFzXSJWh .

Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf .

Lee et al. [2018] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/abdeb6f575ac5c6676b747bca8d09cc2-Paper.pdf .

Liang et al. [2023] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 9493–9500. IEEE, 2023.

Lin et al. [2023] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: from natural language instructions to feasible plans. Autonomous Robots , Nov 2023. ISSN 1573-7527. doi: 10.1007/s10514-023-10131-7 . URL https://doi.org/10.1007/s10514-023-10131-7 .

Liu et al. [2020] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems , 2020.

Luo et al. [2021] Rachel Luo, Shengjia Zhao, Jonathan Kuck, Boris Ivanovic, Silvio Savarese, Edward Schmerling, and Marco Pavone. Sample-efficient safety assurances using conformal prediction, 2021. URL https://arxiv.org/abs/2109.14082 .

Miller et al. [2021] John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: On the strong correlation between out-of-distribution and in-distribution generalization, 2021. URL https://arxiv.org/abs/2107.04649 .

[32] M Minderer, A Gritsenko, A Stone, M Neumann, D Weissenborn, A Dosovitskiy, A Mahendran, A Arnab, M Dehghani, Z Shen, et al. Simple open-vocabulary object detection with vision transformers. arxiv 2022. arXiv preprint arXiv:2205.06230 .

Ovadia et al. [2019] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , Red Hook, NY, USA, 2019. Curran Associates Inc.

Panigrahi et al. [2023] Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. arXiv preprint arXiv:2302.06600 , 2023.

Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748–8763. PMLR, 2021.

Rahman et al. [2021] Quazi Marufur Rahman, Peter Corke, and Feras Dayoub. Run-time monitoring of machine learning for robotic perception: A survey of emerging trends. IEEE Access , 9:20067–20075, 2021. doi: 10.1109/ACCESS.2021.3055015 .

Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 5389–5400. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/recht19a.html .

Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084 .

Richter and Roy [2017] Charles Richter and Nicholas Roy. Safe visual navigation via deep learning and novelty detection. In RSS , July 2017.

Ruff et al. [2021] Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G. Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE , 109(5):756–795, 2021. doi: 10.1109/JPROC.2021.3052449 .

Sagawa et al. [2020] Shiori Sagawa, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=ryxGuJrFvS .

Salehi et al. [2021] Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges, 2021. URL https://arxiv.org/abs/2110.14051 .

Shah et al. [2023] Dhruv Shah, Błażej Osiński, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning , pages 492–504. PMLR, 2023.

Sharma et al. [2021] Apoorva Sharma, Navid Azizan, and Marco Pavone. Sketching curvature for efficient out-of-distribution detection for deep neural networks. CoRR , abs/2102.12567, 2021. URL https://arxiv.org/abs/2102.12567 .

Sinha et al. [2022] Rohan Sinha, Apoorva Sharma, Somrita Banerjee, Thomas Lew, Rachel Luo, Spencer M Richards, Yixiao Sun, Edward Schmerling, and Marco Pavone. A system-level view on out-of-distribution data in robotics. arXiv preprint arXiv:2212.14020 , 2022. Available at https://arxiv.org/abs/2212.14020 .

Sinha et al. [2023] Rohan Sinha, Edward Schmerling, and Marco Pavone. Closing the loop on runtime monitors with fallback-safe mpc. In 2023 62nd IEEE Conference on Decision and Control (CDC) , pages 6533–6540, 2023. doi: 10.1109/CDC49753.2023.10383965 .

Song et al. [2020] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems , 33:16857–16867, 2020.

Stellato et al. [2020] B. Stellato, G. Banjac, P. Goulart, A. Bemporad, and S. Boyd. OSQP: an operator splitting solver for quadratic programs. Mathematical Programming Computation , 12(4):637–672, 2020. doi: 10.1007/s12532-020-00179-2 . URL https://doi.org/10.1007/s12532-020-00179-2 .

Sun et al. [2023] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 , 2023.

Tagliabue et al. [2023] Andrea Tagliabue, Kota Kondo, Tong Zhao, Mason Peterson, Claudius T Tewari, and Jonathan P How. Real: Resilience and adaptation using large language models on autonomous aerial robots. arXiv preprint arXiv:2311.01403 , 2023.

Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Wang et al. [2023] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368 , 2023.

Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems , 35:24824–24837, 2022.

Wortsman et al. [2022] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7959–7971, 2022.

Xiao et al. [2023] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087–38099. PMLR, 2023.

Xiao et al. [2022] Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:2211.11736 , 2022.

Yu et al. [2023a] Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302.11550 , 2023a.

Yu et al. [2023b] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08647 , 2023b.

Zhao et al. [2021] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning , pages 12697–12706. PMLR, 2021.

-A Notation and Glossary

-A 符号与术语表

We include a glossary of all the notation and symbols used in this paper in Table V.

我们给出本文所用全部符号与记号的术语表。

	Symbol	Description
	$x$	unless explicitly defined otherwise, scalar variables are lowercase
	$\mathbf{x}$	vectors are boldfaced
	$\mathcal{X}$	sets are caligraphic
	$x_{t}$	time-varying quantities are indexed with a subscript $t\in\mathbb{N}_{\geq 0}$
Notation and conventions	$\bm{x}_{0:t}$	Shorthand to index subsequences: $\bm{x}_{0:t}:=\{\bm{x}_{0},\dots,\bm{x}_{t}\}$
	$\lambda,\delta,\epsilon,\theta$	hyperparameters (regardless of their type) are lowercase Greek characters
	$\bm{x}_{t+k|t}$	Predicted quantities at $k$ time steps into the future computed at time step $t$. Read $\bm{x}_{t+k|t}$ as “the predicted value of $\bm{x}$ at time $t+k$ given time $t$.”
	$\bm{x}$	System state
	$\bm{u}$	Input
	$\bm{o}$	Observation
	$\bm{f}$	Dynamics
	$\bm{h}$	Anomaly Detector
	$\bm{w}$	Generative reasoner
	$\bm{e}$	Embedding vector
	$\phi$	Embedding model
	$\mathcal{X}$	State constraint set
	$\mathcal{U}$	Input constraint set
	$\mathcal{O}$	Observation space
	$\tau$	Anomaly detection threshold
	$\bm{s}$	Anomaly score function
	$C$	control objective function for the MPC Section IV-B
	$\mathcal{D}_{\mathrm{nom}}$	Dataset of nominal observations
	$N$	Number of nominal observations in $\mathcal{D}_{\mathrm{nom}}$
Variables	$\mathcal{X}_{R}^{i}$	the $i$’th recovery region
	$d$	Number of recovery regions
	$\mathcal{D}_{e}$	Embedding vector cache constructed from $\mathcal{D}_{\mathrm{nom}}$
	$\alpha$	Quantile hyperparameter to select the anomaly detector threshold
	$q$	Optimization variable used to define the empirical $\alpha$-quantile
	$\mathcal{Y}$	Subset of $\{1,\dots,d\}$ indicating a selection of recovery regions
	$K$	Upper bound on the latency of the slow reasoner
	$T$	Time horizon of the MPC Section IV-B
	$J$	Objective value associated with the solution of the MPC problem Section IV-B
	$\bm{x}^{\star},\ \bm{u}^{\star}$	Starred quantities denote the optimal values of the decision variables in the MPC problem Section IV-B
	$t_{\mathrm{anom}}$	Time step at which the anomaly detector triggers

TABLE V: Glossary of notation and symbols used in this paper.

（符号表，符号与英文说明保持与原文一致以便对照。）

	Symbol	Description
	$x$	unless explicitly defined otherwise, scalar variables are lowercase
	$\mathbf{x}$	vectors are boldfaced
	$\mathcal{X}$	sets are caligraphic
	$x_{t}$	time-varying quantities are indexed with a subscript $t\in\mathbb{N}_{\geq 0}$
Notation and conventions	$\bm{x}_{0:t}$	Shorthand to index subsequences: $\bm{x}_{0:t}:=\{\bm{x}_{0},\dots,\bm{x}_{t}\}$
	$\lambda,\delta,\epsilon,\theta$	hyperparameters (regardless of their type) are lowercase Greek characters
	$\bm{x}_{t+k|t}$	Predicted quantities at $k$ time steps into the future computed at time step $t$. Read $\bm{x}_{t+k|t}$ as “the predicted value of $\bm{x}$ at time $t+k$ given time $t$.”
	$\bm{x}$	System state
	$\bm{u}$	Input
	$\bm{o}$	Observation
	$\bm{f}$	Dynamics
	$\bm{h}$	Anomaly Detector
	$\bm{w}$	Generative reasoner
	$\bm{e}$	Embedding vector
	$\phi$	Embedding model
	$\mathcal{X}$	State constraint set
	$\mathcal{U}$	Input constraint set
	$\mathcal{O}$	Observation space
	$\tau$	Anomaly detection threshold
	$\bm{s}$	Anomaly score function
	$C$	control objective function for the MPC Section IV-B
	$\mathcal{D}_{\mathrm{nom}}$	Dataset of nominal observations
	$N$	Number of nominal observations in $\mathcal{D}_{\mathrm{nom}}$
Variables	$\mathcal{X}_{R}^{i}$	the $i$’th recovery region
	$d$	Number of recovery regions
	$\mathcal{D}_{e}$	Embedding vector cache constructed from $\mathcal{D}_{\mathrm{nom}}$
	$\alpha$	Quantile hyperparameter to select the anomaly detector threshold
	$q$	Optimization variable used to define the empirical $\alpha$-quantile
	$\mathcal{Y}$	Subset of $\{1,\dots,d\}$ indicating a selection of recovery regions
	$K$	Upper bound on the latency of the slow reasoner
	$T$	Time horizon of the MPC Section IV-B
	$J$	Objective value associated with the solution of the MPC problem Section IV-B
	$\bm{x}^{\star},\ \bm{u}^{\star}$	Starred quantities denote the optimal values of the decision variables in the MPC problem Section IV-B
	$t_{\mathrm{anom}}$	Time step at which the anomaly detector triggers

TABLE V: Glossary of notation and symbols used in this paper.

-B Background: Anomaly Detection

-B 背景：异常检测

In essence, an anomaly detector is a classifier $\bm{h}:\mathcal{O}\to\{\mathrm{nominal},\ \mathrm{anomaly}\}$ that maps observations to a detection at runtime. However, limited access to anomalous examples (after all, it is their dissimilarity from prior experiences that makes them anomalous) typically precludes us from training a classifier with an obvious decision boundary using supervised learning [3, 40]. Instead, anomaly detection algorithms require two steps: First, we must construct a scalar score function $\bm{s}(o)\in\mathbb{R}$ from an observation, where a higher score indicates that the sample is “more” anomalous. Second, we need to calibrate a decision threshold $\tau\in\mathbb{R}$ on the score function such that:

本质上，异常检测器是分类器 $\bm{h}:\mathcal{O}\to\{\mathrm{True},\mathrm{False}\}$，输出观测是否应视为异常。实践中常先定义分数函数，再相对阈值做判定。本文考察若干分数函数，并在附录中比较其经验表现。

$\bm{h}(o)=\begin{cases}\mathrm{anomaly}&\text{if\bm{s}(o)>\tau$}\
\mathrm{nominal}&\text{if $\bm{s}(o)\leq\tau$}\end{cases}.$$

We investigate several score functions in this paper and compare their downstream utility in improving the safety and reliability of an autonomous robot. This necessitates instantiating the anomaly detectors with specific thresholds and measuring the accuracy of the subsequent calibrated classifier, since generic measures of a score functions’ expressiveness are not guaranteed to capture the overall impact on an autonomy stack.

本文考察若干分数函数，并比较它们在合成与真实感域上的经验表现。

-C Background: Mechanics of LLMs

-C 背景：LLM 机制

The decoder-only LLM architecture typically stacks together large numbers of Transformer modules to construct a mapping from a sequence of input tokens $x_{0:t}$ to a contextual embedding matrix $\phi(x_{0:t})\in\mathbb{R}^{n\times t+1}$. That is, each input token gets mapped to a corresponding contextual embedding. It is well-known that the contextual embeddings are generally useful for prediction tasks themselves and often exhibit interesting properties. For example, some models are trained with contrastive losses to ensure embeddings with similar semantic meaning cluster closely in the embedding space, enabling retrieval of relevant information via similarity search. However, current research in robotics focuses on using LLMs to generate strings of text through autoregressive generation, i.e., next token prediction. To do so, a linear classification head is typically added onto the output contextual embedding to define a probability distribution over the next token

仅解码器 LLM 架构通常堆叠 Transformer 块，对 token 序列做自注意力，并输出下一 token 的概率分布，再采样并追加到序列，重复直至停止。自回归生成长序列使模型能逐步推理，但也带来显著时延。

$$x_{t+1}\sim p_{\mathrm{llm}}(x_{t+1}|\phi(x_{0:t})):=\mathrm{softmax}(W_{\mathrm{out}}\phi(x_{0:t})_{t}), \tag{4}$$

which is then sampled and appended to the token sequence, after which the next token can be sampled. It should be clear that generating output sequences carries a computational cost that scales superlinearly in the cost of computing an embedding. That is, the zero-shot reasoning capabilities of LLM generation come at a computational cost, whereas direct learning on embeddings requires a source of supervision.

随后被采样并追加到 token 序列，重复该过程直至生成停止符或达到长度上限。

-D Additional Results: Synthetics

-D 补充结果：合成实验

The main goal of the synthetics experiments is to analyze the performance characteristics of our fast anomaly detector across robotics domains with diverse observational and task semantics. Here, we extend our synthetics results and analysis (Section V-A) to evaluate three such performance characteristics. First is the embedding-based anomaly detector’s quality as measured by area under the receiver operating characteristic (AUROC) curve (Section -D1). Second is the anomaly detector’s sensitivity with respect to varying detection thresholds (Section -D2). Lastly, we evaluate the anomaly detector’s performance with respect to the choice of similarity score function (Section -D3).

合成实验的主要目标是分析快异常检测器相对生成推理基线的表现，并研究嵌入模型规模、概念覆盖与分数函数选择的影响。

Fig. 5: Embedding-based anomaly detection results for the manipulation, autonomous vehicle, and VTOL domains. The top row of figures plot the AUROC as a function of experiences sampled IID from the respective domain datasets. The bottom row of figures plot accuracy as a function of the concepts sampled from the respective domain datasets.

图 5：操作、自动驾驶与 VTOL 域上基于嵌入的异常检测结果。上行：异常检测准确率相对 IID 采样经验数；下行：准确率相对采样概念覆盖。

-D1 Fast Anomaly Detector ROC Analysis

We follow the synthetics evaluation scheme presented in the main results, which compares the performance of our fast anomaly detector over nine language models, using top-$5$ scoring, with a detection threshold set to the $95$-th quantile of the scores in the nominal dataset (Eq. 2). Instead of reporting accuracy, we now measure performance in terms of AUROC, which more holistically reflects the detector’s performance across varying detection thresholds.

-D1 快异常检测器 ROC 分析

我们沿用合成域设定，绘制不同嵌入模型的 ROC 曲线以比较检测性能。

The results are shown in Fig. 5. We observe similar performance trends to the previous results (Fig. 2, measured in terms of accuracy), where MPNet closely rivals the top performing 7B parameter models, Mistral and Llama 2, followed by the OpenAI embedding models, and lastly, the BERT models. These trends become increasingly clear as domain complexity increases (i.e., from the Manipulation to VTOL domains, left to right). In the most challenging VTOL domain, we observe that concept coverage is key to attaining strong performance. This is perhaps a byproduct of the more nuanced concept shifts in complex domains, which necessitate comprehensive coverage of nominal concepts to deduce anomalies. Consider how, in the VTOL domain, the anomalous “swarming flock of birds” can be mistakenly interpreted as similar to the nominal “flying bird” without the additional grounding from other nominal concepts such as a “blimp” or “quadcopter.”

结果见图 5。我们观察到与正文类似的趋势：概念覆盖充足时，面向嵌入训练的较小模型可匹敌甚至超过更大生成模型用于异常检测的表现。

-D2 Detection Threshold Analysis

To construct the anomaly detector (Section -B), we need to select or calibrate a detection threshold $\tau$ to differentiate anomalous from nominal observations. Many techniques exist for calibrating detection thresholds, several of which offer specific guarantees on e.g., false positive rates, such as conformal prediction [3].

-D2 检测阈值分析

为构造异常检测器，我们将阈值设为标称分数的经验分位数。

In this experiment, we show the performance variation of our fast anomaly detector with respect to a range of detection thresholds (i.e., empirical quantiles) in an attempt to capture the sensitivity of our method to, for example, well or poorly calibrated thresholds. We report anomaly detection accuracy on the VTOL domain because, as shown by VTOL’s relatively slowly increasing AUROC trends in Fig. 5, we may expect larger variances in performance across thresholds. We evaluate three language models, including OpenAI Ada 002, MPNet, and Mistral (7B), and randomly (IID) sample $80$% of the full nominal dataset to construct the embedding cache for the anomaly detector. Results are reported over $5$ random seeds.

本实验展示改变分位数阈值时检测器性能的变化。

The results are shown in Table VI. First, we observe a positive relationship between anomaly detection accuracy and threshold quantile across all methods. Once again, MPNet performs nearly identically to Mistral (7B), while OpenAI Ada 002 begins to plateau at the $85$-th quantile. Both MPNet and Mistral (7B) consistently outperform generative reasoning with GPT-4 (with the exception of the $75$-th quantile). From the relatively small (yet non-negligible) performance improvements among increasing quantiles, we conclude that 1) while our anomaly detector is reasonably robust to the choice of detection threshold, 2) performance can be improved through the use of more sophisticated calibration techniques. Lastly, all methods produce negligible standard deviations across the random seeds, likely because the sampled embedding cache ($80$% of the full nominal dataset) provides $\sim 100$% coverage over all nominal concepts, which Fig. 5 shows heavily influences performance.

结果见表 VI。首先，我们观察到安全率与代价之间存在权衡：更高分位数阈值降低假阳性，但可能增加漏检。

	Embedding		Generative
Threshold:	75-Quantile	85-Quantile	90-Quantile	95-Quantile		GPT-4	GPT-4 CoT
OpenAI Ada 002	0.8 (0.002)	0.84 (0.002)	0.85 (0.002)	0.85 (0.002)		0.75	0.86
MPNet	0.83 (0.002)	0.89 (0.002)	0.91 (0.002)	0.93 (0.001)
Mistral (7B)	0.83 (0.001)	0.89 (0.001)	0.91 (0.001)	0.94 (0.001)

TABLE VI: Calibration results for a selection of embedding models in the VTOL domain. The table reports the mean anomaly detection accuracy thresholding the top-5 score function at different quantile thresholds. Standard deviations are provided in parentheses. Statistics were computed over multiple samplings of 80% of the available nominal data samples. Accuracies for GPT-4 single-token and CoT queries are provided for comparison.

	Embedding		Generative
Threshold:	75-Quantile	85-Quantile	90-Quantile	95-Quantile		GPT-4	GPT-4 CoT
OpenAI Ada 002	0.8 (0.002)	0.84 (0.002)	0.85 (0.002)	0.85 (0.002)		0.75	0.86
MPNet	0.83 (0.002)	0.89 (0.002)	0.91 (0.002)	0.93 (0.001)
Mistral (7B)	0.83 (0.001)	0.89 (0.001)	0.91 (0.001)	0.94 (0.001)

表 VI：Calibration results for a selection of embedding models in the VTOL domain. The table reports the mean anomaly detection accuracy thresholding the top-5 score function at different quantile thresholds. Standard deviations are provided in parentheses. Statistics were computed over multiple samplings of 80% of the available nominal data samples. Accuracies for GPT-4 single-token and CoT queries are provided for comparison.

Fig. 6: Score function comparison for a selection of embedding models applied to the VTOL domain. We compare the top-$k$ (for $k\in\{1,3,5,10\}$) and Mahalanobis distance score functions for MPNet, Mistral (7B), Llama2 Chat (7B), and OpenAI Ada 002.

图 6：若干嵌入模型在 VTOL 域上的分数函数比较。我们比较 top-$k$ 余弦相似度与马氏距离等分数。

-D3 Similarity Score Function Analysis

Our fast anomaly detector can be instantiated with an arbitrary choice of similarity score function $\bm{s}(\bm{e}_{t};\mathcal{D}_{e})$. For brevity, our main results exclusively featured the use of top-$5$ scoring. Thus, we conduct an ablation experiment to test the robustness of our approach to the choice of score function.

-D3 相似度分数函数分析

快异常检测器依赖分数函数 $\bm{s}$ 度量当前嵌入相对标称缓存的差异。

Our ablation includes top-$k$ scoring for $k$ in $\{1,3,5,10\}$, which quantifies the distance between the input embedding $\bm{e}_{t}$ and the $k$ closest embeddings in the nominal embedding cache $\mathcal{D}_{e}=\{\bm{e}_{i}\}_{i=1}^{N}$, and 2) the Mahalanobis distance score between the current embedding $\bm{e}_{t}$ and the multivariate Gaussian distribution formed from the mean and covariance of $\mathcal{D}_{e}=\{\bm{e}_{i}\}_{i=1}^{N}$. As before, we experiment on VTOL synthetic due to its size and complexity, evaluating four language models of varying size and function: OpenAI Ada 002, MPNet, Llama 2 Chat (7B), Mistral (7B).

消融包括 $k\in\{1,3,5,10\}$ 的 top-$k$ 打分，以及马氏距离等替代分数。

The results are shown in Fig. 6. We first observe that the accuracy difference between all score functions at $100$% concept coverage is within approximately $5$% across all evaluated language models except OpenAI Ada 002. At first glance, this might suggest that the choice of score function is, to an extent, irrelevant for achieving high detection accuracies. Upon closer analysis, we see that the top-$1$ and Mahalanobis score functions perform quite poorly in the low-data regime for most language models, while MPNet is able to utilize Mahalanobis better than others. Overall, we find that top-$k$ for $k$ in $\{3,5,10\}$ are all strong choices of score functions that demonstrate competitive performance in several data regimes.

结果见图 6。我们首先观察到各分数函数之间差异不大，表明方法对具体启发式度量选择相对稳健；top-$5$ 在准确率与校准稳定性上表现良好。

-D4 Safety Assessment in Manipulation Domain

Recall, once an anomaly has been detected by the fast-reasoner, the slower reasoner is tasked with assessing whether the observation requires a safety-preserving fallback. In the main body of the report, we present results for a safety assessment of anomalies in VTOL domain with various SoTA language models. In Table VII, we extend our safety assessment to the manipulation domain. Similar to the trend observed in Fig. 2, safety assessment is significantly easier in the manipulation domain than in the VTOL. Further, we see strong performance gains in using GPT-4 to correctly recognize an inconsequential anomaly in comparison to GPT-3.5.

-D4 操作域中的安全评估

回顾一旦检测到异常，慢推理器须评估是否需要保全安全干预。我们在操作域重复 VTOL 的评估协议。

Domain	Method	TPR	FPR	Accuracy
Manip.	GPT-3.5 Turbo	1.0	0.73	0.64
GPT-3.5 Turbo CoT	1.0	0.52	0.74
GPT-4	1.0	0.0	1.0
GPT-4 CoT	1.0	0.0	1.0

TABLE VII: Slow Generative Reasoning for Anomaly Assessment in the Warehouse Manipulation Domain.

Domain	Method	TPR	FPR	Accuracy
Manip.	GPT-3.5 Turbo	1.0	0.73	0.64
GPT-3.5 Turbo CoT	1.0	0.52	0.74
GPT-4	1.0	0.0	1.0
GPT-4 CoT	1.0	0.0	1.0

表 VII：Slow Generative Reasoning for Anomaly Assessment in the Warehouse Manipulation Domain.

-E Additional Results: Quadrotor Simulation

-E 补充结果：四旋翼仿真

In this section, we include additional details about our quadrotor simulation and describe our implementation of the MPC solver used in Algorithm 1 (AESOP). We show the qualitative behavior and improvement of our algorithm in comparison with two baseline methods in Fig. 7 in addition to annotated videos of the trajectories in the supplementary files. We also quantitatively evaluate the rate at which AESOP and the baselines successfully recover in over a set of $500$ randomized scenarios.

本节给出四旋翼仿真实验的额外细节，包括实现、基线与更多定性/定量结果。

-E1 Simulation and Implementation Details

We simulate the full 12-state dynamics of a quadrotor with an arm length of $.25\mathrm{m}$, a mass of $1\mathrm{kg}$ and an inertia matrix $J=\mathrm{diag}(.45,.45,.7)$. In contrast with the simulation in Section V, where the velocity of the quadrotor was unconstrained, we constrain the drone to move at a maximum of $1.5\mathrm{m/s}$ along each of the principle axes’ directions. As shown in green in Fig. 7, we provide the planner with four recovery regions, encoded as polytopic constraints on states, representing potential landing zones. We label the first two, around $x\approx 8\mathrm{m}$, as grassy fields. We label the third and fourth landing zones around $x\approx 3\mathrm{m}$ as a building rooftop and a parking lot respectively. We use these labels to simulate the EVTOL synthetic in closed loop, abstracting perception into a pure-text observation at each timestep. The quadrotor plans trajectories of length $T=4\mathrm{s}$ using a time discretization of $\mathrm{dt}=.1\mathrm{s}$ and we assume the slow LLM reasoner takes at most $K=1.5\mathrm{s}$ to output a decision. In these simulations, the goal of the quadrotor is to fly to a goal point, denoted by the blue star in Fig. 7.

-E1 仿真与实现细节

我们仿真四旋翼飞向并降落在目标着陆区的投递任务。动力学以 $\mathrm{dt}=0.1\mathrm{s}$ 离散化，规划时域 $T=4\mathrm{s}$，慢推理时延上界 $K=1.5\mathrm{s}$。异常观测从 VTOL 合成集随机注入。恢复区编码为停车场与开阔田野的多面体约束。

We implement the MPC Section IV-B core to the AESOP algorithm in Python using the OSQP [48] solver using linearized dynamics and a quadratic objective. Note that AESOP (Algorithm 1) requires a methodology to select a set of recovery regions to constrain the MPC Section IV-B at each timestep, and the MPC is guaranteed to be feasible for at least one such choice by Theorem 1. In principle, one could select the optimal subset of recovery regions using mixed-integer programming. Here, we adopt the simpler approach proposed in [46] where we solve multiple versions of Section IV-B at each timestep, each associated with a different combination of at least two recovery sets. Then, we select the trajectory plan associated with the feasible solution to Section IV-B with least cost. This allows the quadrotor to dynamically select various safety interventions to maximally make progress towards the goal.

我们实现第 IV-B 节 MPC 作为 AESOP 算法核心，使用标准二次规划/非线性规划求解器在滚动时域上维护标称与恢复轨迹树，并施加 $K$ 步一致性约束。

Overall, this implementation runs at approximately $42\mathrm{Hz}$ on the Nvidia Jetson Orin AGX’s CPU, with some variance induced by re-initialization of solver warm-starts when the recovery sets change. Combined with the inference latency of the fast anomaly detectors in Table III, this means that the AESOP framework can comfortably run in real-time.

总体而言，该实现约以 $42\mathrm{Hz}$ 运行，远快于慢推理器时延。

(a) Closed loop trajectory of the naive baseline MPC (black). Also shown are the nominal predicted trajectory (blue) and the minimum constraint violating recovery trajectory (red) at the timestep that the slow LLM reasoner outputs.

(a) 朴素基线 MPC 的闭环轨迹（黑）。亦显示恢复相关轨迹。

-E2 Baselines

We compare AESOP to two baselines that also use the slow LLM-based reasoner to select a recovery set on cue of the fast anomaly detector.

-E2 基线

我们将 AESOP 与两种同样尝试在异常后恢复的基线比较。

Naive MPC: The first is a naive MPC algorithm that only plans a single nominal trajectory towards the goal during nominal timesteps (i.e., when $\bm{h}(\bm{o}_{t})=0$) without maintaining feasible recovery plans. This baseline continues nominal operation until the LLM returns a choice of recovery set, at which point the MPC attempts to plan a recovery trajectory.

朴素 MPC： 第一种仅在检测到危险事件后事后计算恢复计划，忽略监视器时延。

Fallback-Safe MPC (FS-MPC) [46]: We base the second baseline on the Fallback-safe MPC proposed in [46]. This algorithm maintains several feasible recovery trajectories at all times in a similar fashion to the AESOP algorithm. However, this algorithm does not account for the latency associated with the slow LLM-based reasoner (i.e., setting $K=0$ in Section IV-B), and only engages a recovery plan once the LLM returns a decision.

回退安全 MPC（FS-MPC）[46]： 第二种基于 [46] 的回退安全规划，维护恢复选项但不显式为慢 LLM 推理预留 $K$ 步一致时域。

Our implementations of both these algorithms rely on slack variables, so that they return a trajectory plan that minimally violates constraints in case it is dynamically impossible to compute a recovery trajectory that reaches the chosen set.

这两种算法的实现都依赖松弛变量以处理瞬时不可行。

-E3 Results

Qualitative Figures: In Fig. 7, we show the qualitative differences in the behavior of AESOP and the baseline methods.

-E3 结果

定性图： 图 7 展示各方法的代表性闭环轨迹。

Firstly, Fig. 7(c) shows the trajectory of the quadrotor using AESOP in an episode where no anomalies occur and therefore, where the fast anomaly detector raises no alarms. As such, Fig. 7(c) shows that AESOP does not interfere significantly with the nominal operation of the quadrotor: It still reaches the goal location with little impediment.

首先，图 7(c) 展示 AESOP 下四旋翼轨迹：在快检测触发后于一致时域内减速，并在慢推理返回后提交所选恢复区。

Secondly, Fig. 7(a) shows the trajectory of the naive MPC baseline in an episode where the fast reasoner detects an anomaly at $t=3.0\mathrm{s}$ and the slow LLM reasoner returns it’s output $1.5\mathrm{s}$ later. The naive MPC assumes that all the recovery regions are always reachable from the current state, nor does it account for the latency of the LLMs reasoning. Therefore, once the LLM outputs that the quadrotor should land at a grassy recovery region (around $x=8\mathrm{m}$ in Fig. 7(c)), it can no longer plan a dynamically feasible path to the recovery set and crash lands in an unsafe ground region. In contrast Fig. 7(f) shows that AESOP recovers the robot to the desired safe region. This example showcases that it is necessary to plan multiple trajectories even in nominal scenarios to ensure that the safety-preserving interventions selected by the LLM can be executed safely.

其次，图 7(a) 展示朴素 MPC 基线轨迹：因未预留推理时间，常在慢决策返回前丧失可行恢复，导致碰撞或无法到达恢复区。

Third, Fig. 7(b) shows the trajectory of the fallback-safe MPC (FSMPC) algorithm from [46] on the same example as Fig. 7(a) and Fig. 7(f). While the FSMPC algorithm maintains several feasible recovery plans during nominal operations, Fig. 7(b) shows that this is not sufficient to ensure safe recovery: The FSMPC algorithm does not account for the time delay of the LLM-based reasoner. Instead it continues it’s nominal operations until the LLM returns. As shown in Fig. 7(b), the feasible recovery strategies changed in the time interval between the detection of the anomaly detector and the output of the LLM. As a result, the FSMPC algorithm tried to find a dynamically feasible recovery trajectory to the grassy areas as instructed by the LLM, but was instead forced to crash land in an unsafe region. In contrast, by accounting for the latency of the LLM, AESOP was able to safely land (i.e., Fig. 7(f)), thus showing the necessity of accounting for LLM inference latency in control design for dynamic robotic systems.

第三，图 7(b) 展示回退安全 MPC 轨迹：相对朴素基线有所改善，但因未按 LLM 时延设置一致时域，在部分场景仍无法保证慢推理返回时选项仍可用。

Finally, Fig. 7(d) shows the behavior of the AESOP algorithm when the fast anomaly detector detects an anomalous scenario, but the LLM determines the anomaly is inconsequential to the robot’s safety and returns the system to nominal operations. The quadrotor descends and slows down during the inference time of the LLM, and continues its flight towards the goal thereafter. As such, Fig. 7(d) shows that by leveraging the LLM, AESOP minimally impedes goal completion when anomalies are not immediate safety risks. Moreover, Fig. 7(f) and Fig. 7(e) show that AESOP safely recovers the system when the anomaly is consequential to safety.

最后，图 7(d) 展示 AESOP 在异常被判定无害时的行为：慢推理返回 $y=0$ 后恢复标称运行，无需提交恢复区。

Quantitative Evaluation: We quantitatively ablate the improvement of our proposed approach in comparison with the naive MPC and FSMPC by simulating $500$ trajectories with a consequential anomaly appearing at a random timestep. To do so, we uniformly sample an initial condition with zero velocity and rotation in a box with width $2\mathrm{m}$ around $(x,y,z)=(10,2,2)$ and fix the goal state as in Fig. 7. We then uniformly select a timestep at which the anomaly appears in the interval $t=[1\mathrm{s},4\mathrm{s}]$. Table II shows the fraction of scenarios in which AESOP and the baselines safely reach the recovery set chosen by the LLM. By design, AESOP successfully lands the quadrotor in the chosen recovery region each time. While the FSMPC algorithm improves over the naive baseline, Table VIII shows that it is essential to account for the LLM’s latency to achieve reliability.

定量评估： 我们在 $500$ 个场景上定量消融计入慢推理时延的影响，比较成功恢复率；AESOP 达到 $100\%$，显著高于朴素 MPC 与 FS-MPC。

	Naive MPC	FS-MPC [46]	AESOP
Successful Recovery Rate	15%	23%	100%

TABLE VIII: Percentage of trajectories where the quadrotor successfully recovered to the LLM’s choice of recovery region.

	Naive MPC	FS-MPC [46]	AESOP
Successful Recovery Rate	15%	23%	100%

表 VIII：Percentage of trajectories where the quadrotor successfully recovered to the LLM’s choice of recovery region.

-F Additional Results: Autonomous Vehicle Simulation

-F 补充结果：自动驾驶仿真

(a) Vehicle approaches a nominal stop sign.

(a) 车辆接近标称停车标志。

In our ablations on self-driving scenarios, we adopt the semantic anomaly dataset presented in [12]. This dataset includes two classes of semantic anomalies with multiple instantiations in a set of independent experiments. In nominal experiments, the vehicle approaches a stop sign, as in Fig. 8(a), or a traffic light, as in Fig. 8(b), in an environment with common-place observations. In anomalous experiments, the vehicle approaches an image of a stop sign on a billboard, as in Fig. 9(a), or a truck transporting an inactive traffic light, as in Fig. 9(b). In this setting, we aim to use observations gathered from trajectories including nominal stop signs and traffic lights to identify when either anomaly is present in a novel observation. We adapt our anomaly detection pipeline using a two-step approach with language embeddings and a direct end-to-end method with multi-modal CLIP embeddings.

在自动驾驶场景消融中，我们采用 [12] 的语义异常设置：用 CARLA 生成受已记录感知失效启发的异常观测。全部异常都要求保全安全干预。

Supplemental to our discussion in the paper’s main body, our ablations provide four interesting observations: 1) ground truth scene descriptions are necessary to overcome misclassifications by the object detector, 2) a large model, on the scale of Mistral (7B), is necessary to capture the presence of an anomaly in a semantically rich environment such as self-driving, 3) embeddings from the two-stage pipeline using LM embeddings only capture semantics and do not pick up on visual novelty, whereas multi-modal CLIP embeddings incur false positives on visually novel but semantically nominal observations, and 4) that a smaller model, like MPNet, struggles to correctly detect anomalies as the number of objects in an observation increases.

作为正文讨论的补充，我们的消融比较文本嵌入、多模态嵌入与生成推理在高维 RGB 观测上的表现，并研究检测错误与场景复杂度的影响。

(a) Vehicle approaches a stop sign anomaly.

(a) 车辆接近停车标志异常。

-F1 Object Detection vs. Ground-truth Detections

Recall, the two-step pipeline constructs a textual prompt using scene descriptions output by OWL-ViT on each CARLA observation. Our initial experimentation found processing language embeddings from the raw scene descriptions returned by OWL-ViT was insufficient to surpass GPT-4V’s CoT accuracy which is demonstrated in Fig. 10. As noted in [12], because CARLA’s synthetic visual features represent a distribution shift from the realistic images on which OWL-ViT was trained, OWL-ViT periodically hallucinates object detections, such as the presence of an anomaly when none is present, or misses an anomaly detection entirely. Most commonly, we noticed the OWL-ViT characterized the rendered images as e.g., “an image of a stop sign” or “a picture of a stop sign.” Therefore, independent of the model’s size, the language embeddings on anomalous scene descriptions are not significantly different from that of nominal observations: at best, we achieve 0.70 mean accuracy with Mistral (7B) which is inferior to GPT-4V CoT. Instead, if we post-process the raw scene descriptions to contain ground-truth detections, then the resultant language embeddings are semantically different between the nominal and anomalous observations. We do this by removing false positive detections and introducing a true positive detection if missed by the detector. We demonstrate this finding in Fig. 10, which shows Mistral (7B) achieves a mean accuracy of 0.94 surpassing GPT-4V CoT by 4% absolute. While post-processing an object detector’s output is not feasible at deployment, this ablation serves as a proof of concept for our algorithm in a real-world environment where object detection is presumably reliable.

-F1 物体检测 vs. 真值检测

回顾文本嵌入流水线依赖 OWL-ViT 检测构造提示。我们比较使用检测器输出与真值场景描述时的异常检测准确率。由于仿真到现实域偏移，检测器有时产生不准确描述，从而影响较小嵌入模型；更大模型（如 Mistral）对此更鲁棒。

Also, we notice that when using ground truth scene descriptions, the model’s size creates a spread in performance in Fig. 10. Interestingly, with the introduction of ground truth detections, MPNet and BERT-large achieve similar accuracy to Mistral (7B) processing raw scene descriptions. With Mistral (7B), which is 64x and 19x larger than MPNet and BERT-large, respectively, access to ground truth detections completely unblocks anomaly detection. These results suggest that MPNet and BERT-large, with limited expression due to model size, produce language embeddings that are not semantically rich enough to capture the presence of an anomaly among numerous nominal objects. Therefore, our two-step pipeline requires high-fidelity scene descriptions and a sufficiently large language model, on the scale of Mistral (7B), for anomaly detection with high-dimensional observations characteristic of the real world.

此外，使用真值场景描述时，各文本嵌入检测器准确率上升，表明两阶段流水线的瓶颈部分来自感知而非嵌入异常分数本身。

Fig. 10: Anomaly detection with MPNet, BERT-large and Mistral (7B) w/ and w/out ground truth scene descriptions against generative CoT reasoning with GPT-4V.

图 10：MPNet、BERT-large 与 Mistral（7B）在有/无真值场景描述时的异常检测。

-F2 Calibration Ablation on Withheld Trajectories

Method	TPR	FPR	Bal. Accuracy
(Lang.) MPNet Abl.	0.55	0.11	0.72
(Lang.) Mistral Abl.	0.96	0.19	0.89
(Vision) CLIP Abl.	0.99	0.57	0.71

TABLE IX: Accuracy of embedding detectors when withholding nominal data from CARLA routes with anomalies. All methods are our own.

-F2 在留出轨迹上的校准消融

Method	TPR	FPR	Bal. Accuracy
(Lang.) MPNet Abl.	0.55	0.11	0.72
(Lang.) Mistral Abl.	0.96	0.19	0.89
(Vision) CLIP Abl.	0.99	0.57	0.71

表 IX：Accuracy of embedding detectors when withholding nominal data from CARLA routes with anomalies. All methods are our own.

The image observations in the CARLA dataset from [12] were constructed by driving a car along simulated routes in several different maps. Some of these routes pass by anomalous objects, e.g., a stop sign on a billboard, which trick the vehicle into making unsafe decision. This means that episodes wherein the vehicle takes unsafe actions include both nominal and anomalous observations, and that the routes appear both with and without anomalous objects. In the main evaluations in Table IV and Section -F the embedding caches therefore contain nominal embeddings associated with all the routes, i.e., capturing a setting wherein anomalies suddenly appear on roads the AV often drives. As shown in Table IV, the two-stage detectors (using MPNet and Mistral) and the single stage multi-modal detector (using CLIP) perform well at detecting anomalies in this scenario.

CARLA 数据集 [12] 中的图像观测沿若干路线采集；异常物体被放置在路线上以诱使不安全决策。因此同一路线既出现标称观测也出现异常观测，校准策略会影响嵌入缓存的构成。

Therefore, we also run an ablation wherein we withhold all nominal data from routes wherein anomalies occur when constructing the embedding cache, leaving only routes in which anomalies never occur. This resembles a setting where the vehicle drives novel routes that contain sporadic anomalies. As shown in Table IV, the false positive rate of the CLIP embedding-based approach significantly increases in this scenario. This is most likely because the CLIP embeddings contain both visual features (e.g., those suitable for object detection as used in [49]) and semantic features. Therefore, the visual novelty in the previously unseen trajectories causes false positives. In contrast, the two-stage approaches, which first describe the scene with an object detector before prompting an LM embedding model, do not attend to visual features and therefore do not suffer such performance drops. In [23], the authors argue that multimodal models for robotics should explicitly balance visual and semantic features, and in line with those insights, we argue future work investigating how to differentiate visual and semantic features can make multi-modal anomaly detectors more robust.

因此我们还做消融：在构造嵌入缓存时扣留所有经过异常物体的路线上的标称观测，仅用从未经过异常的路线。该更严格校准下，纯视觉 CLIP 嵌入假阳性上升，而基于语言的两阶段检测更稳健。

-F3 Accuracy vs. Complexity of Observations

Here, we further investigate the discrepancy in accuracy between the smaller MPNet embedding model (110M) and the larger Mistral embedding model (7B) in Table IV. We do so because on the synthetic tasks in Section V-A, where the observations contain descriptions of at most three objects, both models performed more comparably. As shown in Fig. 11, we see that the accuracy of MPNet drops off when the total number of objects within each image observation increases, which largely explains their difference in performance. However, an interesting nuance is that MPNet’s accuracy is again comparable to Mistral when there are 7-8 objects in the observation. Fig. 12 supports the hypothesis that this may be because the imbalance between nominal and anomalous images is larger for observations with many objects, so that large numbers of observations may correlate with nominal conditions. Moreover, there are significantly fewer observations with many obstacles. Overall, these results suggest that larger models are needed to reason about the anomalousness of more complex scenes with many objects.

-F3 准确率 vs. 观测复杂度

这里我们进一步研究文本嵌入检测器准确率如何随每帧检测物体数变化。较小模型（如 MPNet）随物体数增加准确率下降；较大模型（如 Mistral）更稳定。

Fig. 11: Accuracy of the text-based embedding detectors as a function of the number of objects within each observation for the CARLA dataset.

图 11：文本嵌入检测器准确率随每观测帧内物体数的变化。

Fig. 12: Representation of nominal (nom.) and anomalous (anom.) observations (obs.) in the our CARLA evaluation dataset.

图 12：CARLA 评估数据集中标称（nom.）与异常（anom.）观测（obs.）的表示。

-G Prompting Details

-G 提示细节

We emphasize that all the prompts used in our experiments can be found in the repositories listed on our project webpage, https://sites.google.com/view/aesop-llm. However, for completeness, we briefly describe our prompting strategy here. For all language-based tasks (i.e., the synthetics in Section V-A, the hardware in Section V-C, and the text-based evaluations in Section V-D), we parse the current task description of the robot and a list of all observed objects into a prompt template. The template first provides a brief description of the robot that is being monitored and defines the monitoring task, followed by the current task and observation. The prompt finally concludes with an instruction on the monitor’s output (e.g., a single token nominal/anomaly classification or a chain-of-thought safety assessment). For multi-modal models, we only provide the robot’s task. In early experiments, we found that it was necessary to provide the descriptions of the robot and the monitoring task so that the model interprets the observations from the context of the robot’s task, whereas e.g., simply embedding an object label or asking an LLM whether an object is anomalous fails to capture its relation to the robot and other objects in the environment. As a concrete example, we use the following template for the synthetic VTOL task in Section V-A:

我们强调实验所用全部提示可通过项目页获取。总体上，快检测不依赖生成提示，而慢推理提示提供任务上下文、可用干预列表，并要求模型按严格模板输出分类，以便稳定输出长度并界定时延上界 $K$。合成域与硬件实验使用相同结构，仅场景描述与干预选项不同。

-H Proof of Theorem 1

-H 定理 1 的证明

Finally, we prove prove Theorem 1, which establishes the properties of the closed loop system formed by AESOP (Algorithm 1) and Eq. 1. To do so, the assumption that $\mathcal{X}_{R}^{1},\dots,\mathcal{X}_{R}^{d}$ are control invariant sets, which we made in the problem formulation (see Section III), is critical. Therefore, to make this paper as self-contained as possible, we first reiterate the standard definition of a control invariant set.

最后，我们证明定理 1，它确立算法 1 与第 IV-B 节 MPC 构成的闭环系统的性质。

Definition 1(Control Invariant Set [4]).

A set $\mathcal{X}_{R}\subseteq\mathcal{X}$ for the dynamical system Eq. 1 subject to state and input constraints $\mathcal{X}\subseteq\mathbb{R}^{n}$, $\mathcal{U}\subseteq\mathbb{R}^{m}$ is a control invariant set if for every $\bm{x}\in\mathcal{X}_{R}$, there exists a $\bm{u}\in\mathcal{U}$ such that $\bm{f}(\bm{x},\bm{u})\in\mathcal{X}_{R}$.

定义 1（控制不变集 [4]）。

若对任意 $x\in\mathcal{S}$，存在 $u\in\mathcal{U}$ 使 $f(x,u)\in\mathcal{S}$，则集合 $\mathcal{S}\subseteq\mathcal{X}$ 控制不变。

As shown in our experiments in Section V-B, Section V-B, it is often straightforward to identify recovery regions that are control invariant: For example, states in which the quadrotor has landed are control invariant. We now restate Theorem 1 and provide its proof, which relies on a recursive feasibility argument [4].

如第 V-B 节实验所示，手工设计的恢复区域在我们的设定中可保持持久可行。

See 1

见定义 1

Proof.

Suppose that the MPC in Section IV-B is feasible for some set of recovery strategies $\mathcal{Y}\subseteq\{1,\dots,d\}$ at some time step $t<t_{\mathrm{anom}}$. Let $\bm{x}_{t:t+T+1|t}^{i,\star}$ denote optimal predicted trajectories and let $\bm{u}_{t:t+T|t}^{i,\star}$ be the optimal predicted input sequences associated with a) the nominal trajectory, $i=0$, and b) recovery strategies, $i\in\mathcal{Y}$, that minimize Section IV-B at time $t$. Then, it holds that $\bm{x}_{t}\in\mathcal{X}$ and $\bm{u}_{t}=\bm{u}_{t|t}^{0,\star}\in\mathcal{U}$ by construction of Section IV-B.

证明。

设第 IV-B 节 MPC 在 $t=0$ 对某 $\mathcal{Y}$ 可行。算法 1 在标称模式施加标称轨迹的第一输入；由一致性约束，该输入对所有恢复轨迹亦可行，故状态–输入约束满足，且至少一种干预保持可行。

Furthermore, since we assume that each recovery set $\mathcal{X}_{R}^{i}$ is a control invariant set, there exists an input $\bm{u}_{t+T+1|t+1}^{i}\in\mathcal{U}$ such that the input sequence $\bm{u}_{t+1:t+T+1|t+1}^{i}:=[\bm{u}_{t+1:t+T|t}^{i,\star};\bm{u}_{t+T+1|t+1}^{i}]$ and its associated state sequence $\bm{x}_{t+1:t+T+2|t+2}^{i}:=[\bm{x}_{t+1};\bm{x}_{t+2:t+T+1|t}^{i,\star};\bm{f}(\bm{x}_{t+T+1|t}^{i,\star},\bm{u}_{t+T+1|t+1}^{i})]$ satisfy state and input constraints with $\bm{x}_{t+T+2|t+1}^{i}\in\mathcal{X}_{R}^{i}$ for each $i\in\mathcal{Y}$. This implies that 1) there exists a set $\mathcal{Y}^{\prime}\subseteq\{1,\dots,d\}$ with $|\mathcal{Y}^{\prime}|\geq 1$ for which the MPC Section IV-B is feasible at time $t+1$ and 2) we therefore satisfy $\bm{x}_{t+1}\in\mathcal{X}$ and $\bm{u}_{t+1}\in\mathcal{U}$. Therefore, we have 1) that $\bm{x}_{t}\in\mathcal{X}$ and $\bm{u}_{t}\in\mathcal{U}$ and 2) there exists at least one safety intervention for which the MPC Section IV-B is feasible for all $t\leq t_{\mathrm{anom}}$.