SymSkill: Symbol and Skill Co-Invention for Data-Efficient and Reactive Long-Horizon Manipulation

SymSkill：符号与技能共同发明，用于省数据且可反应的长程操作

Yifei Simon Shao    Yuchen Zheng    Sunan Sun    Pratik Chaudhari    Vijay Kumar    Nadia Figueroa Affiliation:  GRASP Laboratory, University of Pennsylvania, Philadelphia, PA, 19104 USA Affiliation:  yishao, zhengyc, sunan, pratikac, kumar, nadiafig@seas.upenn.edu

Abstract

摘要

Multi-step manipulation in dynamic environments remains challenging. Imitation learning (IL) is reactive but lacks compositional generalization, since monolithic policies do not decide which skill to reuse when scenes change. Classical task-and-motion planning (TAMP) offers compositionality, but its high planning latency prevents real-time failure recovery. We introduce SymSkill, a unified framework that jointly learns predicates, operators, and skills from unlabeled, unsegmented demonstrations, combining compositional generalization with real-time recovery. Offline, SymSkill learns symbolic abstractions and goal-oriented skills directly from demonstrations. Online, given a conjunction of learned predicates, it uses a symbolic planner to compose and reorder skills to achieve symbolic goals while recovering from failures at both the motion and symbolic levels in real time. Coupled with a compliant controller, SymSkill supports safe execution under human and environmental disturbances. In RoboCasa simulation, SymSkill executes 12 single-step tasks with 85% success and composes them into multi-step plans without additional data. On a real Franka robot, it learns from 5 minutes of play data and performs 12-step tasks from goal specifications. Code and additional analysis are available at https://symskill.github.io/.

在动态环境中完成多步操作仍然很困难。模仿学习（IL）反应快，但缺少组合泛化，因为整段策略在场景变化时并不能决定该复用哪一项技能。经典的任务与运动规划（TAMP）具有组合性，但其较高的规划时延妨碍了实时失败恢复。我们提出 SymSkill，这是一个统一框架，从无标注、未切分的演示中共同学习谓词、算子和技能，把组合泛化与实时恢复结合起来。离线时，SymSkill 直接从演示中学习符号抽象和面向目标的技能。在线时，给定已学习谓词的合取，它用符号规划器组合并重排技能，以达成符号目标，同时在运动层和符号层上实时恢复失败。再配上柔顺控制器，SymSkill 能够在人和环境扰动下安全执行。在 RoboCasa 仿真中，SymSkill 以 85% 的成功率完成 12 项单步任务，并在不增加新数据的情况下把它们组成多步计划。在真实 Franka 机器人上，它从 5 分钟的玩乐式数据中学习，并根据目标设定执行 12 步任务。代码与补充分析见 https://symskill.github.io/。

I Introduction

I 引言

Enabling robots to perform complex, long‑horizon manipulation in the real world remains challenging. Recent imitation‑learning (IL) approaches [1, 2] excel at reproducing skills given large, high‑quality datasets, but tend to learn monolithic policies rather than reusable skills and predicates that compose into multi‑step plans. Historically, Task and Motion Planning (TAMP) bridges this gap by decomposing problems into symbolic planning over predicates/operators and continuous motion generation [3]. However, two factors limit TAMP scalability in practice. 1) Symbols and skills are often hand‑engineered and tuned per environment, which is labor‑intensive. 2) TAMP takes tens to hundreds of seconds to solve a large problem in a realistic contact-rich simulation environments[4], making it infeasible to plan in dynamic environments with moving objects, or achieve real-time failure recovery at the symbolic or motion level.

让机器人在真实世界中完成复杂的长程操作仍然很困难。近来的模仿学习（IL）方法[1, 2]在拥有大规模、高质量数据时，擅长复现技能，但往往学到的是整段策略，而不是可以组成多步计划的可复用技能和谓词。历史上，任务与运动规划（TAMP）通过把问题分解为对谓词/算子的符号规划以及连续运动生成，来弥合这一差距[3]。但有两个因素限制了 TAMP 在实践中的可扩展性。1）符号和技能常常要按环境手写和调参，耗费人力。2）在现实的、接触丰富的仿真环境中，TAMP 求解一个较大问题往往需要数十到数百秒[4]，因而难以在有移动物体的动态环境中规划，也难以在符号层或运动层上实现实时失败恢复。

Fig. 1: Illustration of the SymSkill predicate and skill co-invention process on a DoorOpen task. Left: In the premotion segment (end-effector only motion), the object in motion in the next segment is treated as the object of interest ${o_{\textrm{int}}}$, and its frame serves as the reference for both predicate and skill learning. End-effector trajectories in this frame are used to fit SE(3) LPV-DS skills, and their endpoints are clustered to yield object-gripper relative pose predicates ${}^{{o_{\textrm{int}}}}\psi_{ee}$. Right: In the motion segment (gripper + object moving), a reference object ${o_{\textrm{ref}}}$ is selected by querying a VLM on frames from the segment. Gripper trajectories are then expressed in the ${o_{\textrm{ref}}}$ frame and used to fit a DS skill. Endpoints of the manipulated object trajectory in the ${o_{\textrm{ref}}}$ frame are clustered to yield object–object relative pose predicates ${}^{o_{\textrm{ref}}}\psi_{o_{\textrm{int}}}$.

图 1：在开门（DoorOpen）任务上，对 SymSkill 谓词与技能共同发明过程的示意。左：在预运动片段（只有末端执行器在运动）中，下一段里正在移动的物体被当作兴趣物体 ${o_{\textrm{int}}}$，其坐标系同时作为谓词学习和技能学习的参考。该坐标系下的末端执行器轨迹用来拟合 SE(3) LPV-DS 技能，轨迹端点再聚类，得到物体与夹爪之间的相对位姿谓词 ${}^{{o_{\textrm{int}}}}\psi_{ee}$。右：在运动片段（夹爪与物体一起运动）中，通过在该片段的图像上查询视觉语言模型（VLM），选定参考物体 ${o_{\textrm{ref}}}$。夹爪轨迹再被表示在 ${o_{\textrm{ref}}}$ 坐标系中，用来拟合一个动力系统技能。被操作物体轨迹在 ${o_{\textrm{ref}}}$ 坐标系中的端点再聚类，得到物体与物体之间的相对位姿谓词 ${}^{o_{\textrm{ref}}}\psi_{o_{\textrm{int}}}$。

Symbol and Skill Co-Invention methods, such as [5], aim to combine the benefits of IL and TAMP by learning reusable symbols and skills from robot demonstrations and planning over them at runtime. As shown in [6, 5], there is a delicate trade-off between inventing long-horizon operators that are too general to support effective planning and inventing operators that are too fine-grained to admit robust skill learning from limited data. Existing methods therefore rely on propose-and-down-select optimization procedures for predicate selection, but these searches can become slow as the number of objects and demonstrations grows, and may still fail to recover semantically meaningful predicates. We instead exploit a simpler structural prior: many manipulation interactions follow a small number of recurring object-centric patterns, where robots approach an object through a limited set of relative poses and manipulated objects come to rest in a small number of meaningful poses relative to nearby reference objects. Inspired by recent works that use VLMs to identify task-relevant objects, we use a VLM only in a lightweight offline role to identify the relevant stationary reference object in each demonstration, enabling relative-frame predicate and skill learning without relying on VLMs for online reasoning or control.

符号与技能共同发明方法，例如[5]，希望结合模仿学习和 TAMP 的优点，从机器人演示中学习可复用的符号和技能，并在运行时对它们做规划。如[6, 5]所示，这里存在一种微妙的权衡：发明出的长程算子如果过于一般，就难以支持有效规划；如果过于细碎，又很难从有限数据中学到稳健的技能。因此，已有方法依赖“先提出再筛选”的优化流程来选择谓词，但随着物体和演示数量增加，这类搜索会变慢，而且仍可能无法恢复出语义上有意义的谓词。我们转而利用一个更简单的结构先验：许多操作交互都遵循少数反复出现的、以物体为中心的模式——机器人通过有限的一组相对位姿接近某个物体，被操作物体也会相对于附近的参考物体，停在少数有意义的位姿上。受近来用视觉语言模型识别任务相关物体的工作启发，我们只让视觉语言模型在离线阶段承担一个轻量角色：识别每条演示中相关的静止参考物体，从而在不依赖视觉语言模型做在线推理或控制的情况下，学习相对坐标系下的谓词和技能。

Fig. 2: SymSkill offline pipeline (top half) and the online pipeline (bottom half). Subsection V-A (purple) describes segmentation and reference frame selection. Subsection V-B (orange) describes how predicates are learned for each segment. Subsection V-C (green) learns the operators for online planning. Subsection V-D (blue) describes how each operator’s skill is learned. Subsection V-E (yellow) describes how SymSkill operates online.

图 2：SymSkill 的离线流程（上半部分）和在线流程（下半部分）。第 V-A 小节（紫色）描述切分与参考坐标系选择。第 V-B 小节（橙色）描述如何为每个片段学习谓词。第 V-C 小节（绿色）学习用于在线规划的算子。第 V-D 小节（蓝色）描述如何学习每个算子的技能。第 V-E 小节（黄色）描述 SymSkill 如何在线运行。

To this end, we propose SymSkill, a unified framework that learns predicates, operators, and goal-oriented skills in an unsupervised manner from unsegmented robot demonstration data, requiring as few as 5 demonstrations per task. At the symbolic level, SymSkill identifies the object each trajectory segment moves toward using a VLM and automatically defines predicates as relative-pose classifiers. At the motion level, we adopt a dynamical-system (DS)-based approach to learn stable motion policies from minimal demonstration data. At execution time, given a symbolic goal specified by the learned predicates, SymSkill uses a symbolic planner to compose skills into long-horizon plans that generalize across goals. Because replanning occurs only at the symbolic level, SymSkill supports real-time error recovery. Together with the learned DS skills and a compliant passive DS controller, SymSkill is robust to continuous-state perturbations without requiring replanning. In RoboCasa, SymSkill achieves an 85% success rate on single-step tasks. Without additional data, it composes these skills to perform multi-step tasks. We also validate the approach on a Franka Panda robot, where SymSkill learns 11 operators from 5 minutes of play data and achieves user-specified symbolic goals in real time.

为此，我们提出 SymSkill，这是一个统一框架，能够以无监督方式从未切分的机器人演示数据中学习谓词、算子和面向目标的技能，每项任务最少只需约 5 条演示。在符号层，SymSkill 用视觉语言模型识别每个轨迹片段所朝向的物体，并自动把谓词定义成相对位姿分类器。在运动层，我们采用基于动力系统（DS）的方法，从极少的演示数据中学习稳定的运动策略。执行时，给定由已学习谓词指定的符号目标，SymSkill 用符号规划器把技能组成可跨目标泛化的长程计划。由于重规划只发生在符号层，SymSkill 支持实时错误恢复。再结合学到的动力系统技能和柔顺的被动动力系统控制器，SymSkill 能够在不必重规划的情况下，对连续状态扰动保持稳健。在 RoboCasa 中，SymSkill 在单步任务上达到 85% 的成功率。在不增加新数据的情况下，它还能把这些技能组合成多步任务。我们还在 Franka Panda 机器人上验证了该方法：SymSkill 从 5 分钟的玩乐式数据中学到 11 个算子，并实时达成用户指定的符号目标。

Contributions 1) a framework for joint discovery and learning of symbols and goal-oriented DS skills from unlabeled and unsegmented demonstrations of short and long-horizon tasks, 2) online execution and failure recovery with reactive planning at the task and motion level, and 3) an open-source implementation for out-of-the-box robot-learning in RoboCasa [7] with its original demonstrations.

贡献 1）一个框架，能够从短程和长程任务的无标注、未切分演示中，共同发现并学习符号以及面向目标的动力系统技能；2）在任务层和运动层上，通过反应式规划实现在线执行和失败恢复；3）一套开源实现，可直接在 RoboCasa[7] 上、使用其原始演示进行机器人学习。

II Problem Statement

II 问题陈述

We consider the problem of learning from play in deterministic, fully observed manipulation domains. Let $\mathcal{O}$ be the set of objects, where each object $o\in\mathcal{O}$ is assigned to a type $\mathbf{\lambda}(o)$ drawn from a predefined finite set $\Lambda$. Let $\mathcal{F}=\{ee\}\cup\{o:o\in\mathcal{O}\}$ denote the set of kinematic frames of end-effector and object frames.

我们考虑在确定性、完全可观测的操作域中，从玩乐式数据里学习的问题。设 $\mathcal{O}$ 为物体集合，每个物体 $o\in\mathcal{O}$ 被赋予一个类型 $\mathbf{\lambda}(o)$，该类型取自预先给定的有限集合 $\Lambda$。设 $\mathcal{F}=\{ee\}\cup\{o:o\in\mathcal{O}\}$ 表示末端执行器坐标系和物体坐标系构成的运动学坐标系集合。

A pose $\mathbf{T}\in SE(3)$ comprises position and orientation; ${}^{A}\mathbf{T}_{B}\in SE(3)$ denotes the pose of frame $B$ expressed in frame $A$. At time $t$, the continuous state in world frame is

位姿 $\mathbf{T}\in SE(3)$ 由位置和朝向组成；${}^{A}\mathbf{T}_{B}\in SE(3)$ 表示坐标系 $B$ 在坐标系 $A$ 中表示的位姿。在时刻 $t$，世界坐标系下的连续状态为

$$\mathbf{x}_{t}\;=\;\Big(\mathbf{T}_{ee},\,\{\mathbf{T}_{(o)}\}_{o\in\mathcal{O}},\,\{\mathbf{\lambda}(o)\}_{o\in\mathcal{O}}\Big).$$

Consistent with related works that also assume access to complete object states in simulation or via fiducial-based perception systems [5, 6], we assume an having a perception module that provides tracked 6D poses and object types for all task-relevant objects at each timestep.

与同样假定可以在仿真中、或通过基于标识的感知系统获得完整物体状态的相关工作一致[5, 6]，我们假定存在一个感知模块，能够在每个时间步为所有任务相关物体提供被跟踪的 6 维位姿和物体类型。

Problem Setup. We are given $N$ unlabeled and unsegmented robot demonstrations,

问题设定。 给定 $N$ 条无标注且未切分的机器人演示，

$$\mathcal{D}=\{\tau_{i}\}_{i=1}^{N},\qquad\tau_{i}=\big\{\mathbf{x}_{t}\big\}_{t=0}^{T_{i}}.$$

Each $\tau_{i}$ of length $T_{i}$ contains one or more demonstration trajectories of arbitrary object manipulating in the scene. For each trajectory, we record a time-synchronized RGB video of the workspace that keeps all task-relevant objects in view. A low-level skill policy outputs a 6D end-effector twist,

每条长度为 $T_{i}$ 的 $\tau_{i}$，包含场景中对任意物体进行操作的一条或多条演示轨迹。对每条轨迹，我们还记录与时间对齐的工作空间 RGB 视频，并使所有任务相关物体都保持在视野中。底层技能策略输出 6 维末端执行器旋量，

$$\begin{bmatrix}v,\omega\end{bmatrix}^{T}=f(\cdot), \tag{1}$$

where $v,\omega$ are the linear and angular end-effector velocities, respectively. We further assume gripper action $g=\{\text{open},\text{closed}\}$ to be either open or closed throughout the policy. At test time, given initial state $\mathbf{x}_{0}$ and goal state $\mathbf{x}_{G}$, we seek to apply sequentially a number of policy tuple $\langle f,g\rangle$ so that the $\mathbf{x}_{G}$ is achieved, while monitoring and recovering from failure in real-time. The robot action defined by policy $f$ is tracked by the following passive impedance controller

其中 $v,\omega$ 分别是末端执行器的线速度和角速度。我们进一步假定，夹爪动作 $g=\{\text{open},\text{closed}\}$ 在策略执行过程中始终为打开或关闭。测试时，给定初始状态 $\mathbf{x}_{0}$ 和目标状态 $\mathbf{x}_{G}$，我们希望依次施加若干策略元组 $\langle f,g\rangle$，以使 $\mathbf{x}_{G}$ 达成，同时实时监测并从失败中恢复。由策略 $f$ 定义的机器人动作，由下面的被动阻抗控制器跟踪

$$\displaystyle F_{ee}=G-D(\dot{\mathbf{T}}_{ee}-f(\cdot)), \tag{2}$$

where $G\in\mathbb{R}^{6}$ is the gravity compensation term, $\dot{\mathbf{T}}_{ee}\in\mathbb{R}^{6}$ is the end-effector velocity and $D$ is the damping gain ensuring the control input is energy dissipating in the directions orthogonal to the desired velocities, as in [8].

其中 $G\in\mathbb{R}^{6}$ 是重力补偿项，$\dot{\mathbf{T}}_{ee}\in\mathbb{R}^{6}$ 是末端执行器速度，$D$ 是阻尼增益，用于保证控制输入在与期望速度正交的方向上是耗能的，如[8]所述。

III Preliminary

III 预备知识

III-A Learning Stable SE(3) Policy in Relative Frame

III-A 在相对坐标系中学习稳定的 SE(3) 策略

When learning skills, we use dynamical system-based motion policy[9, 10, 11]. By leveraging redundancy of solutions from demonstration data, a learned dynamical system (DS) can be used as a stable motion policy that is robust to both temporal and spatial uncertainty. Specifically, we implement SE(3) LPV-DS [12] combined with convex policy learning[13], which requires only a small amount of data and is used later to fit one stable skill policy per learned operator. The framework consists of a linear Parameter Varying DS (LPV-DS), $f_{p}$, for position control and a Quaternion-DS, $f_{o}$, for orientation control:

在学习技能时，我们使用基于动力系统的运动策略[9, 10, 11]。通过利用演示数据中解的冗余性，学到的动力系统（DS）可以用作对时间和空间不确定性都稳健的稳定运动策略。具体来说，我们实现了与凸策略学习[13]相结合的 SE(3) LPV-DS[12]，它只需要少量数据，随后会为每个已学习算子拟合一个稳定的技能策略。该框架由用于位置控制的线性参数变化动力系统（LPV-DS）$f_{p}$，以及用于朝向控制的四元数动力系统 $f_{o}$ 组成：

$$\displaystyle v=f_{p}(x;\ \Theta_{p}),\,\,\,\,\omega=f_{o}(\mathbb{q};\ \Theta_{o}),$$

where the inputs are position $x\in\mathbb{R}^{3}$ and orientation $\mathbb{q}\in SO(3)$ represented as quaternions, and each function is parameterized by $\Theta_{*}$. Using the LPV-DS as an example, the function $f_{p}$ has the form of a mixture of continuous linear time-invariant (LTI) system:

其中输入是位置 $x\in\mathbb{R}^{3}$ 以及用四元数表示的朝向 $\mathbb{q}\in SO(3)$，每个函数由 $\Theta_{*}$ 参数化。以 LPV-DS 为例，函数 $f_{p}$ 具有连续线性时不变（LTI）系统混合的形式：

$$v=\sum_{k=1}^{K}\gamma_{k}(x)\mathbb{A}_{k}\left(x-x^{*}\right), \tag{4}$$

where $K$ represents the total number of LTI systems and $\gamma_{k}(x)$ is the mixing function that assigns the weight of each LTI system. $\gamma_{k}(x)$ is characterized by the Gaussian Mixture Model (GMM) parameters $\{\pi_{k},\mu_{k},\mathbb{\Sigma}_{k}\}_{k=1}^{K}$, which are estimated by fitting a GMM to the reference trajectories. Subsequently, each LTI system $\mathbb{A}_{k}$ is learned by solving a semi-definite program (SDP) with constraints enforcing globally asymptotic stability. For more details on SE(3) LPV-DS, please refer to [10, 11, 12].

其中 $K$ 表示 LTI 系统的总数，$\gamma_{k}(x)$ 是为每个 LTI 系统分配权重的混合函数。$\gamma_{k}(x)$ 由高斯混合模型（GMM）参数 $\{\pi_{k},\mu_{k},\mathbb{\Sigma}_{k}\}_{k=1}^{K}$ 刻画，这些参数通过对参考轨迹拟合 GMM 估计得到。随后，每个 LTI 系统 $\mathbb{A}_{k}$ 通过求解一个半定规划（SDP）来学习，其中的约束用于保证全局渐近稳定。关于 SE(3) LPV-DS 的更多细节，请参见[10, 11, 12]。

III-B Symbolic Abstraction and Task and Motion Planning

III-B 符号抽象与任务和运动规划

A predicate, in this work, is a function, $\psi(A,B)$, that takes a tuple of frames as input and maps to a truth value as,

在本工作中，谓词是一个函数 $\psi(A,B)$，它以一个坐标系元组为输入，并映射到真值，即

$$\psi_{\lambda_{1},\lambda_{2}}(A,B)\rightarrow\{\text{True},\text{False}\} \tag{5}$$

such that $\mathbf{\lambda}(A)=\lambda_{1}$ and $\mathbf{\lambda}(B)=\lambda_{2}$. Instantiating all predicates over all type-consistent tuples in $\mathbf{x}_{t}$ yields the symbolic state $s_{t}$ (the set of true ground atoms).

并满足 $\mathbf{\lambda}(A)=\lambda_{1}$ 且 $\mathbf{\lambda}(B)=\lambda_{2}$。在 $\mathbf{x}_{t}$ 上对所有类型一致的元组实例化全部谓词，就得到符号状态 $s_{t}$（为真的基础原子集合）。

An operator $\alpha=\langle\text{params},\text{pre},\text{eff},\text{maintain},\text{\emph{skill}}\rangle$ is a typed template defined over objects/frames. It consists of: (i) parameters $\text{params}=[\lambda_{1},\lambda_{2},\dots]$ specifying the required types of objects/frames, (ii) preconditions $\text{pre}(\alpha)$, the set of predicates that must hold in the symbolic state before the operator can be executed, (iii) effects $\text{eff}(\alpha)$, consisting of add effects (predicates made true) and delete effects (predicates made false) after execution, and (iv) maintenance conditions $\text{maintain}(\alpha)$, the set of predicates that must hold throughout execution. (v) skill a low-level policy tuple $\langle f,g\rangle$, such as the DS policy for $f(\cdot)$ (Sec. III-A) and grasping action $g$, that realizes this transition on the robot.

一个算子 $\alpha=\langle\text{params},\text{pre},\text{eff},\text{maintain},\text{\emph{skill}}\rangle$ 是定义在物体/坐标系上的带类型模板。它包括：(i) 参数 $\text{params}=[\lambda_{1},\lambda_{2},\dots]$，指定所需物体/坐标系的类型；(ii) 前置条件 $\text{pre}(\alpha)$，即算子执行前必须在符号状态中成立的谓词集合；(iii) 效果 $\text{eff}(\alpha)$，由添加效果（变为真的谓词）和删除效果（变为假的谓词）组成；(iv) 保持条件 $\text{maintain}(\alpha)$，即在整个执行过程中必须成立的谓词集合；(v) 技能，即实现这一转移的底层策略元组 $\langle f,g\rangle$，例如第 III-A 节中的动力系统策略 $f(\cdot)$ 以及抓取动作 $g$。

Formally, a grounded operator defines a transition on symbolic states,

形式上看，一个已落地的算子在符号状态上定义一次转移，

$$\alpha([o_{1},o_{2},\dots],s_{0})\rightarrow s_{1}, \tag{6}$$

where each parameter is assigned a type-consistent object, i.e., $\lambda(o_{i})=\lambda_{i}$. If the grounded preconditions are satisfied in $s_{0}$, the operator can be applied, producing $s_{1}$ through its effects while requiring its maintenance conditions to hold during execution. The typical planning process is a slower than real-time search and optimization process, with methods like interleaved planning [14] or Search & Sample (SeSame) [15].

其中每个参数都被赋予一个类型一致的物体，即 $\lambda(o_{i})=\lambda_{i}$。若已落地的前置条件在 $s_{0}$ 中得到满足，该算子便可应用，通过其效果产生 $s_{1}$，同时要求保持条件在执行过程中成立。典型的规划过程是慢于实时的搜索与优化过程，方法包括交错规划[14]或 Search & Sample（SeSame）[15]。

Approach Predicates Skills # of Demos Planning Time SymSkill (Ours) Relative Pose Cluster (Start/End Motion) SE(3) LPV-DS [12] 1-10 <100ms NSIL [5] Relative Pose Cluster (Low Relative Velocity) MLP BC 200 <100ms LAMP [4] Relational Critical Regions Motion Planning (MP) 200 > 50 s NOD-TAMP [16] NDF Features Optimization + MP 1-10 > 50 s

Approach Predicates Skills # of Demos Planning Time SymSkill (Ours) Relative Pose Cluster (Start/End Motion) SE(3) LPV-DS [12] 1-10 <100ms NSIL [5] Relative Pose Cluster (Low Relative Velocity) MLP BC 200 <100ms LAMP [4] Relational Critical Regions Motion Planning (MP) 200 > 50 s NOD-TAMP [16] NDF Features Optimization + MP 1-10 > 50 s

TABLE I: Comparison of predicate and skill learning methods.

方法 谓词 技能 演示数量 规划时间 SymSkill（本文） 相对位姿聚类（运动起止） SE(3) LPV-DS [12] 1-10 <100ms NSIL [5] 相对位姿聚类（低相对速度） MLP 行为克隆 200 <100ms LAMP [4] 关系关键区域 运动规划（MP） 200 > 50 s NOD-TAMP [16] NDF 特征 优化 + 运动规划 1-10 > 50 s

方法 谓词 技能 演示数量 规划时间 SymSkill（本文） 相对位姿聚类（运动起止） SE(3) LPV-DS [12] 1-10 <100ms NSIL [5] 相对位姿聚类（低相对速度） MLP 行为克隆 200 <100ms LAMP [4] 关系关键区域 运动规划（MP） 200 > 50 s NOD-TAMP [16] NDF 特征 优化 + 运动规划 1-10 > 50 s

表 I：谓词与技能学习方法的比较。

IV Related Work

IV 相关工作

We categorize related work below, and compare the most relevant works to ours in Table I. Note that all methods in the table learn predicates in relative frames, which has proven a necessity for generalizable manipulation learning frameworks. Of all the methods, ours is the only one that plans in real time and requires fewer than 10 demonstrations.

我们在下面归类相关工作，并在表 I 中把与本文最相关的工作加以比较。需要注意，表中所有方法都在相对坐标系中学习谓词，这已被证明是可泛化操作学习框架的必要因素。在所有这些方法中，只有本文的方法能够实时规划，并且所需演示少于 10 条。

Data Generation for Visuomotor Policies: Related to our approach are recent works that leverage relative frames for data generation [17, 18, 19]. These methods typically segment human demonstrations into sub-trajectories and then stitch them, either through simulation or direct perception editing, to augment data and train visuomotor policies from moderately sized datasets. While effective for scaling data, they do not learn the underlying task dependencies from demonstrations, but instead reproduce rigid subtask sequences.

面向视觉运动策略的数据生成： 与我们方法相关的，是近来利用相对坐标系来生成数据的工作[17, 18, 19]。这些方法通常把人类演示切成子轨迹，再通过仿真或直接的感知编辑把它们拼接起来，以扩增数据，并在中等规模数据集上训练视觉运动策略。它们在扩大数据规模上是有效的，但并没有从演示中学习底层的任务依赖，而只是复现固定的子任务序列。

Hierarchical Imitation Learning: Current imitation learning (IL) strategies such as Diffusion Policy[1] excels at reproducing complex multi-modal skills, but they often degenerate on long-horizon tasks that require sequencing multiple skills. To address this, hierarchical IL methods [20, 21] decompose demonstrations into a high-level planner over skills and low-level controllers for execution. While this structure improves tractability and performance, the high-level planners provide no symbolic guarantees that the composed sequence of skills will achieve goal completion. Instead, their plans are statistical predictions from latent distributions, lacking logical verification or explicit reasoning over task dependencies.

层次模仿学习： 当前的模仿学习（IL）策略，例如 Diffusion Policy[1]，擅长复现复杂的多模态技能，但在需要把多项技能按顺序组合的长程任务上，往往会出现退化。为应对这一点，层次模仿学习方法[20, 21]把演示分解为对技能的高层规划器和用于执行的底层控制器。这种结构改善了可处理性和性能，但高层规划器并不提供符号上的保证，无法确保组合出的技能序列一定能完成目标。相反，它们的计划是来自潜在分布的统计预测，缺少逻辑校验，也缺少对任务依赖的显式推理。

Symbol Learning with Skill Label: One thread of work invents symbols with pre-defined skills and skill-labeled data [22, 23, 24, 25]. More recently, [26] proposed to learn the neural effect predicates of operators and classifiers for these predicates together. [16] (NOD-TAMP in Table I) uses NDF features [27] for learning grasping predicates. However, labeling is tedious, requiring teleoperating with pre-programmed skills, or performing direct operational-space teleoperation followed by skill labeling.

带技能标签的符号学习： 一条工作线是在预先定义的技能和带技能标签的数据上发明符号[22, 23, 24, 25]。近来，[26] 提出同时学习算子的神经效果谓词，以及这些谓词的分类器。[16]（表 I 中的 NOD-TAMP）使用 NDF 特征[27]来学习抓取谓词。不过，标注很繁琐，需要用预先编程的技能做遥操作，或者先做操作空间遥操作，再进行技能标注。

Symbol Learning with Unsegmented Data: This class of methods propose a candidate pool of predicates using enumeration or VLM, and then sub-select using an objective function [6, 28]. When learning from a limited number of demonstrations or when the number of features for each object is high, these approaches often fail due to limited data or extended running time, as shown in Results. Notably, [4] (LAMP in Table I) proposes Relational Critical Regions (RCR) as predicates without performing the optimization. However, it still opts to use motion planning as the skill, making real-time failure recovery difficult.

从未切分数据中学习符号： 这类方法用枚举或视觉语言模型提出候选谓词池，再用目标函数做筛选[6, 28]。当演示数量有限，或每个物体的特征数量很多时，这些方法常常会因数据不足或运行时间过长而失败，如结果部分所示。值得注意的是，[4]（表 I 中的 LAMP）提出把关系关键区域（RCR）作为谓词，而不做这一优化。不过，它仍然选择用运动规划作为技能，使实时失败恢复变得困难。

Predicates/Operator/Skill Co-invention: Among prior works, NSIL [5] (NSIL in Table I) is the closest to ours. It uses relative low-velocity regions of the trajectory as meaningful candidate predicates. However, as shown in our experiment, this method fails to produce correct and semantically meaningful predicates and still requires the error-prone down-select optimization mentioned above.

谓词/算子/技能共同发明： 在已有工作中，NSIL[5]（表 I 中的 NSIL）与本文最接近。它把轨迹中相对低速的区域当作有意义的候选谓词。但如我们的实验所示，该方法无法产生正确且语义上有意义的谓词，并且仍然需要上面提到的、容易出错的筛选优化。

V Methods

V 方法

SymSkill jointly learns predicates $\psi$, operators $\alpha$, and skills $\mathbf{\Theta}$ from unsegmented demonstrations $\mathcal{D}$ and leverages them for real-time task execution. Demonstrations are segmented into end-effector–only (premotion) and end-effector–object (motion) segments, expressed in relative frames (Sec.V-A). From these segments, we cluster endpoints to invent relative-pose predicates (Sec.V-B). Then the operators are derived by tracking predicate transitions (Sec.V-C). Lastly, DS policies skill for each operator is learned (Sec.V-D). At test time, symbolic goals are achieved by composing operators into skill sequences. Closed-loop DS policy ensures stability and disturbance rejection, while online monitoring and replanning enable real-time recovery. Fig. 2 shows the offline and online pipeline of SymSkill.

SymSkill 从未切分演示 $\mathcal{D}$ 中共同学习谓词 $\psi$、算子 $\alpha$ 和技能 $\mathbf{\Theta}$，并利用它们做实时任务执行。演示被切成仅含末端执行器的片段（预运动）以及末端执行器与物体一起运动的片段（运动），并表示在相对坐标系中（第 V-A 节）。从这些片段出发，我们对端点聚类，以发明相对位姿谓词（第 V-B 节）。随后，通过跟踪谓词转移来导出算子（第 V-C 节）。最后，为每个算子学习动力系统策略技能（第 V-D 节）。测试时，通过把算子组成技能序列来达成符号目标。闭环的动力系统策略保证稳定性和扰动抑制，在线监测和重规划则实现实时恢复。图 2 给出了 SymSkill 的离线和在线流程。

V-A Demo Segmentation and Reference-Frame Selection

V-A 演示切分与参考坐标系选择

We assume a demonstration $\tau=\{\mathbf{x}_{t}\}_{t=0}^{T}$ comprises of unordered episodes of skills, each with premotion $\rightarrow$ motion segments. A premotion segment is the motion of the end-effector gripper towards an object prior to making contact, while during a motion segment we assume at most one non-gripper object moves concurrently with the gripper. This holds in typical single-arm demonstrations for both rigid-object transport and single-joint articulated-object interactions.

我们假定一条演示 $\tau=\{\mathbf{x}_{t}\}_{t=0}^{T}$ 由无序的技能片段组成，每个技能都有预运动 $\rightarrow$ 运动这样的片段。预运动片段是末端执行器夹爪在接触之前朝向某个物体的运动；而在运动片段中，我们假定最多有一个非夹爪物体与夹爪同时运动。这在典型的单臂演示中成立，既包括刚体搬运，也包括单关节铰接物体的交互。

For each demonstration $\tau$, we compute linear and angular velocities for all frames $o$ and detect change points using a fixed threshold on either velocities. For gripper end-effector $ee$ and object $o\in\mathcal{O}$, let $t^{\text{start}}$ and $t^{\text{stop}}$ denote the times at which some $o$ begins and ceases motion. We call this object the motion object ${o_{\textrm{int}}}$ for that episode. We then extract two contiguous segments:

对每条演示 $\tau$，我们计算所有坐标系 $o$ 的线速度和角速度，并用其中一个速度上的固定阈值检测变化点。对夹爪末端执行器 $ee$ 和物体 $o\in\mathcal{O}$，设 $t^{\text{start}}$ 和 $t^{\text{stop}}$ 表示某个 $o$ 开始运动和停止运动的时刻。我们把该物体称为这一段的运动物体 ${o_{\textrm{int}}}$。然后提取两个连续片段：

$$\underbrace{\mathcal{S}^{\mathrm{pre}}_{{o_{\textrm{int}}}}=[t_{0},\,t^{\text{start}})}_{\text{gripper-only motion ({premotion})}}\quad\text{and}\quad\underbrace{\mathcal{S}^{\mathrm{mot}}_{{o_{\textrm{int}}}}=[t^{\text{start}},\,t^{\text{stop}}]}_{\text{gripper+object motion ({motion})}}.$$

Here $t_{0}$ is the maximal time before $t^{\text{start}}$ such that no object other than $ee$ is moving in $[t_{0},t^{\text{start}})$.

这里 $t_{0}$ 是 $t^{\text{start}}$ 之前的最大时刻，使得在 $[t_{0},t^{\text{start}})$ 内除 $ee$ 外没有其他物体在运动。

For premotion segments, we express trajectories in the frame of the motion object and treat the frame of ${o_{\textrm{int}}}$ as the reference frame:

对预运动片段，我们在运动物体的坐标系中表示轨迹，并把 ${o_{\textrm{int}}}$ 的坐标系当作参考坐标系：

$$\texttt{{premotion}:}\quad\bigl\{{}^{{o_{\textrm{int}}}}\mathbf{T}_{ee}(t)\bigr\}_{t\in\mathcal{S}^{\mathrm{pre}}_{{o_{\textrm{int}}}}}.$$

For motion segments, both $ee$ and ${o_{\textrm{int}}}$ are in motion in the world frame. We do not assume rigid contact between them, since manipulating articulated items often involves non-prehensile movements. We assume ${o_{\textrm{int}}}$ motion is typically organized around one or a few reference objects, each denoted as ${o_{\textrm{ref}}}$ (e.g., transporting a cup into a sink, rotating a door w.r.t. its cabinet). To obtain all reference objects for motion segments individually for each motion episode, while capturing semantically meaningful reference objects, we query the Gemini-2.5-Pro[29] VLM on $n$ evenly spaced frames from $\mathcal{S}^{\mathrm{mot}}_{{o_{\textrm{int}}}}$ with a structured output constrained to scene objects, as in Fig. 3. This structured output limits hallucination and enforces selection among known candidates. With all ${o_{\textrm{ref}}}$ fixed, we retain motion-segment trajectories in that frame:

对运动片段，$ee$ 和 ${o_{\textrm{int}}}$ 都在世界坐标系中运动。我们并不假定它们之间存在刚性接触，因为操作铰接物体时常常包含非抓取式运动。我们假定 ${o_{\textrm{int}}}$ 的运动通常围绕一个或少数几个参考物体组织，每个记为 ${o_{\textrm{ref}}}$（例如把杯子运进水槽，或相对柜体转动一扇门）。为了给每个运动片段分别得到全部参考物体，同时抓住语义上有意义的参考物体，我们在 $\mathcal{S}^{\mathrm{mot}}_{{o_{\textrm{int}}}}$ 中均匀间隔的 $n$ 帧上查询 Gemini-2.5-Pro[29] 视觉语言模型，并使用被约束到场景物体上的结构化输出，如图 3 所示。这种结构化输出限制了幻觉，并强制在已知候选中选择。在所有 ${o_{\textrm{ref}}}$ 确定之后，我们保留该坐标系下的运动片段轨迹：

$$\texttt{motion:}\quad\bigl\{{}^{{o_{\textrm{int}}}}\mathbf{T}_{ee}(t),\ {}^{{o_{\textrm{ref}}}}\mathbf{T}_{ee}(t),\ {}^{{o_{\textrm{ref}}}}\mathbf{T}_{{o_{\textrm{int}}}}(t)\bigr\}_{t\in\mathcal{S}^{\mathrm{mot}}_{{o_{\textrm{int}}}}}.$$

We assume that objects of the same type can be manipulated in a similar manner, and that interactions between the same object type and reference type share common trajectory structures that can be exploited during learning. For now, we assume each object has a predefined type and $\mathbf{\lambda}({o_{\textrm{int}}})\in\Lambda,\mathbf{\lambda}({o_{\textrm{ref}}})\in\Lambda$, but we can also easily expand to a open-object setting using a VLM for classification, as in [28].

我们假定同一类型的物体可以用相似的方式操作，并且同一物体类型与参考类型之间的交互，共享可在学习中利用的共同轨迹结构。目前，我们假定每个物体都有预先定义的类型，且 $\mathbf{\lambda}({o_{\textrm{int}}})\in\Lambda,\mathbf{\lambda}({o_{\textrm{ref}}})\in\Lambda$，但也可以很容易地扩展到开放物体设定，用视觉语言模型做分类，如[28]所述。

Outputs: aggregating across demonstrations produces:

输出： 跨演示汇总后得到：

$$\displaystyle\mathcal{D}_{\textrm{pre}}(\mathbf{\lambda}_{o_{\textrm{int}}}) \\ \displaystyle=\bigl\{\,\bigl({}^{{o_{\textrm{int}}}}\mathbf{T}_{ee}(t)\bigr)_{t\in\mathcal{S}^{\mathrm{pre}}_{{o_{\textrm{int}}}}}\;\big|\;\mathbf{\lambda}({o_{\textrm{int}}})=\mathbf{\lambda}_{o_{\textrm{int}}}\bigr\} \tag{7}$$

$$\begin{aligned} \mathcal{D}{\textrm{motion}}\bigl(\mathbf{\lambda}{o_{\textrm{int}}},\mathbf{\lambda}{o{\textrm{ref}}}\bigr)&=\bigl{,\bigl({}^{{o_{\textrm{int}}}}\mathbf{T}{ee}(t),{}^{{o{\textrm{ref}}}}\mathbf{T}{ee}(t),{}^{{o{\textrm{ref}}}}\mathbf{T}{{o{\textrm{int}}}}(t)\bigr){t\in\mathcal{S}^{\mathrm{mot}}{{o_{\textrm{int}}}}}\
&\big|;\mathbf{\lambda}({o_{\textrm{int}}})=\mathbf{\lambda}{o{\textrm{int}}}\cap\mathbf{\lambda}({o_{\textrm{ref}}})=\mathbf{\lambda}{o{\textrm{ref}}}\bigr}.\end{aligned} \tag{8}$$

Fig. 3: The VLM prompt used for the real-world learning-from-play experiment proceeds as follows. First, the initial image is used to obtain text descriptions of all objects in view. Next, four equally spaced images from each motion segment are provided to Gemini together with the required output enumeration object, using the structured output feature. The returned text is then mapped back to the corresponding object name.

图 3：真实世界玩乐式学习实验中使用的视觉语言模型提示如下进行。首先，用初始图像得到视野中所有物体的文本描述。接着，把每个运动片段中四张等间隔图像，连同所需的枚举输出对象一起提供给 Gemini，并使用结构化输出功能。返回的文本再被映射回相应的物体名称。

V-B Relative Pose Predicate Learning

V-B 相对位姿谓词学习

We seek to capture distributions of relative poses that serve as meaningful symbolic predicates. We consider the relative pose of the end-effector with respect to the motion object, ${}^{{o_{\textrm{int}}}}\mathbf{T}_{ee}$, aggregated across $\mathcal{D}_{\textrm{pre}}(\mathbf{\lambda}_{o_{\textrm{int}}})$. Rather than taking the last frame of each trajectory, which is unreliable under small datasets or non-prehensile motions, we fit normal distributions over the collection of poses observed in motion segments $\{{}^{{o_{\textrm{int}}}}\mathbf{T}_{ee}(t)\}_{t\in\mathcal{S}^{\mathrm{mot}}_{{o_{\textrm{int}}}}}$. These are two independent Gaussians over translation ${}^{{o_{\textrm{int}}}}p_{ee}\!\sim\!\mathcal{N}(\mu^{{o_{\textrm{int}}},ee}_{\text{pos}},\Sigma^{{o_{\textrm{int}}},ee}_{\text{pos}})$ and orientation $\log({}^{{o_{\textrm{int}}}}R_{ee})\!\sim\!\mathcal{N}(\mu^{{o_{\textrm{int}}},ee}_{\text{ori}},\Sigma^{{o_{\textrm{int}}},ee}_{\text{ori}})$. Given a new relative pose, we compute Mahalanobis distances to the respective means: $d_{\text{pos}}({}^{{o_{\textrm{int}}}}p_{ee}),\;\;d_{\text{ori}}(\log({}^{{o_{\textrm{int}}}}R_{ee}))$. We declare the predicate ${}^{{o_{\textrm{int}}}}\psi_{ee}$ to hold if both distances $\epsilon_{\text{pos}},\epsilon_{\text{ori}}$:

我们希望刻画那些可以作为有意义符号谓词的相对位姿分布。我们考虑末端执行器相对运动物体的位姿 ${}^{{o_{\textrm{int}}}}\mathbf{T}_{ee}$，并在 $\mathcal{D}_{\textrm{pre}}(\mathbf{\lambda}_{o_{\textrm{int}}})$ 上汇总。我们并不取每条轨迹的最后一帧，因为在小数据集或非抓取式运动下，这样做并不可靠；而是对运动片段中观测到的位姿集合 $\{{}^{{o_{\textrm{int}}}}\mathbf{T}_{ee}(t)\}_{t\in\mathcal{S}^{\mathrm{mot}}_{{o_{\textrm{int}}}}}$ 拟合正态分布。它们是平移 ${}^{{o_{\textrm{int}}}}p_{ee}\!\sim\!\mathcal{N}(\mu^{{o_{\textrm{int}}},ee}_{\text{pos}},\Sigma^{{o_{\textrm{int}}},ee}_{\text{pos}})$ 和朝向 $\log({}^{{o_{\textrm{int}}}}R_{ee})\!\sim\!\mathcal{N}(\mu^{{o_{\textrm{int}}},ee}_{\text{ori}},\Sigma^{{o_{\textrm{int}}},ee}_{\text{ori}})$ 上的两个独立高斯。给定一个新的相对位姿，我们计算到相应均值的马氏距离：$d_{\text{pos}}({}^{{o_{\textrm{int}}}}p_{ee}),\;\;d_{\text{ori}}(\log({}^{{o_{\textrm{int}}}}R_{ee}))$。若两个距离都不超过阈值 $\epsilon_{\text{pos}},\epsilon_{\text{ori}}$，我们就判定谓词 ${}^{{o_{\textrm{int}}}}\psi_{ee}$ 成立：

$${}^{{o_{\textrm{int}}}}\psi_{ee}(\mathbf{x})=\mathbf{1}\!\left[d_{\text{pos}}\leq\epsilon_{\text{pos}}\wedge d_{\text{ori}}\leq\epsilon_{\text{ori}}\right].$$

fall below thresholds. Similarly, object–object relative pose predicates ${}^{o_{\textrm{ref}}}\psi_{o_{\textrm{int}}}$ are obtained by fitting Gaussian distributions over $\{{}^{{o_{\textrm{ref}}}}\mathbf{T}_{{o_{\textrm{int}}}}(t)\}_{t\in\mathcal{S}^{\mathrm{mot}}_{{o_{\textrm{int}}}}}$, augmented with a short ($\approx$2s) post-motion window to stabilize end-pose estimation. The resulting ellipsoids not only define predicates but also serve as samplers for downstream goal-pose resampling during online recovery (Sec. V-E).

同样，物体与物体之间的相对位姿谓词 ${}^{o_{\textrm{ref}}}\psi_{o_{\textrm{int}}}$，通过对 $\{{}^{{o_{\textrm{ref}}}}\mathbf{T}_{{o_{\textrm{int}}}}(t)\}_{t\in\mathcal{S}^{\mathrm{mot}}_{{o_{\textrm{int}}}}}$ 拟合高斯分布得到，并附上一段很短（约 2 秒）的后运动窗口，以稳定末端位姿估计。得到的椭球不仅定义谓词，也作为在线恢复时下游目标位姿重采样的采样器（第 V-E 节）。

Outputs: Collecting these components yields the predicate libraries

输出： 汇集这些构件，就得到谓词库

$$\Psi_{\mathrm{pre}}(\mathbf{\lambda}_{o_{\textrm{int}}})=\{{}^{o_{\textrm{int}}}\psi_{ee}\},\qquad\Psi_{\mathrm{motion}}(\mathbf{\lambda}_{o_{\textrm{int}}},\mathbf{\lambda}_{o_{\textrm{ref}}})=\{{}^{o_{\textrm{ref}}}\psi_{o_{\textrm{int}}}\}.$$

V-C Operator Learning using Learned Predicates

V-C 用已学习谓词学习算子

After we learn the relative‑pose predicates, we re‑evaluate all demonstration trajectories with $\Psi_{\text{pre}}(\mathbf{\lambda}_{{o_{\textrm{int}}}})$ and $\Psi_{\text{motion}}(\mathbf{\lambda}_{{o_{\textrm{int}}}},\mathbf{\lambda}_{{o_{\textrm{ref}}}})$ and invent symbolic operators using the method of [30]: We first convert each demonstration into an abstract state sequence by evaluating all learned predicates at every demonstration segmentation boundary. We denote the abstract states immediately before and after a transition as $s_{0}$ and $s_{1}$, respectively. Across these sequences, we identify recurring transition groups by finding segments with the same effects, where effects are defined as

在学到相对位姿谓词之后，我们用 $\Psi_{\text{pre}}(\mathbf{\lambda}_{{o_{\textrm{int}}}})$ 和 $\Psi_{\text{motion}}(\mathbf{\lambda}_{{o_{\textrm{int}}}},\mathbf{\lambda}_{{o_{\textrm{ref}}}})$ 重新评估所有演示轨迹，并采用[30]的方法发明符号算子：我们先在每个演示切分边界上评估全部已学习谓词，把每条演示转换成抽象状态序列。我们把一次转移前后的抽象状态分别记为 $s_{0}$ 和 $s_{1}$。在这些序列中，我们通过寻找具有相同效果的片段来识别反复出现的转移组，其中效果定义为

$$\text{add}(\alpha)=\bigcap_{(s_{0},s_{1})\in\mathcal{T}}(s_{1}\setminus s_{0}),\qquad\text{del}(\alpha)=\bigcap_{(s_{0},s_{1})\in\mathcal{T}}(s_{0}\setminus s_{1}),$$

and

以及

$$\text{pre}(\alpha)=\bigcap_{(s_{0},s_{1})\in\mathcal{T}}s_{0}.$$

for each group of matched transition $\mathcal{T}$. Because our system must monitor continuous states $\mathbf{x}$ online, we augment each operator with a set of maintain conditions to be the intersection of all continuous‑state predicates that hold throughout the interval between $s_{0}$ and $s_{1}$,

对每一组匹配的转移 $\mathcal{T}$ 都如此定义。由于系统必须在线监测连续状态 $\mathbf{x}$，我们还为每个算子增加一组保持条件，取为在 $s_{0}$ 与 $s_{1}$ 之间整个区间内都成立的连续状态谓词的交，

$$\displaystyle\text{maintain}\;=\;\bigcap_{\,t(s_{0})\leq t<t(s_{1})}\!\!\mathbf{x}(t). \tag{9}$$

Together, we obtain a new operator

合在一起，我们得到一个新算子

$$\alpha=\langle\text{params},\text{pre},\text{eff},\text{maintain},\text{\emph{skill}}\rangle,$$

where params($\alpha$) are ordered and typed inputs that are automatically aggregated from all elements above. Tab. III shows the operators learned for the real-world learning-from-play experiment.

其中 params($\alpha$) 是从上述全部要素中自动汇总得到的、有序且带类型的输入。表 III 给出了真实世界玩乐式学习实验中学到的算子。

Outputs: We call the collection of operators $\Omega$, where each operator $\alpha$ has trajectory segments from the dataset. Each operator’s skill will be learned in the next subsection.

输出： 我们把算子集合称为 $\Omega$，其中每个算子 $\alpha$ 都带有来自数据集的轨迹片段。每个算子的技能将在下一小节中学习。

Fig. 4: The visualization of demonstrations and SE(3) LPV-DS policy rollout for Op3 in Tab.III. The left figure shows multiple collected trajectories placing a thing type item from various locations into the pan. The multimodal nature of the data is captured by 4 distinct Gaussians shown in different colors following the policy learning outlined in Sec. III-A. The right figure shows the reconstruction results of the learned policy starting from the same initial conditions, where the policy pose attractor in the pan frame is marked as an axis. All demonstrations converge on the attractor.

图 4：表 III 中 Op3 的演示以及 SE(3) LPV-DS 策略展开的可视化。左图给出多条把 thing 类型物品从不同位置放入锅中的采集轨迹。数据的多模态性质，由按第 III-A 节所述策略学习得到的、以不同颜色标出的 4 个不同高斯来刻画。右图给出从相同初始条件出发、对学到策略的重建结果，锅坐标系中的策略位姿吸引子用坐标轴标出。所有演示都收敛到该吸引子。

V-D SE(3) Skill Learning

V-D SE(3) 技能学习

Each operator $\alpha\in\Omega$ requires a skill = $\langle f,g\rangle$ for controlling the pose and gripper action of end-effector. We parameterize the policy $f_{\alpha}$ as a concatenated function of Eq. (3). For operators that model the premotion segments, we follow the learning procedures outlined in Sec. III-A, and use the demonstration data $\{{}^{{o_{\textrm{int}}}}\mathbf{T}_{ee}\}$ from Eq. (7) to obtain the corresponding policy:

每个算子 $\alpha\in\Omega$ 都需要一个技能 $= \langle f,g\rangle$，用来控制末端执行器的位姿和夹爪动作。我们把策略 $f_{\alpha}$ 参数化成公式 (3) 的拼接函数。对刻画预运动片段的算子，我们沿用第 III-A 节中的学习流程，并用公式 (7) 中的演示数据 $\{{}^{{o_{\textrm{int}}}}\mathbf{T}_{ee}\}$ 得到相应策略：

$$\displaystyle{}^{o_{\textrm{int}}}f_{\alpha}({}^{{o_{\textrm{int}}}}\mathbf{T}_{ee};\Theta_{p},\Theta_{o}). \tag{10}$$

For operators consisting of motion trajectories, the policies are expressed in ${o_{\textrm{ref}}}$ frame following the same learning procedure using the demonstration data $\{{}^{{o_{\textrm{ref}}}}\mathbf{T}_{ee}\}$ from Eq. (8):

对由运动轨迹组成的算子，策略被表示在 ${o_{\textrm{ref}}}$ 坐标系中，并用公式 (8) 中的演示数据 $\{{}^{{o_{\textrm{ref}}}}\mathbf{T}_{ee}\}$，按同样的学习流程得到：

$$\displaystyle{}^{o_{\textrm{ref}}}f_{\alpha}({}^{{o_{\textrm{ref}}}}\mathbf{T}_{ee};\Theta_{p},\Theta_{o}). \tag{11}$$

For motion segments, specifically the ones including non-prehensile motions, we find that policies using relative pose trajectories between $ee,{o_{\textrm{ref}}}$ perform significantly better than using relative pose trajectories between ${o_{\textrm{int}}},{o_{\textrm{ref}}}$, hence justifying the use of $\{{}^{{o_{\textrm{ref}}}}\mathbf{T}_{ee}\}$ in Eq. (11). As introduced in Sec. II, the output of each learned policy is tracked by a task-space passive controller [8] as in Eq. (2). One visualized policy is shown in Fig. 4.

对运动片段，尤其是包含非抓取式运动的片段，我们发现，使用 $ee$ 与 ${o_{\textrm{ref}}}$ 之间的相对位姿轨迹，明显优于使用 ${o_{\textrm{int}}}$ 与 ${o_{\textrm{ref}}}$ 之间的相对位姿轨迹，这也说明了公式 (11) 中使用 $\{{}^{{o_{\textrm{ref}}}}\mathbf{T}_{ee}\}$ 的合理性。如第 II 节所述，每个学到策略的输出都由任务空间被动控制器[8]按公式 (2) 跟踪。一个可视化策略见图 4。

V-E Online Execution Monitoring and Adaptation

V-E 在线执行监测与适应

The online algorithm requires a symbolic goal state $s_{g}$[^1], expressed as a conjunction of one or more learned predicates. Given the current continuous state $\mathbf{x}_{0}$, we first compute its symbolic abstraction $s_{0}$. We then perform symbolic planning using A∗ search with the learned operators, producing a plan skeleton $\alpha_{1},\alpha_{2},\ldots,\alpha_{n}$ from $s_{0}$ to $s_{g}$, if one exists. We then sequentially execute skill in $\alpha$, requiring little computation during execution. Since each skill is a stable feedback policy, when $f_{\alpha}$ outputs zero velocity, we advance to the next operator.

在线算法需要一个符号目标状态 $s_{g}$[^1]，表示为至少一个已学习谓词的合取。给定当前连续状态 $\mathbf{x}_{0}$，我们先计算它的符号抽象 $s_{0}$。然后，我们用已学习算子做 A* 搜索的符号规划，如果存在从 $s_{0}$ 到 $s_{g}$ 的计划，就得到计划骨架 $\alpha_{1},\alpha_{2},\ldots,\alpha_{n}$。随后，我们依次执行 $\alpha$ 中的技能，执行过程中几乎不需要额外计算。由于每个技能都是稳定的反馈策略，当 $f_{\alpha}$ 输出零速度时，我们就进入下一个算子。

During execution, we monitor i) that the maintain conditions hold and ii) new expected effect satisfaction when each skill ends. If failure occurs we replan from the current state. See project website for online algorithm. We summarize the elements that enable reliable recovery and eventual plan completion.

在执行过程中，我们监测：i) 保持条件是否成立；ii) 每个技能结束时，新的期望效果是否得到满足。如果发生失败，我们就从当前状态重新规划。在线算法见项目网站。我们概括使可靠恢复并最终完成计划成为可能的要素。

Obstacle Avoidance For each object in the scene $\mathcal{O}_{-{o_{\textrm{int}}}}$, excluding the ones that the gripper is holding or approaching, we model them as an ellipsoid and apply the local modulation introduced in [9] during skill execution:

避障 对场景中除夹爪正在持有或正在接近的物体之外的每个物体 $\mathcal{O}_{-{o_{\textrm{int}}}}$，我们把它们建模成椭球，并在技能执行时施加[9]中引入的局部调制：

$$\displaystyle f^{\prime}=\mathbf{M}(\mathcal{O}_{-{o_{\textrm{int}}}})f(\mathbf{T};\mathbf{\Theta}), \tag{12}$$

where the modulated policy $f^{\prime}$ incorporates the obstacle avoidance behavior and the modulation matrix $\mathbf{M}$ is constructed through eigenvalue decomposition with the normal and tangent directions of the defined ellipsoid boundaries.

其中被调制的策略 $f^{\prime}$ 纳入了避障行为，调制矩阵 $\mathbf{M}$ 通过对所定义椭球边界的法向和切向做特征值分解来构造。

Resampling after failure If a robot fails to execute a task on a given object, attempting it again without replanning will typically lead to another failure. Inspired by TAMP, a policy $f$ can be modified online by performing a frame transform when detecting failure during skill execution. Formally, we directly transform the policy: $f^{\prime}=\mathbf{T}f$, where $\mathbf{T}$ is the pose sampled from the effect normal distribution as introduced in Sec. V-B. 1) When the maintain effect is lost, such as losing the grasp of an object, we assume the previous skill needs to be resampled; 2) When effects of current $\alpha$ is not satisfied at the end of skill, we assume the attractor of the current skill needs to be resampled. Therefore, depending on the operator sequence, we draw samples from ${}^{o_{\textrm{int}}}\psi_{ee}$ or ${}^{o_{\textrm{ref}}}\psi_{o_{\textrm{int}}}$ to apply transformation. This strategy enables autonomous recovery from external disturbances, such as the robot regrasping a dropped object or reopening a closed cabinet door.

失败后的重采样 如果机器人在给定物体上执行任务失败，不重规划就再次尝试，通常还会再次失败。受 TAMP 启发，当在技能执行中检测到失败时，可以通过做一个坐标系变换，在线修改策略 $f$。形式上看，我们直接变换策略：$f^{\prime}=\mathbf{T}f$，其中 $\mathbf{T}$ 是从第 V-B 节引入的效果正态分布中采样得到的位姿。1) 当保持效果丢失时，例如失去对物体的抓取，我们假定需要重采样前一个技能；2) 当当前 $\alpha$ 的效果在技能结束时未被满足时，我们假定需要重采样当前技能的吸引子。因此，根据算子序列，我们从 ${}^{o_{\textrm{int}}}\psi_{ee}$ 或 ${}^{o_{\textrm{ref}}}\psi_{o_{\textrm{int}}}$ 中采样以施加变换。这一策略使系统能够从外部扰动中自主恢复，例如重新抓起掉落的物体，或重新打开已关上的柜门。

VI Experimental Results

VI 实验结果

We evaluate our method in RoboCasa[7] simulation environment, and on the real Franka robot with motion capture and a webcam during learning.

我们在 RoboCasa[7] 仿真环境中评估该方法，并在真实 Franka 机器人上，于学习阶段使用动作捕捉和网络摄像头进行评估。

VI-A Single Step Simulation Result

VI-A 单步仿真结果

We exclusively use the demonstrations collected by the authors of the RoboCasa paper to ensure reproducibility. For single-step tasks, we reduce the variation in the demonstrations by filtering to keep only one variant of fixture per task, such as those OpenSingleDoor demonstrations with a cabinet that opens to the left. At test time, we also only generate environment with reduced task variation. Each task still have some randomness such as object initial poses. Table II shows the result of the proposed method by learning from 5-10 demonstrations per task: Proposed w/o monitoring removes online maintain/effect checking and replanning, executing the policies in open loop once the initial symbolic plan is produced. Proposed w/ DP shows SymSkill when the low level policy is replaced by state-input U-Net-based Diffusion Policy (DP).

为保证可复现，我们只使用 RoboCasa 论文作者采集的演示。对单步任务，我们通过筛选来减少演示中的变化，每项任务只保留一种夹具变体，例如只保留柜门向左开的 OpenSingleDoor 演示。测试时，我们也只生成任务变化被限制后的环境。每项任务仍保留一些随机性，例如物体的初始位姿。表 II 给出了每项任务从 5–10 条演示中学习后的结果：Proposed w/o monitoring 去掉在线的保持/效果检查和重规划，一旦得到初始符号计划，就开环执行策略。Proposed w/ DP 表示把底层策略替换为以状态为输入、基于 U-Net 的 Diffusion Policy（DP）时的 SymSkill。

Task Success Rate %	Proposed	Proposed w/o Monitoring	Proposed w/ DP
OpenSingleDoor	100	100	0
CloseSingleDoor	100	80	0
PnPCounterToCab	80	70	0
PnPCabToCounter	100	40	0
PnPStoveToCounter	70	30	0
PnPCounterToStove	20	0	0
OpenDrawer	100	100	0
CloseDrawer	70	50	40
TurnOnStove	100	100	0
TurnOffStove	80	30	0
TurnOnSinkFaucet	100	100	0
TurnOffSinkFaucet	100	90	0
Average	85.0	65.0	3.3

TABLE II: RoboCasa simulation result on 10 trials per task

任务成功率 %	本文方法	本文方法 无监测	本文方法 配 DP
OpenSingleDoor	100	100	0
CloseSingleDoor	100	80	0
PnPCounterToCab	80	70	0
PnPCabToCounter	100	40	0
PnPStoveToCounter	70	30	0
PnPCounterToStove	20	0	0
OpenDrawer	100	100	0
CloseDrawer	70	50	40
TurnOnStove	100	100	0
TurnOffStove	80	30	0
TurnOnSinkFaucet	100	100	0
TurnOffSinkFaucet	100	90	0
平均	85.0	65.0	3.3

表 II：RoboCasa 仿真结果，每项任务 10 次试验

Operators	Human-Interpretable Summary	Preconditions	Effects	Maintain Conditions
Op7	Pick lid from cabinet	GripperOpen, Lid-in-cabinet	Gripper-in-lid, $\lnot$Lid-in-cabinet, $\lnot$GripperOpen	Lid-in-cabinet, GripperOpen
Op11	Pick lid from cookware	GripperOpen, Lid-in-cookware	Gripper-in-lid, $\lnot$Lid-in-cookware, $\lnot$GripperOpen	Lid-in-cookware, GripperOpen
Op1	Place lid $\to$ cabinet	Gripper-in-lid	Lid-in-cabinet, $\lnot$Gripper-in-lid, GripperOpen	Gripper-in-lid
Op8	Place lid $\to$ cookware	Gripper-in-lid	Lid-in-cookware, $\lnot$Gripper-in-lid, GripperOpen	Gripper-in-lid
Op9	Pick thing from drawer	GripperOpen, Thing-in-container, Thing-in-drawer	Gripper-in-thing, $\lnot$Thing-in-drawer, $\lnot$GripperOpen	Thing-in-container, Thing-in-drawer, GripperOpen
Op5	Pick thing from cookware	GripperOpen, Lid-in-cabinet, Thing-in-cookware	Gripper-in-thing, $\lnot$Thing-in-cookware, $\lnot$GripperOpen	Thing-in-cookware, Lid-in-cabinet, GripperOpen
Op10	Pick thing from container	GripperOpen, Thing-in-container	Gripper-in-thing, $\lnot$Thing-in-container, $\lnot$GripperOpen	Thing-in-container, GripperOpen
Op4	Place thing $\to$ drawer	Gripper-in-thing, Thing-in-cookware	Thing-in-drawer, $\lnot$Gripper-in-thing, GripperOpen	Gripper-in-thing, Thing-in-cookware
Op3	Place thing $\to$ cookware	Gripper-in-thing, Lid-in-cabinet	Thing-in-cookware, $\lnot$Gripper-in-thing, GripperOpen	Gripper-in-thing, Lid-in-cabinet
Op6	Place thing $\to$ container	Gripper-in-thing	Thing-in-container, $\lnot$Gripper-in-thing, GripperOpen	Gripper-in-thing

TABLE III: Learned Operators from play data: each couples symbolic transitions with SE(3) DS skills. Operators are arranged by semantic affinity.

算子	可被人理解的摘要	前置条件	效果	保持条件
Op7	从柜中拿起盖子	GripperOpen, Lid-in-cabinet	Gripper-in-lid, $\lnot$Lid-in-cabinet, $\lnot$GripperOpen	Lid-in-cabinet, GripperOpen
Op11	从炊具上拿起盖子	GripperOpen, Lid-in-cookware	Gripper-in-lid, $\lnot$Lid-in-cookware, $\lnot$GripperOpen	Lid-in-cookware, GripperOpen
Op1	把盖子放到柜中	Gripper-in-lid	Lid-in-cabinet, $\lnot$Gripper-in-lid, GripperOpen	Gripper-in-lid
Op8	把盖子放到炊具上	Gripper-in-lid	Lid-in-cookware, $\lnot$Gripper-in-lid, GripperOpen	Gripper-in-lid
Op9	从抽屉中拿起物品	GripperOpen, Thing-in-container, Thing-in-drawer	Gripper-in-thing, $\lnot$Thing-in-drawer, $\lnot$GripperOpen	Thing-in-container, Thing-in-drawer, GripperOpen
Op5	从炊具中拿起物品	GripperOpen, Lid-in-cabinet, Thing-in-cookware	Gripper-in-thing, $\lnot$Thing-in-cookware, $\lnot$GripperOpen	Thing-in-cookware, Lid-in-cabinet, GripperOpen
Op10	从容器中拿起物品	GripperOpen, Thing-in-container	Gripper-in-thing, $\lnot$Thing-in-container, $\lnot$GripperOpen	Thing-in-container, GripperOpen
Op4	把物品放到抽屉中	Gripper-in-thing, Thing-in-cookware	Thing-in-drawer, $\lnot$Gripper-in-thing, GripperOpen	Gripper-in-thing, Thing-in-cookware
Op3	把物品放到炊具中	Gripper-in-thing, Lid-in-cabinet	Thing-in-cookware, $\lnot$Gripper-in-thing, GripperOpen	Gripper-in-thing, Lid-in-cabinet
Op6	把物品放到容器中	Gripper-in-thing	Thing-in-container, $\lnot$Gripper-in-thing, GripperOpen	Gripper-in-thing

表 III：从玩乐式数据中学到的算子：每个算子都把符号转移与 SE(3) 动力系统技能耦合在一起。算子按语义相近程度排列。

SymSkill correctly segments trajectories and identifies the object in motion. The VLM is almost always able to determine the reference object correctly. For RoboCasa tasks, we take the identified ${o_{\textrm{int}}}$ and ${o_{\textrm{ref}}}$ and select the most frequent assignment across demonstrations, which yields perfect accuracy. Goal is specified by abstracting the symbolic state at the end of the majority of demonstrations. Failure cases arise primarily in PnP tasks, where the randomly generated containers are sometimes too tall (e.g., a salad bowl). In such cases, the arm carrying the item collides with the container, causing task failure.

SymSkill 能够正确切分轨迹，并识别正在运动的物体。视觉语言模型几乎总能正确判定参考物体。对 RoboCasa 任务，我们取已识别的 ${o_{\textrm{int}}}$ 和 ${o_{\textrm{ref}}}$，并在各条演示中选择最频繁的指派，从而得到完全正确的结果。目标通过抽象多数演示结束时的符号状态来指定。失败情形主要出现在拾放任务中，随机生成的容器有时过高（例如沙拉碗）。此时，带着物品的手臂会与容器相撞，导致任务失败。

With only 5–10 demonstrations per task, DP is severely data-limited. In particular, premotion skills start from widely varying initial poses but occupy only a narrow funnel near the target object, so test-time states quickly fall out of distribution and execution often fails before reaching the object. Some low-variance motion skills can be reproduced qualitatively, but task success remains near zero because both phases must succeed. In contrast, the SE(3) LPV-DS skill defines a convergent feedback field in the learned reference frame, enabling steady progress toward the goal under perturbations. We also evaluated DP with data augmentation from the DS policy, as detailed on the project website, but found no success with DP either. In contrast, the $\mathrm{SE}(3)$ LPV-DS controller induces a convergent vector field in the learned reference frame; its closed-loop stability prevents stalling and ensures steady progress to the goal even under perturbations.

每项任务只有 5–10 条演示时，扩散策略受到严重的数据限制。尤其是，预运动技能从差异很大的初始位姿出发，但在接近目标物体时只占一个很窄的漏斗，因此测试时的状态很快就会离开训练分布，执行也常常在到达物体前就失败。一些低方差的运动技能可以在外观上被复现，但由于两个阶段都必须成功，任务成功率仍接近零。相比之下，SE(3) LPV-DS 技能在学到的参考坐标系中定义了一个收敛的反馈场，使系统在扰动下仍能稳定朝目标推进。我们还评估了用动力系统策略做数据扩增后的扩散策略，细节见项目网站，但扩散策略仍然没有成功。与此相对，$\mathrm{SE}(3)$ LPV-DS 控制器在学到的参考坐标系中诱导出一个收敛的向量场；其闭环稳定性可以防止停滞，并保证即使在扰动下也能稳定朝目标推进。

For the symbol–skill co-invention baseline, we re-implemented NSIL [5] for qualitative comparison on OpenSingleDoor and PnPCounterToCab. In our low-data, multi-object setting, NSIL frequently selected spurious relative-pose predicates that explained the demonstrations but were not semantically meaningful for planning. The method was also sensitive to suboptimal demonstrations and slightly non-prehensile interactions, where useful contacts were often not included among the candidate predicates. As a result, NSIL failed to recover reusable predicates in these tasks, and the limited data was also insufficient for learning an effective low-level policy.

作为符号—技能共同发明的基线，我们在 OpenSingleDoor 和 PnPCounterToCab 上重新实现了 NSIL[5]，做定性比较。在我们这种低数据、多物体的设定中，NSIL 常常选出虽然能解释演示、但对规划并无语义意义的虚假相对位姿谓词。该方法对次优演示和略带非抓取性质的交互也很敏感，有用的接触往往没有被纳入候选谓词。因此，NSIL 未能在这些任务中恢复出可复用的谓词，有限的数据也不足以学到有效的底层策略。

VI-B Performing Multi-Step Task With No Additional Data

VI-B 在不增加新数据的情况下执行多步任务

We created a new task, StoreCheese, in RoboCasa. The task is successful when the robot picks the cheese from the cabinet, places it on the counter, and closes the cabinet door. To execute this task, we reuse only the previously learned predicates, operators, and skills from the constituent short-horizon tasks; we update operator preconditions via predicate evaluation across demonstrations. The operator from PnPCabToCounter task thus has the predicate OpenSingleDoor-RelPose(Door, Cabinet), meaning door being open, as a precondition (illustrated as ${}^{o_{\textrm{ref}}}\psi_{o_{\textrm{int}}}$ in Fig.1). We then manually specify the goal predicates as {CloseSingleDoor-RelPose(Door, Cabinet), PnPCabToCounter-RelPose(Cheese, Counter)}. With this setup, the Franka robot successfully plans the operator sequence: open the door, pick and place the cheese, and finally close the door. It completes the task by chaining together six skills and recovering from symbolic errors multiple times. Video of the experiment can be found on the project website.

我们在 RoboCasa 中创建了一项新任务 StoreCheese。当机器人从柜中取出奶酪放到台面上，并关上柜门时，任务成功。为执行这一任务，我们只复用组成该任务的短程任务中已经学到的谓词、算子和技能；我们通过对演示做谓词评估来更新算子的前置条件。因此，来自 PnPCabToCounter 任务的算子，把谓词 OpenSingleDoor-RelPose(Door, Cabinet)（表示门是开着的）作为前置条件（在图 1 中示意为 ${}^{o_{\textrm{ref}}}\psi_{o_{\textrm{int}}}$）。然后，我们手动把目标谓词指定为 {CloseSingleDoor-RelPose(Door, Cabinet)，PnPCabToCounter-RelPose(Cheese, Counter)}。在这一设定下，Franka 机器人成功规划出算子序列：开门、拾放奶酪，最后关门。它通过串起六项技能完成任务，并多次从符号错误中恢复。实验视频见项目网站。

Fig. 5: Real-world data collection pipeline. We use a motion capture system to record object interactions in the workspace. Here we show one motion episode with a sequence of timestamped images; the manipulated object (${o_{\textrm{int}}}$) is a banana. Frames with orange, yellow, and green banners denote the premotion, motion, and post-motion segments, respectively.

图 5：真实世界数据采集流程。我们用动作捕捉系统记录工作空间中的物体交互。这里展示一个运动片段及其带时间戳的图像序列；被操作物体（${o_{\textrm{int}}}$）是一根香蕉。带橙色、黄色和绿色条的帧，分别表示预运动、运动和后运动片段。

Fig. 6: Real-world execution of SymSkill toward the manually specified symbolic goal {RelPose(banana, plate)}. During execution, we introduce three disturbances to highlight the three recovery mechanisms of SymSkill: closing the lid triggers symbolic-level recovery, moving the plate needs needs no explicit recovery with DS skills, and adding an obstacle is handled by modulation. SymSkill successfully addresses all three and completes the task.

图 6：SymSkill 朝向手动指定的符号目标 {RelPose(banana, plate)} 的真实世界执行。执行过程中，我们引入三种扰动，以突出 SymSkill 的三种恢复机制：盖上盖子会触发符号层恢复，移动盘子时动力系统技能本身不需要显式恢复，加入障碍则由调制来处理。SymSkill 成功应对了这三种情形并完成任务。

VI-C Learning From Play In Real-World

VI-C 在真实世界中从玩乐式数据学习

We demonstrate our method can learn from play data in the real world. We set up a scene with block and banana (thing_type), red plate (drawer_type), white plate (container_type), dishrack (cabinet_type), lid (lid_type), and pan (cookware_type). During play data collection, the demonstrator uses a UMI gripper[31] to perform sequences of manipulation tasks such as closing the pan with a lid or placing the banana on a plate. We obtain the pose of objects and the gripper from a motion capture system and record the video data from a webcam. Fig. 5 shows the data collection process. Fig. 3 shows selecting reference frame process and the VLM prompt. We find that with minimal prompt engineering, VLM can correctly identify the reference frame ${o_{\textrm{ref}}}$, leading to correct learned predicates. Table III summarizes the learned operators from approximately 5 minutes of unsegmented real-world play. Our method learns semantically meaningful and logical operators from unsegmented data, such as recognizing that picking items from the pan requires first removing the lid to place it on the dishrack. An example is shown in Fig. 6.

我们展示该方法可以在真实世界中从玩乐式数据学习。我们布置的场景包括积木和香蕉（thing_type）、红盘子（drawer_type）、白盘子（container_type）、沥水架（cabinet_type）、盖子（lid_type）以及平底锅（cookware_type）。在采集玩乐式数据时，演示者使用 UMI 夹爪[31]完成一系列操作，例如用盖子盖住锅，或把香蕉放到盘子上。我们用动作捕捉系统获得物体和夹爪的位姿，并用网络摄像头记录视频。图 5 给出采集过程。图 3 给出选择参考坐标系的过程以及视觉语言模型提示。我们发现，只需很少的提示工程，视觉语言模型就能正确识别参考坐标系 ${o_{\textrm{ref}}}$，从而学到正确的谓词。表 III 汇总了从大约 5 分钟未切分的真实世界玩乐中学到的算子。我们的方法能够从未切分数据中学到语义上有意义、逻辑上也合理的算子，例如认识到从锅中取物前，要先把盖子放到沥水架上。一个例子见图 6。

We also observe non-obvious but semantically consistent preconditions. For example, Op9 includes ‘Thing-in-container’ as a precondition for picking from the red plate because this relation holds in all demonstrations. Although counterintuitive, this illustrates that SymSkill captures dataset-specific task structure directly from play rather than relying on generic language priors. We also demonstrate reacting to human external disturbance and recovering from failure in a OpenSingleDoor task. The video of the experiment is on the project website.

我们还观察到一些并不直观、但在语义上一致的前置条件。例如，Op9 把 “Thing-in-container” 作为从红盘子中拾取的前置条件，因为这一关系在所有演示中都成立。尽管这有些反直觉，但它说明 SymSkill 是直接从玩乐中捕获数据集特有的任务结构，而不是依赖一般的语言先验。我们还展示了在 OpenSingleDoor 任务中，对来自人的外部扰动作出反应并从失败中恢复。实验视频见项目网站。

VII Conclusion and Future Work

VII 结论与未来工作

We presented SymSkill, a symbol–skill co-invention framework that jointly learns relative-pose predicates for planning and DS-based skills for execution. Our results in simulation and on real robots show that SymSkill is significantly more sample-efficient and faster to learn than existing baselines, while enabling robust long-horizon manipulation. As future work, we plan to extend our framework to learn directly from egocentric video and to scale toward mobile manipulation scenarios, further broadening its applicability to real-world generalist robots.

我们提出了 SymSkill，这是一个符号—技能共同发明框架，同时学习用于规划的相对位姿谓词，以及用于执行的、基于动力系统的技能。仿真和真实机器人上的结果表明，与已有基线相比，SymSkill 明显更省样本，学习也更快，同时能够实现稳健的长程操作。作为未来工作，我们计划把框架扩展到直接从第一人称视频中学习，并走向移动操作场景，从而进一步拓宽它在真实世界通用机器人上的适用性。

Acknowledgment: We thank Bowen Li, Nishanth Kumar, Tom Silver and Rachel Holladay for the helpful discussions at various stages of the project. We thank Peng Qiu for helping out with setting up the simulator.

致谢：我们感谢 Bowen Li、Nishanth Kumar、Tom Silver 和 Rachel Holladay 在项目不同阶段的有益讨论。我们感谢 Peng Qiu 帮助搭建仿真器。

References

参考文献

[1] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2023) Diffusion policy: visuomotor policy learning via action diffusion . The International Journal of Robotics Research , pp. 02783649241273668 . Cited by: §I , §IV .

[2] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn (2023) Learning fine-grained bimanual manipulation with low-cost hardware . Proceedings of Robotics: Science and Systems (RSS) . Cited by: §I .

[3] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez (2021) Integrated task and motion planning . Annual review of control, robotics, and autonomous systems 4 ( 1 ), pp. 265–293 . Cited by: §I .

[4] N. Shah, J. Nagpal, P. Verma, and S. Srivastava (2024) From reals to logic and back: inventing symbolic vocabularies, actions, and models for planning from raw data . arXiv preprint arXiv:2402.11871 . Cited by: §I , TABLE I , §IV .

[5] L. Keller, D. Tanneberg, and J. Peters (2025) Neuro-symbolic imitation learning: discovering symbolic abstractions for skill learning . In IEEE International Conference on Robotics and Automation (ICRA) , Cited by: Appendix B , §I , §II , TABLE I , §IV , §VI-A .

[6] T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Pérez, L. Kaelbling, and J. B. Tenenbaum (2023) Predicate invention for bilevel planning . In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37 , pp. 12120–12129 . Cited by: §I , §II , §IV .

[7] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu (2024) RoboCasa: large-scale simulation of everyday tasks for generalist robots . In Robotics: Science and Systems , Cited by: §I , §VI .

[8] K. Kronander and A. Billard (2015) Passive interaction control with dynamical systems . IEEE Robotics and Automation Letters 1 ( 1 ), pp. 106–113 . Cited by: §II , §V-D .

[9] S. M. Khansari-Zadeh and A. Billard (2012) A dynamical system approach to realtime obstacle avoidance . Autonomous Robots 32 ( 4 ), pp. 433–454 . External Links: ISSN 1573-7527 , Document Cited by: §III-A , §V-E .

[10] N. Figueroa and A. Billard (2018) A physically-consistent bayesian non-parametric mixture model for dynamical system learning. . In CoRL , pp. 927–946 . Cited by: §III-A , §III-A .

[11] A. Billard, S. Mirrazavi, and N. Figueroa (2022) Learning for adaptive and reactive robot control: a dynamical systems approach . The MIT Press . External Links: ISBN 9780262046169 Cited by: §III-A , §III-A .

[12] S. Sun and N. Figueroa (2024) SE(3) linear parameter varying dynamical systems for globally asymptotically stable end-effector control . In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , Vol. , pp. 5152–5159 . External Links: Document Cited by: §III-A , §III-A , TABLE I .

[13] T. Li, S. Sun, S. S. Aditya, and N. Figueroa (2025) Elastic motion policy: an adaptive dynamical system for robust and efficient one-shot imitation learning . In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , Cited by: §III-A .

[14] C. R. Garrett, T. Lozano-Pérez, and L. P. Kaelbling (2020) Pddlstream: integrating symbolic planners and blackbox samplers via optimistic adaptive planning . In Proceedings of the international conference on automated planning and scheduling , Vol. 30 , pp. 440–448 . Cited by: §III-B .

[15] J. Mendez-Mendez, L. P. Kaelbling, and T. Lozano-Perez (2023) Embodied lifelong learning for task and motion planning . In Proceedings of the 7th Conference on Robot Learning (CoRL-23) , Cited by: §III-B .

[16] S. Cheng, C. R. Garrett, A. Mandlekar, and D. Xu (2024) NOD-tamp: generalizable long-horizon planning with neural object descriptors . In 8th Annual Conference on Robot Learning , Cited by: TABLE I , §IV .

[17] Z. Xue, S. Deng, Z. Chen, Y. Wang, Z. Yuan, and H. Xu (2025) DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning . In Proceedings of Robotics: Science and Systems , LosAngeles, CA, USA . External Links: Document Cited by: §IV .

[18] A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox (2023) MimicGen: a data generation system for scalable robot learning using human demonstrations . In 7th Annual Conference on Robot Learning , Cited by: §IV .

[19] C. Garrett, A. Mandlekar, B. Wen, and D. Fox (2024) SkillMimicGen: automated demonstration generation for efficient skill learning and deployment . In 8th Annual Conference on Robot Learning , Cited by: §IV .

[20] W. Wan, Y. Zhu, R. Shah, and Y. Zhu (2024) LOTUS: continual imitation learning for robot manipulation through unsupervised skill discovery . In 2024 IEEE International Conference on Robotics and Automation (ICRA) , Vol. , pp. 537–544 . External Links: Document Cited by: §IV .

[21] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar (2023) MimicPlay: long-horizon imitation learning by watching human play . In 7th Annual Conference on Robot Learning , Cited by: §IV .

[22] L. P. Kaelbling and T. Lozano-Pérez (2017) Learning composable models of parameterized skills . In 2017 IEEE International Conference on Robotics and Automation (ICRA) , pp. 886–893 . Cited by: §IV .

[23] G. Konidaris, L. P. Kaelbling, and T. Lozano-Perez (2018) From skills to symbols: learning symbolic representations for abstract high-level planning . Journal of Artificial Intelligence Research 61 , pp. 215–289 . Cited by: §IV .

[24] S. James and B. Rosman (2022) Autonomous learning of object-centric abstractions for high-level planning . In Proceedings of the Tenth International Conference on Learning Representations (ICLR) , External Links: Link Cited by: §IV .

[25] W. Liu, N. Nie, R. Zhang, J. Mao, and J. Wu (2024) BLADE: learning compositional behaviors from demonstration and language . In CoRL , Cited by: §IV .

[26] B. Li, T. Silver, S. Scherer, and A. Gray (2025) Bilevel Learning for Bilevel Planning . In Proceedings of the Robotics: Science and Systems (RSS) , Cited by: §IV .

[27] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitzmann (2022) Neural descriptor fields: se (3)-equivariant object representations for manipulation . In 2022 International Conference on Robotics and Automation (ICRA) , pp. 6394–6400 . Cited by: §IV .

[28] A. Athalye, N. Kumar, T. Silver, Y. Liang, T. Lozano-Pérez, and L. P. Kaelbling (2024) Predicate invention from pixels via pretrained vision-language models . arXiv preprint arXiv:2501.00296 . Cited by: §IV , §V-A .

[29] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities . arXiv preprint arXiv:2507.06261 . Cited by: §V-A .

[30] R. Chitnis, T. Silver, J. B. Tenenbaum, T. Lozano-Perez, and L. P. Kaelbling (2022) Learning neuro-symbolic relational transition models for bilevel planning . In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 4166–4173 . Cited by: §V-C .

[31] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song (2024) Universal manipulation interface: in-the-wild robot teaching without in-the-wild robots . In Proceedings of Robotics: Science and Systems (RSS) , Cited by: §VI-C .

Appendix A Online Algorithm

附录 A 在线算法

We show the pseudo-code of our online algorithm.

我们给出在线算法的伪代码。

1: Current State $\mathbf{x}_{0}$, Goal atoms $s_{\text{g}}$, Learned operators ${\Omega}$ (each operator $\alpha$ = $\langle$ pre, maintain, eff, skill $\rangle$), all objects $\mathcal{O}$
2: Initialize replan_count $\leftarrow 0$,$failmem\leftarrow\{\}$$\triangleright$ failure memory
3: CurrentPlan$(\alpha_{1},\ldots,\alpha_{n})\leftarrow\textsc{SymbolicPlanner}(s_{0},s_{g},\Omega)$
4: if $s_{g}$ unreachable or replan_count $\geq 20$ then
5:    return Failure
6: end if
7: $\alpha_{exe}\leftarrow\text{CurrentPlan}[i]$
8: $\alpha_{prev}\leftarrow\text{CurrentPlan}[i-1]$
9: if $\alpha_{exe}\in failmem$ then
10:    Goal Pose $\leftarrow$ sample(eff($\alpha_{exe}$)) $\triangleright$ Sample in ${}^{{o_{\textrm{int}}}}\psi_{ee}$ or ${}^{{o_{\textrm{ref}}}}\psi_{{o_{\textrm{int}}}}$
11: else
12:    Goal Pose $\leftarrow 0$
13: end if
14: while maintain$(\alpha_{exe})\in\text{abstract}(\mathbf{x})$ do
15:    $<f,g>\leftarrow$skill$(\alpha_{exe})$
16:    $f^{\prime}\leftarrow\mathbf{M}(\mathcal{O}_{-{o_{\textrm{int}}}})f$ $\triangleright$ Obstacle Avoidance
17:    if $f^{\prime}<\epsilon$ then $\triangleright$ Current skill ending
18:     if eff($\alpha_{exe}\notin\text{abstract}(\mathbf{x})$) then
19:       Add $(\alpha_{exe})$ to $failmem$ $\triangleright$ Current skill failure
20:       Go To Line: 3 $\triangleright$ Current skill success
21:     end if
22:     $i\leftarrow i+1$
23:     Go to Line: 7
24:    end if
25: end while
26: Add $(\alpha_{prev})$ to $failmem$ $\triangleright$ Failure of maintain predicates
27: Go To Line: 3

1: 当前状态 $\mathbf{x}_{0}$，目标原子 $s_{\text{g}}$，已学习算子 ${\Omega}$（每个算子 $\alpha$ = $\langle$ pre, maintain, eff, skill $\rangle$），全部物体 $\mathcal{O}$
2: 初始化 replan_count $\leftarrow 0$，$failmem\leftarrow\{\}$$\triangleright$ 失败记忆
3: CurrentPlan$(\alpha_{1},\ldots,\alpha_{n})\leftarrow\textsc{SymbolicPlanner}(s_{0},s_{g},\Omega)$
4: 如果 $s_{g}$ 不可达或 replan_count $\geq 20$ 则
5:    返回 失败
6: 结束 如果
7: $\alpha_{exe}\leftarrow\text{CurrentPlan}[i]$
8: $\alpha_{prev}\leftarrow\text{CurrentPlan}[i-1]$
9: 如果 $\alpha_{exe}\in failmem$ 则
10:    目标位姿 $\leftarrow$ sample(eff($\alpha_{exe}$)) $\triangleright$ 在 ${}^{{o_{\textrm{int}}}}\psi_{ee}$ 或 ${}^{{o_{\textrm{ref}}}}\psi_{{o_{\textrm{int}}}}$ 中采样
11: 否则
12:    目标位姿 $\leftarrow 0$
13: 结束 如果
14: 当 maintain$(\alpha_{exe})\in\text{abstract}(\mathbf{x})$ 时
15:    $<f,g>\leftarrow$skill$(\alpha_{exe})$
16:    $f^{\prime}\leftarrow\mathbf{M}(\mathcal{O}_{-{o_{\textrm{int}}}})f$ $\triangleright$ 避障
17:    如果 $f^{\prime}<\epsilon$ 则 $\triangleright$ 当前技能即将结束
18:     如果 eff($\alpha_{exe}\notin\text{abstract}(\mathbf{x})$) 则
19:       把 $(\alpha_{exe})$ 加入 $failmem$ $\triangleright$ 当前技能失败
20:       转到第 3 行 $\triangleright$ 当前技能成功
21:     结束 如果
22:     $i\leftarrow i+1$
23:     转到第 7 行
24:    结束 如果
25: 结束 当
26: 把 $(\alpha_{prev})$ 加入 $failmem$ $\triangleright$ 保持谓词失败
27: 转到第 3 行

Appendix B Extended Analysis of [5]

附录 B 对[5]的补充分析

Difficulty 1: Assumption of Optimal Demonstrations.

NSIL requires that candidate predicate sets yield plans whose length matches the demonstration length. In practice, demonstrations in RoboCasa often contain suboptimal behaviors, such as multiple approaches before grasping. For instance, in OpenSingleDoor or PnPCounterToCab, demonstrations sometimes include repeated approaches, leading to longer trajectories than the optimal plan. As a result, valid predicate sets are rejected because the demonstration is not optimal. We relaxed this by introducing a penalty for mismatched lengths rather than strict rejection, but the issue remains fundamental.

困难 1：对最优演示的假定。

NSIL 要求候选谓词集合产生的计划长度与演示长度一致。实践中，RoboCasa 中的演示常常包含次优行为，例如在抓取前多次接近。例如，在 OpenSingleDoor 或 PnPCounterToCab 中，演示有时包含重复接近，使轨迹长于最优计划。结果是，有效的谓词集合会因演示并非最优而被拒绝。我们通过引入长度不匹配的惩罚、而不是严格拒绝来放宽这一点，但问题仍然是根本性的。

Difficulty 2: Ambiguity from Distractor Objects.

Including irrelevant objects significantly increases ambiguity. For a simple pick-and-place task with one distractor, low-speed analysis generated 13 candidate predicates. Beam search often selected fragile ones, e.g., $\text{RelPose}(\text{DoorHandle},\text{Object})$, because incorrect relationships (object to door) has the same cost as semantically meaningful ones (object to cabinet). Such predicates fail to generalize if the distractor moves. Our method mitigates this by using VLM-based semantic grounding to select reference objects, bypassing spurious associations.

困难 2：干扰物体带来的歧义。

纳入不相关物体，会显著增加歧义。对一个带有一个干扰物的简单拾放任务，低速分析产生了 13 个候选谓词。束搜索常常选出脆弱的谓词，例如 $\text{RelPose}(\text{DoorHandle},\text{Object})$，因为不正确的关系（物体到门）与语义上有意义的关系（物体到柜子）具有相同的代价。如果干扰物发生移动，这类谓词就无法泛化。我们的方法通过基于视觉语言模型的语义锚定来选择参考物体，从而绕开这些虚假关联。

Difficulty 3: Noisy Non-Prehensile Interactions.

For articulated objects, grasping is frequently non-prehensile (e.g., pushing a door handle rather than firmly holding it). This makes $\text{RelPose}(\text{Gripper},\text{Handle})$ a noisy predicate and artificially inflates the demonstration symbolic sequence length. In our trials with four demonstrations of door opening, the optimization incorrectly favored gripper-to-cabinet relations, which do not reflect the actual manipulation. This highlights NSIL’s difficulty in capturing non-prehensile skills robustly.

困难 3：带噪声的非抓取式交互。

对铰接物体来说，抓取常常是非抓取式的（例如推动门把手，而不是牢牢握住它）。这使 $\text{RelPose}(\text{Gripper},\text{Handle})$ 成为一个带噪声的谓词，并人为拉长了演示的符号序列长度。在我们用四条开门演示做的试验中，优化错误地偏向夹爪与柜子之间的关系，而这并不反映实际操作。这凸显了 NSIL 在稳健刻画非抓取式技能上的困难。

Summary.

Overall, NSIL’s reliance on optimal demonstrations, its vulnerability to distractors, and its fragility in non-prehensile settings limit its robustness in realistic environments such as RoboCasa. By contrast, our framework leverages semantic grounding (via VLMs) for predicate discovery and SE(3) DS policies with stability guarantees for skill execution, making it more robust under limited demonstrations and realistic variations.

小结。

总体来看，NSIL 对最优演示的依赖、对干扰物的脆弱，以及在非抓取设定中的不稳健，限制了它在 RoboCasa 这类真实环境中的表现。相比之下，我们的框架借助语义锚定（通过视觉语言模型）来发现谓词，并用带稳定性保证的 SE(3) 动力系统策略来执行技能，因而在演示有限、变化更贴近现实时更加稳健。

Appendix C DP with Data Augmentation

附录 C 带数据扩增的扩散策略

We used the trained $\mathrm{SE}(3)$ LPV-DS policies to generate 100 rollouts for each skill as additional training data for the diffusion policy (DP). Initial end-effector poses were sampled from an $\mathrm{SE}(3)$ Gaussian fit to the initial poses in the training set. Example training data, augmented data, and a rollout generated by the trained DP are shown in Figure 7.

我们用训练好的 $\mathrm{SE}(3)$ LPV-DS 策略为每项技能生成 100 条展开，作为扩散策略（DP）的额外训练数据。初始末端执行器位姿从拟合训练集约初始位姿的 $\mathrm{SE}(3)$ 高斯中采样。示例训练数据、扩增数据，以及由训练好的扩散策略生成的一条展开，见图 7。

We used trained $\mathrm{SE}(3)$ LPV-DS policies to generate rollouts as additional training data for diffusion policy (DP). Initial end-effector poses were sampled from an $\mathrm{SE}(3)$ Gaussian fit to the initial poses in the training set. All training data, augmented data, and a rollout from the trained DP are shown in Figure 7.

我们用训练好的 $\mathrm{SE}(3)$ LPV-DS 策略生成展开，作为扩散策略（DP）的额外训练数据。初始末端执行器位姿从拟合训练集约初始位姿的 $\mathrm{SE}(3)$ 高斯中采样。全部训练数据、扩增数据，以及训练好的扩散策略给出的一条展开，见图 7。

Fig. 7: DP rollout for the premotion segment of OpenSingleDoor. Purple trajectories are 4 training demonstrations used for SymSkill, yellow trajectories are augmented data generated by $\mathrm{SE}(3)$ LPV-DS, and the black curve is a rollout from the trained DP. The poses (red, green, blue axes) indicate end-effector orientation at several timestamps; the larger pose denotes the averaged final orientation across all demonstrations.

图 7：OpenSingleDoor 预运动片段上的扩散策略展开。紫色轨迹是用于 SymSkill 的 4 条训练演示，黄色轨迹是由 $\mathrm{SE}(3)$ LPV-DS 生成的扩增数据，黑色曲线是训练好的扩散策略给出的一条展开。位姿（红、绿、蓝轴）标出若干时刻的末端执行器朝向；较大的位姿表示所有演示上平均后的最终朝向。

We evaluated DP with data augmentation in RoboCasa. Although the rollout in Figure 7 appears successful, we observed degraded orientation performance in simulation, leading to a zero success rate. We hypothesize this is due to the robot’s kinematic constraints, which push the end-effector into regions outside the training distribution. DP succeeds on relatively simple tasks (e.g., CloseDrawer) that do not require precise orientation control, but fails on tasks such as OpenSingleDoor and PnPCounterToCab, where the policy consistently approaches the ${o_{\textrm{int}}}$ but cannot achieve the grasp orientation required for success.

我们在 RoboCasa 中评估了带数据扩增的扩散策略。尽管图 7 中的展开看起来是成功的，我们在仿真中观察到朝向表现变差，成功率因此为零。我们猜想，这是由于机器人的运动学约束把末端执行器推到了训练分布之外的区域。扩散策略在相对简单、不需要精确朝向控制的任务上可以成功（例如 CloseDrawer），但在 OpenSingleDoor 和 PnPCounterToCab 这类任务上会失败：策略能够持续接近 ${o_{\textrm{int}}}$，却无法达到成功所需的抓取朝向。