FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control
FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control
FlashSAC:面向高维机器人控制的快速、稳定 Off-Policy 强化学习
Holiday Robotics KAIST KRAFTON Turing Inc TU Darmstadt hessian.AI KTH Royal Institute of Technology German Research Center for AI (DFKI) Robotics Institute Germany (RIG)
- equal contribution
Correspondence to: Hojoon Lee hojoon.lee@holiday-robotics.com
Abstract
摘要
Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable. On-policy methods such as Proximal Policy Optimization (PPO) are widely used for their stability, but their reliance on narrowly distributed on-policy data limits accurate policy evaluation in high-dimensional state and action spaces. Off-policy methods can overcome this limitation by learning from a broader state-action distribution, yet suffer from slow convergence and instability, as fitting a value function over diverse data requires many gradient updates, causing critic errors to accumulate through bootstrapping. We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic. Motivated by scaling laws observed in supervised learning, FlashSAC sharply reduces gradient updates while compensating with larger models and higher data throughput. To maintain stability at increased scale, FlashSAC explicitly bounds weight, feature and gradient norms, curbing critic error accumulation. Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency, with the largest gains on high-dimensional tasks such as dexterous manipulation. In sim-to-real humanoid locomotion, FlashSAC reduces training time from hours to minutes, demonstrating the promise of off-policy RL for sim-to-real transfer.
在缺少专家示范时,强化学习(RL)是机器人控制的一条核心路径。Proximal Policy Optimization(PPO)这类 on-policy 方法因为训练稳定而被广泛使用,但它们依赖分布较窄的 on-policy 数据,在高维状态与动作空间里很难准确评估策略。Off-policy 方法可以从更广的状态–动作分布中学习,从而克服这一限制,却又常常收敛慢、训练不稳:要在多样化数据上拟合价值函数,往往需要大量梯度更新,critic 误差会经自举不断累积。我们提出 FlashSAC,一种建立在 Soft Actor-Critic 之上、既快又稳的 off-policy 强化学习算法。受监督学习中缩放规律的启发,FlashSAC 大幅减少梯度更新次数,改用更大的模型和更高的数据通量来补偿。为在更大规模下保持稳定,它显式约束权重、特征与梯度范数,抑制 critic 误差累积。在 10 个仿真器、60 余项任务上,FlashSAC 在最终性能和训练效率上都稳定优于 PPO 以及较强的 off-policy 基线,提升最明显的是灵巧操作这类高维任务。在人形运动的仿真到真机迁移中,它把训练时间从数小时压到数分钟,显示出 off-policy 强化学习用于仿真到真机迁移的潜力。
Project page: https://holiday-robot.github.io/FlashSAC
Public code: https://github.com/Holiday-Robot/FlashSAC
Fig. 1: Results Overview. Tasks grouped by state–action dimensionality, with representative examples shown for each category. (a) State-based, Low DoF: Gripper manipulation and quadruped locomotion tasks from IsaacLab, ManiSkill, and Genesis. In low-dimensional settings, FlashSAC achieves performance comparable to PPO. (b) State-based, High DoF: Dexterous manipulation and humanoid locomotion tasks from IsaacLab and MuJoCo Playground. In high-dimensional settings, FlashSAC substantially outperforms PPO in both asymptotic return and wall-clock efficiency. (c) Sim-to-Real: Humanoid locomotion on the Unitree G1 platform. FlashSAC enables sim-to-real transfer within minutes, whereas PPO requires hours of training.
图 1:结果总览。任务按状态–动作维数分组,每类给出代表性示例。(a)基于状态、低自由度:来自 IsaacLab、ManiSkill 和 Genesis 的夹爪操作与四足运动。在低维设定下,FlashSAC 的表现与 PPO 相当。(b)基于状态、高自由度:来自 IsaacLab 和 MuJoCo Playground 的灵巧操作与人形运动。在高维设定下,FlashSAC 在渐近回报和墙钟效率上都明显超过 PPO。(c)仿真到真机:Unitree G1 平台上的人形运动。FlashSAC 能在数分钟内完成仿真到真机迁移,而 PPO 需要数小时训练。
I. Introduction
I. 引言
The long-standing goal of robot learning is to develop agents that generalize across a wide range of tasks in the real world. While large-scale imitation learning from real-world data has recently yielded impressive results in robotic control [38, 97], reinforcement learning (RL) from simulation remains a core paradigm when expert demonstrations are unavailable, incomplete, or insufficient [39, 3, 114].
机器人学习长期以来的目标,是得到能在真实世界中跨多种任务泛化的智能体。大规模真实示范上的模仿学习最近在机器人控制中取得了令人印象深刻的结果 [38, 97],但当专家示范缺失、不完整或不充分时,从仿真中做强化学习(RL)仍是一条核心路径 [39, 3, 114]。
To date, sim-to-real RL has been most successful in relatively constrained domains such as quadruped locomotion [32, 81] and gripper-based manipulation [2], which are characterized by low-dimensional state–action spaces and extremely high-throughput simulators [95, 61, 4, 112]. In this regime, on-policy methods such as Proximal Policy Optimization (PPO) [85, 86] have proven effective: PPO is stable, easy to tune, and its data inefficiency is acceptable when fresh on-policy data can be collected cheaply.
迄今为止,仿真到真机的强化学习最成功的,还是四足运动 [32, 81] 和夹爪操作 [2] 这类相对受限的领域:状态–动作空间维度较低,仿真器通量又极高 [95, 61, 4, 112]。在这一体制下,Proximal Policy Optimization(PPO)[85, 86] 这类 on-policy 方法被证明很有效:PPO 稳定、易调参,而且只要能便宜地采到新的 on-policy 数据,样本效率偏低也还可以接受。
However, this regime is becoming less representative of modern robot learning. Emerging applications—including humanoid locomotion [89], dexterous manipulation [11, 108], and vision-based control [66, 36]—involve much higher-dimensional state and action spaces, where policy evaluation and improvement from narrowly distributed on-policy data become substantially harder. Simultaneously, simulation grows increasingly expensive due to complex contact dynamics [52, 109], and larger policy architectures further raise rollout costs [38, 97]. In this setting, repeatedly discarding past experience in favor of freshly collected data becomes increasingly inefficient in both sample complexity and wall-clock time.
不过,这一体制已经越来越不像当今机器人学习的典型场景。人形运动 [89]、灵巧操作 [11, 108] 和基于视觉的控制 [66, 36] 等新应用,状态和动作空间的维数要高得多,仅靠分布很窄的 on-policy 数据去做策略评估与改进会困难得多。与此同时,复杂接触动力学让仿真越来越昂贵 [52, 109],更大的策略网络又进一步抬高了 rollout 成本 [38, 97]。在这种条件下,反复丢掉过往经验、只保留新采到的数据,在样本复杂度和墙钟时间上都越来越不划算。
Off-policy RL offers a natural alternative. By reusing diverse experience from a replay buffer, off-policy methods can achieve substantially higher data efficiency than on-policy approaches [62, 51, 22]. This advantage is particularly appealing in high-dimensional robotic tasks, where broader data coverage can support better policy evaluation and improvement. Yet despite this promise, off-policy RL has not become the default choice for sim-to-real transfer, as it often suffers from slow training and instability [17, 47].
Off-policy 强化学习提供了一条自然的替代路径。通过在回放缓冲区中复用多样化经验,off-policy 方法往往能比 on-policy 方法高得多的数据效率 [62, 51, 22]。在高维机器人任务里,更广的数据覆盖有助于更好地评估和改进策略,这一优势尤其有吸引力。尽管如此,off-policy 强化学习仍未成为仿真到真机迁移的默认选择,因为它常常训练缓慢且不稳定 [17, 47]。
A central challenge is learning an accurate value function from broad replay data. Off-policy methods train a critic by minimizing a bootstrapped Bellman objective,
一个核心困难,是从宽分布的回放数据中学到准确的价值函数。Off-policy 方法通过最小化带自举的 Bellman 目标来训练 critic ,
where transitions are sampled from a replay buffer and . In high-dimensional settings, fitting this critic accurately over diverse replay data often requires many gradient updates, which not only increases training time but also compounds estimation errors through repeated bootstrapping, as the critic is optimized toward targets that depend on its own predictions [94, 101].
其中转移 从回放缓冲区 中采样,且 。在高维设定下,要在多样化的回放数据上把这个 critic 拟合准确,往往需要很多次梯度更新;这不但拉长训练时间,还会因反复自举而放大估计误差——因为 critic 是在朝着依赖于自身预测的目标做优化 [94, 101]。
In this paper, we present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic. Motivated by scaling trends in supervised learning, FlashSAC sharply reduces the number of gradient updates while compensating with larger models and higher data throughput, improving training efficiency and better matching modern large-scale simulation pipelines. However, larger critics can further exacerbate instability under bootstrapping. To maintain stability, FlashSAC explicitly controls critic update dynamics by bounding weight, feature, and gradient norms, thereby preventing the accumulation of critic errors.
本文提出 FlashSAC,一种建立在 Soft Actor-Critic 之上、既快又稳的 off-policy 强化学习算法。受监督学习中缩放趋势的启发,FlashSAC 大幅减少梯度更新次数,改用更大的模型和更高的数据通量来补偿,从而提高训练效率,也更契合当代大规模仿真流水线。不过,更大的 critic 在自举下可能让不稳定性进一步加重。为保持稳定,FlashSAC 通过约束权重、特征与梯度范数,显式控制 critic 的更新动态,从而防止 critic 误差累积。
We evaluate FlashSAC on more than 60 locomotion and manipulation tasks across 10 simulators, spanning high-dimensional state-based control, vision-based control, and sim-to-real humanoid locomotion. Across this benchmark suite, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency, with the largest gains on the most challenging tasks such as dexterous manipulation and humanoid locomotion. In sim-to-real humanoid walking, FlashSAC reduces training time from hours to minutes while maintaining stable real-world deployment, demonstrating that off-policy RL can be both fast and stable for scalable sim-to-real robot learning.
我们在 10 个仿真器、60 余项运动与操作任务上评估 FlashSAC,覆盖高维基于状态的控制、基于视觉的控制,以及人形运动的仿真到真机迁移。在这一基准套件上,FlashSAC 在最终性能和训练效率上都稳定优于 PPO 以及较强的 off-policy 基线,提升最大的是灵巧操作和人形运动这类最难的任务。在人形行走的仿真到真机迁移中,FlashSAC 把训练时间从数小时压到数分钟,同时还能稳定部署到真实世界,说明 off-policy 强化学习可以既快又稳,支撑可扩展的仿真到真机机器人学习。
II. Related Work
II. 相关工作
A. On-Policy Reinforcement Learning
A. On-Policy 强化学习
On-policy RL has been the dominant paradigm for simulation-based robot learning when environment interaction is cheap and massively parallelizable. Among on-policy methods, PPO [85] is particularly popular for its stability, ease of implementation, and robustness to hyperparameter choices. Combined with modern high-throughput simulators [58, 95, 61, 4], PPO has enabled successful sim-to-real transfer in relatively constrained domains such as quadruped locomotion [32, 81, 86] and rigid-body manipulation [2]. Its widespread adoption has motivated a line of work improving upon it, such as stronger trust-region guarantees [110], better exploration via expanding action space [50], and lower gradient variance via pathwise estimates [107].
当环境交互便宜、而且可以大规模并行时,on-policy 强化学习一直是仿真机器人学习的主导范式。在 on-policy 方法中,PPO [85] 因其稳定、实现简单、对超参不那么敏感而尤为常用。结合当代高通量仿真器 [58, 95, 61, 4],PPO 已在四足运动 [32, 81, 86] 和刚体操作 [2] 这类相对受限的领域实现了成功的仿真到真机迁移。它的广泛采用也带动了一系列改进工作,例如更强的信赖域保证 [110]、通过扩展动作空间改善探索 [50],以及用路径式估计降低梯度方差 [107]。
However, on-policy methods fundamentally rely on freshly collected data and discard experience generated by earlier policies [94]. As task dimensionality increases, achieving sufficient state–action coverage via on-policy rollouts becomes increasingly expensive. While importance sampling can, in principle, correct for policy mismatch and enable data reuse [90, 57], importance weights in high-dimensional continuous action spaces exhibit extremely high variance, rendering this strategy impractical in modern robotic learning.
然而,on-policy 方法本质上依赖新采集的数据,并丢弃早期策略产生的经验 [94]。随着任务维数升高,仅靠 on-policy rollout 获得足够的状态–动作覆盖会越来越贵。重要性采样原则上可以校正策略不一致、从而复用数据 [90, 57],但在高维连续动作空间里,重要性权重的方差极高,这使得这一策略在当代机器人学习中并不实用。
B. Off-Policy Reinforcement Learning
B. Off-Policy 强化学习
Off-policy RL decouples data collection from policy optimization by storing transitions in a replay buffer and reusing them across updates [94, 62]. This is especially appealing in high-dimensional robotic tasks, where diverse experience supports better policy evaluation than narrowly distributed on-policy rollouts.
Off-policy 强化学习把数据采集与策略优化解耦:把转移存进回放缓冲区,并在多次更新中复用 [94, 62]。在高维机器人任务中,这一点尤其有吸引力,因为多样化经验比分布很窄的 on-policy rollout 更能支撑策略评估。
Off-Policy Model-Based RL: Model-based RL further improves sample efficiency by learning environment dynamics and using them for planning or imagined rollouts [93, 34]. Recent approaches such as DreamerV3 [23] and TD-MPC2 [24] have demonstrated strong performance in vision-based domains by planning in learned latent spaces. However, learning accurate dynamics models and performing repeated planning procedures significantly increases per-step training cost [56], often limiting their scalability in wall-clock critical settings.
Off-Policy 基于模型的强化学习:基于模型的强化学习通过学习环境动力学,并用于规划或想象 rollout,从而进一步提高样本效率 [93, 34]。DreamerV3 [23] 和 TD-MPC2 [24] 等近期方法在学到的潜空间中做规划,已在视觉领域展示出很强的表现。不过,学习准确的动力学模型并反复执行规划,会显著增加每步训练成本 [56],在墙钟时间吃紧的场景里,可扩展性往往受限。
Off-Policy Model-Free RL: Model-free off-policy algorithms such as DDPG [51], TD3 [17], and SAC [22] learn policies and value functions directly from replayed experience without explicit dynamics models. Their simplicity and data reuse make them attractive for robotic control. However, as discussed in Section I, off-policy model-free RL suffers from three persistent challenges: slow training, unstable training dynamics, and exploration in high-dimensional action spaces. The first two challenges stem from the bootstrapped Bellman objective illustrated in Equation 1: fitting a critic over diverse replay data in high-dimensional state-action spaces requires many gradient updates, directly increasing training time. Because critic targets depend on the critic’s own predictions, approximation and extrapolation errors at poorly supported state-action pairs compound across updates [94, 101]. The third arises because the maximum-entropy formulation of SAC alone is often insufficient to maintain coherent exploration in high-dimensional action spaces [14, 29], motivating dedicated noise mechanisms and exploration schemes.
Off-Policy 无模型强化学习:DDPG [51]、TD3 [17] 和 SAC [22] 这类无模型 off-policy 算法,不显式建模动力学,直接从回放经验中学习策略和价值函数。它们形式简单、又能复用数据,对机器人控制很有吸引力。但如第 I 节所述,无模型 off-policy 强化学习仍面临三个持续存在的挑战:训练慢、训练动态不稳定,以及高维动作空间中的探索。前两个困难来自公式 (1) 所示的自举 Bellman 目标:要在高维状态–动作空间里、对多样化回放数据拟合 critic,需要大量梯度更新,训练时间会直接被拉长。又因为 critic 的目标依赖于它自己的预测,在支撑不足的状态–动作对上,逼近误差和外推误差会随更新不断叠加 [94, 101]。第三个困难则是:仅靠 SAC 的最大熵形式,往往不足以在高维动作空间里维持连贯的探索 [14, 29],因而需要专门的噪声机制和探索方案。
Prior work has primarily addressed each challenge in isolation. To improve speed, one line of work scales data throughput via parallel simulation and large replay buffers [48, 77, 78, 88, 87, 70]. For example, FastTD3 [88] and FastSAC [87] achieve strong wall-clock efficiency in humanoid locomotion but relies on small networks (0.2M parameters), which limits its asymptotic performance. Scaling to larger networks is difficult in this setting, as increased model capacity exacerbates instability under bootstrapped training.
已有工作大多是分别应对其中某一个挑战。为了提高速度,一条路线借助并行仿真和大型回放缓冲区来扩大数据通量 [48, 77, 78, 88, 87, 70]。例如,FastTD3 [88] 和 FastSAC [87] 在人形运动上取得了很强的墙钟效率,但依赖大约 0.2M 参数的小网络,限制了渐近性能。在这种设定下把网络做大并不容易,因为模型容量增加会加重自举训练下的不稳定性。
To improve stability, a second line of work constrains value-function sensitivity by bounding feature, weight, and gradient norms [8, 42, 20, 67, 44, 45, 55, 71, 72, 68, 15, 31, 103], or by other measures such as reinitialization [69, 91], distillation [105], ensembling [10, 27], and alternative critic target networks [106, 26]. These constraints limit error amplification under distribution shift and repeated bootstrapping, enabling training with larger networks that achieve higher asymptotic performance. However, the increased model capacity requires more gradient updates to converge, resulting in slower training in data-rich simulation regimes.
为了提高稳定性,第二条路线通过约束特征、权重和梯度范数,来限制价值函数的敏感度 [8, 42, 20, 67, 44, 45, 55, 71, 72, 68, 15, 31, 103],或采用重新初始化 [69, 91]、蒸馏 [105]、集成 [10, 27] 以及替代的 critic 目标网络 [106, 26] 等其他手段。这些约束能限制分布偏移和反复自举下的误差放大,从而允许训练更大的网络,得到更高的渐近性能。不过,模型容量增大后,收敛往往需要更多梯度更新,在数据充裕的仿真体制下,训练反而更慢。
To improve exploration, a third line of work exploits off-policy RL’s ability to decouple data collection from policy optimization, injecting temporally-correlated action noise to obtain coherent trajectories such as Ornstein–Uhlenbeck processes [29], pink noise [14], parameter-space noise [75], state-dependent exploration [80, 79], and temporally-extended action repetition [12]. However, exploration mechanisms alone leave the bottlenecks of slow and unstable training untouched, limiting their impact as model capacity and data throughput scale.
为了改善探索,第三条路线利用 off-policy 强化学习把数据采集与策略优化解耦的能力,注入时间相关的动作噪声,以得到连贯轨迹,例如 Ornstein–Uhlenbeck 过程 [29]、粉红噪声 [14]、参数空间噪声 [75]、依赖状态的探索 [80, 79],以及时间上延展的动作重复 [12]。但仅靠探索机制,并没有触及训练慢和不稳定这两个瓶颈,因而在模型容量和数据通量扩大时,它们的作用仍然有限。
FlashSAC unifies these directions: it achieves fast training by sharply reducing gradient updates while scaling model capacity and data throughput, maintains stable training dynamics by jointly bounding weight, feature, and gradient norms, and adopts a lightweight noise-repetition scheme that produces temporally-correlated exploration without per-environment overhead.
FlashSAC 把这几条方向统一起来:它通过大幅减少梯度更新、同时扩大模型容量和数据通量,实现快速训练;通过联合约束权重、特征与梯度范数,维持稳定的训练动态;并采用一种轻量的噪声重复方案,在不增加按环境开销的前提下,产生时间相关的探索。
III. Preliminary
III. 预备知识
In this section, we introduce the RL framework and algorithmic foundation upon which FlashSAC is built.
本节介绍 FlashSAC 所依据的强化学习框架与算法基础。
A. Markov Decision Process (MDP)
A. 马尔可夫决策过程(MDP)
We model robotic control as a discounted Markov Decision Process (MDP), , where denotes the state space, denotes the continuous action space, denotes the transition dynamics, denotes the reward function, and is the discount factor.
我们将机器人控制建模为带折扣的马尔可夫决策过程(MDP),其中 为状态空间, 为连续动作空间, 为转移动力学, 为奖励函数, 为折扣因子。
At each timestep , the agent observes , samples an action , receives a reward , and transitions to the next state . The goal is to learn a policy that maximizes the discounted sum of rewards.
在每个时间步 ,智能体观测到 ,采样动作 ,得到奖励 ,并转移到下一状态 。目标是学习策略 ,使折扣奖励之和最大化。
B. Soft Actor Critic (SAC)
B. Soft Actor-Critic(SAC)
FlashSAC builds upon SAC [22], a widely used off-policy RL algorithm. SAC stores transitions collected under past policies in a replay buffer , and trains the policy using samples drawn from this buffer.
FlashSAC 建立在被广泛使用的 off-policy 算法 SAC [22] 之上。SAC 把过往策略采集到的转移 存入回放缓冲区 ,并用从该缓冲区抽出的样本来训练策略。
Beyond maximizing expected return, SAC incorporates an entropy regularization term that encourages exploration. This entropy maximization is particularly important in high-dimensional state–action spaces, where insufficient exploration can lead to poor coverage of the replay buffer and exacerbate approximation and extrapolation errors.
除了最大化期望回报,SAC 还加入一项鼓励探索的熵正则。在高维状态–动作空间里,这种熵最大化尤其重要:探索不足会使回放缓冲区覆盖变差,并加重逼近误差和外推误差。
To reduce approximation errors in bootstrapped value learning, SAC commonly employs clipped double Q-learning [17], maintaining two action-value functions and . The minimum of the two estimates is used when forming targets, reducing the impact of optimistic value errors.
为减少自举价值学习中的逼近误差,SAC 通常采用 clipped double Q-learning [17],同时维护两个动作价值函数 和 。构造目标时取二者的较小值,以降低乐观价值误差的影响。
Concretely, the policy is optimized by minimizing
具体而言,策略 通过最小化下式来优化
where controls the relative importance of entropy.
其中 控制熵项的相对重要程度。
Each critic is trained by minimizing a bootstrapped Bellman error using slowly updated target networks and , which are updated via exponential moving average:
每个 critic 通过最小化带自举的 Bellman 误差来训练,并使用缓慢更新的目标网络 和 ,后者按指数滑动平均更新:
where is the target update rate.
其中 是目标网络的更新速率。
For , the critic weights are optimized by minimizing the Bellman loss
对 ,critic 权重 通过最小化 Bellman 损失来优化
where the target value is
其中目标值为
IV. FlashSAC
FlashSAC is a fast and stable off-policy RL algorithm for high-dimensional robotic control. It achieves strong asymptotic performance with fast wall-clock time through three complementary mechanisms: (i) fast training by scaling data and model while reducing gradient updates (§IV-A), (ii) stable training by constraining critic update dynamics (§IV-B), and (iii) broad exploration for diverse data coverage (§IV-C).
FlashSAC 是一种面向高维机器人控制、既快又稳的 off-policy 强化学习算法。它通过三项互补机制,在较短墙钟时间内达到较强的渐近性能:(i)扩大数据与模型规模、同时减少梯度更新,从而实现快速训练(§IV-A);(ii)约束 critic 的更新动态,从而实现稳定训练(§IV-B);(iii)进行广泛探索,以覆盖多样化数据(§IV-C)。
A. Fast Training
A. 快速训练
On-policy methods such as PPO discard all collected data after each iteration. In high-dimensional robotic tasks where simulation is expensive, this data inefficiency becomes a critical bottleneck. Off-policy RL reuses past experience from a replay buffer, but conventionally requires many gradient updates per transition to extract sufficient learning signal, which slows wall-clock time and compounds bootstrapping errors.
PPO 这类 on-policy 方法会在每次迭代后丢弃全部已采集数据。在仿真昂贵的高维机器人任务中,这种数据浪费会成为关键瓶颈。Off-policy 强化学习会从回放缓冲区复用过往经验,但按常规做法,每条转移往往要做很多次梯度更新才能抽出足够的学习信号,既拖慢墙钟时间,又会叠加自举误差。
FlashSAC takes a different approach inspired by the scaling trends observed in supervised learning: under a fixed compute budget, larger models trained with larger batches and fewer updates converge faster than smaller models with frequent updates [37]. This principle has been difficult to apply in off-policy RL, because increased model capacity tends to amplify critic instability under bootstrapping. FlashSAC resolves this tension by stabilizing critic training through constrained update dynamics (§IV-B), enabling a regime of high data throughput, large models, and infrequent gradient updates.
FlashSAC 采取了另一条受监督学习缩放趋势启发的路线:在固定计算预算下,用更大模型、更大 batch、更少更新去训练,往往比小模型频繁更新收敛得更快 [37]。这一原则很难直接搬到 off-policy 强化学习里,因为模型容量增加,往往会加重自举下的 critic 不稳定。FlashSAC 通过约束更新动态来稳定 critic 训练(§IV-B),从而能够同时采用高数据通量、大模型和稀疏的梯度更新。
- Massively Parallel Simulation: We collect data using parallel simulation environments, enabling rapid accumulation of diverse trajectories. While many off-policy RL setups rely on a small number of environments [22, 17], high-throughput data collection is critical for maintaining adequate coverage of the state-action space in high-dimensional tasks.
- 大规模并行仿真:我们用 个并行仿真环境采集数据,从而快速积累多样化轨迹。许多 off-policy 强化学习设置只使用少量环境 [22, 17],而高通量数据采集对于在高维任务中维持足够的状态–动作覆盖至关重要。
- Large-Capacity Replay Buffer: FlashSAC uses a replay buffer of up to 10M transitions, an order of magnitude larger than the 1M commonly used in standard off-policy configurations [44, 72]. In high-dimensional tasks, rare but important state-action pairs can be easily overwritten in smaller buffers, leading to catastrophic forgetting and inducing extrapolation error. A larger buffer preserves such long-tail experiences and maintains the diversity of training data available to the critic throughout learning [16].
- 大容量回放缓冲区:FlashSAC 使用最多 1000 万条转移的回放缓冲区,比常见 off-policy 配置中常用的 100 万条大约高一个数量级 [44, 72]。在高维任务中,稀少但重要的状态–动作对在较小缓冲区里很容易被覆盖,从而导致灾难性遗忘并引发外推误差。更大的缓冲区能够留住这类长尾经验,并在整个学习过程中维持 critic 可用训练数据的多样性 [16]。
- Large Model, Large Batch, Fewer Updates: Standard off-policy RL baselines use small MLPs (0.2-0.5M parameters, 2-3 layers) to avoid instability [22, 88]. In contrast, FlashSAC employs a 2.5M-parameter, 6-layer network for both the actor and critic, paired with a batch size of 2048 that nearly saturates GPU utilization. The updates-to-data ratio is set to 2/1024, meaning only 2 gradient updates are performed per 1024 new transitions. Although such infrequent updates are typically ineffective in off-policy RL, the combination of large batches, higher learning rates, and increased model capacity enables fast convergence with fewer updates.
- 大模型、大 batch、更少更新:标准 off-policy 强化学习基线为避免不稳定,通常使用小规模 MLP(0.2–0.5M 参数、2–3 层)[22, 88]。相比之下,FlashSAC 的 actor 和 critic 都采用 2.5M 参数、6 层网络,并配上几乎占满 GPU 的 batch size 2048。更新–数据比设为 2/1024,也就是每新增 1024 条转移只做 2 次梯度更新。尽管如此稀疏的更新在 off-policy 强化学习中通常效果不佳,但大 batch、更高学习率和更大模型容量结合在一起,使得用更少更新也能快速收敛。
- Code Optimization: FlashSAC is implemented in PyTorch [73], with both training and inference JIT-compiled to minimize Python overhead. We use mixed-precision throughout training [59], which reduces wall-clock time by 5-10%.
- 代码优化:FlashSAC 用 PyTorch [73] 实现,训练和推理都经 JIT 编译,以尽量减少 Python 开销。训练全程使用混合精度 [59],可将墙钟时间再缩短 5–10%。
B. Stable Training
B. 稳定训练
Scaling data and model accelerates training but does not prevent instability arising from bootstrapped critic updates. In the Bellman backup, estimation errors at next-state action pairs propagate into the current Q-value targets and can be recursively amplified through repeated updates. This problem worsens with both state-action dimensionality and model capacity, making stability a prerequisite for scaling. FlashSAC addresses this by constraining weight, feature, and gradient norms throughout training via the following mechanisms.
把数据和模型做大可以加快训练,但并不能消除来自自举 critic 更新的不稳定。在 Bellman 回退中,下一状态–动作对上的估计误差会进入当前 Q 值目标,并可能在反复更新中被递归放大。这一问题会随状态–动作维数和模型容量一并加重,因此稳定是缩放的前提。FlashSAC 通过下述机制,在整个训练过程中约束权重、特征与梯度范数。
图 2:FlashSAC 结构。该结构由堆叠的倒残差块组成,采用激活前的批归一化,以及后置 RMS 归一化。
- Inverted Residual Backbone: The backbone stacks inverted residual blocks inspired by the Transformer feedforward block [104] (Figure 2). Each block expands features to a higher dimension via an inverted bottleneck [30], projects back to the original dimension, and adds a residual connection [25] to stabilize gradient propagation. After the final block, we apply RMSNorm [113] to bound per-sample feature norms before value heads, preventing out-of-distribution inputs from producing unbounded activations that destabilize bootstrapping.
- 倒残差主干:主干堆叠了受 Transformer 前馈块启发的倒残差块 [104](图 2)。每个块先通过倒瓶颈把特征扩到更高维 [30],再投影回原维度,并加上残差连接 [25] 以稳定梯度传播。在最后一块之后,我们用 RMSNorm [113] 在价值头之前约束每个样本的特征范数,避免分布外输入产生无界激活、进而破坏自举。
- Pre-activation Batch Normalization: Replay data is collected by a mixture of evolving policies, inducing non-stationary input distributions. Without normalization, feature activations can saturate (e.g., dead ReLUs [1]), degrading gradient flow [13, 54, 43]. We apply batch normalization [33] before each nonlinearity to keep activations well-scaled. We choose batch normalization over layer normalization [5] because it exploits large-batch statistics from diverse replay data, yielding a smoother loss landscape with a lower effective condition number [83, 8, 72].
- 激活前批归一化:回放数据来自不断演化的多条策略,输入分布因此非平稳。若不做归一化,特征激活可能饱和(例如死亡 ReLU [1]),从而损害梯度流动 [13, 54, 43]。我们在每个非线性之前施加批归一化 [33],使激活保持合适的尺度。我们选择批归一化而不是层归一化 [5],是因为它能利用多样化回放数据上的大 batch 统计量,得到更平滑、有效条件数更低的损失曲面 [83, 8, 72]。
- Cross-Batch Value Prediction: Batch normalization computes statistics per batch, so the predicted Q-values and target Q-values receive different normalization when computed in separate forward passes. Following [8], we concatenate current and next-state transitions into a single batch so that both share the same statistics, ensuring consistency in the Bellman update.
- 跨 batch 价值预测:批归一化按 batch 计算统计量,因此若当前 Q 值和目标 Q 值在两次独立前向中计算,它们会受到不同的归一化。沿用 [8],我们把当前转移和下一状态转移拼成同一个 batch,使二者共享统计量,从而保证 Bellman 更新的一致性。
- Distributional Critic with Adaptive Reward Scaling: Following [45, 72], we represent the Q-value as a categorical distribution over atoms uniformly spaced on . The network predicts atom probabilities and is trained via cross-entropy loss against the projected Bellman target [7]. This distributional formulation smooths the optimization landscape and reduces sensitivity to noisy targets [72].
- 带自适应奖励缩放的分布式 critic:沿用 [45, 72],我们把 Q 值表示成在 上均匀排布的 个原子上的类别分布。网络预测各原子的概率,并用投影后的 Bellman 目标做交叉熵训练 [7]。这种分布式形式能平滑优化曲面,并降低对噪声目标的敏感度 [72]。
To keep returns within the distributional critic’s fixed support, we normalize rewards directly rather than centering returns [65] or scaling losses [84]. We track the running discounted return variance and maximum magnitude , and scale as:
为了把回报留在分布式 critic 的固定支撑之内,我们直接对奖励做归一化,而不是对回报做中心化 [65] 或对损失做缩放 [84]。我们跟踪折扣回报的运行方差 以及最大幅度 ,并按如下方式缩放:
This bounds effective returns while maintaining a consistent scale throughout training.
这样既能约束有效回报,又能在整个训练过程中保持一致的尺度。
- Weight Normalization: Uncontrolled weight growth increases Q-value variance and amplifies estimation errors under bootstrapping [55]. After each gradient step, we project each weight vector onto the unit-norm sphere [102, 53, 45, 71, 72] and each normalization parameter vector to norm . This constrains the network to encode information through direction rather than scale.
- 权重归一化:权重不受控制地增长会增大 Q 值方差,并在自举下放大估计误差 [55]。每次梯度步骤之后,我们把每个权重向量投影到单位范数球面 [102, 53, 45, 71, 72],并把每个归一化参数向量 投影到范数 。这迫使网络通过方向而不是尺度来编码信息。
C. Exploration
C. 探索
Off-policy RL can decouple data collection from policy optimization, allowing exploration strategies that pursue broad state-action coverage independently of the current policy. FlashSAC employs two complementary mechanisms.
Off-policy 强化学习可以把数据采集与策略优化解耦,从而允许探索策略在独立于当前策略的情况下追求更广的状态–动作覆盖。FlashSAC 采用两种互补机制。
- Unified Entropy Target: Maximum-entropy RL with automatic temperature tuning [22] encourages sustained exploration, but requires specifying a target entropy. Standard practice sets this target per task, which is impractical across embodiments with varying action dimensions. We instead parameterize the target entropy via a fixed action standard deviation . For a Gaussian policy with diagonal covariance, this gives:
- 统一的熵目标:带自动温度调节的最大熵强化学习 [22] 有助于持续探索,但需要指定目标熵。通行做法是按任务分别设定该目标,这在动作维数各不相同的多种本体上并不现实。我们改为用一个固定的动作标准差 来参数化目标熵。对对角协方差的高斯策略,这给出:
which scales linearly with action dimension, ensuring consistent exploration across embodiments without per-task tuning. We set in all experiments.
它随动作维数线性变化,从而在不同本体上保持一致的探索,无需按任务调参。所有实验中我们都取 。
- Noise Repetition: Temporally correlated action noise is commonly used to improve exploration in sparse-reward settings, with pink noise [14] and Ornstein–Uhlenbeck noise [29] being widely used. However, these methods are ill-suited to massively parallel simulations, as they require per-environment correlated-noise processes, which incur substantial computational and memory overhead.
- 噪声重复:时间相关的动作噪声常被用来改善稀疏奖励设定下的探索,粉红噪声 [14] 和 Ornstein–Uhlenbeck 噪声 [29] 都较为常用。不过这些方法不太适合大规模并行仿真,因为它们需要为每个环境维护相关噪声过程,计算和内存开销都很大。
We propose Noise Repetition, a lightweight alternative that induces temporal correlation using minimal local state. At each repetition interval, a noise vector is sampled for action selection and held constant for consecutive steps. The repetition length is drawn from a Zeta distribution with probability mass function [12], favoring short repeat intervals while occasionally producing long, correlated action sequences.
我们提出噪声重复(Noise Repetition),这是一种用极少局部状态就能引入时间相关性的轻量替代。在每个重复区间开始时,为动作选择采样一个噪声向量 ,并在随后 步内保持不变。重复长度 从 Zeta 分布中抽取,概率质量函数为 [12],它偏好较短的重复间隔,同时偶尔产生较长的、相关的动作序列。
V. Experiments
V. 实验
We evaluate FlashSAC on a diverse suite of robotic control tasks, measuring both asymptotic performance and wall-clock time (measured on a single RTX 5090 GPU). Our experiments span low- and high-dimensional state-based control, vision-based control, and sim-to-real humanoid locomotion.
我们在一套多样化的机器人控制任务上评估 FlashSAC,同时衡量渐近性能和墙钟时间(在单块 RTX 5090 GPU 上测定)。实验覆盖低维与高维的基于状态控制、基于视觉的控制,以及人形运动的仿真到真机迁移。
Fig. 3: Results on State-Based RL, GPU-based Simulators. Learning curves on select tasks from IsaacLab [61], ManiSkill [95], Genesis [4], and MuJoCo Playground [112]. We evaluate performance on (a) low-dimensional tasks with gripper manipulation and quadruped locomotion, and (b) high-dimensional tasks involving dexterous manipulation and humanoid locomotion. While comparable to PPO [86] in low-dimensional tasks, FlashSAC significantly outperforms PPO in high-dimensional tasks.
图 3:基于状态的强化学习、GPU 仿真器上的结果。选自 IsaacLab [61]、ManiSkill [95]、Genesis [4] 和 MuJoCo Playground [112] 的若干任务学习曲线。我们评估(a)夹爪操作与四足运动等低维任务,以及(b)灵巧操作与人形运动等高维任务。在低维任务上与 PPO [86] 相当,FlashSAC 在高维任务上则明显超过 PPO。
A. State-Based RL on GPU-based Simulators
A. 基于状态的强化学习:GPU 仿真器
Experimental Setup: We evaluate on 25 state-based control tasks drawn from four GPU-based simulators: IsaacLab [61], MuJoCo Playground [112], ManiSkill3 [95], and Genesis [4], all of which enable large-scale sample collection at minimal wall-clock cost.
实验设置:我们评估来自四个 GPU 仿真器的 25 项基于状态的控制任务:IsaacLab [61]、MuJoCo Playground [112]、ManiSkill3 [95] 和 Genesis [4]。这些仿真器都能以很低的墙钟成本做大规模样本采集。
The tasks span a wide range of state–action dimensionalities:
这些任务覆盖了很宽的状态–动作维数范围:
- Low-dim (15 tasks): Gripper-based manipulation (Franka) and quadruped locomotion (AnyMal-C/D, Unitree Go2).
- 低维(15 项任务):夹爪操作(Franka)和四足运动(AnyMal-C/D、Unitree Go2)。
- High-dim (10 tasks): Dexterous manipulation (Allegro, Shadow Hand) and humanoid locomotion (Unitree G1, H1, Booster T1).
- 高维(10 项任务):灵巧操作(Allegro、Shadow Hand)和人形运动(Unitree G1、H1、Booster T1)。
A complete task list is provided in § X.
完整任务列表见 § X。
We compare FlashSAC against strong, widely adopted baselines:
我们把 FlashSAC 与强且被广泛采用的基线作比较:
- PPO [86]: A highly optimized on-policy implementation from RSL-RL, representative of current best practices in sim-to-real robotic RL.
- PPO [86]:来自 RSL-RL 的高度优化 on-policy 实现,代表当前仿真到真机机器人强化学习中的通行做法。
- FastTD3 [88]: A wall-clock–optimized off-policy method designed for high throughput simulations.
- FastTD3 [88]:为高通量仿真设计的、针对墙钟时间优化的 off-policy 方法。
Whenever available, we report published results; otherwise, we reproduce results using official implementations.
只要有已发表结果,我们就直接报告;否则用官方实现复现。
Off-policy methods (FlashSAC and FastTD3) are trained for 50M environment steps. To probe asymptotic performance, PPO is trained for 200M steps, requiring approximately the compute of FlashSAC. While baseline methods use task-specific hyperparameter tuning, FlashSAC is evaluated using a single unified configuration across all tasks, varying only the discount factor to match simulator defaults (e.g., for IsaacLab, for Playground).
Off-policy 方法(FlashSAC 和 FastTD3)训练 5000 万环境步。为探测渐近性能,PPO 训练 2 亿步,计算量大约是 FlashSAC 的 。基线方法使用按任务调节的超参,而 FlashSAC 在全部任务上采用同一套统一配置,只把折扣因子 调到与各仿真器默认值一致(例如 IsaacLab 为 ,Playground 为 )。
Experimental Results: Figure 3 summarizes performance on representative tasks, with full results in § XII.
实验结果:图 3 汇总了代表性任务上的表现,完整结果见 § XII。
On low-dimensional tasks, FlashSAC slightly outperforms PPO (Figure 3.a). As consistent with prior findings, on-policy methods remain effective when state–action spaces are small, and simulation throughput is high enough to collect a large volume of samples.
在低维任务上,FlashSAC 略优于 PPO(图 3.a)。这与已有观察一致:当状态–动作空间较小、仿真通量又高到足以采集大量样本时,on-policy 方法仍然有效。
On high-dimensional tasks, FlashSAC demonstrates a clear and consistent advantage (Figure 3.b). Across dexterous manipulation and humanoid locomotion benchmarks, FlashSAC converges reliably to higher asymptotic performance while requiring substantially less wall-clock time than PPO.
在高维任务上,FlashSAC 展现出清晰而稳定的优势(图 3.b)。在灵巧操作和人形运动基准上,FlashSAC 都能可靠地收敛到更高的渐近性能,所需墙钟时间还明显少于 PPO。
Compared to FastTD3, FlashSAC is markedly more stable, converging across all tasks where FastTD3 frequently fails or underperforms (e.g., Go2Walk, Franka Pull Cube). When both methods converge, FlashSAC achieves higher asymptotic performance, with the largest gains observed in humanoid locomotion, where larger model capacity is particularly beneficial.
与 FastTD3 相比,FlashSAC 明显更稳,能在 FastTD3 经常失败或表现不佳的全部任务上收敛(例如 Go2Walk、Franka Pull Cube)。当两种方法都能收敛时,FlashSAC 的渐近性能更高,提升最大的是人形运动——在那里,更大的模型容量尤为有益。
Fig. 4: Results on State-Based RL, CPU-based Simulators. Learning curves on select tasks from MuJoCo [99, 98], DMC [96], HumanoidBench [89] and MyoSuite [9]. We primarily evaluate high-dimensional tasks, involving dexterous manipulation and humanoid locomotion. FlashSAC significantly outperforms PPO, as well as strong off-policy RL and model-based RL baselines in both compute efficiency and asymptotic performance.
图 4:基于状态的强化学习、CPU 仿真器上的结果。选自 MuJoCo [99, 98]、DMC [96]、HumanoidBench [89] 和 MyoSuite [9] 的若干任务学习曲线。我们主要评估涉及灵巧操作和人形运动的高维任务。无论是计算效率还是渐近性能,FlashSAC 都明显优于 PPO,以及较强的 off-policy 强化学习和基于模型的强化学习基线。
B. State-Based RL on CPU-based Simulators
B. 基于状态的强化学习:CPU 仿真器
Experimental Setup: We further evaluate FlashSAC on 40 single-environment, CPU-based continuous-control tasks drawn from four established benchmarks: MuJoCo [98], DeepMind Control Suite [96], MyoSuite [9], and HumanoidBench [89]. Unlike GPU-based simulators, these benchmarks use a single environment instance, placing greater emphasis on sample efficiency rather than wall-clock throughput.
实验设置:我们进一步在 40 项单环境、基于 CPU 的连续控制任务上评估 FlashSAC,这些任务来自四个既有基准:MuJoCo [98]、DeepMind Control Suite [96]、MyoSuite [9] 和 HumanoidBench [89]。与 GPU 仿真器不同,这些基准使用单个环境实例,因此更看重样本效率,而不是墙钟通量。
We compare FlashSAC against strong sample-efficient baselines:
我们把 FlashSAC 与较强的样本高效基线作比较:
- PPO [86]: A highly optimized on-policy implementation from RSL-RL, included to assess whether on-policy methods remain viable in the low-sample regime.
- PPO [86]:来自 RSL-RL 的高度优化 on-policy 实现,用来检验 on-policy 方法在低样本体制下是否仍然可行。
- XQC [72]: A recent off-policy method coupled with batch-normalization designed for high sample efficiency.
- XQC [72]:近期一种结合批归一化、面向高样本效率的 off-policy 方法。
- SimbaV2 [45]: An improved variant of Simba [44] for stable off-policy learning.
- SimbaV2 [45]:Simba [44] 的改进变体,用于稳定的 off-policy 学习。
- TD-MPC2 [24]: A model-based method that combines off-policy RL with model-predictive planning.
- TD-MPC2 [24]:把 off-policy 强化学习与模型预测规划结合的基于模型方法。
- MR.Q [19]: A recent model-free method using a model-based objective for better representation learning.
- MR.Q [19]:近期一种无模型方法,借助基于模型的目标来改进表征学习。
As in the GPU-based setting, FlashSAC uses a single unified configuration across all tasks, with only minimal adjustments to match each benchmark’s conventions. Since sample collection is considerably slower with a single environment, the CPU-based configuration differs from the GPU setting by reducing the batch size from 2048 to 512 and setting the update-to-data ratio to 1, reflecting the lower data throughput.
与 GPU 设定一样,FlashSAC 在全部任务上使用同一套统一配置,只做极少调整以匹配各基准的惯例。由于单环境下样本采集慢得多,CPU 配置把 batch size 从 2048 降到 512,并把更新–数据比设为 1,以反映更低的数据通量。
Experimental Results: As illustrated in Figure V-B, FlashSAC consistently outperforms all baselines across representative tasks in this sample-efficient regime. Full per-task results appear in § XII. PPO performs particularly poorly here, as on-policy methods cannot reuse experience and thus suffer under limited sample budgets.
实验结果:如图 V-B 所示,在这种强调样本效率的体制下,FlashSAC 在代表性任务上稳定优于所有基线。逐任务完整结果见 § XII。PPO 在这里表现尤其差,因为 on-policy 方法无法复用经验,在有限样本预算下会吃亏。
These results confirm that the design choices in FlashSAC generalize beyond massively parallel GPU-based simulation: even in the classical single-environment setting, where sample efficiency is the primary bottleneck, FlashSAC matches or exceeds dedicated sample-efficient methods without task-specific tuning.
这些结果说明,FlashSAC 中的设计选择并不只适用于大规模并行的 GPU 仿真:即便在样本效率才是主要瓶颈的经典单环境设定下,它也能在不做按任务调参的情况下,达到或超过专门面向样本效率的方法。
C. Vision-based RL
C. 基于视觉的强化学习
Experimental Setup: We extend our evaluation to vision-based control, where high rendering cost and low environment throughput severely limit the number of transitions collected per unit time, making data efficiency critical. We evaluate on 8 tasks from the DMControl Suite [100], spanning manipulation and mono/bi-pedal locomotion. A complete task list is provided in § X.
实验设置:我们把评估扩展到基于视觉的控制。这里渲染成本高、环境通量低,单位时间内能采到的转移很少,因而数据效率至关重要。我们在 DMControl Suite [100] 的 8 项任务上评估,覆盖操作以及单足/双足运动。完整任务列表见 § X。
Given the low throughput of visual environments, we focus on off-policy baselines:
鉴于视觉环境通量较低,我们主要与 off-policy 基线比较:
- DrQ-v2 [111]: A DDPG-based [51] method that improves data efficiency through image augmentation [41].
- DrQ-v2 [111]:一种基于 DDPG [51] 的方法,通过图像增强来提高数据效率 [41]。
- MR.Q [19]: An off-policy method that incorporates a dynamics modeling objective to improve representation learning.
- MR.Q [19]:一种加入动力学建模目标以改进表征学习的 off-policy 方法。
As in the CPU-based state experiments (§ V-B), sample collection is slow; we reuse the same hyperparameters from that setup, adapting only the following to match the standard DrQ-v2 configuration [111]: (i) a lightweight convolutional encoder (3 convolutional layers followed by a linear bottleneck), (ii) frame stacking of the three most recent frames (84 × 84 × 9) for temporal reasoning without recurrent architectures, and (iii) 3-step returns for better credit assignment. All methods are trained for 1M environment steps with an action repeat of 2. Full details are in § X.
与基于 CPU 的状态实验(§ V-B)一样,样本采集很慢;我们复用那一套超参,只做如下调整以对齐 DrQ-v2 的标准配置 [111]:(i)轻量卷积编码器(3 个卷积层再加一个线性瓶颈);(ii)堆叠最近 3 帧()以在不使用循环结构的情况下做时间推理;(iii)3 步回报,以改善信用分配。所有方法均训练 100 万环境步,动作重复为 2。完整细节见 § X。
Fig. 5: Results on Vision-Based RL. Learning curves on selected tasks from vision-based DMControl Suite [96]. We assess learning performance in low-dimensional environments, including pendulum manipulation and bipedal locomotion. FlashSAC achieves better compute efficiency and higher asymptotic performance.
图 5:基于视觉的强化学习结果。选自视觉版 DMControl Suite [96] 的若干任务学习曲线。我们评估低维环境中的学习表现,包括摆的操作和双足运动。FlashSAC 取得了更好的计算效率和更高的渐近性能。
Experimental Results: Figure 5 shows representative learning curves, with full results in § XII. Across tasks, FlashSAC matches or exceeds all baselines in asymptotic performance while converging faster in wall-clock time. DrQ-v2 is sample-efficient but unstable, failing to converge in several environments (e.g., Finger Turn Hard). MR.Q achieves high final performance but incurs additional computational cost from its auxiliary dynamics model. In contrast, FlashSAC achieves competitive or superior results with a single set of hyperparameters, without task-specific exploration or auxiliary objectives.
实验结果:图 5 给出了代表性学习曲线,完整结果见 § XII。在各项任务上,FlashSAC 的渐近性能都不低于各基线,墙钟时间上还收敛得更快。DrQ-v2 样本效率高但不稳定,在若干环境中无法收敛(例如 Finger Turn Hard)。MR.Q 最终性能高,但其辅助动力学模型会带来额外计算成本。相比之下,FlashSAC 用同一套超参就能取得相当或更好的结果,既不需要按任务调节探索,也不需要辅助目标。
We note that the stabilization techniques in FlashSAC are orthogonal to such extensions; for example, MR.Q’s representation learning objective could be layered on top for further gains in visual feature learning.
需要指出,FlashSAC 中的稳定化技术与这类扩展是正交的;例如,可以把 MR.Q 的表征学习目标叠加上去,以进一步改进视觉特征学习。
D. Sim-to-Real Transfer
D. 仿真到真机迁移
Off-policy RL is often regarded as unreliable for sim-to-real transfer, particularly in high-dimensional systems, where training instability can lead to unsafe behaviors [63]. We evaluate whether FlashSAC enables reliable sim-to-real transfer on a challenging 29-DoF Unitree G1 humanoid performing blind locomotion.
Off-policy 强化学习常被认为不宜用于仿真到真机迁移,在高维系统中尤其如此,因为训练不稳定可能导致不安全行为 [63]。我们评估 FlashSAC 能否在一项具有挑战性的任务上实现可靠的仿真到真机迁移:29 自由度的 Unitree G1 人形机器人做盲运动。
Experimental Setup: We train blind locomotion policies in simulation using a terrain curriculum comprising pyramid stairs, discrete grids, waves, and pits. The curriculum consists of 10 terrain levels with stair heights ranging from 0 to 23cm (step width 32cm, platform width 3m). Terrain difficulty is increased automatically using a game-inspired curriculum [82]. To facilitate sim-to-real transfer, we apply large-scale domain randomization alongside the terrain curriculum; full details are provided in § XI.
实验设置:我们在仿真中用地形课程训练盲运动策略,课程包括金字塔楼梯、离散网格、波浪和坑洞。课程共 10 级地形,台阶高度从 0 到 23cm(踏步宽 32cm,平台宽 3m)。地形难度按游戏式课程自动提升 [82]。为便于仿真到真机迁移,我们在地形课程之外施加大规模域随机化;完整细节见 § XI。
As a baseline, we use PPO with the sim-to-real pipeline of [64]. FlashSAC adopts the same sim-to-real adaptation techniques for a fair comparison. Both methods use implicit system identification via a context estimator [64] and an asymmetric actor–critic formulation [74], where the critic receives privileged information (e.g., contact states and height maps) during training. Both methods share identical reward design and coefficients, combining velocity tracking with regularization terms penalizing foot slip, excessive torque, action discontinuities, and orientation instability. FlashSAC uses the same architecture and hyperparameters as in the state-based experiments V-B.
基线采用 [64] 的仿真到真机流水线上的 PPO。为公平比较,FlashSAC 使用同一套仿真到真机适应技术。两种方法都通过上下文估计器做隐式系统辨识 [64],并采用不对称的 actor–critic 形式 [74],训练时 critic 能看到接触状态和高度图等特权信息。两种方法使用完全相同的奖励设计与系数,在速度跟踪之外,还对脚底打滑、过大扭矩、动作不连续和姿态不稳加以正则。FlashSAC 的网络结构和超参与基于状态的实验 V-B 相同。
Experimental Results: On flat terrain (Figure 1.(c)), FlashSAC achieves stable real-world locomotion after approximately 20 minutes of training, whereas PPO requires about 3 hours to reach comparable performance. The learned policy supports omnidirectional locomotion (forward, backward, and lateral) without re-training.
实验结果:在平坦地形上(图 1.(c)),FlashSAC 大约训练 20 分钟就能在真实世界中稳定行走,而 PPO 大约需要 3 小时才能达到相当的表现。学到的策略支持全向运动(前进、后退和侧向),无需再训练。
The advantage of FlashSAC is more pronounced on rough terrain (Figure 6). In the real-world setup, the robot faces stairs of 15cm height, 60cm width, and 1.5m platform width—conditions unseen during training, which uses different stair dimensions. FlashSAC successfully climbs these stairs after approximately 4 hours of training, while PPO requires nearly 20 hours to achieve a similar capability.
在崎岖地形上,FlashSAC 的优势更加明显(图 6)。真实场景中,机器人面对的是高度 15cm、宽度 60cm、平台宽 1.5m 的楼梯——这些条件在训练中并未出现,训练时的楼梯尺寸不同。FlashSAC 大约训练 4 小时就能爬上这些楼梯,而 PPO 几乎需要 20 小时才能达到相近能力。
Overall, FlashSAC reduces the training time for sim-to-real humanoid locomotion by nearly an order of magnitude compared to PPO while maintaining stable and safe behaviors.
总体而言,与 PPO 相比,FlashSAC 把人形运动仿真到真机迁移的训练时间缩短了将近一个数量级,同时仍能保持稳定、安全的行为。
图 6:Unitree G1 上的仿真到真机爬楼。FlashSAC 仅在仿真中训练 4 小时,就能在真实世界中稳定爬楼,而 PPO 几乎需要 20 小时才能达到同样能力。
VI. Analysis
VI. 分析
In this section, we analyze the factors underlying FlashSAC’s performance across four aspects. We first examine how off-policy learning yields broader state–action coverage than on-policy methods (§VI-A). We then investigate three design choices central to FlashSAC: scaling data collection and model capacity for faster training (§VI-B), architectural ablations that improve training stability (§VI-C), and the effect of entropy and temporal correlation on exploration (§VI-D). All experiments are conducted in four IsaacLab environments [61]: cube reorientation with the Allegro and Shadow Hands, and flat and rough terrain locomotion with the G1.
本节从四个方面分析 FlashSAC 表现背后的因素。我们首先考察 off-policy 学习为何能比 on-policy 方法获得更广的状态–动作覆盖(§VI-A)。随后研究 FlashSAC 中三项关键设计:通过扩大数据采集和模型容量来加快训练(§VI-B)、改善训练稳定性的结构消融(§VI-C),以及熵与时间相关性对探索的影响(§VI-D)。全部实验在四个 IsaacLab 环境中进行 [61]:Allegro 和 Shadow Hand 的立方体重定向,以及 G1 在平坦与崎岖地形上的运动。
Fig. 7: Off-Policy vs. On-Policy Data Coverage. We train FlashSAC for 1M steps on the Shadow Hand cube reorientation task. 2D density plots show the joint distribution of object y-position and fingertip joint actions for (a) off-policy samples from the replay buffer and (b) on-policy samples collected by rolling out the final policy for an additional 1M steps. Off-policy data covers a substantially broader region of the state–action space.
图 7:Off-policy 与 on-policy 的数据覆盖。我们在 Shadow Hand 立方体重定向任务上把 FlashSAC 训练 100 万步。二维密度图给出物体 y 位置与指尖关节动作的联合分布:(a)来自回放缓冲区的 off-policy 样本;(b)把最终策略再 rollout 100 万步得到的 on-policy 样本。Off-policy 数据覆盖了状态–动作空间中明显更广的区域。
A. Off-Policy vs On-Policy
A. Off-Policy 与 On-Policy
We train FlashSAC for 1M steps on the IsaacLab Shadow Hand task with a replay buffer of size 1M, then collect 1M additional on-policy transitions by rolling out the final policy.
我们在 IsaacLab 的 Shadow Hand 任务上把 FlashSAC 训练 100 万步,回放缓冲区大小为 100 万,再把最终策略 rollout,额外采集 100 万条 on-policy 转移。
Figure 7 compares the state–action coverage of the two datasets via 2D density plots of finger actions versus object y-position. Off-policy data (Figure 7.(a)) covers a substantially broader region of the state–action space, reflecting experience accumulated across diverse behavior policies stored in the replay buffer. On-policy data (Figure 7.(b)), by contrast, is tightly concentrated around the final policy’s distribution. This disparity suggests that limited state–action coverage is a key factor in the reduced effectiveness of on-policy methods on high-dimensional tasks, where achieving comparable coverage would require substantially more data collection.
图 7 用手指动作相对物体 y 位置的二维密度图,比较两份数据的状态–动作覆盖。Off-policy 数据(图 7.(a))覆盖了状态–动作空间中明显更广的区域,反映的是回放缓冲区中由多样化行为策略积累下来的经验。相比之下,on-policy 数据(图 7.(b))紧紧集中在最终策略的分布附近。这一差异表明,有限的状态–动作覆盖是 on-policy 方法在高维任务上效果下降的关键因素:要达到相当的覆盖,需要采集的数据会多得多。
B. Scaling Ablation for Faster Training
B. 加速训练的缩放消融
We study how scaling data, model capacity, and reducing the number of gradient updates (§IV-A) affects the compute efficiency of FlashSAC. We perform univariate ablations over five hyperparameters: batch size, replay buffer size, network width, network depth, and update-to-data (UTD) ratio.
我们研究扩大数据、增大模型容量并减少梯度更新次数(§IV-A)如何影响 FlashSAC 的计算效率。我们对五个超参做单变量消融:batch size、回放缓冲区大小、网络宽度、网络深度,以及更新–数据比(UTD)。
Figure 8 shows learning curves plotted against wall-clock time. Increasing replay buffer size improves performance up to 10M transitions by stabilizing training (Figure 8.(a)). However, overly large buffers (e.g., 50M) slow learning because recent high-quality samples are drawn less frequently, though they can achieve slightly higher asymptotic performance given sufficient training time.
图 8 给出相对墙钟时间的学习曲线。把回放缓冲区增大到 1000 万条转移,能通过稳定训练来提升性能(图 8.(a))。但过大的缓冲区(例如 5000 万)会拖慢学习,因为近期的高质量样本被抽到的频率下降;若训练时间足够,它们仍可能达到略高的渐近性能。
Figures 8.(b)–(e) exhibit trends consistent with established scaling laws [37]: increasing batch size and model capacity, along with reducing the UTD ratio, accelerate convergence. Most existing off-policy RL methods rely on small architectures for training stability (e.g., width 128 with inverted bottlenecks and block depth 1), which limits convergence speed. The scaling mechanisms of FlashSAC enable higher-capacity models, resulting in substantially faster convergence.
图 8.(b)–(e) 呈现出与既有缩放规律一致的趋势 [37]:增大 batch size 和模型容量,并降低 UTD 比,都能加快收敛。现有大多数 off-policy 强化学习方法为了训练稳定而使用较小结构(例如宽度 128、带倒瓶颈、块深度为 1),这限制了收敛速度。FlashSAC 的缩放机制使得可以采用更高容量的模型,从而显著加快收敛。
图 8:缩放结果。(a)回放缓冲区大小在训练稳定性与效率之间折中。(b–e)增大 batch size 和模型容量,同时降低 UTD 比,可以加快收敛。
图 9:结构消融。从标准 MLP 出发,逐项加入组件,直至得到完整的 FlashSAC。每加入一项,都通过约束权重、特征与梯度范数来稳定训练,并降低条件数。
C. Architectural Ablation for Stable Training
C. 稳定训练的结构消融
We analyze the contribution of each architectural component in FlashSAC (§IV-B) to determine whether the proposed design stabilizes training. Beyond final task performance, we measure parameter, feature, and gradient norms throughout training as indicators of optimization stability. Following [72], we also measure the condition number of the critic loss landscape, where larger values correspond to poorly conditioned updates that can exacerbate critic error amplification.
我们分析 FlashSAC 中各结构组件(§IV-B)的贡献,以判断所提设计是否稳定了训练。除最终任务性能外,我们在整个训练过程中测量参数、特征和梯度范数,作为优化稳定性的指标。沿用 [72],我们还测量 critic 损失曲面的条件数:数值越大,意味着更新的条件越差,越容易加重 critic 误差放大。
Starting from a standard MLP critic, we incrementally add: Residual Blocks, Batch Normalization, Post RMSNorm, Distributional Critics with Reward Scaling, and Weight Normalization. Figure 9 summarizes the results. As components are added, parameter, feature, and gradient norms remain bounded throughout training with no uncontrolled growth, indicating well-behaved critic updates and reduced error amplification. The condition number also decreases monotonically, reaching its lowest value with the full FlashSAC architecture.
我们从标准 MLP critic 出发,依次加入:残差块、批归一化、后置 RMSNorm、带奖励缩放的分布式 critic,以及权重归一化。图 9 汇总了结果。随着组件加入,参数、特征与梯度范数在整个训练中都保持有界、没有失控增长,说明 critic 更新行为良好,误差放大得到抑制。条件数也单调下降,在完整 FlashSAC 结构下达到最低。
These gains in optimization stability directly translate to improved task performance (Figure 9.(a)), underscoring the importance of controlling update dynamics in off-policy RL. While weight normalization alone yields modest gains, it improves robustness in sample-limited regimes and is therefore retained in the final design.
这些优化稳定性上的收益,直接转化为更好的任务性能(图 9.(a)),说明在 off-policy 强化学习中控制更新动态非常重要。权重归一化单独带来的提升虽然有限,但能改善样本受限体制下的稳健性,因此被保留在最终设计中。
D. Exploration Ablation
D. 探索消融
We analyze FlashSAC’s exploration strategy (§IV-C): unifying the entropy target and noise repetition.
我们分析 FlashSAC 的探索策略(§IV-C):统一熵目标 ,以及噪声重复。
Figure 10(a) shows the effect of varying the entropy target across values {0.05,0.1,0.15,0.2,0.25}. Performance is largely insensitive to this hyperparameter, with all settings converging to similar asymptotic scores. This robustness simplifies tuning in practice, as the default value (=0.15) performs well without task-specific adjustment.
图 10.(a) 给出在 上改变熵目标 的影响。性能对这一超参大体不敏感,各种设定都收敛到相近的渐近分数。这种稳健性简化了实际调参,因为默认值()无需按任务调节也能表现良好。
Figure 10(b) compares training with and without noise repetition. Disabling noise repeat leads to slower convergence and lower aggregate scores, confirming that temporally correlated exploration is crucial for FlashSAC. Repeating sampled action noise across consecutive steps produces coherent exploratory trajectories rather than uncorrelated perturbations that are quickly averaged out by the dynamics in high-dimensional control tasks.
图 10.(b) 比较了有无噪声重复的训练。关掉噪声重复后,收敛变慢,总体分数也更低,说明时间相关的探索对 FlashSAC 至关重要。把采样到的动作噪声在连续数步中重复,能产生连贯的探索轨迹,而不是很快被高维控制任务中的动力学平均掉的不相关扰动。
Fig. 10: Exploration Ablation Results. (a) Unified entropy target: The optimal entropy target lies in the range 0.15 to 0.2 across tasks, enabling a unified setting without task-specific tuning. (b) Noise repetition: Repeating action noise accelerates convergence and improves asymptotic performance.
图 10:探索消融结果。(a)统一熵目标:最优熵目标 在各项任务上都落在 0.15 到 0.2 之间,因而可以统一设定、不必按任务调节。(b)噪声重复:重复动作噪声能加快收敛,并提高渐近性能。
VII. Lessons and Opportunities
VII. 启示与展望
We presented FlashSAC, a fast and stable off-policy RL framework for high-dimensional robotics. As the robotics community moves toward high-dimensional [21], perception-rich [66, 36], and contact-intensive tasks [109], the scalability of on-policy RL becomes increasingly constrained. Off-policy RL is an appealing alternative, but its adoption has been limited by slow training speed and instability in critic learning arising from function approximation error and bootstrapped updates. FlashSAC addresses these challenges through two complementary mechanisms: scaling data and model capacity while reducing the number of gradient updates for faster training, and integrating explicit architectural constraints on critic updates for stable optimization. Together, these yield strong asymptotic performance and up to an order-of-magnitude reduction in wall-clock time compared to on-policy methods.
我们提出了 FlashSAC,一个面向高维机器人、既快又稳的 off-policy 强化学习框架。随着机器人社区走向更高维 [21]、更依赖感知 [66, 36]、接触更密集的任务 [109],on-policy 强化学习的可扩展性愈发受限。Off-policy 强化学习是一条有吸引力的替代路径,但其采用一直受限于训练速度慢,以及函数逼近误差和自举更新造成的 critic 学习不稳定。FlashSAC 用两项互补机制应对这些挑战:扩大数据与模型容量并减少梯度更新,以实现更快训练;并对 critic 更新施加明确的结构约束,以实现稳定优化。二者结合,带来了较强的渐近性能,以及相对 on-policy 方法最多一个数量级的墙钟时间下降。
Stabilized off-policy learning opens new opportunities for robot learning. Improved data efficiency makes it feasible to train larger policies, incorporate vision and other rich sensory inputs [92], and leverage slower but more realistic simulators [76]. Off-policy methods also naturally support learning from a mixture of demonstrations and self-collected experience [6]. While this work focuses on state-based control, extending these critic-stabilization principles to tactile-based learning is a promising direction for future work.
稳定下来的 off-policy 学习,为机器人学习打开了新的空间。更高的数据效率使得训练更大的策略、引入视觉和其他丰富传感输入 [92],以及使用更慢但更真实的仿真器 [76] 成为可能。Off-policy 方法也天然支持从示范与自行采集经验的混合中学习 [6]。本文主要关注基于状态的控制,把这些 critic 稳定化原则扩展到基于触觉的学习,是一个值得继续探索的方向。
Acknowledgements
致谢
We would like to express our gratitude to Younggyo Seo and Yekyung Nah for their valuable feedback on this paper. This work was supported by the Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2019- II190075, Artificial Intelligence Graduate School Program (KAIST)). This research was also funded by the research cluster “Third Wave of AI”, funded by the excellence program of the Hessian Ministry of Higher Education, Science, Research and the Arts, hessian.AI and by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy (EXC-3057/1 “Reasonable Artificial Intelligence”, Project No. 533677015). It was further partially supported by the German Federal Ministry of Research, Technology and Space (BMFTR) under the Robotics Institute Germany (RIG).
我们感谢 Younggyo Seo 和 Yekyung Nah 对本文提供的宝贵意见。本工作得到韩国政府(MSIT)资助的 Institute for Information & communications Technology Planning & Evaluation(IITP)项目支持(RS-2019- II190075,Artificial Intelligence Graduate School Program (KAIST))。研究同时得到研究集群 “Third Wave of AI” 的资助,该集群由 Hessian Ministry of Higher Education, Science, Research and the Arts 的卓越计划、hessian.AI 以及 Deutsche Forschungsgemeinschaft(DFG, German Research Foundation)在德国卓越战略(EXC-3057/1 “Reasonable Artificial Intelligence”,项目号 533677015)下资助。工作还部分得到德国联邦研究、技术与空间部(BMFTR)通过 Robotics Institute Germany(RIG)的支持。
References
参考文献
[1] Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on lifelong learning agents, pages 620–636. PMLR, 2023.
[2] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020.
[3] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866, 2017.
[4] Genesis Authors. Genesis: A generative and universal physics engine for robotics and beyond, December 2024. https://github.com/Genesis-Embodied-AI/Genesis .
[5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[6] Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577–1594. PMLR, 2023.
[7] Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pages 449–458. PMLR, 2017.
[8] Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. International Conference on Learning Representations (ICLR), 2024.
[9] Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite–a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600, 2022.
[10] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.
[11] Yuanpei Chen, Yaodong Yang, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen Marcus McAleer, Hao Dong, and Song-Chun Zhu. Towards human-level bimanual dexterous manipulation with reinforcement learning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. https://openreview.net/forum?id=D29JbExncTP .
[12] Will Dabney, Georg Ostrovski, and André Barreto. Temporally-extended { { \ \backslash epsilon } } -greedy exploration. arXiv preprint arXiv:2006.01782, 2020.
[13] Shibhansh Dohare, J Fernando Hernandez-Garcia, Parash Rahman, Richard S Sutton, and A Rupam Mahmood. Maintaining plasticity in deep continual learning. arXiv preprint arXiv:2306.13812, 2023.
[14] Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[15] Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and A. Rupam Mahmood. Weight clipping for deep continual and reinforcement learning. Reinforcement Learning Journal, 5:2198–2217, 2024.
[16] William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International conference on machine learning, pages 3061–3071. PMLR, 2020.
[17] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018.
[18] Scott Fujimoto, Wei-Di Chang, Edward J Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. arXiv preprint arXiv:2306.02451, 2023.
[19] Scott Fujimoto, Pierluca D’Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat. Towards general-purpose model-free reinforcement learning. arXiv preprint arXiv:2501.16142, 2025.
[20] Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811, 2024.
[21] Zhaoyuan Gu, Junheng Li, Wenlan Shen, Wenhao Yu, Zhaoming Xie, Stephen McCrory, Xianyi Cheng, Abdulaziz Shamsah, Robert Griffin, C Karen Liu, et al. Humanoid locomotion and manipulation: Current progress and challenges in control, planning, and learning. arXiv preprint arXiv:2501.02116, 2025.
[22] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018.
[23] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.
[24] Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control, 2024.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
[26] Ahmed Hendawy, Henrik Metternich, Théo Vincent, Mahdi Kallel, Jan Peters, and Carlo D’Eramo. Use the online network if you can: Towards fast and stable reinforcement learning. arXiv preprint arXiv:2510.02590, 2025.
[27] Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint arXiv:2110.02034, 2021.
[28] David Hoeller, Nikita Rudin, Dhionis Sako, and Marco Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, 9(88):eadi7566, 2024.
[29] Jakob Hollenstein, Sayantan Auddy, Matteo Saveriano, Erwan Renaudo, and Justus Piater. Action noise in off-policy deep reinforcement learning: Impact on exploration and performance. arXiv preprint arXiv:2206.03787, 2022.
[30] Andrew G Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
[31] Marcel Hussing, Claas Voelcker, Igor Gilitschenski, Amir-massoud Farahmand, and Eric Eaton. Dissecting deep RL with high update ratios: Combatting value divergence. In Reinforcement Learning Conference (RLC), volume 5, 2024.
[32] Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26):eaau5872, 2019.
[33] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. pmlr, 2015.
[34] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[35] Gwanghyeon Ji, Juhyeok Mun, Hyeongjun Kim, and Jemin Hwangbo. Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters, 7(2):4630–4637, April 2022. ISSN 2377-3774. 10.1109/lra.2022.3151396 . http://dx.doi.org/10.1109/LRA.2022.3151396 .
[36] Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Jim Fan, and Yuke Zhu. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 16923–16930. IEEE, 2025.
[37] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[38] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source vision-language-action model. In CoRL, 2025.
[39] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013.
[40] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034, 2021.
[41] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33:19884–19895, 2020.
[42] Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, Se-Young Yun, and Chulhee Yun. Plastic: Improving input and label plasticity for sample efficient reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024a.
[43] Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. arXiv preprint arXiv:2406.02596, 2024b.
[44] Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, 2024c.
[45] Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspherical normalization for scalable deep reinforcement learning. arXiv preprint arXiv:2502.15280, 2025.
[46] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5(47), October 2020. ISSN 2470-9476. 10.1126/scirobotics.abc5986 . http://dx.doi.org/10.1126/scirobotics.abc5986 .
[47] Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting. arXiv preprint arXiv:2304.10466, 2023a.
[48] Zechu Li, Tao Chen, Zhang-Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel Q-learning: Scaling off-policy reinforcement learning under massively parallel simulation. In Proceedings of the International Conference on Machine Learning (ICML), 2023b.
[49] Qiayuan Liao, Takara E Truong, Xiaoyu Huang, Yuman Gao, Guy Tevet, Koushil Sreenath, and C Karen Liu. Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion. arXiv preprint arXiv:2508.08241, 2025.
[50] Shuhao Liao, Peizhuo Li, Xinrong Yang, Linnan Chang, Zhaoxin Fan, Qing Wang, Lei Shi, Yuhong Cao, Wenjun Wu, and Guillaume Sartoretti. Gpo: Growing policy optimization for legged robot locomotion and whole-body control. arXiv preprint arXiv:2601.20668, 2026.
[51] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
[52] Xingyu Lin, Yufei Wang, Jake Olkin, and David Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 432–448. PMLR, 2021.
[53] Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. ngpt: Normalized transformer with representation learning on the hypersphere. arXiv preprint arXiv:2410.01131, 2024.
[54] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. Proc. the International Conference on Machine Learning (ICML), 2023.
[55] Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado P van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. Advances in Neural Information Processing Systems, 37:106440–106473, 2024.
[56] Thomas M. Moerland, Joost Broekens, Aske Plaat, and Catholijn M. Jonker. Model-based reinforcement learning: A survey. Foundations and Trends in Machine Learning, 16(1):1–118, 2023.
[57] A Rupam Mahmood, Hado P Van Hasselt, and Richard S Sutton. Weighted importance sampling for off-policy learning with linear function approximation. Advances in neural information processing systems, 27, 2014.
[58] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021.
[59] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017.
[60] Mayank Mittal, Nikita Rudin, Victor Klemm, Arthur Allshire, and Marco Hutter. Symmetry considerations for learning task symmetric robot policies. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7433–7439. IEEE, 2024.
[61] Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Muñoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, et al. Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831, 2025.
[62] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
[63] J.W. Mock and University of Wyoming. Department of Electrical Engineering. A Comparison of PPO, TD3, and SAC Reinforcement Algorithms for Quadruped Walking Gait Generation and Transfer Learning to a Physical Robot . University of Wyoming, 2023. ISBN 9798379561789. https://books.google.co.kr/books?id=waUG0AEACAAJ .
[64] I Nahrendra, Byeongho Yu, and Hyun Myung. Dreamwaq: Learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning. arXiv preprint arXiv:2301.10602, 2023.
[65] Abhishek Naik, Yi Wan, Manan Tomar, and Richard S Sutton. Reward centering. arXiv preprint arXiv:2405.09999, 2024.
[66] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems, 2024.
[67] Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405.16158, 2024.
[68] Michal Nauman, Marek Cygan, Carmelo Sferrazza, Aviral Kumar, and Pieter Abbeel. Bigger, regularized, categorical: High-capacity value functions are efficient multi-task learners. arXiv preprint arXiv:2505.23150, 2025.
[69] Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
[70] Johan Obando-Ceron, Walter Mayor, Samuel Lavoie, Scott Fujimoto, Aaron Courville, and Pablo Samuel Castro. Simplicial embeddings improve sample efficiency in actor-critic agents. arXiv preprint arXiv:2510.13704, 2025.
[71] Daniel Palenicek, Florian Vogt, Joe Watson, and Jan Peters. Scaling off-policy reinforcement learning with batch and weight normalization. Advances in Neural Information Processing Systems (NeurIPS), 2025.
[72] Daniel Palenicek, Florian Vogt, Joe Watson, Ingmar Posner, and Jan Peters. XQC: Well-conditioned optimization accelerates deep reinforcement learning. International Conference on Learning Representations (ICLR), 2026.
[73] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[74] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542, 2017.
[75] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[76] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724, 2023.
[77] Antonin Raffin. Getting SAC to work on a massive parallel simulator: An RL journey with off-policy algorithms (part I). https://araffin.github.io/post/sac-massive-sim/, February 2025a.
[78] Antonin Raffin. Getting SAC to work on a massive parallel simulator: Tuning for speed (part II). https://araffin.github.io/post/tune-sac-isaac-sim/, July 2025b.
[79] Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Proceedings of the 5th Conference on Robot Learning (CoRL), volume 164 of Proceedings of Machine Learning Research, pages 1634–1644, 2022.
[80] Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber. State-dependent exploration for policy gradient methods. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), 2008.
[81] Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on robot learning, pages 91–100. PMLR, 2022a.
[82] Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on robot learning, pages 91–100. PMLR, 2022b.
[83] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? Advances in neural information processing systems, 31, 2018.
[84] Tom Schaul, Georg Ostrovski, Iurii Kemaev, and Diana Borsa. Return-based scaling: Yet another normalisation trick for deep rl. arXiv preprint arXiv:2105.05347, 2021.
[85] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[86] Clemens Schwarke, Mayank Mittal, Nikita Rudin, David Hoeller, and Marco Hutter. Rsl-rl: A learning library for robotics research. arXiv preprint arXiv:2509.10771, 2025.
[87] Younggyo Seo, Carmelo Sferrazza, Juyue Chen, Guanya Shi, Rocky Duan, and Pieter Abbeel. Learning sim-to-real humanoid locomotion in 15 minutes, 2025a. https://arxiv.org/abs/2512.01996 .
[88] Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint arXiv:2505.22642, 2025b.
[89] Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403.10506, 2024.
[90] Christian Robert Shelton. Importance sampling for reinforcement learning with multiple objectives. PhD thesis, Massachusetts Institute of Technology, 2001.
[91] Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
[92] Entong Su, Chengzhe Jia, Yuzhe Qin, Wenxuan Zhou, Annabella Macaluso, Binghao Huang, and Xiaolong Wang. Sim2real manipulation on unknown objects with tactile-based reinforcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 9234–9241. IEEE, 2024.
[93] Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216–224. Elsevier, 1990.
[94] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[95] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, et al. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint arXiv:2410.00425, 2024.
[96] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
[97] Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020, 2025.
[98] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012. 10.1109/IROS.2012.6386109 .
[99] Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024.
[100] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
[101] Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648, 2018.
[102] Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350, 2017.
[103] Gautham Vasan, Mohamed Elsayed, Alireza Azimi, Jiamin He, Fahim Shariar, Colin Bellinger, Martha White, and A. Rupam Mahmood. Deep policy gradient methods without batch updates, target networks, or replay buffers. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[104] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[105] Théo Vincent, Tim Faust, Yogesh Tripathi, Jan Peters, and Carlo D’Eramo. Eau de Q-network: Adaptive distillation of neural networks in deep reinforcement learning. In Reinforcement Learning Journal / Reinforcement Learning Conference (RLC), 2025a.
[106] Théo Vincent, Yogesh Tripathi, Tim Faust, Abdullah Akgül, Yaniv Oren, Melih Kandemir, Jan Peters, and Carlo D’Eramo. Bridging the performance gap between target-free and target-based reinforcement learning. arXiv preprint arXiv:2506.04398, 2025b.
[107] Claas Voelcker, Axel Brunnbauer, Marcel Hussing, Michal Nauman, Pieter Abbeel, Eric Eaton, Radu Grosu, Amir-massoud Farahmand, and Igor Gilitschenski. Relative entropy pathwise policy optimization. arXiv preprint arXiv:2507.11019, 2025.
[108] Jun Wang, Ying Yuan, Haichuan Che, Haozhi Qi, Yi Ma, Jitendra Malik, and Xiaolong Wang. Lessons from learning to spin" pens". arXiv preprint arXiv:2407.18902, 2024.
[109] Yuran Wang, Ruihai Wu, Yue Chen, Jiarui Wang, Jiaqi Liang, Ziyu Zhu, Haoran Geng, Jitendra Malik, Pieter Abbeel, and Hao Dong. Dexgarmentlab: Dexterous garment manipulation environment with generalizable policy. arXiv preprint arXiv:2505.11032, 2025.
[110] Zhengpeng Xie, Qiang Zhang, Fan Yang, Marco Hutter, and Renjing Xu. Simple policy optimization. arXiv preprint arXiv:2401.16025, 2024.
[111] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.
[112] Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujoco playground. arXiv preprint arXiv:2502.08844, 2025.
[113] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 32, 2019.
[114] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737–744. IEEE, 2020.









