1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

面向自监督强化学习的 1000 层网络：扩展深度可以带来新的目标到达能力

Kevin Wang (Princeton University, kw6487@princeton.edu); Ishaan Javali (Princeton University, ijavali@princeton.edu); Michał Bortkiewicz (Warsaw University of Technology, michal.bortkiewicz.dokt@pw.edu.pl); Tomasz Trzciński (Warsaw University of Technology; Tooploox; IDEAS Research Institute); Benjamin Eysenbach (Princeton University, eysenbach@princeton.edu)

Kevin Wang（普林斯顿大学，kw6487@princeton.edu）；Ishaan Javali（普林斯顿大学，ijavali@princeton.edu）；Michał Bortkiewicz（华沙理工大学，michal.bortkiewicz.dokt@pw.edu.pl）；Tomasz Trzciński（华沙理工大学；Tooploox；IDEAS 研究所）；Benjamin Eysenbach（普林斯顿大学，eysenbach@princeton.edu）

（译注：本对照依据 arXiv:2503.14858v5 的 HTML 版本，论文发表于 NeurIPS 2025，获最佳论文奖。）

Abstract

摘要

Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 – 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by $2\times$ – $50\times$, outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/.

扩大自监督学习的规模，推动了语言和视觉领域的突破，但强化学习（reinforcement learning, RL）一直没有取得类似的进展。本文研究自监督强化学习的几个基本构件，它们能大幅提升可扩展性，其中网络深度是关键因素。近年来大多数强化学习论文都使用浅层架构（大约 2–5 层），而我们证明，把深度增加到 1024 层可以显著提升性能。我们的实验在无监督的目标条件设定下进行：没有示范，也没有奖励，智能体必须（从零开始）探索，并学会如何最大化到达指定目标的可能性。在模拟的运动和操作任务上评估，我们的方法把自监督对比强化学习算法的性能提高了 $2\times$ – $50\times$，超过了其他目标条件基线。增加模型深度不仅提高了成功率，还从质上改变了学到的行为。项目网页和代码见：https://wang-kevin3290.github.io/scaling-crl/。

Figure 1: Scaling network depth yields performance gains across a suite of locomotion, navigation, and manipulation tasks, ranging from doubling performance to 50× improvements on Humanoid-based tasks. Notably, rather than scaling smoothly, performance often jumps at specific critical depths (e.g., 8 layers on Ant Big Maze, 64 on Humanoid U-Maze), which correspond to the emergence of qualitatively distinct policies (see Section 4).

图 1：在一系列运动、导航和操作任务上，扩展网络深度都带来了性能提升，幅度从翻倍到人形机器人类任务上的 50 倍不等。性能往往不是平滑增长，而是在特定的临界深度处跃升（例如 Ant Big Maze 上的 8 层、Humanoid U-Maze 上的 64 层），这些跃升对应着质上不同的策略的出现（见第 4 节）。

1 Introduction

1 引言

While scaling model size has been an effective recipe in many areas of machine learning, its role and impact in reinforcement learning (RL) remain unclear. The typical model size for state-based RL tasks is between 2 to 5 layers (Raffin et al., 2021; Huang et al., 2022). In contrast, it is not uncommon to use very deep networks in other domain areas; Llama 3 (Dubey et al., 2024) and Stable Diffusion 3 (Esser et al., 2024) have hundreds of layers. In fields such as vision (Radford et al., 2021; Zhai et al., 2021; Dehghani et al., 2023) and language (Srivastava et al., 2023), models often only acquire the ability to solve certain tasks once they are larger than a critical scale. In the RL setting, many researchers have searched for similar emergent phenomena (Srivastava et al., 2023), but these papers typically report only small marginal benefits and typically only on tasks where small models already achieve some degree of success (Nauman et al., 2024b; Lee et al., 2024; Farebrother et al., 2024). A key open question in RL today is whether it is possible to achieve similar jumps in performance by scaling RL networks.

扩大模型规模在机器学习的许多领域都是行之有效的办法，但它在强化学习中的作用和影响仍不清楚。基于状态的强化学习任务，典型模型规模在 2 到 5 层之间（Raffin et al., 2021; Huang et al., 2022）。相比之下，在其他领域使用非常深的网络并不罕见：Llama 3（Dubey et al., 2024）和 Stable Diffusion 3（Esser et al., 2024）都有数百层。在视觉（Radford et al., 2021; Zhai et al., 2021; Dehghani et al., 2023）和语言（Srivastava et al., 2023）等领域，模型往往要超过某个临界规模，才获得解决某些任务的能力。在强化学习中，许多研究者寻找过类似的涌现现象（Srivastava et al., 2023），但这些论文通常只报告很小的边际收益，而且通常只在小模型已经取得一定成功的任务上（Nauman et al., 2024b; Lee et al., 2024; Farebrother et al., 2024）。当前强化学习的一个关键开放问题是：能否通过扩展强化学习网络，获得类似的性能跃升。

At first glance, it makes sense why training very large RL networks should be difficult: the RL problem provides very few bits of feedback (e.g., only a sparse reward after a long sequence of observations), so the ratio of feedback to parameters is very small. The conventional wisdom (LeCun, 2016), reflected in many recent models (Radford, 2018; Chen et al., 2020; Goyal et al., 2019), has been that large AI systems must be trained primarily in a self-supervised fashion and that RL should only be used to finetune these models. Indeed, many of the recent breakthroughs in other fields have been primarily achieved with self-supervised methods, whether in computer vision (Caron et al., 2021; Radford et al., 2021; Liu et al., 2024), NLP (Srivastava et al., 2023), or multimodal learning (Zong et al., 2024). Thus, if we hope to scale reinforcement learning methods, self-supervision will likely be a key ingredient.

乍看之下，训练非常大的强化学习网络之所以困难，原因不难理解：强化学习问题提供的反馈信息量很少（例如，一长串观测之后只有一个稀疏奖励），因此反馈与参数之比非常小。传统观点（LeCun, 2016）认为，大型 AI 系统必须主要以自监督方式训练，强化学习只应用来微调这些模型；许多近期模型也体现了这一观点（Radford, 2018; Chen et al., 2020; Goyal et al., 2019）。确实，其他领域近来的许多突破主要是靠自监督方法取得的，无论是计算机视觉（Caron et al., 2021; Radford et al., 2021; Liu et al., 2024）、自然语言处理（Srivastava et al., 2023）还是多模态学习（Zong et al., 2024）。因此，如果我们希望扩展强化学习方法，自监督很可能是一个关键要素。

In this paper, we will study building blocks for scaling reinforcement learning. Our first step is to rethink the conventional wisdom above: “reinforcement learning” and “self-supervised learning” are not diametric learning rules, but rather can be married together into self-supervised RL systems that explore and learn policies without reference to a reward function or demonstrations (Eysenbach et al., 2021; Eysenbach et al., 2022; Lee et al., 2022). In this work, we use one of the simplest self-supervised RL algorithms, contrastive RL (CRL) (Eysenbach et al., 2022). The second step is to recognize the importance of increasing data availability. We will do this by building on recent GPU-accelerated RL frameworks (Makoviychuk et al., 2021; Rutherford et al., 2023; Rudin et al., 2022; Bortkiewicz et al., 2024). The third step is to increase network depth, using networks that are up to $100\times$ deeper than those typically found in prior work. Stabilizing the training of such networks will require incorporating architectural techniques from prior work, including residual connections (He et al., 2015), layer normalization (Ba et al., 2016), and Swish activation (Ramachandran et al., 2018). Our experiments will also study the relative importance of batch size and network width.

本文研究扩展强化学习的基本构件。第一步是重新审视上述传统观点：“强化学习”和“自监督学习”并不是截然对立的学习规则，二者可以结合成自监督强化学习系统，在不参考奖励函数或示范的情况下探索并学习策略（Eysenbach et al., 2021; Eysenbach et al., 2022; Lee et al., 2022）。本文使用最简单的自监督强化学习算法之一：对比强化学习（contrastive RL, CRL）（Eysenbach et al., 2022）。第二步是认识到增加可用数据的重要性，为此我们以近来的 GPU 加速强化学习框架为基础（Makoviychuk et al., 2021; Rutherford et al., 2023; Rudin et al., 2022; Bortkiewicz et al., 2024）。第三步是增加网络深度，所用网络比以往工作中常见的深最多 $100\times$。要稳定训练这样的网络，需要吸收以往工作中的架构技术，包括残差连接（He et al., 2015）、层归一化（Ba et al., 2016）和 Swish 激活（Ramachandran et al., 2018）。我们的实验还将研究批大小和网络宽度的相对重要性。

The primary contribution of this work is to show that a method that integrates these building blocks into a single RL approach exhibits strong scalability:

本文的主要贡献是表明，把这些基本构件整合到同一种强化学习方法中，可以表现出很强的可扩展性：

• Empirical Scalability: We observe a significant performance increase, more than $20\times$ in half of the environments and outperforming other standard goal-conditioned baselines. These performance gains correspond to qualitatively distinct policies that emerge as the scale increases.
• Scaling Depth in Network Architecture: While many prior RL works have primarily focused on increasing network width, they often report limited or even negative returns when expanding depth (Lee et al., 2024; Nauman et al., 2024b). In contrast, our approach unlocks the ability to scale along the axis of depth, yielding performance improvements that surpass those from scaling width alone (see Sec. 4).
• Empirical Analysis: We conduct an extensive analysis of the key components in our scaling approach, uncovering critical factors and offering new insights.

• 实证上的可扩展性：我们观察到性能大幅提升，在一半的环境中超过 $20\times$，并超过其他标准的目标条件基线。这些性能提升对应着随规模增大而出现的、质上不同的策略。
• 扩展网络架构的深度：以往许多强化学习工作主要关注增加网络宽度，而增加深度时往往报告收益有限甚至为负（Lee et al., 2024; Nauman et al., 2024b）。相比之下，我们的方法打开了沿深度轴扩展的能力，带来的性能提升超过只扩展宽度（见第 4 节）。
• 实证分析：我们对扩展方法中的关键组件做了大量分析，找出了关键因素，并给出新的认识。

We anticipate that future research may build on this foundation by uncovering additional building blocks.

我们预期，未来的研究可以在此基础上发现更多的基本构件。

2 Related Work

2 相关工作

Natural Language Processing (NLP) and Computer Vision (CV) have recently converged in adopting similar architectures (i.e. transformers) and shared learning paradigms (i.e self-supervised learning), which together have enabled transformative capabilities of large-scale models (Vaswani et al., 2017; Srivastava et al., 2023; Zhai et al., 2021; Dehghani et al., 2023; Wei et al., 2022). In contrast, achieving similar advancements in reinforcement learning (RL) remains challenging. Several studies have explored the obstacles to scaling large RL models, including parameter underutilization (Obando-Ceron et al., 2024), plasticity and capacity loss (Lyle et al., 2024; Lyle et al., 2022), data sparsity (Andrychowicz et al., 2017; LeCun, 2016), and training instabilities (Ota et al., 2021; Henderson et al., 2018; Van Hasselt et al., 2018; Nauman et al., 2024a). As a result, current efforts to scale RL models are largely restricted to specific problem domains, such as imitation learning (Tuyls et al., 2024), multi-agent games (Neumann and Gros, 2022), language-guided RL (Driess et al., 2023; Ahn et al., 2022), and discrete action spaces (Obando-Ceron et al., 2024; Schwarzer et al., 2023).

自然语言处理（NLP）和计算机视觉（CV）近来趋于采用相似的架构（即 Transformer）和共同的学习范式（即自监督学习），二者共同成就了大规模模型的变革性能力（Vaswani et al., 2017; Srivastava et al., 2023; Zhai et al., 2021; Dehghani et al., 2023; Wei et al., 2022）。相比之下，在强化学习中取得类似进展仍然困难。一些研究探讨了扩展大型强化学习模型的障碍，包括参数利用不足（Obando-Ceron et al., 2024）、可塑性与容量损失（Lyle et al., 2024; Lyle et al., 2022）、数据稀疏（Andrychowicz et al., 2017; LeCun, 2016）以及训练不稳定（Ota et al., 2021; Henderson et al., 2018; Van Hasselt et al., 2018; Nauman et al., 2024a）。因此，目前扩展强化学习模型的努力大多局限在特定问题领域，例如模仿学习（Tuyls et al., 2024）、多智能体博弈（Neumann and Gros, 2022）、语言引导的强化学习（Driess et al., 2023; Ahn et al., 2022）和离散动作空间（Obando-Ceron et al., 2024; Schwarzer et al., 2023）。

Recent approaches suggest several promising directions, including new architectural paradigms (Obando-Ceron et al., 2024), distributed training approaches (Ota et al., 2021; Espeholt et al., 2018), distributional RL (Kumar et al., 2023), and distillation (Team et al., 2023). Compared to these approaches, our method makes a simple extension to an existing self-supervised RL algorithm. The most recent works in this vein include Lee et al. (2024) and Nauman et al. (2024b), which leverage residual connections to facilitate the training of wider networks. These efforts primarily focus on network width, noting limited gains from additional depth, thus both works use architectures with only four MLP layers. In our method, we find that scaling width indeed improves performance (Section 4.4); however, our approach also enables scaling along depth, proving to be more powerful than width alone.

近期的方法提出了几个有前景的方向，包括新的架构范式（Obando-Ceron et al., 2024）、分布式训练方法（Ota et al., 2021; Espeholt et al., 2018）、分布式强化学习（distributional RL）（Kumar et al., 2023）和蒸馏（Team et al., 2023）。与这些方法相比，我们的方法只是对现有的自监督强化学习算法做了一个简单扩展。这一方向上最近的工作是 Lee et al.（2024）和 Nauman et al.（2024b），它们利用残差连接来帮助训练更宽的网络。这些工作主要关注网络宽度，并指出增加深度收益有限，因此两者使用的架构都只有四层 MLP。在我们的方法中，扩展宽度确实能提升性能（第 4.4 节）；但我们的方法也能沿深度扩展，而且证明比只扩展宽度更有效。

One notable effort to train deeper networks is described by Farebrother et al. (2024), who cast value-based RL into a classification problem by discretizing the TD objective into a categorical cross-entropy loss. This approach draws on the conjecture that classification-based methods can be more robust and stable and thus may exhibit better scaling properties than their regressive counterparts (Torgo and Gama, 1996; Farebrother et al., 2024). The CRL algorithm that we use effectively uses a cross-entropy loss as well (Eysenbach et al., 2022). Its InfoNCE objective is a generalization of the cross-entropy loss, thereby performing RL tasks by effectively classifying whether current states and actions belong to the same or different trajectory that leads toward a goal state. In this vein, our work serves as a second piece of evidence that classification, much like cross-entropy’s role in the scaling success in NLP, could be a potential building block in RL.

Farebrother et al.（2024）描述了一项训练更深网络的重要尝试：他们把 TD 目标离散化为类别交叉熵损失，把基于价值的强化学习转化为分类问题。这一方法依据的猜想是，基于分类的方法可能比回归方法更稳健、更稳定，因而可能有更好的扩展特性（Torgo and Gama, 1996; Farebrother et al., 2024）。我们使用的 CRL 算法实际上也用了交叉熵损失（Eysenbach et al., 2022）。它的 InfoNCE 目标是交叉熵损失的推广，通过判断当前状态和动作与通往某个目标状态的轨迹是否属于同一条轨迹，来完成强化学习任务。从这个意义上说，我们的工作是第二份证据，表明分类可能成为强化学习的一个基本构件，就像交叉熵在 NLP 扩展成功中所起的作用一样。

3 Preliminaries

3 预备知识

This section introduces notation and definitions for goal-conditioned RL and contrastive RL. Our focus is on online RL, where a replay buffer stores the most recent trajectories, and the critic is trained in a self-supervised manner.

本节介绍目标条件强化学习和对比强化学习的记号与定义。我们关注在线强化学习：回放缓冲区（replay buffer）存储最近的轨迹，评论家（critic）以自监督方式训练。

Goal-Conditioned Reinforcement Learning

We define a goal-conditioned MDP as tuple $\mathcal{M}_{g}=(\mathcal{S},\mathcal{A},p_{0},p,p_{g},r_{g},\gamma)$, where the agent interacts with the environment to reach arbitrary goals (Kaelbling, 1993; Andrychowicz et al., 2017; Blier et al., 2021). At every time step $t$, the agent observes state $s_{t}\in\mathcal{S}$ and performs a corresponding action $a_{t}\in\mathcal{A}$. The agent starts interaction in states sampled from $p_{0}(s_{0})$, and the interaction dynamics are defined by the transition probability distribution $p(s_{t+1}\mid s_{t},a_{t})$. Goals $g\in\mathcal{G}$ are defined in a goal space $\mathcal{G}$, which is related to $\mathcal{S}$ via a mapping $f:\mathcal{S}\to\mathcal{G}$. For example, $\mathcal{G}$ may correspond to a subset of state dimensions. The prior distribution over goals is defined by $p_{g}(g)$. The reward function is defined as the probability density of reaching the goal in the next time step $r_{g}(s_{t},a_{t})\triangleq(1-\gamma)p(s_{t+1}=g\mid s_{t},a_{t})$, with discount factor $\gamma$.

目标条件强化学习

我们把目标条件 MDP 定义为元组 $\mathcal{M}_{g}=(\mathcal{S},\mathcal{A},p_{0},p,p_{g},r_{g},\gamma)$，智能体与环境交互以到达任意目标（Kaelbling, 1993; Andrychowicz et al., 2017; Blier et al., 2021）。在每个时间步 $t$，智能体观测状态 $s_{t}\in\mathcal{S}$，并执行相应的动作 $a_{t}\in\mathcal{A}$。智能体从按 $p_{0}(s_{0})$ 采样的状态开始交互，交互动态由转移概率分布 $p(s_{t+1}\mid s_{t},a_{t})$ 定义。目标 $g\in\mathcal{G}$ 定义在目标空间 $\mathcal{G}$ 中，它通过映射 $f:\mathcal{S}\to\mathcal{G}$ 与 $\mathcal{S}$ 关联。例如，$\mathcal{G}$ 可以对应状态维度的一个子集。目标的先验分布由 $p_{g}(g)$ 定义。奖励函数定义为下一时间步到达目标的概率密度 $r_{g}(s_{t},a_{t})\triangleq(1-\gamma)p(s_{t+1}=g\mid s_{t},a_{t})$，其中 $\gamma$ 为折扣因子。

In this setting, the goal-conditioned policy $\pi(a\mid s,g)$ receives both the current observation of the environment as well as a goal. We define the discounted state visitation distribution as $p^{\pi\left(\cdot\mid\cdot,g\right)}_{\gamma}(s)\triangleq(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}p^{\pi\left(\cdot\mid\cdot,g\right)}_{t}(s)$, where $p_{t}^{\pi}(s)$ is the probability that policy $\pi$ visits $s$ after exactly $t$ steps, when conditioned with $g$. This last expression is precisely the $Q$-function of the policy $\pi(\cdot\mid\cdot,g)$ for the reward $r_{g}$: $Q^{\pi}_{g}(s,a)\triangleq p^{\pi\left(\cdot\mid\cdot,g\right)}_{\gamma}(g\mid s,a)$. The objective is to maximize the expected reward:

在这一设定中，目标条件策略 $\pi(a\mid s,g)$ 同时接收环境的当前观测和一个目标。我们把折扣状态访问分布定义为 $p^{\pi\left(\cdot\mid\cdot,g\right)}_{\gamma}(s)\triangleq(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}p^{\pi\left(\cdot\mid\cdot,g\right)}_{t}(s)$，其中 $p_{t}^{\pi}(s)$ 是以 $g$ 为条件时，策略 $\pi$ 恰好在 $t$ 步之后访问 $s$ 的概率。后一个表达式恰好就是策略 $\pi(\cdot\mid\cdot,g)$ 在奖励 $r_{g}$ 下的 $Q$ 函数：$Q^{\pi}_{g}(s,a)\triangleq p^{\pi\left(\cdot\mid\cdot,g\right)}_{\gamma}(g\mid s,a)$。目标是最大化期望奖励：

$$\max_{\pi}\mathbb{E}_{p_{0}(s_{0}),p_{g}\left(g\right),\pi\left(\cdot\mid\cdot,g\right)}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{g}\left(s_{t},a_{t}\right)\right]. \tag{1}$$

Contrastive Reinforcement Learning.

Our experiments will use the contrastive RL algorithm (Eysenbach et al., 2022) to solve goal-conditioned problems. Contrastive RL is an actor-critic method; we will use $f_{\phi,\psi}(s,a,g)$ to denote the critic and $\pi_{\theta}(a\mid s,g)$ to denote the policy. The critic is parametrized with two neural networks that return state, action pair embedding $\phi(s,a)$ and goal embedding $\psi(g)$. The critic’s output is defined as the $l^{2}$-norm between these embeddings: $f_{\phi,\psi}(s,a,g)=\|\phi(s,a)-\psi(g)\|_{2}$. The critic is trained with the InfoNCE objective (Sohn, 2016) as in previous works (Eysenbach et al., 2022; Eysenbach et al., 2021; Zheng et al., 2023; Zheng et al., 2024; Myers et al., 2024; Bortkiewicz et al., 2024). Training is conducted on batches $\mathcal{B}$, where ${\color{#008C00}s_{i},a_{i},g_{i}}$ represent the state, action, and goal (future state) sampled from the same trajectory, while ${\color{#BF0000}g_{j}}$ represents a goal sampled from a different, random trajectory. The objective function is defined as:

对比强化学习。

我们的实验用对比强化学习算法（Eysenbach et al., 2022）求解目标条件问题。对比强化学习是一种行动者—评论家（actor-critic）方法；我们用 $f_{\phi,\psi}(s,a,g)$ 表示评论家，用 $\pi_{\theta}(a\mid s,g)$ 表示策略。评论家由两个神经网络参数化，分别输出状态—动作对嵌入 $\phi(s,a)$ 和目标嵌入 $\psi(g)$。评论家的输出定义为这两个嵌入之间的 $l^{2}$ 范数：$f_{\phi,\psi}(s,a,g)=\|\phi(s,a)-\psi(g)\|_{2}$。与以往工作（Eysenbach et al., 2022; Eysenbach et al., 2021; Zheng et al., 2023; Zheng et al., 2024; Myers et al., 2024; Bortkiewicz et al., 2024）一样，评论家用 InfoNCE 目标（Sohn, 2016）训练。训练在批次 $\mathcal{B}$ 上进行，其中 ${\color{#008C00}s_{i},a_{i},g_{i}}$ 表示从同一条轨迹中采样的状态、动作和目标（未来状态），而 ${\color{#BF0000}g_{j}}$ 表示从另一条随机轨迹中采样的目标。目标函数定义为：

$$\min_{\phi,\psi}\mathbb{E}_{\mathcal{B}}\left[-\sum\nolimits_{{\color{#008C00}i=1}}^{|\mathcal{B}|}\log\biggl({\frac{e^{f_{\phi,\psi}({\color{#008C00}s_{i}},{\color{#008C00}a_{i}},{\color{#008C00}g_{i}})}}{\sum\nolimits_{{\color{#BF0000}j=1}}^{K}e^{f_{\phi,\psi}({\color{#008C00}s_{i}},{\color{#008C00}a_{i}},{\color{#BF0000}g_{j}})}}}\biggr)\right].$$

The policy $\pi_{\theta}(a\mid s,g)$ is trained to maximize the critic:

策略 $\pi_{\theta}(a\mid s,g)$ 的训练目标是最大化评论家的输出：

$$\max_{\pi_{\theta}}\mathbb{E}{\begin{subarray}{c}p{0}(s_{0}),p(s_{t+1}\mid s_{t},a_{t}),\
p_{g}\left(g\right),\pi_{\theta}(a\mid s,g)\end{subarray}}\left[f_{\phi,\psi}(s,a,g)\right].$$

Residual Connections

Figure 2: Architecture. Our approach integrates residual connections into both the actor and critic networks of the Contrastive RL algorithm. The depth of this residual architecture is defined as the total number of Dense layers across the residual blocks, which, with our residual block size of 4, equates to $4N$.

残差连接

图 2：架构。我们的方法在对比强化学习算法的行动者网络和评论家网络中都加入了残差连接。这一残差架构的深度定义为所有残差块中 Dense 层的总数；我们的残差块大小为 4，因此深度等于 $4N$。

We incorporate residual connections (He et al., 2015) into our architecture, following their successful use in RL (Farebrother et al., 2024; Lee et al., 2024; Nauman et al., 2024b). A residual block transforms a given representation $\mathbf{h}_{i}$ by adding a learned residual function $F_{i}(\mathbf{h}_{i})$ to the original representation. Mathematically, this is expressed as:

我们在架构中加入残差连接（He et al., 2015），这是沿用它在强化学习中的成功经验（Farebrother et al., 2024; Lee et al., 2024; Nauman et al., 2024b）。残差块把一个学到的残差函数 $F_{i}(\mathbf{h}_{i})$ 加到原始表示 $\mathbf{h}_{i}$ 上，以此变换该表示。数学上写作：

$$\mathbf{h}_{i+1}=\mathbf{h}_{i}+F_{i}\left(\mathbf{h}_{i}\right)$$

where $\mathbf{h}_{i+1}$ is the output representation, $\mathbf{h}_{i}$ is the input representation, and $F_{i}(\mathbf{h}_{i})$ is a transformation learned through the network (e.g., using one or more layers). The addition ensures that the network learns modifications to the input rather than entirely new transformations, helping to preserve useful features from earlier layers. Residual connections improve gradient propagation by introducing shortcut paths (He et al., 2016; Veit et al., 2016), enabling more effective training of deep models.

其中 $\mathbf{h}_{i+1}$ 是输出表示，$\mathbf{h}_{i}$ 是输入表示，$F_{i}(\mathbf{h}_{i})$ 是网络学到的变换（例如由一层或多层实现）。这种相加使网络学习对输入的修改，而不是全新的变换，有助于保留前面各层的有用特征。残差连接通过引入捷径路径改善梯度传播（He et al., 2016; Veit et al., 2016），使深层模型能更有效地训练。

4 Experiments

4 实验

4.1 Experimental Setup

4.1 实验设置

Environments.

All RL experiments use the JaxGCRL codebase (Bortkiewicz et al., 2024), which facilitates fast online GCRL experiments based on Brax (Freeman et al., 2021) and MJX (Todorov et al., 2012) environments. The specific environments used are a range of locomotion, navigation, and robotic manipulation tasks, for details see Appendix B. We use a sparse reward setting, with $r=1$ only when the agent is in the goal proximity. For evaluation, we measure the number of time steps (out of 1000) that the agent is near the goal. When reporting an algorithm’s performance as a single number, we compute the average score over the last five epochs of training.

环境。

所有强化学习实验都使用 JaxGCRL 代码库（Bortkiewicz et al., 2024），它基于 Brax（Freeman et al., 2021）和 MJX（Todorov et al., 2012）环境，便于快速开展在线 GCRL 实验。所用的具体环境是一系列运动、导航和机器人操作任务，详见附录 B。我们采用稀疏奖励设定，只有智能体处于目标附近时 $r=1$。评估时，我们统计智能体在（共 1000 个）时间步中位于目标附近的步数。用单个数字报告算法性能时，我们取训练最后五个轮次的平均分数。

Architectural Components

We employ residual connections from the ResNet architecture (He et al., 2015), with each residual block consisting of four repeated units of a Dense layer, a Layer Normalization (Ba et al., 2016) layer, and Swish activation (Ramachandran et al., 2018). We apply the residual connections immediately following the final activation of the residual block, as shown in Figure 2. In this paper, we define the depth of the network as the total number of Dense layers across all residual blocks in the architecture. In all experiments, the depth refers to the configuration of the actor network and both critic encoder networks, which are scaled jointly, except for the ablation experiment in Section 4.4.

架构组件

我们采用 ResNet 架构（He et al., 2015）中的残差连接，每个残差块由四个重复单元组成，每个单元包括一个 Dense 层、一个层归一化（Ba et al., 2016）层和 Swish 激活（Ramachandran et al., 2018）。如图 2 所示，残差连接紧接在残差块最后一个激活之后。本文把网络深度定义为架构中所有残差块的 Dense 层总数。在所有实验中，除第 4.4 节的消融实验外，深度指行动者网络和两个评论家编码器网络的配置，三者同时扩展。

4.2 Scaling Depth in Contrastive RL

4.2 在对比强化学习中扩展深度

We start by studying how increasing network depth can increase performance. Both the JaxGCRL benchmark and relevant prior work (Lee et al., 2024; Nauman et al., 2024b; Zheng et al., 2024) use MLPs with a depth of 4, and as such we adopt it as our baseline. In contrast, we will study networks of depth 8, 16, 32, and 64. The results in Figure 1 demonstrate that deeper networks achieve significant performance improvements across a diverse range of locomotion, navigation, and manipulation tasks. Compared to the 4-layer models typical in prior work, deeper networks achieve $2-5\times$ gains in robotic manipulation tasks, over $20\times$ gains in long-horizon maze tasks such as Ant U4-Maze and Ant U5-Maze, and over $50\times$ gains in humanoid-based tasks. The full table of performance increases up to depth 64 is provided in Table 1.

我们先研究增加网络深度如何提升性能。JaxGCRL 基准和相关的以往工作（Lee et al., 2024; Nauman et al., 2024b; Zheng et al., 2024）都使用深度为 4 的 MLP，因此我们把它作为基线。我们则研究深度为 8、16、32 和 64 的网络。图 1 的结果表明，在各种运动、导航和操作任务上，更深的网络都取得了显著的性能提升。与以往工作中常见的 4 层模型相比，更深的网络在机器人操作任务上提升 $2-5\times$，在 Ant U4-Maze 和 Ant U5-Maze 等长时程迷宫任务上提升超过 $20\times$，在人形机器人类任务上提升超过 $50\times$。深度扩展到 64 时完整的性能提升表见表 1。

In Figure 12, we present results the same 10 environments, but compared against SAC, SAC+HER, TD3+HER, GCBC, and GCSL. Scaling CRL leads to substantial performance improvements, outperforming all other baselines in 8 out of 10 tasks. The only exception is SAC on the Humanoid Maze environments, where it exhibits greater sample efficiency early on; however, scaled CRL eventually reaches comparable performance. These results highlight that scaling the depth of the CRL algorithm enables state-of-the-art performance in goal-conditioned reinforcement learning.

在图 12 中，我们在同样的 10 个环境上给出结果，但与 SAC、SAC+HER、TD3+HER、GCBC 和 GCSL 比较。扩展后的 CRL 带来大幅性能提升，在 10 个任务中的 8 个上超过所有其他基线。唯一的例外是 Humanoid Maze 环境中的 SAC，它在早期样本效率更高；但扩展后的 CRL 最终达到了相当的性能。这些结果表明，扩展 CRL 算法的深度能在目标条件强化学习中取得最先进的性能。

4.3 Emergent Policies Through Depth

4.3 随深度涌现的策略

Figure 3: Increasing depth results in new capabilities: Row 1: A depth-4 agent collapses and throws itself toward the goal. Row 2: A depth-16 agent walks upright. Row 3: A depth-64 agent struggles and falls. Row 4: A depth-256 agent vaults the wall acrobatically.

图 3：增加深度带来新能力。第 1 行：深度为 4 的智能体倒下，把自己扑向目标。第 2 行：深度为 16 的智能体直立行走。第 3 行：深度为 64 的智能体挣扎着摔倒。第 4 行：深度为 256 的智能体以杂技般的动作翻越墙壁。

A closer examination of the results from the performance curves in Figure 1 reveals a notable pattern: instead of a gradual improvement in performance as depth increases, there are pronounced jumps that occur once a critical depth threshold is reached (also shown in Figure 6). The critical depths vary by environment, ranging from 8 layers (e.g. Ant Big Maze) to 64 layers in the Humanoid U-Maze task, with further jumps occurring even at depths of 1024 layers (see the Testing Limits section, Section 4.4).

仔细查看图 1 中的性能曲线，可以发现一个值得关注的规律：性能并不随深度增加而逐渐提高，而是在达到某个临界深度阈值后出现明显的跃升（另见图 6）。临界深度因环境而异，从 8 层（例如 Ant Big Maze）到 Humanoid U-Maze 任务中的 64 层不等，甚至在 1024 层深度时还会出现进一步的跃升（见第 4.4 节的“测试极限”部分）。

Prompted by this observation, we visualized the learned policies at various depths and found qualitatively distinct skills and behaviors exhibited. This is particularly pronounced in the humanoid-based tasks, as illustrated in Figure 3. Networks with a depth of 4 exhibit rudimentary policies where the agent either falls or throws itself toward the target. Only at a critical depth of 16 does the agent develop the ability to walk upright into the goal. In the Humanoid U-Maze environment, networks of depth 64 struggle to navigate around the intermediary wall, collapsing on the ground. Remarkably at a depth of 256, the agent learns unique behaviors on Humanoid U-Maze. These behaviors include folding forward into a leveraged position to propel itself over walls and shifting into a seated posture over the intermediary obstacle to worm its way toward the goal (one of these policies is illustrated in the fourth row of Figure 3). To the best of our knowledge, this is the first goal-conditioned approach to document such behaviors on the humanoid environment.

受这一观察启发，我们把不同深度下学到的策略可视化，发现它们表现出质上不同的技能和行为。这在人形机器人类任务中尤其明显，如图 3 所示。深度为 4 的网络表现出初级的策略，智能体要么摔倒，要么把自己扑向目标。直到临界深度 16，智能体才学会直立走进目标。在 Humanoid U-Maze 环境中，深度为 64 的网络难以绕过中间的墙，会瘫倒在地。值得一提的是，在深度为 256 时，智能体在 Humanoid U-Maze 上学会了独特的行为。这些行为包括向前折叠身体形成杠杆姿势，把自己撑过墙，以及在中间障碍物上转成坐姿，像虫子一样蠕动着向目标前进（图 3 第四行展示了其中一种策略）。据我们所知，这是第一个在人形机器人环境中记录到此类行为的目标条件方法。

4.4 What Matters for CRL Scaling

4.4 什么对 CRL 扩展至关重要

Width vs. Depth

Figure 4: Scaling network width vs. depth. Here, we reflect findings from previous works (Lee et al., 2024; Nauman et al., 2024b) which suggest that increasing network width can enhance performance. However, in contrast to prior work, our method is able to scale depth, yielding more impactful performance gains. For instance, in the Humanoid environment, raising the width to 2048 (depth=4) fails to match the performance achieved by simply doubling the depth to 8 (width=256). The comparative advantage of scaling depth is more pronounced as the observational dimensionality increases.

宽度与深度

图 4：扩展网络宽度与扩展深度的比较。这里我们再现了以往工作（Lee et al., 2024; Nauman et al., 2024b）的发现，即增加网络宽度可以提升性能。但与以往工作不同，我们的方法能够扩展深度，带来更大的性能提升。例如在 Humanoid 环境中，把宽度提高到 2048（深度为 4），也比不上只把深度翻倍到 8（宽度为 256）所达到的性能。观测维度越高，扩展深度的相对优势越明显。

Past literature has shown that scaling network width can be effective  (Lee et al., 2024; Nauman et al., 2024b). In Figure 4, we find that scaling width is also helpful in our experiments: wider networks consistently outperform narrower networks (depth held constant at 4). However, depth seems to be a more effective axis for scaling: simply doubling the depth to 8 (width held constant at 256) outperforms the widest networks in all three environments. The advantage of depth scaling is most pronounced in the Humanoid environment (observation dimension 268), followed by Ant Big Maze (dimension 29) and Arm Push Easy (dimension 17), suggesting that the comparative benefit may increase with higher observation dimensionality.

以往文献表明，扩展网络宽度是有效的（Lee et al., 2024; Nauman et al., 2024b）。在图 4 中，我们发现扩展宽度在我们的实验中也有帮助：更宽的网络始终优于更窄的网络（深度固定为 4）。但深度似乎是更有效的扩展维度：只把深度翻倍到 8（宽度固定为 256），在全部三个环境中都优于最宽的网络。深度扩展的优势在 Humanoid 环境（观测维度 268）中最明显，其次是 Ant Big Maze（维度 29）和 Arm Push Easy（维度 17），这表明观测维度越高，相对收益可能越大。

Note additionally that the parameter count scales linearly with width but quadratically with depth. For comparison, a network with 4 MLP layers and 2048 hidden units has roughly 35M parameters, while one with a depth of 32 and 256 hidden units has only around 2M. Therefore, when operating under a fixed FLOP compute budget or specific memory constraints, depth scaling may be a more computationally efficient approach to improving network performance.

另外请注意，参数量随宽度线性增长，随深度则呈二次增长（译注：原文如此；按常规计算，MLP 参数量随宽度呈二次增长、随深度线性增长，与后文的例子一致）。作为对比，一个 4 层 MLP、2048 个隐藏单元的网络约有 35M 参数，而深度为 32、256 个隐藏单元的网络只有约 2M 参数。因此，在固定的 FLOP 计算预算或特定内存限制下，扩展深度可能是提升网络性能计算效率更高的途径。

Scaling the Actor vs. Critic Networks

Figure 5: Critical depth and residual connections. Incrementally increasing depth results in marginal performance gains (left). However, once a critical threshold is reached, performance improves dramatically (right) for networks with residual connections.

扩展行动者网络与评论家网络

图 5：临界深度与残差连接。逐步增加深度只带来边际的性能提升*（左）。但一旦达到临界阈值，带残差连接的网络性能会大幅提高（右）*。

To investigate the role of scaling in the actor and critic networks, Figure 6 presents the final performance for various combinations of actor and critic depths across three environments. Prior work (Nauman et al., 2024b; Lee et al., 2024) focuses on scaling the critic network, finding that scaling the actor degrades performance. In contrast, while we do find that scaling the critic is more impactful in two of the three environments (Humanoid, Arm Push Easy), our method benefits from scaling the actor network jointly, with one environment (Ant Big Maze) demonstrating actor scaling to be more impactful. Thus, our method suggests that scaling both the actor and critic networks can play a complementary role in enhancing performance.

为了研究扩展行动者网络和评论家网络各自的作用，图 6 给出了三个环境中不同行动者深度与评论家深度组合的最终性能。以往工作（Nauman et al., 2024b; Lee et al., 2024）关注扩展评论家网络，并发现扩展行动者会降低性能。相比之下，虽然我们确实发现扩展评论家在三个环境中的两个（Humanoid、Arm Push Easy）作用更大，但我们的方法也能从同时扩展行动者网络中获益，而且在一个环境（Ant Big Maze）中，扩展行动者的作用更大。因此，我们的方法表明，扩展行动者网络和评论家网络可以在提升性能上相辅相成。

Deep Networks Unlock Batch Size Scaling

Figure 7: Deeper networks unlock batch size scaling. We find that as depth increases from 4 to 64 in Humanoid, larger networks can effectively leverage batch size scaling to achieve further improvements.

深层网络打开了批大小扩展的空间

图 7：更深的网络打开了批大小扩展的空间。我们发现，在 Humanoid 中，随着深度从 4 增加到 64，更大的网络能有效利用批大小扩展取得进一步提升。

Scaling batch size has been well-established in other areas of machine learning (Chen et al., 2022; Zhang et al., 2024). However, this approach has not translated as effectively to reinforcement learning (RL), and prior work has even reported negative impacts on value-based RL (Obando-Ceron et al., 2023). Indeed, in our experiments, simply increasing the batch size for the original CRL networks yields only marginal differences in performance (Figure 7, top left).

扩大批大小在机器学习的其他领域已是成熟做法（Chen et al., 2022; Zhang et al., 2024）。但这一做法在强化学习中效果没有那么好，以往工作甚至报告它对基于价值的强化学习有负面影响（Obando-Ceron et al., 2023）。确实，在我们的实验中，只增加原始 CRL 网络的批大小，性能差别很小（图 7 左上）。

At first glance, this might seem counterintuitive: since reinforcement learning typically involves fewer informational bits per piece of training data (LeCun, 2016), one might expect higher variance in batch loss or gradients, suggesting the need for larger batch sizes to compensate. At the same time, this possibility hinges on whether the model in question can actually make use of a bigger batch size—in domains of ML where scaling has been successful, larger batch sizes usually bring the most benefit when coupled with sufficiently large models (Zhang et al., 2024; Chen et al., 2022). One hypothesis is that the small models traditionally used in RL may obscure the underlying benefits of larger batch size.

乍看之下，这似乎有违直觉：由于强化学习中每条训练数据通常携带的信息量较少（LeCun, 2016），人们可能预期批损失或梯度的方差更大，因而需要更大的批大小来弥补。与此同时，这种可能性取决于所用模型能否真正利用更大的批大小：在扩展取得成功的机器学习领域，更大的批大小通常要与足够大的模型配合才能带来最大收益（Zhang et al., 2024; Chen et al., 2022）。一种假设是，强化学习中传统上使用的小模型可能掩盖了更大批大小的潜在好处。

To test this hypothesis, we study the effect of increasing the batch size for networks of varying depths. As shown in Figure 7, scaling the batch size becomes effective as network depth grows. This finding offers evidence that by scaling network capacity, we may simultaneously unlock the benefits of larger batch size, potentially making it an important component in the broader pursuit of scaling self-supervised RL.

为了检验这一假设，我们研究了在不同深度的网络上增加批大小的效果。如图 7 所示，随着网络深度增加，扩大批大小变得有效。这一发现提供了证据：扩展网络容量的同时，我们或许也能释放更大批大小的好处，这可能使批大小成为扩展自监督强化学习这一更大目标中的重要组成部分。

Training Contrastive RL with 1000+ Layers

We next study whether further increasing depth beyond 64 layers further improves performance. We use the Humanoid maze tasks as these are both the most challenging environments in the benchmark and also seem to benefit from the deepest scaling. The results, shown in Figure 12, indicate that performance continues to substantially improve as network depth reaches 256 and 1024 layers in the Humanoid U-Maze environment. While we were unable to scale beyond 1024 layers due to computational constraints, we expect to see continued improvements with even greater depths, especially on the most challenging tasks.

用 1000 层以上的网络训练对比强化学习

接下来我们研究，把深度增加到 64 层以上能否进一步提升性能。我们使用 Humanoid 迷宫任务，因为它们既是基准中最具挑战性的环境，也似乎最能从最深的扩展中获益。图 12 的结果表明，在 Humanoid U-Maze 环境中，网络深度达到 256 层和 1024 层时，性能继续大幅提升。受计算资源所限，我们没能扩展到 1024 层以上，但我们预计在更深的网络上性能还会继续提升，尤其是在最具挑战性的任务上。

Figure 8: We disentangle the effects of exploration and expressivity on depth scaling by training three networks in parallel: a “collector,” plus one deep and one shallow learner that train only from the collector’s shared replay buffer. In all three environments, when using a deep collector (i.e. good data coverage), the deep learner outperforms the shallow learner, indicating that expressivity is crucial when controlling for good exploration. With a shallow collector (poor exploration), even the deep learner cannot overcome the limitations of insufficient data coverage. As such, the benefits of depth scaling arise from a combination of improved exploration and increased expressivity working jointly.

图 8：我们并行训练三个网络，以区分探索和表达能力在深度扩展中各自的作用：一个“收集者”（collector），加上一个深层学习者和一个浅层学习者，两个学习者只用收集者的共享回放缓冲区训练。在全部三个环境中，使用深层收集者（即数据覆盖良好）时，深层学习者都优于浅层学习者，说明在控制了探索质量之后，表达能力至关重要。使用浅层收集者（探索不足）时，即使深层学习者也无法克服数据覆盖不足的限制。因此，深度扩展的好处来自探索改善和表达能力增强两者的共同作用。

4.5 Why Scaling Happens

4.5 扩展为什么有效

Depth Enhances Contrastive Representations

Figure 9: Deeper Q-functions are qualitatively different. In the U4-Maze, the start and goal positions are indicated by the $\circledcirc$ and G symbols respectively, and the visualized Q values are computed via the $L_{2}$ distance in the learned representation space, i.e., $Q(s,a,g)=\|\phi(s,a)-\psi(g)\|_{2}$. The shallow depth 4 network (left) naively relies on Euclidean proximity, showing high Q values near the start despite a maze wall. In contrast, the depth 64 network (right) clusters high Q values at the goal, gradually tapering along the interior.

深度增强了对比表示

图 9：更深的 Q 函数在质上不同。在 U4-Maze 中，起点和目标位置分别用 $\circledcirc$ 和 G 符号标出，可视化的 Q 值通过学到的表示空间中的 $L_{2}$ 距离计算，即 $Q(s,a,g)=\|\phi(s,a)-\psi(g)\|_{2}$。深度为 4 的浅层网络*（左）简单地依赖欧氏距离上的接近程度，尽管隔着一堵迷宫墙，起点附近的 Q 值仍然很高。相比之下，深度为 64 的网络（右）*把高 Q 值集中在目标处，并沿迷宫内侧逐渐减弱。

The long-horizon setting has been a long-standing challenge in RL particularly in unsupervised goal-conditioned settings where there is no auxiliary reward feedback (Gupta et al., 2019). The family of U-Maze environments requires a global understanding of the maze layout for effective navigation. We consider a variant of the Ant U-Maze environment, the U4-maze, in which the agent must initially move in the direction opposite the goal to loop around and ultimately reach it. As shown in Figure 9, we observe a qualitative difference in the behavior of the shallow network (depth 4) compared to the deep network (depth 64). The visualized Q-values computed from the critic encoder representations reveal that the depth 4 network seemingly relies on Euclidean distance to the goal as a proxy for the Q value, even when a wall obstructs the direct path. In contrast, the depth 64 critic network learns richer representations, enabling it to effectively capture the topology of the maze as visualized by the trail of high Q values along the inner edge. These findings suggest that increasing network depth leads to richer learned representations, enabling deeper networks to better capture environment topology and achieve more comprehensive state-space coverage in a self-supervised manner.

长时程设定一直是强化学习中的老难题，在没有辅助奖励反馈的无监督目标条件设定中尤其如此（Gupta et al., 2019）。U-Maze 系列环境要求对迷宫布局有全局理解才能有效导航。我们考虑 Ant U-Maze 环境的一个变体 U4-maze，智能体必须先朝与目标相反的方向移动，绕一圈才能最终到达目标。如图 9 所示，我们观察到浅层网络（深度 4）与深层网络（深度 64）的行为在质上不同。由评论家编码器表示计算出的 Q 值可视化显示，即使有墙挡住直线路径，深度为 4 的网络似乎仍依赖到目标的欧氏距离作为 Q 值的替代。相比之下，深度为 64 的评论家网络学到了更丰富的表示，能有效捕捉迷宫的拓扑结构，可视化中沿内侧边缘的一串高 Q 值就体现了这一点。这些发现表明，增加网络深度会带来更丰富的学习表示，使更深的网络能以自监督方式更好地捕捉环境拓扑，并更全面地覆盖状态空间。

Depth Enhances Exploration and Expressivity in a Synergized Way

Our earlier results suggested that deeper networks achieve greater state-action coverage. To better understand why scaling works, we sought to determine to whether improved data alone explains the benefits of scaling, or whether it acts in conjunction with other factors. Thus, we designed an experiment in Figure 8 in which we train three networks in parallel: one network, the “collector," interacts with the environment and writes all experience to a shared replay buffer. Alongside it, two additional "learners", one deep and one shallow, train concurrently. Crucially, these two learners never collect their own data; they train only from the collector’s buffer. This design holds the data distribution constant while varying the model’s capacity, so any performance gap between the deep and shallow learners must come from expressivity rather than exploration. When the collector is deep (e.g., depth 32), across all three environments the deep learner substantially outperforms the shallow one across all three environments, indicating that the expressivity of the deep networks is critical. On the other hand, we repeat the experiment with shallow collectors (e.g., depth 4), which explores less effectively and therefore populates the buffer with low-coverage experience. Here, both the deep and shallow learners struggle and achieve similarly poor performance, which indicates that the deep network’s additional capacity does not overcome the limitations of insufficient data coverage. As such, scaling depth enhances exploration and expressivity in a synergized way: stronger learning capacity drives more extensive exploration, and strong data coverage is essential to fully realize the power of stronger learning capacity. Both aspects jointly contribute to improved performance.

深度协同增强探索与表达能力

前面的结果表明，更深的网络能达到更大的状态—动作覆盖范围。为了更好地理解扩展为什么有效，我们想确定：单靠数据改善能否解释扩展的好处，还是数据改善与其他因素共同起作用。为此我们设计了图 8 的实验，并行训练三个网络：一个网络是“收集者”，与环境交互，并把全部经验写入共享回放缓冲区。与此同时，另外两个“学习者”（一深一浅）同步训练。关键在于，这两个学习者从不自己收集数据，只用收集者的缓冲区训练。这一设计在改变模型容量的同时保持数据分布不变，因此深层与浅层学习者之间的任何性能差距都只能来自表达能力，而不是探索。当收集者是深层网络时（例如深度 32），在全部三个环境中，深层学习者都大幅优于浅层学习者，说明深层网络的表达能力至关重要。另一方面，我们用浅层收集者（例如深度 4）重复了这一实验，它探索效果较差，缓冲区里填的是覆盖范围小的经验。这时，深层和浅层学习者都举步维艰，表现同样很差，说明深层网络额外的容量无法克服数据覆盖不足的限制。因此，扩展深度以协同的方式增强了探索和表达能力：更强的学习能力推动更广泛的探索，而良好的数据覆盖是充分发挥更强学习能力的必要条件。两方面共同促成了性能提升。

Figure 10: We visualize state-action embeddings from shallow (depth 4) and deep (depth 64) networks along a successful trajectory in the Humanoid task. Near the goal, embeddings from the deep network expand across a curved surface, while those from the shallow network form a tight cluster. This suggests that deeper networks may devote greater representational capacity to regions of the state space that are more frequently visited and play a more critical role in successful task completion.

图 10：我们沿 Humanoid 任务中的一条成功轨迹，可视化浅层（深度 4）和深层（深度 64）网络的状态—动作嵌入。在目标附近，深层网络的嵌入在一个曲面上铺展开，而浅层网络的嵌入聚成一团。这表明更深的网络可能把更多的表示容量分配给状态空间中访问更频繁、对成功完成任务更关键的区域。

Deep Networks Learn to Allocate Greater Representational Capacity to States Near the Goal

In Figure 10 we take a successful trajectory in the Humanoid environment and visualize the embeddings of state-action encoder along this trajectory for both deep vs. shallow networks. While the shallow network (Depth 4) tends to cluster near-goal states tightly together, the deep network produces more "spread out" representations. This distinction is important: in a self-supervised setting, we want our representations to separate states that matter—particularly future or goal-relevant states—from random ones. As such, we want to allocate more representational capacity to such critical regions. This suggests that deep networks may learn to allocate representational capacity more effectively to state regions that matter most for the downstream task.

深层网络学会把更多表示容量分配给目标附近的状态

在图 10 中，我们取 Humanoid 环境中的一条成功轨迹，分别可视化深层和浅层网络沿这条轨迹的状态—动作编码器嵌入。浅层网络（深度 4）倾向于把接近目标的状态紧紧聚在一起，深层网络则产生更“分散”的表示。这一区别很重要：在自监督设定中，我们希望表示能把重要的状态（尤其是未来状态或与目标相关的状态）与随机状态区分开，因此希望把更多的表示容量分配给这些关键区域。这表明深层网络可能学会了更有效地把表示容量分配给对下游任务最重要的状态区域。

Deeper Networks Enable Partial Experience Stitching

Figure 11: Deeper networks exhibit improved generalization. (Top left) We modify the training setup of the Ant U-Maze environment such that start-goal pairs are separated by $\leq 3$ units. This design guarantees that no evaluation pairs (Top right) were encountered during training, testing the ability for combinatorial generalization via stitching. (Bottom) Generalization ability improves as network depth grows from 4 to 16 to 64 layers.

更深的网络能实现部分经验拼接

图 11：更深的网络泛化能力更强。*（左上）我们修改 Ant U-Maze 环境的训练设置，使起点—目标对之间的距离 $\leq 3$ 个单位。这一设计保证评估用的起点—目标对（右上）在训练中从未出现过，以测试通过拼接实现组合泛化的能力。（下）*随着网络深度从 4 层增加到 16 层再到 64 层，泛化能力不断提高。

Another key challenge in reinforcement learning is learning policies that can generalize to tasks unseen during training. To evaluate this setting, we designed a modified version of the Ant U-Maze environment. As shown in Figure 11 (top right), the original JaxGCRL benchmark assesses the agent’s performance on the three farthest goal positions located on the opposite side of the wall. However, instead of training on all possible subgoals (a superset of the evaluation state-goal pairs), we modified the setup to train on start-goal pairs that are at most 3 units apart, ensuring that none of the evaluation pairs ever appear in the training set. Figure 11 demonstrates that depth 4 networks show limited generalization, solving only the easiest goal (4 units away from the start). Depth 16 networks achieve moderate success, while depth 64 networks excel, sometimes solving the most challenging goal position. These results suggest that the increasing network depth results in some degree of stitching, combining $\leq$3-unit pairs to navigate the 6-unit span of the U-Maze.

强化学习中的另一个关键挑战，是学到能泛化到训练中未见任务的策略。为了评估这一设定，我们设计了 Ant U-Maze 环境的修改版本。如图 11（右上）所示，原始 JaxGCRL 基准在位于墙另一侧的三个最远目标位置上评估智能体的表现。但我们不在所有可能的子目标（评估用状态—目标对的超集）上训练，而是改为只在相距不超过 3 个单位的起点—目标对上训练，确保评估用的配对从未出现在训练集中。图 11 显示，深度为 4 的网络泛化能力有限，只能完成最简单的目标（距起点 4 个单位）。深度为 16 的网络取得中等程度的成功，而深度为 64 的网络表现出色，有时能完成最具挑战性的目标位置。这些结果表明，增加网络深度会带来一定程度的拼接（stitching），把 $\leq$3 个单位的配对组合起来，走完 U-Maze 6 个单位的跨度。

The (CRL) Algorithm is Key

In Appendix A, we show that scaled CRL outperforms other baseline goal-conditioned algorithms and advance the SOTA for goal-conditioned RL. We observe that for temporal difference methods (SAC, SAC+HER, TD3+HER), the performance saturates for networks of depth 4, and there is either zero or negative performance gains from deeper networks. This is in line with previous research showing that these methods benefit mainly from width (Lee et al., 2024; Nauman et al., 2024b). These results suggest that the self-supervised CRL algorithm is critical.

（CRL）算法是关键

在附录 A 中，我们表明扩展后的 CRL 优于其他目标条件基线算法，并推进了目标条件强化学习的最高水平。我们观察到，对于时序差分方法（SAC、SAC+HER、TD3+HER），性能在深度为 4 的网络上就已饱和，更深的网络带来的收益为零甚至为负。这与以往研究一致，即这些方法主要从宽度中获益（Lee et al., 2024; Nauman et al., 2024b）。这些结果表明，自监督的 CRL 算法至关重要。

We also experiment with scaling more self-supervised algorithms, namely Goal-Conditioned Behavioral Cloning (GCBC) and Goal-Conditioned Supervised Learning (GCSL). While these methods yield zero success in certain environments, they show some utility in arm manipulation tasks. Interestingly, even a very simple self-supervised algorithm like GCBC benefits from increased depth. This points to a promising direction for future work of further investigating other self-supervised methods to uncover potentially different or complementary recipes for scaling self-supervised RL.

我们还尝试扩展更多自监督算法，即目标条件行为克隆（Goal-Conditioned Behavioral Cloning, GCBC）和目标条件监督学习（Goal-Conditioned Supervised Learning, GCSL）。这些方法在某些环境中成功率为零，但在机械臂操作任务上有一定作用。有意思的是，即使像 GCBC 这样非常简单的自监督算法，也能从增加深度中获益。这为未来工作指出了一个有前景的方向：进一步研究其他自监督方法，发现扩展自监督强化学习的可能不同或互补的做法。

Finally, recent work has augmented goal-conditioned RL with quasimetric architectures, leveraging the fact that temporal distances satisfy a triangle inequality–based invariance. In Appendix A, we also investigate whether the depth scaling effect persists when applied to these quasimetric networks.

最后，近期工作利用时间距离满足基于三角不等式的不变性这一事实，用拟度量（quasimetric）架构增强目标条件强化学习。在附录 A 中，我们也研究了深度扩展效应在用于这些拟度量网络时是否仍然存在。

Figure 12: Testing the limits of scale. We extend the results from Figure 1 by scaling networks even further on the challenging Humanoid maze environments. We observe continued performance improvements with network depths of 256 and 1024 layers on Humanoid U-Maze. Note that for the 1024-layer networks, we observed the actor loss exploding at the onset of training, so we maintained the actor depth at 512 while using 1024-layer networks only for the two critic encoders.

图 12：测试规模的极限。我们在具有挑战性的 Humanoid 迷宫环境中进一步扩展网络，延伸图 1 的结果。在 Humanoid U-Maze 上，网络深度达到 256 层和 1024 层时，性能持续提升。注意，对于 1024 层的网络，我们观察到行动者损失在训练开始时就爆炸，因此我们把行动者深度保持在 512 层，只对两个评论家编码器使用 1024 层网络。

4.6 Does Depth Scaling Improve Offline Contrastive RL?

4.6 深度扩展能否改进离线对比强化学习？

In preliminary experiments, we evaluated depth scaling in the offline goal-conditioned setting using OGBench  (Park et al., 2024). We found little evidence that increasing the network depth of CRL improves performance in this offline setting. To further investigate this, we conducted ablations: (1) scaling critic depth while holding the actor at 4 or 8 layers, and (2) applying cold initialization to the final layers of the critic encoders (Zheng et al., 2024). In all cases, baseline depth 4 networks often had the highest success. A key direction for future work is to see if our method can be adapted to enable scaling in the offline setting.

在初步实验中，我们用 OGBench（Park et al., 2024）评估了离线目标条件设定下的深度扩展。我们几乎没有发现证据表明，在这一离线设定中增加 CRL 的网络深度能提升性能。为了进一步研究，我们做了消融：(1) 扩展评论家深度，同时把行动者固定在 4 层或 8 层；(2) 对评论家编码器的最后几层使用冷初始化（Zheng et al., 2024）。在所有情况下，深度为 4 的基线网络往往成功率最高。未来工作的一个重要方向，是看我们的方法能否经过调整，在离线设定中实现扩展。

5 Conclusion

5 结论

Arguably, much of the success of vision and language models today is due to the emergent capabilities they exhibit from scale (Srivastava et al., 2023), leading to many systems reducing the RL problem to a vision or language problem. A critical question for large AI models is: where does the data come from? Unlike supervised learning paradigms, RL methods inherently address this by jointly optimizing both the model and the data collection process through exploration. Ultimately, determining effective ways of building RL systems that demonstrate emergent capabilities may be important for transforming the field into one that trains its own large models. We believe that our work is a step towards these systems. By integrating key components for scaling up RL into a single approach, we show that model performance consistently improves as scale increases in complex tasks. In addition, deep models exhibit qualitatively better behaviors which might be interpreted as implicitly acquired skills necessary to reach the goal.

可以说，今天视觉和语言模型的成功很大程度上源于它们随规模涌现的能力（Srivastava et al., 2023），这使许多系统把强化学习问题归结为视觉或语言问题。大型 AI 模型面临的一个关键问题是：数据从哪里来？与监督学习范式不同，强化学习方法通过探索同时优化模型和数据收集过程，本身就回答了这个问题。归根到底，找到构建能展现涌现能力的强化学习系统的有效方法，对于把这一领域变成一个训练自己的大模型的领域可能很重要。我们相信本文是朝这类系统迈出的一步。通过把扩展强化学习的关键组件整合到同一种方法中，我们表明在复杂任务上，模型性能随规模增大而持续提升。此外，深层模型表现出质上更好的行为，可以理解为隐式习得了到达目标所需的技能。

Limitations.

The primary limitations of our results are that scaling network depth comes at the cost of compute. An important direction for future work is to study how distributed training might be used to leverage even more compute, and how techniques such as pruning and distillation might be used to decrease the computational costs.

局限。

我们结果的主要局限在于，扩展网络深度要付出计算代价。未来工作的一个重要方向，是研究如何利用分布式训练来调用更多计算资源，以及如何用剪枝和蒸馏等技术降低计算成本。

Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

影响声明

本文的目标是推动机器学习领域的发展。我们的工作可能产生许多社会影响，但我们认为没有哪一项需要在此特别强调。

Acknowledgments.

We gratefully acknowledge Nathaniel Chen, Galen Collier, and the full staff of Princeton Research Computing for their invaluable assistance. We also thank Colin Lu for his discussions and contributions to this work. This research was also partially supported by the National Science Centre, Poland (grant no. 2023/51/D/ST6/01609); the Princeton Laboratory for Artificial Intelligence under Award 2025-97; and the Warsaw University of Technology through the Excellence Initiative: Research University (IDUB) program. Finally, we would also like to thank Jens Tuyls and Harshit Sikchi for providing helpful commends and feedback on the manuscript.

致谢。

衷心感谢 Nathaniel Chen、Galen Collier 以及普林斯顿研究计算部门全体员工的宝贵帮助。也感谢 Colin Lu 对本工作的讨论和贡献。本研究还得到了波兰国家科学中心（资助号 2023/51/D/ST6/01609）、普林斯顿人工智能实验室（Award 2025-97）以及华沙理工大学“卓越计划：研究型大学”（IDUB）项目的部分资助。最后，感谢 Jens Tuyls 和 Harshit Sikchi 对稿件提出的有益意见和反馈。

References

参考文献

Ahn et al. (2022) M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. C. Julian, D. Kalashnikov, Y. Kuang, K. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, and M. Yan Do as i can, not as i say: grounding language in robotic affordances . Conference on Robot Learning . Cited by: §2 .

Andrychowicz et al. (2017) M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba Hindsight Experience Replay . In Neural Information Processing Systems , Vol. 30 . Cited by: §2 , §3 .

Ba et al. (2016) J. L. Ba, J. R. Kiros, and G. E. Hinton Layer normalization . arXiv preprint arXiv: 1607.06450 . Cited by: §1 , §4.1 .

Blier et al. (2021) L. Blier, C. Tallec, and Y. Ollivier Learning Successor States and Goal-Dependent Values: a Mathematical Viewpoint . arXiv . External Links: 2101.07123 Cited by: §3 .

Bortkiewicz et al. (2024) M. Bortkiewicz, W. Pałucki, V. Myers, T. Dziarmaga, T. Arczewski, Ł. Kuciński, and B. Eysenbach Accelerating goal-conditioned rl algorithms and research . arXiv preprint arXiv:2408.11052 . Cited by: Figure 19 , §B.2 , §1 , §3 , §4.1 .

Caron et al. (2021) M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin Emerging properties in self-supervised vision transformers . arXiv preprint arXiv: 2104.14294 . Cited by: §1 .

Chang et al. (2018) B. Chang, L. Meng, E. Haber, F. Tung, and D. Begert Multi-level residual networks from dynamical systems view . In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings , External Links: Link Cited by: §A.7 .

Chen et al. (2022) C. Chen, J. Zhang, Y. Xu, L. Chen, J. Duan, Y. Chen, S. D. Tran, B. Zeng, and T. Chilimbi Why do we need large batchsizes in contrastive learning? a gradient-bias perspective . In Advances in Neural Information Processing Systems , A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.) , External Links: Link Cited by: §4.4 , §4.4 .

Chen et al. (2020) T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton Big self-supervised models are strong semi-supervised learners . Advances in neural information processing systems 33 , pp. 22243–22255 . Cited by: §1 .

Dehghani et al. (2023) M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. Steiner, M. Caron, R. Geirhos, I. M. Alabdulmohsin, R. Jenatton, L. Beyer, M. Tschannen, A. Arnab, X. Wang, C. Riquelme, M. Minderer, J. Puigcerver, U. Evci, M. Kumar, S. van Steenkiste, G. F. Elsayed, A. Mahendran, F. Yu, A. Oliver, F. Huot, J. Bastings, M. Collier, A. Gritsenko, V. Birodkar, C. Vasconcelos, Y. Tay, T. Mensink, A. Kolesnikov, F. Paveti’c, D. Tran, T. Kipf, M. Luvci’c, X. Zhai, D. Keysers, J. Harmsen, and N. Houlsby Scaling vision transformers to 22 billion parameters . International Conference on Machine Learning . External Links: Document Cited by: §1 , §2 .

Driess et al. (2023) D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. R. Florence PaLM-e: an embodied multimodal language model . International Conference on Machine Learning . External Links: Document Cited by: §2 .

Dubey et al. (2024) A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models . arXiv preprint arXiv:2407.21783 . Cited by: §1 .

Espeholt et al. (2018) L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: scalable distributed deep-rl with importance weighted actor-learner architectures . In International conference on machine learning , pp. 1407–1416 . Cited by: §2 .

Esser et al. (2024) P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis . In Forty-first International Conference on Machine Learning , Cited by: §1 .

Eysenbach et al. (2021) B. Eysenbach, R. Salakhutdinov, and S. Levine C-Learning: Learning to Achieve Goals via Recursive Classification . In International Conference on Learning Representations , Cited by: §1 , §3 .

Eysenbach et al. (2022) B. Eysenbach, T. Zhang, S. Levine, and R. R. Salakhutdinov Contrastive learning as goal-conditioned reinforcement learning . Advances in Neural Information Processing Systems 35 , pp. 35603–35620 . Cited by: §1 , §2 , §3 .

Farebrother et al. (2024) J. Farebrother, J. Orbay, Q. Vuong, A. A. Taïga, Y. Chebotar, T. Xiao, A. Irpan, S. Levine, P. S. Castro, A. Faust, A. Kumar, and R. Agarwal Stop Regressing: Training Value Functions via Classification for Scalable Deep RL . arXiv . Cited by: §1 , §2 , §3 .

Freeman et al. (2021) C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem Brax – a Differentiable Physics Engine for Large Scale Rigid Body Simulation . In NeurIPS Datasets and Benchmarks , External Links: Link Cited by: §4.1 .

Goyal et al. (2019) P. Goyal, D. Mahajan, A. Gupta, and I. Misra Scaling and benchmarking self-supervised visual representation learning . In Proceedings of the ieee/cvf International Conference on computer vision , pp. 6391–6400 . Cited by: §1 .

Gupta et al. (2019) A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman Relay policy learning: solving long-horizon tasks via imitation and reinforcement learning . Conference on Robot Learning . Cited by: §4.5 .

He et al. (2015) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition . Computer Vision and Pattern Recognition . External Links: Document Cited by: §1 , §3 , §4.1 .

He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Identity mappings in deep residual networks . pp. 630–645 . External Links: Document , Link Cited by: §3 .

Henderson et al. (2018) P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger Deep reinforcement learning that matters . In Proceedings of the AAAI conference on artificial intelligence , Vol. 32 . Cited by: §2 .

Huang et al. (2022) S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and J. G.M. Araújo CleanRL: high-quality single-file implementations of deep reinforcement learning algorithms . Journal of Machine Learning Research 23 ( 274 ), pp. 1–18 . External Links: Link Cited by: §1 .

Kaelbling (1993) L. P. Kaelbling Learning to achieve goals . In IJCAI , Vol. 2 , pp. 1094–8 . Cited by: §3 .

Kumar et al. (2023) A. Kumar, R. Agarwal, X. Geng, G. Tucker, and S. Levine Offline q-learning on diverse multi-task data both scales and generalizes . In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 , External Links: Link Cited by: §2 .

LeCun (2016) Y. LeCun Predictive learning . Note: Invited talk at the 30th Conference on Neural Information Processing Systems (NIPS)Barcelona, Spain External Links: Link Cited by: §1 , §2 , §4.4 .

Lee et al. (2024) H. Lee, D. Hwang, D. Kim, H. Kim, J. J. Tai, K. Subramanian, P. R. Wurman, J. Choo, P. Stone, and T. Seno SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning . Cited by: §A.2 , 2nd item , §1 , §2 , §3 , Figure 4 , Figure 4 , §4.2 , §4.4 , §4.4 , §4.5 .

Lee et al. (2022) K. Lee, O. Nachum, M. Yang, L. Lee, D. Freeman, W. Xu, S. Guadarrama, I. Fischer, E. Jang, H. Michalewski, and I. Mordatch Multi-Game Decision Transformers . arXiv . Cited by: §1 .

Liu et al. (2023) B. Liu, Y. Feng, Q. Liu, and P. Stone Metric Residual Networks for Sample Efficient Goal-Conditioned Reinforcement Learning . Cited by: §A.4 .

Liu et al. (2024) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning . Advances in neural information processing systems 36 . Cited by: §1 .

Lyle et al. (2022) C. Lyle, M. Rowland, and W. Dabney Understanding and preventing capacity loss in reinforcement learning . arXiv preprint arXiv:2204.09560 . Cited by: §2 .

Lyle et al. (2024) C. Lyle, Z. Zheng, K. Khetarpal, H. van Hasselt, R. Pascanu, J. Martens, and W. Dabney Disentangling the causes of plasticity loss in neural networks . arXiv preprint arXiv:2402.18762 . Cited by: §2 .

Makoviychuk et al. (2021) V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: high performance gpu-based physics simulation for robot learning . arXiv preprint arXiv:2108.10470 . Cited by: §1 .

Myers et al. (2024) V. Myers, C. Zheng, A. Dragan, S. Levine, and B. Eysenbach Learning temporal distances: contrastive successor features can provide a metric structure for decision-making . International Conference on Machine Learning . External Links: Document Cited by: §A.4 , §3 .

Nauman et al. (2024a) M. Nauman, M. Bortkiewicz, P. Milos, T. Trzcinski, M. Ostaszewski, and M. Cygan Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 , External Links: Link Cited by: §2 .

Nauman et al. (2024b) M. Nauman, M. Ostaszewski, K. Jankowski, P. Miłoś, and M. Cygan Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control . arXiv . Cited by: §A.2 , 2nd item , §1 , §2 , §3 , Figure 4 , Figure 4 , §4.2 , §4.4 , §4.4 , §4.5 .

Neumann and Gros (2022) O. Neumann and C. Gros Scaling laws for a multi-agent reinforcement learning model . arXiv preprint arXiv:2210.00849 . Cited by: §2 .

Obando-Ceron et al. (2023) J. Obando-Ceron, M. G. Bellemare, and P. S. Castro Small batch deep reinforcement learning . Neural Information Processing Systems . Note: Published at NeurIPS 2023 External Links: Link Cited by: §4.4 .

Obando-Ceron et al. (2024) J. Obando-Ceron, G. Sokar, T. Willi, C. Lyle, J. Farebrother, J. N. Foerster, G. Dziugaite, D. Precup, and P. S. Castro Mixtures of experts unlock parameter scaling for deep rl . International Conference on Machine Learning . External Links: Document Cited by: §2 , §2 .

Ota et al. (2021) K. Ota, D. K. Jha, and A. Kanezaki Training larger networks for deep reinforcement learning . arXiv preprint arXiv:2102.07920 . Cited by: §2 , §2 .

Park et al. (2024) S. Park, K. Frans, B. Eysenbach, and S. Levine OGBench: benchmarking offline goal-conditioned rl . arXiv preprint arXiv: 2410.20092 . Cited by: Figure 18 , §4.6 .

Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision . International Conference on Machine Learning . Cited by: §1 , §1 .

Radford (2018) A. Radford Improving language understanding by generative pre-training . Cited by: §1 .

Raffin et al. (2021) A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann Stable-baselines3: reliable reinforcement learning implementations . Journal of Machine Learning Research 22 ( 268 ), pp. 1–8 . External Links: Link Cited by: §1 .

Ramachandran et al. (2018) P. Ramachandran, B. Zoph, and Q. V. Le Searching for activation functions . In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings , External Links: Link Cited by: §1 , §4.1 .

Rudin et al. (2022) N. Rudin, D. Hoeller, P. Reist, and M. Hutter Learning to walk in minutes using massively parallel deep reinforcement learning . In Conference on Robot Learning , pp. 91–100 . Cited by: §1 .

Rutherford et al. (2023) A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. Ingvarsson, T. Willi, A. Khan, C. S. de Witt, A. Souly, et al. JaxMARL: multi-agent rl environments and algorithms in jax . arXiv preprint arXiv:2311.10090 . Cited by: §1 .

Schwarzer et al. (2023) M. Schwarzer, J. S. Obando-Ceron, A. C. Courville, M. G. Bellemare, R. Agarwal, and P. S. Castro Bigger, better, faster: human-level atari with human-level efficiency . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.) , Proceedings of Machine Learning Research , Vol. 202 , pp. 30365–30380 . External Links: Link Cited by: §2 .

Sohn (2016) K. Sohn Improved Deep Metric Learning With Multi-Class N-Pair Loss Objective . In Neural Information Processing Systems , Vol. 29 . Cited by: §3 .

Srivastava et al. (2023) A. Srivastava, A. Rastogi, A. Rao, et al. Beyond the imitation game: quantifying and extrapolating the capabilities of language models . Trans. Mach. Learn. Res. . Cited by: §1 , §1 , §2 , §5 .

Team et al. (2023) A. A. Team, J. Bauer, K. Baumli, S. Baveja, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg, M. Chang, N. Clay, A. Collister, et al. Human-timescale adaptation in an open-ended task space . arXiv preprint arXiv:2301.07608 . Cited by: §2 .

Todorov et al. (2012) E. Todorov, T. Erez, and Y. Tassa Mujoco: a Physics Engine for Model-Based Control . In IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 5026–5033 . Cited by: §4.1 .

Torgo and Gama (1996) L. Torgo and J. Gama Regression by classification . In Advances in Artificial Intelligence: 13th Brazilian Symposium on Artificial Intelligence, SBIA’96 Curitiba, Brazil, October 23–25, 1996 Proceedings 13 , pp. 51–60 . Cited by: §2 .

Tuyls et al. (2024) J. Tuyls, D. Madeka, K. Torkkola, D. Foster, K. Narasimhan, and S. Kakade Scaling Laws for Imitation Learning in Single-Agent Games . arXiv . Cited by: §2 .

Van Hasselt et al. (2018) H. Van Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil Deep reinforcement learning and the deadly triad . arXiv preprint arXiv:1812.02648 . Cited by: §2 .

Vaswani et al. (2017) A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin Attention is all you need . nips . Cited by: §2 .

Veit et al. (2016) A. Veit, M. Wilber, and S. Belongie Residual networks behave like ensembles of relatively shallow networks . arXiv preprint arXiv: 1605.06431 . Cited by: §3 .

Wang et al. (2023a) T. Wang, A. Torralba, P. Isola, and A. Zhang Optimal goal-reaching reinforcement learning via quasimetric learning . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.) , Proceedings of Machine Learning Research , Vol. 202 , pp. 36411–36430 . External Links: Link Cited by: §A.4 .

Wang et al. (2023b) T. Wang, A. Torralba, P. Isola, and A. Zhang Optimal goal-reaching reinforcement learning via quasimetric learning . External Links: 2304.01203 , Link Cited by: §A.3 .

Wei et al. (2022) J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus Emergent abilities of large language models . Trans. Mach. Learn. Res. . External Links: Document Cited by: §2 .

Zhai et al. (2021) X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer Scaling vision transformers . Computer Vision and Pattern Recognition . External Links: Document Cited by: §1 , §2 .

Zhang et al. (2024) H. Zhang, D. Morwani, N. Vyas, J. Wu, D. Zou, U. Ghai, D. Foster, and S. Kakade How does critical batch size scale in pre-training? . arXiv preprint arXiv: 2410.21676 . Cited by: §4.4 , §4.4 .

Zheng et al. (2024) C. Zheng, B. Eysenbach, H. Walke, P. Yin, K. Fang, R. Salakhutdinov, and S. Levine Stabilizing Contrastive RL: Techniques for Offline Goal Reaching . In International Conference on Learning Representations , Cited by: §3 , §4.2 , §4.6 .

Zheng et al. (2023) C. Zheng, R. Salakhutdinov, and B. Eysenbach Contrastive Difference Predictive Coding . In Twelfth International Conference on Learning Representations , Cited by: §3 .

Zong et al. (2024) Y. Zong, O. M. Aodha, and T. Hospedales Self-supervised multimodal learning: a survey . External Links: 2304.01008 , Link Cited by: §1 .

Appendix A Additional Experiments

附录 A 补充实验

A.1 Scaled CRL Outperforms All Other Baselines on 8 out of 10 Environments

A.1 扩展后的 CRL 在 10 个环境中的 8 个上优于所有其他基线

Figure 12: Scaled CRL (Ours) outperforms baselines CRL (original), SAC, SAC+HER, TD3+HER, GCSL, and GCBC in 8 out 10 environments.

图 12：扩展后的 CRL（本文方法）在 10 个环境中的 8 个上优于基线 CRL（原始版本）、SAC、SAC+HER、TD3+HER、GCSL 和 GCBC。

In Figure 1, we demonstrated that increasing the depth of the CRL algorithm leads to significant performance improvements over the original CRL (see also Table 1). Here, we show that these gains translate to state-of-the-art results in online goal-conditioned RL, with Scaled CRL outperforming both standard TD-based methods such as SAC, SAC+HER, and TD3+HER, as well as self-supervised imitation-based approaches like GCBC and GCSL.

在图 1 中，我们展示了增加 CRL 算法的深度能比原始 CRL 带来显著的性能提升（另见表 1）。这里我们表明，这些提升转化为在线目标条件强化学习中的最先进结果：扩展后的 CRL 既优于 SAC、SAC+HER 和 TD3+HER 等标准的基于 TD 的方法，也优于 GCBC 和 GCSL 等基于模仿的自监督方法。

A.2 The CRL Algorithm is Key: Depth Scaling is Not Effective on Other Baselines

A.2 CRL 算法是关键：深度扩展在其他基线上无效

Figure 13: Depth scaling yields limited gains for SAC, SAC+HER, TD3+HER, GCSL, and GCBC.

图 13：深度扩展对 SAC、SAC+HER、TD3+HER、GCSL 和 GCBC 的收益有限。

Next, we investigate whether increasing network depth in the baseline algorithms yields similar performance improvements as observed in CRL. We find that SAC, SAC+HER, and TD3+HER do not benefit from depths beyond four layers, which is consistent with prior findings (Lee et al., 2024; Nauman et al., 2024b). Additionally, GCSL and GCBC fail to achieve any meaningful performance on the Humanoid and Ant Big Maze tasks. Interestingly, we do observe one exception, as GCBC exhibits improved performance with increased depth in the Arm Push Easy environment.

接下来我们研究，在基线算法中增加网络深度，是否能带来与 CRL 类似的性能提升。我们发现 SAC、SAC+HER 和 TD3+HER 无法从四层以上的深度中获益，这与以往的发现一致（Lee et al., 2024; Nauman et al., 2024b）。此外，GCSL 和 GCBC 在 Humanoid 和 Ant Big Maze 任务上没有取得任何有意义的表现。有意思的是，我们确实观察到一个例外：在 Arm Push Easy 环境中，GCBC 的性能随深度增加而提高。

Task	Dim	D = 4	D = 64	Imprv.
Arm Binpick Hard	17	$38$ $\mathbf{\pm}4$	$219$ $\mathbf{\pm}15$	$5.7\times$
Arm Push Easy		$308$ $\mathbf{\pm}33$	$762$ $\mathbf{\pm}30$	$2.5\times$
Arm Push Hard		$171$ $\mathbf{\pm}11$	$410$ $\mathbf{\pm}13$	$2.4\times$
Ant U4-Maze	29	$11.4$ $\mathbf{\pm}4.1$	$286$ $\mathbf{\pm}36$	$25\times$
Ant U5-Maze		$0.97$ $\mathbf{\pm}0.7$	$61$ $\mathbf{\pm}18$	$63\times$
Ant Big Maze		$61$ $\mathbf{\pm}20$	$441$ $\mathbf{\pm}25$	$7.3\times$
Ant Hardest Maze		$215$ $\mathbf{\pm}8$	$387$ $\mathbf{\pm}21$	$1.8\times$
Humanoid	268	$12.6$ $\mathbf{\pm}1.3$	$649$ $\mathbf{\pm}19$	$52\times$
Humanoid U-Maze		$3.2$ $\mathbf{\pm}1.2$	$159$ $\mathbf{\pm}33$	$50\times$
Humanoid Big Maze		$0.06$ $\mathbf{\pm}0.04$	$59$ $\mathbf{\pm}21$	$1051\times$

Table 1: Increasing network depth (depth $D=4\rightarrow 64$) increases performance on CRL (Figure 1). Scaling depth exhibits the greatest benefits on tasks with the largest observation dimension (Dim).

任务	维度	D = 4	D = 64	提升
Arm Binpick Hard	17	$38$ $\mathbf{\pm}4$	$219$ $\mathbf{\pm}15$	$5.7\times$
Arm Push Easy		$308$ $\mathbf{\pm}33$	$762$ $\mathbf{\pm}30$	$2.5\times$
Arm Push Hard		$171$ $\mathbf{\pm}11$	$410$ $\mathbf{\pm}13$	$2.4\times$
Ant U4-Maze	29	$11.4$ $\mathbf{\pm}4.1$	$286$ $\mathbf{\pm}36$	$25\times$
Ant U5-Maze		$0.97$ $\mathbf{\pm}0.7$	$61$ $\mathbf{\pm}18$	$63\times$
Ant Big Maze		$61$ $\mathbf{\pm}20$	$441$ $\mathbf{\pm}25$	$7.3\times$
Ant Hardest Maze		$215$ $\mathbf{\pm}8$	$387$ $\mathbf{\pm}21$	$1.8\times$
Humanoid	268	$12.6$ $\mathbf{\pm}1.3$	$649$ $\mathbf{\pm}19$	$52\times$
Humanoid U-Maze		$3.2$ $\mathbf{\pm}1.2$	$159$ $\mathbf{\pm}33$	$50\times$
Humanoid Big Maze		$0.06$ $\mathbf{\pm}0.04$	$59$ $\mathbf{\pm}21$	$1051\times$

表 1：增加网络深度（深度 $D=4\rightarrow 64$）提高了 CRL 的性能（图 1）。在观测维度（维度）最大的任务上，扩展深度的收益最大。

A.3 Additional Scaling Experiments: Offline GCBC, BC, and QRL

A.3 补充扩展实验：离线 GCBC、BC 和 QRL

We further investigate several additional scaling experiments. As shown in Figure 14, our approach successfully scales with depth in the offline GCBC setting on the antmaze-medium-stitch task from OGBench. We find that our the combination of layer normalization, residual connections, and Swish activations is critical, suggesting that our architectural choices may be applied to unlock depth scaling in other algorithms and settings. We also attempt to scale depth for behavioral cloning and the QRL (Wang et al., 2023b) algorithm—in both of these cases, however, we observe negative results.

我们进一步做了几项补充扩展实验。如图 14 所示，在 OGBench 的 antmaze-medium-stitch 任务上，我们的方法在离线 GCBC 设定中成功实现了深度扩展。我们发现，层归一化、残差连接和 Swish 激活的组合至关重要，这表明我们的架构选择或许可以用来在其他算法和设定中打开深度扩展的空间。我们也尝试为行为克隆（BC）和 QRL（Wang et al., 2023b）算法扩展深度，但在这两种情况下都得到了负面结果。

Figure 14: Our approach successfully scales depth in offline GCBC on antmaze-medium-stitch (OGBench). In contrast, scaling depth for BC (antmaze-giant-navigate, expert SAC data) and for both online (FetchPush) and offline QRL (pointmaze-giant-stitch, OGBench) yield negative results.

图 14：我们的方法在 OGBench 的 antmaze-medium-stitch 上成功扩展了离线 GCBC 的深度。相比之下，为 BC（antmaze-giant-navigate，专家 SAC 数据）以及在线（FetchPush）和离线 QRL（pointmaze-giant-stitch，OGBench）扩展深度，都得到了负面结果。

A.4 Can Depth Scaling also be Effective for Quasimetric Architectures?

A.4 深度扩展对拟度量架构是否同样有效？

Prior work (Wang et al., 2023a; Liu et al., 2023) has found that temporal distances satisfy an important invariance property, suggesting the use of quasimetric architectures when learning temporal distances. Our next experiment tests whether changing the architecture affects the scaling properties of self-supervised RL. Specifically, we use the CMD-1 algorithm (Myers et al., 2024), which employs a backward NCE loss with MRN representations. The results indicate that scaling benefits are not limited to a single neural network parametrization. However, MRN’s poor performance on the Ant U5-Maze task suggests further innovation is needed for consistent scaling with quasimetric models.

以往工作（Wang et al., 2023a; Liu et al., 2023）发现，时间距离满足一个重要的不变性，因此建议在学习时间距离时使用拟度量架构。我们的下一个实验检验改变架构是否会影响自监督强化学习的扩展特性。具体来说，我们使用 CMD-1 算法（Myers et al., 2024），它采用带 MRN 表示的反向 NCE 损失。结果表明，扩展的收益并不局限于某一种神经网络参数化。不过，MRN 在 Ant U5-Maze 任务上表现不佳，说明要让拟度量模型稳定地扩展，还需要进一步的创新。

Figure 15: Performance of depth scaling on CRL augmented with quasimetric architectures (CMD-1).

图 15：在用拟度量架构增强的 CRL（CMD-1）上进行深度扩展的性能。

A.5 Additional Architectural Ablations: Layer Norm and Swish Activation

A.5 补充架构消融：层归一化与 Swish 激活

We conduct ablation experiments to validate the architectural choices of layer norm and swish activation. Figure 16 shows that removing layer normalization performs significantly worse. Additionally, scaling with ReLU significantly hampers scalability. These results, along with Figure 6 show that all of our architectural components—residual connections, layer norm, and swish activations—are jointly essential to unlocking the full performance of depth scaling.

我们做了消融实验，验证层归一化和 Swish 激活这两项架构选择。图 16 显示，去掉层归一化后效果明显变差。此外，使用 ReLU 进行扩展会严重削弱可扩展性。这些结果连同图 6 表明，我们所有的架构组件（残差连接、层归一化和 Swish 激活）缺一不可，共同释放了深度扩展的全部性能。

Figure 16: (Left) Layer Norm is essential for scaling depth. (Right) Scaling with ReLU activations leads to worse performance compared to Swish activations.

图 16：（左）层归一化对扩展深度至关重要。（右）与 Swish 激活相比，使用 ReLU 激活进行扩展性能更差。

A.6 Can We Integrate Novel Architectural Innovations from the Emerging RL Scaling Literature?

A.6 能否吸收新兴强化学习扩展文献中的架构创新？

Recently, Simba-v2 proposed a new architecture for scalable RL. Its key innovation is the replacement of layer normalization with hyperspherical normalization, which projects network weights onto the unit-norm hypersphere after each gradient update. As shown, the same depth-scaling trends hold when adding hyperspherical normalization to our architecture, and it further improves the sample efficiency of depth scaling. This demonstrates that our method can naturally incorporate new architectural innovations emerging in the RL scaling literature.

最近，Simba-v2 提出了一种可扩展强化学习的新架构。它的关键创新是用超球面归一化取代层归一化，即在每次梯度更新后把网络权重投影到单位范数超球面上。如表所示，在我们的架构中加入超球面归一化后，同样的深度扩展趋势依然成立，而且深度扩展的样本效率进一步提高。这表明我们的方法可以自然地吸收强化学习扩展文献中出现的新架构创新。

Steps to reach $\geq$200 success
Depth	4	16	32
With	–	50	42
Without	–	64	54

Steps to reach $\geq$400 success
Depth	4	16	32
With	–	62	48
Without	–	75	64

Steps to reach $\geq$600 success
Depth	4	16	32
With	–	77	67
Without	–	–	77

Table 2: Integrating hyperspherical normalization in our architecture enhances the sample efficiency of depth scaling.

达到 $\geq$200 成功所需步数
深度	4	16	32
使用	–	50	42
不使用	–	64	54

达到 $\geq$400 成功所需步数
深度	4	16	32
使用	–	62	48
不使用	–	75	64

达到 $\geq$600 成功所需步数
深度	4	16	32
使用	–	77	67
不使用	–	–	77

表 2：在我们的架构中加入超球面归一化，提高了深度扩展的样本效率。

A.7 Residuals Norms in Deep Networks

A.7 深层网络中的残差范数

Prior work has noted decreasing residual activation norms in deeper layers (Chang et al., 2018). We investigate whether this pattern also holds in our setting. For the critic, the trend is generally evident, especially in very deep architectures (e.g., depth 256). The effect is not as pronounced in the actor.

以往工作注意到，越深的层残差激活的范数越小（Chang et al., 2018）。我们考察这一规律在我们的设定中是否同样成立。对于评论家，这一趋势总体上很明显，在非常深的架构（例如深度 256）中尤其如此。在行动者中，这一效应没那么明显。

Figure 17: L2 norms of residual activations in networks with depths of 32, 64, 128, and 256.

图 17：深度为 32、64、128 和 256 的网络中残差激活的 L2 范数。

A.8 Scaling Depth for Offline Goal-conditioned RL

A.8 为离线目标条件强化学习扩展深度

Figure 18: To evaluate the scalability of our method in the offline setting, we scaled model depth on OGBench (Park et al., 2024). In two out of three environments, performance drastically declined as depth scaled from 4 to 64, while a slight improvement was seen on antmaze-medium-stitch-v0. Successfully adapting our method to scale offline GCRL is an important direction for future work.

图 18：为了评估我们的方法在离线设定中的可扩展性，我们在 OGBench（Park et al., 2024）上扩展了模型深度。在三个环境中的两个上，深度从 4 扩展到 64 时性能急剧下降，而在 antmaze-medium-stitch-v0 上略有提升。让我们的方法成功扩展离线 GCRL，是未来工作的一个重要方向。

Appendix B Experimental Details

附录 B 实验细节

B.1 Environment Setup and Hyperparameters

B.1 环境设置与超参数

Figure 19: The scaling results of this paper are demonstrated on the JaxGCRL benchmark, showing that they replicate across a diverse range of locomotion, navigation, and manipulation tasks. These tasks are set in the online goal-conditioned setting where there are no auxiliary rewards or demonstrations. Figure taken from (Bortkiewicz et al., 2024).

图 19：本文的扩展结果在 JaxGCRL 基准上得到验证，表明它们在各种运动、导航和操作任务上都能复现。这些任务处于在线目标条件设定中，没有辅助奖励或示范。图片取自（Bortkiewicz et al., 2024）。

Our experiments use the JaxGCRL suite of GPU-accelerated environments, visualized in Figure 19, and a contrastive RL algorithm with hyperparameters reported in Table 7. In particular, we use 10 environments, namely: ant_big_maze, ant_hardest_maze, arm_binpick_hard, arm_push_easy, arm_push_hard, humanoid, humanoid_big_maze, humanoid_u_maze, ant_u4_maze, ant_u5_maze.

我们的实验使用 JaxGCRL 的 GPU 加速环境套件（见图 19），以及对比强化学习算法，超参数见表 7。具体来说，我们使用 10 个环境，即：ant_big_maze、ant_hardest_maze、arm_binpick_hard、arm_push_easy、arm_push_hard、humanoid、humanoid_big_maze、humanoid_u_maze、ant_u4_maze、ant_u5_maze。

B.2 Python Environment Differences

B.2 Python 环境差异

In all plots presented in the paper, we used MJX 3.2.6 and Brax 0.10.1 to ensure a fair and consistent comparison. During development, we noticed discrepancies in physics behavior between the environment versions we employed (the CleanRL version of JaxGCRL) and the version recommended in a more recent commit of JaxGCRL (Bortkiewicz et al., 2024). Upon examination, the performance differences (shown in Figure 20) stem from a difference in versions in the MJX and Brax packages. Nonetheless, in both sets of MJX and Brax versions, performance scales monotonically with depth.

为确保比较公平一致，本文所有图中都使用 MJX 3.2.6 和 Brax 0.10.1。在开发过程中，我们注意到所用的环境版本（JaxGCRL 的 CleanRL 版本）与 JaxGCRL 较新提交中推荐的版本（Bortkiewicz et al., 2024）在物理行为上存在差异。经检查，性能差异（见图 20）源于 MJX 和 Brax 包的版本不同。不过在两组 MJX 和 Brax 版本下，性能都随深度单调提升。

Figure 20: Scaling behavior for humanoid in two different python environments: MJX=3.2.3, Brax=0.10.5 and MJX=3.2.6, Brax=0.10.1 (ours) version of JaxGCRL. Scaling depth improves the performance significantly for both versions. In the environment we used, training requires fewer environment steps to reach a marginally better performance than in other Python environment.

图 20：JaxGCRL 在两种不同 Python 环境下 humanoid 的扩展表现：MJX=3.2.3、Brax=0.10.5，以及 MJX=3.2.6、Brax=0.10.1（本文所用）。在两种版本下，扩展深度都显著提升了性能。在我们使用的环境中，训练达到略好的性能所需的环境步数比另一种 Python 环境更少。

B.3 Wall-clock Time of Our Approach

B.3 本文方法的实际运行时间

We report the wall-clock time of our approach in Table 3. The table shows results for depths of 4, 8, 16, 32, and 64 across all ten environments, and for the Humanoid U-Maze environment, scaling up to 1024 layers. Overall, wall-clock time increases approximately linearly with depth beyond a certain point.

表 3 报告了本文方法的实际运行时间。表中给出所有十个环境在深度为 4、8、16、32 和 64 时的结果，以及 Humanoid U-Maze 环境扩展到 1024 层的结果。总体而言，超过一定深度后，实际运行时间大致随深度线性增长。

Environment	Depth 4	Depth 8	Depth 16	Depth 32	Depth 64
Humanoid	1.48 $\pm$ 0.00	2.13 $\pm$ 0.01	3.40 $\pm$ 0.01	5.92 $\pm$ 0.01	10.99 $\pm$ 0.01
Ant Big Maze	2.12 $\pm$ 0.00	2.77 $\pm$ 0.00	4.04 $\pm$ 0.01	6.57 $\pm$ 0.02	11.66 $\pm$ 0.03
Ant U4-Maze	1.98 $\pm$ 0.27	2.54 $\pm$ 0.01	3.81 $\pm$ 0.01	6.35 $\pm$ 0.01	11.43 $\pm$ 0.03
Ant U5-Maze	9.46 $\pm$ 1.75	10.99 $\pm$ 0.02	16.09 $\pm$ 0.01	31.49 $\pm$ 0.34	46.40 $\pm$ 0.12
Ant Hardest Maze	5.11 $\pm$ 0.00	6.39 $\pm$ 0.00	8.94 $\pm$ 0.01	13.97 $\pm$ 0.01	23.96 $\pm$ 0.06
Arm Push Easy	9.97 $\pm$ 1.03	11.02 $\pm$ 1.29	12.20 $\pm$ 1.43	14.94 $\pm$ 1.96	19.52 $\pm$ 1.97
Arm Push Hard	9.74 $\pm$ 1.05	10.55 $\pm$ 1.20	11.98 $\pm$ 1.49	14.40 $\pm$ 1.64	18.53 $\pm$ 0.06
Arm Binpick Hard	18.41 $\pm$ 2.16	17.48 $\pm$ 1.88	19.47 $\pm$ 0.05	21.91 $\pm$ 1.93	29.64 $\pm$ 6.10
Humanoid U-Maze	8.72 $\pm$ 0.01	11.29 $\pm$ 0.01	16.36 $\pm$ 0.03	26.48 $\pm$ 0.05	46.74 $\pm$ 0.04
Humanoid Big Maze	12.45 $\pm$ 0.02	15.02 $\pm$ 0.01	20.34 $\pm$ 0.01	30.61 $\pm$ 0.05	50.33 $\pm$ 0.05

Table 3: Wall-clock time (in hours) for Depth 4, 8, 16, 32, and 64 across all 10 environments.

环境	深度 4	深度 8	深度 16	深度 32	深度 64
Humanoid	1.48 $\pm$ 0.00	2.13 $\pm$ 0.01	3.40 $\pm$ 0.01	5.92 $\pm$ 0.01	10.99 $\pm$ 0.01
Ant Big Maze	2.12 $\pm$ 0.00	2.77 $\pm$ 0.00	4.04 $\pm$ 0.01	6.57 $\pm$ 0.02	11.66 $\pm$ 0.03
Ant U4-Maze	1.98 $\pm$ 0.27	2.54 $\pm$ 0.01	3.81 $\pm$ 0.01	6.35 $\pm$ 0.01	11.43 $\pm$ 0.03
Ant U5-Maze	9.46 $\pm$ 1.75	10.99 $\pm$ 0.02	16.09 $\pm$ 0.01	31.49 $\pm$ 0.34	46.40 $\pm$ 0.12
Ant Hardest Maze	5.11 $\pm$ 0.00	6.39 $\pm$ 0.00	8.94 $\pm$ 0.01	13.97 $\pm$ 0.01	23.96 $\pm$ 0.06
Arm Push Easy	9.97 $\pm$ 1.03	11.02 $\pm$ 1.29	12.20 $\pm$ 1.43	14.94 $\pm$ 1.96	19.52 $\pm$ 1.97
Arm Push Hard	9.74 $\pm$ 1.05	10.55 $\pm$ 1.20	11.98 $\pm$ 1.49	14.40 $\pm$ 1.64	18.53 $\pm$ 0.06
Arm Binpick Hard	18.41 $\pm$ 2.16	17.48 $\pm$ 1.88	19.47 $\pm$ 0.05	21.91 $\pm$ 1.93	29.64 $\pm$ 6.10
Humanoid U-Maze	8.72 $\pm$ 0.01	11.29 $\pm$ 0.01	16.36 $\pm$ 0.03	26.48 $\pm$ 0.05	46.74 $\pm$ 0.04
Humanoid Big Maze	12.45 $\pm$ 0.02	15.02 $\pm$ 0.01	20.34 $\pm$ 0.01	30.61 $\pm$ 0.05	50.33 $\pm$ 0.05

表 3：所有 10 个环境在深度为 4、8、16、32 和 64 时的实际运行时间（小时）。

Depth	Time (h)
4	3.23 $\pm$ 0.001
8	4.19 $\pm$ 0.003
16	6.07 $\pm$ 0.003
32	9.83 $\pm$ 0.006
64	17.33 $\pm$ 0.003
128	32.67 $\pm$ 0.124
256	73.83 $\pm$ 2.364
512	120.88 $\pm$ 2.177
1024	134.15 $\pm$ 0.081

Table 4: Total wall-clock time (in hours) for training from Depth 4 up to Depth 1024 in the Humanoid U-Maze environment.

深度	时间（小时）
4	3.23 $\pm$ 0.001
8	4.19 $\pm$ 0.003
16	6.07 $\pm$ 0.003
32	9.83 $\pm$ 0.006
64	17.33 $\pm$ 0.003
128	32.67 $\pm$ 0.124
256	73.83 $\pm$ 2.364
512	120.88 $\pm$ 2.177
1024	134.15 $\pm$ 0.081

表 4：在 Humanoid U-Maze 环境中，深度从 4 到 1024 训练所需的总实际运行时间（小时）。

B.4 Wall-clock Time: Comparison to Baselines

B.4 实际运行时间：与基线比较

Since the baselines use standard sized networks, naturally our scaled approach incurs higher raw wall-clock time per environment step (Table 5). However, a more practical metric is the time required to reach a given performance level. As shown in Table 6, our approach outperforms the strongest baseline, SAC, in 7 of 10 environments while requiring less wall-clock time.

由于基线使用标准大小的网络，我们扩展后的方法每个环境步的原始运行时间自然更长（表 5）。不过，更实用的指标是达到给定性能水平所需的时间。如表 6 所示，在 10 个环境中的 7 个上，我们的方法用更少的实际时间超过了最强基线 SAC。

Environment	Scaled CRL	SAC	SAC+HER	TD3	GCSL	GCBC
Humanoid	11.0 $\pm$ 0.0	0.5 $\pm$ 0.0	0.6 $\pm$ 0.0	0.8 $\pm$ 0.0	0.4 $\pm$ 0.0	0.6 $\pm$ 0.0
Ant Big Maze	11.7 $\pm$ 0.0	1.6 $\pm$ 0.0	1.6 $\pm$ 0.0	1.7 $\pm$ 0.0	1.5 $\pm$ 0.3	1.4 $\pm$ 0.1
Ant U4-Maze	11.4 $\pm$ 0.0	1.2 $\pm$ 0.0	1.3 $\pm$ 0.0	1.3 $\pm$ 0.0	0.7 $\pm$ 0.0	1.1 $\pm$ 0.1
Ant U5-Maze	46.4 $\pm$ 0.1	5.7 $\pm$ 0.0	6.1 $\pm$ 0.0	6.2 $\pm$ 0.0	2.8 $\pm$ 0.1	5.6 $\pm$ 0.5
Ant Hardest Maze	24.0 $\pm$ 0.0	4.3 $\pm$ 0.0	4.5 $\pm$ 0.0	5.0 $\pm$ 0.0	2.1 $\pm$ 0.6	4.4 $\pm$ 0.5
Arm Push Easy	19.5 $\pm$ 0.6	8.3 $\pm$ 0.0	8.5 $\pm$ 0.0	8.4 $\pm$ 0.0	6.4 $\pm$ 0.1	8.3 $\pm$ 0.3
Arm Push Hard	18.5 $\pm$ 0.0	8.5 $\pm$ 0.0	8.6 $\pm$ 0.0	8.3 $\pm$ 0.1	5.2 $\pm$ 0.3	7.4 $\pm$ 0.5
Arm Binpick Hard	29.6 $\pm$ 1.3	20.7 $\pm$ 0.1	20.7 $\pm$ 0.0	18.4 $\pm$ 0.3	8.0 $\pm$ 0.9	16.2 $\pm$ 0.4
Humanoid U-Maze	46.7 $\pm$ 0.0	3.0 $\pm$ 0.0	3.5 $\pm$ 0.0	5.4 $\pm$ 0.0	3.1 $\pm$ 0.1	7.2 $\pm$ 0.8
Humanoid Big Maze	50.3 $\pm$ 0.0	8.6 $\pm$ 0.0	9.3 $\pm$ 0.0	7.5 $\pm$ 1.1	5.1 $\pm$ 0.0	11.4 $\pm$ 1.9

Table 5: Wall-clock training time comparison of our method vs. baselines across all 10 environments.

环境	扩展后的 CRL	SAC	SAC+HER	TD3	GCSL	GCBC
Humanoid	11.0 $\pm$ 0.0	0.5 $\pm$ 0.0	0.6 $\pm$ 0.0	0.8 $\pm$ 0.0	0.4 $\pm$ 0.0	0.6 $\pm$ 0.0
Ant Big Maze	11.7 $\pm$ 0.0	1.6 $\pm$ 0.0	1.6 $\pm$ 0.0	1.7 $\pm$ 0.0	1.5 $\pm$ 0.3	1.4 $\pm$ 0.1
Ant U4-Maze	11.4 $\pm$ 0.0	1.2 $\pm$ 0.0	1.3 $\pm$ 0.0	1.3 $\pm$ 0.0	0.7 $\pm$ 0.0	1.1 $\pm$ 0.1
Ant U5-Maze	46.4 $\pm$ 0.1	5.7 $\pm$ 0.0	6.1 $\pm$ 0.0	6.2 $\pm$ 0.0	2.8 $\pm$ 0.1	5.6 $\pm$ 0.5
Ant Hardest Maze	24.0 $\pm$ 0.0	4.3 $\pm$ 0.0	4.5 $\pm$ 0.0	5.0 $\pm$ 0.0	2.1 $\pm$ 0.6	4.4 $\pm$ 0.5
Arm Push Easy	19.5 $\pm$ 0.6	8.3 $\pm$ 0.0	8.5 $\pm$ 0.0	8.4 $\pm$ 0.0	6.4 $\pm$ 0.1	8.3 $\pm$ 0.3
Arm Push Hard	18.5 $\pm$ 0.0	8.5 $\pm$ 0.0	8.6 $\pm$ 0.0	8.3 $\pm$ 0.1	5.2 $\pm$ 0.3	7.4 $\pm$ 0.5
Arm Binpick Hard	29.6 $\pm$ 1.3	20.7 $\pm$ 0.1	20.7 $\pm$ 0.0	18.4 $\pm$ 0.3	8.0 $\pm$ 0.9	16.2 $\pm$ 0.4
Humanoid U-Maze	46.7 $\pm$ 0.0	3.0 $\pm$ 0.0	3.5 $\pm$ 0.0	5.4 $\pm$ 0.0	3.1 $\pm$ 0.1	7.2 $\pm$ 0.8
Humanoid Big Maze	50.3 $\pm$ 0.0	8.6 $\pm$ 0.0	9.3 $\pm$ 0.0	7.5 $\pm$ 1.1	5.1 $\pm$ 0.0	11.4 $\pm$ 1.9

表 5：本文方法与各基线在所有 10 个环境中的实际训练时间比较。

Environment	SAC	Scaled CRL (Depth 64)
Humanoid	0.46	6.37
Ant Big Maze	1.55	0.00
Ant U4-Maze	1.16	0.00
Ant U5-Maze	5.73	0.00
Ant Hardest Maze	4.33	0.45
Arm Push Easy	8.32	1.91
Arm Push Hard	8.50	6.65
Arm Binpick Hard	20.70	4.43
Humanoid U-Maze	3.04	N/A*
Humanoid Big Maze	8.55	N/A*

Table 6: Wall-clock time (in hours) for our approach to surpass SAC’s final performance. As shown, our approach surpasses SAC performance in less wall-clock time in 7 out of 10 environments. The N/A* entries are because in those environments, scaled CRL doesn’t outperform SAC.

环境	SAC	扩展后的 CRL（深度 64）
Humanoid	0.46	6.37
Ant Big Maze	1.55	0.00
Ant U4-Maze	1.16	0.00
Ant U5-Maze	5.73	0.00
Ant Hardest Maze	4.33	0.45
Arm Push Easy	8.32	1.91
Arm Push Hard	8.50	6.65
Arm Binpick Hard	20.70	4.43
Humanoid U-Maze	3.04	N/A*
Humanoid Big Maze	8.55	N/A*

表 6：本文方法超过 SAC 最终性能所需的实际运行时间（小时）。如表所示，在 10 个环境中的 7 个上，本文方法用更少的实际时间超过了 SAC。标为 N/A* 的条目表示在这些环境中，扩展后的 CRL 没有超过 SAC。

Hyperparameter	Value
num_timesteps	100M-400M (varying across tasks)
update-to-data (UTD) ratio	1:40
max_replay_size	10,000
min_replay_size	1,000
episode_length	1,000
discounting	0.99
num_envs	512
batch_size	512
policy_lr	3e-4
critic_lr	3e-4
contrastive_loss_function	InfoNCE
energy_function	L2
logsumexp_penalty	0.1
Network depth	depends on the experiment
Network width	depends on the experiment
representation dimension	64

Table 7: Hyperparameters

超参数	取值
num_timesteps	100M–400M（因任务而异）
update-to-data (UTD) ratio	1:40
max_replay_size	10,000
min_replay_size	1,000
episode_length	1,000
discounting	0.99
num_envs	512
batch_size	512
policy_lr	3e-4
critic_lr	3e-4
contrastive_loss_function	InfoNCE
energy_function	L2
logsumexp_penalty	0.1
Network depth	视实验而定
Network width	视实验而定
representation dimension	64

表 7：超参数

NeurIPS Paper Checklist

NeurIPS 论文核查清单

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The abstract contains 3 main claims: (1) Depth scaled to 1024 layers; (2) Performance increases 2-50x on CRL and outperforms other goal-conditioned baselines. (3) These performance gains leads to qualitatively new learned behaviors. Each of these claims are clearly substantiated in the main text in Section 4.

Guidelines:

• The answer NA means that the abstract and introduction do not include the claims made in the paper.
• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

1. 论断

问题：摘要和引言中的主要论断是否准确反映了论文的贡献和范围？

回答：[是]

说明：摘要包含 3 项主要论断：（1）深度扩展到 1024 层；（2）CRL 的性能提高 2–50 倍，并超过其他目标条件基线；（3）这些性能提升带来质上全新的习得行为。每项论断都在正文第 4 节中得到了明确的支撑。

指南：

• 回答 NA 表示摘要和引言中没有包含论文提出的论断。
• 摘要和/或引言应清楚陈述论文的论断，包括论文的贡献以及重要的假设和局限。对这一问题回答“否”或 NA，审稿人不会有好印象。
• 论断应与理论和实验结果相符，并反映结果在多大程度上可望推广到其他设定。
• 可以把尚未实现的目标作为动机写进去，只要说清楚论文并未实现这些目标即可。

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We included a Limitations section that describes the main limitation of our paper, which is latency of deep networks. We also multiple times in the paper demarcated where our research can be extended by future work.

Guidelines:

• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
• The authors are encouraged to create a separate "Limitations" section in their paper.
• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

2. 局限

问题：论文是否讨论了作者所做工作的局限？

回答：[是]

说明：我们设有“局限”一节，说明了论文的主要局限，即深层网络的延迟。我们还多次在文中标明了研究可由未来工作扩展的地方。

指南：

• 回答 NA 表示论文没有局限，回答“否”表示论文有局限但没有讨论。
• 鼓励作者在论文中单独设一个“局限”小节。
• 论文应指出所有强假设，以及结果在这些假设被违反时有多稳健（例如独立性假设、无噪声设定、模型设定正确、只在局部成立的渐近近似）。作者应思考这些假设在实践中可能如何被违反，以及会有什么后果。
• 作者应反思论断的范围，例如方法是否只在少数数据集上或只用少数几次运行做过测试。一般来说，实证结果往往依赖隐含的假设，这些假设应当讲清楚。
• 作者应反思影响方法性能的因素。例如，图像分辨率低或光线不足时，人脸识别算法可能表现很差；语音转文字系统可能因为无法处理专业术语，不能可靠地为在线讲座提供字幕。
• 作者应讨论所提算法的计算效率，以及它们如何随数据集规模扩展。
• 如适用，作者应讨论其方法在隐私和公平问题上可能存在的局限。
• 作者可能担心，完全坦诚地交代局限会被审稿人当作拒稿理由，但更糟的结果可能是审稿人发现了论文没有承认的局限。作者应作出最佳判断，并认识到个人的透明行为对形成维护学术共同体诚信的规范起着重要作用。审稿人会被明确告知，不得因作者坦诚交代局限而扣分。

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [N/A]

Justification: This is an empirical paper. As such, no theoretical results that require assumptions or proofs.

Guidelines:

• The answer NA means that the paper does not include theoretical results.
• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
• All assumptions should be clearly stated or referenced in the statement of any theorems.
• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
• Theorems and Lemmas that the proof relies upon should be properly referenced.

3. 理论假设与证明

问题：对每项理论结果，论文是否给出了完整的假设集合和完整（且正确）的证明？

回答：[N/A]

说明：这是一篇实证论文，因此没有需要假设或证明的理论结果。

指南：

• 回答 NA 表示论文不包含理论结果。
• 论文中所有定理、公式和证明都应编号并相互引用。
• 所有假设都应在定理陈述中写明或引用。
• 证明可以放在正文或补充材料中；如果放在补充材料中，鼓励作者给出简短的证明梗概以帮助理解。
• 反过来，正文中给出的任何非正式证明，都应在附录或补充材料中配以正式证明。
• 证明所依赖的定理和引理应当注明出处。

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, documentation for reproducing the experiments is included alongside the anonymous code.

Guidelines:

• The answer NA means that the paper does not include experiments.
• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

4. 实验结果的可复现性

问题：论文是否充分披露了复现主要实验结果所需的全部信息，以达到影响论文主要论断和/或结论的程度（无论是否提供代码和数据）？

回答：[是]

说明：是的，匿名代码附有复现实验的说明文档。

指南：

• 回答 NA 表示论文不包含实验。
• 如果论文包含实验，对这一问题回答“否”，审稿人不会有好印象：无论是否提供代码和数据，让论文可复现都很重要。
• 如果贡献是数据集和/或模型，作者应说明为使结果可复现或可验证所采取的步骤。
• 根据贡献的不同，可以用多种方式实现可复现性。例如，如果贡献是一种新架构，完整描述架构可能就够了；如果贡献是某个具体模型及其实证评估，可能需要让他人能用同一数据集复制该模型，或者提供对模型的访问。一般来说，公开代码和数据往往是一个好办法，但也可以通过详细的复现说明、对托管模型的访问（例如大语言模型的情形）、发布模型检查点，或其他适合该研究的方式来实现可复现性。
• NeurIPS 虽然不要求公开代码，但要求所有投稿都提供某种合理的可复现途径，具体可能取决于贡献的性质。例如：(a) 如果贡献主要是一种新算法，论文应说清如何复现该算法。(b) 如果贡献主要是一种新的模型架构，论文应清楚完整地描述该架构。(c) 如果贡献是一个新模型（例如大语言模型），则应提供访问该模型以复现结果的途径，或者复现该模型的途径（例如使用开源数据集，或给出如何构建数据集的说明）。(d) 我们理解有些情况下可复现性比较棘手，这时欢迎作者说明他们提供可复现性的具体方式。对于闭源模型，访问可能受到某种限制（例如仅限注册用户），但其他研究者应当有某种途径来复现或验证结果。

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See link to anonymous code in Abstract.

Guidelines:

• The answer NA means that paper does not include experiments requiring code.
• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.
• While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.
• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

5. 数据和代码的开放获取

问题：论文是否开放了数据和代码，并在补充材料中附有足以忠实复现主要实验结果的说明？

回答：[是]

说明：见摘要中的匿名代码链接。

指南：

• 回答 NA 表示论文不包含需要代码的实验。
• 详情请参阅 NeurIPS 代码和数据提交指南（https://nips.cc/public/guides/CodeSubmissionPolicy）。
• 我们鼓励公开代码和数据，但也理解这可能无法做到，因此回答“否”是可以接受的。除非代码是贡献的核心（例如一个新的开源基准），论文不会仅仅因为没有附代码而被拒。
• 说明应包含复现结果所需的确切命令和环境。详情请参阅 NeurIPS 代码和数据提交指南（https://nips.cc/public/guides/CodeSubmissionPolicy）。
• 作者应说明数据的获取和准备方式，包括如何获取原始数据、预处理数据、中间数据和生成数据等。
• 作者应提供复现新方法和各基线全部实验结果的脚本。如果只有部分实验可复现，应说明脚本中省略了哪些实验以及原因。
• 投稿时，为保持匿名，作者应发布匿名版本（如适用）。
• 建议在补充材料（附在论文后）中提供尽可能多的信息，但也允许给出数据和代码的 URL。

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Experiments section and Appendix on Experimental Details

Guidelines:

• The answer NA means that the paper does not include experiments.
• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
• The full details can be provided either with the code, in appendix, or as supplemental material.

6. 实验设置与细节

问题：论文是否给出了理解结果所需的全部训练和测试细节（例如数据划分、超参数及其选择方式、优化器类型等）？

回答：[是]

说明：见“实验”一节和关于实验细节的附录。

指南：

• 回答 NA 表示论文不包含实验。
• 实验设置应在正文中交代到足以理解和领会结果的详细程度。
• 完整细节可以随代码、在附录中或作为补充材料提供。

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars in figures depict one standard error across random seeds. We used 5 seeds in Figure 1. For other figures in the main text, we could only run 3 seeds because of computational constraints.

Guidelines:

• The answer NA means that the paper does not include experiments.
• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
• The assumptions made should be given (e.g., Normally distributed errors).
• It should be clear whether the error bar is the standard deviation or the standard error of the mean.
• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

7. 实验的统计显著性

问题：论文是否恰当、正确地报告了误差线，或其他有关实验统计显著性的适当信息？

回答：[是]

说明：图中的误差线表示不同随机种子之间的一个标准误。图 1 使用了 5 个种子。正文其他图受计算资源限制，只能跑 3 个种子。

指南：

• 回答 NA 表示论文不包含实验。
• 如果结果附有误差线、置信区间或统计显著性检验，至少对支撑论文主要论断的实验是如此，作者就应回答“是”。
• 应清楚说明误差线所刻画的变异来源（例如训练/测试划分、初始化、某个参数的随机抽取，或在给定实验条件下的整体运行）。
• 应说明误差线的计算方法（闭式公式、调用某个库函数、自助法等）。
• 应给出所作的假设（例如误差服从正态分布）。
• 应说清误差线是标准差还是均值的标准误。
• 报告 1-sigma 误差线是可以的，但应说明。如果误差的正态性假设没有经过验证，作者最好报告 2-sigma 误差线，而不是声称有 96% 置信区间。
• 对于非对称分布，作者应注意不要在表格或图中给出会导致结果超出取值范围的对称误差线（例如负的错误率）。
• 如果在表格或图中报告了误差线，作者应在正文中说明其计算方式，并引用相应的图表。

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute resources are detailed in the appendix.

Guidelines:

• The answer NA means that the paper does not include experiments.
• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

8. 实验计算资源

问题：对每个实验，论文是否提供了复现实验所需计算资源的充分信息（计算单元类型、内存、执行时间）？

回答：[是]

说明：计算资源详见附录。

指南：

• 回答 NA 表示论文不包含实验。
• 论文应说明计算单元的类型（CPU 或 GPU）、内部集群或云服务商，以及相关的内存和存储。
• 论文应给出每次单独实验运行所需的计算量，并估算总计算量。
• 论文应披露整个研究项目所需的计算是否多于论文中报告的实验（例如没有写进论文的初步实验或失败实验）。

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: No known violations of the Code of Ethics.

Guidelines:

• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

9. 伦理准则

问题：论文中开展的研究是否在各方面都符合 NeurIPS 伦理准则 https://neurips.cc/public/EthicsGuidelines？

回答：[是]

说明：没有已知的违反伦理准则之处。

指南：

• 回答 NA 表示作者没有审阅 NeurIPS 伦理准则。
• 如果作者回答“否”，应说明需要偏离伦理准则的特殊情况。
• 作者应确保保持匿名（如适用），例如因其所在司法辖区的法律法规而有特殊考虑的情况。

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The Conclusion notes that there are no immediately societal impacts of the work.

Guidelines:

• The answer NA means that there is no societal impact of the work performed.
• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

10. 更广泛的影响

问题：论文是否讨论了所做工作潜在的正面和负面社会影响？

回答：[是]

说明：结论部分指出，这项工作没有直接的社会影响。

指南：

• 回答 NA 表示所做工作没有社会影响。
• 如果作者回答 NA 或“否”，应说明为什么他们的工作没有社会影响，或者为什么论文没有讨论社会影响。
• 负面社会影响的例子包括潜在的恶意或非预期用途（例如虚假信息、生成虚假个人资料、监控）、公平问题（例如部署可能对特定群体作出不公平决定的技术）、隐私问题和安全问题。
• 会议预计许多论文是基础研究，与特定应用无关，更不涉及部署。但如果存在通向任何负面应用的直接路径，作者应当指出。例如，指出生成模型质量的提升可能被用来生成用于散布虚假信息的深度伪造内容，是合理的。另一方面，不必指出一个优化神经网络的通用算法可能让人更快地训练出生成深度伪造内容的模型。
• 作者应考虑：技术按预期使用且运行正常时可能产生的危害；技术按预期使用但给出错误结果时可能产生的危害；以及因（有意或无意的）滥用而产生的危害。
• 如果存在负面社会影响，作者还可以讨论可能的缓解策略（例如分级发布模型、在提供攻击方法的同时提供防御方法、监控滥用的机制、监控系统如何随时间从反馈中学习的机制、提高机器学习的效率和可及性）。

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: No immediate impact to high-risk applications.

Guidelines:

• The answer NA means that the paper poses no such risks.
• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

11. 防护措施

问题：对于滥用风险高的数据或模型（例如预训练语言模型、图像生成器或爬取的数据集），论文是否说明了为负责任地发布而采取的防护措施？

回答：[N/A]

说明：对高风险应用没有直接影响。

指南：

• 回答 NA 表示论文不存在此类风险。
• 滥用或两用风险高的已发布模型，应在发布时配备必要的防护措施以便受控使用，例如要求用户遵守使用准则或访问模型的限制，或部署安全过滤器。
• 从互联网爬取的数据集可能带来安全风险。作者应说明他们如何避免发布不安全的图像。
• 我们认识到提供有效的防护措施很有挑战性，许多论文也不需要这样做，但我们鼓励作者考虑这一点，并尽最大努力。

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [N/A]

Justification: Benchmarks used are appropriately cited in the main text.

Guidelines:

• The answer NA means that the paper does not use existing assets.
• The authors should cite the original paper that produced the code package or dataset.
• The authors should state which version of the asset is used and, if possible, include a URL.
• The name of the license (e.g., CC-BY 4.0) should be included for each asset.
• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
• If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

12. 现有资产的许可

问题：论文中使用的资产（例如代码、数据、模型）的创建者或原始所有者是否得到了恰当的致谢，其许可和使用条款是否得到明确说明和恰当遵守？

回答：[N/A]

说明：所用基准已在正文中恰当引用。

指南：

• 回答 NA 表示论文没有使用现有资产。
• 作者应引用产生该代码包或数据集的原始论文。
• 作者应说明所用资产的版本，并尽可能给出 URL。
• 每项资产都应注明许可名称（例如 CC-BY 4.0）。
• 对于从特定来源（例如网站）爬取的数据，应提供该来源的版权和服务条款信息。
• 如果发布资产，应提供包内的许可、版权信息和使用条款。对于常用数据集，paperswithcode.com/datasets 整理了部分数据集的许可，其许可指南可以帮助确定数据集的许可。
• 对于重新打包的现有数据集，应同时提供原始许可和衍生资产的许可（如有变化）。
• 如果这些信息在网上找不到，鼓励作者联系资产的创建者。

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: Datasets and benchmark used are all from prior work and appropriately cited.

Guidelines:

• The answer NA means that the paper does not release new assets.
• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
• The paper should discuss whether and how consent was obtained from people whose asset is used.
• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

13. 新资产

问题：论文中引入的新资产是否有完善的文档，文档是否随资产一起提供？

回答：[N/A]

说明：所用数据集和基准都来自以往工作，并已恰当引用。

指南：

• 回答 NA 表示论文没有发布新资产。
• 研究者应通过结构化模板说明数据集/代码/模型的细节，包括训练、许可、局限等。
• 论文应讨论是否以及如何征得了其资产被使用者的同意。
• 投稿时，记得对资产做匿名处理（如适用）。可以创建一个匿名 URL，或附上一个匿名的 zip 文件。

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: No crowdsourcing experiments.

Guidelines:

• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

14. 众包与人类被试研究

问题：对于众包实验和人类被试研究，论文是否给出了提供给参与者的完整说明文本和截图（如适用），以及报酬细节（如有）？

回答：[N/A]

说明：没有众包实验。

指南：

• 回答 NA 表示论文不涉及众包，也不涉及人类被试研究。
• 可以把这些信息放在补充材料中，但如果论文的主要贡献涉及人类被试，则应在正文中给出尽可能多的细节。
• 根据 NeurIPS 伦理准则，参与数据收集、整理或其他劳动的工作人员，报酬至少应达到数据收集者所在国家的最低工资。

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: No human subject experiments

Guidelines:

• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

15. 人类被试研究的机构审查委员会（IRB）批准或同等批准

问题：论文是否说明了研究参与者可能承担的风险、这些风险是否已告知被试，以及是否获得了机构审查委员会（IRB）批准（或根据所在国家或机构要求的同等批准/审查）？

回答：[N/A]

说明：没有人类被试实验。

指南：

• 回答 NA 表示论文不涉及众包，也不涉及人类被试研究。
• 根据研究开展所在的国家，任何人类被试研究都可能需要 IRB 批准（或同等批准）。如果获得了 IRB 批准，应在论文中写明。
• 我们认识到这方面的程序因机构和地区而异，我们期望作者遵守 NeurIPS 伦理准则及其所在机构的指南。
• 初次投稿时，不要包含任何会破坏匿名性的信息（如适用），例如开展研究的机构。

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: LLMs were not used in writing the paper, and were only used for occasional code debugging.

Guidelines:

• The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
• Please refer to our LLM policy (https://neurips.cc/Conferences/2025/LLM) for what should or should not be described.

16. 大语言模型使用声明

问题：如果大语言模型是本研究核心方法中重要的、原创的或非标准的组成部分，论文是否说明了其使用情况？注意，如果大语言模型只用于写作、编辑或排版，不影响研究的核心方法、科学严谨性或原创性，则无需声明。

回答：[N/A]

说明：撰写论文时没有使用大语言模型，只在偶尔调试代码时用过。

指南：

• 回答 NA 表示本研究的核心方法开发不涉及将大语言模型作为任何重要、原创或非标准的组成部分。
• 关于哪些情况应当说明、哪些不必说明，请参阅我们的大语言模型政策（https://neurips.cc/Conferences/2025/LLM）。