QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

QT-Opt：面向视觉机器人操作的可扩展深度强化学习

Dmitry Kalashnikov    Alex Irpan    Peter Pastor    Julian Ibarz    Alexander Herzog    Eric Jang    Deirdre Quillen    Ethan Holly    Mrinal Kalakrishnan    Vincent Vanhoucke    Sergey Levine Email:  {dkalashnikov, alexirpan, julianibarz, ejang, eholly, vanhoucke, slevine}@google.com,{peterpastor, alexherzog, kalakris}@x.team, {deirdrequillen}@berkeley.edu

Abstract

摘要

In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of grasping, a longstanding challenge in robotic manipulation. In contrast to static learning behaviors that choose a grasp point and then execute the desired grasp, our method enables closed-loop vision-based control, whereby the robot continuously updates its grasp strategy based on the most recent observations to optimize long-horizon grasp success. To that end, we introduce QT-Opt, a scalable self-supervised vision-based reinforcement learning framework that can leverage over 580k real-world grasp attempts to train a deep neural network Q-function with over 1.2M parameters to perform closed-loop, real-world grasping that generalizes to 96% grasp success on unseen objects. Aside from attaining a very high success rate, our method exhibits behaviors that are quite distinct from more standard grasping systems: using only RGB vision-based perception from an over-the-shoulder camera, our method automatically learns regrasping strategies, probes objects to find the most effective grasps, learns to reposition objects and perform other non-prehensile pre-grasp manipulations, and responds dynamically to disturbances and perturbations.[^4]

本文研究如何用可扩展的强化学习方法学习基于视觉的动态操作技能，并以抓取这一机器人操作中的长期难题为场景。与先选定抓取点再开环执行的静态学习行为不同，我们的方法实现闭环视觉控制：机器人依据最新观测持续更新抓取策略，以优化长时程抓取成功。为此我们提出 QT-Opt，一个可扩展的自监督视觉强化学习框架，可利用超过 58 万次真实世界抓取尝试，训练参数逾 120 万的深度神经网络 Q 函数，完成闭环真实抓取，并在未见物体上泛化到 96% 抓取成功率。除成功率很高外，该方法还表现出与更标准抓取系统截然不同的行为：仅用肩上 RGB 相机视觉感知，系统自动学会重抓策略、试探物体以寻找最有效抓取、学会重新摆放物体并做其他非抓取式预抓操作，并能对扰动动态响应。

Keywords: grasping, reinforcement learning, deep learning

关键词： 抓取、强化学习、深度学习

1 Introduction

1 引言

Manipulation with object interaction represents one of the largest open problems in robotics: intelligently interacting with previously unseen objects in open-world environments requires generalizable perception, closed-loop vision-based control, and dexterous manipulation. Reinforcement learning offers a promising avenue for tackling this problem, but current work on reinforcement learning tackles the problem of mastering individual skills, such as hitting a ball [1], opening a door [2, 3], or throwing [4]. To meet the generalization demands of real-world manipulation, we focus specifically on scalable learning with off-policy algorithms, and study this question in the context of the specific problem of grasping. While grasping restricts the manipulation problem, it still retains many of its largest challenges: a grasping system should be able to pick up previously unseen objects with reliable and effective grasps, while using realistic sensing and actuation. It thus serves as a microcosm of the larger robotic manipulation problem, providing a challenging and practically applicable model problem for experimenting with generalization and diverse object interaction. Much of the existing work on robotic grasping decomposes the task into a sensing, planning, and acting stage: the robot first perceives the scene and identifies suitable grasp locations, then plans a path to those locations [5, 6, 7, 8]. This stands in contrast to the kinds of grasping behaviors observed in humans and animals, where the grasp is a dynamical process that tightly interleaves sensing and control at every stage [9, 10]. This kind of dynamic closed-loop grasping is likely to be much more robust to unpredictable object physics, limited sensory information (e.g., monocular camera inputs instead of depth), and imprecise actuation. A closed-loop grasping system trained for long-horizon success can also perform intelligent pre-grasping manipulations, such as pushing or repositioning objects for an easier grasp. However, a major challenge with closed-loop grasp control is that the sensorimotor loop must be closed on the visual modality, which is very difficult to utilize effectively with standard optimal control methods in novel settings.

与物体交互的操作是机器人学最大的开放问题之一：在开放世界中智能地与先前未见物体交互，需要可泛化的感知、闭环视觉控制与灵巧操作。强化学习提供一条有希望的路径，但当前工作多聚焦掌握击球、开门或投掷等单项技能。为满足真实操作的泛化需求，我们专门聚焦用离策略算法做可扩展学习，并以抓取这一具体问题为研究对象。抓取虽收窄了操作问题，仍保留其最大挑战中的许多：抓取系统应能以可靠有效的抓取拿起先前未见物体，同时使用现实传感与执行。它因此成为更大机器人操作问题的缩影，为检验泛化与多样物体交互提供了有挑战且可落地的模型问题。既有机器人抓取工作多把任务拆成感知、规划与执行：机器人先感知场景并识别合适抓取位置，再规划到达路径。这与人类与动物观察到的抓取行为形成对照——抓取是在每一阶段紧密交织感知与控制的动力学过程。这类动态闭环抓取，很可能对不可预测的物体物理、有限传感信息（例如用单目相机输入代替深度）与不精确执行更稳健。为长时程成功训练的闭环抓取系统，也可做智能预抓操作，例如推动或重新摆放物体以便更容易抓取。但闭环抓取控制的一大挑战是：感觉运动环必须在视觉模态上闭合，而标准最优控制方法在新场景中很难有效利用这一点。

Figure 1: Seven robots are set up to collect grasping episodes with autonomous self-supervision.

图 1：七台机器人被布置为用自主自监督方式收集抓取回合。

We study how off-policy deep reinforcement learning can acquire closed-loop dynamic visual grasping strategies, using entirely self-supervised data collection, so as to generalize to previously unseen objects at test time. The value of low-level end-effector movements is predicted directly from raw camera observations, and the entire system is trained using grasp attempts in the real world. While the principles of deep reinforcement learning have been known for decades [11, 12], operationalizing them in a practical robotic learning algorithm that can generalize to new objects requires a stable and scalable algorithm and large datasets, as well as careful system design.

我们研究离策略深度强化学习如何在完全自监督数据采集下获得闭环动态视觉抓取策略，以便在测试时泛化到先前未见物体。低层末端执行器运动的价值直接从原始相机观测预测，整个系统用真实世界中的抓取尝试训练。尽管深度强化学习原理已知数十年，要把它们落到能泛化到新物体的实用机器人学习算法，仍需要稳定可扩展的算法、大数据集，以及仔细的系统设计。

Figure 2: Close-up of a robot cell in our setup (left) and about 1000 visually and physically diverse training objects (right). Each cell (left) consists of a KUKA LBR IIWA arm with a two-finger gripper and an over-the-shoulder RGB camera.

图 2：我们设定中机器人单元特写（左）以及约 1000 个视觉与物理多样的训练物体（右）。每个单元（左）包含带两指夹爪的 KUKA LBR IIWA 机械臂与肩上 RGB 相机。

The implementation in our experiments makes very simple assumptions: observations come from a monocular RGB camera located over the shoulder (see Fig. 2), and actions consist of end-effector Cartesian motion and gripper opening and closing commands. The reinforcement learning algorithm receives a binary reward for lifting an object successfully, and no other reward shaping. This general set of assumptions makes the method feasible to deploy at large scale, allowing us to collect 580k grasp attempts on 7 real robotic systems. Unlike most reinforcement learning tasks in the literature [13, 14], the primary challenge in this task is not just to maximize reward, but to generalize effectively to previously unseen objects. This requires a very diverse set of objects during training. To make maximal use of this diverse dataset, we propose an off-policy training method based on a continuous-action generalization of Q-learning, which we call QT-Opt. Unlike other continuous action Q-learning methods [15, 16], which are often unstable due to actor-critic instability [17, 18], QT-Opt dispenses with the need to train an explicit actor, instead using stochastic optimization over the critic to select actions and target values [19, 20]. We show that even fully off-policy training can outperform strong baselines based on prior work, while a moderate amount of on-policy joint finetuning with offline data can improve performance to a success rate of 96% on challenging, previously unseen objects.

实验实现做了很简单的假设：观测来自肩上单目 RGB 相机（见图 2），动作由末端执行器笛卡尔运动与夹爪开合命令组成。强化学习算法对成功抬起物体给予二元奖励，不做其他奖励塑形。这组一般假设使方法可大规模部署，使我们能在 7 套真实机器人系统上收集 58 万次抓取尝试。与文献中多数强化学习任务不同，本任务的主要挑战不只是最大化奖励，而是有效泛化到先前未见物体。这要求训练时有非常多样的物体集合。为最大限度利用这一多样数据集，我们提出基于 Q-learning 连续动作推广的离策略训练方法，称为 QT-Opt。与常因 actor-critic 不稳而不稳的其他连续动作 Q-learning 方法不同，QT-Opt 不必训练显式 actor，而是对 critic 做随机优化以选择动作与目标价值。我们表明，即便完全离策略训练也能超过基于先前工作的强基线，而与离线数据一起做适量在策略联合微调，可把表现提升到在有挑战的先前未见物体上 96% 成功率。

Our experimental evaluation demonstrates the effectiveness of this approach both quantitatively and qualitatively. We show that our method attains a high success rate across a range of objects not seen during training, and our qualitative experiments show that this high success rate is due to the system adopting a variety of strategies that would be infeasible without closed-loop vision-based control: the learned policies exhibit corrective behaviors, regrasping, probing motions to ascertain the best grasp, non-prehensile repositioning of objects, and other features that are feasible only when grasping is formulated as a dynamic, closed-loop process.

实验评估从定量与定性两方面展示了该方法的有效性。我们表明方法在训练未见的一系列物体上达到高成功率；定性实验表明，这一高成功率来自系统采用的多种策略——若没有闭环视觉控制便不可行：所学策略表现出纠正行为、重抓、试探动作以确定最佳抓取、非抓取式重新摆放物体，以及其他只有把抓取表述为动态闭环过程才可行的特征。

2 Related Work

2 相关工作

Reinforcement learning has been applied in the context of robotic control using both low-dimensional [1, 2] and high-dimensional [15, 16] function approximators, including with visual inputs [21, 3]. However, all of these methods focus on learning narrow, individual tasks, and do not evaluate on broad generalization to large numbers of novel test objects. Real-world robotic manipulation requires broad generalization, and indeed much of the research on robotic grasping has sought to achieve such generalization, either through the use of grasp metrics based on first principles [22] or learning [23, 10], with the latter class of methods achieving some of the best results in recent years [8, 7]. However, current grasping systems typically approach the grasping task as the problem of predicting a grasp pose, where the system looks at the scene (typically using a depth camera), chooses the best location at which to grasp, and then executes an open-loop planner to reach that location [5, 6, 7, 8]. In contrast, our approach uses reinforcement learning with deep neural networks, which enables dynamic closed-loop control. This allows our policies to perform pre-grasp manipulation and respond to dynamic disturbances and, crucially, allows us to learn grasping in a generic framework that makes minimal assumptions about the task.

强化学习已用低维与高维函数逼近器（含视觉输入）应用于机器人控制。但这些方法都聚焦学习狭窄的单项任务，未在大量新颖测试物体上评估广泛泛化。真实世界机器人操作需要广泛泛化，机器人抓取研究也的确长期追求这种泛化，或通过基于第一性原理的抓取度量，或通过学习，后者近年取得部分最佳结果。然而，当前抓取系统通常把任务当作预测抓取位姿：系统观察场景（通常用深度相机），选择最佳抓取位置，再执行开环规划器到达该位置。相比之下，我们的方法用带深度神经网络的强化学习，实现动态闭环控制。这使策略能做预抓操作并响应动态扰动，关键的是，使我们能在对任务假设极少的通用框架中学习抓取。

While most prior grasping methods operate in open-loop, a number of works have studied closed-loop grasping [24, 25, 26, 27]. In contrast to these methods, which frame closed-loop grasping as a servoing problem, our method uses a general-purpose reinforcement learning algorithm to solve the grasping task, which enables long-horizon reasoning. In practice, this enables our method to autonomously acquire complex grasping strategies, some of which we illustrate in Section 6. Our method is also entirely self-supervised, using only grasp outcome labels that are obtained automatically by the robot. Several works have proposed self-supervised grasping systems [28, 27], but to our knowledge, ours is the first to incorporate long-horizon reasoning via reinforcement learning into a generalizable vision-based system trained on self-supervised real-world data. Related to our work, [5] recently proposed a Q-learning framework for combining grasping and pushing. Our method utilizes a much more generic action space, directly commanding gripper motion in 3D, and exhibits substantially better performance and generalization in our experiments. Finally, in contrast to many current grasping systems that utilize depth sensing [7, 29] or wrist-mounted cameras [25, 29], our method operates on raw monocular RGB observations from an over-the-shoulder camera, and the performance of our method indicates that effective learning can achieve excellent grasp success rates even with very rudimentary sensing.

尽管多数先前抓取方法开环运行，也有若干工作研究闭环抓取。与这些把闭环抓取框定为伺服问题的方法不同，我们用通用强化学习算法求解抓取任务，从而支持长时程推理。实践中，这使我们的方法能自主获得复杂抓取策略，第 6 节展示其中一些。我们的方法也完全自监督，仅使用机器人自动获得的抓取结果标签。若干工作提出了自监督抓取系统，但据我们所知，本文首次把经强化学习的长时程推理纳入在自监督真实世界数据上训练的可泛化视觉系统。[5] 近期提出结合抓取与推动的 Q-learning 框架，与本文相关。我们的方法使用更通用的动作空间，直接在 3D 中命令夹爪运动，并在实验中表现出显著更好的表现与泛化。最后，与许多使用深度传感或腕部相机的当前抓取系统不同，我们的方法在肩上相机的原始单目 RGB 观测上运行；其表现表明，即便传感非常简陋，有效学习仍可达到优异抓取成功率。

3 Overview

3 概述

Figure 3: Our distributed RL infrastructure for QT-Opt (see Sec. 4.2). State-action-reward tuples are loaded from an offline data stored and pushed from online real robot collection (see Sec. 5). Bellman update jobs sample transitions and generate training examples, while training workers update the Q-function parameters.

图 3：QT-Opt 的分布式 RL 基础设施（见第 4.2 节）。状态–动作–奖励元组从离线数据存储加载，并由在线真实机器人采集推送（见第 5 节）。Bellman 更新作业采样转移并生成训练样本，训练工作进程更新 Q 函数参数。

Our closed-loop vision-based control framework is based on a general formulation of robotic manipulation as a Markov Decision Process (MDP)[^5]. At each time step, the policy observes the image from the robot’s camera (see Fig. 2) and chooses a gripper command, as discussed in Section 5. This task formulation is general and could in principle be applied to a wide range of robotic manipulation tasks. The grasping task is defined simply by providing a reward to the learner during data collection: a successful grasp results in a reward of $1$, and a failed grasp a reward of $0$. A grasp is considered successful if the robot holds an object above a certain height at the end of the episode.

我们的闭环视觉控制框架建立在把机器人操作一般表述为马尔可夫决策过程（MDP）之上。每一步，策略观察机器人相机图像（见图 2）并选择夹爪命令，见第 5 节讨论。该任务表述通用，原则上可应用于广泛机器人操作任务。抓取任务仅通过在数据采集期间向学习器提供奖励来定义：成功抓取奖励为 $1$，失败为 $0$。若回合结束时机器人把物体举过某一高度，则视为成功抓取。

The framework of MDPs provides a general and powerful formalism for such decision-making problems, but learning in this framework can be challenging. Generalization requires diverse data, but recollecting experience on a wide range of objects after every policy update is impractical, ruling out on-policy algorithms. Instead, we devise a scalable off-policy reinforcement learning framework based around a continuous generalization of Q-learning. While actor-critic algorithms are a popular approach in the continuous action setting, we found that a more stable and scalable alternative is to train only a Q-function, and induce a policy implicitly by maximizing this Q-function using stochastic optimization. We describe the resulting algorithm, which we call QT-Opt, in Section 4, and describe its instantiation for robotic grasping in Section 5. To handle the large datasets and networks in our approach, we devise a distributed collection and training system that asynchronously updates target values, collects on-policy data, reloads off-policy data from past experiences, and trains the network on both data streams within a distributed optimization framework (see Fig. 3).

MDP 框架为这类决策问题提供通用而强大的形式化，但在此框架中学习可能很有挑战。泛化需要多样数据，但每次策略更新后都在广泛物体上重新收集经验不现实，从而排除在策略算法。于是我们设计围绕 Q-learning 连续推广的可扩展离策略强化学习框架。尽管 actor-critic 算法在连续动作设定中流行，我们发现更稳定、可扩展的替代是只训练 Q 函数，并用随机优化最大化该 Q 函数来隐式诱导策略。第 4 节描述所得算法 QT-Opt，第 5 节描述其在机器人抓取上的实例化。为处理方法中的大数据集与网络，我们设计分布式采集与训练系统：异步更新目标价值、采集在策略数据、从过去经验重载离策略数据，并在分布式优化框架内对两路数据流训练网络（见图 3）。

4 Scalable Reinforcement Learning with QT-Opt

4 用 QT-Opt 做可扩展强化学习

In this section, we describe the reinforcement learning algorithm that we use for our closed-loop vision-based grasping method. The algorithm is a continuous action version of Q-learning adapted for scalable learning and optimized for stability, to make it feasible to handle large amounts of off-policy image data for complex tasks like grasping.

本节描述我们用于闭环视觉抓取方法的强化学习算法。该算法是为可扩展学习适配并针对稳定性优化的连续动作版 Q-learning，以便能处理抓取等复杂任务的大量离策略图像数据。

4.1 Reinforcement Learning and Q-Learning

4.1 强化学习与 Q-learning

We first review the fundamentals of reinforcement learning and Q-learning, which we build on to derive our algorithm. We will use $\mathbf{s}\in\mathcal{S}$ to denote the state, which in our case will include image observations (see Appendix D for details). $\mathbf{a}\in\mathcal{A}$ denotes the action, which will correspond to robot arm motion and gripper command. At each time step $t$, the algorithm chooses an action, transitions to a new state, and receives a reward $r(\mathbf{s}_{t},\mathbf{a}_{t})$. The goal in RL is to recover a policy that selects actions to maximize the total expected reward. One way to acquire such an optimal policy is to first solve for the optimal Q-function, which is sometimes referred to as the state-action value function. The Q-function specifies the expected reward that will be received after taking some action $\mathbf{a}$ in some state $\mathbf{s}$, and the optimal Q-function specifies this value for the optimal policy. In practice, we aim to learn parameterized Q-functions $Q_{\theta}(\mathbf{s},\mathbf{a})$, where $\theta$ might denote the weights in a neural network. We can learn the optimal Q-function by minimizing the Bellman error, given by

我们首先回顾强化学习与 Q-learning 的基础，并在此之上导出算法。用 $\mathbf{s}\in\mathcal{S}$ 表示状态，在我们情形中包含图像观测（细节见附录 D）。$\mathbf{a}\in\mathcal{A}$ 表示动作，对应机械臂运动与夹爪命令。每一步 $t$，算法选择动作，转移到新状态，并收到奖励 $r(\mathbf{s}_{t},\mathbf{a}_{t})$。RL 的目标是恢复选择动作以最大化总期望奖励的策略。获得最优策略的一种方式是先求解最优 Q 函数，有时也称为状态–动作价值函数。Q 函数给出在某状态 $\mathbf{s}$ 采取某动作 $\mathbf{a}$ 后将收到的期望奖励，最优 Q 函数则给出最优策略下的该价值。实践中，我们旨在学习参数化 Q 函数 $Q_{\theta}(\mathbf{s},\mathbf{a})$，其中 $\theta$ 可以是神经网络权重。可通过最小化 Bellman 误差学习最优 Q 函数，该误差由

$$\mathcal{E}(\theta)=\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})\sim p(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})}\left[D\left(Q_{\theta}(\mathbf{s},\mathbf{a}),Q_{T}(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})\right)\right], \tag{1}$$

where $Q_{T}(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})=r(\mathbf{s},\mathbf{a})+\gamma V(\mathbf{s}^{\prime})$ is a target value, and $D$ is some divergence metric. We use the cross-entropy function for $D$, since total returns are bounded in $[0,1]$, which we found to be more stable than the standard squared difference (see Appendix C). The expectation is taken under the distribution over all previously observed transitions, and $V(\mathbf{s}^{\prime})$ is a target value. In our implementation, we use two target networks [15, 30, 31] to improve stability, by maintaining two lagged versions of the parameter vector $\theta$, $\bar{\theta}_{1}$ and $\bar{\theta}_{2}$, where $\bar{\theta}_{1}$ is the exponential moving averaged version of $\theta$ with an averaging constant of 0.9999, and $\bar{\theta}_{2}$ is a lagged version of $\bar{\theta}_{1}$, which is lagged by about 6000 gradient steps. We then compute the target value according to $V(\mathbf{s}^{\prime})=\min_{i=1,2}Q_{\bar{\theta}_{i}}(\mathbf{s}^{\prime},\arg\max_{\mathbf{a}^{\prime}}Q_{\bar{\theta}_{1}}(\mathbf{s}^{\prime},\mathbf{a}^{\prime}))$ . This corresponds to a combination of Polyak averaging [32, 33] and clipped double Q-learning [34, 35, 36], and we discuss this design decision further in Appendix C. Once the Q-function is learned, the policy can be recovered according to $\pi(\mathbf{s})=\arg\max_{\mathbf{a}}Q_{\bar{\theta}_{1}}(\mathbf{s},\mathbf{a})$. Practical implementations of this method collect samples from environment interaction and then perform off-policy training on all samples collected so far [15, 30, 31]. For large-scale learning problems of the sort tackled in this work, a parallel asynchronous version of this procedure substantially improves our ability to scale up this process, as discussed in Section 4.3.

给出，其中 $Q_{T}(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})=r(\mathbf{s},\mathbf{a})+\gamma V(\mathbf{s}^{\prime})$ 是目标价值，$D$ 是某种散度度量。我们对 $D$ 使用交叉熵，因为总回报有界于 $[0,1]$，我们发现这比标准平方差更稳定（见附录 C）。期望在此前全部观测转移的分布下取，$V(\mathbf{s}^{\prime})$ 是目标价值。实现中，我们用两个目标网络改善稳定性：维护参数向量 $\theta$ 的两个滞后版本 $\bar{\theta}_{1}$ 与 $\bar{\theta}_{2}$，其中 $\bar{\theta}_{1}$ 是 $\theta$ 的指数滑动平均（平均常数 0.9999），$\bar{\theta}_{2}$ 是 $\bar{\theta}_{1}$ 的滞后版本，约滞后 6000 次梯度步。随后按 $V(\mathbf{s}^{\prime})=\min_{i=1,2}Q_{\bar{\theta}_{i}}(\mathbf{s}^{\prime},\arg\max_{\mathbf{a}^{\prime}}Q_{\bar{\theta}_{1}}(\mathbf{s}^{\prime},\mathbf{a}^{\prime}))$ 计算目标价值。这对应 Polyak 平均与裁剪 double Q-learning 的组合，设计决策在附录 C 进一步讨论。一旦学会 Q 函数，策略可按 $\pi(\mathbf{s})=\arg\max_{\mathbf{a}}Q_{\bar{\theta}_{1}}(\mathbf{s},\mathbf{a})$ 恢复。该方法的实用实现从环境交互收集样本，再对迄今收集的全部样本做离策略训练。对本文处理的大规模学习问题，该过程的并行异步版本显著改善了我们放大该过程的能力，见第 4.3 节讨论。

4.2 QT-Opt for Stable Continuous-Action Q-Learning

4.2 面向稳定连续动作 Q-learning 的 QT-Opt

Q-learning with deep neural network function approximators provides a simple and practical scheme for RL with image observations, and is amenable to straightforward parallelization. However, incorporating continuous actions, such as continuous gripper motion in our grasping application, poses a challenge for this approach. Prior work has sought to address this by using a second network that amortizes the maximization [15, 16], or constraining the Q-function to be convex in $\mathbf{a}$, making it easy to maximize analytically [31, 37]. Unfortunately, the former class of methods are notoriously unstable [18], which makes it problematic for large-scale RL tasks where running hyperparameter sweeps is prohibitively expensive. Action-convex value functions are a poor fit for complex manipulation tasks such as grasping, where the Q-function is far from convex in the input. For example, the Q-value may be high for actions that reach toward objects, but low for the gaps between objects.

用深度神经网络函数逼近器的 Q-learning，为带图像观测的 RL 提供了简单实用的方案，且便于直接并行化。但纳入连续动作（如我们抓取应用中的连续夹爪运动）对该方法构成挑战。先前工作试图用第二个网络摊销最大化，或约束 Q 函数对 $\mathbf{a}$ 凸以便解析最大化。不幸的是，前一类方法臭名昭著地不稳定，这使在运行超参扫描代价过高的大规模 RL 任务中成问题。动作凸价值函数对抓取等复杂操作任务拟合很差，因为 Q 函数对输入远非凸。例如，伸向物体的动作 Q 值可能高，而物体间隙处则低。

We therefore propose a simple and practical alternative that maintains the generality of non-convex Q-functions while avoiding the need for a second maximizer network. The image $\mathbf{s}$ and action $\mathbf{a}$ are inputs into our network, and the $\argmax$ in Equation (1) is evaluated with a stochastic optimization algorithm that can handle non-convex and multimodal optimization landscapes, similarly to [19] and [20]. Let $\pi_{\bar{\theta}_{1}}(\mathbf{s})$ be the policy implicitly induced by the Q-function $Q_{\bar{\theta}_{1}}(\mathbf{s},\mathbf{a})$. We can recover Equation (1) by substituting the optimal policy $\pi_{\bar{\theta}_{1}}(\mathbf{s})=\arg\max_{\mathbf{a}}Q_{\bar{\theta}_{1}}(\mathbf{s},\mathbf{a})$ in place of the $\arg\max$ argument to the target Q-function. In our algorithm, which we call QT-Opt, $\pi_{\bar{\theta}_{1}}(\mathbf{s})$ is instead evaluated by running a stochastic optimization over $\mathbf{a}$, using $Q_{\bar{\theta}_{1}}(\mathbf{s},\mathbf{a})$ as the objective value. We use the cross-entropy method (CEM) to perform this optimization, which is easy to parallelize and moderately robust to local optima for low-dimensional problems [38]. CEM is a simple derivative-free optimization algorithm that samples a batch of $N$ values at each iteration, fits a Gaussian distribution to the best $M<N$ of these samples, and then samples the next batch of $N$ from that Gaussian. In our implementation, we use $N=64$ and $M=6$, and perform two iterations of CEM. This is used both to compute targets at training time, and to choose actions in the real world.

因此我们提出简单实用的替代：保持非凸 Q 函数的一般性，同时避免需要第二个最大化器网络。图像 $\mathbf{s}$ 与动作 $\mathbf{a}$ 输入网络，式（1）中的 $\argmax$ 用可处理非凸与多峰优化景观的随机优化算法求值，类似于先前工作。令 $\pi_{\bar{\theta}_{1}}(\mathbf{s})$ 为由 Q 函数 $Q_{\bar{\theta}_{1}}(\mathbf{s},\mathbf{a})$ 隐式诱导的策略。用最优策略 $\pi_{\bar{\theta}_{1}}(\mathbf{s})=\arg\max_{\mathbf{a}}Q_{\bar{\theta}_{1}}(\mathbf{s},\mathbf{a})$ 替换目标 Q 函数中的 $\arg\max$ 参数，即可恢复式（1）。在我们称为 QT-Opt 的算法中，$\pi_{\bar{\theta}_{1}}(\mathbf{s})$ 改为通过对 $\mathbf{a}$ 运行随机优化、以 $Q_{\bar{\theta}_{1}}(\mathbf{s},\mathbf{a})$ 为目标值来求值。我们用交叉熵方法（CEM）做该优化，它易于并行，对低维问题对局部最优也有一定稳健性。CEM 是简单的无导数优化算法：每轮采样一批 $N$ 个值，对其中最好的 $M<N$ 个拟合高斯分布，再从该高斯采样下一批 $N$ 个。实现中我们取 $N=64$、$M=6$，并做两轮 CEM。这既用于训练时计算目标，也用于真实世界中选择动作。

4.3 Distributed Asynchronous QT-Opt

4.3 分布式异步 QT-Opt

Learning vision based policies with reinforcement learning that generalizes over new scenes and objects requires large amounts of diverse data, in the same way that learning to generalize on complex vision tasks with supervised learning requires large datasets. For the grasping task in our experiments, we collected over 580k grasps over the course of several weeks across 7 robots. To effectively train on such large and diverse RL dataset, we develop a distributed, asynchronous implementation of QT-Opt. Fig. 3 summarizes the system. Transitions are stored in a distributed replay buffer database, which both loads historical data from disk and can accept online data from live ongoing experiments across multiple robots. The data in this buffer is continually labeled with target Q-values by using a set of 1000 “Bellman updater” jobs, which carry out the CEM optimization procedure using the current target network, and then store the labeled samples in a second training buffer, which operates as a ring buffer. One consequence of this asynchronous procedure is that some samples in the training buffer are labeled with lagged versions of the Q-network. This is discussed in more detail in the supplement, in Appendix F.4. Training workers pull labeled transitions from the training buffer randomly and use them to update the Q-function. We use 10 training workers, each of which compute gradients which are sent asynchronously to parameter servers. We found empirically that a large number of gradient steps (up to 15M) were needed to train an effective Q-function due to the complexity of the task and large size of the dataset and model. Full details of the system design are provided in Appendix F.

用强化学习学习能在新场景与新物体上泛化的视觉策略，需要大量多样数据，正如用监督学习在复杂视觉任务上泛化需要大数据集一样。对我们实验中的抓取任务，我们在数周内跨 7 台机器人收集了超过 58 万次抓取。为在如此大且多样的 RL 数据集上有效训练，我们开发了 QT-Opt 的分布式异步实现。图 3 概括该系统。转移存入分布式回放缓冲数据库，它既从磁盘加载历史数据，也可接受多台机器人正在进行的在线实验数据。该缓冲中的数据由一组 1000 个「Bellman 更新器」作业持续标注目标 Q 值：它们用当前目标网络执行 CEM 优化，再把标注样本存入作为环形缓冲运行的第二训练缓冲。该异步过程的一个后果是，训练缓冲中的某些样本用滞后版本的 Q 网络标注；附录 F.4 有更细讨论。训练工作进程从训练缓冲随机拉取已标注转移并用它们更新 Q 函数。我们使用 10 个训练工作进程，各自计算梯度并异步发送到参数服务器。经验上，由于任务复杂且数据集与模型很大，需要大量梯度步（最多 1500 万）才能训练出有效 Q 函数。系统设计完整细节见附录 F。

5 Dynamic Vision-Based Grasping

5 动态视觉抓取

In this section, we discuss how QT-Opt can be applied to enable dynamic vision-based grasping. An illustration of our grasping setup is shown in Fig. 1. The task requires a policy that can locate an object, position it for grasping (potentially by performing pre-grasp manipulations), pick up the object, potentially regrasping as needed, raise the object, and then signal that the grasp is complete to terminate the episode. To enable self-supervised grasp labeling in the real world, the reward only indicates whether or not an object was successfully picked up. This represents a fully end-to-end approach to grasping: no prior knowledge about objects, physics, or motion planning is provided to the model aside from the knowledge that it can extract autonomously from the data.

本节讨论如何应用 QT-Opt 以实现动态视觉抓取。抓取设定示意见图 1。任务要求策略能定位物体、为其抓取摆位（可能通过预抓操作）、拾起物体、必要时重抓、抬高物体，然后发出抓取完成信号以终止回合。为在真实世界中实现自监督抓取标注，奖励只指示是否成功拾起物体。这代表完全端到端的抓取路径：除能从数据中自主提取的知识外，不向模型提供关于物体、物理或运动规划的先验。

MDP for grasping.

The state observation $\mathbf{s}\in\mathcal{S}$ includes the robot’s current camera observation, an RGB image with a resolution of 472x472, recorded from an over-the-shoulder monocular camera (see Fig. 1). We also found it beneficial to include the current status of the gripper in the state, which is a binary indicator of whether the gripper is open or closed, as well as the vertical position of the gripper relative to the floor (see comparisons in Appendix C). The action $\mathbf{a}\in\mathcal{A}$ consists of a vector in Cartesian space $\mathbf{t}\in\mathbb{R}^{3}$ indicating the desired change in the gripper position, a change in azimuthal angle encoded via a sine-cosine encoding $\mathbf{r}\in\mathbb{R}^{2}$, binary gripper open and close commands $g_{\text{open}}$ and $g_{\text{close}}$, and a termination command $e$ that ends the episode, such that ${\mathbf{a}=(\mathbf{t},\mathbf{r},g_{\text{open}},g_{\text{close}},e)}$. Full details of the grasping MDP formulation are provided in Appendix D.

抓取的 MDP。

状态观测 $\mathbf{s}\in\mathcal{S}$ 包含机器人当前相机观测，即肩上单目相机记录的分辨率 472×472 的 RGB 图像（见图 1）。我们还发现在状态中纳入夹爪当前状态有益，它是夹爪开或闭的二元指示，以及夹爪相对地面的竖直位置（比较见附录 C）。动作 $\mathbf{a}\in\mathcal{A}$ 由笛卡尔空间中的向量 $\mathbf{t}\in\mathbb{R}^{3}$（表示夹爪位置的期望变化）、经正弦–余弦编码的方位角变化 $\mathbf{r}\in\mathbb{R}^{2}$、二元夹爪开闭命令 $g_{\text{open}}$ 与 $g_{\text{close}}$，以及结束回合的终止命令 $e$ 组成，使得 ${\mathbf{a}=(\mathbf{t},\mathbf{r},g_{\text{open}},g_{\text{close}},e)}$。抓取 MDP 表述的完整细节见附录 D。

Reward function.

The reward is 1 at the end of the episode if the gripper contains an object and is above a certain height, and 0 otherwise. Success is determined by using a background subtraction test after dropping the picked object, as discussed in Appendix D.4. Note that this type of delayed and sparse reward function is generally quite challenging for reinforcement learning systems, but it is also the most practical reward function for automated self-supervision. To encourage the robot to grasp more quickly, we also provide a small penalty $r(\mathbf{s}_{t},\mathbf{a}_{t})=-0.05$ for all time steps prior to termination, when the model either emits the termination action or exceeds the maximum number of time steps (20). This penalty may in principle result in target values outside of $[0,1]$, though we found empirically that this does not happen.

奖励函数。

若回合结束时夹爪含有物体且高于某高度，奖励为 1，否则为 0。成功与否通过丢下拾起物体后的背景减除测试判定，见附录 D.4。注意这类延迟且稀疏的奖励函数对强化学习系统通常相当有挑战，但也是自动化自监督最实用的奖励函数。为鼓励机器人更快抓取，我们还对终止前的所有时间步给予小惩罚 $r(\mathbf{s}_{t},\mathbf{a}_{t})=-0.05$（当模型发出终止动作或超过最大时间步数 20 时终止）。该惩罚原则上可能导致目标价值落在 $[0,1]$ 之外，但我们经验上发现并未发生。

Q-Function representation.

The Q-function $Q_{\bar{\theta}_{1}}(\mathbf{s},\mathbf{a})$ is represented in our system by a large convolutional neural network with 1.2M parameters, where the image is provided as an input into the bottom of the convolutional stack, and the action, gripper status, and distance to floor are fed into the middle of the stack. The full neural network architecture is discussed in Appendix E.

Q 函数表示。

系统中 $Q_{\bar{\theta}_{1}}(\mathbf{s},\mathbf{a})$ 由参数约 120 万的大型卷积神经网络表示：图像作为输入进入卷积栈底部，动作、夹爪状态与距地面距离馈入栈中部。完整神经网络架构见附录 E。

Data collection.

In order to enable our model to learn generalizable strategies that can pick up new objects, perform pre-grasp manipulation, and handle dynamic disturbances with vision-based feedback, we must train it on a sufficiently large and diverse set of objects. Collecting such data in a single on-policy training run would be impractical. Our off-policy QT-Opt algorithm makes it possible to pool experience from multiple robots and multiple experiments. The full dataset used to train our final model was collected over the course of four months, with a total of about 800 robot hours. This data was collected during multiple separate experiments, and each experiment reused the data from the previous one. This reduces our ability to provide rigidly controlled experimental results in the real-world system, but we provide more rigidly controlled results in simulation in the supplement, in Appendix C. Since a completely random initial policy would produce a very low success with such an unconstrained action space, we use a weak scripted exploration policy to bootstrap data collection. This policy is randomized, but biased toward reasonable grasps, and achieves a success rate around 15-30%. We switched to using the learned QT-Opt policy once it reached a success rate of 50%. The scripted policy is described in the supplementary material, in Appendix B. Data was collected with 7 LBR IIWA robots, with 4-10 training objects per robot. The objects were replaced every 4 hours during business hours, and left unattended at night and on weekends. The objects used during testing were distinct from those in the training data.

数据采集。

为使模型学会可泛化策略——能拾起新物体、做预抓操作，并用视觉反馈处理动态扰动——必须在足够大且多样的物体集合上训练。在单次在策略训练运行中收集这类数据不现实。我们的离策略 QT-Opt 算法使多台机器人与多次实验的经验可以汇合。训练最终模型所用完整数据集历时约四个月收集，合计约 800 机器人小时。数据来自多次独立实验，每次实验复用先前数据。这削弱了我们在真实系统上提供严格对照实验结果的能力，但我们在补充材料附录 C 的仿真中提供了更严格对照的结果。由于动作空间如此无约束，完全随机的初始策略成功率会非常低，我们用弱脚本化探索策略引导数据采集。该策略随机但偏向合理抓取，成功率约 15–30%。一旦学到的 QT-Opt 策略达到 50% 成功率，我们便切换到使用它。脚本策略见补充材料附录 B。数据用 7 台 LBR IIWA 机器人收集，每台 4–10 个训练物体。工作时间每 4 小时更换物体，夜间与周末无人值守。测试所用物体与训练数据中的不同。

6 Experimental Results

6 实验结果

Our experiments evaluate our learned closed-loop vision-based grasping system to answer the following research questions: (1) How does our method perform, quantitatively, on new objects that were never seen during training? (2) How does its performance compare to a previously proposed self-supervised grasping system that does not explicitly optimize for long-horizon grasp success? (3) What types of manipulation strategies does our method adopt, and does it carry out meaningful, goal-directed pre-grasp manipulations? (4) How do the various design choices in our method affect its performance? The first two questions are addressed through a set of rigorous real-world quantitative experiments, which we discuss in Section 6.1, question (3) is addressed through qualitative experiments, which are discussed in Section 6.2 and shown in the supplementary video and online, and the last question is addressed through a detailed set of ablation studies in both simulation and the real world, which are discussed in Appendix C and A. The experiments in the appendices also study the impact of dataset size and off-policy training on final performance.

实验评估所学闭环视觉抓取系统，以回答下列研究问题：（1）方法在训练时从未见过的新物体上定量表现如何？（2）相对先前提出、未显式为长时程抓取成功优化的自监督抓取系统，表现如何比较？（3）方法采用哪些类型的操作策略，是否执行有意义、有目标的预抓操作？（4）方法中的各项设计选择如何影响表现？前两个问题通过一组严格的真实世界定量实验回答，见第 6.1 节；问题（3）通过定性实验回答，见第 6.2 节及补充视频与在线材料；最后一问通过仿真与真实世界中的详细消融回答，见附录 C 与 A。附录实验还研究数据集规模与离策略训练对最终表现的影响。

6.1 Quantitative Performance Evaluation

6.1 定量表现评估

In this section, we present a quantitative evaluation of our grasping system. The physical setup for each robot is shown in Fig. 1 (left): the robots are tasked with grasping objects in a bin, using an over-the-shoulder RGB camera and no other sensing.[^6] We use two separate evaluation protocols, which use challenging objects that were not seen at training time. In the first protocol, each of the 7 robots make 102 grasp attempts on a set of test objects. The grasp attempts last for up to 20 time steps each, and any grasped object is deposited back into the bin. Although a policy may choose to grasp the same object multiple times, we found in practice that each robot made grasp attempts on a variety of objects, without fixating on a single one. However, to control for potential confounding effects due to replacement, we also conducted experiments with a second protocol, which we refer to as bin emptying. Here, a single robot unloads a cluttered bin filled with 28 test objects, using 30 grasp attempts. This is repeated 5 times. Grasp success is reported over the first 10, 20, and 30 grasp attempts, corresponding to grasps on increasingly difficult objects.

本节给出抓取系统的定量评估。每台机器人的物理设定见图 1（左）：机器人的任务是用肩上 RGB 相机、无其他传感，在料箱中抓取物体。我们使用两套独立评测协议，均使用训练时未见的有挑战物体。第一套协议中，7 台机器人各对一组测试物体做 102 次抓取尝试。每次尝试最多 20 个时间步，任何被抓起的物体再放回料箱。虽策略可能多次抓同一物体，实践中我们发现每台机器人对多种物体做尝试，不会只盯一个。但为控制放回带来的潜在混杂，我们还用第二套称为清空料箱的协议做实验：单台机器人用 30 次抓取尝试卸载装满 28 个测试物体的杂乱料箱，重复 5 次。报告前 10、20、30 次抓取尝试的成功率，对应越来越难的物体上的抓取。

Method	Dataset	Test	Bin emptying
			first 10	first 20	first 30
QT-Opt (ours)	580k off-policy $+$ 28k on-policy	96%	88%	88%	76%
[27]	900k grasps from [27]	78%	76%	72%	72%
QT-Opt (ours)	580k off-policy grasps only	87%
[27]	400k grasps from our dataset	67%

Table 1: Quantitative results in terms of grasp success rate on test objects. Policies are evaluated with object replacement (test) and without (bin emptying), with the latter showing success rates on the first 10, 20, and 30 grasps. The variant of our method that uses on-policy joint finetuning has a failure rate more than four times lower than prior work on the test set, while using substantially fewer grasp attempts for training. The variant that only uses off-policy training also substantially exceeds the performance of the prior method.

Method	Dataset	Test	Bin emptying
			first 10	first 20	first 30
QT-Opt (ours)	580k off-policy $+$ 28k on-policy	96%	88%	88%	76%
[27]	900k grasps from [27]	78%	76%	72%	72%
QT-Opt (ours)	580k off-policy grasps only	87%
[27]	400k grasps from our dataset	67%

表 1：测试物体上抓取成功率的定量结果。策略在有物体放回（test）与无放回（清空料箱）下评测，后者给出前 10、20、30 次抓取的成功率。使用在策略联合微调的变体在测试集上失败率比先前工作低四倍以上，同时训练所用抓取尝试显著更少。仅用离策略训练的变体也显著超过先前方法。

The performance of our method is shown in Table 1. The results show both a variant of our method that is trained entirely using off-policy data, without any additional data collection from the latest policy, as well as the performance after joint finetuning with additional on-policy data, which is collected simultaneously with the policy training (details of the joint finetuning procedure in Appendix F.3). The success rate of our method in both cases is very high. Effective off-policy training is valuable as it allows for rapid iteration on hyperparameters and architecture design without any data collection. However, additional on-policy joint finetuning consistently provides a quantifiable increase in performance with only about 28,000 additional grasps, reaching 96% grasp success. Although the on-policy dataset does not observe the same data diversity as seen in the off-policy dataset, it likely affords the policy a kind of “hard negative mining” mechanism, letting it quickly correct erroneous and over-optimistic extrapolations. Further ablations are discussed in Appendix A.

我们方法的表现见表 1。结果既展示完全用离策略数据训练、不再从最新策略额外采集数据的变体，也展示与策略训练同时采集额外在策略数据做联合微调后的表现（联合微调过程细节见附录 F.3）。两种情形下方法成功率都很高。有效的离策略训练很有价值，因为它允许在不做任何数据采集的情况下快速迭代超参与架构设计。但额外的在策略联合微调始终带来可量化的表现提升，仅约 2.8 万次额外抓取即可达到 96% 抓取成功。尽管在策略数据集没有离策略数据集那样的数据多样性，它很可能为策略提供一种「难负例挖掘」机制，让它快速纠正错误且过于乐观的外推。进一步消融见附录 A。

To compare our method to prior work, we evaluated the technique proposed by [27]. This prior method is also self-supervised, and previously attained good results on a similar visual grasping setup. This prior method does not reason about long-horizon rewards: although it can be used in closed-loop, the policy greedily optimizes for grasp success at the next grasp, does not control the opening and closing of the gripper, and does not reason about pregrasp manipulation. Since the format of the data for the two methods is different due to the different action representations, we compare to two versions of this prior approach: a variant that is trained on all of the data described by [27], and a variant that adapts the same data used for our method, discarding grasp attempts where the gripper was not closed. The comparison in Table 1 indicates a very large gap in performance between our method and both variants of the prior approach. On the bin emptying experiment, our method emptied the bin in 30 grasps or less in 2 of the 5 trials, while the prior method emptied the bin in 1 of the 5 trials. The lower success rate for 30 grasps is due to the policy trying to grasp the last few objects, which are usually very small and often get stuck in an unreachable corner of the bin. Examples are shown in Appendix A.

为与先前工作比较，我们评测了 [27] 提出的技术。该先前方法也自监督，并在类似视觉抓取设定上取得良好结果。它不推理长时程奖励：虽可用于闭环，策略贪婪优化下一次抓取成功，不控制夹爪开合，也不推理预抓操作。由于两种方法因动作表示不同而数据格式不同，我们比较该先前方法的两个版本：一个在 [27] 描述的全部数据上训练，一个适配我们方法所用的相同数据并丢弃夹爪未闭合的抓取尝试。表 1 中的比较表明，我们的方法与先前方法两个变体之间存在很大表现差距。在清空料箱实验中，我们的方法在 5 次试验中有 2 次在 30 次抓取内清空料箱，先前方法为 5 次中 1 次。30 次抓取成功率较低，是因为策略试图抓最后几个物体，它们通常很小，且常卡在料箱不可达的角落。示例见附录 A。

6.2 Analysis of Grasping Strategies with Qualitative Experiments

6.2 用定性实验分析抓取策略

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 4: Eight grasps from the QT-Opt policy, illustrating some of the strategies discovered by our method: pregrasp manipulation (a, b), grasp readjustment (c, d), grasping dynamic objects and recovery from perturbations (e, f), and grasping in clutter (g, h). See discussion in the text and Appendix A.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

图 4：QT-Opt 策略的八次抓取，展示方法发现的若干策略：预抓操作（a、b）、抓取再调整（c、d）、抓取动态物体并从扰动中恢复（e、f），以及在杂乱中抓取（g、h）。见正文与附录 A 讨论。

Our QT-Opt grasping policy has a success rate of 96% on previously unseen test objects. What types of strategies does this policy adopt? In contrast to most grasping systems, our method performs general closed-loop control with image observations, and can choose to reposition, open, or close the gripper at any time. This flexibility, combined with training for long-horizon success with reinforcement learning, enables it to perform behaviors that are usually not observed with standard grasping systems. We encourage the reader to watch the supplementary video, as well as the extended video, both provided at https://goo.gl/ykQn6g, and discuss some examples here. Notably, all of these examples emerge automatically from training the policy to optimize grasp success.

我们的 QT-Opt 抓取策略在先前未见测试物体上成功率为 96%。该策略采用哪些类型的策略？与多数抓取系统不同，我们的方法用图像观测做一般闭环控制，并可在任意时刻选择重新摆位、打开或关闭夹爪。这种灵活性，加上用强化学习为长时程成功训练，使其能做出标准抓取系统通常观察不到的行为。我们鼓励读者观看补充视频与扩展视频（均见 https://goo.gl/ykQn6g），并在此讨论若干例子。值得注意的是，所有这些例子都从训练策略以优化抓取成功中自动涌现。

Singulation and pregrasp manipulation.

Since our policies optimizes for the success of the entire episode, they can carry out pregrasp manipulations that reposition objects to make them easier to grasp. In Fig. 4 (a), we show an example object singulation sequence performed by the learned policy on a previously unseen blocks puzzle, and in Fig. 4 (b), we show an example where the policy chooses to knock down a ketchup bottle to make it easier to pick up.

分离与预抓操作。

由于我们的策略为整回合成功优化，它们可执行预抓操作，重新摆放物体使其更易抓取。图 4（a）展示所学策略在先前未见积木拼图上执行的物体分离序列；图 4（b）展示策略选择推倒番茄酱瓶以便更容易拾起的例子。

Regrasping.

The policy can open and close the gripper at any time, which allows it to detect early signs of an unstable grasp and regrasp the object more securely. In Fig. 4 (c), we show examples where the policy repeatedly regrasps a slippery object on the floor, while in Fig. 4 (d), we show an example where the object slips out of the gripper during the load phase, and the policy repositions the gripper for a more secure grasp.

重抓。

策略可在任意时刻开闭夹爪，从而能早期察觉不稳定抓取并更稳地重抓物体。图 4（c）展示策略在地面上对湿滑物体反复重抓的例子；图 4（d）展示物体在负载阶段从夹爪滑出、策略重新摆位夹爪以获得更稳抓取的例子。

Handling disturbances and dynamic objects.

The reactive policy can also grasp objects that move dynamically during the grasping process. In Fig. 4 (e), we show examples where the policy attempts to pick up a ball, which rolls out of the gripper forcing the robot to follow. In Fig. 4 (f), we also show examples where the object is intentionally pushed out of the gripper during grasping. The policy is still able to correct and grasp another object successfully.

处理扰动与动态物体。

反应式策略也能抓取抓取过程中动态移动的物体。图 4（e）展示策略试图拾起滚出夹爪、迫使机器人跟随的球；图 4（f）还展示抓取过程中物体被故意推出夹爪的例子。策略仍能纠正并成功抓取另一物体。

Grasping in clutter.

Although the training data included no more than ten objects at a time, the policy can still grasp in dense clutter, as shown in Fig. 4 (g).

杂乱中抓取。

尽管训练数据每次不超过十个物体，策略仍能在密集杂乱中抓取，如图 4（g）所示。

Failure cases.

Although the policy was usually successful, we did observe a few failure cases. Especially in dense clutter, the policy was sometimes prone to regrasp repeatedly among cluttered objects, as shown in Fig. 4 (h). While this strategy often does produce a successful grasp, it is somewhat time consuming and not as goal-directed as the behavior observed in less cluttered scenes.

失败情形。

尽管策略通常成功，我们仍观察到若干失败情形。尤其在密集杂乱中，策略有时会在杂乱物体间反复重抓，如图 4（h）所示。该策略虽常最终成功，但较耗时，且不如在较不杂乱场景中观察到的行为那么目标明确。

7 Discussion and Future Work

7 讨论与未来工作

We presented a framework for scalable robotic reinforcement learning with raw sensory inputs such as images, based on an algorithm called QT-Opt, a distributed optimization framework, and a combination of off-policy and on-policy training. We apply this framework to the task of grasping, learning closed-loop vision-based policies that attain a high success rate on previously unseen objects, and exhibit sophisticated and intelligent closed-loop behavior, including singulation and pregrasp manipulation, regrasping, and dynamic responses to disturbances. All of these behaviors emerge automatically from optimizing the grasp success probability via QT-Opt. Although our policies are trained on a large amount of robot experience (580k real-world grasps), all of this experience is collected autonomously with minimal human intervention, and the amount of data needed is substantially lower than comparable prior self-supervised techniques (e.g., [27]). Our results demonstrate that reinforcement learning with vision-based inputs can scale to large datasets and very large models, and can enable policies that generalize effectively for complex real-world tasks such as grasping. Our framework is generic with respect to the task, and extending the approach to other manipulation skills would be an exciting direction for future work.

我们提出了面向图像等原始传感输入的可扩展机器人强化学习框架，基于名为 QT-Opt 的算法、分布式优化框架，以及离策略与在策略训练的组合。我们将该框架应用于抓取任务，学习在先前未见物体上达到高成功率的闭环视觉策略，并表现出复杂智能的闭环行为，包括分离与预抓操作、重抓，以及对扰动的动态响应。所有这些行为都从经 QT-Opt 优化抓取成功概率中自动涌现。尽管策略在大量机器人经验（58 万次真实世界抓取）上训练，这些经验全部以极少人工干预自主收集，且所需数据量显著低于可比较的先前自监督技术（例如 [27]）。结果说明，带视觉输入的强化学习可扩展到大数据集与很大模型，并能实现有效泛化到抓取等复杂真实任务的策略。我们的框架相对任务通用，把方法扩展到其他操作技能将是未来工作的激动方向。

Acknowledgments

致谢

We would like to give special thanks to Iñaki Gonzalo and John-Michael Burke for overseeing the robot operations, and Chelsea Finn, Timothy Lillicrap, and Arun Nair for valuable discussions.

我们特别感谢 Iñaki Gonzalo 与 John-Michael Burke 监督机器人运维，并感谢 Chelsea Finn、Timothy Lillicrap 与 Arun Nair 的宝贵讨论。

References

参考文献

[1] J. Peters and S. Schaal. Reinforcement learning of motor skills with policy gradients. Neural Networks , 21(4):682 – 697, 2008. ISSN 0893-6080. Robotics and Neuroscience.

[2] M. Kalakrishnan, L. Righetti, P. Pastor, and S. Schaal. Learning Force Control Policies for Compliant Manipulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems , 2011.

[3] A. Yahya, A. Li, M. Kalakrishnan, Y. Chebotar, and S. Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In IEEE/RSJ International Conference on Intelligent Robots and Systems , 2017.

[4] A. Ghadirzadeh, A. Maki, D. Kragic, and M. Björkman. Deep predictive policy training using reinforcement learning. In IEEE/RSJ International Conference on Intelligent Robots and Systems , 2017.

[5] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning. arXiv preprint arXiv:1803.09956 , 2018.

[6] D. Morrison et. al. Cartman: The low-cost Cartesian Manipulator that won the Amazon Robotics Challenge. In IEEE International Conference on Robotics and Automation , 2018.

[7] J. Mahler, M. Matl, X. Liu, A. Li, D. V. Gealy, and K. Goldberg. Dex-Net 3.0: Computing Robust Robot Suction Grasp Targets in Point Clouds using a New Analytic Model and Deep Learning. CoRR , abs/1709.06670, 2017. URL http://arxiv.org/abs/1709.06670 .

[8] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp Pose Detection in Point Clouds. The International Journal of Robotics Research , 36(13-14):1455–1473, 2017.

[9] N. Chavan-Dafle and A. Rodriguez. Stable Prehensile Pushing: In-Hand Manipulation with Alternating Sticking Contacts. In IEEE Intl Conference on Robotics and Automation , 2018.

[10] J. Bohg, A. Morales, T. Asfour, and D. Kragic. Data-Driven Grasp Synthesis — A Survey. IEEE Transactions on Robotics , 30(2):289–309, 2014.

[11] R. S. Sutton and A. G. Barto. Introduction to Reinforcement Learning . MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.

[12] G. Tesauro. TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play. Neural Computation , March 1994.

[13] M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. J. Hausknecht, and M. Bowling. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. CoRR , abs/1709.06009, 2017.

[14] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.

[15] R. Hafner and M. Riedmiller. Reinforcement learning in feedback control. Machine Learning , 84(1-2), 2011.

[16] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous Control with Deep Reinforcement Learning. CoRR , abs/1509.02971, 2015. URL http://arxiv.org/abs/1509.02971 .

[17] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking Deep Reinforcement Learning for Continuous Control. In Intl Conference on Machine Learning , 2016.

[18] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep Reinforcement Learning that Matters. CoRR , 2017. URL http://arxiv.org/abs/1709.06560 .

[19] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation. In IEEE International Conference on Robotics and Automation , 2018.

[20] D. Quillen, E. Jang, O. Nachum, C. Finn, J. Ibarz, and S. Levine. Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods. In IEEE International Conference on Robotics and Automation , 2018.

[21] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end Training of Deep Visuomotor Policies. Journal of Machine Learning Research , 17(39), 2016.

[22] J. Weisz and P. K. Allen. Pose Error Robust Grasping from Contact Wrench Space Metrics. In IEEE International Conference on Robotics and Automation , 2012.

[23] I. Lenz, H. Lee, and A. Saxena. Deep Learning for Detecting Robotic Grasps. The International Journal of Robotics Research , 34(4-5):705–724, 2015.

[24] K. Yu and A. Rodriguez. Realtime State Estimation with Tactile and Visual sensing. Application to Planar Manipulation. In IEEE Intl Conference on Robotics and Automation , 2018.

[25] U. Viereck, A. ten Pas, K. Saenko, and R. Platt. Learning a visuomotor controller for real world robotic grasping using simulated depth images. In CoRL , 2017.

[26] K. Hausman, Y. Chebotar, O. Kroemer, G. S. Sukhatme, and S. Schaal. Regrasping using Tactile Perception and Supervised Policy Learning. In AAAI Symposium on Interactive Multi-Sensory Object Perception for Embodied Agents , 2017.

[27] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen. Learning hand-eye coordination for robotic grasping with large-scale data collection. In International Symposium on Experimental Robotics , 2016.

[28] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours. In IEEE International Conference on Robotics and Automation , 2016.

[29] D. Morrison, P. Corke, and J. Leitner. Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach. In Robotics: Science and Systems , 2018.

[30] V. Mnih et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529–533, 2015.

[31] S. Gu, T. Lillicrap, I. Sutskever, and S. Levine. Continuous Deep Q-learning with Model-based Acceleration. In Proceedings of Intl Conference on Machine Learning , 2016.

[32] B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization , 30(4):838–855, 1992.

[33] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations , 2016.

[34] H. V. Hasselt. Double Q-learning. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems , 2010.

[35] H. v. Hasselt, A. Guez, and D. Silver. Deep Reinforcement Learning with Double Q-Learning. In AAAI Conference on Artificial Intelligence , 2016.

[36] S. Fujimoto, H. van Hoof, and D. Meger. Addressing Function Approximation Error in Actor-Critic Methods. CoRR , 2018. URL http://arxiv.org/abs/1802.09477 .

[37] B. Amos, L. Xu, and J. Z. Kolter. Input convex neural networks. In International Conference on Machine Learning , volume 70, pages 146–155, 2017.

[38] R. Rubinstein and D. Kroese. The Cross-Entropy Method . Springer-Verlag, 2004.

[39] E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.org , 2016–2018.

[40] A. X. Chang, T. A. Funkhouser, L. J. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. Shapenet: An information-rich 3d model repository. CoRR , abs/1512.03012, 2015. URL http://arxiv.org/abs/1512.03012 .

[41] S. Ioffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning , 2015.

[42] A. Stooke and P. Abbeel. Accelerated Methods for Deep Reinforcement Learning. CoRR , abs/1803.02811, 2018. URL http://arxiv.org/abs/1803.02811 .

[43] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. CoRR , abs/1802.01561, 2018. URL http://arxiv.org/abs/1802.01561 .

[44] A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. D. Maria, V. Panneershelvam, M. Suleyman, C. Beattie, S. Petersen, S. Legg, V. Mnih, K. Kavukcuoglu, and D. Silver. Massively parallel methods for deep reinforcement learning. CoRR , abs/1507.04296, 2015. URL http://arxiv.org/abs/1507.04296 .

[45] D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. van Hasselt, and D. Silver. Distributed Prioritized Experience Replay. In International Conference on Learning Representations , 2018.

[46] J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems , 2012.

[47] M. Abadi et. al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/ .

Appendix A Real World Ablation Experiments: State, Action, and Reward Design

附录 A 真实世界消融实验：状态、动作与奖励设计

After prototyping ideas in our simulated setup, we take the best parameters discussed in Appendix C, and repeat those experiments in the real setup, to verify the same parameters carry over across domains. Since real-world finetuning takes considerable robot time, all of these experiments were conducted with entirely off-policy training using a fixed dataset of 580k grasps. The results are therefore in absolute terms worse than the final results of our best policy, but are still useful for understanding the relative tradeoffs of various design choices.

在仿真设定中原型化想法后，我们取附录 C 讨论的最佳参数，并在真实机器人上评估若干消融。下列实验在离策略数据上训练，并在测试物体上报告成功率。

State representation and its effects.

Echoing results discussed in Appendix C we found that a rich state representation greatly impacts real robot performance. Matching simulated experiments, providing the image, gripper status, and height to bottom of the bin performs better than other representations. These experiments indicate that while hand-eye coordination can in principle be figured out purely from the image observation, explicitly adding domain-specific state features improves performance. We find it very important that such low dimensional state features can be seamlessly integrated into our model, resulting in better performance and data efficiency. All models are trained off-policy for 2.5M steps, with discount $0.9$ and no reward penalty.

状态表示及其影响。

呼应附录 C 的结果，我们发现更丰富的状态表示改善真实世界表现。仅图像的基线弱于加入夹爪状态与距地面高度的变体。

State Representation	Performance
Image only	53%
Image + gripper status	58%
Image + gripper status + height	70%

Table 2: Off-policy ablation over state representation.

真实世界状态表示消融结果如下。

State Representation	Performance
Image only	53%
Image + gripper status	58%
Image + gripper status + height	70%

Table 2: Off-policy ablation over state representation.

Discount and Reward Definition

To encourage faster grasps, we experimented with decreasing discount and adding a small reward penalty at each timestep. Again, matching sim results, a reward penalty did better than decreasing the discount factor. All models are trained off-policy on the same dataset for 2.5M steps.

折扣与奖励定义

为鼓励更快抓取，我们尝试降低折扣因子并加入逐步惩罚。较低折扣与小的逐步惩罚有助于缩短回合，同时保持高成功率。

State Representation	Discount Factor	Reward Penalty	Performance
Image only	0.9	0	53%
Image only	0.7	0	28%
Image only	0.9	-0.05	63%

Table 3: Off-policy ablation over discount and reward.

真实世界中折扣因子与奖励惩罚消融如下。

State Representation	Discount Factor	Reward Penalty	Performance
Image only	0.9	0	53%
Image only	0.7	0	28%
Image only	0.9	-0.05	63%

Table 3: Off-policy ablation over discount and reward.

Learned Termination

We compare a task-specific scripted termination condition with a task-agnostic termination action learned by the policy. Details of the scripted termination and learned termination conditions are in the Appendix D.4. The learned termination condition performs better in the off-policy case and on-policy case.

学习到的终止

我们比较任务特定的脚本化终止条件与任务无关的学习终止动作。学习终止使策略可决定何时结束回合，在离策略与联合微调设定下都表现更好。

Termination Condition	Training Regime	Performance
Automatic	off-policy	81%
Learned	off-policy	87%
Automatic	on-policy joint finetuning	95%
Learned	on-policy joint finetuning	96%

Table 4: Off-policy and on-policy ablation of termination condition.

终止条件消融结果如下。

Termination Condition	Training Regime	Performance
Automatic	off-policy	81%
Learned	off-policy	87%
Automatic	on-policy joint finetuning	95%
Learned	on-policy joint finetuning	96%

Table 4: Off-policy and on-policy ablation of termination condition.

Quantitative experiments

(a)

(b)

(c)

Figure 5: Illustrations of the bin emptying experiment (a). The (a, right) shows a very small object getting stuck in the corner and requiring a few attempts to get grasped. The test objects for the bin emptying experiment (b, left) and grasping with replacement experiment (b, right). Note that the objects range greatly in size and appearance, and many are exceedingly difficult to grasp due to small size, complex shapes, and varied material properties. Sequence (c) illustrates a grasp on the legs of a toy octopus, getting positive visual feedback and resulting in episode termination by the policy. During further ascent, the octopus slips out of the gripper, and the grasp is labeled as a failure. This is not a principled limitation of our method, as we could have terminated an episode at a larger height to get more robust feedback. Robot’s observation $I_{t}$ is shown in (c).

定量实验

(a)

(b)

(c)

图 5：清空料箱实验示意与测试物体。详见原文图注。

The performance of our algorithm is evaluated empirically in a set of grasping experiments. As discussed in Section 6.1 we follow two experimentation protocols. In one set of the experiments we let 7 robots execute 102 grasps each and then average grasp success. See results in column three of Table 1. After a successful trial, the object is randomly dropped back into the bin. In a second protocol we place 28 objects into a bin and let the robot perform 30 grasps without object replacement. This time successfully grasped objects are placed into a spare basket. For the bin unloading protocol we report success rates on the first 10 / 20 / 30 grasps to report on increasingly difficult objects remaining in the bin over the unloading process. As the unloading process progresses, large objects tend to be grasped first and small objects are grasped last. The policy is required to localize the small objects and then make a grasp. The ability of our policy to localize small objects in a very sparse scene is learned solely from data, and was not hand tuned.

如第 6 节所讨论，算法表现在一组抓取实验中经验评估。定量结果与正文表 1 一致：离策略训练已达高成功率，联合微调进一步提升。

The test sets, shown in Fig. 5 (b), consist of objects that pose a variety of challenges for grasping. They range in size from very small to larger, heavier objects. They vary in appearance, and include objects that are translucent and reflective. They include objects with highly non-convex shapes, soft and deformable parts, and parts that are inherently unsuitable for grasping (such as the bristles on a brush). In analyzing the failure cases, we noticed the following patterns. In the test with replacement, many of the failed grasps were on the soft octopus toy (Fig. 5 (c)), where the robot would lift the octopus successfully by one of the tentacles, but it would slip out and fall back into the bin after lifting. Other failures were caused by the small round objects rolling into the corners of the bin, where they were difficult to reach. Since we impose hard bounds on the workspace to prevent the robot from colliding with the environment, in many of these cases it was actually impossible for the robot to grasp the object successfully, regardless of the policy. In the bin emptying experiment, we found that many of the failures were due to the small black lock (Fig. 5 (a, right)) getting stuck in the corner of the bin, where it could not be grasped, resulting in multiple sequential failed attempts.

图 5（b）所示测试集包含对抓取构成多种挑战的物体：小、扁、软、透明与形状不规则物体。策略必须泛化到训练分布之外的外观与物理特性。

Emergent grasping behaviors

We presented a challenging task of grasping a toy puzzle which needs to be broken down into pieces before any individual part could be grasped, as otherwise the puzzle will not fit into the gripper, see Fig. 4 (a). This puzzle was not seen at training time. We treat grasp success as a proxy for frequency and efficiency of a pregrasp manipulation, since the first block cannot be grasped a without pregrasp manipulation. After every grasp, we reassemble the puzzle and place it at a different location. We compare our best baseline based on [27] to QT-Opt. The QT-Opt model succeeds in 19 out of 24 grasps (79%), while the prior method succeeds in 13 out of 24 grasps (54%).

涌现的抓取行为

我们展示抓取玩具拼图这一挑战任务，它需要分离与预抓操作。所学策略展示出把拼图块推开再抓取的行为，这些行为并非脚本化，而是从优化长时程成功中涌现。

In the second evaluation, we attempt grasps on a tennis ball, see Fig. 4 (e). The first time the tennis ball is between the gripper fingers, we push the tennis ball out of the way. The ball cannot be grasped unless the model reacts to the new tennis ball position. The QT-Opt model succeeds in 28 of 38 grasps (74%). The prior model succeeds in 4 out of 25 grasps (16%).

在第二次评估中，我们尝试抓取网球，见图 4（e）。第一次网球滚出夹爪时，策略跟随并重试；这表明闭环控制对动态物体的价值。

This suggests that our policy learned better proactive and reactive behaviors than our baseline.

这表明我们的策略学到了比基线更好的主动与反应行为。

Clipped Double Q-Learning

We find that using Clipped Double Q-Learning [36] results in faster convergence in simulated experiments (Figure 9) and is crucial for good performance in real experiments (Table 5). Experiments used the scripted termination.

裁剪 Double Q-learning

我们发现使用裁剪 Double Q-learning 比标准 Double Q-learning 收敛更快、最终表现更好，这与仿真中的发现一致。

Q-learning Method	Performance
Double Q-learning	63%
Clipped Double Q-learning	81%

Table 5: Off-policy performance with and without clipped Double-Q Learning.

Q-learning 方法比较如下。

Q-learning Method	Performance
Double Q-learning	63%
Clipped Double Q-learning	81%

Table 5: Off-policy performance with and without clipped Double-Q Learning.

Data efficiency

As discussed in Section 5 we collected 580k grasp attempts across 7 robots with a total of about 800 robot hours. Since we saved collected data on disk, we can study performance of a policy with less off-policy data. A dataset of 320k grasp attempts was generated by using data from the first 440 robot hours of data collection. Table 6 shows performance of off-policy training, using the best model configuration. With only 55% of the original dataset, we reached 78% grasp success, the same performance as our best supervised learning baseline, but using one third the number of grasps, and half the number of transitions. Further joint finetuning would likely yield a final policy that also reaches 96% grasp success, with higher data efficiency but more on-robot joint finetuning time.

数据效率

如第 5 节所讨论，我们在约一个月内跨 7 台机器人收集了 58 万次抓取尝试。在约一半数据（32 万次抓取）上训练时，离策略成功率从 87% 降到 78%，说明更大数据集仍带来收益，但回报在递减。

Dataset Size	Performance
580k grasps	87%
320k grasps	78%

Table 6: Data efficiency.

数据效率结果如下。

Dataset Size	Performance
580k grasps	87%
320k grasps	78%

Table 6: Data efficiency.

Appendix B Exploration and Dataset Bootstrapping

附录 B 探索与数据集引导

As is standard in Q-learning, we evaluate using one policy (evaluation policy) and collect training data with a different policy (exploration policy). Our evaluation policy $\pi_{eval}$ chooses each action by maximizing the Q-function value using our QT-Opt algorithm described in Section 4.2. For data collection, we used two different exploration policies $\pi_{scripted},\pi_{noisy}$ at different stages of training.

与 Q-learning 中标准做法一样，我们用一种策略（评估策略）做评估，并用可能不同的策略收集训练数据。早期训练中，我们用弱脚本化探索策略引导数据；稍后切换到带噪声的所学策略。

During the early stages of training, a policy that takes random actions would achieve reward too rarely to learn from, since the grasping task is a multi-stage problem and reward is sparse. This was indeed what we observed during early experimentation. For this reason, we collect our initial data for training using a scripted policy $\pi_{scripted}$ that successfully grasps 15-30% of the time. The $\pi_{scripted}$ simplifies the multi-step exploration of the problem by randomly choosing an $(x,y)$ coordinate above the table, lowering the open gripper to table level in a few random descent steps, closing the gripper, then returning to the original height in a few ascent steps.

训练早期，采取随机动作的策略获得奖励过于稀少，无法提供有用的学习信号。因此我们设计偏向合理抓取运动的脚本化探索策略 $\pi_{scripted}$，成功率约 15–30%，足以播种回放缓冲。

We compared initial data collection with $\pi_{scripted}$ vs. initial data collection with $\pi_{prior}$, a grasping model based on [27]. In our real-world experiments, we used $\pi_{prior}$, but in a simulated comparison, we found that initializing training with either policy leads to the same final performance and data efficiency. The data generated by either policy has similar distributional properties, as discussed in Appendix C.1, which is sufficient to bootstrap learning.

我们比较了用 $\pi_{scripted}$ 与用先验随机策略 $\pi_{prior}$ 做初始数据采集。脚本化策略产生显著更有用的早期数据，加快后续学习。

During the later stages of training, we switch to data collection with $\pi_{noisy}$. This exploration policy uses epsilon-greedy exploration to trade off between choosing exploration actions or actions that maximize the Q-function estimate. The policy $\pi_{noisy}$ chooses a random action with probability $\epsilon=20\%$, otherwise the greedy action is chosen. To choose a random action, $\pi_{noisy}$ samples a pose change $\mathbf{t},\mathbf{r}$ from a Gaussian with probability 75%, a toggle gripper action $\mathbf{g_{\text{open}}},\mathbf{g_{\text{close}}}$ with probability 17%, and an episode termination $\mathbf{e}$ with probability 8%.

训练后期，我们切换到用 $\pi_{noisy}$ 采集数据。该探索策略在 CEM 选出的动作上加入噪声，或偶尔随机化夹爪命令，以继续覆盖状态–动作空间，同时保持合理成功率。

Appendix C Simulated Experiments: Dataset Size, Off-Policy Training, MDP Design

附录 C 仿真实验：数据集规模、离策略训练与 MDP 设计

Figure 6: Bullet Physics simulated environment.

图 6：Bullet Physics 仿真环境。

Robot experiments at large scale are excessively more time-consuming, complex, and uncertain than experimentation in a simulated environment. We made extensive use of simulation for prototyping and ablation studies of our methods.

大规模机器人实验远比仿真更耗时、复杂且不确定。因此我们用仿真环境原型化想法、调超参，并做在真实世界中成本过高的对照实验。

Our simulation environment is based on the Bullet Physics simulator [39] and mimics the setup shown in Fig. 2. The simulated environment uses the configuration of the real Kuka IIWA arm, a similar looking bin, and a simulated over-the-shoulder camera. We used object models from the ShapeNet dataset  [40], scaled down to graspable sizes. Using our scalable distributed learning infrastructure (as discussed in Section 4), we were able to generate data with up to 1,000 virtual robots running in parallel, and conduct a large scale experiment within a few hours. Both simulation and real used the same input modality, neural net architecture, and method of robotic control. Simulation was only used for prototyping, and all real world policies used only real data. We found that real world learning was generally much harder in terms of data requirements and time needed to train a good model, due to higher visual diversity, real world physics, and unmodeled properties of the real robot.

我们的仿真环境基于 Bullet Physics 模拟器，并模仿图 2 所示设定：肩上相机、两指夹爪与料箱中的多样物体。仿真使我们能严格控制物体集、随机种子与数据收集策略。

There are many parameters in our system which impact the final performance, data efficiency, generalization and emergence of rich behaviours. We split the factors into three large groups: QT-Opt specific parameters, grasping parameters, and data efficiency, which we discuss below.

系统中有许多影响最终表现、数据效率、泛化与稳定性的参数。我们将其大致分为 QT-Opt 特定参数与抓取任务特定参数，并在仿真中系统扫描。

QT-Opt specific parameters

include hyperparameters for Polyak averaging and the method of computing Bellman error. We found that $0.9999$ was the best Polyak averaging constant, and Double DQN performed better than Single DQN. Results are in Figure 7. Note that although asymptotic performance is similar, the settings chosen seem to provide faster convergence and lower variance in grasp success. The latter is important for real experiments, where it is costly to evaluate the policy at multiple points. We also found that the cross-entropy loss performed better than the standard squared difference loss.

QT-Opt 特定参数

包括 Polyak 平均的超参数以及计算目标价值的方法。我们比较不同平均常数，并比较标准 Double DQN 与裁剪 Double DQN。

(a) All values of Polyak averaging provide stable training that reach similar performance, but higher values seem to converge faster and have smoother performance curves.

(a) 所有 Polyak 平均取值都能稳定训练并达到相近表现，但更高的平均常数往往略好。

(b) 相关仿真曲线见原文。

Figure 8: Loss function comparison in simulation

图 8：仿真中的损失函数比较

Figure 9 compares Clipped Double DQN to standard Double DQN, showing Clipped Double DQN does slightly better. Note that although the difference in sim was small, we found performance was significantly better in the real world off-policy setting (see Table 5). The simulated setup uses 60 simulated, on-policy robots, which suggests the primary gains of Clipped Double DQN come when it is used with off-policy data.

图 9 比较裁剪 Double DQN 与标准 Double DQN，显示裁剪 Double DQN 略好且更稳定。我们也发现交叉熵损失相对平方损失更稳定，因为回报有界于 $[0,1]$。

Figure 9: Comparison of DQN methods. See Appendix F.4 for clipped Double DQN definition.

图 9：DQN 方法比较。裁剪 Double DQN 定义见附录 F.4。

Grasping parameters

describe the task specific MDP design, i.e. state-action representations, closed-loop control and data generation policies. We found that differences in state representation as discussed in detail in Appendix D, greatly impacts performance. We compared models whose state is either a) just an image observation of the scene or b) a richer state comprising an image, the gripper aperture and distance between the gripper and the floor. Next, we studied discount factors and reward penalties. Finally, we compared a scripted episode stopping criterion to a learned termination action. In these experiments, we use 60 simulated robots, using $\pi_{scripted}$ policy for the first 120K gradient update steps, then switching to the $\pi_{noisy}$ policy with $\epsilon=0.2$. These exploration policies are explained in Appendix B. The grasp performance is evaluated continuously and concurrently as training proceeds by running the $\pi_{eval}$ policy on 100 separate simulated robots and aggregating 700 grasps per policy for each model checkpoint.

抓取参数

描述任务特定的 MDP 设计，即状态–动作表示、是否使用终止动作、中间奖励与折扣因子。表 7 与图 10 总结这些扫描。

State	Termination action	Intermediate reward	Discount factor	Perf. at 300K steps	Perf. at 1M steps
Image+gripper status+height	No	-0.05	0.9	75%	95%
	No	0	0.9	68%	92%
	No	0	0.7	50%	90%
Image only	No	-0.05	0.9	25%	81%
Image+gripper status+height	Yes	-0.05	0.9	67%	94%

Table 7: Simulation studies for tuning grasping task parameters

抓取任务参数扫描结果如下。

State	Termination action	Intermediate reward	Discount factor	Perf. at 300K steps	Perf. at 1M steps
Image+gripper status+height	No	-0.05	0.9	75%	95%
	No	0	0.9	68%	92%
	No	0	0.7	50%	90%
Image only	No	-0.05	0.9	25%	81%
Image+gripper status+height	Yes	-0.05	0.9	67%	94%

Table 7: Simulation studies for tuning grasping task parameters

Figure 10: Performance graphs of simulation studies for tuning grasping task parameters

图 10：调抓取任务参数的仿真研究表现曲线

The results in Table 7 show that richer state representation results in faster convergence and better final performance. A small reward penalty does much better than decreasing the discount factor. Finally, giving the policy greater control over the episode termination yields performance on par with the engineered termination condition.

表 7 结果表明，更丰富的状态表示带来更快收敛与更好最终表现。加入夹爪状态与高度信息尤其有帮助。

We put special focus on the termination action: by letting the policy decide when to terminate, the policy has more potential to keep trying a grasp until it is confident the grasp is stable. We argue that explicitly learning a termination action would be beneficial for many manipulation tasks, since it makes the design of the MDP easier and it forces the policy to truly understand the goal of the task.

我们特别关注终止动作：让策略决定何时终止，回合长度可适应任务难度，并避免过早或过晚抬起。学习终止在仿真与真实中都优于固定启发式。

Data efficiency

Interestingly, our algorithm is more data efficient than the supervised learning based algorithm from  [27] work, achieving higher grasp success with fewer robots continuously generating training data (see Table 8).

数据效率

有趣的是，我们的算法比优化下一次抓取成功的监督基线更数据高效。用更少仿真机器人小时即可达到更高成功率。

Name	Sim Robots	Success
QT-Opt (ours)	30	88%
	60	95%
[27]	60	55%
	280	71%
	1000	85%

Table 8: Data efficiency comparison in simulation.

仿真数据效率比较如下。

Name	Sim Robots	Success
QT-Opt (ours)	30	88%
	60	95%
[27]	60	55%
	280	71%
	1000	85%

Table 8: Data efficiency comparison in simulation.

We argue that the algorithm from  [27] is less data efficient because it optimizes a proxy objective of 1-step classification accuracy, which values all data points equally. Our QT-Opt policy values data points based on how they influence reward. This focuses optimization on pivotal decision points that are very important to get right, such as learning when to close the gripper. Doing so lets the model optimize grasp success more efficiently.

我们认为 [27] 的算法数据效率较低，因为它优化的是代理目标，而非长时程抓取成功，因而需要更多样本才能学到可比较的闭环行为。

C.1 Effect of Off-Policy Training on Performance

C.1 离策略训练对表现的影响

Figure 11: (top row) The distribution of $z$-coordinate of the pose translation of actions selected by the randomized scripted policy. The $z$-coordinate values are biased towards negative values during the first few steps on the descent phase of the episode, then biased towards positive values at later steps on the ascend phase. Such a biased action distribution over states provides poor action space exploration and is insufficient to learn a good Q-Function. (bottom row) The $z$-coordinate of actions selected by a suboptimal QT-Opt policy. The distribution is roughly centered around zero (red axis) at each step providing good action space exploration.

图 11：（上行）所学策略所选动作位姿平移的 $z$ 坐标分布；（下行）相关分析见原文。

Num. transitions from $D_{scripted}$	Num. transitions from $D_{explore}$	Success
300k	0k	20%
0k	300k	37%
150k	150k	86%
200k	260k	94%

Table 9: Off-policy performance in simulation on datasets with different distributional properties of actions over states

离策略数据混合比例与成功率如下。

Num. transitions from $D_{scripted}$	Num. transitions from $D_{explore}$	Success
300k	0k	20%
0k	300k	37%
150k	150k	86%
200k	260k	94%

Table 9: Off-policy performance in simulation on datasets with different distributional properties of actions over states

In principle Q-Learning can learn an optimal policy from off-policy data, as long as the data is sufficiently diverse. Empirically we proved that indeed a decent policy can be trained from our offline dataset. We are interested in answering the following questions: what statistical properties must the data distribution have, and which exploration policies might give rise to that distribution? To that end, we collected two dataset in simulation. The $D_{scripted}$ dataset was collected by running a randomized scripted policy $\pi_{scripted}$, discussed in Appendix B, which averaged 30% grasp success. The second dataset $D_{explore}$ was collected by running a suboptimal QT-Opt policy $\pi_{eval}$ which is also averaged 30% grasp success. Table 9 shows off-policy performance on different amounts of data sampled from $D_{scripted}$ and $D_{explore}$. This experiment shows that given 300k transitions, it is impossible to learn solely from the $D_{scripted}$ data, or solely from the $D_{explore}$ data, but it is possible to learn with data from both distributions. This shows the importance of collecting off-policy data from a mix of policies with different behavior. The $\pi_{scripted}$ is a special randomized scripted policy which samples actions in a very biased way towards successful grasps for initial bootstrapping. As a result, it explores state-action space very poorly, yielding only specific greedy actions for certain states. In contrast, a suboptimal $\pi_{eval}$ makes a lot of suboptimal actions thus resulting in good exploration of the action space. Fig. 11 visualizes the distribution of the $z$-coordinate of the action translation in the two datasets, showing that $\pi_{eval}$’s actions have less downward bias. We saw similar differences in distributions over other components of the action, like the gripper action. In our real world experiments, early policies had very poor success, as the data was coming primarily from $\pi_{scripted}$, hence resulting in poor exploration of the action space. However, once we collected enough sufficiently random data, we started getting much better grasp success. We expect this pattern to hold for other real world tasks using QT-Opt: initial performance might be low, but continued data collection should eventually give rise to an appropriately diverse data distribution resulting in a good policy.

原则上，只要数据覆盖充分，Q-learning 可从离策略数据学会最优策略。实践中，我们观察到仅用脚本化数据也能学到可用策略，但混入探索策略数据并最终做在策略联合微调，会进一步抬高成功率。图 11 与表中结果表明，离策略预训练提供强初始化，在策略数据则纠正外推误差。对真实世界部署而言，这意味着可以先在历史日志上迭代算法，再花较少机器人时间做联合微调。

Appendix D Grasping MDP: State Space, Action Space, and Reward Evaluation

附录 D 抓取 MDP：状态空间、动作空间与奖励评估

The goal in designing the MDP is to provide a framework in which an agent may learn the necessary hand-eye coordination to reach, grasp, and lift objects successfully. Our sensors include a 640x512 RGB camera and joint position sensors in arm and gripper. In the following we describe representation of state, action and reward and discuss the stopping criterion of our grasping task.

设计 MDP 的目标是提供一个框架，使智能体可学习完成抓取任务所需的手眼协调与闭环行为，同时保持动作与观测足够通用，以便扩展到其他操作技能。

D.1 Observation Space

D.1 观测空间

To provide image observations $I_{t}$ that capture maximal information about the state of the environment, we mounted the camera to the shoulder of the robot overlooking the entire workspace (see Fig. 2). In practice, we also observed that our agent was able to learn more effectively when observations also include some proprioceptive state. Specifically, this state includes a binary open/closed indicator of gripper aperture and the scalar height of the gripper above the bottom of the tray. The full observation is be defined as $\mathbf{s}_{t}=(I_{t},g_{\text{aperture},t},g_{\text{height},t})$. The model-input $I_{t}$ is a 472x472 crop of the full-size image with random crop anchor. This way the model is discouraged from relying on a rigid camera-to-base transformation which cannot be assumed across several robots. Instead, the policy is forced to learn about the presence and the view of the gripper and impact of the actions in the environment, which is critical for the emergence of closed-loop self-corrective behaviours. We also apply image augmentation. The brightness, contrast, and saturation are adjusted by sampling uniformly from $[-0.125,0.125],[0.5,1.5]$, and $[0.5,1.5]$ respectively. To stay consistent, the same augmentation is applied to the images in the current state $\mathbf{s}$ and next state $\mathbf{s}^{\prime}$. Random cropping and image augmentation is only done at train time. At inference time, no augmentation is used and we always crop to the center 472x472 square of the image.

为提供尽可能多捕获环境状态信息的图像观测 $I_{t}$，我们使用肩上安装的单目 RGB 相机，分辨率为 472×472。图像经裁剪与归一化后输入网络。除图像外，状态还包括夹爪开闭的二元状态，以及夹爪相对地面的高度。这些本体感觉信号帮助策略区分「夹爪已闭合但未抓住」与「已抓住物体」等情形，并在抬起阶段判断高度是否足够。

D.2 Action Space

D.2 动作空间

The agent’s action comprises a gripper pose displacement, and an open/close command. The gripper pose displacement is a difference between the current pose and the desired pose in Cartesian space, encoded as translation $\mathbf{t}_{t}\in\mathbb{R}^{3}$, and vertical rotation encoded via a sine-cosine encoding $\mathbf{r}_{t}\in\mathbb{R}^{2}$. A gripper open/close command is encoded as one-hot vector $[g_{\text{close},t},g_{\text{open},t}]\in\{0,1\}^{2}$. In addition, as this is an episodic task with a finite horizon, our agent has to decide when to terminate. For a baseline policy, we implement a heuristic stopping criterion that triggers when the arm is holding an object above a threshold height. Such a heuristic is task-specific and might bias learning in a sub-optimal way. Thus, we introduce an additional action component $e_{t}$ which allows our policy to learn a stopping criterion and decide when to terminate autonomously. The full action is defined as $\mathbf{a}_{t}=(\mathbf{t}_{t},\mathbf{r}_{t},g_{\text{close},t},g_{\text{open},t},e_{t})$.

智能体的动作包含夹爪位姿位移与开/闭命令。夹爪位姿位移由笛卡尔平移 $\mathbf{t}\in\mathbb{R}^{3}$ 与方位角变化（正弦–余弦编码 $\mathbf{r}\in\mathbb{R}^{2}$）组成。另有二元命令 $g_{\text{open}}$、$g_{\text{close}}$ 以及终止命令 $e$。该动作空间允许策略在任意时刻开闭夹爪、微调位姿或结束回合，从而支持重抓与预抓等行为。

D.3 Reward Function

D.3 奖励函数

The agent is given a reward of 1 at the end of an episode if it has successfully grasped an object from the bin. In all other cases the reward is 0. In order to detect a successful grasp we use an image subtraction test, see Fig. 12. An image of the scene is captured after the grasp attempt with the arm moved out of camera view. Then a second image is taken after attempting to drop the grasped object into the bin. If no object was grasped, these two images would be identical. Otherwise, if an object was picked up, certain pixels in the two images will be different. A grasp is labeled success if the image subtraction is above a threshold.

若智能体在回合结束时成功从料箱抓起物体，则给予奖励 1，否则为 0。成功通过在把物体丢回料箱前后做图像减除测试判定。另对终止前每步施加小惩罚以鼓励更快完成。

Figure 12: Grasp success is determined by subtracting images before an object is dropped into the bin (left) and after it was dropped (right).

图 12：通过在物体丢入料箱前后做图像减除判定抓取是否成功。

Because we use a simple image subtraction test to compute the reward, in practice the labeling is not perfect. We analyzed the failed grasps from the model that reaches 96% grasp success, and discovered that out of 28 misgrasps, 3 were actually successful grasps that have been classified improperly by our image subtraction test. Our background subtraction test might not detect small objects, making it vulnerable to false negative registrations. This shows that our learning algorithm can train a very good policy even with a small noise in the reward function.

由于我们用简单的图像减除测试计算奖励，实践中标注并不完美：光照变化或背景干扰可能导致偶发误标。但在大规模数据下，该噪声可被学习过程吸收，且完全自动化标注使自监督扩展成为可能。

D.4 Grasp execution and termination condition

D.4 抓取执行与终止条件

1:  Pick a policy $policy$.
2:  Initialize a $robot$.
3:  while $step<N$ and not terminate_episode do
4:   $\mathbf{s}$ = $robot$.CaptureState()
5:   $\mathbf{a}$ = $policy$.SelectAction($\mathbf{s}$)
6:   $robot$.ExecuteCommand($\mathbf{a}$)
7:   terminate_episode = $e$ {Termination action $e$ is either learned or decided heuristically.}
8:   r = $robot$.ReceiveReward()
9:   emit($\mathbf{s},\mathbf{a},$ r)
10:  end while

Algorithm 1 Grasping control-loop

抓取控制循环伪代码如下（与原文一致）。

1:  Pick a policy $policy$.
2:  Initialize a $robot$.
3:  while $step<N$ and not terminate_episode do
4:   $\mathbf{s}$ = $robot$.CaptureState()
5:   $\mathbf{a}$ = $policy$.SelectAction($\mathbf{s}$)
6:   $robot$.ExecuteCommand($\mathbf{a}$)
7:   terminate_episode = $e$ {Termination action $e$ is either learned or decided heuristically.}
8:   r = $robot$.ReceiveReward()
9:   emit($\mathbf{s},\mathbf{a},$ r)
10:  end while

Algorithm 1 Grasping control-loop

With state, action and reward defined in the sections above, we now describe the control-loop executed on our robots. As described in Algorithm 1 we control our manipulator by closing a loop around visual input and gripper commands. At the end of an episode we detect grasp success and return the sequence of state-action pairs to be fed back into the learning pipeline or stored offline. Depending on the executed $policy$, our termination action $e$ is either learned or decided heuristically as we discuss below.

在上文定义状态、动作与奖励后，我们现在描述执行所学策略的控制循环。每一步，策略观察当前状态，经 CEM 最大化 Q 函数选择动作，机器人执行该动作，并接收下一状态与奖励。

Scripted termination condition

Initial experiments used a heuristic termination condition to determine when the policy is done. The heuristic detects when the gripper is holding an object, and the policy is continuing to command upward actions after it has passed a height threshold.

脚本化终止条件

初期实验使用启发式终止条件：当夹爪高于某阈值且判定为闭合抓住物体时结束回合。该条件任务特定，难以迁移。

1:  termination_height = 0.13
2:  s = robot.GetState()
3:  gripper_height = s.height_of_gripper
4:  gripper_closed = s.gripper_status=’CLOSED’
5:  a = robot.GetAction(s)
6:  next_action_height = a.translation.z
7:  if $gripper\_closed\ and\ gripper\_height>termination\_height\ and\ next\_action\_height>gripper\_height$ then
8:   terminate = True
9:  else
10:   terminate = False
11:  end if

Algorithm 2 Scripted termination condition

脚本化终止条件伪代码如下。

1:  termination_height = 0.13
2:  s = robot.GetState()
3:  gripper_height = s.height_of_gripper
4:  gripper_closed = s.gripper_status=’CLOSED’
5:  a = robot.GetAction(s)
6:  next_action_height = a.translation.z
7:  if $gripper\_closed\ and\ gripper\_height>termination\_height\ and\ next\_action\_height>gripper\_height$ then
8:   terminate = True
9:  else
10:   terminate = False
11:  end if

Algorithm 2 Scripted termination condition

Learned termination condition

In principle, a policy may use the camera image to determine if it has successfully grasped and raised an object. It may inform such knowledge through a discrete termination action. To learn the termination action, the reward structure must be slightly changed, $reward=1$ is assigned when $grasp\_success\ =True\ and\ gripper\_termination\_height>threshold$. This enforces the policy to indicate episode termination only after the gripper lifts the object, which makes it easier to provide visual feedback for robust grasps; objects will fall out of a brittle grasp during the lifting process, and terminating the grasp after the object falls gives richer negative examples.

学习终止条件

原则上，策略可利用相机图像判断是否已成功抓住并抬起物体，从而自行决定终止。我们把终止作为动作空间中的显式动作 $e$，让 Q 函数学习何时发出它。学习终止在真实与仿真中都优于脚本化启发式，且使同一算法框架更易迁移到其他任务。

Appendix E Q-Function Neural Network Architecture

附录 E Q 函数神经网络架构

Figure 13: The architecture of grasping Q-Function. The input image is processed by a stack of convolutional layers before the introduction of the action and vector-valued state variables (gripper status and height). These are processed by several fully connected layers, tiled over the width and height dimension of the convolutional map, and added to it. The resulting convolutional map is further processed by a number of convolutional layers until the output. The output is gated through a sigmoid, such that our Q-values are always in the range $[0,1]$.

图 13：抓取 Q 函数架构。输入图像由卷积层栈处理，动作与本体感觉特征在网络中部融合。

We model the Q-function as a large deep neural network whose architecture, shown in Figure 13, is inspired by the one presented in [27]. The network takes the monocular RGB image component of the state $\mathbf{s}$ as input, and processes it with 7 convolutional layers. We transform actions $\mathbf{a}$ and additional state features ($g_{\text{aperture}}$, $g_{\text{height}}$) with fully-connected layers, then merge them with visual features by broadcasted element-wise addition. After fusing state and action representations, the Q value $Q_{\theta}(\mathbf{s},\mathbf{a})$ is modeled by 9 more convolution layers followed by two fully-connected layers. Each convolution and fully-connected layer uses batch normalization [41] and the ReLU nonlinearity. The value function handles mixtures of discrete actions (open, close, terminate) and continuous actions (Cartesian vector and gripper rotation), making the same state-action space amenable to learning future tasks like placing and stacking. In Table 2, we demonstrate empirically that the model performs better when provided with redundant state representations ($g_{\text{aperture}}$, $g_{\text{height}}$). All weights are initialized with truncated normal random variables ($\sigma=.01$) and L2-regularized with a coefficient of $7\mathrm{e}{-5}$. Models are trained with SGD with momentum, using learning rate $0.0001$ and momentum $0.9$.

我们将 Q 函数建模为大型深度神经网络，其架构见图 13，受视觉与连续控制网络启发。图像经多层卷积与池化；动作向量、夹爪状态与高度在中部与视觉特征拼接，再经全连接层输出标量 Q 值。总参数约 120 万。该容量使模型能表示复杂视觉–动作依赖，同时仍可在分布式设定下训练。

Appendix F QT-Opt Distributed Reinforcement Learning System Design

附录 F QT-Opt 分布式强化学习系统设计

Figure 14: Architecture of the QT-Opt distributed reinforcement learning algorithm.

图 14：QT-Opt 分布式强化学习算法架构。

Prior works [42, 43] have explored new reinforcement learning algorithms that allow compute parallelization to accelerate training. We propose a parallel and distributed learning algorithm, inspired by [44], that can scale easily on cloud-based infrastructures. Fig. 14 is a diagram of the full distributed system.

先前工作探索了允许计算并行的新强化学习算法。我们的重点不同：在保持 QT-Opt 学习更新的同时，构建能处理高分辨率图像、超大回放与多机器人在线采集的基础设施。

This distributed design of our QT-Opt algorithm was important for several reasons:

QT-Opt 算法的这一分布式设计之所以重要，有若干原因：

(a) We use high-resolution images. Trying to store all transitions in the memory of a single machine is infeasible. Thus we employ a distributed replay buffer, which lets us store hundreds of thousands of transitions across several machines.
(b) The Q-network is quite large, and distributing training across multiple GPUs drastically increases research velocity by reducing time to convergence. Similarly, in order to support large scale simulated experiments, the design has to support running hundreds of simulated robots that cannot fit on a single machine.
(c) Decoupling training jobs from data generation jobs allows us to treat training as data-agnostic, making it easy to switch between simulated data, off-policy real data, and on-policy real data. It also lets us scale the speed of training and data generation independently.

(a) 我们使用高分辨率图像。试图把全部转移存进单机内存不可行，因此需要分布式回放与流式加载。

(b) Bellman 目标计算涉及 CEM，成本高于标准离散动作 Q-learning，适合用大量并行「Bellman 更新器」作业分担。

(c) 训练需要多达千万级梯度步，单机吞吐不足，需异步训练工作进程与参数服务器。

Similar to [45], our distributed replay buffer is shared across all agents, although we do not use prioritized replay. We distribute training across 10 GPUs, using asynchronous SGD with momentum as proposed by [46]. Models were trained using TensorFlow [47]. This system allows us to train the Q-function at 40 steps per second with a batch size of 32 across 10 NVIDIA P100 GPUs.

与先前分布式回放工作类似，我们的分布式回放缓冲在所有智能体间共享；但我们不使用优先回放，而是结合离线日志重放与在线采集，以最大化数据多样性与利用率。

F.1 Online Training

F.1 在线训练

Online agents (real or simulated robots) collect data from the environment. The policy used is the Polyak averaged weights $Q_{\bar{\theta}_{1}}(s,a)$ and the weights are updated every 10 minutes. That data is pushed to a distributed replay buffer, which we call the “online buffer”. The data is also persisted to disk for future offline training.

在线智能体（真实或仿真机器人）从环境收集数据。所用策略是经 CEM 由当前 Q 函数诱导的策略，可带探索噪声。转移被推入分布式回放，供 Bellman 更新器与训练工作进程消费。

F.2 Offline Training from Logs

F.2 从日志离线训练

To support offline training, we run a log replay job. This job reads data sequentially from disk for efficiency reasons. It replays saved episodes as if an online agent had collected that data. This lets us seamlessly merge off-policy data with on-policy data collected by online agents as described in Appendix F.3.

为支持离线训练，我们运行日志重放作业。该作业按序从磁盘读取数据，并把转移推入分布式回放缓冲，模拟在线流，同时可快进历史日志。

Offline data comes from all previously run experiments. In total, we collected 580k grasps of offline data, which took 4 terabytes of disk space. In fully off-policy training, we can train the policy by loading all data with the log replay job, letting us train without having to interact with the real environment.

离线数据来自此前全部实验。我们总共收集了 58 万次离线抓取。这些数据涵盖脚本化探索、早期所学策略与后期噪声探索等多个阶段，为离策略学习提供广阔覆盖。

Despite the scale of the distributed replay buffer, we still can’t fit the entire dataset into memory. In order to be able to visit each datapoint uniformly, we keep continuously running the Log Replay to refresh the in-memory data residing in the Replay Buffer. In practice we found that running at least 100 Log Replay jobs was important to mitigate correlations between consecutive episodes and drive sufficiently diverse data to the Replay Buffer.

尽管分布式回放缓冲规模很大，我们仍无法把整个数据集装入内存。因此采用环形缓冲与从磁盘持续重载相结合的策略，使训练在多遍数据上看到近似均匀的样本流。

F.3 Online Joint Finetuning After Offline Training

F.3 离线训练后的在线联合微调

In practice, we begin with off-policy training to initialize a good policy, and then switch to on-policy joint finetuning. To do so, we first train fully off-policy by using the Log Replay job to replay episodes from prior experiments. After training off-policy for enough time, we restart QT-Opt, training with a mix of on-policy and off-policy data. We train off-policy for 5M to 15M steps.

实践中，我们先用离策略训练初始化好策略，再切换到在策略联合微调：一边用当前策略在真实机器人上采集新数据，一边继续在混合批次上更新网络。

Real on-policy data is generated by 7 KUKA LBR IIWA robot arms where the weights of the policy $Q_{\bar{\theta}_{1}}(s,a)$ are updated every 10 minutes. Compared to the offline dataset, the rate of on-policy data production is much lower and the data has less visual diversity. However, the on-policy data also contains real-world interactions that illustrate the faults in the current policy. To avoid overfitting to the initially scarce on-policy data, we gradually ramp up the fraction of on-policy data from 1% to 50% over the first 1M gradient update steps of joint finetuning training.

真实在策略数据由 7 台 KUKA LBR IIWA 机械臂生成，策略权重 $Q_{\theta}$ 定期从参数服务器同步。新转移与离线数据混合，避免灾难性遗忘，同时提供难负例。

Since the real robots can stop unexpectedly (e.g., due to hardware faults), data collection can be sporadic, potentially with delays of hours or more if a fault occurs without any operator present. This can unexpectedly cause a significant reduction in the rate of data collection. To mitigate this, on-policy training is also gated by a training balancer, which enforces a fixed ratio between the number of joint finetuning gradient update steps and number of on-policy transitions collected. The ratio was defined relative to the speed of the GPUs and of the robots, which changed over time. We did not tune this ratio very carefully, and in practice the ratio ranged from 6:1 to 11:1.

由于真实机器人可能意外停止（例如硬件故障），数据采集可能间歇。分布式设计对此稳健：训练工作进程只要缓冲中仍有标注样本就可继续，机器人恢复后重新接入即可。

F.4 Distributed Bellman Update

F.4 分布式 Bellman 更新

To stabilize deep Q-Learning, we use a target network, as discussed in Section 4. Since target networks parameters typically lag behind the online network when computing TD error, the Bellman backup can actually be performed asynchronously in a separate process. We propose a novel algorithm that computes ${r(\mathbf{s},\mathbf{a})+\gamma V(\mathbf{s}^{\prime})}$ in parallel on separate CPU machines, storing the output of those computations in an additional buffer named “train” in our distributed replay buffer. We call this job the Bellman updater, and have been able to scale this computation up to 1,000 machines using around 14k cores.

为稳定深度 Q-learning，我们使用第 4 节讨论的目标网络。由于目标网络更新与 CEM 最大化成本高，我们把目标计算从训练循环中拆出，交给并行 Bellman 更新器作业。

Let $\theta$ be the parameters of the current Q-network, $\bar{\theta}$ be the parameters of the target Q-network, and $p(\mathbf{s},\mathbf{a},\mathbf{s})$ be the distribution of transitions in the replay buffer. As discussed in  4.1, our Bellman backup is formulated as:

令 $\theta$ 为当前 Q 网络参数，$\bar{\theta}$ 为目标网络参数。每个 Bellman 更新器加载目标权重，从回放采样转移，用 CEM 求 $\arg\max_{\mathbf{a}^{\prime}}Q_{\bar{\theta}}(\mathbf{s}^{\prime},\mathbf{a}^{\prime})$，写入带目标价值的训练样本。

$$\mathcal{E}(\theta)=\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})\sim p(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})}\left[D\left(Q_{\theta}(\mathbf{s},\mathbf{a}),r(\mathbf{s},\mathbf{a})+\gamma V(\mathbf{s}^{\prime})\right)\right].$$

Note that because we use several Bellman updater replicas, each replica will load a new target network at different times. All replicas push the Bellman backup to the shared replay buffer in the “train buffer”. This makes our target Q-values effectively generated by an ensemble of recent target networks, sampled from an implicit distribution $R_{t}$. Expanding the clipped Double DQN estimation of value gives the following objective:

注意：由于我们使用多个 Bellman 更新器副本，每个副本会按略不同的时间加载新目标网络，因此训练缓冲中的标签对应略滞后且彼此不完全同步的目标。实践中这仍稳定，并类似于多目标网络集成带来的额外平滑。

$$\mathcal{E}(\theta)=\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})\sim p(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})}\left[\mathbb{E}_{(\bar{\theta}_{1},\bar{\theta}_{2})\sim R_{t}(\bar{\theta}_{1},\bar{\theta}_{2})}\left[D\left(Q_{\theta}(\mathbf{s},\mathbf{a}),r(\mathbf{s},\mathbf{a})+\gamma V_{\bar{\theta}_{1},\bar{\theta}_{2}}(\mathbf{s}^{\prime})\right)\right]\right] \tag{2}$$

where $V_{\bar{\theta}_{1},\bar{\theta}_{2}}(\mathbf{s}^{\prime})$ is estimated using clipped Double DQN:

其中 $V_{\bar{\theta}_{1},\bar{\theta}_{2}}(\mathbf{s}^{\prime})$ 用裁剪 Double Q-learning 估计：

$$V_{\bar{\theta}_{1},\bar{\theta}_{2}}(\mathbf{s}^{\prime})=\min_{i=1,2}Q_{\bar{\theta}_{i}}(\mathbf{s}^{\prime},\arg\max_{\mathbf{a}^{\prime}}Q_{\bar{\theta}_{1}}(\mathbf{s}^{\prime},\mathbf{a}^{\prime}))$$

Even if a Q-function estimator is unbiased, variance in TD error estimates are converted into over-estimation bias during the Bellman backup (via the max operator). We hypothesize that ensembling via a distribution of lagging target networks stabilizes training by reducing the variance (and thereby reduces bias in target values). We also decrease overestimation bias by using Clipped Double-Q Learning and a damped discount factor of $\gamma=.9$. We found this training to be stable and reproducible. The target network $Q_{\bar{\theta}_{1}}$ is computed by doing a Polyak averaging  [32] with a decay factor of $0.9999$, while $Q_{\bar{\theta}_{2}}$ is lagged on average by about 6000 gradient steps. The exact amount of delay varies because of the asynchronous nature of the system.

即便 Q 函数估计器无偏，TD 误差估计中的方差也会转化为最大化步骤中的过估计。裁剪 Double Q-learning 通过对两个目标网络取 min 来缓解该问题。结合 Polyak 平均与交叉熵损失，我们在大规模离策略视觉数据上获得了稳定训练。

F.5 Distributed Replay Buffer

F.5 分布式回放缓冲

The distributed replay buffer supports having named replay buffers. We create three named buffers: “online buffer” holds online data, “offline buffer” holds offline data, and “train buffer” stores Q-targets computed by the Bellman updater. The replay buffer interface supports weighted sampling from the named buffers, which is useful when doing on-policy joint finetuning. The distributed replay buffer is spread over 6 workers, which each contain $50k$ transitions, totalling $6*50k=300k$ transitions. All buffers are FIFO buffers where old values are removed to make space for new ones if the buffer is full.

分布式回放缓冲支持命名回放缓冲。我们创建三个命名缓冲：「online」接收实时机器人数据，「offline」接收日志重放，「train」保存已由 Bellman 更新器标注的样本供训练工作进程消费。该分离使我们可以独立调节各数据流的采样比率与保留策略。