Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Dex-Net 2.0：用合成点云与解析抓取度量深度学习规划稳健抓取

Jeffrey Mahler 1 , Jacky Liang 1 , Sherdil Niyaz 1 , Michael Laskey 1 , Richard Doan 1 , Xinyu Liu 1 , Juan Aparicio Ojea 2 , and Ken Goldberg 1 Affiliation:  1 Dept. of EECS, University of California, Berkeley Email: {jmahler, jackyliang, sniyaz, laskeymd, rdoan, xinyuliu, goldberg}@berkeley.edu Affiliation:  2 Siemens Corporation, Corporate Technology Email: juan.aparicio@siemens.com

Abstract

摘要

To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table. We use the resulting dataset, Dex-Net 2.0, to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly predicts the probability of success of grasps from depth images, where grasps are specified as the planar position, angle, and depth of a gripper relative to an RGB-D sensor. Experiments with over 1,000 trials on an ABB YuMi comparing grasp planning methods on singulated objects suggest that a GQ-CNN trained with only synthetic data from Dex-Net 2.0 can be used to plan grasps in 0.8$s$ with a success rate of 93$\%$ on eight known objects with adversarial geometry and is 3$\times$ faster than registering point clouds to a precomputed dataset of objects and indexing grasps. The Dex-Net 2.0 grasp planner also has the highest success rate on a dataset of 10 novel rigid objects and achieves 99$\%$ precision (one false positive out of 69 grasps classified as robust) on a dataset of 40 novel household objects, some of which are articulated or deformable. Code, datasets, videos, and supplementary material are available at http://berkeleyautomation.github.io/dex-net.

为缩短稳健机器人抓取规划深度学习的数据采集时间，我们探索在合成数据集上训练：由 Dex-Net 1.0 中数千个 3D 模型在桌面上随机位姿生成 670 万点云、抓取与解析抓取度量。用所得数据集 Dex-Net 2.0 训练抓取质量卷积神经网络（GQ-CNN），从深度图快速预测抓取成功概率；抓取由相对 RGB-D 传感器的平面位置、角度与夹爪深度指定。在 ABB YuMi 上超过 1000 次孤立物体抓取试验中对比多种规划方法表明：仅用 Dex-Net 2.0 合成数据训练的 GQ-CNN 可在 0.8 秒内规划抓取，在八个几何对抗的已知物体上成功率 93%，且比把点云配准到预计算物体库再索引抓取快 3 倍。Dex-Net 2.0 抓取规划器在 10 个新刚体物体集上成功率最高，在 40 个新家用物体（部分可铰接或可变形）上达到 99% 精度（69 次判为稳健的抓取中仅 1 次假正）。代码、数据、视频与补充材料见 http://berkeleyautomation.github.io/dex-net。

I Introduction

I 引言

Reliable robotic grasping is challenging due to imprecision in sensing and actuation, which leads to uncertainty about properties such as object shape, pose, material properties, and mass. Recent results suggest that deep neural networks trained on large datasets of human grasp labels [31] or physical grasp outcomes [40] can be used to plan grasps that are successful across a wide variety of objects directly from images or point clouds, similar to generalization results in computer vision [28]. However, data collection requires either tedious human labeling [25] or months of execution time on a physical system [32].

可靠的机器人抓取很难：传感与执行不精确带来物体形状、位姿、材料与质量等不确定性。近期结果表明，在大量人类抓取标注 [31] 或物理抓取结果 [40] 上训练的深度网络，可直接从图像或点云规划跨多种物体成功的抓取，类似计算机视觉中的泛化 [28]。但采集要么需要繁琐人工标注 [25]，要么要在物理系统上跑数月 [32]。

An alternative approach is to plan grasps using physics-based analyses such as caging [46], grasp wrench space (GWS) analysis [44], robust GWS analysis [56], or simulation [25], which can be rapidly computed using Cloud Computing [27]. However, these methods assume a separate perception system that estimates properties such as object shape or pose either perfectly [44] or according to known Gaussian distributions [34]. This is prone to errors [2], may not generalize well to new objects, and can be slow to match point clouds to known models during execution [13]. In this paper we instead consider predicting grasp success directly from depth images by training a deep Convolutional Neural Network (CNN) on a massive dataset of parallel-jaw grasps, grasp metrics, and rendered point clouds generated using analytic models of robust grasping and image formation [18, 35], building upon recent research on classifying force closure grasps [15, 51] and the outcomes of dynamic grasping simulations [24, 25, 54].

另一路径是用基于物理的分析规划抓取，如笼络 [46]、抓取扳手空间（GWS）分析 [44]、稳健 GWS 分析 [56] 或仿真 [25]，并可借助云计算快速计算 [27]。但这些方法假设另有感知系统完美估计形状或位姿 [44]，或按已知高斯分布估计 [34]。这易出错 [2]、对新物体泛化差，执行时把点云匹配到已知模型也可能很慢 [13]。本文改为：用稳健抓取与成像的解析模型 [18, 35] 生成海量平行颚抓取、抓取度量与渲染点云，训练深度 CNN 直接从深度图预测抓取成功，承接力闭合抓取分类 [15, 51] 与动态抓取仿真结果分类 [24, 25, 54] 的近期研究。

Fig. 1: Dex-Net 2.0 Architecture. (Center) The Grasp Quality Convolutional Neural Network (GQ-CNN) is trained offline to predict the robustness candidate grasps from depth images using a dataset of 6.7 million synthetic point clouds, grasps, and associated robust grasp metrics computed with Dex-Net 1.0. (Left) When an object is presented to the robot, a depth camera returns a 3D point cloud, where pairs of antipodal points identify a set of several hundred grasp candidates. (Right) The GQ-CNN rapidly determines the most robust grasp candidate, which is executed with the ABB YuMi robot.

图 1：Dex-Net 2.0 架构。（中）抓取质量卷积神经网络（GQ-CNN）离线训练，用 Dex-Net 1.0 算出的 670 万合成点云、抓取及相关稳健抓取度量，从深度图预测候选抓取稳健性。（左）物体呈现给机器人时，深度相机返回 3D 点云，对跖点对识别出数百个抓取候选。（右）GQ-CNN 快速判定最稳健候选，由 ABB YuMi 执行。

Our primary contributions are: 1) the Dexterity Network (Dex-Net) 2.0, a dataset associating 6.7 million point clouds and analytic grasp quality metrics with parallel-jaw grasps planned using robust quasi-static GWS analysis on a dataset of 1,500 3D object models, 2) a Grasp Quality Convolutional Neural Network (GQ-CNN) model trained to classify robust grasps in depth images using expected epsilon quality as supervision, where each grasp is specified as a planar pose and depth relative to a camera, and 3) a grasp planning method that samples antipodal grasp candidates and ranks them with a GQ-CNN.

主要贡献：1）Dexterity Network（Dex-Net）2.0 数据集——在 1500 个 3D 物体模型上用稳健准静态 GWS 分析规划平行颚抓取，关联 670 万点云与解析抓取质量度量；2）GQ-CNN 模型——以期望 epsilon 质量为监督，在深度图中分类稳健抓取，每个抓取由相对相机的平面位姿与深度指定；3）采样对跖抓取候选并用 GQ-CNN 排序的抓取规划方法。

In over 1,000 physical trials of grasping single objects on a tabletop with an ABB YuMi robot, we compare Dex-Net 2.0 to image-based grasp heuristics, a random forest [51], an SVM [52], and a baseline that recognizes objects, registers their 3D pose [13], and indexes Dex-Net 1.0 [34] for the most robust grasp to execute. We find that the Dex-Net 2.0 grasp planner is 3$\times$ faster than the registration-based method, 93$\%$ successful on objects seen in training (the highest of learning-based methods), and is the best performing method on novel objects, achieving 99$\%$ precision on a dataset of 40 household objects despite being trained entirely on synthetic data.

在 ABB YuMi 桌面抓取单个物体超过 1000 次物理试验中，我们把 Dex-Net 2.0 与基于图像的抓取启发式、随机森林 [51]、SVM [52]，以及识别物体、配准 3D 位姿 [13] 并索引 Dex-Net 1.0 [34] 执行最稳健抓取的基线对比。发现 Dex-Net 2.0 规划器比基于配准的方法快 3 倍，在训练见过的物体上成功率 93%（学习方法中最高），在新物体上表现最好，在 40 个家用物体上达 99% 精度——尽管完全在合成数据上训练。

II Related Work

II 相关工作

Grasp Planning. Given an object and reachability constraints due to the environment, grasp planning considers finding a gripper configuration that maximizes a success (or quality) metric. Methods fall into one of two categories based on success criteria: analytic methods [44], which consider performance according to physical models such as the ability to resist external wrenches [42], and empirical (or data-driven) methods [4], which typically use human labels [2] or the ability to lift the object in physical trials [40].

抓取规划。 给定物体与环境可达约束，抓取规划考虑寻找最大化成功（或质量）度量的夹爪配置。按成功准则分两类：解析方法 [44] 按物理模型（如抵抗外扳手能力 [42]）衡量；经验（或数据驱动）方法 [4] 通常用人类标注 [2] 或物理试验中能否抬起物体 [40]。

Analytic Methods. Analytic approaches typically assume that object and contact locations are known exactly and consider either the ability to resist external wrenches [44] or the ability to constrain the object’s motion [46]. To execute grasps on a physical robot, a common approach is to precompute a database of known 3D objects labeled with grasps and quality metrics such as GraspIt! [14].Precomputed grasps are indexed using point cloud registration: matching point clouds to known 3D object models in the database using visual and geometric similarity [4, 5, 7, 13, 20, 22, 27] and executing the highest quality grasp for the estimated object instance.

解析方法。 解析方法通常假设物体与接触位置精确已知，考虑抵抗外扳手 [44] 或约束物体运动 [46] 的能力。要在物理机器人上执行，常见做法是预计算标有抓取与质量度量的已知 3D 物体库，如 GraspIt! [14]。预计算抓取通过点云配准索引：用视觉与几何相似性把点云匹配到库中已知 3D 模型 [4, 5, 7, 13, 20, 22, 27]，再执行估计物体实例的最高质量抓取。

Robust grasp planning (RGP) methods maximize grasp robustness, or the expected value of an analytic metric under uncertainty in sensing and control. This typically involves labeling grasps on a database 3D object models with robust metrics such as probability of force closure  [27] or the pose error robust metric [56] and using registration-based planning. Recent research has demonstrated that the sampling complexity of RGP can be improved using Multi-Armed Bandits [29] and datasets of prior 3D objects and robust grasps, such as the Dexterity Network (Dex-Net) 1.0 [34]. In this work we make a major extension to Dex-Net 1.0 by associating synthetic point clouds with robust grasps and training a Convolutional Neural Network to map point clouds and candidates grasps to estimated robustness.

稳健抓取规划（RGP）最大化抓取稳健性，即传感与控制不确定性下解析度量的期望。通常在 3D 物体模型库上用稳健度量（如力闭合概率 [27] 或位姿误差稳健度量 [56]）标注抓取，再做基于配准的规划。近期研究表明可用多臂老虎机 [29] 与先验 3D 物体及稳健抓取数据集（如 Dexterity Network（Dex-Net）1.0 [34]）改善 RGP 采样复杂度。本工作对 Dex-Net 1.0 做重大扩展：把合成点云与稳健抓取关联，并训练 CNN 把点云与候选抓取映射到估计稳健性。

Empirical Methods. Empirical approaches typically use machine learning to develop models that map from robotic sensor readings directly to success labels from humans or physical trials. Human labels have become popular due to empirical correlation with physical success [2], although they may be expensive to aquire for large datasets. Research in this area has largely focused on associating human labels with graspable regions in RGB-D images [31] or point clouds [9, 21, 25]. Lenz et al. [31] created a dataset of over 1k RGB-D images with human labels of successful and unsuccessful grasping regions, which has been used to train fast CNN-based detection models [45].

经验方法。 经验方法通常用机器学习建立从机器人传感器读数直接到人类或物理试验成功标签的模型。因与物理成功经验相关 [2]，人类标注很流行，但对大数据集获取可能昂贵。该领域研究多聚焦把人类标注与 RGB-D 图像 [31] 或点云 [9, 21, 25] 中的可抓区域关联。Lenz 等 [31] 创建含 1000 多张 RGB-D 图、带成功/失败抓取区域人类标注的数据集，已用于训练快速 CNN 检测模型 [45]。

Another line of research on empirical grasp planning has attempted to optimize success in physical trials directly. The time cost of generating samples on a physical robot led to the development active methods for acquiring grasping experiences such as Multi-Armed Bandits using Correlated Beta Processes [37] or Prior Confidence Bounds [39]. Recently Pinto and Gupta [40] scaled up data collection by recording over 40k grasping experiences on a Baxter and iteratively training CNNs to predict lifting successes or to resist grasp perturbations caused by an adversary [41]. Levine et al. [32] scaled up dataset collection even futher, collecting over 800k datapoints with a set of continuously running robotic arms and using deep learning to predict end effector poses. However, this required over 2 months of training across up to 14 robots.

经验抓取规划的另一线研究试图直接优化物理试验成功率。在物理机器人上生成样本的时间成本催生了用相关 Beta 过程多臂老虎机 [37] 或先验置信界 [39] 等主动获取抓取经验的方法。Pinto 与 Gupta [40] 在 Baxter 上记录超 4 万次抓取经验并迭代训练 CNN 预测抬起成功或抵抗对手扰动 [41]，从而扩大采集规模。Levine 等 [32] 进一步扩大：多条持续运行的机械臂采集超 80 万数据点，用深度学习预测末端位姿，但需最多 14 台机器人训练两个多月。

Computer Vision Techniques in Robot Grasping. Grasps for a physical robot are typically planned from images of target objects. Analytic grasp planning methods register images of rigid objects to a known database of 3D models, which typically involves segmentation, classification, and geometric pose estimation [7, 22, 49, 58] from 3D point cloud data in order to index precomputed grasps. However, this multi-stage approach can be prone to many hyperparameters that are difficult to tune and errors can compound across modules.

机器人抓取中的计算机视觉。 物理机器人抓取通常从目标物体图像规划。解析抓取规划把刚体物体图像配准到已知 3D 模型库，通常涉及从 3D 点云做分割、分类与几何位姿估计 [7, 22, 49, 58] 以索引预计算抓取。但多阶段方法超参数难调，误差会跨模块累积。

An alternative approach is to use deep learning to estimate 3D object shape and pose directly from color and depth images [17, 59]. Recent research in robotics has focused on how to improve accuracy in object recognition by structuring the way the neural network fuses the separate color and depth streams from images [43] and adding synthetic noise to synthetic training images [10].Another approach is to detect graspable regions directly in images without explicitly representing object shape and pose [31, 38, 40, 50], as it may not always be necessary to explicitly recognize objects and their pose to perform a grasp.

另一路径是用深度学习直接从彩色与深度图估计 3D 物体形状与位姿 [17, 59]。机器人学近期研究关注：如何通过结构化融合图像中分离的彩色与深度流提升物体识别精度 [43]，以及向合成训练图像加合成噪声 [10]。还有做法是不显式表示物体形状与位姿，直接在图像中检测可抓区域 [31, 38, 40, 50]，因为执行抓取未必总需要显式识别物体及其位姿。

Since training on real images may require significant data collection time, an alternative approach is to learn on simulated images [54, 55] and to adapt the representation to real data [53, 60]. Recent research suggests that in some cases it may be sufficient to train on datasets generated using perturbations to the parameters of the simulator [15, 48]. Notably, Johns et al. [24] used rendered depth images with simulated noise to train a CNN-based detector for planar grasps using dynamic simulations as supervision. We build upon these results by using robust analytic grasp metrics as supervision, using the gripper’s distance from the camera in predictions, and performing extensive evaluations on a physical robot.

由于在真实图像上训练可能需要大量采集时间，另一路径是在仿真图像上学习 [54, 55] 再把表示适配到真实数据 [53, 60]。近期研究表明，有时在仿真器参数扰动生成的数据集上训练就够 [15, 48]。值得注意的是，Johns 等 [24] 用带仿真噪声的渲染深度图，以动态仿真为监督训练面向平面抓取的 CNN 检测器。我们在这些结果上推进：用稳健解析抓取度量作监督，在预测中使用夹爪相对相机的距离，并在物理机器人上做广泛评估。

III Problem Statement

III 问题陈述

We consider the problem of planning a robust planar parallel-jaw grasp for a singulated rigid object resting on a table based on point clouds from a depth camera. We learn a function that takes as input a candidate grasp and a depth image and outputs an estimate of robustness [27, 56], or probability of success under uncertainty in sensing and control.

我们考虑：基于深度相机点云，为桌上孤立刚体规划稳健平面平行颚抓取。学习一个函数，输入候选抓取与深度图，输出稳健性估计 [27, 56]，即传感与控制不确定性下的成功概率。

III-A Assumptions

III-A 假设

We assume a parallel-jaw gripper, rigid objects singulated on a planar worksurface, and single-view (2.5D) point clouds taken with a depth camera. For generating datasets, we assume a known gripper geometry and a single overhead depth camera with known intrinsics.

假设平行颚夹爪、平面工作台上的孤立刚体，以及深度相机拍摄的单视角（2.5D）点云。生成数据集时，假设已知夹爪几何，以及内参已知的单台俯视深度相机。

III-B Definitions

III-B 定义

Fig. 2: Graphical model for robust parallel-jaw grasping of objects on a table surface based on point clouds. Blue nodes are variables included in the state representation. Object shapes $\mathcal{O}$ are uniformly distributed over a discrete set of object models and object poses $T_{o}$ are distributed over the object’s stable poses and a bounded region of a planar surface. Grasps $\mathbf{u}=(\mathbf{p},\varphi)$ are sampled uniformly from the object surface using antipodality constraints. Given the coefficient of friction $\gamma$ we evaluate an analytic success metric $S$ for a grasp on an object. A synthetic 2.5D point cloud $\mathbf{y}$ is generated from 3D meshes based on the camera pose $T_{c}$, object shape, and pose and corrupted with multiplicative and Gaussian Process noise.

图 2：基于点云的桌面物体稳健平行颚抓取图模型。蓝节点为状态表示中的变量。物体形状 $\mathcal{O}$ 在离散物体模型集上均匀分布；物体位姿 $T_{o}$ 分布在物体稳定位姿与平面有界区域内。抓取 $\mathbf{u}=(\mathbf{p},\varphi)$ 用对跖约束从物体表面均匀采样。给定摩擦系数 $\gamma$，评估抓取在物体上的解析成功度量 $S$。合成 2.5D 点云 $\mathbf{y}$ 由相机位姿 $T_{c}$、物体形状与位姿基于 3D 网格生成，并加乘性与高斯过程噪声。

States. Let $\mathbf{x}=(\mathcal{O},T_{o},T_{c},\gamma)$ denote a state describing the variable properties of the camera and objects in the environment, where $\mathcal{O}$ specifies the geometry and mass properties of an object, $T_{o},T_{c}$ are the 3D poses of the object and camera, respectively, and $\gamma\in\mathbb{R}$ is the coefficient of friction between the object and gripper.

状态。 令 $\mathbf{x}=(\mathcal{O},T_{o},T_{c},\gamma)$ 表示描述环境中相机与物体可变属性的状态，其中 $\mathcal{O}$ 指定物体几何与质量属性，$T_{o},T_{c}$ 分别为物体与相机的 3D 位姿，$\gamma\in\mathbb{R}$ 为物体与夹爪间的摩擦系数。

Grasps. Let $\mathbf{u}=(\mathbf{p},\varphi)\in\mathbb{R}^{3}\times\mathcal{S}^{1}$ denote a parallel-jaw grasp in 3D space specified by a center $\mathbf{p}=(x,y,z)\in\mathbb{R}^{3}$ relative to the camera and an angle in the table plane $\varphi\in\mathcal{S}^{1}$.

抓取。 令 $\mathbf{u}=(\mathbf{p},\varphi)\in\mathbb{R}^{3}\times\mathcal{S}^{1}$ 表示 3D 空间中的平行颚抓取，由相对相机的中心 $\mathbf{p}=(x,y,z)\in\mathbb{R}^{3}$ 与桌面平面内角度 $\varphi\in\mathcal{S}^{1}$ 指定。

Point Clouds. Let $\mathbf{y}=\mathbb{R}_{+}^{H\times W}$ be a 2.5D point cloud represented as a depth image with height $H$ and width $W$ taken by a camera with known intrinsics [18], and let $T_{c}$ be the 3D pose of the camera.

点云。 令 $\mathbf{y}=\mathbb{R}_{+}^{H\times W}$ 为内参已知相机拍摄的、表示为高 $H$ 宽 $W$ 深度图的 2.5D 点云 [18]，$T_{c}$ 为相机 3D 位姿。

Robust Analytic Grasp Metircs. Let $S(\mathbf{u},\mathbf{x})\in\{0,1\}$ be a binary-valued grasp success metric, such as force closure or physical lifting. Let $p(S,\mathbf{u},\mathbf{x},\mathbf{y})$ be a joint distribution on grasp success, grasps, states, and point clouds modeling imprecision in sensing and control. For example, $p$ could be defined by noisy sensor readings of a known set of industrial parts coming down a conveyor belt in arbitrary poses. Let the robustness of a grasp given an observation [5, 56] be the expected value of the metric, or probability of success under uncertainty in sensing and control: $Q(\mathbf{u},\mathbf{y})=\mathbb{E}\left[S\mid\mathbf{u},\mathbf{y}\right].$

稳健解析抓取度量。 令 $S(\mathbf{u},\mathbf{x})\in\{0,1\}$ 为二值抓取成功度量，如力闭合或物理抬起。令 $p(S,\mathbf{u},\mathbf{x},\mathbf{y})$ 为对抓取成功、抓取、状态与点云的联合分布，建模传感与控制不精确。例如，$p$ 可由传送带上任意位姿的已知工业零件的噪声传感器读数定义。给定观测的抓取稳健性 [5, 56] 为度量期望，即传感与控制不确定性下的成功概率：$Q(\mathbf{u},\mathbf{y})=\mathbb{E}\left[S\mid\mathbf{u},\mathbf{y}\right].$

Fig. 3: Dex-Net 2.0 pipeline for training dataset generation. (Left) The database contains 1,500 3D object mesh models. (Top) For each object, we sample hundreds of parallel-jaw grasps to cover the surface and evaluate robust analytic grasp metrics using sampling. For each stable pose of the object we associate a set of grasps that are perpendicular to the table and collision-free for a given gripper model. (Bottom) We also render point clouds of each object in each stable pose, with the planar object pose and camera pose sampled uniformly at random. Every grasp for a given stable pose is associated with a pixel location and orientation in the rendered image. (Right) Each image is rotated, translated, cropped, and scaled to align the grasp pixel location with the image center and the grasp axis with the middle row of the image, creating a $32\times 32$ grasp image. The full dataset contains over 6.7 million grasp images.

图 3：Dex-Net 2.0 训练数据集生成管线。（左）数据库含 1500 个 3D 物体网格模型。（上）对每个物体采样数百个平行颚抓取以覆盖表面，并用采样评估稳健解析抓取度量。对每个稳定位姿，关联垂直于桌面且对给定夹爪模型无碰撞的一组抓取。（下）还对每个稳定位姿渲染点云，平面物体位姿与相机位姿均匀随机采样。给定稳定位姿的每个抓取关联到渲染图像中的像素位置与朝向。（右）每张图经旋转、平移、裁剪与缩放，使抓取像素对齐图像中心、抓取轴对齐图像中间行，得到 $32\times 32$ 抓取图像。完整数据集含超过 670 万张抓取图像。

III-C Objective

III-C 目标

Our goal is to learn a robustness function $Q_{\theta^{*}}(\mathbf{u},\mathbf{y})\in[0,1]$ over many possible grasps, objects, and images that classifies grasps according to the binary success metric:

目标是在许多可能的抓取、物体与图像上学习稳健性函数 $Q_{\theta^{*}}(\mathbf{u},\mathbf{y})\in[0,1]$，按二值成功度量分类抓取：

$$\displaystyle\theta^{*}=\underset{\theta\in\Theta}{\text{argmin }}\mathbb{E}_{p(S,\mathbf{u},\mathbf{x},\mathbf{y})}\left[\mathcal{L}(S,Q_{\theta}(\mathbf{u},\mathbf{y}))\right] \tag{III.1}$$

where $\mathcal{L}$ is the cross-entropy loss function and $\Theta$ defines the parameters of the Grasp Quality Convolutional Network (GQ-CNN) described in Section IV-B. This objective is motivated by that fact that $Q_{\theta^{*}}=Q$ for all possible grasps and images when there exists some $\theta\in\Theta$ such that $Q_{\theta}=Q$ [36]. The estimated robustness function can be used in a grasping policy that maximizes $Q_{\theta^{*}}$ over a set of candidate grasps: $\pi_{\theta}(\mathbf{y})=\text{argmax}_{\mathbf{u}\in\mathcal{C}}Q_{\theta}(\mathbf{u},\mathbf{y})$, where $\mathcal{C}$ specifies constraints on the set of available grasps, such as collisions or kinematic feasibility. Learning $Q$ rather than directly learning the policy allows us to enforce task-specific constraints without having to update the learned model.

其中 $\mathcal{L}$ 为交叉熵损失，$\Theta$ 定义第 IV-B 节所述 GQ-CNN 的参数。该目标的动机是：当存在 $\theta\in\Theta$ 使 $Q_{\theta}=Q$ 时，对所有可能抓取与图像有 $Q_{\theta^{*}}=Q$ [36]。估计的稳健性函数可用于在候选抓取集上最大化 $Q_{\theta^{*}}$ 的抓取策略：$\pi_{\theta}(\mathbf{y})=\text{argmax}_{\mathbf{u}\in\mathcal{C}}Q_{\theta}(\mathbf{u},\mathbf{y})$，其中 $\mathcal{C}$ 指定可用抓取集上的约束，如碰撞或运动学可行。学习 $Q$ 而非直接学策略，使我们可在不必更新所学模型的情况下施加任务特定约束。

IV Learning a Grasp Robustness Function

IV 学习抓取稳健性函数

Solving for the grasp robustness function in objective B.1 is challenging for several reasons. First, we may need a huge number of samples to approximate the expectation over a large number of possible objects. We address this by generating Dex-Net 2.0, a training dataset of 6.7 million synthetic point clouds, parallel-jaw grasps, and robust analytic grasp metrics across 1,500 3D models sampled from the graphical model in Fig. 2. Second, the relationship between point clouds, grasps, and metrics over a large datset of objects may be complex and difficult to learn with linear or kernelized models. Consequently, we develop a Grasp Quality Convolutional Neural Network (GQ-CNN) model that classifies robust grasp poses in depth images and train the model on data from Dex-Net 2.0.

求解目标 B.1 中的抓取稳健性函数有几重困难。首先，在大量可能物体上近似期望可能需要海量样本。我们通过生成 Dex-Net 2.0 应对：按图 2 图模型从 1500 个 3D 模型采样，得到 670 万合成点云、平行颚抓取与稳健解析抓取度量的训练数据集。其次，跨大物体库的点云、抓取与度量关系可能复杂，线性或核化模型难学。因此我们开发 GQ-CNN，在深度图中分类稳健抓取位姿，并在 Dex-Net 2.0 数据上训练。

IV-A Dataset Generation

IV-A 数据集生成

We estimate $Q_{\theta^{*}}$ using a sample approximation [11] of the objective in Equation B.1 using i.i.d samples $(S_{1},\mathbf{u}_{1},\mathbf{x}_{1},\mathbf{y}_{1}),...,(S_{N},\mathbf{u}_{N},\mathbf{x}_{N},\mathbf{y}_{N})\sim p(S,\mathbf{u},\mathbf{x},\mathbf{y})$ from our generative graphical model for images, grasps, and success metrics:

我们用图像、抓取与成功度量的生成图模型的 i.i.d. 样本 $(S_{1},\mathbf{u}_{1},\mathbf{x}_{1},\mathbf{y}_{1}),...,(S_{N},\mathbf{u}_{N},\mathbf{x}_{N},\mathbf{y}_{N})\sim p(S,\mathbf{u},\mathbf{x},\mathbf{y})$，对公式 B.1 目标做样本近似 [11] 以估计 $Q_{\theta^{*}}$：

$$\displaystyle\hat{\theta}=\underset{\theta\in\Theta}{\text{argmin }}\sum\limits_{i=1}^{N}\mathcal{L}(S_{i},Q_{\theta}(\mathbf{u}_{i},\mathbf{y}_{i})).$$

IV-A1 Graphical Model

Our graphical model is illustrated in Fig. 2 and models $p(S,\mathbf{u},\mathbf{x},\mathbf{y})$ as the product of a state distribution $p(\mathbf{x})$, an observation model $p(\mathbf{y}|\mathbf{x})$, a grasp candidate model $p(\mathbf{u}|\mathbf{x})$, and an analytic model of grasp success $p(S|\mathbf{u},\mathbf{x})$.

IV-A1 图模型

我们的图模型见图 2，把 $p(S,\mathbf{u},\mathbf{x},\mathbf{y})$ 建模为状态分布 $p(\mathbf{x})$、观测模型 $p(\mathbf{y}|\mathbf{x})$、抓取候选模型 $p(\mathbf{u}|\mathbf{x})$ 与抓取成功解析模型 $p(S|\mathbf{u},\mathbf{x})$ 的乘积。

Distribution	Description
$p(\gamma)$	truncated Gaussian distribution over friction coefficients
$p(\mathcal{O})$	discrete uniform distribution over 3D object models
$p(T_{o}|\mathcal{O})$	continuous uniform distribution over the discrete set of object stable poses and planar poses on the table surface	continuous uniform distribution over the discrete set of	object stable poses and planar poses on the table surface
continuous uniform distribution over the discrete set of
object stable poses and planar poses on the table surface
$p(T_{c})$	continuous uniform distribution over spherical coordinates for radial bounds $[r_{\ell},r_{u}]$ and polar angle in $[0,\delta]$	continuous uniform distribution over spherical coordinates	for radial bounds $[r_{\ell},r_{u}]$ and polar angle in $[0,\delta]$
continuous uniform distribution over spherical coordinates
for radial bounds $[r_{\ell},r_{u}]$ and polar angle in $[0,\delta]$

continuous uniform distribution over the discrete set of
object stable poses and planar poses on the table surface

continuous uniform distribution over spherical coordinates
for radial bounds $[r_{\ell},r_{u}]$ and polar angle in $[0,\delta]$

TABLE I: Details of the distributions used in the Dex-Net 2.0 graphical model for generating the Dex-Net training dataset.

分布	描述
$p(\gamma)$	摩擦系数上的截断高斯分布
$p(\mathcal{O})$	3D 物体模型上的离散均匀分布
$p(T_{o}|\mathcal{O})$	物体稳定位姿离散集与桌面平面位姿上的连续均匀分布
$p(T_{c})$	径向界 $[r_{\ell},r_{u}]$、极角 $[0,\delta]$ 的球坐标上连续均匀分布

表 I：Dex-Net 2.0 图模型中用于生成训练数据集的分布细节。

We model the state distribution as

我们将状态分布建模为

$$\displaystyle p(\mathbf{x}) \\ \displaystyle=p(\gamma)p(\mathcal{O})p(T_{o}|\mathcal{O})p(T_{c})$$

where the distributions are detailed in Table I. Our grasp candidate model $p(\mathbf{u}\mid\mathbf{x})$ is a uniform distribution over pairs of antipodal contact points on the object surface that form a grasp axis parallel to the table plane. Our observation model is $\mathbf{y}=\alpha\hat{\mathbf{y}}+\epsilon$ where $\hat{\mathbf{y}}$ is a rendered depth image for a given object in a given pose, $\alpha$ is a Gamma random variable modeling depth-proportional noise, and $\epsilon$ is zero-mean Gaussian Process noise over pixel coordinates with bandwidth $\ell$ and measurement noise $\sigma$ modeling additive noise [35]. We model grasp success as:

各分布详见表 I。抓取候选模型 $p(\mathbf{u}\mid\mathbf{x})$ 是物体表面上形成平行于桌面的抓取轴的对跖接触点对上的均匀分布。观测模型为 $\mathbf{y}=\alpha\hat{\mathbf{y}}+\epsilon$，其中 $\hat{\mathbf{y}}$ 为给定物体在给定位姿下的渲染深度图，$\alpha$ 为建模与深度成正比噪声的 Gamma 随机变量，$\epsilon$ 为像素坐标上带宽 $\ell$、测量噪声 $\sigma$ 的零均值高斯过程噪声，建模加性噪声 [35]。我们将抓取成功建模为：

$$\displaystyle S(\mathbf{u},\mathbf{x})=\left{\begin{array}[]{cc}1&E_{Q}>\delta\text{ and }collfree(\mathbf{u},\mathbf{x})\
0&otherwise\end{array}\right.$$

where $E_{Q}$ is the robust epsilon quality defined in [51], a variant of the pose error robust metric [56] that includes uncertainty in friction and gripper pose, and $collfree(\mathbf{u},\mathbf{x})$ indicates that the gripper does not collide with the object or table. The supplemental file details the parameters of these distributions.

其中 $E_{Q}$ 为 [51] 定义的稳健 epsilon 质量——包含摩擦与夹爪位姿不确定性的位姿误差稳健度量 [56] 变体；$collfree(\mathbf{u},\mathbf{x})$ 表示夹爪不与物体或桌面碰撞。补充材料详述这些分布的参数。

Fig. 4: (Left) Architecture of the Grasp Quality Convolutional Neural Network (GQ-CNN). Planar grasp candidates $\mathbf{u}=(i,j,\varphi,z)$ are generated from a depth image and transformed to align the image with the grasp center pixel $(i,j)$ and orientation $\varphi$. The architecture contains four convolutional layers in pairs of two separated by ReLU nonlinearities followed by 3 fully connected layers and a separate input layer for the $z$, the distance of the gripper from the camera. The use of convolutional layers was motivated by the relevance of depth edges as features for learning in previous research [3, 31, 34] and the use of ReLUs was motivated by image classification results [28]. The network estimates the probability of grasp success (robustness) $Q_{\theta}\in[0,1]$, which can be used to rank grasp candidates. (Right) The first layer of convolutional filters learned by the GQ-CNN on Dex-Net 2.0. The filters appear to compute oriented image gradients at various scales, which may be useful for inferring contact normals and collisions between the gripper and object.

图 4：（左）抓取质量卷积神经网络（GQ-CNN）架构。平面抓取候选 $\mathbf{u}=(i,j,\varphi,z)$ 从深度图生成，并变换使图像与抓取中心像素 $(i,j)$ 及朝向 $\varphi$ 对齐。架构含两两成对的四层卷积（中间隔 ReLU），后接三层全连接，以及夹爪相对相机距离 $z$ 的单独输入层。使用卷积层的动机是先前研究中深度边缘作为学习特征的相关性 [3, 31, 34]；使用 ReLU 的动机是图像分类结果 [28]。网络估计抓取成功概率（稳健性）$Q_{\theta}\in[0,1]$，可用于排序候选。（右）在 Dex-Net 2.0 上 GQ-CNN 学到的第一层卷积滤波器。滤波器似乎在多尺度计算有向图像梯度，或有助于推断接触法向及夹爪与物体碰撞。

IV-A2 Database

Dex-Net 2.0 contains 6.7 million datapoints generated using the pipeline of Fig. 3.

IV-A2 数据库

Dex-Net 2.0 含用图 3 管线生成的 670 万数据点。

3D Models. The dataset contains a subset of 1,500 mesh models from Dex-Net 1.0: 1,371 synthetic models from 3DNet [57] and 129 laser scans from the KIT object database [26]. Each mesh is aligned to a standard frame of reference using the principal axes, rescaled to fit within a gripper width of $5.0cm$ (the opening width of an ABB YuMi gripper), and assigned a mass of $1.0kg$ centered in the object bounding box since some meshes are nonclosed. For each object we also compute a set of stable poses [12] and store all stable poses with probability of occurence above a threshold.

3D 模型。 数据集含 Dex-Net 1.0 中 1500 个网格子集：1371 个来自 3DNet [57] 的合成模型，129 个来自 KIT 物体库 [26] 的激光扫描。每个网格用主轴对齐到标准参考系，缩放到适合夹爪开口宽 5.0 cm（ABB YuMi 夹爪开口），并因部分网格非封闭而在包围盒中心赋质量 1.0 kg。对每个物体还计算稳定位姿集 [12]，并存储出现概率高于阈值的全部稳定位姿。

Parallel-Jaw Grasps. Each object is labeled with a set of up to 100 parallel-jaw grasps. The grasps are sampled using the rejection sampling method for antipodal point pairs developed in Dex-Net 1.0 [34] with constraints to ensure coverage of the object surface [33]. For each grasp we evaluate the expected epsilon quality $E_{Q}$ [42] under object pose, gripper pose, and friction coefficient uncertainty using Monte-Carlo sampling [51],

平行颚抓取。 每个物体标注最多 100 个平行颚抓取。抓取用 Dex-Net 1.0 [34] 的对跖点对拒绝采样，并加约束以保证覆盖物体表面 [33]。对每个抓取，在物体位姿、夹爪位姿与摩擦系数不确定性下用蒙特卡洛采样 [51] 评估期望 epsilon 质量 $E_{Q}$ [42]，

Rendered Point Clouds. Every object is also paired with a set of 2.5D point clouds (depth images) for each object stable pose, with camera poses and planar object poses sampled according to the graphical model described in Section IV-A1. Images are rendered using a pinhole camera model and perspective projection with known camera intrinsics, and each rendered image is centered on the object of interest using pixel transformations. Noise is added to the images during training as described in Section IV-B3.

渲染点云。 每个物体还与各稳定位姿的一组 2.5D 点云（深度图）配对，相机位姿与平面物体位姿按第 IV-A1 节图模型采样。图像用针孔相机模型与已知内参的透视投影渲染，并用像素变换把每张渲染图以感兴趣物体为中心。训练时按第 IV-B3 节向图像加噪声。

IV-B Grasp Quality Convolutional Neural Network

IV-B 抓取质量卷积神经网络

IV-B1 Architecture

The Grasp Quality Convolutional Neural Network (GQ-CNN) architecture, illustrated in Fig. 4 and detailed in the caption, defines the set of parameters $\Theta$ used to represent the grasp robustness function $Q_{\theta}$. The GQ-CNN takes as input the gripper depth from the camera $z$ and a depth image centered on the grasp center pixel $\mathbf{v}=(i,j)$ and aligned with the grasp axis orientation $\varphi$. The image-gripper alignment removes the need to learn rotational invariances that can be modeled by known, computationally-efficient image transformations (similar to spatial transformer networks [23]) and allows the network to evaluate any grasp orientation in the image rather than a predefined discrete set as in [24, 40]. Following standard preprocessing conventions, we normalize the input data by subtracting the mean and dividing by the standard deviation of the training data and then pass the image and gripper depth through the network to estimate grasp robustness. The GQ-CNN has approximately 18 million parameters.

IV-B1 架构

图 4 所示并在图注详述的 GQ-CNN 架构定义了表示抓取稳健性函数 $Q_{\theta}$ 的参数集 $\Theta$。GQ-CNN 输入为夹爪相对相机的深度 $z$，以及以抓取中心像素 $\mathbf{v}=(i,j)$ 为中心、与抓取轴朝向 $\varphi$ 对齐的深度图。图像–夹爪对齐去掉学习可用已知高效图像变换建模的旋转不变性的需要（类似空间变换网络 [23]），并允许网络评估图像中任意抓取朝向，而非 [24, 40] 中预定义离散集。按标准预处理约定，输入数据减训练均值、除训练标准差，再把图像与夹爪深度送入网络估计抓取稳健性。GQ-CNN 约有 1800 万参数。

IV-B2 Training Dataset

GQ-CNN training datasets are generated by associating grasps with a pixel $\mathbf{v}$, orientation $\varphi$, and depth $z$ relative to rendered depth images as illustrated in Fig. 3. We compute these parameters by transforming grasps into the camera frame of reference using the camera pose $T_{c}$ and projecting the 3D grasp position and orientation onto the imaging plane of the camera [18]. We then transform all pairs of images and grasp configurations to a single image centered on $\mathbf{v}$ and oriented along $\varphi$ (see the left panel of Fig. 4 for an illustration). The Dex-Net 2.0 training dataset contains 6.7 million datapoints and approximately $21.2\%$ positive examples for the thresholded robust epsilon quality with threshold $\delta=0.002$ [25] and a custom YuMi gripper.

IV-B2 训练数据集

GQ-CNN 训练数据集通过把抓取与相对渲染深度图的像素 $\mathbf{v}$、朝向 $\varphi$ 与深度 $z$ 关联生成，如图 3。我们用相机位姿 $T_{c}$ 把抓取变换到相机参考系，并把 3D 抓取位置与朝向投影到相机成像平面 [18] 来计算这些参数。再把所有图像与抓取配置对变换到以 $\mathbf{v}$ 为中心、沿 $\varphi$ 定向的单张图像（见图 4 左）。Dex-Net 2.0 训练数据集含 670 万数据点，对阈值 $\delta=0.002$ 的阈值化稳健 epsilon 质量 [25] 与定制 YuMi 夹爪，约 21.2% 为正例。

IV-B3 Optimization

We optimize the parameters of the GQ-CNN using backpropagation with stochastic gradient descent and momentum [28]. We initialize the weights of the model by sampling from a zero mean gaussian with variance $\frac{2}{n_{i}}$, where $n_{i}$ is the number of inputs to the $i$-th network layer [19]. To augment the dataset, we reflect the image about its vertical and horizontal axes and rotate each image by $180^{\circ}$ since these lead to equivalent grasps. We also adaptively sample image noise from our noise model (see Section IV-A1) before computing the batch gradient for new samples during training to model imaging noise without explicitly storing multiple versions of each image. To speed up noise sampling we approximate the Gaussian Process noise by upsampling an array of uncorrelated zero-mean Gaussian noise using bilinear interpolation. We set hyperparameters based on the performance on a randomize synthetic validation set as described in Section VI-C.

IV-B3 优化

我们用带动量的随机梯度下降反向传播 [28] 优化 GQ-CNN 参数。权重从零均值、方差 $\frac{2}{n_{i}}$ 的高斯采样初始化，其中 $n_{i}$ 为第 $i$ 层输入数 [19]。为增强数据，关于竖直与水平轴反射图像，并将每张图旋转 $180^{\circ}$，因这些对应等价抓取。训练时还在计算新样本批梯度前，按噪声模型（见第 IV-A1 节）自适应采样图像噪声，以建模成像噪声而不显式存储每图多版本。为加速噪声采样，用双线性插值上采样不相关零均值高斯噪声阵列来近似高斯过程噪声。超参数按第 VI-C 节所述在随机化合成验证集上的表现设定。

V Grasp Planning

V 抓取规划

The Dex-Net 2.0 grasp planner uses the robust grasping policy $\pi_{\theta}(\mathbf{y})=\text{argmax}_{\mathbf{u}\in\mathcal{C}}Q_{\theta}(\mathbf{u},\mathbf{y})$ illustrated in Fig. 1. The set $\mathcal{C}$ is a discrete set of antipodal candidate grasps [6] sampled uniformly at random in image space for surface normals defined by the depth image gradients. Each grasp candidate is evaluated by the GQ-CNN, and the most robust grasp that is (a) kinematically reachable and (b) not in collision with the table is executed. The supplemental file contains additional details.

Dex-Net 2.0 抓取规划器使用图 1 所示稳健抓取策略 $\pi_{\theta}(\mathbf{y})=\text{argmax}_{\mathbf{u}\in\mathcal{C}}Q_{\theta}(\mathbf{u},\mathbf{y})$。集合 $\mathcal{C}$ 是在图像空间中按深度图梯度定义的表面法向均匀随机采样的离散对跖候选抓取集 [6]。每个候选由 GQ-CNN 评估，执行（a）运动学可达且（b）不与桌面碰撞的最稳健抓取。补充材料含更多细节。

VI Experiments

VI 实验

We evaluated clasification performance on both real and synthetic data and performed extensive physical evaluations on an ABB YuMi with custom silicone gripper tips designed by Guo et al. [16] to benchmark the performance of grasping a single object. All experiments ran on a Desktop running Ubuntu 14.04 with a 2.7 GHz Intel Core i5-6400 Quad-Core CPU and an NVIDIA GeForce 980, and we used an NVIDIA GeForce GTX 1080 for training large models.

我们在真实与合成数据上评估分类性能，并在配备 Guo 等 [16] 设计的定制硅胶指尖的 ABB YuMi 上做广泛物理评估，以基准单个物体抓取表现。全部实验在 Ubuntu 14.04 台式机上运行，CPU 为 2.7 GHz Intel Core i5-6400 四核，GPU 为 NVIDIA GeForce 980；训练大模型另用 NVIDIA GeForce GTX 1080。

Fig. 5: (Left) The experimental platform for benchmarking grasping with the ABB YuMi. We registered the camera to the robot with a chessboard before each experiment. In each trial a human operator sampled an object pose by shaking the object in a box and placing it upside down in the workspace. We then took an RGB-D image with a Primsense Carmine 1.08, filled in the image using inpainting [24], segmented the object using color background subtraction, and formed a bounding box for the detected object. The grasp planner under evaluation then planned a gripper pose and the YuMi executed the grasp. Grasps were considered successful if the gripper held the object after lifting, transporting, and shaking the object. (Top-Right) The training set of 8 objects with adversarial geometric features such as smooth curved surfaces and narrow openings for grasping known objects. (Bottom-Right) The test set of 10 household objects not seen during training.

图 5：（左）ABB YuMi 抓取基准实验平台。每次实验前用棋盘把相机配准到机器人。每轮试验由人工操作员把物体在盒中摇动后倒置放入工作区以采样物体位姿。随后用 Primsense Carmine 1.08 拍 RGB-D 图，用修复 [24] 填补图像，用颜色背景减除分割物体，并为检测到的物体形成包围盒。待评抓取规划器规划夹爪位姿，YuMi 执行抓取。若抬起、搬运并摇动后夹爪仍握住物体，则视为成功。（右上）训练用的 8 个具有光滑曲面、狭窄开口等对抗几何特征的物体，用于已知物体评估。（右下）训练未见的 10 个家用测试物体。

VI-A Physical Benchmark Description

VI-A 物理基准描述

We created a benchmark for grasping single objects on a tabletop to compare grasp planning methods. The setup is illustrated in Fig. 5 and the experimental procedure is described in the caption and shown in the supplemental video[^1]. Each grasp planner received as input a color image, depth image, bounding box containing the object, and camera intrinsics, and output a target grasping pose for the gripper. A human operator was required to reset the object in the workspace on each trial, and therefore blinded operators from which grasp planning method was being tested in order to remove bias.

我们创建桌面单物体抓取基准以对比抓取规划方法。设置见图 5，实验流程见图注并在补充视频[^1]展示。每个抓取规划器输入彩色图、深度图、含物体的包围盒与相机内参，输出夹爪目标抓取位姿。每轮试验需人工重置物体，因此对操作员盲测所用规划方法以去除偏差。

We compared performance on this benchmark with the following metrics:

我们在该基准上用下列指标对比性能：

1. Success Rate: The percentage of grasps that were able to lift, transport, and hold a desired object after shaking.

1. 成功率： 能抬起、搬运并在摇动后握住目标物体的抓取百分比。

2. Precision: The success rate on grasps that are have an estimated robustness higher than 50$\%$. This measures performance when the robot can decide not to grasp an object, which could be useful when the robot has other actions (e.g. pushing) available.

2. 精度： 估计稳健性高于 50% 的抓取上的成功率。度量机器人可选择不抓时的表现，在另有动作（如推）可用时有用。

3. Robust Grasp Rate: The percentage of planned grasps with an estimated robustness higher than 50$\%$.

3. 稳健抓取率： 估计稳健性高于 50% 的已规划抓取百分比。

4. Planning Time: The time in seconds between receiving an image and returning a planned grasp.

4. 规划时间： 收到图像到返回已规划抓取的秒数。

VI-B Datasets

VI-B 数据集

Fig. 5 illustrates the physical object datasets used in the benchmark:

图 5 展示基准所用物理物体数据集：

1. Train: A validation set of 8 3D-printed objects with adversarial geometric features such as smooth, curved surfaces. This is used to set model parameters and to evaluate performance on known objects.

1. Train： 8 个具有光滑曲面等对抗几何特征的 3D 打印物体验证集，用于设定模型参数并评估已知物体表现。

2. Test: A set of 10 household objects similar to models in Dex-Net 2.0 with various material, geometric, and specular properties. This is used to evaluate generalization to unknown objects.

2. Test： 10 个与 Dex-Net 2.0 模型相似、材料/几何/镜面属性各异的家用物体，用于评估对未知物体的泛化。

We chose objects based on geometric features under three constraints: (a) small enough to fit within the workspace, (b) weight less than 0.25$kg$, the payload of the YuMi, and (c) height from the table greater than 1.0$cm$ due to a limitation of the silicone gripper fingertips.

选物基于几何特征并受三重约束：(a) 足够小以放入工作区；(b) 重量小于 0.25 kg（YuMi 载荷）；(c) 相对桌面高度大于 1.0 cm（硅胶指尖限制）。

We used four different GQ-CNN training datasets to study the effect on performance, each with a 80-20 image-wise training and validation split:

我们用四个不同 GQ-CNN 训练数据集研究对性能的影响，各按图像做 80-20 训练/验证划分：

1. Adv-Synth: Synthetic images and grasps for the adversarial objects in Train (189k datapoints).

1. Adv-Synth： Train 中对抗物体的合成图像与抓取（18.9 万数据点）。

2. Adv-Phys: Outcomes of executing random antipodal grasps with random gripper height and friction coefficient of $\mu=0.5$ in 50 physical trials per object in Train (400 datapoints).

2. Adv-Phys： 在 Train 每个物体上以随机夹爪高度、$\mu=0.5$ 摩擦系数执行随机对跖抓取的结果（每物 50 次物理试验，共 400 数据点）。

3. Dex-Net-Small: A subset of data from 150 models sampled uniformly from Dex-Net 2.0 (670k datapoints).

3. Dex-Net-Small： 从 Dex-Net 2.0 均匀采样的 150 个模型数据子集（67 万数据点）。

4. Dex-Net-Large: Data from all 1500 models in Dex-Net 2.0 (6.7m datapoints).

4. Dex-Net-Large： Dex-Net 2.0 全部 1500 个模型的数据（670 万数据点）。

Fig. 7: Receiver operating characteristic comparing the performance of learning models on Adv-Synth. The GQ-CNN models all perform similarly and have a significantly higher true positive rate when compared to ML-RF and ML-SVM. Model Accuracy ($\%$) ML-SVM 89.7 ML-RF 90.5 GQ-S-Adv 97.8 GQ-L-Adv 97.8 GQ-Adv 98.1 TABLE IV: The classification accuracy of each model on Adv-Synth. We see that the GQ-CNN methods have less than 2.5$\%$ test error while ML-RF and ML-SVM are closer to 10$\%$ error. Pretraining does not appear to affect performance.

Fig. 7: Receiver operating characteristic comparing the performance of learning models on Adv-Synth. The GQ-CNN models all perform similarly and have a significantly higher true positive rate when compared to ML-RF and ML-SVM. Model Accuracy ($\%$) ML-SVM 89.7 ML-RF 90.5 GQ-S-Adv 97.8 GQ-L-Adv 97.8 GQ-Adv 98.1 TABLE IV: The classification accuracy of each model on Adv-Synth. We see that the GQ-CNN methods have less than 2.5$\%$ test error while ML-RF and ML-SVM are closer to 10$\%$ error. Pretraining does not appear to affect performance.

图 7：各学习模型在 Adv-Synth 上的 ROC 对比。GQ-CNN 模型表现相近，相对 ML-RF 与 ML-SVM 真阳性率显著更高。表 IV：各模型在 Adv-Synth 上的分类精度。GQ-CNN 方法测试误差低于 2.5%，而 ML-RF 与 ML-SVM 接近 10%。预训练似乎不影响性能。模型精度（%）：ML-SVM 89.7；ML-RF 90.5；GQ-S-Adv 97.8；GQ-L-Adv 97.8；GQ-Adv 98.1。

	Comparions of Methods	Comparions of Methods	GQ-CNN Parameter Sensitivity	GQ-CNN Parameter Sensitivity
Comparions of Methods
GQ-CNN Parameter Sensitivity
	Random	Random	IGQ	IGQ	ML-RF	ML-RF	ML-SVM	ML-SVM	REG	REG	GQ-L-Adv	GQ-L-Adv	GQ-S-Adv	GQ-S-Adv	GQ-Adv	GQ-Adv	GQ-Adv-Phys	GQ-Adv-Phys	GQ-Adv-FC	GQ-Adv-FC	GQ-Adv-LowU	GQ-Adv-LowU	GQ-Adv-HighU	GQ-Adv-HighU
Random
IGQ
ML-RF
ML-SVM
REG
GQ-L-Adv
GQ-S-Adv
GQ-Adv
GQ-Adv-Phys
GQ-Adv-FC
GQ-Adv-LowU
GQ-Adv-HighU
Success Rate ($\%$)	58$\pm$11	70$\pm$10	75$\pm$9	80$\pm$9	95$\pm$5	93$\pm$6	85$\pm$8	83$\pm$8	80$\pm$9	83$\pm$8	78$\pm$9	86$\pm$8
Precision ($\%$)	N/A	N/A	100	100	N/A	94	90	91	80	89	90	92
Robust Grasp Rate ($\%$)	N/A	N/A	5	0	N/A	43	60	44	100	89	53	64
Planning Time (sec)	N/A	1.9	0.8	0.9	2.6	0.8	0.9	0.8	0.8	0.7	0.8	0.9

Comparions of Methods

GQ-CNN Parameter Sensitivity

Random

IGQ

ML-RF

ML-SVM

REG

GQ-L-Adv

GQ-S-Adv

GQ-Adv

GQ-Adv-Phys

GQ-Adv-FC

GQ-Adv-LowU

GQ-Adv-HighU

TABLE V: Performance of grasp planning methods on the Train dataset with 95$\%$ confidence intervals for the success rate. Each method was tested for 80 trials (10 trials per object). Details on the methods used for comparison can be found in Section VI-C. We see that REG (point cloud registration) has the highest success rate at 95$\%$ but the GQ-L-Adv performs comparably at 93$\%$ and is 3$\times$ faster. Performance of the GQ-CNN drops to 80$\%$ when trained on the Adv-Phys dataset (GQ-Adv-Phys), likely due to the small number of training examples, and drops to 78$\%$ when no noise is added to the images during training (GQ-Adv-LowU).

表 V：抓取规划方法在 Train 对抗物体集上的性能对比，以及 GQ-CNN 对数据集规模、预训练与成功度量等参数的敏感性。成功率给出 95% 置信区间。在全部 Dex-Net 2.0 上训练的 GQ（GQ-L）在已知物体上达到最高成功率 93%，规划约 0.8 秒，比基于配准的方法快约 3 倍。详见第 VI-C 节方法说明。

VI-C Grasp Planning Methods Used for Comparison

VI-C 用于对比的抓取规划方法

We compared a number of grasp planning methods on simulated and real data. We tuned the parameters of each method based on synthetic classification performance and physical performance on the training objects. All methods other than point cloud registration used the antipodal grasp sampling method described in Section V with the same set of parameters to generate candidate grasps, and each planner executes the highest-ranked grasp according to the method. Additional details on the methods and their parameters can be found in the supplemental file.

我们在仿真与真实数据上对比多种抓取规划方法，以评估架构、数据集与训练程序的影响。

Image-based Grasp Quality Metrics (IGQ). We sampled a set of force closure grasp candidates by finding antipodal points on the object boundary [6] using edge detection and ranked grasps by the distance from the center of the jaws to the centroid of the object segmentation mask. We set the gripper depth using a fixed offset from the depth of the grasp center pixel.

基于图像的抓取质量度量（IGQ）。 我们采样一组力闭合对跖抓取，并用深度图梯度估计接触法向来评估图像空间中的解析度量，作为快速启发式。

Point-Cloud Registration (REG). We also compared with grasp planning based on point cloud registration, a state-of-the-art method for using precomputed grasps [13, 20]. We first coarsely estimated the object instance and pose based on the top 3 most similar synthetic images from Dex-Net 2.0, where similarity is measured as distance between AlexNet conv5 features [13, 34]. After coarse matching, we finetuned the pose of the object in the table plane using Iterated Closest Point [17, 27] with a point-to-plane cost. Finally, we retrieved the most robust gripper pose from Dex-Net 2.0 for the estimated object. The system had a median translational error of 4.5$mm$ a median rotational error of 3.5*∘* in the table plane for known objects.

点云配准（REG）。 我们还与基于把观测点云配准到已知 3D 模型数据库并索引预计算稳健抓取的规划对比 [13, 34]。该方法对已知物体可很准，但配准慢，且对新物体可能失败。

Alternative Machine Learning Models (ML). We also compared the performance of a Random Forest with 200 trees of depth up to 10 (ML-RF) motivated by the results of [51] and a Support Vector Machine with the RBF kernel and a regularization parameter of 1 (ML-SVM) motivated by the results of [3, 50, 52]. For the RF we used the raw transformed images and gripper depths normalized by the mean and standard deviation across all pixels as features. For the SVM we used a Histogram of Oriented Gradients (HOG) [8] feature representation. Both methods were trained using scikit-learn on the Adv-Synth dataset.

替代机器学习模型（ML）。 我们还对比随机森林（ML-RF）与支持向量机（ML-SVM）在与 GQ-CNN 相同特征上的表现，特征包括抓取图像梯度统计等。

Grasp Quality CNNs (GQ). We trained the GQ-CNN (abbrev. GQ) using the thresholded robust epsilon metric with $\delta=0.002$ [25] for 5 epochs on Dex-Net-Large (all of Dex-Net 2.0) using Gaussian process image noise with standard deviation $\sigma=0.005$. We used TensorFlow [1] with a batch size of $128$, a momentum term of $0.9$, and an exponentially decaying learning rate with step size $0.95$. Training took approximately 48 hours on an NVIDIA GeForce 1080. The first layer of $7\times 7$ convolution filters are shown in the right panel of Fig. 4, and suggest that the network learned finegrained vertical edge detectors and coarse oriented gradients. We hypothesize that vertical filters help to detect antipodal contact normals and the coarse oriented gradients estimate collisions.

抓取质量 CNN（GQ）。 我们用第 IV-B 节所述程序在不同数据集上训练 GQ-CNN（简称 GQ）。除非另有说明，默认 GQ 指在 Dex-Net-Large 上训练的模型。

To benchmark the architecture outside of our datasets, we trained on the Cornell Grasping Dataset [31] (containing 8,019 examples) and achieved a $93.0\%$ recognition rate using grayscale images and an $80-20$ imagewise training-validation split compared to $93.7\%$ on RGB-D images in the original paper. We also trained several variants to evaluate sensitivity to several parameters:

为在我们的数据集之外基准架构，我们在 Cornell 抓取数据集上训练，并与先前报告结果对比。

Dataset Size. We trained a GQ-CNN on Dex-Net-Small for 15 epochs (GQ-S).

数据集规模。 我们在 Dex-Net-Small 上训练 GQ-CNN 15 个 epoch（GQ-S），以研究更少合成数据的影响。

Amount of Pretraining We trained three GQ-CNNs on the synthetic dataset of adversarial training objects (Adv-Synth) to study the effect of pretraining with Dex-Net for a new, known set of objects. The model GQ-Adv was trained on only Adv-Synth for 25 epochs. The models GQ-L-Adv and GQ-S-Adv were initialized with the weights of GQ and GQ-S, respectively, and finetuned for 5 epochs on Adv-Synth.

预训练量。 我们在合成数据上训练三个 GQ-CNN，并在 Adv-Phys 物理数据上微调不同轮数，以研究从仿真到现实的迁移。

Success Metric. We trained a GQ-CNN using probability of force closure thresholded at 25$\%$ (GQ-Adv-FC), which is a robust version of the antipodality metric of [15, 52], and and labels for 400 random grasp attempts on the Train objects using a physical robot [32, 40] (GQ-Adv-Phys).

成功度量。 我们用阈值为 0.5 的力闭合概率作为成功度量训练 GQ-CNN，以与稳健 epsilon 质量对比。

Noise Levels. We trained a GQ-CNN with zero noise $\sigma=0$ (GQ-Adv-LowU) and high noise with $\sigma=0.01$ (GQ-Adv-HighU).

噪声水平。 我们以零噪声 $\sigma=0$（GQ-Adv-Zero）训练 GQ-CNN，以研究训练时图像噪声的影响。

VI-D Classification of Synthetic Data

VI-D 合成数据分类

The GQ-CNN trained on all of Dex-Net 2.0 had an accuracy of 85.7$\%$ on a held out validation set of approximately 1.3 million datapoints. Due to the memory and time requirements of training SVMs, we compared synthetic classification performance across methods on the smaller Adv-Synth dataset. Fig. 7 shows the receiver operating characteristic curve comparing the performance of GQ-L-Adv, GQ-S-Adv, GQ-Adv, ML-SVM, and ML-RF on a held-out validation set and Table 7 details the classification accuracy for the various methods. The GQ-CNNs outperformed ML-RF and ML-SVM, achieving near-perfect validation accuracy.

在全部 Dex-Net 2.0 上训练的 GQ-CNN 在留出合成测试集上精度 85.7%。图 7 与表 IV 显示，在 Adv-Synth 上 GQ-CNN 方法精度约 98%，显著高于 ML-RF 与 ML-SVM。

VI-E Performance Comparison on Known Objects

VI-E 已知物体上的性能对比

We evaluated the performance of the grasp planning methods on known objects from Train. Each grasp planner had 80 trials (10 per object). The left half of Table V compares the performance with other grasp planning methods and the right half compares the performance of the GQ-CNN varations. We found that GQ planned grasps 3$\times$ faster than REG and achieved a high 93$\%$ success rate and 94$\%$ precision. The results also suggest that training on the full Dex-Net 2.0 dataset was necessary to achieve higher than $90\%$ success.

我们在 Train 的已知对抗物体上评估各抓取规划方法。表 V 详述结果。在 Dex-Net-Large 上训练的 GQ 达到最高成功率 93%，规划约 0.8 秒，比 REG 快约 3 倍。较小合成数据集或仅物理数据训练会降低性能。

VI-F Performance Comparison on Novel Objects

VI-F 新物体上的性能对比

We also compared the performance of the methods on the ten novel test objects from Test to evaluate generalization to novel objects. Each method was run for 50 trials (5 per object). The parameters of each method were set based on Train object performance without knowledge of the performance on Test. Table VI details the results. GQ performed best with an 80$\%$ success rate and $100\%$ precision (zero false positives over $29$ grasps classified as robust).

我们还在 Test 的十个新测试物体上对比各方法以评估对新物体的泛化。每方法运行 50 次试验（每物 5 次）。各方法参数按 Train 物体表现设定，不看 Test 表现。表 VI 详述结果。GQ 最好：成功率 80%，精度 100%（29 次判为稳健的抓取中零假正）。

	IGQ	IGQ	REG	REG	GQ-Adv-Phys	GQ-Adv-Phys	GQ-Adv	GQ-Adv	GQ-S	GQ-S	GQ	GQ
IGQ
REG
GQ-Adv-Phys
GQ-Adv
GQ-S
GQ
Success Rate ($\%$)	60$\pm$13	52$\pm$14	68$\pm$13	74$\pm$12	72$\pm$12	80$\pm$11
Precision ($\%$)	N/A	N/A	68	87	92	100
Robust Grasp Rate ($\%$)	N/A	N/A	100	30	48	58
Planning Time (sec)	1.8	3.4	0.7	0.7	0.8	0.8

IGQ

REG

GQ-Adv-Phys

GQ-Adv

GQ-S

GQ

TABLE VI: Performance of grasp planning methods on our grasping benchmark with the test dataset of 10 household objects with 95$\%$ confidence intervals for the success rate. Each method was tested for 50 trials, and details on the methods used for comparison can be found in Section VI-C. GQ performs best in terms of success rate and precision, with 100$\%$ precision (zero false positives among $29$ positive classifications). Performance decreases with smaller training datasets, but the GQ-CNN methods outperform the image-based grasp quality metrics (IGQ) and point cloud registration (REG).

方法	成功率 (%)	精度 (%)	稳健抓取率 (%)	规划时间 (秒)
IGQ	60±13	N/A	N/A	1.8
REG	52±14	N/A	N/A	3.4
GQ-Adv-Phys	68±13	68	100	0.7
GQ-Adv	74±12	87	30	0.7
GQ-S	72±12	92	48	0.8
GQ	80±11	100	58	0.8

表 VI：在 10 个家用测试物体抓取基准上的性能，成功率含 95% 置信区间。每方法 50 次试验；方法细节见第 VI-C 节。GQ 在成功率与精度上最好，精度 100%（29 次正分类中零假正）。训练数据变少时性能下降，但 GQ-CNN 方法仍优于基于图像的抓取质量度量（IGQ）与点云配准（REG）。

Fig. 8: (Left) The test set of 40 household objects used for evaluating the generalization performance of the Dex-Net 2.0 grasp planner. The dataset contains rigid, articulated, and deformable objects. (Right) The experimental setup for order fulfillment with the ABB YuMi. The goal is to grasp and transport three target objects to a shipping container (box on right).

图 8：（左）用于评估 Dex-Net 2.0 抓取规划器泛化的 40 个家用物体测试集，含刚体、可铰接与可变形物体。（右）ABB YuMi 订单履约实验设置。目标是把三个目标物体抓取并运到运输容器（右侧盒）。

VI-G Generalization Ability of the Dex-Net 2.0 Grasp Planner

VI-G Dex-Net 2.0 抓取规划器的泛化能力

We evaluated the generalization performance of GQ in 100 grasping trials on the 40 object test set illustrated in Fig. 8, which contains articulated (e.g. can opener) and deformable (e.g. washcloth) objects. We used the cross entropy method (CEM) [32], which iteratively samples a set of candidate grasps and re-fits the candidate grasp distribution to the grasps with the highest predicted robustness, in order to find better maxima of the robust grasping policy. More details can be found in the supplemental file. The CEM-augmented Dex-Net 2.0 grasp planner achieved 94$\%$ success and 99$\%$ precision (68 successes out of 69 grasps classified as robust), and it took an average of 2.5$s$ to plan grasps.

我们在图 8 所示含可铰接（如开罐器）与可变形（如抹布）物体的 40 物测试集上做 100 次抓取试验，评估 GQ 泛化。使用交叉熵方法（CEM）[32]：迭代采样候选抓取集，并把候选分布重新拟合到预测稳健性最高的抓取，以更好寻找稳健抓取策略的极大值。更多细节见补充材料。经 CEM 增强的 Dex-Net 2.0 抓取规划器达到 94% 成功率与 99% 精度（69 次判为稳健的抓取中 68 次成功），平均规划 2.5 秒。

VI-H Application: Order Fulfillment

VI-H 应用：订单履约

To demonstrate the modularity of the Dex-Net 2.0 grasp planner, we used it in an order fulfillment application with the ABB YuMi. The goal was to grasp and transport a set of three target objects to a shipping box in the presence of three distractor objects when starting with the objects in a pile on a planar worksurface, illustrated in Fig. 8. Since the Dex-Net 2.0 grasp planner assumes singulated objects, the YuMi first separated the objects using a policy learned from human demonstrations mapping binary images to push locations [30]. When the robot detected an object with sufficient clearance from the pile, it identified the object based on color and used GQ-L-Adv to plan a robust grasp. The robot then transported the object to either the shipping box or a reject box, depending on whether or not the object was a distractor. The system successfully placed the correct objects in the box on 4 out of 5 attempts and was successful in grasping on 93$\%$ of 27 total attempts.

为展示 Dex-Net 2.0 抓取规划器的模块性，我们在 ABB YuMi 上用于订单履约。目标是在三个干扰物存在时，把三个目标物体从平面工作台上的一堆抓取并运到运输箱，见图 8。因 Dex-Net 2.0 规划器假设孤立物体，YuMi 先用从人类示范学到的、把二值图像映射到推位置的策略 [30] 分开物体。当检测到与堆有足够间隙的物体时，按颜色识别物体并用 GQ-L-Adv 规划稳健抓取。再按是否为干扰物运到运输箱或拒绝箱。系统在 5 次尝试中 4 次正确装箱，27 次总尝试中抓取成功 93%。

VI-I Failure Modes

VI-I 失败模式

Fig. 9 displays some common failures of the GQ-CNN grasp planner. One failure mode occured when the RGB-D sensor failed to measure thin parts of the object geometry, making these regions seem accessible. A second type of failure occured due to collisions with the object. It appears that the network was not able to fully distinguish collision-free grasps in narrow parts of the object geometry. This suggests that performance could be improved with more accurate depth sensing and using analytic methods to prune grasps in collsion.

图 9 展示 GQ-CNN 抓取规划器的一些常见失败。一种失败发生在 RGB-D 传感器未能测到物体几何的薄部位，使这些区域看似可及。另一类失败来自与物体碰撞：网络似乎未能充分区分物体几何狭窄部位的无碰撞抓取。这表明更准确的深度传感、并用解析方法剪除碰撞中的抓取，或可改善性能。

Fig. 9: Four examples of failed grasps planned using the GQ-CNN from Dex-Net 2.0. The most common failure modes were related to: (left) missing sensor data for an important part of the object geometry, such as thin parts of the object surface, and (right) collisions with the object that are misclassified as robust.

图 9：用 Dex-Net 2.0 的 GQ-CNN 规划失败的四个抓取例子。最常见失败模式与：（左）物体几何重要部分缺失传感数据（如物体表面薄部位），以及（右）与物体碰撞被误判为稳健相关。

VII Discussion and Future Work

VII 讨论与未来工作

We developed a Grasp Quality Convolutional Neural Network (GQ-CNN) architecture that predicts grasp robustness from a point cloud and trained it on Dex-Net 2.0, a dataset containing 6.7 million point clouds, parallel-jaw grasps, and robust grasp metrics. In over 1,000 physical evaluations, we found that the Dex-Net 2.0 grasp planner is as reliable and 3$\times$ faster a method based on point cloud registration, and had $99\%$ precision on a test set of 40 novel objects.

我们开发了从点云预测抓取稳健性的 GQ-CNN 架构，并在含 670 万点云、平行颚抓取与稳健抓取度量的 Dex-Net 2.0 上训练。超过 1000 次物理评估中，发现 Dex-Net 2.0 抓取规划器与基于点云配准的方法同样可靠且快 3 倍，在 40 个新物体测试集上精度 99%。

In future work, our goal is to approach 100$\%$ success on known objects by using active learning to adaptively acquire grasps using a policy initialized with a GQ-CNN. Additionally, we plan to exend the method to grasp objects in clutter [15, 32] by using simulated piles of rigid objects from Dex-Net and by augmenting the grasping policy with an option to push and separate objects when no robust grasp is available. We also intend to extend the method to use point clouds from multiple viewpoints and in grasping tasks with sequential structure, such as regrasping for assembly. Furthermore, we plan to release a subset of our code, dataset, and the trained GQ-CNN weights to facilitate further research and comparisons.

未来工作中，我们的目标是用主动学习、以 GQ-CNN 初始化的策略自适应获取抓取，使已知物体成功率逼近 100%。此外，计划通过使用 Dex-Net 的仿真刚体堆、并在无稳健抓取可用时用推开物体的选项增强抓取策略，把方法扩展到杂乱中抓取 [15, 32]。还打算扩展到多视角点云以及具有序列结构的抓取任务（如装配中的再抓取）。此外，计划发布部分代码、数据集与训练好的 GQ-CNN 权重，以促进后续研究与对比。

Acknowledgments

致谢

This research was performed at the AUTOLAB at UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab, the Real-Time Intelligent Secure Execution (RISE) Lab, and the CITRIS ”People and Robots” (CPAR) Initiative. The authors were supported in part by the U.S. National Science Foundation under NRI Award IIS-1227536: Multilateral Manipulation by Human-Robot Collaborative Systems, the Department of Defense (DoD) through the National Defense Science & Engineering Graduate Fellowship (NDSEG) Program, the Berkeley Deep Drive (BDD) Program, and by donations from Siemens, Google, Cisco, Autodesk, IBM, Amazon Robotics, and Toyota Robotics Institute. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Sponsors. We thank our colleagues who provided helpful feedback, code, and suggestions, in particular Pieter Abbeel, Ruzena Bajcsy, Brenton Chu, Roy Fox, David Gealy, Ed Johns, Sanjay Krishnan, Animesh Garg, Sergey Levine, Pusong Li, Matt Matl, Stephen McKinley, Andrew Reardon, Vishal Satish, Sammy Staszak, and Nan Tian.

本研究在 UC Berkeley AUTOLAB 完成，隶属于 Berkeley AI Research（BAIR）Lab、Real-Time Intelligent Secure Execution（RISE）Lab 以及 CITRIS「People and Robots」（CPAR）计划。作者部分得到美国国家科学基金会 NRI 奖 IIS-1227536：人机协作系统的多边操作、国防部（DoD）国家国防科学与工程研究生（NDSEG）奖学金、Berkeley Deep Drive（BDD）计划，以及西门子、Google、Cisco、Autodesk、IBM、Amazon Robotics 与 Toyota Robotics Institute 捐赠支持。本文任何意见、发现与结论或建议均属作者，不一定反映资助方观点。感谢提供有益反馈、代码与建议的同事，特别是 Pieter Abbeel、Ruzena Bajcsy、Brenton Chu、Roy Fox、David Gealy、Ed Johns、Sanjay Krishnan、Animesh Garg、Sergey Levine、Pusong Li、Matt Matl、Stephen McKinley、Andrew Reardon、Vishal Satish、Sammy Staszak 与 Nan Tian。

Appendix A Parameters of Graphical Model

附录 A 图模型参数

Our graphical model is illustrated in Fig. 2 and models $p(S,\mathbf{g},\mathbf{x},\mathbf{y})$ as the product of a state distribution $p(\mathbf{x})$, an observation model $p(\mathbf{y}|\mathbf{x})$, a grasp candidate model $p(\mathbf{g}|\mathbf{x})$, and a grasp success model $p(S|\mathbf{g},\mathbf{x})$.

我们的图模型见图 2，把 $p(S,\mathbf{g},\mathbf{x},\mathbf{y})$ 建模为状态分布 $p(\mathbf{x})$、观测模型 $p(\mathbf{y}|\mathbf{x})$、抓取候选模型 $p(\mathbf{g}|\mathbf{x})$ 与抓取成功模型 $p(S|\mathbf{g},\mathbf{x})$ 的乘积。

We model the state distribution as $p(\mathbf{x})=p(\gamma)p(\mathcal{O})p(T_{o}|\mathcal{O})p(T_{c})$. We model $p(\gamma)$ as a Gaussian distribution $\mathcal{N}(0.5,0.1)$ truncated to $[0,1]$. We model $p(\mathcal{O})$ as a discrete uniform distribution over 3D objects in a given dataset. We model $p(T_{o}|\mathcal{O})=p(T_{o}|T_{s})p(T_{s}|\mathcal{O})$, where is $p(T_{s}|\mathcal{O})$ is a discrete uniform distribution over object stable poses and $p(T_{o}|T_{s})$ is uniform distribution over 2D poses: $\mathcal{U}([-0.1,0.1]\times[-0.1,0.1]\times[0,2\pi))$. We compute stable poses using the quasi-static algorithm given by Goldberg et al. [12]. We model $p(T_{c})$ as a uniform distribution on spherical coordinates $r,\theta,\varphi\sim\mathcal{U}([0.65,0.75]\times[0,2\pi)\times[0.05\pi,0.1\pi])$, where the camera optical axis always intersects the center of the table.

我们将状态分布建模为 $p(\mathbf{x})=p(\gamma)p(\mathcal{O})p(T_{o}|\mathcal{O})p(T_{c})$。将 $p(\gamma)$ 建模为截断到 $[0,1]$ 的高斯 $\mathcal{N}(0.5,0.1)$。将 $p(\mathcal{O})$ 建模为给定数据集中 3D 物体上的离散均匀分布。将 $p(T_{o}|\mathcal{O})=p(T_{o}|T_{s})p(T_{s}|\mathcal{O})$，其中 $p(T_{s}|\mathcal{O})$ 为物体稳定位姿上的离散均匀，$p(T_{o}|T_{s})$ 为 2D 位姿上的均匀：$\mathcal{U}([-0.1,0.1]\times[-0.1,0.1]\times[0,2\pi))$。用 Goldberg 等 [12] 的准静态算法计算稳定位姿。将 $p(T_{c})$ 建模为球坐标 $r,\theta,\varphi\sim\mathcal{U}([0.65,0.75]\times[0,2\pi)\times[0.05\pi,0.1\pi])$ 上的均匀分布，相机光轴始终交于桌面中心。

Our distribution over grasps is a uniform distribution over pairs of antipodal points on the object surface that are parallel to the table plane. We sample from this distribution for a fixed coefficient of friction $\mu=0.6$ and reject samples outside the friction cone or non-parallel to the surface.

抓取分布为平行于桌面的物体表面对跖点对上的均匀分布。对固定摩擦系数 $\mu=0.6$ 采样，并拒绝摩擦锥外或非平行于表面的样本。

We model images as $\mathbf{y}=\alpha*\hat{\mathbf{y}}+\epsilon$ where $\hat{\mathbf{y}}$ is a rendered depth image created using OSMesa offscreen rendering. We model $\alpha$ as a Gamma random variable with shape$=1000.0$ and scale=$0.001$. We model $\epsilon$ as Gaussian Process noise drawn with measurement noise $\sigma=0.005$ and kernel bandwidth $\ell=\sqrt{2}px$.

我们将图像建模为 $\mathbf{y}=\alpha*\hat{\mathbf{y}}+\epsilon$，其中 $\hat{\mathbf{y}}$ 为用 OSMesa 离屏渲染创建的渲染深度图。将 $\alpha$ 建模为 shape=1000.0、scale=0.001 的 Gamma 随机变量。将 $\epsilon$ 建模为测量噪声 $\sigma=0.005$、核带宽 $\ell=\sqrt{2}$ px 的高斯过程噪声。

We compute grasp robustness metrics using the graphical model and noise parameters of [51].

我们用图模型与噪声模型计算抓取稳健性度量，细节见正文。

Appendix B Grasp Sampling Methods

附录 B 抓取采样方法

Our goal is to learn policy parameters $\theta$ that maximize the success rate of planned grasps over a distribution on point clouds that can be generated from a set of possible objects $\mathcal{D}$:

目标是学习策略参数 $\theta$，使在可由可能物体集 $\mathcal{D}$ 生成的点云分布上，已规划抓取的成功率最大化：

$$\displaystyle\theta^{*}=\underset{\theta\in\Theta}{\text{argmax }}\mathbb{E}_{p(\mathbf{y}\mid\mathcal{D})}\left[Q(\pi_{\theta},\mathbf{y})\right]. \tag{B.1}$$

We propose to approach the objective of equation B.1 by using a greedy grasping policy $\pi_{\theta}(\mathbf{y})=\text{argmax}_{\mathbf{u}\in\mathcal{C}}Q_{\theta}(\mathbf{u},\mathbf{y})$ with respect to a GQ-CNN robustness function $Q_{\theta}(\mathbf{u},\mathbf{y})$, where $\mathcal{C}$ specifies constraints on candidate grasps such as kinematic feasibility. We explore two implementations of the robust grasping policy: (1) sampling a large, fixed set of antipodal grasps and choosing the most robust one and (2) optimizing for the most robust grasp using derivative free optimization.

我们建议用相对 GQ-CNN 稳健性函数 $Q_{\theta}(\mathbf{u},\mathbf{y})$ 的贪心抓取策略 $\pi_{\theta}(\mathbf{y})=\text{argmax}_{\mathbf{u}\in\mathcal{C}}Q_{\theta}(\mathbf{u},\mathbf{y})$ 逼近公式 B.1 的目标，其中 $\mathcal{C}$ 指定运动学可行等候选约束。我们探索该策略的两种实现：对跖抓取采样与无导数优化。

B-A Antipodal Grasp Sampling

B-A 对跖抓取采样

The antipodal grasp sampling method used in the paper is designed to sample antipodal grasps specified as a planar pose, angle, and height with respect to a table. The algorithm is detailed in Algorithm 1. We first threshold the depth image to find areas of high gradient. Then, we use rejection sampling over pairs of pixels to generate a set of candidate antipodal grasps, incrementally increasing the friction coefficient until a desired number of grasps is reached in case the desired number cannot be achieved with a smaller friction coefficient. We convert antipodal grasps in image space to 3D by assigning discretizing the gripper height between the height of the grasp center pixel relative and the height of the table surface itself.

文中所用对跖抓取采样方法旨在采样相对桌面由平面位姿、角度与高度指定的对跖抓取。算法详见算法 1。先对深度图取阈值找高梯度区域，再在像素对上做拒绝采样生成候选对跖抓取集，并逐步提高摩擦下限直至采满 $M$ 个候选或达到样本上限。

This grasp sampling method is used for all image based grasp planners in the paper. We used $M=1000$, $K$ set to the intrinsics of a Primesense Carmine 1.08, $T_{c}$ determined by chessboard registration, $g=0.0025m$, $\mu_{\ell}=0.4$, $\delta_{\mu}=0.2$, $N=1000$, and $\delta_{h}=0.01m$.

该抓取采样方法用于文中全部基于图像的抓取规划器。我们取 $M=1000$，$K$ 为 Primesense Carmine 1.08 内参，$T_{c}$ 由棋盘配准确定，$g=0.0025$ m，$\mu_{\ell}=0.4$，$\delta_{\mu}=0.2$，$N=1000$，$\delta_{h}=0.01$ m。

Input: Depth image $\mathbf{y}$, Number of grasps $M$, Camera Intrinsics Matrix $K$, Camera pose $T_{c}$, Depth gradient threshold $g$, Min friction coef $\mu_{\ell}$, Friction coef increment $\delta_{\mu}$, Max samples per friction coef $N$, Gripper height resolution $\delta_{h}$ Result: $\mathcal{G}$, set of candidate grasps
1
// Compute depth edges
$G_{x}=\nabla_{x}\mathbf{y},G_{y}=\nabla_{y}\mathbf{y}$; 2
$\mathcal{E}=\{\mathbf{u}\in\mathbb{R}^{2}:G_{x}(\mathbf{u})^{2}+G_{y}(\mathbf{u})^{2}>g\}$; 3
4
// Find antipodal pairs
$\mathcal{G}=\{\}$, $i=0$, $j=0$; 5
while $|\mathcal{G}|<M$ and $\mu<=1.0$* **do** 6
$\mathbf{u},\mathbf{v}=$UniformRandom($\mathcal{E},2$); 7
**if** *Antipodal($\mathbf{u},\mathbf{v},\mu$)* **then** 8
// Compute piont in world coordinates
$\mathbf{c}=0.5*(\mathbf{u}+\mathbf{v})$; 9
$\mathbf{p}_{c}=$Deproject($K,\mathbf{y},\mathbf{c}$); 10
$\mathbf{p}=T_{c}*\mathbf{p}_{c}$; 11
$h=\mathbf{p}.z$; 12
13
// Add all heights
**while** *$h>0$ do 14
$\mathcal{G}=\mathcal{G}\cup\{\mathbf{g}(\mathbf{u},\mathbf{v},h)\}$; 15
$h=h-\delta_{h}$; 16
end while 17
end if 18
$i=i+1$, $j=j+1$; 19
20
// Update friction coef
if $j>=N$ then 21
$\mu=\mu+\delta_{\mu}$; 22
$j=0$; 23
end if 24
end while 25
return $\mathcal{G}$; 26
27

Algorithm 1 Antipodal Grasp Sampling from a Depth Image

输入： 深度图 $\mathbf{y}$，抓取数 $M$，相机内参矩阵 $K$，相机位姿 $T_{c}$，深度梯度阈值 $g$，最小摩擦系数 $\mu_{\ell}$，摩擦系数增量 $\delta_{\mu}$，每摩擦系数最大样本数 $N$，夹爪高度分辨率 $\delta_{h}$
输出： $\mathcal{G}$，候选抓取集

1. 计算深度边缘 $G_{x}=\nabla_{x}\mathbf{y},G_{y}=\nabla_{y}\mathbf{y}$
2. 在高梯度像素对上拒绝采样对跖抓取，递增摩擦下限直至 $|\mathcal{G}|\ge M$ 或达到样本上限
3. 对每个候选在高度网格上评估并加入 $\mathcal{G}$

算法 1 对跖抓取采样

B-B Derivative Free Optimization

B-B 无导数优化

One problem with choosing a grasp from a fixed set of candidates is that the set of candidates may all have a low probability of success. This can be difficult when an object can only be grasped in a small set of precise configurations, such as the example in Fig. 10. Some of these failures can be seen in the right panel of the failure modes figure in the original paper.

从固定候选集中选抓取的一个问题是：候选可能全部成功概率都很低。当物体只能在少数精确配置下被抓住时尤其困难，如图 10 之例。原文失败模式图右栏可见部分此类失败。

In our second generalization study we addressed this problem using the cross entropy method (CEM) [32, 47], a form of derivative-free optimization, to optimize for the most robust grasp by iteratively resampling grasps from a learned distribution over robust grasps and updating the distribution. The method, illustrated in Algorithm 2, models the distribution on promising grasps using a Gaussian Mixture Model (GMM) and seeds the initial set of grasps with antipodal point pairs using Algorithm 1 with no iterative friction coefficient updates. The algorithm takes as input the number of CEM iterations $m$, the number of initial grasps to sample $n$, the number of grasps to resample from the model $c$, the number of GMM mixure components $k$, a friction coefficient $\mu$, and elite percentage $\gamma$, and the GQ-CNN $Q_{\theta}$, and returns an estimate of the most robust grasp $\mathbf{u}$. In our generalization experiment we used $m=3$, $n=100$, $c=50$, $\mu=0.8$, $k=3$, and $\gamma=25\%$. The qualitative performance of our method on several examples from our experiments is illustrated in Fig. 11.

在第二次泛化研究中，我们用交叉熵方法（CEM）[32, 47]——一种无导数优化——应对：通过从学到的稳健抓取分布迭代重采样抓取并更新分布，优化最稳健抓取。方法见算法 2，用高斯混合模型对有希望的抓取分布建模。

Fig. 10: (Left) Grasp robustness predicted by a Grasp Quality Convolutional Neural Network (GQ-CNN) trained with Dex-Net 2.0 over the space of depth images and grasps for a single point cloud collected with a Primesense Carmine. As the center of the gripper moves from the top to the bottom of the image the GQ-CNN prediction stays near zero and spikes on the most robust grasp (right), for which the gripper fits into a small opening on the object surface. This suggests that the GQ-CNN has learned a detailed representation of the collision space between the object and gripper. Furthermore, the sharp spike suggests that it may be difficult to plan robust grasps by randomly sampling grasps in image space. We consider planning the most robust grasp using the cross-entropy method on the GQ-CNN response.

图 10：（左）在用 Primesense Carmine 采集的单个点云上，经 Dex-Net 2.0 训练的 GQ-CNN 在深度图与抓取空间上预测的抓取稳健性。当夹爪中心从图像顶部移到底部时，GQ-CNN 预测接近零，并在最稳健抓取处尖峰（右），该处夹爪对准物体可抓部位。

Fig. 11: Example input color images and maps of the grasp robust estimated by the GQ-CNN over grasp centers for a constant grasp axis angle in image space and height above the table, with the grasp planned by our CEM-based robust grasping policy shown in black. CEM is able to find precise robust grasping locations encoded by the GQ-CNN that are very close to the global maximum for the given grasp axis and height. The GQ-CNN also appears to assign non-zero robustness to several grasps that completely miss the object. This is likely because no such grasps are in the training set, and future work could augment the training dataset to avoid these grasps.

图 11：示例输入彩色图，以及在图像空间固定抓取轴角度与相对桌面高度下、GQ-CNN 对抓取中心估计的稳健性图；黑色为 CEM 稳健抓取策略规划的抓取。CEM 能找到 GQ-CNN 编码的、非常接近给定抓取轴角度全局极大的精确稳健抓取位置。

Input: Num rounds $m$, Num inital samples $n$, Num CEM samples $c$, Num GMM mixture $k$, Friction coef $\mu$, Elite percentage $\gamma$, Robustness function $Q_{\theta}$ Result: $\mathbf{u}$, most robust grasp
$\mathcal{U}\leftarrow$uniform set of $n$ antipodal grasps; 1
for i = 1, …, m do 2
$\mathcal{E}\leftarrow$top $\gamma-$percentile of grasps ranked by $Q_{\theta}$; 3
$M\leftarrow$GMM fit to $\mathcal{E}$ with $k$ mixtures; 4
$G\leftarrow c$ iid samples from $M$; 5
end for 6
return $\underset{\mathbf{u}\in\mathcal{U}}{\text{argmax }}{Q_{\theta}(\mathbf{u},\mathbf{y})}$; 7

Algorithm 2 Robust Grasping Policy using the Cross Entropy Method on a Learned GQ-CNN

输入： 轮数 $m$，初始样本数 $n$，CEM 样本数 $c$，GMM 混合数 $k$，摩擦系数 $\mu$，精英比例 $\gamma$，稳健性函数 $Q_{\theta}$
输出： $\mathbf{u}$，最稳健抓取
$\mathcal{U}\leftarrow$ $n$ 个对跖抓取的均匀集；
for i = 1, …, m do
$\mathcal{E}\leftarrow$ 按 $Q_{\theta}$ 排序的前 $\gamma$ 百分位抓取；
拟合 $k$ 分量 GMM $M$ 到 $\mathcal{E}$；
从 $M$ 采样 $c$ 个抓取更新 $\mathcal{U}$；
end
返回 $\mathcal{U}$ 中 $Q_{\theta}$ 最高的抓取

算法 2 基于 CEM 的稳健抓取优化

References

参考文献

Abadi et al. [2016] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 , 2016.

Balasubramanian et al. [2012] Ravi Balasubramanian, Ling Xu, Peter D Brook, Joshua R Smith, and Yoky Matsuoka. Physical human interactive guidance: Identifying grasping principles from human-planned grasps. IEEE Trans. Robotics , 28(4):899–910, 2012.

Bohg and Kragic [2010] Jeannette Bohg and Danica Kragic. Learning grasping points with shape context. Robotics and Autonomous Systems , 58(4):362–377, 2010.

Bohg et al. [2014] Jeannette Bohg, Antonio Morales, Tamim Asfour, and Danica Kragic. Data-driven grasp synthesis—a survey. IEEE Trans. Robotics , 30(2):289–309, 2014.

Brook et al. [2011] Peter Brook, Matei Ciocarlie, and Kaijen Hsiao. Collaborative grasp planning with multiple object representations. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , pages 2851–2858. IEEE, 2011.

Chen and Burdick [1993] I-Ming Chen and Joel W Burdick. Finding antipodal point grasps on irregularly shaped objects. IEEE Trans. Robotics and Automation , 9(4):507–512, 1993.

Ciocarlie et al. [2014] Matei Ciocarlie, Kaijen Hsiao, Edward Gil Jones, Sachin Chitta, Radu Bogdan Rusu, and Ioan A Şucan. Towards reliable grasping and manipulation in household environments. In Experimental Robotics , pages 241–252. Springer, 2014.

Dalal and Triggs [2005] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , volume 1, pages 886–893. IEEE, 2005.

Detry et al. [2013] Renaud Detry, Carl Henrik Ek, Marianna Madry, and Danica Kragic. Learning a dictionary of prototypical grasp-predicting parts from grasping experience. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , pages 601–608. IEEE, 2013.

Eitel et al. [2015] Andreas Eitel, Jost Tobias Springenberg, Luciano Spinello, Martin Riedmiller, and Wolfram Burgard. Multimodal deep learning for robust rgb-d object recognition. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) , pages 681–687. IEEE, 2015.

Friedman et al. [2001] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning , volume 1. Springer series in statistics Springer, Berlin, 2001.

Goldberg et al. [1999] Ken Goldberg, Brian V Mirtich, Yan Zhuang, John Craig, Brian R Carlisle, and John Canny. Part pose statistics: Estimators and experiments. IEEE Trans. Robotics and Automation , 15(5):849–857, 1999.

Goldfeder and Allen [2011] Corey Goldfeder and Peter K Allen. Data-driven grasping. Autonomous Robots , 31(1):1–20, 2011.

Goldfeder et al. [2009] Corey Goldfeder, Matei Ciocarlie, Hao Dang, and Peter K Allen. The columbia grasp database. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , pages 1710–1716. IEEE, 2009.

Gualtieri et al. [2016] Marcus Gualtieri, Andreas ten Pas, Kate Saenko, and Robert Platt. High precision grasp pose detection in dense clutter. arXiv preprint arXiv:1603.01564 , 2016.

Guo et al. [2017] Menglong Guo, David V Gealy, Jacky Liang, Jeffrey Mahler, Aimee Goncalves, Stephen McKinley, and Ken Goldberg. Design of parallel-jaw gripper tip surfaces for robust grasping. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , 2017.

Gupta et al. [2015] Saurabh Gupta, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Aligning 3d models to rgb-d images of cluttered scenes. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , pages 4731–4740, 2015.

Hartley and Zisserman [2003] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision . Cambridge university press, 2003.

He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proc. IEEE Int. Conf. on Computer Vision (ICCV) , pages 1026–1034, 2015.

Hernandez et al. [2016] Carlos Hernandez, Mukunda Bharatheesha, Wilson Ko, Hans Gaiser, Jethro Tan, Kanter van Deurzen, Maarten de Vries, Bas Van Mil, Jeff van Egmond, Ruben Burger, et al. Team delft’s robot winner of the amazon picking challenge 2016. arXiv preprint arXiv:1610.05514 , 2016.

Herzog et al. [2014] Alexander Herzog, Peter Pastor, Mrinal Kalakrishnan, Ludovic Righetti, Jeannette Bohg, Tamim Asfour, and Stefan Schaal. Learning of grasp selection based on shape-templates. Autonomous Robots , 36(1-2):51–65, 2014.

Hinterstoisser et al. [2011] Stefan Hinterstoisser, Stefan Holzer, Cedric Cagniart, Slobodan Ilic, Kurt Konolige, Nassir Navab, and Vincent Lepetit. Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In Proc. IEEE Int. Conf. on Computer Vision (ICCV) , pages 858–865. IEEE, 2011.

Jaderberg et al. [2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Proc. Advances in Neural Information Processing Systems , pages 2017–2025, 2015.

Johns et al. [2016] Edward Johns, Stefan Leutenegger, and Andrew J Davison. Deep learning a grasp function for grasping under gripper pose uncertainty. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) , pages 4461–4468. IEEE, 2016.

Kappler et al. [2015] Daniel Kappler, Jeannette Bohg, and Stefan Schaal. Leveraging big data for grasp planning. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , 2015.

Kasper et al. [2012] Alexander Kasper, Zhixing Xue, and Rüdiger Dillmann. The kit object models database: An object model database for object recognition, localization and manipulation in service robotics. Int. Journal of Robotics Research (IJRR) , 31(8):927–934, 2012.

Kehoe et al. [2013] Ben Kehoe, Akihiro Matsukawa, Sal Candido, James Kuffner, and Ken Goldberg. Cloud-based robot grasping with the google object recognition engine. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , pages 4263–4270. IEEE, 2013.

Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Proc. Advances in Neural Information Processing Systems , pages 1097–1105, 2012.

Laskey et al. [2015] Michael Laskey, Jeffrey Mahler, Zoe McCarthy, Florian T Pokorny, Sachin Patil, Jur van den Berg, Danica Kragic, Pieter Abbeel, and Ken Goldberg. Multi-armed bandit models for 2d grasp planning with uncertainty. In Proc. IEEE Conf. on Automation Science and Engineering (CASE) . IEEE, 2015.

Laskey et al. [2017] Michael Laskey, Caleb Chuck, Jonathan Lee, Jeffrey Mahler, Sanjay Krishnan, Kevin Jamieson, Anca Dragan, and Ken Goldberg. Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) . IEEE, 2017.

Lenz et al. [2015] Ian Lenz, Honglak Lee, and Ashutosh Saxena. Deep learning for detecting robotic grasps. Int. Journal of Robotics Research (IJRR) , 34(4-5):705–724, 2015.

Levine et al. [2016] Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. arXiv preprint arXiv:1603.02199 , 2016.

Mahler et al. [2016a] Jeffrey Mahler, Brian Hou, Sherdil Niyaz, Florian T Pokorny, Ramu Chandra, and Ken Goldberg. Privacy-preserving grasp planning in the cloud. In Proc. IEEE Conf. on Automation Science and Engineering (CASE) , pages 468–475. IEEE, 2016a.

Mahler et al. [2016b] Jeffrey Mahler, Florian T Pokorny, Brian Hou, Melrose Roderick, Michael Laskey, Mathieu Aubry, Kai Kohlhoff, Torsten Kröger, James Kuffner, and Ken Goldberg. Dex-net 1.0: A cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) . IEEE, 2016b.

Mallick et al. [2014] Tanwi Mallick, Partha Pratim Das, and Arun Kumar Majumdar. Characterizations of noise in kinect depth images: A review. IEEE Sensors Journal , 14(6):1731–1740, 2014.

Miller et al. [1993] John W Miller, Rod Goodman, and Padhraic Smyth. On loss functions which minimize to conditional expected values and posterior probabilities. IEEE Transactions on Information Theory , 39(4):1404–1408, 1993.

Montesano and Lopes [2012] Luis Montesano and Manuel Lopes. Active learning of visual descriptors for grasping using non-parametric smoothed beta distributions. Robotics and Autonomous Systems , 60(3):452–462, 2012.

Nguyen et al. [2016] Anh Nguyen, Dimitrios Kanoulas, Darwin G Caldwell, and Nikos G Tsagarakis. Detecting object affordances with convolutional neural networks. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) , pages 2765–2770. IEEE, 2016.

Oberlin and Tellex [2015] John Oberlin and Stefanie Tellex. Autonomously acquiring instance-based object models from experience. In Int. S. Robotics Research (ISRR) , 2015.

Pinto and Gupta [2016] Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , 2016.

Pinto et al. [2016] Lerrel Pinto, James Davidson, and Abhinav Gupta. Supervision via competition: Robot adversaries for learning tasks. arXiv preprint arXiv:1610.01685 , 2016.

Pokorny and Kragic [2013] Florian T Pokorny and Danica Kragic. Classical grasp quality evaluation: New algorithms and theory. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) , pages 3493–3500. IEEE, 2013.

Porzi et al. [2016] Lorenzo Porzi, Samuel Rota Bulo, Adrian Penate-Sanchez, Elisa Ricci, and Francesc Moreno-Noguer. Learning depth-aware deep representations for robotic perception. IEEE Robotics & Automation Letters , 2016.

Prattichizzo and Trinkle [2008] Domenico Prattichizzo and Jeffrey C Trinkle. Grasping. In Springer handbook of robotics , pages 671–700. Springer, 2008.

Redmon and Angelova [2015] Joseph Redmon and Anelia Angelova. Real-time grasp detection using convolutional neural networks. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , pages 1316–1322. IEEE, 2015.

Rodriguez et al. [2012] Alberto Rodriguez, Matthew T Mason, and Steve Ferry. From caging to grasping. Int. Journal of Robotics Research (IJRR) , page 0278364912442972, 2012.

Rubinstein et al. [2013] Reuven Y Rubinstein, Ad Ridder, and Radislav Vaisman. Fast sequential Monte Carlo methods for counting and optimization . John Wiley & Sons, 2013.

Sadeghi and Levine [2016] Fereshteh Sadeghi and Sergey Levine. (cad)2 rl: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201 , 2016.

Salas-Moreno et al. [2013] Renato F Salas-Moreno, Richard A Newcombe, Hauke Strasdat, Paul HJ Kelly, and Andrew J Davison. Slam++: Simultaneous localisation and mapping at the level of objects. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , pages 1352–1359, 2013.

Saxena et al. [2008] Ashutosh Saxena, Justin Driemeyer, and Andrew Y Ng. Robotic grasping of novel objects using vision. The International Journal of Robotics Research , 27(2):157–173, 2008.

Seita et al. [2016] Daniel Seita, Florian T Pokorny, Jeffrey Mahler, Danica Kragic, Michael Franklin, John Canny, and Ken Goldberg. Large-scale supervised learning of the grasp robustness of surface patch pairs. In Proc. IEEE Int. Conf. on Simulation, Modeling, and Programming of Autonomous Robots (SIMPAR) . IEEE, 2016.

ten Pas and Platt [2015] Andreas ten Pas and Robert Platt. Using geometry to detect grasp poses in 3d point clouds. In Int’l Symp. on Robotics Research , 2015.

Tzeng et al. [2016] Eric Tzeng, Coline Devin, Judy Hoffman, Chelsea Finn, Pieter Abbeel, Sergey Levine, Kate Saenko, and Trevor Darrell. Adapting deep visuomotor representations with weak pairwise constraints. In Workshop on the Algorithmic Foundation of Robotics (WAFR) , 2016.

Varley et al. [2015] Jacob Varley, Jonathan Weisz, Jared Weiss, and Peter Allen. Generating multi-fingered robotic grasps via deep learning. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) , pages 4415–4420. IEEE, 2015.

Veres et al. [2017] Matthew Veres, Medhat Moussa, and Graham W Taylor. Modeling grasp motor imagery through deep conditional generative models. arXiv preprint arXiv:1701.03041 , 2017.

Weisz and Allen [2012] Jonathan Weisz and Peter K Allen. Pose error robust grasping from contact wrench space metrics. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , pages 557–562. IEEE, 2012.

Wohlkinger et al. [2012] Walter Wohlkinger, Aitor Aldoma, Radu B Rusu, and Markus Vincze. 3dnet: Large-scale object class recognition from cad models. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , pages 5384–5391. IEEE, 2012.

Xie et al. [2013] Ziang Xie, Arjun Singh, Justin Uang, Karthik S Narayan, and Pieter Abbeel. Multimodal blending for high-accuracy instance recognition. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) , pages 2214–2221. IEEE, 2013.

Zeng et al. [2016] Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker Jr, Alberto Rodriguez, and Jianxiong Xiao. Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. arXiv preprint arXiv:1609.09475 , 2016.

Zhu et al. [2016] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. arXiv preprint arXiv:1609.05143 , 2016.