Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation
Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation
CSAIL, Massachusetts Institute of Technology
{peteflo,manuelli,russt}@csail.mit.edu
*These authors contributed equally to this work.
Abstract: What is the right object representation for manipulation? We would like robots to visually perceive scenes and learn an understanding of the objects in them that (i) is task-agnostic and can be used as a building block for a variety of manipulation tasks, (ii) is generally applicable to both rigid and non-rigid objects, (iii) takes advantage of the strong priors provided by 3D vision, and (iv) is entirely learned from self-supervision. This is hard to achieve with previous methods: much recent work in grasping does not extend to grasping specific objects or other tasks, whereas task-specific learning may require many trials to generalize well across object configurations or other tasks. In this paper we present Dense Object Nets, which build on recent developments in self-supervised dense descriptor learning, as a consistent object representation for visual understanding and manipulation. We demonstrate they can be trained quickly (approximately 20 minutes) for a wide variety of previously unseen and potentially non-rigid objects. We additionally present novel contributions to enable multi-object descriptor learning, and show that by modifying our training procedure, we can either acquire descriptors which generalize across classes of objects, or descriptors that are distinct for each object instance. Finally, we demonstrate the novel application of learned dense descriptors to robotic manipulation. We demonstrate grasping of specific points on an object across potentially deformed object configurations, and demonstrate using class general descriptors to transfer specific grasps across objects in a class.
摘要:操作应当采用怎样的物体表示?我们希望机器人能够在视觉上感知场景,并学到对其中物体的理解,且这种理解(i)与具体任务无关,能够作为多种操作任务的积木;(ii)普遍适用于刚体和柔性物体;(iii)能够利用三维视觉提供的强先验;(iv)完全通过自监督学习得到。这对已有方法来说并不容易:近期不少抓取工作无法扩展到抓住特定物体或其他任务,而面向单一任务的学习,往往要经过很多次试验,才能在不同物体配置或其他任务上泛化得好。本文提出 Dense Object Nets,它建立在近期自监督稠密描述子学习的进展之上,作为一种用于视觉理解和操作的一致性物体表示。我们表明,对于此前未见、甚至可能是非刚性的各类物体,它可以快速训练(大约 20 分钟)。我们还提出了新的方法以支持多物体描述子学习,并表明只要改动训练步骤,既可以学到在一类物体上通用的描述,也可以学到对每个物体实例都彼此不同的描述。最后,我们展示了把学到的稠密描述子用于机器人操作这一新应用:在物体可能发生变形的配置下抓住指定点,以及用类通用描述把特定抓取迁移到同一类的其他物体上。
Keywords: Visual Descriptor Learning, Self-Supervision, Robot Manipulation
关键词:视觉描述子学习、自监督、机器人操作
1 Introduction
1 引言
What is the right object representation for manipulation? While task-specific reinforcement learning can achieve impressively dexterous skills for a given specific task [1], it is unclear which is the best route to efficiently achieving many different tasks. Other recent work [2, 3] can provide very general grasping functionality but does not address specificity. Achieving specificity, the ability to accomplish specific tasks with specific objects, may require solving the data association problem. At a coarse level the task of identifying individual objects to manipulate can be solved by instance segmentation, as demonstrated in the Amazon Robotics Challenge (ARC) [4, 5] or [6]. Whole-object-level segmentation, however, does not provide any information on the rich structure of the objects themselves, and hence may not be an appropriate representation for solving more complex tasks. While not previously applied to the robotic manipulation domain, recent work has demonstrated advances in learning dense pixel level data association [7, 8], including self-supervision from raw RGBD data [8], which inspired our present work.
操作应当采用怎样的物体表示?面向单一任务的强化学习,虽然能在给定的特定任务上做出相当灵巧的技能[1],但怎样才能高效完成许多不同的任务,目前仍不清楚。另一些近期工作[2, 3]可以提供非常通用的抓取能力,却并不处理“特定性”。若要具备特定性,也就是用特定物体完成特定任务,可能需要先解决数据关联问题。在较粗的层面上,识别出要操作的各个物体,可以用实例分割来完成,这在亚马逊机器人挑战赛(Amazon Robotics Challenge,ARC)[4, 5]或[6]中已有展示。不过,整物体级别的分割并不提供物体自身丰富结构的信息,因此未必适合作为解决更复杂任务的表示。尽管此前尚未被用到机器人操作领域,近期工作已在学习稠密的像素级数据关联上取得进展[7, 8],其中包括从原始 RGBD 数据中做自监督[8],这也启发了我们的工作。
In this paper, we propose and demonstrate using dense visual description as a representation for robotic manipulation. We demonstrate the first autonomous system that can entirely self-supervise to learn consistent dense visual representations of objects, and the first system we know of that is capable of performing the manipulation demonstrations we provide. Specifically, with no human supervision during training, our system can grasp specific locations on deformable objects, grasp semantically corresponding locations on instances in a class, and grasp specific locations on specific instances in clutter. Towards this goal, we also provide practical contributions to dense visual descriptor learning with general computer vision applications outside of robotic manipulation. We call our visual representations Dense Object Nets, which are deep neural networks trained to provide dense (pixelwise) description of objects.
在本文中,我们提出并展示把稠密视觉描述作为机器人操作的一种表示。我们展示了第一套能够完全通过自监督来学习物体一致性稠密视觉表示的自主系统,也是据我们所知第一套能够完成本文所给操作演示的系统。具体来说,训练过程中无需人工监督,我们的系统就能抓住可变形物体上的指定位置,抓住同一类中各实例上语义对应的位置,以及在杂乱场景中抓住特定实例上的指定位置。朝着这一目标,我们还为稠密视觉描述子学习提供了若干实用改进,这些改进在机器人操作之外的一般计算机视觉应用中同样有用。我们把这种视觉表示称作 Dense Object Nets,它们是经过训练、能够提供稠密(像素级)物体描述的深度神经网络。
Code, data, and video available: github.com/RobotLocomotion/pytorch-dense-correspondence
代码、数据与视频见:github.com/RobotLocomotion/pytorch-dense-correspondence
2nd Conference on Robot Learning (CoRL 2018), Zurich, Switzerland.
第 2 届机器人学习会议(CoRL 2018),瑞士苏黎世。
Contributions. We believe our largest contribution is that we introduce dense descriptors as a representation useful for robotic manipulation. We’ve also shown that self-supervised dense visual descriptor learning can be applied to a wide variety of potentially non-rigid objects and classes (47 objects so far, including 3 distinct classes), can be learned quickly (approximately 20 minutes), and can enable new manipulation tasks. In example tasks we grasp specific points on objects across potentially deformed configurations, do so with object instance-specificity in clutter, or transfer specific grasps across objects in a class. We also contribute novel techniques to enable multi-object distinct dense descriptors, and show that by modifying the loss function and sampling procedure, we can either acquire descriptors which generalize across classes of objects, or descriptors that are distinct for each object instance. Finally, we contribute general training techniques for dense descriptors which we found to be critical to achieving good performance in practice.
主要贡献。 我们认为最大的贡献是,把稠密描述子引入为一种对机器人操作有用的表示。我们还表明,自监督的稠密视觉描述子学习可以用于多种多样、甚至可能是非刚性的物体和类别(迄今已有 47 个物体,其中包括 3 个不同的类别),可以快速学会(大约 20 分钟),并能够支持新的操作任务。在示例任务中,我们在可能发生变形的配置下抓住物体上的指定点,在杂乱场景中按物体实例做特定抓取,或把特定抓取迁移到同一类的其他物体上。我们还提出了新的技术,以便得到多物体、彼此可区分的稠密描述子,并表明只要改动损失函数和采样步骤,既可以学到在一类物体上通用的描述,也可以学到对每个物体实例都彼此不同的描述。最后,我们给出若干通用的稠密描述子训练技巧,并发现它们对在实际中取得良好性能至关重要。
Paper Organization. In Section 2 we describe related work. As preliminary in Section 3.1 we describe the general technique for self-supervising dense visual descriptor learning, which is from [8] but reproduced here for clarity. We then describe additional techniques we’ve developed for object-centric visual descriptors in Section 3.2, and Section 3.3 describes techniques for distinct multi-object descriptors. Section 4 describes our experimental setup for our autonomous system, and Section 5 describes our results: our learned visual descriptors for a wide variety of objects (Section 5.1) multi-object descriptors and selective class generalization (Sections 5.2 and 5.3), and robotic manipulation demonstrations (Section 5.4).
论文结构。 第 2 节介绍相关工作。作为预备,第 3.1 节说明自监督稠密视觉描述子学习的一般方法,该方法来自[8],此处为清楚起见再述一遍。随后,第 3.2 节介绍我们为面向物体的视觉描述子发展的额外技术,第 3.3 节介绍用于得到可区分的多物体描述子的技术。第 4 节说明这套自主系统的实验设置,第 5 节介绍结果:在多种多样的物体上学到的视觉描述子(第 5.1 节),多物体描述子与可选择的类内泛化(第 5.2 和 5.3 节),以及机器人操作演示(第 5.4 节)。
2 Related Work
2 相关工作
We review three main areas of related work: learned descriptors, self-supervised visual learning for robots, and robot learning for specific tasks. The task of correspondence estimation from multiple views of the same scene is fundamental in computer vision, whereas dense semantic correspondence across different scenes was popularized by [9]. Recent advances have been made by introducing a pixel-wise variant of contrastive loss [10] combined with deep convolutional networks, as in Choy et al. [7] and Schmidt et al. [8]. For cross-instance semantic correspondence, [7] relies on human annotations, while [8] learns these unsupervised, as we do here. Other work [11] uses image warping to learn descriptors, and most require manually annotated labels [13, 14, 15]. Zeng et al. [12] also uses dense 3D reconstruction to provide automated labeling, but for descriptors of 3D volume patches. Some of these works [8, 11, 15], like ours, learn descriptors for specific object instances or classes, while others [12] learn descriptors for establishing correspondence of arbitrary data. None of these prior works in dense visual learning involve robots.
我们从三个主要方面回顾相关工作:学习得到的描述子、面向机器人的自监督视觉学习,以及面向特定任务的机器人学习。从同一场景的多个视角估计对应关系,是计算机视觉中的基本问题;而跨不同场景的稠密语义对应,则因[9]而得到广泛关注。近期的进展来自把对比损失的像素级变体[10]与深度卷积网络结合起来,如 Choy 等人[7]和 Schmidt 等人[8]。对于跨实例的语义对应,[7]依赖人工标注,而[8]与本文一样,以无监督方式学习这些对应。另有工作[11]用图像扭曲来学习描述子,而且其中多数需要人工标注的标签[13, 14, 15]。Zeng 等人[12]同样用稠密三维重建来自动提供标注,但其描述子针对的是三维体素块。这些工作中,有的[8, 11, 15]和我们一样,为特定物体实例或类别学习描述子;有的[12]则为任意数据建立对应而学习描述子。这些稠密视觉学习方面的已有工作,都没有把机器人纳入其中。
In the area of self-supervised visual robot learning, while some recent work has sought to understand ‘how will the world change given the robot’s action?” [16, 17] in this work we instead ask “what is the current visual state of the robot’s world?”. We address this question with a dense description that is consistent across viewpoints, object configurations and (if desired) object classes. At the coarse level of semantic segmentation several works from the Amazon Robotics Challenge used robots to automate the data collection and annotation process through image-level background subtraction [18, 5, 4]. In contrast this work uses 3D reconstruction-based change detection and dense pixelwise correspondences, which provides a much richer supervisory signal for use during training.
在面向机器人的自监督视觉学习方面,一些近期工作试图理解“给定机器人的动作,世界会怎样变化?”[16, 17];本文则转而问:“机器人所处世界的当前视觉状态是什么?”。我们用一种在视角、物体配置以及(如有需要)物体类别之间都保持一致的稠密描述来回答这个问题。在语义分割这一较粗的层面上,亚马逊机器人挑战赛中的若干工作,用机器人通过图像级背景减除,自动完成数据收集和标注[18, 5, 4]。相比之下,本文使用基于三维重建的变化检测以及稠密的像素级对应,从而在训练中提供丰富得多的监督信号。
In the area of robot learning for a specific task there have been impressive works on end-to-end reinforcement learning [1, 19]. In these papers the goal is to learn a specific task, encoded with a reward function, whereas we learn a general task agnostic visual representation. There have also been several works focusing on grasping from RGB or depth images [3, 18, 2, 20]. These papers focus on successfully grasping any item out of a pile, and are effectively looking for graspable features. They have no consistent object representation or specific location on that object, and thus the robotic manipulation tasks we demonstrate in Section 5.4, e.g. grasping specific points on an object across potentially deformed object configurations, are out of scope for these works.
在面向特定任务的机器人学习方面,已有一些引人注目的端到端强化学习工作[1, 19]。这些论文的目标是学会由奖励函数编码的某一特定任务,而我们学习的是一种通用的、与任务无关的视觉表示。还有若干工作关注从 RGB 或深度图像中抓取[3, 18, 2, 20]。这些论文的重点是成功从一堆物品中抓起任意一件,实际上是在寻找可抓的特征。它们没有一致的物体表示,也没有物体上的指定位置,因此我们在第 5.4 节展示的机器人操作任务,例如在物体可能发生变形的配置下抓住指定点,并不在这些工作的范围内。
3 Methodology
3 方法
3.1 Preliminary: Self-Supervised Pixelwise Contrastive Loss
3.1 预备:自监督的像素级对比损失
We use self-supervised pixelwise contrastive loss, as developed in [7, 8]. This learns a dense visual descriptor mapping which maps a full-resolution RGB image, to a dense descriptor space, , where for each pixel we have a -dimensional descriptor vector. Training is performed in a Siamese fashion, where a pair of RGB images, and are sampled from one RGBD video, and many pixel matches and non-matches are generated from the pair of images. A pixel from image is a match with pixel from image if they correspond to the same vertex of the dense 3D reconstruction (Figure 1 (c-f)). The dense descriptor mapping is trained via pixelwise contrastive loss. The loss function aims to minimize the distance between descriptors corresponding to a match, while descriptors corresponding to a non-match should be at least a distance apart, where is a margin parameter. The dense descriptor mapping is used to map an image to descriptor space . Given a pixel we use to denote the descriptor corresponding to pixel in image . We simply round the real-valued pixel to the closest discrete pixel value , but any continuously-differentiable interpolation can be used for sub-pixel resolution. We denote as the distance between a pair of pixel descriptors: . At each iteration of training, a large number (on the order of 1 million total) of matches and non-matches are generated between images and . The images are mapped to corresponding descriptor images via and the loss function is
我们采用[7, 8]中提出的自监督像素级对比损失。它学习一种稠密视觉描述映射,把全分辨率 RGB 图像 映到稠密描述空间 ,使每个像素都对应一个 维描述向量。训练以孪生网络的方式进行:从同一段 RGBD 视频中采样一对 RGB 图像 和 ,再由这一对图像生成大量匹配与非匹配像素。若图像 中的像素 与图像 中的像素 对应于稠密三维重建中的同一顶点,则二者构成一对匹配(图 1(c–f))。稠密描述映射通过像素级对比损失来训练。损失函数的目标是缩小一对匹配所对应描述之间的距离,而一对非匹配所对应的描述至少应相距间隔 ,其中 是间隔参数。稠密描述映射 把图像 映到描述空间 。给定像素 ,我们用 表示图像 中该像素对应的描述。我们只是把实值像素 取整到最近的离散像素 ,但任何连续可微的插值都可用于亚像素分辨率。我们把一对像素描述之间的 距离记作 :。在每一轮训练中,会在图像 与 之间生成大量(总计约百万量级)匹配 与非匹配 。图像经 映到对应的描述图像,损失函数为
3.2 Training Procedures for Object-Centric Descriptors
3.2 面向物体的描述子训练步骤
Prior work [8] has used dynamic reconstruction [21] of raw RGBD data for only within-scene data association and remarkably showed that even without cross-scene data association, descriptors could be learned that were consistent across many dynamic scenes of the upper body of a human subject. While dynamic reconstruction is powerful, the challenges of topology changes [22] and difficulties of occlusion make it difficult to reliably deploy for an autonomous system. Schmidt et al. [8] also used data associations from static scene reconstructions for the task of relocalization in the same static environment. In contrast we sought to use only static reconstruction but seek consistency for dynamic objects. Other work [11] obtains dense descriptor consistency for a curated dataset of celebrity faces using only image warping for data association.
已有工作[8]对原始 RGBD 数据做动态重建[21],仅用于场景内的数据关联,并令人瞩目地表明:即便没有跨场景的数据关联,也能学到在同一人体上半身的许多动态场景之间保持一致的描述子。动态重建虽然强大,但拓扑变化[22]带来的挑战以及遮挡造成的困难,使它难以在自主系统中可靠部署。Schmidt 等人[8]也使用来自静态场景重建的数据关联,用于在同一静态环境中重定位。与之不同,我们希望只用静态重建,却为动态物体寻求一致性。另有工作[11]仅用图像扭曲做数据关联,便在一套经过整理的名人面孔数据上得到了稠密描述的一致性。
Using our robot mounted camera we are able to reliably collect high quality dense reconstructions for static scenes. Initially we applied only static-scene reconstruction to learn descriptors for specific objects, but we found that the learned object descriptors were not naturally consistent for challenging datasets with objects in significantly different configurations. Subsequently we developed techniques that leverage 3D reconstruction change detection, data augmentation, and loss function balancing to reliably produce consistent object representations with only static-scene data association for the wide variety of objects we have tested. These techniques also improve the precision of correspondences, as is discussed in Section 5.1. While we have tried many other ideas (see Appendix D.2), these are the techniques that were empirically found to significantly improve performance.
借助安装在机器人上的相机,我们能够为静态场景可靠地采集高质量的稠密重建。起初,我们只用静态场景重建来为特定物体学习描述子,但发现在物体处于明显不同配置的困难数据上,学到的物体描述并不能自然保持一致。此后,我们发展了一套技术,借助三维重建变化检测、数据增强和损失函数平衡,仅用静态场景的数据关联,就能在我们测试过的各类物体上可靠地得到一致的物体表示。这些技术也提高了对应的精度,详见第 5.1 节。我们还尝试过许多其他想法(见附录 D.2),但经过实证,真正显著提升性能的正是上述这些技术。
图 1:数据收集与训练流程概览。(a)用机械臂自动采集。(b)利用稠密三维重建做变化检测。(c)–(f)中,匹配以绿色表示,非匹配以红色表示。
Object masking via 3D change detection. Since we are trying to learn descriptors of objects that take up only a fraction of a full image, we observe significant improvements if the representational power of the models are focused on the objects rather than the backgrounds. A image contains 307,200 pixels but an image in our dataset may have as few as 1,000 to 10,000 of those pixels, or .3%–3%, that correspond to the object of interest. Initial testing with human-labeled object masks [23] showed that if matches for data associations were sampled only on the object (while non-matches were sampled from the full image) then correspondence performance was significantly improved. In order to provide autonomous object masking without any human input, we leverage our 3D reconstructions and results from the literature on 3D change detection [24] to recover the object-only part of the reconstruction (Figure 1b). Projecting this geometry into each camera frame yields object masks for each image. We want to emphasize that automatic object masking enables many other techniques in this paper, including: background domain randomization, cross-object loss, and synthetic multi-object scenes.
借助三维变化检测得到物体掩膜。 由于我们要学习的是只占整幅图像一小部分的物体描述,如果把模型的表示能力集中在物体而不是背景上,就会看到明显改进。一幅 的图像有 307,200 个像素,但在我们的数据中,一幅图像里对应于目标物体的像素可能只有 1,000 到 10,000 个,也就是 0.3%–3%。用人工标注的物体掩膜做的初步试验[23]表明:若数据关联中的匹配只在物体上采样(非匹配仍从整幅图像采样),对应性能会显著提高。为了在完全不需要人工输入的情况下自动得到物体掩膜,我们利用三维重建以及三维变化检测方面的已有结果[24],恢复重建中只属于物体的部分(图 1b)。把这部分几何投影到各个相机坐标系,即可得到每幅图像的物体掩膜。我们想强调的是,自动物体掩膜还使本文中的许多其他技术成为可能,包括:背景域随机化、跨物体损失,以及合成的多物体场景。
Background domain randomization. A strategy to encourage cross-scene consistency is to enforce that the learned descriptors are not reliant on the background. Since we have autonomously acquired object masks, we can domain randomize [25] the background (Figure 1c top) to encourage consistency – rather than memorizing the background (i.e. by describing the object by where it is relative to a table edge), the descriptors are forced to be representative of only the object.
背景域随机化。 一种促进跨场景一致性的策略,是迫使学到的描述不依赖背景。既然已经自动得到了物体掩膜,我们就可以对背景做域随机化[25](图 1c 上),以促进一致性——描述子不再去记忆背景(例如根据物体相对桌沿的位置来描述它),而必须只代表物体本身。
Hard-negative scaling. Although as in [8] we originally normalized and by and , respectively, we found that what we call the “hard-negative rate”, i.e. the percentage of sampled non-matches for which would quickly drop well below 1% during training. While not precisely hard-negative mining [26], we empirically measure improved performance if rather than scaling by , we adaptively scale by the number of hard negatives in the non-match sampling, , where is the indicator function:
难负样本缩放。 起初我们与[8]一样,分别用 和 来归一化 和 ,但发现我们所谓的“难负样本率”,也就是采样得到的非匹配中满足 的比例,在训练中会很快降到远低于 1%。这并不精确等同于难负样本挖掘[26],但我们实测发现:若不用 去缩放 ,而是按非匹配采样中难负样本的数量 做自适应缩放,性能会有提升,其中 是示性函数:
Data augmentation and normalization. While we collect only a modest number of scenes (4-10) per object or class, we ensure they are diverse in orientations, crops, and lighting conditions. We also applied synthetic 180-degree rotations randomly to our images. Additionally we find gains in performance by projecting all features to the unit sphere, i.e. when using high-dimensional descriptors spaces (i.e., more than ). This is explained further in Appendix D.1.
数据增强与归一化。 虽然每个物体或类别我们只采集数量不多的场景(4–10 个),但会确保它们在朝向、裁剪和光照条件上足够多样。我们还对图像随机施加合成的 180 度旋转。此外,在使用高维描述空间时(也就是 以上),把所有特征投影到单位球面,即 ,也能带来性能提升。这一点在附录 D.1 中有进一步说明。
3.3 Multi-Object Dense Descriptors
3.3 多物体稠密描述子
We of course would like robots to have dense visual models of more than just one object. When we began this work it wasn’t obvious to us what scale of changes to our training procedure or model architecture would be required in order to simultaneously (a) achieve individual single-object performance comparable to a single-object-only model, while also (b) learn dense visual descriptors for objects that are globally distinct – i.e., the bill of a hat would occupy a different place in descriptor space than the handle of a mug. To achieve distinctness, we introduce three strategies:
我们当然希望机器人拥有的不只是单个物体的稠密视觉模型。开始这项工作时,我们并不清楚需要对训练步骤或模型结构做多大幅度的改动,才能同时(a)使单个物体上的性能达到与只针对单物体的模型相当的水平,并且(b)为各物体学到在全局上可区分的稠密视觉描述——也就是说,帽檐在描述空间中应占据与杯柄不同的位置。为了实现可区分性,我们引入了三种策略:
i. Cross-object loss. The most direct way to ensure that different objects occupy different subsets of descriptor space is to directly impose cross-object loss (Figure 1d). Between two different objects, we know that each and every pair of pixels between them is a non-match. Accordingly we randomly select two images of two different objects, randomly sample many pixels from each object (enabled by object masking), and apply non-match loss (with hard-negative scaling) to all of these pixel pairs.
i. 跨物体损失。 要确保不同物体占据描述空间中不同的子集,最直接的办法就是直接施加跨物体损失(图 1d)。在两个不同物体之间,二者的每一对像素都是非匹配。因此,我们随机选取两个不同物体的两幅图像,从每个物体上随机采样许多像素(这由物体掩膜得以实现),并对所有这些像素对施加非匹配损失(同时做难负样本缩放)。
ii. Direct training on multi-object scenes. A nice property of pixelwise contrastive loss, with data associations provided by 3D geometry, is that we can directly train on multi-object, cluttered scenes without any individual object masks (Figure 1e). This is in contrast with training pixelwise semantic segmentation, which requires labels for each individual object in clutter that may be difficult to attain, i.e. through human labeling. With pixel-level data associations provided instead by 3D geometry, the sampling of matches and the loss function still makes sense, even in clutter.
ii. 直接在多物体场景上训练。 像素级对比损失的一个优点是,在由三维几何提供数据关联的情况下,我们可以直接在多物体、杂乱的场景上训练,而无需各个物体的单独掩膜(图 1e)。这与训练像素级语义分割不同:后者需要杂乱场景中每个物体的标签,而这些标签往往不易获得,例如需要靠人工标注。若改为由三维几何提供像素级数据关联,则匹配的采样和损失函数即使在杂乱场景中也仍然成立。
iii. Synthetic multi-object scenes. We can also synthetically create multi-object scenes by layering object masks [4]. To use dense data associations through synthetic image merging, we prune matches that become occluded during layering (Figure 1f). A benefit of this procedure is that we can create a combinatorial number of “multi-object” scenes from only single object-scenes, and can cover a wide range of occlusion types without collecting physical scenes for each.
iii. 合成的多物体场景。 我们也可以通过叠加物体掩膜来合成多物体场景[4]。为了在合成的图像合并中继续使用稠密数据关联,我们会剔除在叠加过程中被遮挡的匹配(图 1f)。这一做法的好处是,仅从单物体场景就能组合出大量“多物体”场景,并且无需为每一种情形采集真实场景,就能覆盖范围很广的遮挡类型。
4 Experimental
4 实验
Data Collection and Pre-Processing. The minimum requirement for raw data is to collect an RGBD video of an object or objects. Figure 1 shows our experimental setup; we utilize a 7-DOF robot arm (Kuka IIWA LBR) with an RGBD sensor (Primesense Carmine 1.09) mounted at the end-effector. With the robot arm, data collection can be highly automated, and we can achieve reliable camera poses by using forward kinematics along with knowledge of the camera extrinsic calibration. For dense reconstruction we use TSDF fusion [27] of the depth images with camera poses provided by forward kinematics. An alternative route to collecting data which does not require a calibrated robot is to use a dense SLAM method (for example, [28, 29]). In between collecting RGBD videos, the object of interest should be moved to a variety of configurations, and the lighting can be changed if desired. While for many of our data collections a human moved the object between configurations, we have also implemented and demonstrated (see our video) the robot autonomously rearranging the objects, which highly automates the object learning process. We employ a Schunk two-finger gripper and plan grasps directly on the object point cloud (Appendix C). If multiple different objects are used, currently the human must still switch the objects for the robot and indicate which scenes correspond to which object, but even this information could be automated by the robot picking objects from an auxiliary bin.
数据收集与预处理。 对原始数据的最低要求,是采集一段包含一个或多个物体的 RGBD 视频。图 1 给出了我们的实验设置:我们使用一台 7 自由度机械臂(Kuka IIWA LBR),并在末端执行器上安装 RGBD 传感器(Primesense Carmine 1.09)。借助机械臂,数据采集可以高度自动化,并且利用正运动学以及相机的外参标定,就能得到可靠的相机位姿。稠密重建方面,我们用正运动学给出的相机位姿,对深度图像做 TSDF 融合[27]。若不想依赖已标定的机器人,也可以改用稠密 SLAM 方法来采集数据(例如[28, 29])。在各段 RGBD 视频之间,应将目标物体摆放到多种配置,如有需要也可以改变光照。在我们的许多次采集中,是由人在各配置之间移动物体;不过我们也已经实现并演示(见视频)由机器人自主重新摆放物体,从而使物体学习过程高度自动化。我们使用 Schunk 两指夹爪,并直接在物体点云上规划抓取(附录 C)。若使用多个不同物体,目前仍需由人替机器人更换物体,并指明哪些场景对应哪个物体;不过即便这一信息,也可以通过让机器人从辅助料箱中抓取物体来自动完成。
Training Dense Descriptors. For training, at each iteration we randomly sample between some subset of specified image comparison types (Single Object Within Scene, Different Object Across Scene, Multi Object Within Scene, Synthetic Multi Object), and then sample some set of matches and non-matches for each. In this work, we use only static-scene reconstructions, so pixel matches between images can be easily found by raycasting and reprojecting against the dense 3D reconstruction model, and appropriately checking for occlusions and field-of-view constraints. For the dense descriptor mapping we train a 34-layer, stride-8 ResNet pretrained on ImageNet, but we expect any fully-convolutional network (FCN) that has shown effectiveness on semantic segmentation tasks to work well. Additional training details are contained in Appendix D.
训练稠密描述子。 训练时,每一轮我们都从若干指定的图像比较类型中随机采样(单物体场景内、不同物体跨场景、多物体场景内、合成多物体),再为每一种类型采样一组匹配与非匹配。在本工作中,我们只用静态场景重建,因此图像之间的像素匹配很容易通过对稠密三维重建模型做光线投射与再投影,并适当检查遮挡和视场约束来得到。稠密描述映射方面,我们训练一个在 ImageNet 上预训练的 34 层、步幅为 8 的 ResNet,但我们预期任何在语义分割任务上表现良好的全卷积网络(FCN)都能很好地工作。更多训练细节见附录 D。
5 Results
5 结果
Figure 2: Learned object descriptors can be consistent across significant deformation (a) and, if desired, across object classes (b-d). Shown for each (a) and (b-d) are RGB frames (top) and corresponding descriptor images (bottom) that are the direct output of a feed-forward pass through a trained network. (e)-(f) shows that we can learn descriptors for low texture objects, with the descriptors masked for clear visualization. Our object set is also summarized (right).
图 2:学到的物体描述可以在显著变形下保持一致(a),如果需要,也可以在物体类别之间保持一致(b–d)。对(a)以及(b–d),上排是 RGB 帧,下排是对应的描述图像,它们都是训练好的网络一次前向传播的直接输出。(e)–(f)表明我们也能为低纹理物体学习描述,这里对描述做了掩膜以便更清楚地显示。右侧还汇总了我们的物体集合。
5.1 Single-Object Dense Descriptors
5.1 单物体稠密描述子
We observe that with our training procedures described in Section 3.2, for a wide variety of objects we can acquire dense descriptors that are invariant to viewpoint, configuration, and deformation. The variety of objects includes moderately deformable objects such as soft plush toys, shoes, mugs, and hats, and can include very low-texture objects (Figure 2). Many of these objects were just grabbed from around the lab (including the authors’ and labmates’ shoes and hats), and dense visual models can be reliably trained with the same network architecture and training parameters. The techniques in Section 3.2 provide significant improvement in both (a) qualitative consistency over a wide variety of viewpoints, and (b) quantitative precision in correspondences. As with other works that learn pairwise mappings to some descriptor space [30], in practice performance can widely vary based on specific sampling of data associations and non-associations used during training. One way to quantitatively evaluate correspondence precision is with human-labeled (used only for evaluation; never for training) correspondences across two images of an object in different configurations. Given two images containing the same object and pixel locations , corresponding to the same physical point on the object, we can use our dense descriptors to estimate as :
我们观察到,采用第 3.2 节所述的训练步骤,可以在多种多样的物体上得到对视角、配置和变形都保持不变的稠密描述子。这些物体包括中等可变形的物体,例如柔软的毛绒玩具、鞋、马克杯和帽子,也可以包括纹理非常少的物体(图 2)。其中许多物体只是从实验室里随手拿来的(包括作者和同事的鞋和帽子),并且可以用同一套网络结构和训练参数可靠地训练出稠密视觉模型。第 3.2 节中的技术在两方面都带来了显著改进:(a)在多种多样的视角上具有定性的一致性,(b)对应关系具有定量的精度。与其他学习到某一描述空间的成对映射的工作一样[30],实际性能会因训练时数据关联与非关联的具体采样而有较大差异。一种定量评估对应精度的办法,是用人工标注的对应(仅用于评估,从不用于训练),这些对应来自同一物体处于不同配置的两幅图像。给定包含同一物体的两幅图像 ,以及对应于物体上同一物理点的像素位置 、,我们可以用稠密描述子把 估计为 :
Figure 3 (b-c) shows a quantitative comparison of ablative experiments, for four different training procedures described in Figure 3a. Our new standard single-object training procedure (standard-SO) performs significantly better than our implementation of prior work’s training procedures (Schmidt), and we isolate and measure significant improvement in correspondence precision for both object-masking and hard-negative scaling. We also find that for some low-texture objects, orientation randomization and background domain randomization are critical for attaining consistent object descriptors. Otherwise the model may learn to memorize which side of the object is closest the table, rather than a consistent object model (Figure 4b). Background domain randomization is most beneficial for smaller datasets, where it can significantly reduce overfitting and encourage consistency (Figure 4a); it is less critical for high-texture objects and larger datasets.
图 3(b–c)给出了消融实验的定量比较,对应图 3a 中描述的四种不同训练步骤。我们新的标准单物体训练步骤(standard-SO)明显优于我们对已有工作训练步骤的复现(Schmidt);我们还分别分离并测到,物体掩膜和难负样本缩放都能显著提高对应精度。我们还发现,对于一些低纹理物体,朝向随机化和背景域随机化对于得到一致的物体描述至关重要。否则,模型可能会去记住物体的哪一面最靠近桌面,而不是学到一个一致的物体模型(图 4b)。背景域随机化对较小的数据集最有益,能显著减轻过拟合并促进一致性(图 4a);对于高纹理物体和较大的数据集,它就不那么关键。
Figure 3: (a) table describing the network training procedures referenced in experiments. (standard-SO = “standard single object”. standard-SO-P is detailed in Appendix D.1). (b) Plots the cdf of the pixel distance (normalized by image diagonal, 800 for a image) between the best match and the true match , e.g. for standard-SO in 93% of image pairs the normalized pixel distance between and is less than 13%. All networks were trained on the same dataset. (c) Plots the cdf of the fraction of pixels of the object pixels with , i.e. they are closer in descriptor space to than the true match .
图 3:(a)表格说明实验中所引用的网络训练步骤。(standard-SO = “standard single object”,即标准单物体。standard-SO-P 的细节见附录 D.1)。(b)画出最佳匹配 与真实匹配 之间 像素距离(按图像对角线归一化,对 图像该对角线为 800)的累积分布函数;例如,对 standard-SO 而言,在 93% 的图像对中, 与 的归一化像素距离小于 13%。所有网络都在同一数据集上训练。(c)画出物体像素中满足 的像素 所占比例的累积分布函数,也就是在描述空间中比真实匹配 更靠近 的那些像素。
Figure 4: (a), with same axes as Figure 3b, compares standard-SO with without-DR, for which the only difference is that without-DR used no background domain randomization during training. The dataset used for (a) is of three objects, 4 scenes each. (b) shows that for a dataset containing 10 scenes of a drill, learned descriptors are inconsistent without background and orientation randomization during training (middle), but consistent with them (right).
图 4:(a)坐标轴与图 3b 相同,比较了 standard-SO 与 without-DR,二者的唯一差别是 without-DR 在训练时没有做背景域随机化。(a)所用数据集包含三个物体,每个物体 4 个场景。(b)表明,在包含一把电钻的 10 个场景的数据集上,若训练时不做背景和朝向随机化,学到的描述是不一致的(中),加上这些随机化后则是一致的(右)。
5.2 Multi-Object Dense Descriptors
5.2 多物体稠密描述子
An early observation during experimentation was that overlap in descriptor space naturally occurs if the same model is trained simultaneously on different singulated objects, where sampling of matches and non-matches was only performed within scene. Since there is no component of the loss function that requires different objects to occupy different subsets of descriptor space, the model maps them to an overlapping subset of descriptor space, distinct from the background but not each other (Figure 5a). Accordingly we sought to answer the question of whether or not we could separate these objects into unique parts of descriptor space.
实验中一个较早的观察是:如果同一个模型同时在不同的、已分离摆放的物体上训练,并且匹配与非匹配只在场景内采样,描述空间中就会自然出现重叠。因为损失函数中并没有一项要求不同物体占据描述空间中不同的子集,模型会把它们映到描述空间中彼此重叠的一个子集里,该子集与背景不同,但物体之间并不分开(图 5a)。因此我们想回答的问题是:能否把这些物体分到描述空间中各自独有的区域。
Figure 5: Comparison of training without any distinct object loss (a) vs. using cross-object loss (b). In (b), 50% of training iterations applied cross-object loss and 50% applied single-object within-scene loss, whereas (a) is 100% single-object within-scene loss. The plots show a scatter of the descriptors for 10,000 randomly-selected pixels for each of three distinct objects. Networks were trained with to allow direct cluster visualization. (c) Same axes as Figure 3 (a). All networks were trained on the same 3 object dataset. Networks with a number label were trained with cross object loss and the number denotes the descriptor dimension. no-cross-object is a network trained without cross object loss.
图 5:不使用任何使物体可区分的损失进行训练(a),与使用跨物体损失(b)的比较。在(b)中,50% 的训练轮次施加跨物体损失,50% 施加单物体场景内损失;而(a)则是 100% 的单物体场景内损失。图中给出三个不同物体各自随机选取的 10,000 个像素的描述散点。网络以 训练,以便直接观察聚类。(c)坐标轴与图 3(a)相同。所有网络都在同一套 3 物体数据上训练。带数字标注的网络使用了跨物体损失,数字表示描述维度。no-cross-object 是不使用跨物体损失训练的网络。
By applying cross-object loss (Section 3.3.i, training mode specific in Figure 3a), we can convincingly separate multiple objects such that they each occupy distinct subsets of descriptor space (Figure 5b). Note that cross-object loss is an extension of sampling across scene as opposed to only within scene. Given that we can separate objects in descriptor space, we next investigate: does the introduction of object distinctness significantly limit the ability of the models to achieve correspondence precision for each individual object? For multi-object datasets, we observe that there is a measurable decrease in correspondence precision for small-dimensional descriptor spaces when the cross-object loss is introduced, but we can recover correspondence precision by training slightly larger-dimensional descriptor spaces (Figure 5c). For the most part, 3-dimensional descriptor spaces were sufficient to achieve saturated (did not improve with higher-dimension) correspondence precision for single objects, yet this is often not the case for distinct multi-object networks.
通过施加跨物体损失(第 3.3.i 节,即图 3a 中的训练模式 specific),我们可以有说服力地把多个物体分开,使它们各自占据描述空间中不同的子集(图 5b)。需要指出的是,跨物体损失是把采样从仅在场景内扩展到跨场景。既然物体可以在描述空间中被分开,我们接下来考察:引入物体可区分性,是否会显著限制模型在每个单独物体上达到对应精度的能力?对多物体数据,我们观察到,在引入跨物体损失后,低维描述空间上的对应精度会有可测的下降,但只要训练稍高维的描述空间,就可以把对应精度恢复回来(图 5c)。在大多数情况下,三维描述空间已足以使单物体的对应精度达到饱和(再提高维度也不会改进);但对要求可区分的多物体网络来说,情况往往并非如此。
5.3 Selective Class Generalization or Instance Specificity
5.3 可选择的类内泛化或实例特异性
Surprisingly we find that when trained simultaneously on similar items of a class using training mode consistent, the learned descriptors naturally generalize well across sufficiently similar instances of the class. This result of converging descriptors across a class is similar to the surprising generalization observed for human datasets in [8, 11]. Here we show that we can obtain class consistent dense descriptors for 3 different classes of objects (hats, shoes, and mugs) trained with only static-scene data association. We observe that the descriptors are consistent despite considerable differences in color, texture, deformation, and even to some extent underlying shape. The training requirements are reasonably modest – only 6 instances of hats were used for training yet the descriptors generalize well to unseen hats, including a blue hat, a color never observed during training. The generalization extends to instances that a priori we thought would be failure modes: we expected the boot (Figure 6h) to be a failure mode but there is still reasonable consistency with other shoes. Sufficiently different objects are not well generalized, however – for example Baymax and Starbot (Figure 2e,f) are both anthropomorphic toys but we do not attain general descriptors for them. While initially we proposed research into further encouraging consistency within classes, for example by training a Dense Object Net to fool an instance-discriminator, the level of consistency that naturally emerges is remarkable and was sufficient for our desired levels of precision and applications.
出人意料的是,我们发现:当使用训练模式 consistent、同时在一类中相似的物品上训练时,学到的描述会在该类中足够相似的实例之间自然地泛化得很好。这种在一类中描述趋于一致的结果,类似于[8, 11]在人体数据上观察到的出人意料的泛化。这里我们表明,仅用静态场景的数据关联,就可以为 3 个不同的物体类别(帽子、鞋和马克杯)得到类内一致的稠密描述子。我们观察到,尽管在颜色、纹理、变形、甚至在一定程度上在底层形状上都有相当差异,这些描述仍然保持一致。训练需求也相当有限——帽子只用了 6 个实例来训练,描述却能很好地泛化到未见过的帽子,包括一顶蓝色的帽子,而这种颜色在训练中从未出现过。这种泛化还延伸到我们事先以为会失败的实例:我们曾预期靴子(图 6h)会是失败情形,但它与其他鞋之间仍有合理的一致性。不过,差异足够大的物体并不能很好地被泛化——例如 Baymax 和 Starbot(图 2e,f)虽然都是人形玩具,我们并没有为它们得到通用的描述。虽然我们起初还打算研究如何进一步促进类内一致性,例如训练一个 Dense Object Net 去骗过实例判别器,但自然涌现出的一致性已经相当可观,也足以达到我们期望的精度和应用水平。
For other applications, however, instance-specificity is desired. For example, what if you would like your robot to recognize a certain point on hat A as distinct from the comparable point on hat B? Although we could separate very distinct objects in multi-object settings as discussed in the previous section, it wasn’t obvious to us if we could satisfactorily separate objects of the same class. We observe, however, that by applying the multi-object techniques (specific in Figure 3) previously discussed, we can indeed learn distinct descriptors even for very similar objects in a class (Figure 6iv).
不过,在另一些应用中,人们需要的是实例特异性。例如,若希望机器人把帽子 A 上的某一点识别为与帽子 B 上的对应点不同,该怎么办?尽管如前一节所述,我们能够在多物体设定中把差异很大的物体分开,但能否满意地把同一类中的物体分开,对我们来说并不是显而易见的。然而我们观察到,只要采用前面讨论过的多物体技术(图 3 中的 specific),即使是一类中非常相似的物体,也确实能学到彼此不同的描述(图 6iv)。
5.4 Example Applications to Robotic Manipulation: Grasping Specific Points
5.4 机器人操作中的示例应用:抓住指定点
Here we demonstrate a variety of manipulation applications in grasping specific points on objects, where the point of interest is specified in a reference image. We emphasize there could be many other applications, as mentioned in the Conclusion. In our demonstrations, a user clicks on just one pixel in one reference image. Now the robot has the ability to autonomously identify the corresponding point in new scenes via Equation 6. Akin to other works with similarity learning in metric spaces [30], we set a simple threshold to determine whether a valid match exists. If a match is identified in the new scene we can instruct the robot to autonomously grasp this point by looking up the corresponding location in the point cloud and using simple geometric grasping techniques (details in Appendix C).
这里我们展示一类抓住物体上指定点的操作应用,其中兴趣点是在一张参考图像中指定的。我们想强调,还可能有许多其他应用,结论中也会提到。在我们的演示中,用户只需在一张参考图像上点击一个像素 。此后,机器人就能通过公式 6 在新场景中自主找出对应点。与其他在度量空间中做相似性学习的工作类似[30],我们设定一个简单阈值,以判断是否存在有效匹配。若在新场景中找到了匹配,就可以通过在点云中查找对应位置,并使用简单的几何抓取技术,让机器人自主抓住这一点(细节见附录 C)。
The particular novel components of these manipulation demonstrations are in grasping the visual corresponding points for arbitrary pixels that are either in different (potentially deformed) configurations (Fig. 6i-ii), general across instances of classes (Fig. 6iii), or instance-specific in clutter (Fig. 6iv). Our video best displays these tasks. Note that only a dense (as opposed to sparse) method can easily accommodate the arbitrary selection of interaction points, and class-generalization is out of scope for hand-designed descriptors such as SIFT. This is also out of scope for general grasp planners like [3, 18, 2, 20] which lack any visual object representation, and for segmentation based methods [18, 5, 4] since the visual representation provided by segmentation doesn’t capture any information beyond the object mask.
这些操作演示中真正新的部分在于:对任意选定的像素,抓住其视觉对应点,这些点既可以处于不同的(可能已变形的)配置中(图 6i–ii),也可以在一类的各实例之间通用(图 6iii),还可以在杂乱场景中具有实例特异性(图 6iv)。我们的视频最能展示这些任务。需要指出的是,只有稠密(而非稀疏)的方法才能方便地支持任意选择交互点,而且类内泛化也不在 SIFT 这类手工设计描述子的范围内。这对[3, 18, 2, 20]这类缺乏任何视觉物体表示的通用抓取规划器同样不适用;对基于分割的方法[18, 5, 4]也不适用,因为分割给出的视觉表示并不包含物体掩膜以外的信息。
Figure 6: Depiction of “grasp specific point” demonstrations. For each the user specifies a pixel in a single reference image, and the robot automatically grasps the best match in test configurations. For single-object demonstrations, two different points for the caterpillar object are shown: tail (i) and right ear (ii). Note that the “right-ear” demonstration is an example of the ability to break symmetry on reasonably symmetrical objects. For class generalization (iii), trained with consistent, the robot grasps the class-general point on a variety of instances. This was trained on only 4 shoes and extends to unseen instances of the shoe class, for example (iii-i). For instance-specificity (iv) trained with specific and augmented with synthetic multi object scenes (3.3.iii), the robot grasps this point on the specific instance even in clutter.
图 6:“抓住指定点”演示的示意。每一次都由用户在单张参考图像上指定一个像素,机器人再在测试配置中自动抓住最佳匹配。在单物体演示中,给出了毛毛虫玩具上的两个不同点:尾巴(i)和右耳(ii)。需要指出的是,“右耳”这一演示,是在相当对称的物体上打破对称性的一个例子。在类内泛化(iii)中,网络以 consistent 模式训练,机器人会在多种实例上抓住该类通用的点。这次训练只用了 4 只鞋,却能扩展到该类中未见过的实例,例如(iii-i)。在实例特异性(iv)中,网络以 specific 模式训练,并用合成多物体场景做增强(3.3.iii),即使在杂乱场景中,机器人也能抓住特定实例上的这一点。
6 Conclusion
6 结论
This work introduces Dense Object Nets as visual object representations which are useful for robotic manipulation and can be acquired with only robot self-supervision. Building on prior work on learning pixel-level data associations we develop new techniques for object-centricness, multi-object distinct descriptors, and learning dense descriptors by and for robotic manipulation. Without these object centric techniques we found that data associations from static-scene reconstructions were not sufficient to achieve consistent object descriptors. Our approach has enabled automated and reliable descriptor learning at scale for a wide variety of objects (47 objects, and 3 classes). We also show how learned dense descriptors can be extended to the multi object setting. With new contrastive techniques we are able to train Dense Object Nets that map different objects to different parts of descriptor space. Quantitative experiments show we can train these multi object networks while still retaining the performance of networks that do not distinguish objects. We also can learn class-general descriptors which generalize across different object instances, and demonstrated this result for three classes: shoes, hats, and mugs. Using class-general descriptors we demonstrate a robot transferring grasps across different instances of a class. Finally we demonstrate that our distinct-object techniques work even for objects which belong to the same class. This is demonstrated by the robot grasping a specific point on a target shoe in a cluttered pile of shoes. We believe Dense Object Nets can enable many new approaches to robotic manipulation, and are a novel object representation that addresses goals (i-iv) stated in the abstract. In future work we are interested to explore new approaches to solving manipulation problems that exploit the dense visual information that learned dense descriptors provide, and how these dense descriptors can benefit other types of robot learning, e.g. learning how to grasp, manipulate and place a set of objects of interest.
这项工作提出 Dense Object Nets,作为一种对机器人操作有用、并且只需通过机器人自监督就能获得的视觉物体表示。在已有的像素级数据关联学习工作之上,我们发展了面向物体性、多物体可区分描述子,以及为机器人操作、也由机器人操作来学习稠密描述子的新技术。如果没有这些面向物体的技术,我们发现仅凭静态场景重建得到的数据关联,并不足以得到一致的物体描述。我们的方法使描述子学习能够在多种多样的物体上(47 个物体、3 个类别)自动且可靠地规模化进行。我们还表明,学到的稠密描述子可以扩展到多物体设定。借助新的对比技术,我们能够训练出把不同物体映到描述空间不同区域的 Dense Object Nets。定量实验表明,我们可以在训练这些多物体网络的同时,仍保持那些并不区分物体的网络的性能。我们也可以学到在不同物体实例之间泛化的类通用描述,并在鞋、帽子和马克杯这三个类别上展示了这一结果。利用类通用描述,我们演示了机器人把抓取迁移到同一类的不同实例上。最后我们表明,用于区分物体的技术即使对属于同一类的物体也行之有效。这一点由机器人在一堆杂乱的鞋中抓住目标鞋上的指定点来展示。我们相信,Dense Object Nets 能够为机器人操作带来许多新的途径,并且是一种能够回应摘要中目标(i–iv)的新物体表示。在未来工作中,我们希望探索利用学到的稠密描述所提供的稠密视觉信息来解决操作问题的新方法,以及这些稠密描述子如何能够惠及其他类型的机器人学习,例如学习如何抓取、操作并放置一组目标物体。
See video (https://youtu.be/L5UW1VapKNE) for extensive videos of the different types of robot picking.
更多不同类型的机器人抓取视频见 https://youtu.be/L5UW1VapKNE 。
Acknowledgments
致谢
The authors thank Duy-Nguyen Ta (calibration), Alex Alspach (hardware), Yunzhu Li (training techniques), Greg Izatt (robot software), and Pat Marion (perception and visualization) for their help. We also thank Tanner Schmidt for helpful comments in preparing the paper. This work was supported by: Army Research Office, Sponsor Award No. W911NF-15-1-0166; Draper Laboratory Incorporated, Sponsor Award No. SC001-0000001002; Lincoln Laboratory/Air Force, Sponsor Award No. 7000374874; Amazon Research Award, 2D-01029900; Lockheed Martin Corporation, Sponsor Award No. RPP2016-002. Views expressed in the paper are not endorsed by the sponsors.
作者感谢 Duy-Nguyen Ta(标定)、Alex Alspach(硬件)、Yunzhu Li(训练技术)、Greg Izatt(机器人软件)以及 Pat Marion(感知与可视化)提供的帮助。我们也感谢 Tanner Schmidt 在论文撰写过程中给出的有益意见。本工作得到以下资助:Army Research Office,资助编号 W911NF-15-1-0166;Draper Laboratory Incorporated,资助编号 SC001-0000001002;Lincoln Laboratory/Air Force,资助编号 7000374874;Amazon Research Award,2D-01029900;Lockheed Martin Corporation,资助编号 RPP2016-002。文中观点不代表资助方立场。
References
参考文献
[1] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334–1373, 2016.
[2] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv:1703.09312, 2017.
[3] M. Gualtieri, A. ten Pas, K. Saenko, and R. Platt. High precision grasp pose detection in dense clutter. In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on, pages 598–605. IEEE, 2016.
[4] M. Schwarz, C. Lenz, G. M. Garcıa, S. Koo, A. S. Periyasamy, M. Schreiber, and S. Behnke. Fast object learning and dual-arm coordination for cluttered stowing, picking, and packing. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[5] A. Milan, T. Pham, K. Vijay, D. Morrison, A. Tow, L. Liu, J. Erskine, R. Grinover, A. Gurman, T. Hunn, et al. Semantic segmentation from limited training data. arXiv preprint arXiv:1709.07665, 2017.
[6] E. Jang, S. Vijaynarasimhan, P. Pastor, J. Ibarz, and S. Levine. End-to-end learning of semantic grasping. arXiv preprint arXiv:1707.01932, 2017.
[7] C. B. Choy, J. Gwak, S. Savarese, and M. Chandraker. Universal correspondence network. In Advances in Neural Information Processing Systems, pages 2414–2422, 2016.
[8] T. Schmidt, R. Newcombe, and D. Fox. Self-supervised visual descriptor learning for dense correspondence. IEEE Robotics and Automation Letters, 2(2):420–427, 2017.
[9] C. Liu, J. Yuen, and A. Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE transactions on pattern analysis and machine intelligence, 33(5):978–994, 2011.
[10] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1735–1742. IEEE, 2006.
[11] J. Thewlis, H. Bilen, and A. Vedaldi. Unsupervised learning of object frames by dense equivariant image labelling. In Advances in Neural Information Processing Systems, pages 844–855, 2017.
[12] A. Zeng, S. Song, M. Nießner, M. Fisher, J. Xiao, and T. Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 199–208. IEEE, 2017.
[13] J. Taylor, J. Shotton, T. Sharp, and A. Fitzgibbon. The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 103–110. IEEE, 2012.
[14] J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 2930–2937. IEEE, 2013.
[15] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, and C. Rother. Learning 6d object pose estimation using 3d object coordinates. In European conference on computer vision, pages 536–551. Springer, 2014.
[16] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 2786–2793. IEEE, 2017.
[17] A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine. Combining self-supervised learning and imitation for vision-based rope manipulation. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 2146–2153. IEEE, 2017.
[18] A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. arXiv preprint arXiv:1710.01330, 2017.
[19] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. arXiv preprint arXiv:1803.09956, 2018.
[20] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In Robotics and Automation (ICRA), 2016 IEEE International Conference on, pages 3406–3413. IEEE, 2016.
[21] R. A. Newcombe, D. Fox, and S. M. Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 343–352, 2015.
[22] W. Gao and R. Tedrake. Surfelwarp: Efficient non-volumetric single view dynamic reconstruction. Robotics: Science and Systems, 2018.
[23] P. Marion, P. R. Florence, L. Manuelli, and R. Tedrake. Labelfusion: A pipeline for generating ground truth labels for real rgbd data of cluttered scenes. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[24] R. Finman, T. Whelan, M. Kaess, and J. J. Leonard. Toward lifelong object segmentation from change detection in dense rgb-d maps. In Mobile Robots (ECMR), 2013 European Conference on, pages 178–185. IEEE, 2013.
[25] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on, pages 23–30. IEEE, 2017.
[26] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence, 32(9):1627–1645, 2010.
[27] B. Curless and M. Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303–312. ACM, 1996.
[28] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In Mixed and augmented reality (ISMAR), 2011 10th IEEE international symposium on, pages 127–136. IEEE, 2011.
[29] T. Whelan, S. Leutenegger, R. Salas-Moreno, B. Glocker, and A. Davison. Elasticfusion: Dense slam without a pose graph. Robotics: Science and Systems, 2015.
[30] F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
图 7:(a)Kuka IIWA LRB 机械臂。(b)装有 Primesense Carmine 1.09 的 Schunk WSG 50 夹爪。
Appendix A Experimental Hardware
附录 A 实验硬件
All of our data was collected using an RGBD camera mounted on the end effector of a 7 DOF robot arm (see Figure 7). We used a Kuka IIWA LBR robot with a Schunk WSG 50 parallel jaw gripper. A Primesense Carmine 1.09 RGBD sensor was mounted to the Schunk gripper and precisely calibrated for both intrinsics and extrinsics.
我们的全部数据都是用安装在 7 自由度机械臂末端执行器上的 RGBD 相机采集的(见图 7)。我们使用的是配备 Schunk WSG 50 平行爪夹爪的 Kuka IIWA LBR 机械臂。Primesense Carmine 1.09 RGBD 传感器安装在 Schunk 夹爪上,并对其内参和外参都做了精确标定。
Appendix B Experimental Setup: Data Collection and Pre-Processing
附录 B 实验设置:数据收集与预处理
As discussed in the paper all of our data consists of 3D dense reconstructions of a static scene. To collect data for a single scene we place an object (or set of objects) on a table in front of the robot. We then perform a scanning pattern with the robot which allows the camera to capture many different viewpoints of the static scene. This procedure takes about 70 seconds during which approximately 2100 RGBD frames are captured (the sensor outputs at 30Hz). Using the forward kinematics of the robot, together with our camera extrinsic calibration, we also record the precise camera pose corresponding to each image. Since we have the camera poses corresponding to each depth image we use TSDF fusion [27] to obtain a dense 3D reconstruction.
如正文所述,我们的全部数据都是静态场景的三维稠密重建。采集单个场景的数据时,我们把一个(或一组)物体放在机器人前方的桌子上。随后让机器人执行扫描轨迹,使相机能够从许多不同视角拍摄这一静态场景。这一过程大约需要 70 秒,其间会采集大约 2100 帧 RGBD(传感器以 30Hz 输出)。利用机器人的正运动学以及相机外参标定,我们还会记录与每幅图像对应的精确相机位姿。由于每个深度图像都有对应的相机位姿,我们用 TSDF 融合[27]得到稠密的三维重建。
Although we use our robot’s forward kinematics to produce the dense 3D reconstruction together with camera pose tracking, any dense SLAM method (such as [28]) could be used instead. In practice we found that using the robot’s forward kinematics to obtain camera pose estimates produces very reliable 3D reconstructions which are robust to lack of geometric or RGB texture, varying lighting conditions, etc. Next we obtain new depth images for each recorded camera frame by rendering against the 3D reconstruction using our camera pose estimates. This step produces depth images which are globally consistent and free of artifacts (i.e. missing depth values, noise etc.). This keeps the process of finding correspondences across RGB images as a simple operation between poses and depth images. To enable the specific training techniques discussed in the paper we also need to know which parts of the scene correspond to the objects of interest. To do this we implemented the change detection technique of [24]. In practice since all of our data was collected on a tabletop, and our reconstructions can be easily globally aligned (due to the fact that we know the global camera poses from the robot’s forward kinematics) we can simply crop the reconstruction to the area above the table. Once we have isolated the part of the reconstruction corresponding to the objects of interest, we can easily render binary masks via the same procedure as was used to generate the depth images.
虽然我们用机器人的正运动学来生成稠密三维重建并跟踪相机位姿,但也可以改用任何稠密 SLAM 方法(例如[28])。实践中我们发现,用机器人正运动学估计相机位姿,能够得到非常可靠的三维重建,并且对几何或 RGB 纹理不足、光照变化等情况都较为稳健。接下来,我们用估计得到的相机位姿,对三维重建进行渲染,从而为每一帧已记录的相机图像得到新的深度图。这一步产生的深度图在全局上是一致的,并且没有伪影(例如缺失的深度值、噪声等)。这样,在 RGB 图像之间寻找对应就变成了位姿与深度图之间的简单运算。为了支持文中讨论的那些具体训练技术,我们还需要知道场景中哪些部分对应于目标物体。为此,我们实现了[24]中的变化检测技术。实践中,由于我们的全部数据都在桌面上采集,并且重建可以很容易地做全局对齐(因为我们从机器人正运动学中知道全局相机位姿),只需把重建裁到桌面以上的区域即可。一旦分离出重建中对应于目标物体的部分,就可以用与生成深度图相同的步骤,方便地渲染出二值掩膜。
Our RGBD sensor captures images at 30Hz, so we downsample the images to avoid having images which are too visually similar. Specifically we downsample images so that the camera poses are sufficiently different (at least 5cm of translation, or 10 degrees of rotation). After downsampling we are left with approximately 315 images per scene.
我们的 RGBD 传感器以 30Hz 采集图像,因此会对图像做下采样,以避免图像在视觉上过于相似。具体来说,我们下采样到相机位姿有足够差异(至少 5 厘米的平移,或 10 度的旋转)。下采样后,每个场景大约剩下 315 幅图像。
In between collecting RGBD videos, the object of interest should be moved to a variety of configurations, and the lighting can be changed if desired. While for many of our data collections a human moved the object between configurations, we have also implemented and demonstrated (see our video) the robot autonomously rearranging the objects, which highly automates the object learning process. We employ a Schunk two-finger gripper and plan grasps directly on the object point cloud (see Appendix C). If multiple different objects are used, currently the human must still switch the objects for the robot and indicate which scenes correspond to which object, but even this information could be automated by the robot picking objects from an auxiliary bin and use continuity to simply identify which logs are of the same object.
在各段 RGBD 视频之间,应将目标物体摆放到多种配置,如有需要也可以改变光照。在我们的许多次采集中,是由人在各配置之间移动物体;不过我们也已经实现并演示(见视频)由机器人自主重新摆放物体,从而使物体学习过程高度自动化。我们使用 Schunk 两指夹爪,并直接在物体点云上规划抓取(见附录 C)。若使用多个不同物体,目前仍需由人替机器人更换物体,并指明哪些场景对应哪个物体;不过即便这一信息,也可以通过让机器人从辅助料箱中抓取物体,并利用连续性来简单识别哪些记录属于同一物体,从而自动完成。
Appendix C Grasping Pipeline
附录 C 抓取流程
While our learned visual descriptors can help us determine where to grasp, they can’t be used during the bootstrapping phase before visual learning has occurred, and they don’t constrain the 6DOF orientation of the gripper. Accordingly to choose grasps our pipeline employs simple geometric point cloud based techniques. There are two types of grasping that are performed in the paper. The first is performed while grasping the object to automatically reorient it during data collection, during the visual learning process. To achieve this we first use the depth images and camera poses to fuse a point cloud of the scene. We then randomly sample many grasps on the point cloud and prune those that are in collision. The remaining collision free grasps are then scored by a simple geometric criterion that evaluates the grasps for antipodality. The highest scoring grasp is then chosen for execution. The second type of grasping is, as in Section 5.4 of the paper, when we are attempting to grasp a specific point. In this case the robot moves to a handful of pre-specified poses and records those RGBD images. The RGB images are used to look up the best descriptor match and determine a pixel space location to grasp in one image, and the corresponding depth image and camera pose are used to determine the 3D location of this point. The handful of depth images are also fused into a point cloud, and the grasping procedure is almost the same as the first type, with the slight modification that all the randomly sampled grasps are centered around the target grasp point. Although there are a variety of learning based grasping algorithms [3, 2] that could have been used, we found that our simple geometric based grasp planning was sufficient for the tasks at hand.
学到的视觉描述子虽然有助于决定在何处抓取,但在视觉学习尚未发生的启动阶段并不能使用,而且它们也不约束夹爪的 6 自由度朝向。因此,在选择抓取时,我们的流程采用基于点云的简单几何方法。本文中进行的抓取有两种。第一种是在视觉学习过程中、采集数据时为了自动重新摆放物体而去抓取该物体。为此,我们首先用深度图像和相机位姿融合成场景点云,然后在点云上随机采样许多抓取,并剔除发生碰撞的那些。剩下的无碰撞抓取再用一个简单的几何准则打分,该准则评估抓取的对跖性。随后选择得分最高的抓取执行。第二种抓取如正文第 5.4 节所述,是要抓住一个指定点。此时,机器人移动到若干预先指定的位姿,并记录这些 RGBD 图像。RGB 图像用于查找最佳描述匹配,并在其中一幅图像中确定要抓取的像素位置;对应的深度图像和相机位姿则用于确定该点的三维位置。这若干幅深度图像也会融合成点云,抓取步骤与第一种几乎相同,只是略作修改:所有随机采样的抓取都以目标抓取点为中心。虽然也可以使用各种基于学习的抓取算法[3, 2],但我们发现,这种简单的基于几何的抓取规划已足以完成手头的任务。
Appendix D Network Architecture and Training Details
附录 D 网络结构与训练细节
For our network we use 34-layer, stride-8 ResNet (pretrained on ImageNet), and then bilinearly upsample to produce a full resolution image.
网络方面,我们使用 34 层、步幅为 8 的 ResNet(在 ImageNet 上预训练),再双线性上采样,得到全分辨率的 图像。
For training, at each iteration we randomly sample between some specified subset of specified image comparison types (Single Object Within Scene, Different Object Across Scene, Multi Object Within Scene, Synthetic Multi Object). The weighting between how often each type is chosen is done via specifying their probabilities of being selected. Once the type has been sampled we then sample some set of matches and non-matches for each (around 1 million in total). Each step of the optimizer uses a single pair of images. All the networks were trained with the same optimizer settings. Networks were trained for 3500 steps using an Adam optimizer with a weight decay of . Training was performed on a single Nvidia 1080 Ti or Titan Xp, a single step takes approximately 0.22 seconds, i.e. approximately 13 minutes, and so together with collecting a handful of scenes the entire training for a new object can take 20 minutes. The learning rate was set to and dropped by 0.9 every 250 iterations. The networks trained with procedure specific used a 50-50 split of within scene image pairs and across scene image pairs 50% of the time. For the network used to grasp the heel of the red/brown shoe in Section 5.4 we sampled equally the three data types (Single Object Within Scene, Different Object Across Scene, Synthetic Multi Object).
训练时,每一轮我们都从若干指定的图像比较类型中随机采样(单物体场景内、不同物体跨场景、多物体场景内、合成多物体)。各类型被选中的频率,通过指定其被选概率来加权。一旦采样得到类型,再为该类型采样一组匹配与非匹配(总计大约 100 万个)。优化器的每一步使用一对图像。所有网络都用相同的优化器设置训练。网络用 Adam 优化器训练 3500 步,权重衰减为 。训练在一块 Nvidia 1080 Ti 或 Titan Xp 上进行,单步大约需要 0.22 秒,也就是大约 13 分钟;因此再加上采集若干场景,为一个新物体完成全部训练大约需要 20 分钟。学习率设为 ,每 250 次迭代下降为原来的 0.9。以 specific 步骤训练的网络,有一半时间使用场景内图像对与跨场景图像对各占 50% 的划分。对于第 5.4 节中用于抓住红/棕色鞋后跟的网络,我们对三种数据类型做均等采样(单物体场景内、不同物体跨场景、合成多物体)。
D.1 Descriptor Projection to Unit Sphere
D.1 把描述投影到单位球面
There is one additional small feature we discovered prior to camera-ready submission that gives substantial quantitative performance gains, although it was not used nor needed for most of the experiments including all hardware experiments. As is standard in many metric learning works, for example [30], we can add a simple parameterless normalization layer in which we project all features to the unit sphere, . This is contrast to channel-wise normalization mentioned in [7]. Given that there is a projection to the unit-sphere manifold, higher dimensions are needed in order to see improvements. While without unit-sphere projection, we see saturation of performance at around for single-object descriptors, with unit-sphere projection we see significant gains in going from to even for single-object descriptors. Further work could investigate the effect of unit-sphere projection on descriptor consistency across classes. The network marked as standard-SO-P in Figures 3b and 3c was trained identically to standard-SO, but with the unit sphere projection, and with .
在准备相机就绪版本之前,我们还发现一个额外的小设计,能够带来可观的定量性能提升,尽管大多数实验、包括全部硬件实验都没有使用、也并不需要它。与许多度量学习工作中的通行做法一样,例如[30],我们可以加入一个简单的、不含参数的归一化层,把所有特征投影到单位球面,即 。这与[7]中提到的按通道归一化不同。由于投影到了单位球面流形,需要更高的维度才能看到改进。若不做单位球面投影,单物体描述子的性能大约在 时就会饱和;而做了单位球面投影后,即使对单物体描述子,从 增到 也能看到显著提升。后续工作可以考察单位球面投影对跨类别描述一致性的影响。图 3b 和 3c 中标记为 standard-SO-P 的网络,训练方式与 standard-SO 相同,只是加入了单位球面投影,并且取 。
D.2 Additional Approaches Which Did not Improve Performance
D.2 未能提升性能的其他做法
During the course of our experimentation we found that the network architecture and training procedure outlined in Section 3 gave the best performance. However we also tried a variety of other network architectures and loss functions which did not improve performance. We discuss a few of them here.
在实验过程中,我们发现第 3 节给出的网络结构和训练步骤效果最好。不过我们也尝试过多种其他网络结构和损失函数,它们并没有带来性能提升。这里讨论其中几种。
Triplet loss [30] uses a triplet loss instead of the contrastive loss. We implemented a pixelwise version of this triplet loss, but found that it actually reduced performance on our dataset.
三元组损失[30]用三元组损失代替对比损失。我们实现了这种三元组损失的像素级版本,但发现它实际上降低了在我们数据上的性能。
Scaling loss by pixel distance Our loss function tries to ensure that non-matches have descriptors that are at least a margin apart. It can be hard, however, for the network to take two pixels that are next to each other, and assign them significantly different descriptors. In an effort to try to less heavily penalize non-matches that are close to the true match, we introduced an additional scaling term using the pixel distance. Following the notation from Section 3 let , correspond to a non-match. Additionally, using the notation of Equation 6 let correspond to the true match for . Define to be the distance, measured in pixels, between and . The scaled non-match loss is now defined as
按像素距离缩放损失 我们的损失函数试图保证非匹配的描述至少相距间隔 。不过,让网络把相邻的两个像素赋成明显不同的描述,可能相当困难。为了减轻对靠近真实匹配的那些非匹配的惩罚,我们引入了一个利用 像素距离的额外缩放项。沿用第 3 节的记号,设 、 对应于一对非匹配。另外,沿用公式 6 的记号,设 为 的真实匹配。定义 为 与 之间以像素计的 距离。经过缩放的非匹配损失现定义为
where is the pixel distance at which this additional loss component saturates. Parameter sweeps on , where is the original un-scaled loss function, did not show significant differences in performance. A potentially useful extension for future work would be to try scaling by the geodesic distance in the 3D reconstructios.
其中 是这一额外损失项达到饱和时的像素距离。对 做参数扫描时,以 对应于原来未经缩放的损失函数,并未表现出显著的性能差异。对未来工作而言,一个可能有用的扩展是尝试按三维重建中的测地距离来缩放。
Convolutional spatial transformer: We implemented a convolutional spatial transformer, as described in [7]. The convolutional spatial transformer is meant to help the network achieve scale and rotation invariance for each feature. We did not find any significant performance gains with our implementation of such a convolutional spatial transformer. A hypothesis for why we did not see performance gains is that our used network architecture (34-layer ResNet) was significantly deeper than the architecture used in [7], and our data collection and augmentation provided significant variety in scale and rotation – accordingly, our network must approximate scale and rotation invariance in order to fit the training data.
卷积空间变换器: 我们实现了[7]中所述的卷积空间变换器。卷积空间变换器旨在帮助网络使每个特征具有尺度和旋转不变性。以我们的实现来看,这种卷积空间变换器并没有带来显著的性能提升。关于为何未见提升,一个假设是:我们使用的网络结构(34 层 ResNet)明显深于[7]中使用的结构,而且我们的数据采集和增强在尺度和旋转上已提供了相当的多样性——因此,网络为了拟合训练数据,本身就必须近似具备尺度和旋转不变性。
Ratios for sampling non-matches: Given a pixel corresponding to a point on an object, non-matches can either be on the object, or on the background. Let denote object pixels in image , and denote background pixels. During training we experimented with varying the fraction of non-matches which lie in vs. . In general we found performance was insensitive to this ratio as long as the fraction of non-matches in was between 25% to 75%.
采样非匹配的比例: 给定对应于物体上某一点的像素 ,非匹配 既可以落在物体上,也可以落在背景上。用 表示图像 中的物体像素,用 表示背景像素。训练中,我们尝试过改变非匹配 落在 与 中的比例。总体而言,只要落在 中的非匹配比例介于 25% 到 75% 之间,性能对这一比例并不敏感。






