DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表

暂无章节

返回论文列表
目录

DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion

收录于 CVPR 2019 会议论文 / Poster;2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition;DOI 10.1109/cvpr.2019.00346;页码 3343–3352被引 1,453 次
打开论文来源

DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion

Chen Wang² Danfei Xu¹ Yuke Zhu¹ Roberto Martín-Martín¹
Cewu Lu² Li Fei-Fei¹ Silvio Savarese¹

¹Department of Computer Science, Stanford University
²Department of Computer Science, Shanghai Jiao Tong University

Abstract

摘要

A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources. Prior works either extract information from the RGB image and depth separately or use costly post-processing steps, limiting their performances in highly cluttered scenes and real-time applications. In this work, we present DenseFusion, a generic framework for estimating 6D pose of a set of known objects from RGB-D images. DenseFusion is a heterogeneous architecture that processes the two data sources individually and uses a novel dense fusion network to extract pixel-wise dense feature embedding, from which the pose is estimated. Furthermore, we integrate an end-to-end iterative pose refinement procedure that further improves the pose estimation while achieving near real-time inference. Our experiments show that our method outperforms state-of-the-art approaches in two datasets, YCB-Video and LineMOD. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose. Our code and video are available at https://sites.google.com/view/densefusion/.

从 RGB-D 图像估计物体的 6D 位姿,一个关键技术难点是如何把这两种互为补充的数据真正用起来。已有工作要么从彩色图和深度里分别提取信息,要么再接代价很高的后处理,在严重遮挡的场景和实时应用里都会受限。本文提出 DenseFusion,一个从 RGB-D 图像估计一组已知物体 6D 位姿的通用框架。DenseFusion 采用异构结构:两种数据源各自处理,再用一种新的稠密融合网络提取逐像素的稠密特征嵌入,并据此估计位姿。我们还集成了一个可以端到端训练的迭代位姿精炼过程,在进一步提高估计精度的同时,推理速度接近实时。实验表明,该方法在 YCB-Video 和 LineMOD 两个数据集上超过了当时最优的方法。我们还把它部署到真机上,依据估计出的位姿完成抓取和操作。代码与视频见 https://sites.google.com/view/densefusion/。

  1. Introduction
  1. 引言

6D object pose estimation is the crux to many important real-world applications, such as robotic grasping and manipulation [7, 35, 44], autonomous navigation [6, 11, 42], and augmented reality [19, 20]. Ideally, a solution should deal with objects of varying shape and texture, show robustness towards heavy occlusion, sensor noise, and changing lighting conditions, while achieving the speed requirement of real-time tasks. The advent of cheap RGB-D sensors has enabled methods that infer poses of low-textured objects even in poorly-lighted environments more accurately than RGB-only methods. Nonetheless, it is difficult for existing methods to satisfy the requirements of accurate pose estimation and fast inference simultaneously.

6D 物体位姿估计是许多重要实际应用的关键,例如机器人抓取与操作 [7, 35, 44]、自主导航 [6, 11, 42] 以及增强现实 [19, 20]。一套理想的方法应当能处理形状和纹理各异的物体,在严重遮挡、传感器噪声和光照变化下保持稳健,同时又满足实时任务的速度要求。便宜的 RGB-D 传感器出现后,人们已经可以在低纹理、弱光环境里,比纯 RGB 方法更准确地推断位姿。尽管如此,现有方法仍然很难同时做到估计准、推理快。

A robot reaching toward tabletop objects with 6D pose boxes, plus RGB-D insets

Figure 1. We develop an end-to-end deep network model for 6D pose estimation from RGB-D data, which performs fast and accurate predictions for real-time applications such as robot grasping and manipulation.

图 1. 我们提出一个端到端的深度网络,从 RGB-D 数据估计 6D 位姿,能够快速而准确地给出预测,供机器人抓取和操作这类实时应用使用。

Classical approaches first extract features from RGB-D data and perform correspondence grouping and hypothesis verification [3, 12, 13, 15, 26, 33, 38]. However, the reliance on handcrafted features and fixed matching procedures have limited their empirical performances in presence of heavy occlusion and lighting variation. Recent success in visual recognition has inspired a family of data-driven methods that use deep networks for pose estimation from RGB-D inputs, such as PoseCNN [41] and MCN [16].

经典方法先从 RGB-D 数据中提取特征,再做对应分组和假设验证 [3, 12, 13, 15, 26, 33, 38]。不过,它们依赖手工特征和固定的匹配流程,一旦遇到严重遮挡或光照变化,实测表现就会受限。视觉识别方面的进展随后启发了一批数据驱动方法,用深度网络从 RGB-D 输入估计位姿,例如 PoseCNN [41] 和 MCN [16]。

However, these methods require elaborate post-hoc refinement steps to fully utilize the 3D information, such as a highly customized Iterative Closest Point (ICP) [2] procedure in PoseCNN and a multi-view hypothesis verification scheme in MCN. These refinement steps cannot be optimized jointly with the final objective and are prohibitively slow for real-time applications. In the context of autonomous driving, a third family of solutions has been proposed to better exploit the complementary nature of color and depth information from RGB-D data with heterogeneous architectures, i.e. using different network structures to process 2D and 3D data [21, 23, 42]. The idea is to independently extract appearance and geometry information from the RGB image and the point cloud using a CNN and a PointNet [24] architecture, respectively, and then fuse the information of the two data sources. However, the existing methods perform the fusion on a global or bounding-box level, which is not optimal for representing the local appearance and geometry of the objects for 6D pose estimation, especially when the objects are partially occluded.

不过,这些方法要充分用上三维信息,往往还得再接精细的事后精炼,例如 PoseCNN 里高度定制的迭代最近点(ICP)[2] 流程,以及 MCN 里的多视图假设验证。这些精炼步骤不能和最终目标一起优化,对实时应用来说也太慢。在自动驾驶的语境里,又出现了第三类做法:用异构结构更好地利用 RGB-D 中颜色与深度的互补性,也就是用不同的网络分别处理二维和三维数据 [21, 23, 42]。其思路是分别用 CNN 和 PointNet [24] 从彩色图和点云中独立提取外观与几何信息,再把两种数据源融合起来。但现有方法多在全局或包围盒层面做融合,这对表示物体的局部外观和几何并不理想,尤其是在物体被部分遮挡的时候,不利于 6D 位姿估计。

In this work, we propose an end-to-end deep learning approach for estimating 6-DoF poses of known objects from RGB-D inputs. The core of our approach is to embed and fuse RGB values and point clouds at a per-pixel level, as opposed to prior work which uses image crops to compute global features [42] or 2D bounding boxes [23]. This per-pixel fusion scheme enables our model to explicitly reason about the local appearance and geometry information, which is essential to handling heavy occlusion. The dense nature of our fusion scheme also allows us to model the confidence of the pose predictions from each fused image and depth feature, such that the model can make a smarter decision of the final pose estimation. Furthermore, we introduce an iterative pose refinement procedure that can be seamlessly integrated into our end-to-end 6D pose estimation framework. Our method is both efficient and accurate, our inference procedure takes only 0.06 second per frame, more than 200x faster than PoseCNN + ICP [41] (11 seconds per frame), making it suitable for real-time applications (e.g. robotic manipulation).

本文提出一种端到端的深度学习方法,从 RGB-D 输入估计已知物体的六自由度位姿。方法的核心是在逐像素层面嵌入并融合 RGB 值与点云,而不是像已有工作那样用图像裁块计算全局特征 [42],或依赖二维包围盒 [23]。这种逐像素融合使模型能够显式利用局部的外观和几何信息,这对处理严重遮挡至关重要。融合本身是稠密的,我们因而还可以对每个融合后的图像与深度特征所给出的位姿预测建模置信度,让模型在最终位姿上做出更合理的选择。此外,我们引入一个迭代位姿精炼过程,可以无缝接入这个端到端的 6D 位姿估计框架。该方法既快又准:单帧推理只需 0.06 秒,比 PoseCNN + ICP [41](每帧 11 秒)快 200 倍以上,适合机器人操作这类实时应用。

We evaluate our method in two popular benchmarks for 6D pose estimation, YCB-Video [41] and LineMOD [12]. We show that our method outperforms the state-of-the-art PoseCNN after ICP refinement [41] by 3.5% in pose accuracy while being 200x faster in inference time. In particular, we demonstrate its robustness in highly cluttered scenes thanks to our novel dense fusion method. Last, we also showcase how our method can be deployed in a real robot to grasp and manipulate objects based on the estimated poses.

我们在两个常用的 6D 位姿估计基准上评测该方法:YCB-Video [41] 和 LineMOD [12]。结果表明,与经过 ICP 精炼的当时最优方法 PoseCNN [41] 相比,位姿精度高出 3.5%,推理速度则快 200 倍。得益于这种新的稠密融合,它在高度杂乱的场景里也更稳健。最后,我们还展示了如何把它部署到真机上,依据估计出的位姿完成抓取和操作。

In summary, the contributions of this work are two-fold: First, we present a principled way to combine color and depth information from the RGB-D input. We augment the information of each 3D point with 2D information from an embedding space learned for the task and use this new color-depth space to estimate the 6D pose. Second, we integrate an iterative refinement procedure within the neural network architecture, removing the dependency of previous methods of a post-processing ICP step.

概括起来,本文的贡献有两点。第一,我们给出一种有原则的方式,来组合 RGB-D 输入中的颜色与深度信息:用为该任务学到的嵌入空间中的二维信息,去增强每个三维点的表示,并在这个新的颜色–深度空间里估计 6D 位姿。第二,我们把迭代精炼过程直接集成进神经网络结构,从而不再依赖以往方法里那种事后 ICP 步骤。

  1. Related Work
  1. 相关工作

Pose from RGB images. Classical methods rely on detecting and matching keypoints with known object models [1, 7, 9, 27, 44]. Newer methods address the challenge by learning to predict the 2D keypoints [3, 22, 32, 34, 35] and solve the poses by PnP [10]. Though prevail in speed-demanding tasks, these methods become unreliable given low-texture or low-resolution inputs. Other methods propose to directly estimate objects pose from images using CNN-based architectures [28, 36]. Many such methods focus on orientation estimation: Xiang et al. [39, 40] learns a viewpoint-aware pose estimator by clustering 3D features from object models. Mousavian et al. [21] predicts 3D object parameters and recovers poses by single-view geometry constraints. Sundermeyer et al. [31] implicitly encode orientation in a latent space and in test time find the best match in a codebook as the orientation prediction. However, pose estimation in 3D remains a challenge for the lack of depth information. Our method leverages both image and 3D data to estimate object poses in 3D in an end-to-end architecture.

来自 RGB 图像的位姿。 经典方法依赖检测关键点,并与已知物体模型做匹配 [1, 7, 9, 27, 44]。较新的方法则学习预测二维关键点 [3, 22, 32, 34, 35],再用 PnP [10] 求解位姿。这类方法在速度要求高的任务里很常见,但在低纹理或低分辨率输入下会变得不可靠。另一些方法提议用基于 CNN 的结构直接从图像估计物体位姿 [28, 36]。其中不少工作关注朝向估计:Xiang 等人 [39, 40] 通过对物体模型的三维特征聚类,学习一种对视点敏感的位姿估计器;Mousavian 等人 [21] 预测三维物体参数,再靠单视图几何约束恢复位姿;Sundermeyer 等人 [31] 把朝向隐式编码到潜空间里,测试时在码本中找最佳匹配作为朝向预测。不过,缺少深度信息时,三维位姿估计仍然困难。我们的方法同时利用图像和三维数据,在一个端到端结构里估计物体的三维位姿。

Pose from depth / point cloud. Recent studies have proposed to directly tackle the 3D object detection problem in discretized 3D voxel spaces. For example, Song et al. [29, 30] generate 3D bounding box proposals and estimate the poses by featuring the voxelized input with 3D ConvNets. Although the voxel representation effectively encodes geometric information, these methods are often prohibitively expensive: [30] takes nearly 20 seconds for each frame.

来自深度 / 点云的位姿。 近来有工作提议在离散化的三维体素空间里直接做三维物体检测。例如 Song 等人 [29, 30] 先生成三维包围盒候选,再用三维卷积网络对体素化输入提取特征并估计位姿。体素表示能有效编码几何信息,但这类方法往往计算代价过高:[30] 处理每一帧都接近 20 秒。

More recent 3D deep learning architectures have enabled methods that directly performs 6D pose estimation on 3D point cloud data. As an example, both Frustrum PointNets [23] and VoxelNet [43] use a PointNet-like [24] structure and achieved state-of-the-art performances on the KITTI benchmark [11]. Our method also makes use of similar architecture. However, unlike urban driving applications for which point cloud alone provides enough information, generic object pose estimation tasks such as the YCB-Video dataset [41] demands reasoning over both geometric and appearance information. We address such a challenge by proposing a novel 2D-3D sensor fusion architecture.

更新的三维深度学习结构,使得可以直接在三维点云上做 6D 位姿估计。例如,Frustum PointNets [23] 和 VoxelNet [43] 都采用类似 PointNet [24] 的结构,并在 KITTI 基准 [11] 上取得了当时最优的结果。我们的方法也使用了相近的结构。不过,城市驾驶这类应用往往单靠点云就够用;而像 YCB-Video 数据集 [41] 这样更一般的物体位姿估计任务,需要同时利用几何和外观信息。我们用一种新的二维–三维传感器融合结构来应对这一挑战。

Pose from RGB-D data. Classical approaches extract 3D features from the input RGB-D data and perform correspondence grouping and hypothesis verification [3, 12, 13, 15, 26, 33, 38]. However, these features are either hardcoded [12, 13, 26] or learned by optimizing surrogate objectives [3, 33, 38] such as reconstruction [15] instead of the true objective of 6D pose estimation. Newer methods such as PoseCNN [41] directly estimates 6D poses from image data. Li et al. [16] further fuses the depth input as an additional channel to a CNN-based architecture. However, these approaches rely on expensive post-processing steps to make full use of 3D input. In comparison, our method fuses 3D data to 2D appearance feature while retaining the geometric structure of the input space, and we show that it outperforms [41] on the YCB-Video dataset [41] without the post-processing step.

来自 RGB-D 数据的位姿。 经典方法从输入的 RGB-D 数据中提取三维特征,再做对应分组和假设验证 [3, 12, 13, 15, 26, 33, 38]。不过,这些特征要么是写死的 [12, 13, 26],要么是靠优化替代目标学到的 [3, 33, 38],例如重建 [15],而不是 6D 位姿估计这一真正目标。较新的方法如 PoseCNN [41] 直接从图像数据估计 6D 位姿;Li 等人 [16] 进一步把深度当作额外通道送进基于 CNN 的结构。但这些做法要充分利用三维输入,仍然依赖代价很高的后处理。相比之下,我们的方法在保留输入空间几何结构的同时,把三维数据融合进二维外观特征,并在 YCB-Video 数据集 [41] 上表明,即使不做后处理,也能超过 [41]。

Our method is most related to PointFusion [42], in which geometric and appearance information are fused in a heterogeneous architecture. We show that our novel local feature fusion scheme significantly outperforms PointFusion’s naive fusion-by-concatenation method. In addition, we use a novel iterative refinement method to further improve the pose estimation.

与本文最接近的是 PointFusion [42],它也在异构结构里融合几何和外观信息。我们表明,这种新的局部特征融合方案明显优于 PointFusion 那种简单的拼接式融合。此外,我们还用一种新的迭代精炼方法进一步改进位姿估计。

  1. Model
  1. 模型

Our goal is to estimate the 6D pose of a set of known objects present in an RGB-D image of a cluttered scene. Without loss of generality, we represent 6D poses as a homogeneous transformation matrix, p∈SE(3)p \in SE(3)p∈SE(3). In other words, a 6D pose is composed by a rotation R∈SO(3)R \in SO(3)R∈SO(3) and a translation t∈R3t \in \mathbb{R}^3t∈R3, p=[R∣t]p = [R|t]p=[R∣t]. Since we estimate the 6D pose of the objects from camera images, the poses are defined with respect to the camera coordinate frame.

我们的目标是:在杂乱场景的 RGB-D 图像里,估计一组已知物体的 6D 位姿。不失一般性,把 6D 位姿表示成齐次变换矩阵 p∈SE(3)p \in SE(3)p∈SE(3)。也就是说,一个 6D 位姿由旋转 R∈SO(3)R \in SO(3)R∈SO(3) 和平移 t∈R3t \in \mathbb{R}^3t∈R3 组成,p=[R∣t]p = [R|t]p=[R∣t]。由于我们是从相机图像估计物体的 6D 位姿,这些位姿都定义在相机坐标系下。

DenseFusion architecture from RGB-D segmentation to pixel-wise pose prediction

Figure 2. Overview of our 6D pose estimation model. Our model generates object segmentation masks and bounding boxes from RGB images. The RGB colors and point cloud from the depth map are encoded into embeddings and fused at each corresponding pixel. The pose predictor produces a pose estimate for each pixel and the predictions are voted to generate the final 6D pose prediction of the object. (The iterative procedure of our approach is not depicted here for simplicity)

图 2. 我们提出的 6D 位姿估计模型概览。模型先从 RGB 图像生成物体分割掩膜和包围盒。彩色值以及由深度图得到的点云被编码成嵌入,并在每个对应像素上融合。位姿预测器为每个像素给出一个位姿估计,再通过投票得到该物体最终的 6D 位姿预测。(为简明起见,图中未画出迭代过程)

Estimating the pose of a known object in adversarial conditions (e.g. heavy occlusion, poor lighting, . . . ) is only possible by combining the information contained in the color and depth image channels. However, the two data sources reside in different spaces. Extracting features from heterogeneous data sources and fusing them appropriately is the key technical challenge in this domain.

在严重遮挡、光照很差这类不利条件下估计已知物体的位姿,只能把彩色通道和深度通道里的信息结合起来。可这两种数据处在不同的空间里。从异构数据源提取特征并恰当地融合,正是这一领域的关键技术难点。

We address this challenge with (1) a heterogeneous architecture that processes color and depth information differently, retaining the native structure of each data source (Sec. 3.3), and (2) a dense pixel-wise fusion network that performs color-depth fusion by exploiting the intrinsic mapping between the data sources (Sec. 3.4). Finally, the pose estimation is further refined with a differentiable iterative refinement module (Sec. 3.6). In contrast to the expensive post-hoc refinement steps used in [16, 41], our refinement module can be trained jointly with the main architecture and only takes a small fraction of the total inference time.

我们从两方面应对这一挑战:(1)用异构结构以不同方式处理颜色和深度,保留各自数据源的固有结构(第 3.3 节);(2)用稠密的逐像素融合网络,利用两种数据源之间的内在映射来做颜色–深度融合(第 3.4 节)。最后,再用一个可微的迭代精炼模块进一步改进位姿估计(第 3.6 节)。与 [16, 41] 中代价很高的事后精炼不同,该精炼模块可以和主体结构一起训练,只占用总推理时间的一小部分。

3.1. Architecture Overview

3.1. 结构总览

Fig. 2 illustrates the overall proposed architecture. The architecture contains two main stages. The first stage takes color image as input and performs semantic segmentation for each known object category. Then, for each segmented object, we feed the masked depth pixels (converted to 3D point cloud) as well as an image patch cropped by the bounding box of the mask to the second stage.

图 2 给出了整体结构。整个架构包含两个主要阶段。第一阶段以彩色图为输入,对每个已知物体类别做语义分割。然后,对每个分割出的物体,把掩膜内的深度像素(转换成三维点云)以及按掩膜包围盒裁出的图像块送入第二阶段。

The second stage processes the results of the segmentation and estimates the object’s 6D pose. It comprises four components: a) a fully convolutional network that processes the color information and maps each pixel in the image crop to a color feature embedding, b) a PointNet-based [24] network that processes each point in the masked 3D point cloud to a geometric feature embedding, c) a pixel-wise fusion network that combines both embeddings and outputs the estimation of the 6D pose of the object based on an unsupervised confidence scoring, and d) an iterative self-refinement methodology to train the network in a curriculum learning manner and refine the estimation result iteratively. Fig. 2 depicts a), b) and c) and Fig. 3 illustrates d). The details our architecture are described below.

第二阶段处理分割结果并估计物体的 6D 位姿,包含四个部分:a) 一个全卷积网络,处理颜色信息,把图像裁块中的每个像素映射到颜色特征嵌入;b) 一个基于 PointNet [24] 的网络,把掩膜内三维点云中的每个点映射到几何特征嵌入;c) 一个逐像素融合网络,结合这两种嵌入,并基于无监督的置信度打分输出物体的 6D 位姿估计;d) 一套迭代自精炼方法,以课程学习的方式训练网络,并逐步改进估计结果。图 2 给出了 a)、b)、c),图 3 给出了 d)。结构细节如下。

3.2. Semantic Segmentation

3.2. 语义分割

The first step is to segment the objects of interest in the image. Our semantic segmentation network is an encoder-decoder architecture that takes an image as input and generates an N+1N+1N+1-channelled semantic segmentation map. Each channel is a binary mask where active pixels depict objects of each of the NNN possible known classes. The focus of this work is to develop a pose estimation algorithm. Thus we use an existing segmentation architecture proposed by [41].

第一步是在图像中分割出感兴趣的物体。我们的语义分割网络是编码器–解码器结构,输入一幅图像,输出 N+1N+1N+1 个通道的语义分割图。每个通道都是一幅二值掩膜,激活像素对应该 NNN 个已知类别中的一类。本文的重点是开发位姿估计算法,因此分割结构直接采用 [41] 中已有的设计。

3.3. Dense Feature Extraction

3.3. 稠密特征提取

The key technical challenge in this domain is the correct extraction of information from the color and depth channels and their synergistic fusion. Even though color and depth present a similar format in the RGB-D frame, their information resides in different spaces. Therefore, we process them separately to generate color and geometric features from embedding spaces that retain the intrinsic structure of the data sources.

这一领域的关键技术难点,是如何正确地从颜色通道和深度通道提取信息,并把它们协同地融合起来。尽管在 RGB-D 帧里颜色和深度看起来格式相近,它们所携带的信息却处在不同空间。因此我们分开处理,从各自保留数据源固有结构的嵌入空间里生成颜色特征和几何特征。

Dense 3D point cloud feature embedding: Previous approaches have used CNN to process the depth image as an additional image channel [16]. However, such method neglects the intrinsic 3D structure of the depth channel. Instead, we first convert the segmented depth pixels into a 3D point cloud using the known camera intrinsics, and then use a PointNet-like architecture to extract geometric features.

稠密三维点云特征嵌入: 已有方法曾用 CNN 把深度图当作额外的图像通道来处理 [16]。但这种做法忽略了深度通道固有的三维结构。我们改为先利用已知的相机内参,把分割出的深度像素转换成三维点云,再用类似 PointNet 的结构提取几何特征。

PointNet by Qi et al. [24] pioneered the use of a symmetric function (max-pooling) to achieve permutation invariance in processing unordered point sets. The original architecture takes as input a raw point cloud and learns to encode the information about the vicinity of each point and of the point cloud as a whole. The features are shown to be effective in shape classification and segmentation [24] and pose estimation [23, 42]. We propose a geometric embedding network that generates a dense per-point feature by mapping each of the PPP segmented points to a dgeod_{\mathrm{geo}}dgeo​-dimensional feature space. We implement a variant of PointNet architecture that uses average-pooling as opposed to the commonly used max-pooling as the symmetric reduction function.

Qi 等人提出的 PointNet [24] 率先用对称函数(最大池化)来处理无序点集,从而获得置换不变性。原始结构以原始点云为输入,学习编码每个点邻域以及点云整体的信息。这些特征已被证明对形状分类与分割 [24] 以及位姿估计 [23, 42] 有效。我们提出一个几何嵌入网络,把分割得到的 PPP 个点分别映射到 dgeod_{\mathrm{geo}}dgeo​ 维特征空间,从而生成稠密的逐点特征。实现上,我们采用 PointNet 的一个变体,用平均池化代替常用的最大池化,作为对称归约函数。

Dense color image feature embedding: The goal of the color embedding network is to extract per-pixel features such that we can form dense correspondences between 3D point features and image features. The reason for forming these dense correspondences will be clear in the next section. The image embedding network is a CNN-based encoder-decoder architecture that maps an image of size H×W×3H \times W \times 3H×W×3 into a H×W×drgbH \times W \times d_{\mathrm{rgb}}H×W×drgb​ embedding space. Each pixel of the embedding is a drgbd_{\mathrm{rgb}}drgb​-dimensional vector representing the appearance information of the input image at the corresponding location.

稠密彩色图像特征嵌入: 颜色嵌入网络的目标是提取逐像素特征,以便在三维点特征与图像特征之间建立稠密对应。建立这种稠密对应的原因将在下一节说明。图像嵌入网络是基于 CNN 的编码器–解码器,把大小为 H×W×3H \times W \times 3H×W×3 的图像映射到 H×W×drgbH \times W \times d_{\mathrm{rgb}}H×W×drgb​ 的嵌入空间。嵌入中的每个像素都是一个 drgbd_{\mathrm{rgb}}drgb​ 维向量,表示输入图像在对应位置上的外观信息。

3.4. Pixel-wise Dense Fusion

3.4. 逐像素稠密融合

So far we have obtained dense features from both the image and the 3D point cloud inputs; now we need to fuse the information. A naive approach would be to generate a global feature from the dense color and depth features from the segmented area. However, due to heavy occlusion and segmentation errors, the set of features from the previous step may contain features of points/pixels on other objects or parts of the background. Therefore, blindly fusing color and geometric features globally would degrade the performance of the estimation. In the following we describe a novel pixel-wise¹ dense fusion network that effectively combines the extracted features, especially for pose estimation under heavy occlusion and imperfect segmentation.

到目前为止,我们已经分别从图像和三维点云输入得到了稠密特征,接下来需要把这些信息融合起来。一种朴素做法是:由分割区域内的稠密颜色和深度特征生成一个全局特征。可是,由于严重遮挡和分割误差,上一步得到的特征集合里可能混有其他物体或背景上的点/像素。因此,不加区分地在全局层面融合颜色与几何特征,会损害估计效果。下面我们介绍一种新的逐像素¹稠密融合网络,能够有效组合提取到的特征,尤其适用于严重遮挡和分割并不完美时的位姿估计。

Pixel-wise dense fusion: The key idea of our dense fusion network is to perform local per-pixel fusion instead of global fusion so that we can make predictions based on each fused feature. In this way, we can potentially select the predictions based on the visible part of the object and minimize the effects of occlusion and segmentation noise.

逐像素稠密融合: 稠密融合网络的核心想法是做局部的逐像素融合,而不是全局融合,从而可以基于每一个融合特征分别做预测。这样,我们就有可能只根据物体可见部分来选择预测,尽量减小遮挡和分割噪声的影响。

¹Since the mapping between pixels and 3D points is unique, we will use interchangeably pixel-fusion and point-fusion.

¹由于像素与三维点之间的映射是唯一的,我们将交替使用像素融合和点融合这两个说法。

Concretely, our dense fusion procedure first associates the geometric feature of each point to its corresponding image feature pixel based on a projection onto the image plane using the known camera intrinsic parameters. The obtained pairs of features are then concatenated and fed to another network to generate a fixed-size global feature vector using a symmetric reduction function. While we refrained from using a single global feature for the estimation, here we enrich each dense pixel-feature with the global densely-fused feature to provide a global context.

具体而言,稠密融合过程首先利用已知相机内参,把每个点投影到图像平面,从而把它的几何特征与对应的图像特征像素关联起来。得到的特征对再拼接起来,送入另一个网络,用对称归约函数生成固定长度的全局特征向量。虽然我们并不用单个全局特征直接做估计,但在这里会用这个全局稠密融合特征去丰富每个稠密像素特征,以提供全局上下文。

We feed each of the resulting per-pixel features into a final network that predicts the object’s 6D pose. In other words, we will train this network to predict one pose from each densely-fused feature. The result is a set of PPP predicted poses, one per feature. This defines our first learning objective, as we will see in Sec. 3.5. We will now explain our approach to learning to choose the best prediction in a self-supervised manner, inspired by Xu et al. [42].

我们把得到的每个逐像素特征送入最后的网络,用于预测物体的 6D 位姿。也就是说,网络会从每一个稠密融合特征预测一个位姿,最终得到 PPP 个预测,每个特征一个。这构成了第一个学习目标,详见第 3.5 节。接下来说明如何以自监督的方式学习选出最佳预测,这一做法受到 Xu 等人 [42] 的启发。

Per-pixel self-supervised confidence: We would like to train our pose estimation network to decide which pose estimation is likely to be the best hypothesis based on the specific context. To do so, we modify the network to output a confidence score cic_ici​ for each prediction in addition to the pose estimation predictions. We will have to reflect this second learning objective in the overall learning objective, as we will see at the end of the next section.

逐像素的自监督置信度: 我们希望位姿估计网络能够根据具体情形,判断哪一个位姿估计更可能是最好的假设。为此,我们让网络在给出位姿预测的同时,再为每个预测输出一个置信度 cic_ici​。这第二个学习目标需要反映到总体学习目标里,我们将在下一节末尾看到。

3.5. 6D Object Pose Estimation

3.5. 6D 物体位姿估计

Having defined the overall network structure, we now take a closer look at the learning objective. We define the pose estimation loss as the distance between the points sampled on the objects model in ground truth pose and corresponding points on the same model transformed by the predicted pose. Specifically, the loss to minimize for the prediction per dense-pixel is defined as

在给出整体网络结构之后,我们再来仔细看学习目标。位姿估计损失定义为:在真值位姿下从物体模型上采样的点,与同一模型经预测位姿变换后的对应点之间的距离。具体来说,对每个稠密像素的预测,要最小化的损失定义为

Lip=1M∑j∥(Rxj+t)−(R^ixj+t^i)∥(1)L_i^p = \frac{1}{M} \sum_{j} \|(R x_j + t) - (\hat{R}_i x_j + \hat{t}_i)\| \tag{1}Lip​=M1​j∑​∥(Rxj​+t)−(R^i​xj​+t^i​)∥(1)

where xjx_jxj​ denotes the jthj^{th}jth point of the MMM randomly selected 3D points from the object’s 3D model, p=[R∣t]p = [R|t]p=[R∣t] is the ground truth pose, and p^i=[R^i∣t^i]\hat{p}_i = [\hat{R}_i|\hat{t}_i]p^​i​=[R^i​∣t^i​] is the predicted pose generated from the fused embedding of the ithi^{th}ith dense-pixel.

其中 xjx_jxj​ 表示从物体三维模型上随机选出的 MMM 个三维点中的第 jjj 个点,p=[R∣t]p = [R|t]p=[R∣t] 是真值位姿,p^i=[R^i∣t^i]\hat{p}_i = [\hat{R}_i|\hat{t}_i]p^​i​=[R^i​∣t^i​] 则是由第 iii 个稠密像素的融合嵌入生成的预测位姿。

The above loss function is only well-defined for asymmetric objects, where the object shape and/or texture determines a unique canonical frame. Symmetric objects have more than one and possibly an infinite number of canonical frames, which leads to ambiguous learning objectives. Therefore, for symmetric objects, we instead minimize the distance between each point on the estimated model orientation and the closest point on the ground truth model. The loss function becomes:

上述损失函数只对非对称物体定义良好:这时物体的形状和/或纹理能够确定唯一的规范坐标系。对称物体则有不止一个、甚至可能无穷多个规范坐标系,从而带来含糊的学习目标。因此,对对称物体,我们改为最小化估计模型朝向上每个点到真值模型上最近点的距离。损失函数变为:

Iterative pose refinement loop with residual rotation and translation

Figure 3. Iterative Pose Refinement. We introduce an network module that refines the pose estimation in an iterative procedure.

图 3. 迭代位姿精炼。我们引入一个网络模块,以迭代方式精炼位姿估计。

Lip=1M∑jmin⁡0<k<M∥(Rxj+t)−(R^ixk+t^i)∥(2)L_i^p = \frac{1}{M} \sum_{j} \min_{0<k<M} \|(R x_j + t) - (\hat{R}_i x_k + \hat{t}_i)\| \tag{2}Lip​=M1​j∑​0<k<Mmin​∥(Rxj​+t)−(R^i​xk​+t^i​)∥(2)

Optimizing over all predicted per dense-pixel poses would be to minimize the mean of the per dense-pixel losses: L=1N∑iLipL = \frac{1}{N} \sum_{i} L_i^pL=N1​∑i​Lip​. However, as explained before, we would like our network to learn to balance the confidence among the per dense-pixel predictions. To do that we weight the per dense-pixel loss with the dense-pixel confidence, and add a second confidence regularization term:

若对所有逐稠密像素预测的位姿一起优化,就是最小化这些逐稠密像素损失的均值:L=1N∑iLipL = \frac{1}{N} \sum_{i} L_i^pL=N1​∑i​Lip​。不过如前所述,我们希望网络学会在各稠密像素预测之间平衡置信度。为此,我们用稠密像素的置信度去加权相应的损失,并再加一项置信度正则:

L=1N∑i(Lipci−wlog⁡(ci)),(3)L = \frac{1}{N} \sum_{i} (L_i^p c_i - w \log(c_i)), \tag{3}L=N1​i∑​(Lip​ci​−wlog(ci​)),(3)

where NNN is the number of randomly sampled dense-pixel features from the PPP elements of the segment and www is a balancing hyperparameter. Intuitively, low confidence will result in low pose estimation loss but would incur high penalty from the second term, and vice versa. We use the pose estimation that has the highest confidence as the final output.

其中 NNN 是从该分割的 PPP 个元素中随机采样的稠密像素特征个数,www 是平衡用的超参数。直观上看,置信度低会让位姿估计损失变小,但第二项会给出更大的惩罚,反之亦然。我们把置信度最高的那个位姿估计作为最终输出。

3.6. Iterative Refinement

3.6. 迭代精炼

The iterative closest point algorithm (ICP) [2] is a powerful refinement approach used by many 6D pose estimation methods [14, 31, 41]. However, the best-performing ICP implementations are often not efficient enough for real-time applications. Other refinement methods including [17, 18] assume the access to a real-time rendering engine and object models with high-quality textures. Here we propose a neural network-based iterative refinement module based on our densely-fused embedding that can improve the final pose estimation result in a fast and robust manner without additional rendering techniques.

迭代最近点算法(ICP)[2] 是许多 6D 位姿估计方法会用的强力精炼手段 [14, 31, 41]。不过,表现最好的那些 ICP 实现,对实时应用来说往往不够快。其他精炼方法,包括 [17, 18],则假定可以调用实时渲染引擎,并且物体模型带有高质量纹理。这里我们基于稠密融合嵌入,提出一个神经网络形式的迭代精炼模块,能够在不额外使用渲染技术的情况下,又快又稳地改进最终位姿估计。

The goal is to enable the network to correct its own pose estimation error in an iterative manner. The challenge here is training the network to refine the previous prediction as opposed to making new predictions. To do so, we must include the prediction made in the previous iteration as part of the input to the next iteration. Our key idea is to consider the previously predicted pose as an estimate of canonical frame of the target object and transform the input point cloud into this estimated canonical frame. This way, the transformed point cloud implicitly encodes the estimated pose. We then feed the transformed point cloud back into the network and predict a residual pose based on the previously estimated pose. This procedure can be applied iteratively and generate potentially finer pose estimation each iteration.

目标是让网络能够迭代地纠正自己的位姿估计误差。难点在于:训练网络去精炼上一次的预测,而不是重新做一个新的预测。为此,必须把上一轮的预测作为下一轮输入的一部分。我们的关键想法是:把先前预测的位姿看作目标物体规范坐标系的一个估计,并把输入点云变换到这个估计出的规范坐标系中。这样,变换后的点云就隐式编码了已估计的位姿。我们再把变换后的点云送回网络,基于先前估计预测一个残差位姿。这一过程可以迭代进行,每一步都有可能得到更细的位姿估计。

The procedure is illustrated in Fig. 3. Concretely, we train a dedicated pose residual estimator network to perform the refinement given the initial pose estimation from the main network. At each iteration, we reuse the image feature embedding from the main network and perform dense fusion with the geometric features computed for the new transformed point cloud. The pose residual estimator uses as input a global feature from the set of fused pixel features. After KKK iterations, we obtain the final pose estimation as the concatenation of the per-iteration estimations:

该过程如图 3 所示。具体来说,我们训练一个专门的位姿残差估计网络,在主网络给出初始位姿估计后再做精炼。每一次迭代,我们复用主网络得到的图像特征嵌入,并与针对新变换点云算出的几何特征做稠密融合。位姿残差估计器的输入,是从融合后的像素特征集合得到的全局特征。经过 KKK 次迭代后,把各步估计连乘起来,得到最终的位姿估计:

p^=[RK∣tK]⋅[RK−1∣tK−1]⋯[R0∣t0](4)\hat{p} = [R_K|t_K] \cdot [R_{K-1}|t_{K-1}] \cdots [R_0|t_0] \tag{4}p^​=[RK​∣tK​]⋅[RK−1​∣tK−1​]⋯[R0​∣t0​](4)

In our experiments, only after K=2K = 2K=2 iterations, the refined pose is already able to surpass the ICP-refined results of prior works. The pose residual estimator can be trained jointly with the main network. However, the pose estimation at the beginning of the training is too noisy for it to learn anything meaningful. Thus in practice, the joint training starts after the main network has converged.

在我们的实验中,只需 K=2K = 2K=2 次迭代,精炼后的位姿就已经能够超过已有工作再接 ICP 得到的结果。位姿残差估计器可以和主网络一起训练。不过训练初期的位姿估计噪声太大,它很难学到有意义的内容。因此实际操作中,等主网络收敛之后再开始联合训练。

  1. Experiments
  1. 实验

In the experimental section, we would like to answer the following questions: (1) How does the dense fusion network compare to naive global fusion-by-concatenation? (2) Is the dense fusion and prediction scheme robust to heavy occlusion and segmentation errors? (3) Does the iterative refinement module improve the final pose estimation? (4) Is our method robust and efficient enough for downstream tasks such as robotic grasping?

在实验部分,我们希望回答下列问题:(1)稠密融合网络与简单的全局拼接融合相比如何?(2)这种稠密融合与预测方案对严重遮挡和分割误差是否稳健?(3)迭代精炼模块能否改进最终位姿估计?(4)该方法是否足够稳健、足够高效,可以用于机器人抓取这类下游任务?

To answer the first three questions, we evaluate our method on two challenging 6D object pose estimation datasets: YCB-Video Dataset [41] and LineMOD [12]. The YCB-Video Dataset features objects of varying shapes and texture levels under different occlusion conditions. Hence it’s an ideal testbed for our occlusion-resilient multi-modal fusion method. The LineMOD dataset is a widely-used dataset that allows us to compare with a broader range of existing methods. We compare our method with state-of-the-art methods [14, 31, 42] as well as model variants. To answer the last question, we deploy our model to a real robot platform and evaluate the performance of a robot grasping task that uses the predictions from our model.

为回答前三个问题,我们在两个有挑战性的 6D 位姿估计数据集上评测该方法:YCB-Video 数据集 [41] 和 LineMOD [12]。YCB-Video 包含形状和纹理水平各异的物体,并处于不同程度的遮挡下,因此很适合检验我们对遮挡更稳健的多模态融合方法。LineMOD 则是被广泛采用的数据集,便于和更广泛的已有方法比较。我们把该方法与当时最优的方法 [14, 31, 42] 以及若干模型变体进行对照。为回答最后一个问题,我们把模型部署到真机平台上,评测一个使用这些预测结果的机器人抓取任务。

4.1. Datasets

4.1. 数据集

YCB-Video Dataset. The YCB-Video Dataset Xiang et al. [41] features 21 YCB objects Calli et al. [5] of varying shape and texture. The dataset contains 92 RGB-D videos, where each video shows a subset of the 21 objects in different indoor scenes. The videos are annotated with 6D poses and segmentation masks. We follow prior work [41] and split the dataset into 80 videos for training and 2,949 key frames chosen from the rest 12 videos for testing and include the same 80,000 synthetic images released by [41] in our training set. In our experiments, we compare with the result of [41] after depth refinement (ICP) and learning-based depth method [42].

YCB-Video 数据集。 Xiang 等人 [41] 的 YCB-Video 数据集包含 Calli 等人 [5] 的 21 个形状和纹理各异的 YCB 物体。数据集有 92 段 RGB-D 视频,每段视频在不同室内场景中展示这 21 个物体中的一个子集,并标注了 6D 位姿和分割掩膜。我们沿用已有工作 [41] 的划分:80 段视频用于训练,从其余 12 段视频中选出 2,949 个关键帧用于测试,训练集约外还包括 [41] 发布的同一批 80,000 张合成图像。实验中,我们与 [41] 经过深度精炼(ICP)后的结果,以及基于学习的深度方法 [42] 进行比较。

Qualitative 6D pose results on YCB-Video compared with PointFusion and PoseCNN+ICP

Figure 4. Qualitative results on the YCB-Video Dataset. All three methods shown here are tested with the same segmentation masks as in PoseCNN. Each object point cloud in a different color is transformed with the predicted pose and then projected to the 2D image frame. The first two rows are former RGB-D methods and the last row is our approach with dense fusion and iterative refinement (2 iterations).

图 4. YCB-Video 数据集上的定性结果。这里给出的三种方法都使用与 PoseCNN 相同的分割掩膜进行测试。每种颜色的物体点云先按预测位姿变换,再投影到二维图像平面。前两行是已有的 RGB-D 方法,最后一行是我们带稠密融合和迭代精炼(2 次迭代)的方法。

LineMOD Dataset. The LineMOD dataset Hinterstoisser et al. [12] consists of 13 low-textured objects in 13 videos. It is widely adopted by both classical methods [4, 8, 37] and recent learning-based approaches [17, 31, 34]. We use the same training and testing set as prior learning-based works [17, 25, 34] without additional synthetic data and compare with the best ICP-refined results of the state-of-the-art algorithms.

LineMOD 数据集。 Hinterstoisser 等人 [12] 的 LineMOD 数据集包含 13 个低纹理物体,分布在 13 段视频中。经典方法 [4, 8, 37] 和近来的学习方法 [17, 31, 34] 都广泛使用它。我们采用与已有学习方法 [17, 25, 34] 相同的训练和测试划分,不额外使用合成数据,并与当时最优算法经过 ICP 精炼后的最好结果进行比较。

4.2. Metrics

4.2. 评价指标

We use two metrics to report on the YCB-Video Dataset. The average closest point distance (ADD-S) [41] is an ambiguity-invariant pose error metric which takes care of both symmetric and non-symmetric objects into an overall evaluation. Given the estimated pose [R^∣t^][\hat{R}|\hat{t}][R^∣t^] and ground truth pose [R∣t][R|t][R∣t], ADD-S calculates the mean distance from each 3D model point transformed by [R^∣t^][\hat{R}|\hat{t}][R^∣t^] to its closest neighbour on the target model transformed by [R∣t][R|t][R∣t]. We report the area under the ADD-S curve (AUC) following PoseCNN [41]. We follow prior work and set the maximum threshold of AUC to be 0.1m. We also report the percentage of ADD-S smaller than 2cm (<2cm), which measures the predictions under the minimum tolerance for robot manipulation (2cm for most of the robot grippers).

在 YCB-Video 数据集上我们报告两项指标。平均最近点距离(ADD-S)[41] 是一种对歧义不变的位姿误差度量,可以把对称物体和非对称物体放进同一次总体评价里。给定估计位姿 [R^∣t^][\hat{R}|\hat{t}][R^∣t^] 和真值位姿 [R∣t][R|t][R∣t],ADD-S 计算由 [R^∣t^][\hat{R}|\hat{t}][R^∣t^] 变换后的每个三维模型点,到由 [R∣t][R|t][R∣t] 变换后的目标模型上最近点的平均距离。我们沿用 PoseCNN [41],报告 ADD-S 曲线下面积(AUC)。与已有工作一致,AUC 的最大阈值设为 0.1 m。我们还报告 ADD-S 小于 2 cm 的比例(<2cm),用来衡量预测是否落在机器人操作的最小容差之内(对大多数机器人夹爪而言是 2 cm)。

For the LineMOD dataset, we use the Average Distance of Model Points (ADD) [13] for non-symmetric objects and ADD-S for the two symmetric objects (eggbox and glue) following prior works [13, 31, 34].

在 LineMOD 数据集上,我们沿用已有工作 [13, 31, 34],对非对称物体使用模型点平均距离(ADD)[13],对两个对称物体(eggbox 和 glue)则使用 ADD-S。

4.3. Implementation Details

4.3. 实现细节

The image embedding network consists of a Resnet-18 encoder followed by 4 up-sampling layers as the decoder. The PointNet architecture is an MLP followed by an average-pooling reduction function. Both color and geometric dense feature embedding are of dimension 128. We choose w=0.01w = 0.01w=0.01 for Eq. 3 by empirical evaluation. The iterative pose refinement module consists of a 4 fully connected layers that directly output the pose residual from the global dense feature. We use the 2 refinement iterations for all experiments.

图像嵌入网络由 ResNet-18 编码器和作为解码器的 4 个上采样层组成。PointNet 结构是一个 MLP,再接平均池化作为归约函数。颜色和几何的稠密特征嵌入都是 128 维。对式 (3) 中的 www,我们根据实测取 0.010.010.01。迭代位姿精炼模块由 4 个全连接层组成,直接从全局稠密特征输出位姿残差。所有实验都使用 2 次精炼迭代。

4.4. Architectures

4.4. 网络结构

We compare four model variants that showcase the effectiveness of our design choices.

我们比较四种模型变体,以说明各设计选择的效果。

• PointFusion [42] uses a CNN to extract a fixed-size feature vector and fuse by directly concatenating the image feature with the geometry feature. The rest of the network is similar to our architecture. The comparison to this baseline demonstrates the effectiveness of our dense fusion network.

• PointFusion [42] 用 CNN 提取固定长度的特征向量,再把图像特征与几何特征直接拼接来融合。网络的其余部分与我们的结构类似。与这一基线的对比,用来说明稠密融合网络的效果。

• Ours (single) uses our dense fusion network, but instead of performing per-point prediction, it only outputs a single prediction using the global feature vector.

• Ours (single) 使用我们的稠密融合网络,但不做逐点预测,而是只用全局特征向量输出一次预测。

• Ours (per-pixel) performs per-pixel prediction based on each densely fused feature.

• Ours (per-pixel) 基于每一个稠密融合特征做逐像素预测。

• Ours (iterative) is our complete model that uses the iterative refinement (Sec. 3.6) on top of Ours (per-pixel).

• Ours (iterative) 是完整模型,在 Ours (per-pixel) 之上再使用迭代精炼(第 3.6 节)。

Table 1. Quantitative evaluation of 6D pose (ADD-S[41]) on YCB-Video Dataset. Objects with bold name are symmetric.

PointFusion [42] PoseCNN+ICP [41] Ours (single) Ours (per-pixel) Ours (iterative)
AUC <2cm AUC <2cm AUC <2cm AUC <2cm AUC <2cm
002_master_chef_can 90.9 99.8 95.8 100.0 93.9 100.0 95.2 100.0 96.4 100.0
003_cracker_box 80.5 62.6 92.7 91.6 90.8 98.4 92.5 99.3 95.5 99.5
004_sugar_box 90.4 95.4 98.2 100.0 94.4 99.2 95.1 100.0 97.5 100.0
005_tomato_soup_can 91.9 96.9 94.5 96.9 92.9 96.7 93.7 96.9 94.6 96.9
006_mustard_bottle 88.5 84.0 98.6 100.0 91.2 97.8 95.9 100.0 97.2 100.0
007_tuna_fish_can 93.8 99.8 97.1 100.0 94.9 100.0 94.9 100.0 96.6 100.0
008_pudding_box 87.5 96.7 97.9 100.0 88.3 97.2 94.7 100.0 96.5 100.0
009_gelatin_box 95.0 100.0 98.8 100.0 95.4 100.0 95.8 100.0 98.1 100.0
010_potted_meat_can 86.4 88.5 92.7 93.6 87.3 91.4 90.1 93.1 91.3 93.1
011_banana 84.7 70.5 97.1 99.7 84.6 62.0 91.5 93.9 96.6 100.0
019_pitcher_base 85.5 79.8 97.8 100.0 86.9 80.9 94.6 100.0 97.1 100.0
021_bleach_cleanser 81.0 65.0 96.9 99.4 91.6 98.2 94.3 99.8 95.8 100.0
024_bowl 75.7 24.1 81.0 54.9 83.4 55.4 86.6 69.5 88.2 98.8
025_mug 94.2 99.8 95.0 99.8 90.3 94.7 95.5 100.0 97.1 100.0
035_power_drill 71.5 22.8 98.2 99.6 83.1 64.2 92.4 97.1 96.0 98.7
036_wood_block 68.1 18.2 87.6 80.2 81.7 76.0 85.5 93.4 89.7 94.6
037_scissors 76.7 35.9 91.7 95.6 83.6 75.1 96.4 100.0 95.2 100.0
040_large_marker 87.9 80.4 97.2 99.7 91.2 88.6 94.7 99.2 97.5 100.0
051_large_clamp 65.9 50.0 75.2 74.9 70.5 77.1 71.6 78.5 72.9 79.2
052_extra_large_clamp 60.4 20.1 64.4 48.8 66.4 50.2 69.0 69.5 69.8 76.3
061_foam_brick 91.8 100.0 97.2 100.0 92.1 100.0 92.4 100.0 92.5 100.0
MEAN 83.9 74.1 93.0 93.2 88.2 87.9 91.2 95.3 93.1 96.8

表 1. 在 YCB-Video 数据集上对 6D 位姿(ADD-S[41])的定量评价。名称加粗的物体是对称物体。

PointFusion [42] PoseCNN+ICP [41] Ours (single) Ours (per-pixel) Ours (iterative)
AUC <2cm AUC <2cm AUC <2cm AUC <2cm AUC <2cm
002_master_chef_can 90.9 99.8 95.8 100.0 93.9 100.0 95.2 100.0 96.4 100.0
003_cracker_box 80.5 62.6 92.7 91.6 90.8 98.4 92.5 99.3 95.5 99.5
004_sugar_box 90.4 95.4 98.2 100.0 94.4 99.2 95.1 100.0 97.5 100.0
005_tomato_soup_can 91.9 96.9 94.5 96.9 92.9 96.7 93.7 96.9 94.6 96.9
006_mustard_bottle 88.5 84.0 98.6 100.0 91.2 97.8 95.9 100.0 97.2 100.0
007_tuna_fish_can 93.8 99.8 97.1 100.0 94.9 100.0 94.9 100.0 96.6 100.0
008_pudding_box 87.5 96.7 97.9 100.0 88.3 97.2 94.7 100.0 96.5 100.0
009_gelatin_box 95.0 100.0 98.8 100.0 95.4 100.0 95.8 100.0 98.1 100.0
010_potted_meat_can 86.4 88.5 92.7 93.6 87.3 91.4 90.1 93.1 91.3 93.1
011_banana 84.7 70.5 97.1 99.7 84.6 62.0 91.5 93.9 96.6 100.0
019_pitcher_base 85.5 79.8 97.8 100.0 86.9 80.9 94.6 100.0 97.1 100.0
021_bleach_cleanser 81.0 65.0 96.9 99.4 91.6 98.2 94.3 99.8 95.8 100.0
024_bowl 75.7 24.1 81.0 54.9 83.4 55.4 86.6 69.5 88.2 98.8
025_mug 94.2 99.8 95.0 99.8 90.3 94.7 95.5 100.0 97.1 100.0
035_power_drill 71.5 22.8 98.2 99.6 83.1 64.2 92.4 97.1 96.0 98.7
036_wood_block 68.1 18.2 87.6 80.2 81.7 76.0 85.5 93.4 89.7 94.6
037_scissors 76.7 35.9 91.7 95.6 83.6 75.1 96.4 100.0 95.2 100.0
040_large_marker 87.9 80.4 97.2 99.7 91.2 88.6 94.7 99.2 97.5 100.0
051_large_clamp 65.9 50.0 75.2 74.9 70.5 77.1 71.6 78.5 72.9 79.2
052_extra_large_clamp 60.4 20.1 64.4 48.8 66.4 50.2 69.0 69.5 69.8 76.3
061_foam_brick 91.8 100.0 97.2 100.0 92.1 100.0 92.4 100.0 92.5 100.0
MEAN 83.9 74.1 93.0 93.2 88.2 87.9 91.2 95.3 93.1 96.8

Table 2. Runtime breakdown (second per frame on YCB-Video Dataset). Our method is approximately 200x faster than PoseCNN+ICP. Seg means Segmentation, and PE means Pose Estimation.

PoseCNN+ICP [41] Ours
Seg PE ICP ALL Seg PE Refine ALL
0.03 0.17 10.4 10.6 0.03 0.02 0.01 0.06

表 2. 运行时间分解(YCB-Video 数据集上每帧秒数)。我们的方法大约比 PoseCNN+ICP 快 200 倍。Seg 表示分割,PE 表示位姿估计。

PoseCNN+ICP [41] Ours
Seg PE ICP ALL Seg PE Refine ALL
0.03 0.17 10.4 10.6 0.03 0.02 0.01 0.06

Accuracy under increasing occlusion compared with PointFusion and PoseCNN+ICP

Figure 5. Model performance under increasing levels of occlusion. Here the levels of occlusion is estimated by calculating the invisible surface percentage of each object in the image frame. Our methods work more robustly under heavy occlusion compared to baseline methods.

图 5. 在遮挡逐渐加重时的模型表现。这里的遮挡程度,是通过计算每个物体在图像帧中不可见表面的比例来估计的。与基线方法相比,我们的方法在严重遮挡下更为稳健。

4.5. Evaluation on YCB-Video Dataset

4.5. 在 YCB-Video 数据集上的评测

Table 1 shows the evaluation results for all the 21 objects in the YCB-Video Dataset. We report the ADD-S AUC (<0.1m) and the ADD-S<2cm metrics on PoseCNN [41] and our four model variants. To ensure a fair comparison, all methods use the same segmentation masks as in PoseCNN [41]. Among our model variants, Ours (Iterative) achieves the best performance. Our method is able to outperform PoseCNN + ICP [41] even without iterative refinement. In particular, Ours (Iterative) outperforms PoseCNN + ICP by 3.5% on the ADD-S<2cm metric.

表 1 给出了 YCB-Video 数据集全部 21 个物体的评测结果。我们报告 PoseCNN [41] 以及四种模型变体的 ADD-S AUC(<0.1 m)和 ADD-S<2cm 指标。为公平比较,所有方法都使用与 PoseCNN [41] 相同的分割掩膜。在我们的变体中,Ours (Iterative) 表现最好。即便不做迭代精炼,我们的方法也已经能够超过 PoseCNN + ICP [41]。具体来说,Ours (Iterative) 在 ADD-S<2cm 指标上比 PoseCNN + ICP 高 3.5%。

Effect of dense fusion. Both of our dense fusion baselines (Ours (single) and Ours (per-pixel)) outperform PointFusion by a large margin, which shows that dense fusion has a clear advantage over the global fusion-by-concatenation method used in PointFusion.

稠密融合的作用。 两个稠密融合基线(Ours (single) 和 Ours (per-pixel))都大幅超过 PointFusion,说明稠密融合明显优于 PointFusion 所用的全局拼接融合。

Effect of iterative refinement. Table 1 shows that our iterative refinement improves the overall pose estimation performance. In particular, it significantly improves the performances for texture-less symmetric object, e.g., bowl (29%), banana (6%), and extra large clamp (6%) which suffer from orientation ambiguity.

迭代精炼的作用。 表 1 表明,迭代精炼改进了总体的位姿估计表现。尤其是对缺乏纹理的对称物体,例如 bowl(29%)、banana(6%)以及 extra large clamp(6%),这些物体存在朝向歧义,改进更为明显。

Robustness towards occlusion. The main advantage of our dense fusion method is its robustness towards occlusions. To quantify the effect of occlusion on the final performance, we calculate the visible surface ratio of each object instance. Then we calculate how the accuracy (ADD-S<2cm percentage) changes with the extent of occlusion. As shown in Fig. 5, the performances of PointFusion and PoseCNN+ICP degrade significantly as the occlusion increases. In contrast, none of our methods experiences notable performance drop. In particular, the performance of both Ours (per-pixel) and Ours (iterative) only decrease by 2% overall.

对遮挡的稳健性。 稠密融合方法的主要优势,是对遮挡更为稳健。为量化遮挡对最终表现的影响,我们计算每个物体实例的可见表面比例,再观察准确率(ADD-S<2cm 的百分比)如何随遮挡程度变化。如图 5 所示,随着遮挡加重,PointFusion 和 PoseCNN+ICP 的表现明显下降。相比之下,我们的几种方法都没有出现明显下滑。具体来说,Ours (per-pixel) 和 Ours (iterative) 的总体表现只下降了 2%。

Time efficiency. We compare the time efficiency of our model with PoseCNN+ICP in Table 2. We can see that our method is two order of magnitude faster than PoseCNN+ICP. In particular, PoseCNN+ICP spends most of the time on the post processing ICP. In contrast, all of our computation component, namely segmentation (Seg), pose estimation (PE), and iterative refinement (Refine), are equally efficient, and the overall runtime is fast enough for real-time application (16 FPS, about 5 objects in each frame).

时间效率。 表 2 比较了我们的模型与 PoseCNN+ICP 的时间效率。可以看到,我们的方法比 PoseCNN+ICP 快两个数量级。PoseCNN+ICP 的时间主要花在事后 ICP 上。相比之下,我们各个计算部分,即分割(Seg)、位姿估计(PE)和迭代精炼(Refine),效率都比较均衡,总体运行时间也足以支持实时应用(16 FPS,每帧大约 5 个物体)。

Qualitative evaluation. Fig. 4 visualizes some sample predictions made by PoseCNN+ICP, PointFusion, and our iterative refinement model. As we can see, PoseCNN+ICP and PointFusion fail to estimate the correct pose of the bowl in the leftmost column and the cracker box in the middle column due to heavy occlusion, whereas our method remains robust. Another challenging case is the clamp in the middle row due to poor segmentation (not shown in the figure). Our approach localizes the clamp from only the visible part of the object and effectively reduces the dependency on accurate segmentation result.

定性评价。 图 4 展示了 PoseCNN+ICP、PointFusion 以及我们的迭代精炼模型给出的若干预测样本。可以看到,由于严重遮挡,PoseCNN+ICP 和 PointFusion 没能正确估计最左列中碗的位姿,以及中间列中饼干盒的位姿,而我们的方法仍然稳健。另一个困难例子是中间行的夹子,原因是分割较差(图中未画出分割)。我们的方法只根据物体可见部分来定位夹子,从而有效降低了对精确分割结果的依赖。

Table 3. Quantitative evaluation of 6D pose (ADD[13]) on the LineMOD dataset. Objects with bold name are symmetric.

ape ben. cam can cat drill. duck egg. glue hole. iron lamp phone MEAN
RGB BB8 w ref. [25] 40 92 56 64 63 74 44 58 41 67 84 77 54 63
DeepIM [17, 41] 77 98 94 97 82 95 78 97 99 53 98 98 88 89
RGB-D Imp. [31]+ICP 21 64 63 76 72 42 32 99 96 50 63 92 71 65
SSD6D [14]+ICP 65 80 78 86 70 73 66 100 100 49 78 73 79 79
PointFusion [42] 70 81 61 61 79 47 63 100 99 72 83 62 79 74
Ours (per-pixel) 80 84 77 87 89 78 76 100 99 79 92 92 88 86
Ours (iterative) 92 93 94 93 97 87 92 100 100 92 97 95 93 94

表 3. 在 LineMOD 数据集上对 6D 位姿(ADD[13])的定量评价。名称加粗的物体是对称物体。

ape ben. cam can cat drill. duck egg. glue hole. iron lamp phone MEAN
RGB BB8 w ref. [25] 40 92 56 64 63 74 44 58 41 67 84 77 54 63
DeepIM [17, 41] 77 98 94 97 82 95 78 97 99 53 98 98 88 89
RGB-D Imp. [31]+ICP 21 64 63 76 72 42 32 99 96 50 63 92 71 65
SSD6D [14]+ICP 65 80 78 86 70 73 66 100 100 49 78 73 79 79
PointFusion [42] 70 81 61 61 79 47 63 100 99 72 83 62 79 74
Ours (per-pixel) 80 84 77 87 89 78 76 100 99 79 92 92 88 86
Ours (iterative) 92 93 94 93 97 87 92 100 100 92 97 95 93 94

Iterative refinement on LineMOD objects with decreasing ADD error

Figure 6. Iterative refinement performance on LineMOD dataset. We visualize how our iterative refinement procedure corrects initially sub-optimal pose estimation.

图 6. 在 LineMOD 数据集上的迭代精炼表现。我们可视化迭代精炼过程如何纠正最初并不理想的位姿估计。

4.6. Evaluation on LineMOD Dataset

4.6. 在 LineMOD 数据集上的评测

Table 3 compares our method with previous RGB methods with depth refinement (ICP) (results from [31, 34]) on the ADD metric [13]. Even without the iterative refinement step, our method can outperform 7% over the state-of-the-art depth refinement method. After processing the iterative refinement approach, the final result has another 8% improvement, which proves that our learning-based depth method is superior to the sophisticated application of ICP in both accuracy and efficiency. We visualize the estimated 6D pose after each refinement iteration in Fig. 6, where our pose estimation improves by an average of 0.8 cm (ADD) after 2 refinement iterations. The results of some other color-only methods are also listed in Table 3 for reference.

表 3 在 ADD 指标 [13] 上,把我们的方法与以往先用 RGB、再做深度精炼(ICP)的方法进行比较(结果来自 [31, 34])。即便不使用迭代精炼,我们的方法也比当时最优的深度精炼方法高出 7%。再加上迭代精炼后,最终结果又提高了 8%。这说明,这种基于学习的深度方法在精度和效率上,都优于对 ICP 的复杂应用。图 6 可视化了每一次精炼迭代后的 6D 位姿估计:经过 2 次精炼,位姿估计平均改进 0.8 cm(ADD)。表 3 中也列出了一些仅用颜色的方法,供对照参考。

4.7. Robotic Grasping Experiment

4.7. 机器人抓取实验

In our last experiment, we evaluate whether the poses estimated by our approach are accurate enough to enable robot grasping and manipulation. As shown in Fig. 1, we place 5 YCB objects on a table and command the robot to grasp them using the estimated pose. We follow a similar procedure to Tremblay et al. [35]: we place the five objects in four different random locations on the table, at three random orientations, including configurations with partial occlusions. Since the order of picking the objects is not optimized, we do not allow configurations where objects lay on top of each other. The robot attempts 12 grasps on each object, 60 attempts in total. The robot uses the estimated object orientation to compute an alignment of the gripper’s fingers to the object narrower dimension.

在最后一项实验中,我们检验该方法估计出的位姿是否足够准确,能够支持机器人抓取和操作。如图 1 所示,我们把 5 个 YCB 物体放在桌上,并命令机器人依据估计出的位姿去抓取。做法类似于 Tremblay 等人 [35]:把这五个物体放在桌上四个不同的随机位置,以及三种随机朝向,其中也包括部分遮挡的情形。由于抓取顺序未经优化,我们不允许物体叠放。机器人对每个物体尝试 12 次抓取,一共 60 次。机器人利用估计出的物体朝向,把夹爪的手指对齐到物体较窄的那一维。

The robot succeeds on 73% of the grasps using our proposed approach to estimate the pose of the objects. The most difficult object to grasp is the banana (7 out of 12 successful attempts). One possible reason is that our banana model is not exactly the same as in the dataset – ours is plain yellow. This characteristic hinders the estimation, especially of the orientation, and leads to some failed grasp attempts along the longer axis of the object. In spite of this less accurate case, our results indicate that our approach is robust enough to be deployed in real-world robotic tasks without explicit domain adaptation, even with a different RGB-D sensor and in a different background than the ones in the training data.

使用我们提出的方法估计物体位姿时,机器人的抓取成功率为 73%。最难抓的是香蕉(12 次中成功 7 次)。一个可能的原因是,我们使用的香蕉模型和数据集中的并不完全一样——我们的是纯黄色。这一差异妨碍了估计,尤其是朝向估计,从而导致一些沿着物体长轴方向的抓取失败。尽管有这一不够准确的情形,结果仍然表明,该方法稳健到可以直接部署到真实机器人任务中,不必做显式的域适应,即便 RGB-D 传感器和背景都与训练数据不同。

  1. Conclusion
  1. 结论

We presented a novel approach to estimating 6D poses of known objects from RGB-D images. Our approach fuses a dense representation of features that include color and depth information based on the confidence of their predictions. With this dense fusion approach, our method outperforms previous approaches in several datasets, and is significantly more robust against occlusions. Additionally, we demonstrated that a robot can use our proposed approach to grasp and manipulate objects.

我们提出了一种从 RGB-D 图像估计已知物体 6D 位姿的新方法。该方法融合包含颜色和深度信息的稠密特征表示,并依据各预测的置信度来使用它们。凭借这种稠密融合,我们的方法在多个数据集上超过了已有方法,对遮挡也明显更为稳健。此外,我们还展示了机器人可以用该方法来抓取和操作物体。

Acknowledgement

致谢

This work has been partially supported by JD.com American Technologies Corporation (“JD”) under the SAIL-JD AI Research Initiative and by an ONR MURI award (1186514-1-TBCJE). This article solely reflects the opinions and conclusions of its authors and not JD or any entity associated with JD.com. We also want to thank Toyota Research Institute for the Human Support Robot which we used to perform our real robot experiments.

本工作部分得到京东美国技术公司(“JD”)通过 SAIL-JD 人工智能研究计划提供的支持,以及一项 ONR MURI 资助(1186514-1-TBCJE)。本文仅反映作者的观点和结论,并不代表 JD 或任何与 JD.com 相关实体的立场。我们也感谢丰田研究院提供 Human Support Robot,供我们进行真机实验。

References

参考文献

[1] M. Aubry, D. Maturana, A. A. Efros, B. C. Russell, and J. Sivic, “Seeing 3d chairs: Exemplar part-based 2d-3d alignment using a large dataset of cad models,” in Proceedings of the IEEE Computer Vision and Pattern Recognition (cvpr), 2014, pp. 3762–3769.

[2] P. J. Besl and N. D. McKay, “A method for registration of 3-d shapes,” Ieee trans. pattern anal. mach. intell., vol. 14, pp. 239–256, 1992.

[3] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, and C. Rother, “Learning 6d object pose estimation using 3d object coordinates,” in European conference on computer vision, Springer, 2014, pp. 536–551.

[4] A. G. Buch, L. Kiforenko, and D. Kraft, “Rotational subgroup voting and pose clustering for robust 3d object recognition,” in Computer vision (iccv), 2017 ieee international conference on, IEEE, 2017, pp. 4137–4145.

[5] B. Calli, A. Singh, A. Walsman, S. S. Srinivasa, P. Abbeel, and A. M. Dollar, “The ycb object and model set: Towards common benchmarks for manipulation research,” 2015 international conference on advanced robotics (icar), pp. 510–517, 2015.

[6] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, “Multi-view 3d object detection network for autonomous driving,” in Proceedings of the IEEE Computer Vision and Pattern Recognition (cvpr), 2017.

[7] A. Collet, M. Martinez, and S. S. Srinivasa, “The moped framework: Object recognition and pose estimation for manipulation,” The international journal of robotics research, vol. 30, no. 10, pp. 1284–1306, 2011.

[8] B. Drost, M. Ulrich, N. Navab, and S. Ilic, “Model globally, match locally: Efficient and robust 3d object recognition,” in Computer vision and pattern recognition (cvpr), 2010 ieee conference on, Ieee, 2010, pp. 998–1005.

[9] V. Ferrari, T. Tuytelaars, and L. Van Gool, “Simultaneous object recognition and segmentation from single or multiple model views,” International journal of computer vision, vol. 67, no. 2, pp. 159–188, 2006.

[10] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the acm, vol. 24, no. 6, pp. 381–395, 1981.

[11] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in Proceedings of the IEEE Computer Vision and Pattern Recognition (cvpr), IEEE, 2012, pp. 3354–3361.

[12] S. Hinterstoisser, S. Holzer, C. Cagniart, S. Ilic, K. Konolige, N. Navab, and V. Lepetit, “Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes,” Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 858–865, 2011.

[13] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab, “Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes,” in Asian conference on computer vision, Springer, 2012, pp. 548–562.

[14] W. Kehl, F. Manhardt, F. Tombari, S. Ilic, and N. Navab, “Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 22–29.

[15] W. Kehl, F. Milletari, F. Tombari, S. Ilic, and N. Navab, “Deep learning of local rgb-d patches for 3d object detection and 6d pose estimation,” in European conference on computer vision, Springer, 2016, pp. 205–220.

[16] C. Li, J. Bai, and G. D. Hager, “A unified framework for multi-view multi-class object pose estimation,” Arxiv preprint arxiv:1803.08103, 2018.

[17] Y. Li, G. Wang, X. Ji, Y. Xiang, and D. Fox, “Deepim: Deep iterative matching for 6d pose estimation,” Arxiv:1804.00175, 2018.

[18] F. Manhardt, W. Kehl, N. Navab, and F. Tombari, “Deep model-based 6d pose refinement in rgb,” in Eccv, 2018.

[19] E. Marchand, H. Uchiyama, and F. Spindler, “Pose estimation for augmented reality: A hands-on survey,” Ieee transactions on visualization and computer graphics, vol. 22, no. 12, pp. 2633–2651, 2016.

[20] E. Marder-Eppstein, “Project tango,” in Acm siggraph 2016 real-time live!, ser. SIGGRAPH ’16, Anaheim, California: ACM, 2016, 40:25–40:25.

[21] A. Mousavian, D. Anguelov, J. Flynn, and J. Kosecka, “3d bounding box estimation using deep learning and geometry,” in Proceedings of the IEEE Computer Vision and Pattern Recognition (cvpr), 2017.

[22] G. Pavlakos, X. Zhou, A. Chan, K. G. Derpanis, and K. Daniilidis, “6-dof object pose from semantic keypoints,” Arxiv preprint arxiv:1703.04670, 2017.

[23] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, “Frustum pointnets for 3d object detection from rgb-d data,” Arxiv preprint arxiv:1711.08488, 2017.

[24] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” Arxiv preprint arxiv:1612.00593, 2016.

[25] M. Rad and V. Lepetit, “Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth.”

[26] R. Rios-Cabrera and T. Tuytelaars, “Discriminatively trained templates for 3d object detection: A real time scalable approach,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2013, pp. 2048– 2055.

[27] F. Rothganger, S. Lazebnik, C. Schmid, and J. Ponce, “3d object modeling and recognition using local affine-invariant image descriptors and multi-view spatial constraints,” International journal of computer vision, vol. 66, no. 3, pp. 231–259, 2006.

[28] M. Schwarz, H. Schulz, and S. Behnke, “Rgb-d object recognition and pose estimation based on pre-trained convolutional neural network features,” in Robotics and automation (icra), 2015 ieee international conference on, IEEE, 2015, pp. 1329–1335.

[29] S. Song and J. Xiao, “Sliding shapes for 3d object detection in depth images,” in European conference on computer vision, Springer, 2014, pp. 634–651.

[30] ——, “Deep sliding shapes for amodal 3d object detection in rgb-d images,” in Proceedings of the IEEE Computer Vision and Pattern Recognition (cvpr), 2016, pp. 808–816.

[31] M. Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and R. Triebel, “Implicit 3d orientation learning for 6d object detection from rgb images,” in European conference on computer vision, Springer, 2018, pp. 712–729.

[32] S. Suwajanakorn, N. Snavely, J. Tompson, and M. Norouzi, “Discovery of latent 3d keypoints via end-to-end geometric reasoning,” Arxiv preprint arxiv:1807.03146, 2018.

[33] A. Tejani, D. Tang, R. Kouskouridas, and T.-K. Kim, “Latent-class hough forests for 3d object detection and pose estimation,” in Proceedings of the European Conference on Computer Vision, Springer, 2014, pp. 462–477.

[34] B. Tekin, S. N. Sinha, and P. Fua, “Real-Time Seamless Single Shot 6D Object Pose Prediction,” in Proceedings of the IEEE Computer Vision and Pattern Recognition (cvpr), 2018.

[35] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield, “Deep object pose estimation for semantic robotic grasping of household objects,” Arxiv preprint arxiv:1809.10790, 2018.

[36] S. Tulsiani and J. Malik, “Viewpoints and keypoints,” in Proceedings of the IEEE Computer Vision and Pattern Recognition (cvpr), 2015, pp. 1510–1519.

[37] J. Vidal, C.-Y. Lin, and R. Martí, “6d pose estimation using an improved method based on point pair features,” in 2018 4th international conference on control, automation and robotics (iccar), IEEE, 2018, pp. 405–409.

[38] P. Wohlhart and V. Lepetit, “Learning descriptors for object recognition and 3d pose estimation,” in Proceedings of the IEEE Computer Vision and Pattern Recognition (cvpr), 2015, pp. 3109–3118.

[39] Y. Xiang, W. Choi, Y. Lin, and S. Savarese, “Data-driven 3d voxel patterns for object category recognition,” in Proceedings of the IEEE Computer Vision and Pattern Recognition (cvpr), 2015, pp. 1903–1911.

[40] ——, “Subcategory-aware convolutional neural networks for object proposals and detection,” in Applications of computer vision (wacv), 2017 ieee winter conference on, IEEE, 2017, pp. 924–933.

[41] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,” Arxiv preprint arxiv:1711.00199, 2017.

[42] D. Xu, D. Anguelov, and A. Jain, “Pointfusion: Deep sensor fusion for 3d bounding box estimation,” Arxiv preprint arxiv:1711.10871, 2017.

[43] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” Arxiv preprint arxiv:1711.06396, 2017.

[44] M. Zhu, K. G. Derpanis, Y. Yang, S. Brahmbhatt, M. Zhang, C. Phillips, M. Lecce, and K. Daniilidis, “Single image 3d object detection and pose estimation for grasping,” in Robotics and automation (icra), 2014 ieee international conference on, IEEE, 2014, pp. 3936–3943.