PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

PoseCNN：用于杂乱场景中六自由度物体位姿估计的卷积神经网络

Yu Xiang 1,2 , Tanner Schmidt 2 , Venkatraman Narayanan 3 and Dieter Fox 1,2 Affiliation:  1 NVIDIA Research, 2 University of Washington, 3 Carnegie Mellon University yux@nvidia.com, tws10@cs.washington.edu, venkatraman@cs.cmu.edu, dieterf@nvidia.com

Abstract

摘要

Estimating the 6D pose of known objects is important for robots to interact with the real world. The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects. In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation. PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. The 3D rotation of the object is estimated by regressing to a quaternion representation. We also introduce a novel loss function that enables PoseCNN to handle symmetric objects. In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset. Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input. When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset. Our code and dataset are available at https://rse-lab.cs.washington.edu/projects/posecnn/.

估计已知物体的六自由度位姿，对机器人与真实世界交互十分重要。由于物体种类繁多，加上场景中的杂乱以及物体之间的遮挡，这一问题颇具挑战。本文提出 PoseCNN，一种用于六自由度物体位姿估计的新型卷积神经网络。PoseCNN 通过在图像中定位物体中心并预测它与相机的距离，来估计物体的三维平移；物体的三维旋转则通过对四元数表示做回归来估计。我们还提出一种新的损失函数，使 PoseCNN 能够处理对称物体。此外，我们贡献了一个用于六自由度物体位姿估计的大规模视频数据集，命名为 YCB-Video。该数据集提供来自 YCB 数据集的 21 个物体在 92 段视频、共 133,827 帧中的准确六自由度位姿。我们在自己的 YCB-Video 数据集以及 OccludedLINEMOD 数据集上进行了大量实验，表明 PoseCNN 对遮挡非常稳健，能够处理对称物体，并且仅用彩色图像作为输入就能提供准确的位姿估计。当再用深度数据进一步精修位姿时，我们的方法在具有挑战性的 OccludedLINEMOD 数据集上取得了当时的领先结果。代码和数据集见 https://rse-lab.cs.washington.edu/projects/posecnn/。

I INTRODUCTION

I 引言

Recognizing objects and estimating their poses in 3D has a wide range of applications in robotic tasks. For instance, recognizing the 3D location and orientation of objects is important for robot manipulation. It is also useful in human-robot interaction tasks such as learning from demonstration. However, the problem is challenging due to the variety of objects in the real world. They have different 3D shapes, and their appearances on images are affected by lighting conditions, clutter in the scene and occlusions between objects.

识别物体并估计它们在三维中的位姿，在机器人任务中有广泛用途。例如，识别物体的三维位置和朝向对机器人操作很重要；它在人机交互任务中也同样有用，例如从示范中学习。不过，由于真实世界中物体种类繁多，这一问题颇具挑战：它们具有不同的三维形状，其在图像上的外观还会受到光照条件、场景杂乱以及物体之间遮挡的影响。

Traditionally, the problem of 6D object pose estimation is tackled by matching feature points between 3D models and images [20, 25, 8]. However, these methods require that there are rich textures on the objects in order to detect feature points for matching. As a result, they are unable to handle texture-less objects. With the emergence of depth cameras, several methods have been proposed for recognizing texture-less objects using RGB-D data [13, 3, 2, 26, 15]. For template-based methods [13, 12], occlusions significantly reduce the recognition performance. Alternatively, methods that perform learning to regress image pixels to 3D object coordinates in order to establish the 2D-3D correspondences for 6D pose estimation [3, 4] cannot handle symmetric objects.

传统上，六自由度物体位姿估计通过在三维模型与图像之间匹配特征点来完成 [20, 25, 8]。不过，这些方法要求物体具有丰富纹理，才能检测出用于匹配的特征点，因而无法处理无纹理物体。随着深度相机的出现，已有若干方法利用 RGB-D 数据识别无纹理物体 [13, 3, 2, 26, 15]。对基于模板的方法 [13, 12] 来说，遮挡会显著降低识别表现。另一类通过学习把图像像素回归到三维物体坐标、以建立用于六自由度位姿估计的二维–三维对应的方法 [3, 4]，则无法处理对称物体。

Fig. 1: We propose a novel PoseCNN for 6D object pose estimation, where the network is trained to perform three tasks: semantic labeling, 3D translation estimation, and 3D rotation regression.

图 1：我们提出用于六自由度物体位姿估计的新型网络 PoseCNN，该网络被训练去完成三项任务：语义标注、三维平移估计和三维旋转回归。

In this work, we propose a generic framework for 6D object pose estimation where we attempt to overcome the limitations of existing methods. We introduce a novel Convolutional Neural Network (CNN) for end-to-end 6D pose estimation named PoseCNN. A key idea behind PoseCNN is to decouple the pose estimation task into different components, which enables the network to explicitly model the dependencies and independencies between them. Specifically, PoseCNN performs three related tasks as illustrated in Fig. 1. First, it predicts an object label for each pixel in the input image. Second, it estimates the 2D pixel coordinates of the object center by predicting a unit vector from each pixel towards the center. Using the semantic labels, image pixels associated with an object vote on the object center location in the image. In addition, the network also estimates the distance of the object center. Assuming known camera intrinsics, estimation of the 2D object center and its distance enables us to recover its 3D translation $\mathbf{T}$. Finally, the 3D Rotation $\mathbf{R}$ is estimated by regressing convolutional features extracted inside the bounding box of the object to a quaternion representation of $\mathbf{R}$. As we will show, the 2D center voting followed by rotation regression to estimate $\mathbf{R}$ and $\mathbf{T}$ can be applied to textured/texture-less objects and is robust to occlusions since the network is trained to vote on object centers even when they are occluded.

本文提出一个用于六自由度物体位姿估计的通用框架，试图克服已有方法的局限。我们引入一种用于端到端六自由度位姿估计的新型卷积神经网络（CNN），命名为 PoseCNN。PoseCNN 背后的一个关键想法，是把位姿估计任务拆成不同组成部分，使网络能够显式地建模它们之间的依赖与独立关系。具体来说，PoseCNN 完成如图 1 所示的三项相关任务。第一，它为输入图像中的每个像素预测物体标签。第二，它通过预测每个像素指向中心的单位向量，来估计物体中心的二维像素坐标。利用语义标签，属于某一物体的图像像素对该物体在图像中的中心位置投票。此外，网络还估计物体中心的距离。在相机内参已知的前提下，估计二维物体中心及其距离，就能恢复三维平移 $\mathbf{T}$。最后，三维旋转 $\mathbf{R}$ 通过对物体边界框内提取的卷积特征回归到 $\mathbf{R}$ 的四元数表示来估计。我们将表明，这种先对二维中心投票、再回归旋转以估计 $\mathbf{R}$ 和 $\mathbf{T}$ 的做法，既可用于有纹理物体也可用于无纹理物体，并且对遮挡稳健，因为网络被训练成即使中心被遮挡，像素仍能对物体中心投票。

Handling symmetric objects is another challenge for pose estimation, since different object orientations may generate identical observations. For instance, it is not possible to uniquely estimate the orientation of the red bowl or the wood block shown in Fig. 5. While pose benchmark datasets such as the OccludedLINEMOD dataset [17] consider a special symmetric evaluation for such objects, symmetries are typically ignored during network training. However, this can result in bad training performance since a network receives inconsistent loss signals, such as a high loss on an object orientation even though the estimation from the network is correct with respect to the symmetry of the object. Inspired by this observation, we introduce ShapeMatch-Loss, a new loss function that focuses on matching the 3D shape of an object. We will show that this loss function produces superior estimation for objects with shape symmetries.

处理对称物体是位姿估计的另一项挑战，因为物体的不同朝向可能产生相同的观测。例如，无法唯一地估计图 5 中红色碗或木块的朝向。虽然像 OccludedLINEMOD 数据集 [17] 这样的位姿评测集会为这类物体考虑专门的对称性评价，但对称性在网络训练中通常被忽略。这可能导致训练效果不佳：网络会收到不一致的损失信号，例如即使网络的估计相对于物体对称性是正确的，仍会因某个朝向而得到高损失。受这一观察启发，我们提出 ShapeMatch-Loss，一种关注匹配物体三维形状的新损失函数。我们将表明，该损失函数对具有形状对称性的物体能给出更好的估计。

We evaluate our method on the OccludedLINEMOD dataset [17], a benchmark dataset for 6D pose estimation. On this challenging dataset, PoseCNN achieves state-of-the-art results for both color only and RGB-D pose estimation (we use depth images in the Iterative Closest Point (ICP) algorithm for pose refinement). To thoroughly evaluate our method, we additionally collected a large scale RGB-D video dataset named YCB-Video, which contains 6D poses of 21 objects from the YCB object set [5] in 92 videos with a total of 133,827 frames. Objects in the dataset exhibit different symmetries and are arranged in various poses and spatial configurations, generating severe occlusions between them.

我们在 OccludedLINEMOD 数据集 [17] 上评估方法，这是一个六自由度位姿估计的基准数据集。在这个具有挑战性的数据集上，PoseCNN 在仅用彩色图像以及 RGB-D 位姿估计上都取得了当时的领先结果（我们在迭代最近点（ICP）算法中使用深度图像做位姿精修）。为更全面地评估方法，我们另外采集了一个大规模 RGB-D 视频数据集，命名为 YCB-Video，其中包含来自 YCB 物体集 [5] 的 21 个物体在 92 段视频、共计 133,827 帧中的六自由度位姿。数据集中的物体表现出不同的对称性，并以各种位姿和空间配置摆放，从而在物体之间产生严重遮挡。

In summary, our work has the following key contributions:

概括起来，本文的主要贡献如下：

• • We propose a novel convolutional neural network for 6D object pose estimation named PoseCNN. Our network achieves end-to-end 6D pose estimation and is very robust to occlusions between objects.

• 我们提出一种用于六自由度物体位姿估计的新型卷积神经网络，命名为 PoseCNN。该网络实现端到端的六自由度位姿估计，并且对物体之间的遮挡非常稳健。

• • We introduce ShapeMatch-Loss, a new training loss function for pose estimation of symmetric objects.

• 我们提出 ShapeMatch-Loss，一种用于对称物体位姿估计的新训练损失函数。

• • We contribute a large scale RGB-D video dataset for 6D object pose estimation, where we provide 6D pose annotations for 21 YCB objects.

• 我们贡献一个用于六自由度物体位姿估计的大规模 RGB-D 视频数据集，为 21 个 YCB 物体提供六自由度位姿标注。

This paper is organized as follows. After discussing related work, we introduce PoseCNN for 6D object pose estimation, followed by experimental results and a conclusion.

本文结构如下。在讨论相关工作之后，我们介绍用于六自由度物体位姿估计的 PoseCNN，随后给出实验结果和结论。

II RELATED WORK

II 相关工作

6D object pose estimation methods in the literature can be roughly classified into template-based methods and feature-based methods. In template-based methods, a rigid template is constructed and used to scan different locations in the input image. At each location, a similarity score is computed, and the best match is obtained by comparing these similarity scores [12, 13, 6]. In 6D pose estimation, a template is usually obtained by rendering the corresponding 3D model. Recently, 2D object detection methods are used as template matching and augmented for 6D pose estimation, especially with deep learning-based object detectors [28, 23, 16, 29]. Template-based methods are useful in detecting texture-less objects. However, they cannot handle occlusions between objects very well, since the template will have low similarity score if the object is occluded.

文献中的六自由度物体位姿估计方法大致可分为基于模板的方法和基于特征的方法。在基于模板的方法中，会构造一个刚性模板，并用它扫描输入图像中的不同位置。在每个位置计算相似度分数，再通过比较这些分数得到最佳匹配 [12, 13, 6]。在六自由度位姿估计中，模板通常通过对相应三维模型渲染得到。近来，二维目标检测方法被当作模板匹配使用，并被扩展到六自由度位姿估计，尤其是基于深度学习的检测器 [28, 23, 16, 29]。基于模板的方法有助于检测无纹理物体。不过，它们不太能很好地处理物体之间的遮挡，因为一旦物体被遮挡，模板的相似度分数就会变低。

In feature-based methods, local features are extracted from either points of interest or every pixel in the image and matched to features on the 3D models to establish the 2D-3D correspondences, from which 6D poses can be recovered [20, 25, 30, 22]. Feature-based methods are able to handle occlusions between objects. However, they require sufficient textures on the objects in order to compute the local features. To deal with texture-less objects, several methods are proposed to learn feature descriptors using machine learning techniques [32, 10]. A few approaches have been proposed to directly regress to 3D object coordinate location for each pixel to establish the 2D-3D correspondences [3, 17, 4]. But 3D coordinate regression encounters ambiguities in dealing with symmetric objects.

在基于特征的方法中，局部特征从兴趣点或图像中的每个像素提取，并与三维模型上的特征匹配，以建立二维–三维对应，进而恢复六自由度位姿 [20, 25, 30, 22]。基于特征的方法能够处理物体之间的遮挡。不过，它们要求物体具有足够纹理，才能计算局部特征。为处理无纹理物体，已有若干方法用机器学习技术学习特征描述子 [32, 10]。还有一些方法直接把每个像素回归到三维物体坐标位置，以建立二维–三维对应 [3, 17, 4]。但三维坐标回归在处理对称物体时会遇到歧义。

In this work, we combine the advantages of both template-based methods and feature-based methods in a deep learning framework, where the network combines bottom-up pixel-wise labeling with top-down object pose regression. Recently, the 6D object pose estimation problem has received more attention thanks to the competition in the Amazon Picking Challenge (APC). Several datasets and approaches have been introduced for the specific setting in the APC [24, 35]. Our network has the potential to be applied to the APC setting as long as the appropriate training data is provided.

本文在一个深度学习框架中结合基于模板方法和基于特征方法的优点：网络把自下而上的逐像素标注与自上而下的物体位姿回归结合在一起。近来，得益于 Amazon Picking Challenge（APC）中的竞赛，六自由度物体位姿估计问题受到了更多关注。已有若干数据集和方法针对 APC 中的特定设定被提出 [24, 35]。只要提供相应的训练数据，我们的网络也有潜力应用于 APC 设定。

III PoseCNN

Fig. 2: Architecture of PoseCNN for 6D object pose estimation.

图 2：用于六自由度物体位姿估计的 PoseCNN 架构。

Given an input image, the task of 6D object pose estimation is to estimate the rigid transformation from the object coordinate system $O$ to the camera coordinate system $C$. We assume that the 3D model of the object is available and the object coordinate system is defined in the 3D space of the model. The rigid transformation here consists of an SE(3) transform containing a 3D rotation $\mathbf{R}$ and a 3D translation $\mathbf{T}$, where $\mathbf{R}$ specifies the rotation angles around the $X$-axis, $Y$-axis and $Z$-axis of the object coordinate system $O$, and $\mathbf{T}$ is the coordinate of the origin of $O$ in the camera coordinate system $C$. In the imaging process, $\mathbf{T}$ determines the object location and scale in the image, while $\mathbf{R}$ affects the image appearance of the object according to the 3D shape and texture of the object. Since these two parameters have distinct visual properties, we propose a convolutional neural network architecture that internally decouples the estimation of $\mathbf{R}$ and $\mathbf{T}$.

给定输入图像，六自由度物体位姿估计的任务是估计从物体坐标系 $O$ 到相机坐标系 $C$ 的刚体变换。我们假定物体的三维模型可用，并且物体坐标系定义在模型的三维空间中。这里的刚体变换是一个包含三维旋转 $\mathbf{R}$ 和三维平移 $\mathbf{T}$ 的 SE(3) 变换，其中 $\mathbf{R}$ 指定绕物体坐标系 $O$ 的 $X$ 轴、$Y$ 轴和 $Z$ 轴的旋转角，$\mathbf{T}$ 是 $O$ 的原点在相机坐标系 $C$ 中的坐标。在成像过程中，$\mathbf{T}$ 决定物体在图像中的位置和尺度，而 $\mathbf{R}$ 则根据物体的三维形状和纹理影响其图像外观。由于这两个参数具有不同的视觉性质，我们提出一种在内部把 $\mathbf{R}$ 和 $\mathbf{T}$ 的估计解耦开的卷积神经网络架构。

III-A Overview of the Network

III-A 网络概览

Fig. 2 illustrates the architecture of our network for 6D object pose estimation. The network contains two stages. The first stage consists of 13 convolutional layers and 4 max-pooling layers, which extract feature maps with different resolutions from the input image. This stage is the backbone of the network since the extracted features are shared across all the tasks performed by the network. The second stage consists of an embedding step that embeds the high-dimensional feature maps generated by the first stage into low-dimensional, task-specific features. Then, the network performs three different tasks that lead to the 6D pose estimation, i.e., semantic labeling, 3D translation estimation, and 3D rotation regression, as described next.

图 2 展示了我们用于六自由度物体位姿估计的网络架构。网络包含两个阶段。第一阶段由 13 个卷积层和 4 个最大池化层组成，从输入图像中提取不同分辨率的特征图。这一阶段是网络的主干，因为提取出的特征会在网络完成的所有任务之间共享。第二阶段包含一个嵌入步骤，把第一阶段生成的高维特征图嵌入到低维、任务特定的特征中。随后，网络完成通向六自由度位姿估计的三项不同任务，即语义标注、三维平移估计和三维旋转回归，如下所述。

III-B Semantic Labeling

III-B 语义标注

In order to detect objects in images, we resort to semantic labeling, where the network classifies each image pixel into an object class. Compared to recent 6D pose estimation methods that resort to object detection with bounding boxes [23, 16, 29], semantic labeling provides richer information about the objects and handles occlusions better.

为了在图像中检测物体，我们采用语义标注，由网络把每个图像像素分类到一个物体类别。与近来借助边界框做目标检测的六自由度位姿估计方法 [23, 16, 29] 相比，语义标注能提供关于物体的更丰富信息，并且更好地处理遮挡。

The embedding step of the semantic labeling branch, as shown in Fig. 2, takes two feature maps with channel dimension 512 generated by the feature extraction stage as inputs. The resolutions of the two feature maps are $1/8$ and $1/16$ of the original image size, respectively. The network first reduces the channel dimension of the two feature maps to 64 using two convolutional layers. Then it doubles the resolution of the $1/16$ feature map with a deconvolutional layer. After that, the two feature maps are summed and another deconvolutional layer is used to increase the resolution by 8 times in order to obtain a feature map with the original image size. Finally, a convolutional layer operates on the feature map and generates semantic labeling scores for pixels. The output of this layer has $n$ channels with $n$ the number of the semantic classes. In training, a softmax cross entropy loss is applied to train the semantic labeling branch. While in testing, a softmax function is used to compute the class probabilities of the pixels. The design of the semantic labeling branch is inspired by the fully convolutional network in [19] for semantic labeling. It is also used in our previous work for scene labeling [34].

如图 2 所示，语义标注分支的嵌入步骤以特征提取阶段生成的两张通道维数为 512 的特征图作为输入。这两张特征图的分辨率分别是原图的 $1/8$ 和 $1/16$。网络先用两个卷积层把两张特征图的通道维数降到 64，再用一个反卷积层把 $1/16$ 特征图的分辨率加倍。此后两张特征图相加，再用另一个反卷积层把分辨率提高 8 倍，以得到与原图大小相同的特征图。最后，一个卷积层作用在该特征图上，生成像素的语义标注分数。该层的输出有 $n$ 个通道，$n$ 为语义类别数。训练时，对语义标注分支施加 softmax 交叉熵损失；测试时则用 softmax 函数计算像素的类别概率。语义标注分支的设计受到 [19] 中用于语义标注的全卷积网络启发，也用在我们此前的场景标注工作中 [34]。

III-C 3D Translation Estimation

III-C 三维平移估计

Fig. 3: Illustration of the object coordinate system and the camera coordinate system. The 3D translation can be estimated by localizing the 2D center of the object and estimating the 3D center distance from the camera.

图 3：物体坐标系与相机坐标系的示意。三维平移可以通过定位物体的二维中心，并估计三维中心到相机的距离来估计。

As illustrated in Fig. 3, the 3D translation $\mathbf{T}=(T_{x},T_{y},T_{z})^{T}$ is the coordinate of the object origin in the camera coordinate system. A naive way of estimating $\mathbf{T}$ is to directly regress the image features to $\mathbf{T}$. However, this approach is not generalizable since objects can appear in any location in the image. Also, it cannot handle multiple object instances in the same category. Therefore, we propose to estimate the 3D translation by localizing the 2D object center in the image and estimating object distance from the camera. To see, suppose the projection of $\mathbf{T}$ on the image is $\mathbf{c}=(c_{x},c_{y})^{T}$. If the network can localize $\mathbf{c}$ in the image and estimate the depth $T_{z}$, then we can recover $T_{x}$ and $T_{y}$ according to the following projection equation assuming a pinhole camera:

如图 3 所示，三维平移 $\mathbf{T}=(T_{x},T_{y},T_{z})^{T}$ 是物体原点在相机坐标系中的坐标。一种朴素的估计 $\mathbf{T}$ 的方式，是直接把图像特征回归到 $\mathbf{T}$。不过，这种方法难以泛化，因为物体可以出现在图像中的任意位置；它也无法处理同一类别中的多个物体实例。因此，我们提出通过在图像中定位二维物体中心、并估计物体到相机的距离来估计三维平移。设 $\mathbf{T}$ 在图像上的投影为 $\mathbf{c}=(c_{x},c_{y})^{T}$。如果网络能够在图像中定位 $\mathbf{c}$ 并估计深度 $T_{z}$，那么在针孔相机假设下，就可以根据如下投影方程恢复 $T_{x}$ 和 $T_{y}$：

$$\begin{bmatrix}c_{x}\[5.0pt]
c_{y}\end{bmatrix}=\begin{bmatrix}f_{x}\frac{T_{x}}{T_{z}}+p_{x}\[5.0pt]
f_{y}\frac{T_{y}}{T_{z}}+p_{y}\end{bmatrix}, \tag{1}$$

where $f_{x}$ and $f_{y}$ denote the focal lengths of the camera, and $(p_{x},p_{y})^{T}$ is the principal point. If the object origin $O$ is the centroid of the object, we call $\mathbf{c}$ the 2D center of the object.

其中 $f_{x}$ 和 $f_{y}$ 表示相机的焦距，$(p_{x},p_{y})^{T}$ 是主点。如果物体原点 $O$ 是物体的质心，我们就称 $\mathbf{c}$ 为物体的二维中心。

A straightforward way for localizing the 2D object center is to directly detect the center point as in existing key point detection methods [22, 7]. However, these methods would not work if the object center is occluded. Inspired by the traditional Implicit Shape Model (ISM) in which image patches vote for the object center for detection [18], we design our network to regress to the center direction for each pixel in the image. Specifically, for a pixel $\mathbf{p}=(x,y)^{T}$ on the image, it regresses to three variables:

定位二维物体中心的一种直接做法，是像已有关键点检测方法那样直接检测中心点 [22, 7]。不过，如果物体中心被遮挡，这些方法就无法工作。受传统隐式形状模型（ISM）中图像块为检测而对物体中心投票的启发 [18]，我们把网络设计成对图像中每个像素回归中心方向。具体来说，对图像上的像素 $\mathbf{p}=(x,y)^{T}$，它回归到三个变量：

$$(x,y)\rightarrow\Big(n_{x}=\frac{c_{x}-x}{\|\mathbf{c}-\mathbf{p}\|},n_{y}=\frac{c_{y}-y}{\|\mathbf{c}-\mathbf{p}\|},T_{z}\Big). \tag{2}$$

Note that instead of directly regressing to the displacement vector $\mathbf{c}-\mathbf{p}$, we design the network to regress to the unit length vector $\mathbf{n}=(n_{x},n_{y})^{T}=\frac{\mathbf{c}-\mathbf{p}}{\|\mathbf{c}-\mathbf{p}\|}$, i.e., 2D center direction, which is scale-invariant and therefore easier to be trained (as we verified experimentally).

需要注意，我们并不直接回归位移向量 $\mathbf{c}-\mathbf{p}$，而是让网络回归单位长度向量 $\mathbf{n}=(n_{x},n_{y})^{T}=\frac{\mathbf{c}-\mathbf{p}}{\|\mathbf{c}-\mathbf{p}\|}$，也就是二维中心方向。它具有尺度不变性，因而更容易训练（我们也用实验验证了这一点）。

The center regression branch of our network (Fig. 2) uses the same architecture as the semantic labeling branch, except that the channel dimensions of the convolutional layers and the deconvolutional layers are different. We embed the high-dimensional features into a 128-dimensional space instead of 64-dimensional since this branch needs to regress to three variables for each object class. The last convolutional layer in this branch has channel dimension $3\times n$ with $n$ the number of object classes. In training, a smoothed L1 loss function is applied for regression as in [11].

我们网络的中心回归分支（图 2）采用与语义标注分支相同的架构，只是卷积层和反卷积层的通道维数不同。我们把高维特征嵌入到 128 维空间而不是 64 维，因为该分支需要对每个物体类别回归三个变量。该分支最后一个卷积层的通道维数为 $3\times n$，其中 $n$ 是物体类别数。训练时，回归使用与 [11] 中相同的平滑 L1 损失函数。

Fig. 4: Illustration of Hough voting for object center localization: Each pixel casts votes for image locations along the ray predicted from the network.

图 4：用霍夫投票定位物体中心的示意：每个像素沿着网络预测出的射线，为图像中的位置投票。

In order to find the 2D object center $\mathbf{c}$ of an object, a Hough voting layer is designed and integrated into the network. The Hough voting layer takes the pixel-wise semantic labeling results and the center regression results as inputs. For each object class, it first computes the voting score for every location in the image. The voting score indicates how likely the corresponding image location is the center of an object in the class. Specifically, each pixel in the object class adds votes for image locations along the ray predicted from the network (see Fig. 4). After processing all the pixels in the object class, we obtain the voting scores for all the image locations. Then the object center is selected as the location with the maximum score. For cases where multiple instances of the same object class may appear in the image, we apply non-maximum suppression to the voting scores, and then select locations with scores larger than a certain threshold.

为了找到某个物体的二维物体中心 $\mathbf{c}$，我们设计了一个霍夫投票层并将其集成到网络中。霍夫投票层以逐像素语义标注结果和中心回归结果作为输入。对每个物体类别，它首先计算图像中每个位置的投票分数。投票分数表示相应图像位置有多大可能是该类某个物体的中心。具体来说，属于该物体类别的每个像素，会沿着网络预测出的射线为图像位置增加投票（见图 4）。处理完该物体类别的全部像素后，我们就得到了所有图像位置的投票分数。随后，把分数最大的位置选为物体中心。当同一物体类别的多个实例可能出现在图像中时，我们先对投票分数做非极大值抑制，再选出分数超过某一阈值的位置。

After generating a set of object centers, we consider the pixels that vote for an object center to be the inliers of the center. Then the depth prediction of the center, $T_{z}$, is simply computed as the mean of the depths predicted by the inliers. Finally, using Eq. 1, we can estimate the 3D translation $\mathbf{T}$. In addition, the network generates the bounding box of the object as the 2D rectangle that bounds all the inliers, and the bounding box is used for 3D rotation regression.

生成一组物体中心之后，我们把为某个物体中心投票的像素视为该中心的内点。中心的深度预测 $T_{z}$ 就简单地取这些内点所预测深度的均值。最后，利用公式 1，我们就能估计三维平移 $\mathbf{T}$。此外，网络把包围全部内点的二维矩形作为物体的边界框，该边界框再用于三维旋转回归。

III-D 3D Rotation Regression

III-D 三维旋转回归

The lowest part of Fig. 2 shows the 3D rotation regression branch. Using the object bounding boxes predicted from the Hough voting layer, we utilize two RoI pooling layers [11] to “crop and pool” the visual features generated by the first stage of the network for the 3D rotation regression. The pooled feature maps are added together and fed into three Fully-Connected (FC) layers. The first two FC layers have dimension 4096, and the last FC layer has dimension $4\times n$ with $n$ the number of object classes. For each class, the last FC layer outputs a 3D rotation represented by a quaternion.

图 2 的最下方是三维旋转回归分支。利用霍夫投票层预测出的物体边界框，我们使用两个 RoI 池化层 [11]，对网络第一阶段生成的视觉特征进行“裁剪并池化”，以用于三维旋转回归。池化后的特征图相加，再送入三个全连接（FC）层。前两个 FC 层的维数为 4096，最后一个 FC 层的维数为 $4\times n$，其中 $n$ 是物体类别数。对每个类别，最后一个 FC 层输出用四元数表示的三维旋转。

To train the quaternion regression, we propose two loss functions, one of which is specifically designed to handle symmetric objects. The first loss, called PoseLoss (PLoss), operates in the 3D model space and measures the average squared distance between points on the correct model pose and their corresponding points on the model using the estimated orientation. PLoss is defined as

为训练四元数回归，我们提出两种损失函数，其中一种专门用于处理对称物体。第一种损失称为 PoseLoss（PLoss），它在三维模型空间中工作，度量正确模型位姿上的点与使用估计朝向后模型上对应点之间的平均平方距离。PLoss 定义为

$$\text{{PLoss}}(\mathbf{\tilde{q}},\mathbf{q})=\frac{1}{2m}\sum_{\mathbf{x}\in\mathcal{M}}\|R(\mathbf{\tilde{q}})\mathbf{x}-R(\mathbf{q})\mathbf{x}\|^{2}, \tag{3}$$

where $\mathcal{M}$ denotes the set of 3D model points and $m$ is the number of points. $R(\mathbf{\tilde{q}})$ and $R(\mathbf{q})$ indicate the rotation matrices computed from the the estimated quaternion and the ground truth quaternion, respectively. This loss has its unique minimum when the estimated orientation is identical to the ground truth orientation [^1]. Unfortunately, PLoss does not handle symmetric objects appropriately, since a symmetric object can have multiple correct 3D rotations. Using such a loss function on symmetric objects unnecessarily penalizes the network for regressing to one of the alternative 3D rotations, thereby giving possibly inconsistent training signals.

其中 $\mathcal{M}$ 表示三维模型点集，$m$ 是点数。$R(\mathbf{\tilde{q}})$ 和 $R(\mathbf{q})$ 分别表示由估计四元数和真实四元数计算得到的旋转矩阵。该损失在估计朝向与真实朝向完全相同时有唯一最小值 [^1]。遗憾的是，PLoss 并不能恰当地处理对称物体，因为对称物体可以有多个正确的三维旋转。对对称物体使用这样的损失函数，会不必要地惩罚网络回归到另一个等价的三维旋转，从而可能给出不一致的训练信号。

While PLoss could potentially be modified to handle symmetric objects by manually specifying object symmetries and then considering all correct orientations as ground truth options, we here introduce ShapeMatch-Loss (SLoss), a loss function that does not require the specification of symmetries. SLoss is defined as

虽然原则上可以通过手工指定物体对称性、并把所有正确朝向都视为真实选项，来修改 PLoss 以处理对称物体，但我们在此引入 ShapeMatch-Loss（SLoss），一种不需要指定对称性的损失函数。SLoss 定义为

$$\text{{SLoss}}(\mathbf{\tilde{q}},\mathbf{q})=\frac{1}{2m}\sum_{\mathbf{x}_{1}\in\mathcal{M}}\min_{\mathbf{x}_{2}\in\mathcal{M}}\|R(\mathbf{\tilde{q}})\mathbf{x}_{1}-R(\mathbf{q})\mathbf{x}_{2}\|^{2}. \tag{4}$$

As we can see, just like ICP, this loss measures the offset between each point on the estimated model orientation and the closest point on the ground truth model. SLoss is minimized when the two 3D models match each other. In this way, the SLoss will not penalize rotations that are equivalent with respect to the 3D shape symmetry of the object.

可以看到，就像 ICP 一样，该损失度量估计模型朝向上每个点与真实模型上最近点之间的偏移。当两个三维模型彼此匹配时，SLoss 被最小化。这样，SLoss 不会惩罚那些相对于物体三维形状对称性等价的旋转。

Fig. 5: The subset of 21 YCB Objects selected to appear in our dataset.

图 5：被选入我们数据集的 21 个 YCB 物体子集。

IV The YCB-Video Dataset

IV YCB-Video 数据集

Object-centric datasets providing ground-truth annotations for object poses and/or segmentations are limited in size by the fact that the annotations are typically provided manually. For example, the popular LINEMOD dataset [13] provides manual annotations for around 1,000 images for each of the 15 objects in the dataset. While such a dataset is useful for evaluation of model-based pose estimation techniques, it is orders of magnitude smaller than a typical dataset for training state-of-the-art deep neural networks. One solution to this problem is to augment the data with synthetic images. However, care must be taken to ensure that performance generalizes between real and rendered scenes.

提供物体位姿和/或分割真实标注的以物体为中心的数据集，其规模受限于标注通常需要手工完成。例如，常用的 LINEMOD 数据集 [13] 为其中 15 个物体各自提供了大约 1000 张图像的手工标注。这样的数据集对评估基于模型的位姿估计技术很有用，但其规模比训练当代深度神经网络的典型数据集小了几个数量级。一种解决办法是用合成图像增强数据。不过，必须小心确保表现能够在真实场景与渲染场景之间泛化。

IV-A 6D Pose Annotation

IV-A 六自由度位姿标注

To avoid annotating all the video frames manually, we manually specify the poses of the objects only in the first frame of each video. Using Signed Distance Function (SDF) representations of each object, we refine the pose of each object in the first depth frame. Next, the camera trajectory is initialized by fixing the object poses relative to one another and tracking the object configuration through the depth video. Finally, the camera trajectory and relative object poses are refined in a global optimization step.

为避免手工标注所有视频帧，我们只在每段视频的第一帧中手工指定物体位姿。利用每个物体的符号距离函数（SDF）表示，我们在第一帧深度图中精修每个物体的位姿。接下来，通过把物体之间的相对位姿固定下来，并在深度视频中跟踪这一物体配置，来初始化相机轨迹。最后，在一步全局优化中同时精修相机轨迹和物体相对位姿。

IV-B Dataset Characteristics

IV-B 数据集特征

Number of Objects	21
Total Number of Videos	92
Held-out Videos	12
Min Object Count	3
Max Object Count	9
Mean Object Count	4.47
Number of Frames	133,827
Resolution	640 x 480

TABLE I: Statistics of our YCB-Video Dataset

物体数量	21
视频总数	92
留出视频数	12
最少物体数	3
最多物体数	9
平均物体数	4.47
帧数	133,827
分辨率	640 x 480

表 I：我们 YCB-Video 数据集的统计信息

The objects we used are a subset of 21 of the YCB objects [5] as shown in Fig. 5, selected due to high-quality 3D models and good visibility in depth. The videos are collected using an Asus Xtion Pro Live RGB-D camera in fast-cropping mode, which provides RGB images at a resolution of 640x480 at 30 FPS by capturing a 1280x960 image locally on the device and transmitting only the center region over USB. This results in higher effective resolution of RGB images at the cost of a lower FOV, but given the minimum range of the depth sensor this was an acceptable trade-off. The full dataset comprises 133,827 images, two full orders of magnitude larger than the LINEMOD dataset. For more statistics relating to the dataset, see Table I. Fig. 6 shows one annotation example in our dataset where we render the 3D models according to the annotated ground truth pose. Note that our annotation accuracy suffers from several sources of error, including the rolling shutter of the RGB sensor, inaccuracies in the object models, slight asynchrony between RGB and depth sensors, and uncertainties in the intrinsic and extrinsic parameters of the cameras.

我们使用的物体是如图 5 所示的 21 个 YCB 物体 [5] 的子集，选中它们是因为三维模型质量高、并且在深度上可见性较好。视频用 Asus Xtion Pro Live RGB-D 相机在快速裁切模式下采集，该模式以 30 FPS 提供 640x480 分辨率的 RGB 图像：设备在本地捕获 1280x960 图像，再只把中心区域通过 USB 传出。这样可以提高 RGB 图像的有效分辨率，代价是视场更小；但考虑到深度传感器的最小工作距离，这是可以接受的折中。完整数据集包含 133,827 张图像，比 LINEMOD 数据集大约大两个数量级。更多与数据集相关的统计见表 I。图 6 给出了我们数据集中的一个标注示例，其中我们按照标注的真实位姿渲染三维模型。需要说明的是，我们的标注精度会受到若干误差来源影响，包括 RGB 传感器的卷帘快门、物体模型的不准确、RGB 与深度传感器之间轻微的不同步，以及相机内外参的不确定性。

Fig. 6: Left: an example image from the dataset. Right: Textured 3D object models (provided with the YCB dataset) rendered according to the pose annotations for this frame.

图 6：左：数据集中的一张示例图像。右：按照该帧的位姿标注渲染的带纹理三维物体模型（由 YCB 数据集提供）。

V EXPERIMENTS

V 实验

V-A Datasets

V-A 数据集

In our YCB-Video dataset, we use 80 videos for training, and test on 2,949 key frames extracted from the rest 12 test videos. We also evaluate our method on the OccludedLINEMOD dataset [17]. The authors of [17] selected one video with 1,214 frames from the original LINEMOD dataset [13], and annotated ground truth poses for eight objects in that video: Ape, Can, Cat, Driller, Duck, Eggbox, Glue and Holepuncher. There are significant occlusions between objects in this video sequence, which makes this dataset challenging. For training, we use the eight sequences from the original LINEMOD dataset corresponding to these eight objects. In addition, we generate 80,000 synthetic images for training on both datasets by randomly placing objects in a scene.

在我们的 YCB-Video 数据集中，我们用 80 段视频训练，并在其余 12 段测试视频中抽出的 2,949 个关键帧上测试。我们也在 OccludedLINEMOD 数据集 [17] 上评估方法。[17] 的作者从原始 LINEMOD 数据集 [13] 中选出一段包含 1,214 帧的视频，并为该视频中的八个物体标注了真实位姿：Ape、Can、Cat、Driller、Duck、Eggbox、Glue 和 Holepuncher。这段视频序列中物体之间存在显著遮挡，因而使该数据集具有挑战性。训练时，我们使用原始 LINEMOD 数据集中与这八个物体对应的八段序列。此外，我们通过在场景中随机摆放物体，为两个数据集的训练各生成 80,000 张合成图像。

V-B Evaluation Metrics

V-B 评价指标

We adopt the average distance (ADD) metric as proposed in [13] for evaluation. Given the ground truth rotation $\mathbf{R}$ and translation $\mathbf{T}$ and the estimated rotation $\mathbf{\tilde{R}}$ and translation $\mathbf{\tilde{T}}$, the average distance computes the mean of the pairwise distances between the 3D model points transformed according to the ground truth pose and the estimated pose:

我们采用 [13] 提出的平均距离（ADD）指标做评估。给定真实旋转 $\mathbf{R}$ 和平移 $\mathbf{T}$，以及估计旋转 $\mathbf{\tilde{R}}$ 和平移 $\mathbf{\tilde{T}}$，平均距离计算按真实位姿变换后的三维模型点与按估计位姿变换后的对应点之间成对距离的均值：

$$\textsc{ADD}=\frac{1}{m}\sum_{\mathbf{x}\in\mathcal{M}}\|(\mathbf{R}\mathbf{x}+\mathbf{T})-(\mathbf{\tilde{R}}\mathbf{x}+\mathbf{\tilde{T}})\|, \tag{5}$$

where $\mathcal{M}$ denotes the set of 3D model points and $m$ is the number of points. The 6D pose is considered to be correct if the average distance is smaller than a predefined threshold. In the OccludedLINEMOD dataset, the threshold is set to 10% of the 3D model diameter. For symmetric objects such as the Eggbox and Glue, the matching between points is ambiguous for some views. Therefore, the average distance is computed using the closest point distance:

其中 $\mathcal{M}$ 表示三维模型点集，$m$ 是点数。如果平均距离小于预先设定的阈值，就认为六自由度位姿是正确的。在 OccludedLINEMOD 数据集中，该阈值设为三维模型直径的 10%。对 Eggbox 和 Glue 这类对称物体，某些视角下点与点之间的匹配是含糊的。因此，平均距离改用最近点距离计算：

$$\textsc{ADD-S}=\frac{1}{m}\sum_{\mathbf{x}_{1}\in\mathcal{M}}\min_{\mathbf{x}_{2}\in\mathcal{M}}\|(\mathbf{R}\mathbf{x}_{1}+\mathbf{T})-(\mathbf{\tilde{R}}\mathbf{x}_{2}+\mathbf{\tilde{T}})\|. \tag{6}$$

Our design of the loss function for rotation regression is motivated by these two evaluation metrics. Using a fixed threshold in computing pose accuracy cannot reveal how a method performs on these incorrect poses with respect to that threshold. Therefore, we vary the distance threshold in evaluation. In this case, we can plot an accuracy-threshold curve, and compute the area under the curve for pose evaluation.

我们为旋转回归设计损失函数，正是受到这两个评价指标的启发。用固定阈值计算位姿准确率，无法揭示方法在相对于该阈值不正确的那些位姿上表现如何。因此，我们在评估中变化距离阈值。这样就可以画出准确率–阈值曲线，并用曲线下面积来评价位姿。

Instead of computing distances in the 3D space, we can project the transformed points onto the image, and then compute the pairwise distances in the image space. This metric is called the reprojection error that is widely used for 6D pose estimation when only color images are used.

除了在三维空间中计算距离，我们也可以把变换后的点投影到图像上，再在图像空间中计算成对距离。这一指标称为重投影误差，在只使用彩色图像做六自由度位姿估计时被广泛采用。

V-C Implementation Details

V-C 实现细节

PoseCNN is implemented using the TensorFlow library [1]. The Hough voting layer is implemented on GPU as in [31]. In training, the parameters of the first 13 convolutional layers in the feature extraction stage and the first two FC layers in the 3D rotation regression branch are initialized with the VGG16 network [27] trained on ImageNet [9]. No gradient is back-propagated via the Hough voting layer. Stochastic Gradient Descent (SGD) with momentum is used for training.

PoseCNN 用 TensorFlow 库 [1] 实现。霍夫投票层按 [31] 在 GPU 上实现。训练时，特征提取阶段的前 13 个卷积层，以及三维旋转回归分支中前两个 FC 层的参数，用在 ImageNet [9] 上训练的 VGG16 网络 [27] 初始化。梯度不通过霍夫投票层回传。训练使用带动量的随机梯度下降（SGD）。

V-D Baselines

V-D 基线

3D object coordinate regression network. Since the state-of-the-art 6D pose estimation methods mostly rely on regressing image pixels to 3D object coordinates [3, 4, 21], we implement a variation of our network for 3D object coordinate regression for comparison. In this network, instead of regressing to center direction and depth as in Fig. 2, we regress each pixel to its 3D coordinate in the object coordinate system. We can use the same architecture since each pixel still regresses to three variables for each class. Then we remove the 3D rotation regression branch. Using the semantic labeling results and 3D object coordinate regression results, the 6D pose is recovered using the pre-emptive RANSAC as in [4].

三维物体坐标回归网络。 由于当时领先的六自由度位姿估计方法大多依赖把图像像素回归到三维物体坐标 [3, 4, 21]，我们实现了网络的一个变体用于三维物体坐标回归，以便比较。在该网络中，我们不再像图 2 那样回归中心方向和深度，而是把每个像素回归到它在物体坐标系中的三维坐标。由于每个像素仍然为每个类别回归三个变量，我们可以使用相同的架构。随后去掉三维旋转回归分支。利用语义标注结果和三维物体坐标回归结果，再按 [4] 用抢占式 RANSAC 恢复六自由度位姿。

Pose refinement. The 6D pose estimated from our network can be refined when depth is available. We use the Iterative Closest Point (ICP) algorithm to refine the 6D pose. Specifically, we employ ICP with projective data association and a point-plane residual term. We render a predicted point cloud given the 3D model and an estimated pose, and assume that each observed depth value is associated with the predicted depth value at the same pixel location. The residual for each pixel is then the smallest distance from the observed point in 3D to the plane defined by the rendered point in 3D and its normal. Points with residuals above a specified threshold are rejected and the remaining residuals are minimized using gradient descent. Semantic labels from the network are used to crop the observed points from the depth image. Since ICP is not robust to local minimums, we refinement multiple poses by perturbing the estimated pose from the network, and then select the best refined pose using the alignment metric proposed in [33].

位姿精修。 当深度可用时，可以从网络估计出的六自由度位姿出发做精修。我们用迭代最近点（ICP）算法精修六自由度位姿。具体来说，我们采用带投影数据关联和点–平面残差项的 ICP。给定三维模型和一个估计位姿，我们渲染出预测点云，并假定每个观测到的深度值与同一像素位置上的预测深度值相关联。每个像素的残差，就是观测点在三维中到由渲染点及其法向定义的平面的最短距离。残差超过指定阈值的点被剔除，其余残差用梯度下降最小化。网络给出的语义标签用于从深度图中裁出观测点。由于 ICP 对局部极小不稳健，我们通过对网络估计出的位姿做扰动来精修多个位姿，再用 [33] 提出的对齐度量选出最好的精修结果。

V-E Analysis on the Rotation Regress Losses

V-E 对旋转回归损失的分析

Fig. 7: Comparison between the PLoss and the SLoss for 6D pose estimation on three symmetric objects in the YCB-Video dataset.

图 7：在 YCB-Video 数据集的三个对称物体上，比较用于六自由度位姿估计的 PLoss 与 SLoss。

We first conduct experiments to analyze the effect of the two loss functions for rotation regression on symmetric objects. Fig. 7 shows the rotation error histograms for two symmetric objects in the YCB-Video dataset (wood block and large clamp) using the two loss functions in training. The rotation errors of the PLoss for the wood block and the large clamp span from 0 degree to 180 degree. The two histograms indicate that the network is confused by the symmetric objects. While the histograms of the SLoss concentrate on the 180 degree error for the wood block and 0 degree and 180 degree for the large clamp, since they are symmetric with respect to 180 degree rotation around their coordinate axes.

我们首先通过实验分析两种旋转回归损失函数在对称物体上的效果。图 7 给出了 YCB-Video 数据集中两个对称物体（木块和大号夹子）在分别用这两种损失函数训练时的旋转误差直方图。PLoss 下，木块和大号夹子的旋转误差分布在 0 度到 180 度之间。这两张直方图表明，网络被对称物体搞糊涂了。而 SLoss 的直方图则集中在木块的 180 度误差，以及大号夹子的 0 度和 180 度，因为它们相对于绕自身坐标轴旋转 180 度是对称的。

V-F Results on the YCB-Video Dataset

V-F YCB-Video 数据集上的结果

	RGB	RGB-D
	3D Coordinate	PoseCNN	3D Coordinate	3D Coordinate+ICP	PoseCNN+ICP
Object	ADD	ADD-S	ADD	ADD-S	ADD	ADD-S	ADD	ADD-S	ADD	ADD-S
002_master_chef_can	12.3	34.4	50.9	84.0	61.4	90.1	72.7	95.7	69.0	95.8
003_cracker_box	16.8	40.0	51.7	76.9	57.4	77.4	82.7	91.0	80.7	91.8
004_sugar_box	28.7	48.9	68.6	84.3	85.5	93.3	94.6	97.5	97.2	98.2
005_tomato_soup_can	27.3	42.2	66.0	80.9	84.5	92.1	86.1	94.5	81.6	94.5
006_mustard_bottle	25.9	44.8	79.9	90.2	82.8	91.1	97.6	98.3	97.0	98.4
007_tuna_fish_can	5.4	10.4	70.4	87.9	68.8	86.9	76.7	91.4	83.1	97.1
008_pudding_box	14.9	26.3	62.9	79.0	74.8	89.3	86.0	94.9	96.6	97.9
009_gelatin_box	25.4	36.7	75.2	87.1	93.9	97.2	98.2	98.8	98.2	98.8
010_potted_meat_can	18.7	32.3	59.6	78.5	70.9	84.0	78.9	87.8	83.8	92.8
011_banana	3.2	8.8	72.3	85.9	50.7	77.3	73.5	94.3	91.6	96.9
019_pitcher_base	27.3	54.3	52.5	76.8	58.2	83.8	81.1	95.6	96.7	97.8
021_bleach_cleanser	25.2	44.3	50.5	71.9	74.1	89.2	87.2	95.7	92.3	96.8
024_bowl	2.7	25.4	6.5	69.7	8.7	67.4	8.3	77.9	17.5	78.3
025_mug	9.0	20.0	57.7	78.0	57.1	85.3	67.0	91.1	81.4	95.1
035_power_drill	18.0	36.1	55.1	72.8	79.4	89.4	93.2	96.2	96.9	98.0
036_wood_block	1.2	19.6	31.8	65.8	14.6	76.7	21.7	85.2	79.2	90.5
037_scissors	1.0	2.9	35.8	56.2	61.0	82.8	66.0	88.3	78.4	92.2
040_large_marker	0.2	0.3	58.0	71.4	72.4	82.8	74.1	85.5	85.4	97.2
051_large_clamp	6.9	14.6	25.0	49.9	48.0	67.6	54.6	74.9	52.6	75.4
052_extra_large_clamp	2.7	14.0	15.8	47.0	22.1	49.0	25.2	56.4	28.7	65.3
061_foam_brick	0.6	1.2	40.4	87.8	40.0	82.4	46.5	89.9	48.3	97.1
ALL	15.1	29.8	53.7	75.9	64.6	83.7	74.5	90.1	79.3	93.0

TABLE II: Area under the accuracy-threshold curve for 6D pose evaluation on the YCB-Video dataset. Red colored objects are symmetric.

	RGB	RGB-D
	三维坐标	PoseCNN	三维坐标	三维坐标+ICP	PoseCNN+ICP
物体	ADD	ADD-S	ADD	ADD-S	ADD	ADD-S	ADD	ADD-S	ADD	ADD-S
002_master_chef_can	12.3	34.4	50.9	84.0	61.4	90.1	72.7	95.7	69.0	95.8
003_cracker_box	16.8	40.0	51.7	76.9	57.4	77.4	82.7	91.0	80.7	91.8
004_sugar_box	28.7	48.9	68.6	84.3	85.5	93.3	94.6	97.5	97.2	98.2
005_tomato_soup_can	27.3	42.2	66.0	80.9	84.5	92.1	86.1	94.5	81.6	94.5
006_mustard_bottle	25.9	44.8	79.9	90.2	82.8	91.1	97.6	98.3	97.0	98.4
007_tuna_fish_can	5.4	10.4	70.4	87.9	68.8	86.9	76.7	91.4	83.1	97.1
008_pudding_box	14.9	26.3	62.9	79.0	74.8	89.3	86.0	94.9	96.6	97.9
009_gelatin_box	25.4	36.7	75.2	87.1	93.9	97.2	98.2	98.8	98.2	98.8
010_potted_meat_can	18.7	32.3	59.6	78.5	70.9	84.0	78.9	87.8	83.8	92.8
011_banana	3.2	8.8	72.3	85.9	50.7	77.3	73.5	94.3	91.6	96.9
019_pitcher_base	27.3	54.3	52.5	76.8	58.2	83.8	81.1	95.6	96.7	97.8
021_bleach_cleanser	25.2	44.3	50.5	71.9	74.1	89.2	87.2	95.7	92.3	96.8
024_bowl	2.7	25.4	6.5	69.7	8.7	67.4	8.3	77.9	17.5	78.3
025_mug	9.0	20.0	57.7	78.0	57.1	85.3	67.0	91.1	81.4	95.1
035_power_drill	18.0	36.1	55.1	72.8	79.4	89.4	93.2	96.2	96.9	98.0
036_wood_block	1.2	19.6	31.8	65.8	14.6	76.7	21.7	85.2	79.2	90.5
037_scissors	1.0	2.9	35.8	56.2	61.0	82.8	66.0	88.3	78.4	92.2
040_large_marker	0.2	0.3	58.0	71.4	72.4	82.8	74.1	85.5	85.4	97.2
051_large_clamp	6.9	14.6	25.0	49.9	48.0	67.6	54.6	74.9	52.6	75.4
052_extra_large_clamp	2.7	14.0	15.8	47.0	22.1	49.0	25.2	56.4	28.7	65.3
061_foam_brick	0.6	1.2	40.4	87.8	40.0	82.4	46.5	89.9	48.3	97.1
ALL	15.1	29.8	53.7	75.9	64.6	83.7	74.5	90.1	79.3	93.0

表 II：YCB-Video 数据集上六自由度位姿评估的准确率–阈值曲线下面积。红色物体为对称物体。

Fig. 8: (a) Detailed results on the YCB-Video dataset. (b) Accuracy-threshold curves with reprojectin error on the OccludedLINEMOD dataset.

图 8：（a）YCB-Video 数据集上的详细结果。（b）OccludedLINEMOD 数据集上带重投影误差的准确率–阈值曲线。

Table II and Fig. 8(a) presents detailed evaluation for all the 21 objects in the YCB-Video dataset. We show the area under the accuracy-threshold curve using both the ADD metric and the ADD-S metric, where we vary the threshold for the average distance and then compute the pose accuracy. The maximum threshold is set to 10cm.

表 II 和图 8(a) 给出了 YCB-Video 数据集全部 21 个物体的详细评估。我们同时用 ADD 指标和 ADD-S 指标展示准确率–阈值曲线下面积，其中我们变化平均距离的阈值再计算位姿准确率。最大阈值设为 10 厘米。

We can see that i) By only using color images, our network significantly outperforms the 3D coordinate regression network combined with the pre-emptive RANSAC algorithm for 6D pose estimation. When there are errors in the 3D coordinate regression results, the estimated 6D pose can drift far away from the ground truth pose. While in our network, the center localization helps to constrain the 3D translation estimation even if the object is occluded. ii) Refining the poses with ICP significantly improves the performance. PoseCNN with ICP achieves superior performance compared to the 3D coordinate regression network when using depth images. The initial pose in ICP is critical for convergence. PoseCNN provides better initial 6D poses for ICP refinement. iii) We can see that some objects are more difficult to handle such as the tuna fish can that is small and with less texture. The network is also confused by the large clamp and the extra large clamp since they have the same appearance. The 3D coordinate regression network cannot handle symmetric objects very well such as the banana and the bowl.

可以看到：其一，仅使用彩色图像时，我们的网络在六自由度位姿估计上显著优于与抢占式 RANSAC 算法结合的三维坐标回归网络。当三维坐标回归结果存在误差时，估计出的六自由度位姿可能远离真实位姿。而在我们的网络中，即便物体被遮挡，中心定位也有助于约束三维平移估计。其二，用 ICP 精修位姿会显著提升表现。在使用深度图像时，PoseCNN 加 ICP 相对三维坐标回归网络取得了更好的结果。ICP 中的初始位姿对收敛至关重要，而 PoseCNN 能为 ICP 精修提供更好的初始六自由度位姿。其三，可以看到有些物体更难处理，例如体积小、纹理较少的金枪鱼罐头。网络也会被大号夹子和超大号夹子弄混，因为它们外观相同。三维坐标回归网络对香蕉和碗这类对称物体处理得也不太好。

Fig. 9 displays some 6D pose estimation results on the YCB-Video dataset. We can see that the center prediction is quite accurate even if the center is occluded by another object. Our network with color only is already able to provide good 6D pose estimation. With ICP refinement, the accuracy of the 6D pose is further improved.

图 9 展示了 YCB-Video 数据集上的一些六自由度位姿估计结果。可以看到，即使中心被另一物体遮挡，中心预测仍然相当准确。我们的网络仅用彩色图像就已经能够提供不错的六自由度位姿估计。经过 ICP 精修后，六自由度位姿的精度进一步提高。

Method	Michel et al. [21]	Hinterstoisser et al. [14]	Krull et al. [17]	Brachmann et al. [3]	Ours PoseCNN Color	Ours PoseCNN+ICP
Ape	80.7	81.4	68.0	53.1	9.6	76.2
Can	88.5	94.7	87.9	79.9	45.2	87.4
Cat	57.8	55.2	50.6	28.2	0.93	52.2
Driller	94.7	86.0	91.2	82.0	41.4	90.3
Duck	74.4	79.7	64.7	64.3	19.6	77.7
Eggbox	47.6	65.5	41.5	9.0	22.0	72.2
Glue	73.8	52.1	65.3	44.5	38.5	76.7
Holepuncher	96.3	95.5	92.9	91.6	22.1	91.4
MEAN	76.7	76.3	70.3	56.6	24.9	78.0

TABLE III: 6D pose estimation accuracy on the OccludedLINEMOD dataset. Red colored objects are symmetric. All methods use depth except for PoseCNN Color.

方法	Michel et al. [21]	Hinterstoisser et al. [14]	Krull et al. [17]	Brachmann et al. [3]	Ours PoseCNN Color	Ours PoseCNN+ICP
Ape	80.7	81.4	68.0	53.1	9.6	76.2
Can	88.5	94.7	87.9	79.9	45.2	87.4
Cat	57.8	55.2	50.6	28.2	0.93	52.2
Driller	94.7	86.0	91.2	82.0	41.4	90.3
Duck	74.4	79.7	64.7	64.3	19.6	77.7
Eggbox	47.6	65.5	41.5	9.0	22.0	72.2
Glue	73.8	52.1	65.3	44.5	38.5	76.7
Holepuncher	96.3	95.5	92.9	91.6	22.1	91.4
MEAN	76.7	76.3	70.3	56.6	24.9	78.0

表 III：OccludedLINEMOD 数据集上的六自由度位姿估计准确率。红色物体为对称物体。除 PoseCNN Color 外，所有方法都使用深度。

Fig. 9: Examples of 6D object pose estimation results on the YCB-Video dataset from PoseCNN.

图 9：PoseCNN 在 YCB-Video 数据集上的六自由度物体位姿估计结果示例。

V-G Results on the OccludedLINEMOD Dataset

V-G OccludedLINEMOD 数据集上的结果

The OccludedLINEMOD dataset is challenging due to significant occlusions between objects. We first conduct experiments using color images only. Fig. 8(b) shows the accuracy-threshold curves with reprojection error for 7 objects in the dataset, where we compare PoseCNN with [29] that achieves the current state-of-the-art result on this dataset using color images as input. Our method significantly outperforms [29] by a large margin, especially when the reprojection error threshold is small. These results show that PoseCNN is able to correctly localize the target object even under severe occlusions.

由于物体之间存在显著遮挡，OccludedLINEMOD 数据集颇具挑战。我们首先只使用彩色图像做实验。图 8(b) 给出了该数据集中 7 个物体带重投影误差的准确率–阈值曲线，其中我们把 PoseCNN 与 [29] 比较，后者是当时在该数据集上仅用彩色图像作为输入的领先结果。我们的方法以较大幅度显著优于 [29]，尤其是在重投影误差阈值较小时。这些结果表明，即便在严重遮挡下，PoseCNN 仍能正确找到目标物体的位置。

By refining the poses using depth images in ICP, our method also outperforms the state-of-the-art methods using RGB-D data as input. Table III summarizes the pose estimation accuracy on the OccludedLINEMOD dataset. The most improvement comes from the two symmetric objects “Eggbox” and “Glue”. By using our ShapeMatch-Loss for training, PoseCNN is able to correctly estimate the 6D pose of the two objects with respect to symmetry. We also present the result of PoseCNN using color only in Table III. These accuracies are much lower since the threshold here is usually smaller than 2cm. It is very challenging for color-based methods to obtain 6D poses within such small threshold when there are occlusions between objects. Fig. 9 shows two examples of the 6D pose estimation results on the OccludedLINEMOD dataset.

在用深度图像通过 ICP 精修位姿之后，我们的方法也优于以 RGB-D 数据作为输入的当时领先方法。表 III 汇总了 OccludedLINEMOD 数据集上的位姿估计准确率。提升最大的是两个对称物体 “Eggbox” 和 “Glue”。通过用 ShapeMatch-Loss 训练，PoseCNN 能够在考虑对称性的前提下正确估计这两个物体的六自由度位姿。我们在表 III 中也给出了仅使用彩色图像的 PoseCNN 结果。这些准确率要低得多，因为这里的阈值通常小于 2 厘米。当物体之间存在遮挡时，对基于彩色的方法来说，要在如此小的阈值内得到六自由度位姿非常困难。图 9 给出了 OccludedLINEMOD 数据集上六自由度位姿估计结果的两个例子。

VI CONCLUSIONS

VI 结论

In this work, we introduce PoseCNN, a convolutional neural network for 6D object pose estimation. PoseCNN decouples the estimation of 3D rotation and 3D translation. It estimates the 3D translation by localizing the object center and predicting the center distance. By regressing each pixel to a unit vector towards the object center, the center can be estimated robustly independent of scale. More importantly, pixels vote the object center even if it is occluded by other objects. The 3D rotation is predicted by regressing to a quaternion representation. Two new loss functions are introduced for rotation estimation, with the ShapeMatch-Loss designed for symmetric objects. As a result, PoseCNN is able to handle occlusion and symmetric objects in cluttered scenes. We also introduce a large scale video dataset for 6D object pose estimation. Our results are extremely encouraging in that they indicate that it is feasible to accurately estimate the 6D pose of objects in cluttered scenes using vision data only. This opens the path to using cameras with resolution and field of view that goes far beyond currently used depth camera systems. We note that the SLoss sometimes results in local minimums in the pose space similar to ICP. It would be interesting to explore more efficient way in handle symmetric objects in 6D pose estimation in the future.

本文介绍 PoseCNN，一种用于六自由度物体位姿估计的卷积神经网络。PoseCNN 把三维旋转和三维平移的估计解耦开来。它通过定位物体中心并预测中心距离来估计三维平移。通过把每个像素回归为指向物体中心的单位向量，中心可以稳健地、不依赖于尺度地被估计。更重要的是，即便中心被其他物体遮挡，像素仍会对物体中心投票。三维旋转通过对四元数表示做回归来预测。我们为旋转估计引入了两种新的损失函数，其中 ShapeMatch-Loss 专为对称物体设计。因此，PoseCNN 能够在杂乱场景中处理遮挡和对称物体。我们还引入了一个用于六自由度物体位姿估计的大规模视频数据集。我们的结果相当鼓舞人心：它们表明，仅用视觉数据就准确估计杂乱场景中物体的六自由度位姿是可行的。这为使用分辨率和视场远超当前深度相机系统的普通相机开辟了道路。我们注意到，SLoss 有时会像 ICP 一样在位姿空间中陷入局部极小。未来探索更有效的方式来处理六自由度位姿估计中的对称物体，会很有意思。

ACKNOWLEDGMENTS

致谢

This work was funded in part by Siemens and by NSF STTR grant 63-5197 with Lula Robotics.

本工作部分由 Siemens 资助，并由与 Lula Robotics 合作的 NSF STTR 基金 63-5197 资助。

References

参考文献

[1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. TensorFlow: A system for large-scale machine learning. In OSDI , volume 16, pages 265–283, 2016.

[2] Liefeng Bo, Xiaofeng Ren, and Dieter Fox. Learning hierarchical sparse features for RGB-D object recognition. International Journal of Robotics Research (IJRR) , 33(4):581–599, 2014.

[3] Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6D object pose estimation using 3D object coordinates. In European Conference on Computer Vision (ECCV) , pages 536–551, 2014.

[4] Eric Brachmann, Frank Michel, Alexander Krull, Michael Ying Yang, Stefan Gumhold, and Carsten Rother. Uncertainty-driven 6D pose estimation of objects and scenes from a single RGB image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3364–3372, 2016.

[5] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The YCB object and model set: Towards common benchmarks for manipulation research. In International Conference on Advanced Robotics (ICAR) , pages 510–517, 2015.

[6] Zhe Cao, Yaser Sheikh, and Natasha Kholgade Banerjee. Real-time scalable 6DOF pose estimation for textureless objects. In IEEE International Conference on Robotics and Automation (ICRA) , pages 2441–2448, 2016.

[7] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2D pose estimation using part affinity fields. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

[8] Alvaro Collet, Manuel Martinez, and Siddhartha S Srinivasa. The MOPED framework: Object recognition and pose estimation for manipulation. International Journal of Robotics Research (IJRR) , 30(10):1284–1306, 2011.

[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 248–255, 2009.

[10] Andreas Doumanoglou, Vassileios Balntas, Rigas Kouskouridas, and Tae-Kyun Kim. Siamese regression networks with efficient mid-level feature extraction for 3D object pose estimation. arXiv preprint arXiv:1607.02257 , 2016.

[11] Ross Girshick. Fast R-CNN. In IEEE International Conference on Computer Vision (ICCV) , pages 1440–1448, 2015.

[12] Stefan Hinterstoisser, Cedric Cagniart, Slobodan Ilic, Peter Sturm, Nassir Navab, Pascal Fua, and Vincent Lepetit. Gradient response maps for real-time detection of textureless objects. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 34(5):876–888, 2012a.

[13] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In Asian Conference on Computer Vision (ACCV) , pages 548–562, 2012b.

[14] Stefan Hinterstoisser, Vincent Lepetit, Naresh Rajkumar, and Kurt Konolige. Going further with point pair features. In European Conference on Computer Vision (ECCV) , pages 834–848, 2016.

[15] Wadim Kehl, Fausto Milletari, Federico Tombari, Slobodan Ilic, and Nassir Navab. Deep learning of local RGB-D patches for 3D object detection and 6D pose estimation. In European Conference on Computer Vision (ECCV) , pages 205–220, 2016.

[16] Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab. SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again. In IEEE International Conference on Computer Vision (ICCV) , pages 1521–1529, 2017.

[17] Alexander Krull, Eric Brachmann, Frank Michel, Michael Ying Yang, Stefan Gumhold, and Carsten Rother. Learning analysis-by-synthesis for 6D pose estimation in RGB-D images. In IEEE International Conference on Computer Vision (ICCV) , pages 954–962, 2015.

[18] Bastian Leibe, Ales Leonardis, and Bernt Schiele. Combined object categorization and segmentation with an implicit shape model. In ECCV Workshop on statistical learning in computer vision , 2004.

[19] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3431–3440, 2015.

[20] David G Lowe. Object recognition from local scale-invariant features. In IEEE International Conference on Computer Vision (ICCV) , volume 2, pages 1150–1157, 1999.

[21] Frank Michel, Alexander Kirillov, Erix Brachmann, Alexander Krull, Stefan Gumhold, Bogdan Savchynskyy, and Carsten Rother. Global hypothesis generation for 6D object pose estimation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

[22] Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G Derpanis, and Kostas Daniilidis. 6-DOF object pose from semantic keypoints. IEEE International Conference on Robotics and Automation (ICRA) , 2017.

[23] Mahdi Rad and Vincent Lepetit. BB8: A scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth. In IEEE International Conference on Computer Vision (ICCV) , 2017.

[24] Colin Rennie, Rahul Shome, Kostas E Bekris, and Alberto F De Souza. A dataset for improved RGBD-based object detection and pose estimation for warehouse pick-and-place. IEEE Robotics and Automation Letters , 1(2):1179–1185, 2016.

[25] Fred Rothganger, Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. 3D object modeling and recognition using local affine-invariant image descriptors and multi-view spatial constraints. International Journal of Computer Vision (IJCV) , 66(3):231–259, 2006.

[26] Max Schwarz, Hannes Schulz, and Sven Behnke. RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features. In IEEE International Conference on Robotics and Automation (ICRA) , pages 1329–1335, 2015.

[27] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

[28] Hao Su, Charles R Qi, Yangyan Li, and Leonidas J Guibas. Render for CNN: Viewpoint estimation in images using CNNs trained with rendered 3D model views. In IEEE International Conference on Computer Vision (ICCV) , pages 2686–2694, 2015.

[29] Bugra Tekin, Sudipta N Sinha, and Pascal Fua. Real-time seamless single shot 6D object pose prediction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

[30] Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1510–1519, 2015.

[31] Gert-Jan van den Braak, Cedric Nugteren, Bart Mesman, and Henk Corporaal. Fast Hough transform on GPUs: Exploration of algorithm trade-offs. In International Conference on Advanced Concepts for Intelligent Vision Systems , pages 611–622, 2011.

[32] Paul Wohlhart and Vincent Lepetit. Learning descriptors for object recognition and 3D pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3109–3118, 2015.

[33] Jay M Wong, Vincent Kee, Tiffany Le, Syler Wagner, Gian-Luca Mariottini, Abraham Schneider, Lei Hamilton, Rahul Chipalkatty, Mitchell Hebert, David Johnson, et al. SegICP: Integrated deep semantic segmentation and pose estimation. arXiv preprint arXiv:1703.01661 , 2017.

[34] Yu Xiang and Dieter Fox. DA-RNN: Semantic mapping with data associated recurrent neural networks. In Robotics: Science and Systems (RSS) . 2017.

[35] Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker, Alberto Rodriguez, and Jianxiong Xiao. Multi-view self-supervised deep learning for 6D pose estimation in the amazon picking challenge. In IEEE International Conference on Robotics and Automation (ICRA) , pages 1386–1383, 2017.