This paper has been accepted for publication in IEEE Transactions on Robotics.

本文已被 IEEE Transactions on Robotics 接收发表。

DOI: 10.1109/TRO.2017.2705103

IEEE Xplore: http://ieeexplore.ieee.org/document/7946260/

©2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

©2017 IEEE。允许出于个人目的使用本材料。除此以外的所有用途，包括在当前或未来任何媒介中为广告或促销目的重印／再发表本材料、创作新的汇编作品、转售或再分发到服务器或列表，或在其他作品中重用本作品的任何受版权保护的部分，均须获得 IEEE 许可。

ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras

ORB-SLAM2：面向单目、双目和 RGB-D 相机的开源 SLAM 系统

Raúl Mur-Artal    Juan D. Tardós † † thanks: This work was supported by the Spanish government under Project DPI2015-67275, the Aragón regional governmnet under Project DGA T04-FSE and the Ministerio de Educación Scholarship FPU13/04175. † † thanks: R. Mur-Artal was with the Instituto de Investigación en Ingeniería de Aragón (I3A), Universidad de Zaragoza, 50018 Zaragoza, Spain, until January 2017. He is currently with Oculus Research, Redmond, WA 98052 USA (e-mail: raul.murartal@oculus.com). † † thanks: J. D. Tardós is with the Instituto de Investigación en Ingeniería de Aragón (I3A), Universidad de Zaragoza, 50018 Zaragoza, Spain (e-mail: tardos@unizar.es).

Abstract

摘要

We present ORB-SLAM2 a complete SLAM system for monocular, stereo and RGB-D cameras, including map reuse, loop closing and relocalization capabilities. The system works in real-time on standard CPUs in a wide variety of environments from small hand-held indoors sequences, to drones flying in industrial environments and cars driving around a city. Our back-end based on bundle adjustment with monocular and stereo observations allows for accurate trajectory estimation with metric scale. Our system includes a lightweight localization mode that leverages visual odometry tracks for unmapped regions and matches to map points that allow for zero-drift localization. The evaluation on 29 popular public sequences shows that our method achieves state-of-the-art accuracy, being in most cases the most accurate SLAM solution. We publish the source code, not only for the benefit of the SLAM community, but with the aim of being an out-of-the-box SLAM solution for researchers in other fields.

我们提出 ORB-SLAM2，这是一套面向单目、双目和 RGB-D 相机的完整 SLAM 系统，具备地图复用、回环检测与重定位能力。系统可在普通 CPU 上实时运行，适用范围很广：从小范围手持室内序列，到工业环境中飞行的无人机，再到在城市中行驶的汽车。后端基于带有单目和双目观测的光束法平差（bundle adjustment），能够以米制尺度准确地估计轨迹。系统还包含一种轻量定位模式：在未建图区域借助视觉里程计跟踪，并与地图点匹配，从而实现零漂移定位。在 29 条常用公开序列上的评估表明，该方法达到当时最先进的精度，并且在多数情况下是最准确的 SLAM 方案。我们公布源代码，不仅是为了惠及 SLAM 社区，也希望成为其他领域研究者即拿即用的 SLAM 方案。

I Introduction

I 引言

Simultaneous Localization and Mapping (SLAM) has been a hot research topic in the last two decades in the Computer Vision and Robotics communities, and has recently attracted the attention of high-technological companies. SLAM techniques build a map of an unknown environment and localize the sensor in the map with a strong focus on real-time operation. Among the different sensor modalities, cameras are cheap and provide rich information of the environment that allows for robust and accurate place recognition. Therefore Visual SLAM solutions, where the main sensor is a camera, are of major interest nowadays. Place recognition is a key module of a SLAM system to close loops (i.e. detect when the sensor returns to a mapped area and correct the accumulated error in exploration) and to relocalize the camera after a tracking failure, due to occlusion or aggressive motion, or at system re-initialization.

同时定位与地图构建（SLAM）在过去二十年里一直是计算机视觉和机器人社区的热门研究课题，近来也引起了高科技公司的关注。SLAM 技术在未知环境中建立地图，并把传感器定位到该地图中，尤其强调实时运行。在各类传感器中，相机价格低廉，并能提供丰富的环境信息，从而支持稳健而准确的地点识别。因此，以相机为主要传感器的视觉 SLAM 方案在当下尤为重要。地点识别是 SLAM 系统中的关键模块，用于闭环（即检测传感器何时回到已建图区域，并纠正探索过程中累积的误差），也用于在遮挡或剧烈运动导致跟踪失败之后，或在系统重新初始化时，对相机进行重定位。

Visual SLAM can be performed by using just a monocular camera, which is the cheapest and smallest sensor setup. However as depth is not observable from just one camera, the scale of the map and estimated trajectory is unknown. In addition the system bootstrapping require multi-view or filtering techniques to produce an initial map as it cannot be triangulated from the very first frame. Last but not least, monocular SLAM suffers from scale drift and may fail if performing pure rotations in exploration. By using a stereo or an RGB-D camera all these issues are solved and allows for the most reliable Visual SLAM solutions.

视觉 SLAM 可以只用一台单目相机完成，这是最便宜、体积也最小的传感器配置。但由于仅凭一台相机无法观测深度，地图和估计轨迹的尺度是未知的。此外，系统启动需要多视图或滤波技术来生成初始地图，因为无法从最初的一帧就三角化。最后，单目 SLAM 会受尺度漂移之苦，若在探索中做纯旋转还可能失败。使用双目或 RGB-D 相机可以解决这些问题，从而得到更可靠的视觉 SLAM 方案。

(a) Stereo input: trajectory and sparse reconstruction of an urban environment with multiple loop closures.

(b) RGB-D input: keyframes and dense pointcloud of a room scene with one loop closure. The pointcloud is rendered by backprojecting the sensor depth maps from estimated keyframe poses. No fusion is performed.

Fig. 1: ORB-SLAM2 processes stereo and RGB-D inputs to estimate camera trajectory and build a map of the environment. The system is able to close loops, relocalize, and reuse its map in real-time on standard CPUs with high accuracy and robustness.

（a）双目输入：带有多次闭环的城市场景轨迹与稀疏重建。

（b）RGB-D 输入：带一次闭环的房间场景关键帧与稠密点云。点云由估计的关键帧位姿反投影传感器深度图渲染而成，未做融合。

图 1：ORB-SLAM2 处理双目和 RGB-D 输入，以估计相机轨迹并建立环境地图。系统能够在普通 CPU 上实时地闭环、重定位并复用地图，同时保持高准确性和稳健性。

In this paper we build on our monocular ORB-SLAM [1] and propose ORB-SLAM2 with the following contributions:

本文以我们的单目 ORB-SLAM [1] 为基础，提出 ORB-SLAM2，贡献如下：

• • The first open-source^1 SLAM system for monocular, stereo and RGB-D cameras, including loop closing, relocalization and map reuse.

• • 据我们所知，这是首个面向单目、双目和 RGB-D 相机的开源^1 SLAM 系统，包含闭环、重定位和地图复用。

• • Our RGB-D results show that by using Bundle Adjustment (BA) we achieve more accuracy than state-of-the-art methods based on ICP or photometric and depth error minimization.

• • 我们的 RGB-D 结果表明，使用光束法平差（BA）可以比基于 ICP 或光度与深度误差最小化的当时最先进方法更准确。

• • By using close and far stereo points and monocular observations our stereo results are more accurate than the state-of-the-art direct stereo SLAM.

• • 通过同时使用近处和远处双目点以及单目观测，我们的双目结果比当时最先进的直接法双目 SLAM 更准确。

• • A lightweight localization mode that can effectively reuse the map with mapping disabled.

• • 一种在关闭建图的情况下仍能有效复用地图的轻量定位模式。

Fig. 1 shows examples of ORB-SLAM2 output from stereo and RGB-D inputs. The stereo case shows the final trajectory and sparse reconstruction of the sequence 00 from the KITTI dataset [2]. This is an urban sequence with multiple loop closures that ORB-SLAM2 was able to successfully detect. The RGB-D case shows the keyframe poses estimated in sequence fr1_room from the TUM RGB-D Dataset [3], and a dense pointcloud, rendered by backprojecting sensor depth maps from the estimated keyframe poses. Note that our SLAM does not perform any fusion like KinectFusion [4] or similar, but the good definition indicates the accuracy of the keyframe poses. More examples are shown on the attached video.

图 1 给出了 ORB-SLAM2 在双目和 RGB-D 输入上的输出示例。双目情形展示的是 KITTI 数据集 [2] 序列 00 的最终轨迹与稀疏重建。这是一条带有多次闭环的城市序列，ORB-SLAM2 成功检测到了这些闭环。RGB-D 情形展示的是 TUM RGB-D 数据集 [3] 序列 fr1_room 中估计的关键帧位姿，以及由估计关键帧位姿反投影传感器深度图渲染得到的稠密点云。需要注意，我们的 SLAM 并不像 KinectFusion [4] 或类似方法那样做融合，但点云轮廓清晰，说明关键帧位姿是准确的。更多示例见附件视频。

In the rest of the paper, we discuss related work in Section II, we describe our system in Section III, then present the evaluation results in Section IV and end with conclusions in Section V.

后文安排如下：第 II 节讨论相关工作，第 III 节描述系统，第 IV 节给出评估结果，第 V 节作结。

II Related Work

II 相关工作

In this section we discuss related work on stereo and RGB-D SLAM. Our discussion, as well as the evaluation in Section IV is focused only on SLAM approaches.

本节讨论双目和 RGB-D SLAM 的相关工作。我们的讨论以及第 IV 节的评估只聚焦于 SLAM 方法。

II-A Stereo SLAM

II-A 双目 SLAM

A remarkable early stereo SLAM system was the work of Paz et al. [5]. Based on Conditionally Independent Divide and Conquer EKF-SLAM it was able to operate in larger environments than other approaches at that time. Most importantly, it was the first stereo SLAM exploiting both close and far points (i.e. points whose depth cannot be reliably estimated due to little disparity in the stereo camera), using an inverse depth parametrization [6] for the latter. They empirically showed that points can be reliably triangulated if their depth is less than ${\sim}40$ times the stereo baseline. In this work we follow this strategy of treating in a different way close and far points, as explained in Section III-A.

早期一套出色的双目 SLAM 系统是 Paz et al. [5] 的工作。它基于条件独立的分治 EKF-SLAM，能够在比当时其他方法更大的环境中运行。更重要的是，它是首个同时利用近处和远处点（即因双目视差过小而无法可靠估计深度的点）的双目 SLAM，对后者使用逆深度参数化 [6]。他们经验性地表明，若点的深度小于双目基线的约 ${\sim}40$ 倍，就可以可靠地三角化。本文沿用这一策略，对近处和远处点区别对待，见第 III-A 节。

Most modern stereo SLAM systems are keyframe-based [7] and perform BA optimization in a local area to achieve scalability. The work of Strasdat et al. [8] performs a joint optimization of BA (point-pose constraints) in an inner window of keyframes and pose-graph (pose-pose constraints) in an outer window. By limiting the size of these windows the method achieves constant time complexity, at the expense of not guaranteeing global consistency. The RSLAM of Mei et al. [9] uses a relative representation of landmarks and poses and performs relative BA in an active area which can be constrained for constant-time. RSLAM is able to close loops which allow to expand active areas at both sides of a loop, but global consistency is not enforced. The recent S-PTAM by Pire et al. [10] performs local BA, however it lacks large loop closing. Similar to these approaches we perform BA in a local set of keyframes so that the complexity is independent of the map size and we can operate in large environments. However our goal is to build a globally consistent map. When closing a loop, our system aligns first both sides, similar to RSLAM, so that the tracking is able to continue localizing using the old map and then performs a pose-graph optimization that minimizes the drift accumulated in the loop, followed by full BA.

多数现代双目 SLAM 系统是基于关键帧的 [7]，并在局部区域做 BA 优化以实现可扩展性。Strasdat et al. [8] 的工作在内窗口的关键帧上联合优化 BA（点–位姿约束），并在外窗口上优化位姿图（位姿–位姿约束）。通过限制这些窗口的大小，该方法达到常数时间复杂度，代价是不保证全局一致性。Mei et al. [9] 的 RSLAM 使用路标和位姿的相对表示，并在可被约束为常数时间的活动区域内做相对 BA。RSLAM 能够闭环，从而在闭环两侧扩展活动区域，但并不强制全局一致性。Pire et al. [10] 近期的 S-PTAM 做局部 BA，但缺少大尺度闭环。与这些方法类似，我们在一组局部关键帧上做 BA，使复杂度与地图规模无关，从而能在大环境中运行。不过我们的目标是建立全局一致的地图。闭环时，系统首先像 RSLAM 那样对齐两侧，使跟踪能够继续用旧地图定位，然后做位姿图优化以最小化闭环中累积的漂移，再接着做完整 BA。

The recent Stereo LSD-SLAM of Engel et al. [11] is a semi-dense direct approach that minimizes photometric error in image regions with high gradient. Not relying on features, the method is expected to be more robust to motion blur or poorly-textured environments. However as a direct method its performance can be severely degraded by unmodeled effects like rolling shutter or non-lambertian reflectance.

Engel et al. [11] 近期的 Stereo LSD-SLAM 是一种半稠密直接方法，在高梯度图像区域最小化光度误差。由于不依赖特征，该方法有望对运动模糊或弱纹理环境更稳健。但作为直接方法，其性能可能被卷帘快门或非朗伯反射等未建模效应严重降低。

(a) System Threads and Modules.

(b) Input pre-processing

Fig. 2: ORB-SLAM2 is composed of three main parallel threads: tracking, local mapping and loop closing, which can create a fourth thread to perform full BA after a loop closure. The tracking thread pre-processes the stereo or RGB-D input so that the rest of the system operates independently of the input sensor. Although it is not shown in this figure, ORB-SLAM2 also works with a monocular input as in [1].

（a）系统线程与模块。

（b）输入预处理

图 2：ORB-SLAM2 由三个主要并行线程组成：跟踪、局部建图和闭环；闭环后可以再创建第四个线程来做完整 BA。跟踪线程对双目或 RGB-D 输入做预处理，使系统其余部分的运行与输入传感器无关。尽管本图未画出，ORB-SLAM2 也可以像 [1] 那样使用单目输入。

II-B RGB-D SLAM

One of the earliest and most famed RGB-D SLAM systems was the KinectFusion of Newcombe et al. [4]. This method fused all depth data from the sensor into a volumetric dense model that is used to track the camera pose using ICP. This system was limited to small workspaces due to its volumetric representation and the lack of loop closing. Kintinuous by Whelan et al. [12] was able to operate in large environments by using a rolling cyclical buffer and included loop closing using place recognition and pose graph optimization.

最早也最广为人知的 RGB-D SLAM 系统之一是 Newcombe et al. [4] 的 KinectFusion。该方法把传感器的全部深度数据融合进体素稠密模型，并用 ICP 跟踪相机位姿。由于采用体表示且没有闭环，该系统限于较小的工作空间。Whelan et al. [12] 的 Kintinuous 使用滚动循环缓冲，从而能在大环境中运行，并借助地点识别和位姿图优化实现闭环。

Probably the first popular open-source system was the RGB-D SLAM of Endres et al. [13]. This is a feature-based system, whose front-end computes frame-to-frame motion by feature matching and ICP. The back-end performs pose-graph optimization with loop closure constraints from a heuristic search. Similarly the back-end of DVO-SLAM by Kerl et al. [14] optimizes a pose-graph where keyframe-to-keyframe constraints are computed from a visual odometry that minimizes both photometric and depth error. DVO-SLAM also searches for loop candidates in a heuristic fashion over all previous frames, instead of relying on place recognition.

大概最早流行的开源系统是 Endres et al. [13] 的 RGB-D SLAM。这是一套基于特征的系统，前端通过特征匹配和 ICP 计算帧间运动。后端用启发式搜索得到的闭环约束做位姿图优化。类似地，Kerl et al. [14] 的 DVO-SLAM 后端优化位姿图，其中关键帧到关键帧的约束来自同时最小化光度和深度误差的视觉里程计。DVO-SLAM 也在所有先前帧上以启发式方式搜索闭环候选，而不是依赖地点识别。

The recent ElasticFusion of Whelan et al. [15] builds a surfel-based map of the environment. This is a map-centric approach that forget poses and performs loop closing applying a non-rigid deformation to the map, instead of a standard pose-graph optimization. The detailed reconstruction and localization accuracy of this system is impressive, but the current implementation is limited to room-size maps as the complexity scales with the number of surfels in the map.

Whelan et al. [15] 近期的 ElasticFusion 建立基于 surfels 的环境地图。这是一种以地图为中心的方法，它忘掉位姿，并通过对地图做非刚性变形来闭环，而不是做标准的位姿图优化。该系统的精细重建和定位精度令人印象深刻，但当前实现受限于房间尺度的地图，因为复杂度随地图中 surfel 的数量增长。

As proposed by Strasdat et al. [8] our ORB-SLAM2 uses depth information to synthesize a stereo coordinate for extracted features on the image. This way our system is agnostic of the input being stereo or RGB-D. Differently to all above methods our back-end is based on bundle adjustment and builds a globally consistent sparse reconstruction. Therefore our method is lightweight and works with standard CPUs. Our goal is long-term and globally consistent localization instead of building the most detailed dense reconstruction. However from the highly accurate keyframe poses one could fuse depth maps and get accurate reconstruction on-the-fly in a local area or post-process the depth maps from all keyframes after a full BA and get an accurate 3D model of the whole scene.

如 Strasdat et al. [8] 所提出的，我们的 ORB-SLAM2 利用深度信息为图像上提取的特征合成一个双目坐标。这样，系统并不区分输入是双目还是 RGB-D。与上述所有方法不同，我们的后端基于光束法平差，并建立全局一致的稀疏重建。因此该方法很轻量，可在普通 CPU 上运行。我们的目标是长期、全局一致的定位，而不是构建最细致的稠密重建。不过，从高度准确的关键帧位姿出发，人们可以在局部区域在线融合深度图以得到准确重建，也可以在完整 BA 之后对所有关键帧的深度图做后处理，得到整个场景的准确三维模型。

III ORB-SLAM2

ORB-SLAM2 for stereo and RGB-D cameras is built on our monocular feature-based ORB-SLAM [1], whose main components are summarized here for reader convenience. A general overview of the system is shown in Fig. 2. The system has three main parallel threads: 1) the tracking to localize the camera with every frame by finding feature matches to the local map and minimizing the reprojection error applying motion-only BA, 2) the local mapping to manage the local map and optimize it, performing local BA, 3) the loop closing to detect large loops and correct the accumulated drift by performing a pose-graph optimization. This thread launches a fourth thread to perform full BA after the pose-graph optimization, to compute the optimal structure and motion solution.

面向双目和 RGB-D 相机的 ORB-SLAM2 建立在我们基于特征的单目 ORB-SLAM [1] 之上，这里为方便读者概括其主要组成部分。系统总览见图 2。系统有三个主要并行线程：1）跟踪，对每一帧通过寻找与局部地图的特征匹配，并用仅运动 BA 最小化重投影误差来定位相机；2）局部建图，管理局部地图并做局部 BA 加以优化；3）闭环，检测大回环并通过位姿图优化纠正累积漂移。该线程在位姿图优化之后再启动第四个线程做完整 BA，以计算结构和运动的最优解。

The system has embedded a Place Recognition module based on DBoW2 [16] for relocalization, in case of tracking failure (e.g. an occlusion) or for reinitialization in an already mapped scene, and for loop detection. The system maintains a covisibiliy graph [8] that links any two keyframes observing common points and a minimum spanning tree connecting all keyframes. These graph structures allow to retrieve local windows of keyframes, so that tracking and local mapping operate locally, allowing to work on large environments, and serve as structure for the pose-graph optimization performed when closing a loop.

系统内嵌基于 DBoW2 [16] 的地点识别模块，用于跟踪失败（例如遮挡）时的重定位，或在已建图场景中重新初始化，以及用于回环检测。系统维护一个共视（covisibility）图 [8]，把观测到共同点的任意两个关键帧连接起来，以及一棵连接所有关键帧的最小生成树。这些图结构使得可以取出关键帧的局部窗口，从而使跟踪和局部建图在局部运行，能够在大环境中工作，并作为闭环时位姿图优化的结构。

The system uses the same ORB features [17] for tracking, mapping and place recognition tasks. These features are robust to rotation and scale and present a good invariance to camera auto-gain and auto-exposure, and illumination changes. Moreover they are fast to extract and match allowing for real-time operation and show good precision/recall performance in bag-of-word place recognition [18].

系统在跟踪、建图和地点识别任务中使用同一套 ORB 特征 [17]。这些特征对旋转和尺度稳健，对相机自动增益、自动曝光以及光照变化也有较好的不变性。此外它们提取和匹配都很快，能够支持实时运行，并在词袋地点识别中表现出良好的精度／召回 [18]。

In the rest of this section we present how stereo/depth information is exploited and which elements of the system are affected. For a detailed description of each system block, we refer the reader to our monocular publication [1].

本节其余部分介绍如何利用双目／深度信息，以及系统中哪些部分受到影响。各系统模块的详细描述，请读者参阅我们的单目论文 [1]。

III-A Monocular, Close Stereo and Far Stereo Keypoints

III-A 单目、近处双目与远处双目关键点

ORB-SLAM2 as a feature-based method pre-processes the input to extract features at salient keypoint locations, as shown in Fig. 2b. The input images are then discarded and all system operations are based on these features, so that the system is independent of the sensor being stereo or RGB-D. Our system handles monocular and stereo keypoints, which are further classified as close or far.

作为基于特征的方法，ORB-SLAM2 先对输入做预处理，在显著关键点位置提取特征，如图 2b 所示。随后丢弃输入图像，系统的全部操作都基于这些特征，从而使系统与传感器是双目还是 RGB-D 无关。系统处理单目和双目关键点，后者还进一步分为近处和远处。

Stereo keypoints are defined by three coordinates $\mathbf{x}_{\mathrm{s}}=\left(u_{L},v_{L},u_{R}\right)$, being $(u_{L},v_{L})$ the coordinates on the left image and $u_{R}$ the horizontal coordinate in the right image. For stereo cameras, we extract ORB in both images and for every left ORB we search for a match in the right image. This can be done very efficiently assuming stereo rectified images, so that epipolar lines are horizontal. We then generate the stereo keypoint with the coordinates of the left ORB and the horizontal coordinate of the right match, which is subpixel refined by patch correlation. For RGB-D cameras, we extract ORB features on the RGB image and, as proposed by Strasdat et al. [8], for each feature with coordinates $\left(u_{L},v_{L}\right)$ we transform its depth value $d$ into a virtual right coordinate:

双目关键点由三个坐标 $\mathbf{x}_{\mathrm{s}}=\left(u_{L},v_{L},u_{R}\right)$ 定义，其中 $(u_{L},v_{L})$ 是左图坐标，$u_{R}$ 是右图中的水平坐标。对双目相机，我们在两幅图像上都提取 ORB，并对每个左图 ORB 在右图中搜索匹配。若图像已经过立体校正，极线水平，这一步可以做得很高效。然后用左图 ORB 的坐标和右图匹配的水平坐标生成双目关键点，后者通过块相关做亚像素精化。对 RGB-D 相机，我们在 RGB 图像上提取 ORB 特征，并如 Strasdat et al. [8] 所提出的，对每个坐标为 $\left(u_{L},v_{L}\right)$ 的特征，把它的深度值 $d$ 变换成虚拟右坐标：

$$u_{R}=u_{L}-\frac{f_{x}b}{d} \tag{1}$$

where $f_{x}$ is the horizontal focal length and $b$ is the baseline between the structured light projector and the infrared camera, which we approximate to 8cm for Kinect and Asus Xtion. The uncertainty of the depth sensor is represented by the uncertainty of the virtual right coordinate. In this way, features from stereo and RGB-D input are handled equally by the rest of the system.

其中 $f_{x}$ 是水平焦距，$b$ 是结构光投射器与红外相机之间的基线，对 Kinect 和 Asus Xtion 我们近似取为 8 厘米。深度传感器的不确定性由虚拟右坐标的不确定性来表示。这样，来自双目和 RGB-D 输入的特征便由系统其余部分以相同方式处理。

A stereo keypoint is classified as close if its associated depth is less than 40 times the stereo/RGB-D baseline, as suggested in [5], otherwise it is classified as far. Close keypoints can be safely triangulated from one frame as depth is accurately estimated and provide scale, translation and rotation information. On the other hand far points provide accurate rotation information but weaker scale and translation information. We triangulate far points when they are supported by multiple views.

若双目关键点对应的深度小于双目／RGB-D 基线的 40 倍，则将其分类为近处点，如 [5] 所建议；否则分类为远处点。近处关键点的深度估计准确，可以从单帧可靠地三角化，并提供尺度、平移和旋转信息。另一方面，远处点能提供准确的旋转信息，但尺度和平移信息较弱。当远处点得到多个视图支持时，我们再对它们做三角化。

Monocular keypoints are defined by two coordinates $\mathbf{x}_{\mathrm{m}}=\left(u_{L},v_{L}\right)$ on the left image and correspond to all those ORB for which a stereo match could not be found or that have an invalid depth value in the RGB-D case. These points are only triangulated from multiple views and do not provide scale information, but contribute to the rotation and translation estimation.

单目关键点由左图上的两个坐标 $\mathbf{x}_{\mathrm{m}}=\left(u_{L},v_{L}\right)$ 定义，对应于所有未能找到双目匹配的 ORB，或在 RGB-D 情形下深度值无效的那些点。这些点只从多个视图三角化，不提供尺度信息，但对旋转和平移估计有贡献。

III-B System Bootstrapping

III-B 系统启动

One of the main benefits of using stereo or RGB-D cameras is that, by having depth information from just one frame, we do not need a specific structure from motion initialization as in the monocular case. At system startup we create a keyframe with the first frame, set its pose to the origin, and create an initial map from all stereo keypoints.

使用双目或 RGB-D 相机的一个主要好处是：仅从一帧就能得到深度信息，因而不再需要像单目情形那样做专门的运动恢复结构初始化。系统启动时，我们用第一帧创建一个关键帧，把它的位姿设为原点，并由所有双目关键点创建初始地图。

III-C Bundle Adjustment with Monocular and Stereo Constraints

III-C 带单目和双目约束的光束法平差

Our system performs BA to optimize the camera pose in the tracking thread (motion-only BA), to optimize a local window of keyframes and points in the local mapping thread (local BA), and after a loop closure to optimize all keyframes and points (full BA). We use the Levenberg–Marquardt method implemented in g2o [19].

我们的系统用 BA 在跟踪线程中优化相机位姿（仅运动 BA），在局部建图线程中优化局部窗口内的关键帧和点（局部 BA），并在闭环之后优化所有关键帧和点（完整 BA）。我们使用 g2o [19] 中实现的 Levenberg–Marquardt 方法。

Motion-only BA optimizes the camera orientation $\mathbf{R}\in SO(3)$ and position $\mathbf{t}\in\mathbb{R}^{3}$, minimizing the reprojection error between matched 3D points $\mathbf{X}^{i}\in\mathbb{R}^{3}$ in world coordinates and keypoints $\mathbf{x}^{i}_{\mathrm{(\cdot)}}$, either monocular $\mathbf{x}^{i}_{\mathrm{m}}\in\mathbb{R}^{2}$ or stereo $\mathbf{x}^{i}_{\mathrm{s}}\in\mathbb{R}^{3}$, with $i\in\mathcal{X}$ the set of all matches:

仅运动 BA 优化相机方位 $\mathbf{R}\in SO(3)$ 和位置 $\mathbf{t}\in\mathbb{R}^{3}$，最小化已匹配的世界坐标三维点 $\mathbf{X}^{i}\in\mathbb{R}^{3}$ 与关键点 $\mathbf{x}^{i}_{\mathrm{(\cdot)}}$ 之间的重投影误差；关键点可以是单目 $\mathbf{x}^{i}_{\mathrm{m}}\in\mathbb{R}^{2}$ 或双目 $\mathbf{x}^{i}_{\mathrm{s}}\in\mathbb{R}^{3}$，其中 $i\in\mathcal{X}$ 是全部匹配的集合：

$$\{\mathbf{R},\mathbf{t}\}=\argmin_{\mathbf{R},\mathbf{t}}\sum_{i\in\mathcal{X}}\rho\left(\left\|\mathbf{x}^{i}_{\mathrm{(\cdot)}}-\pi_{\mathrm{(\cdot)}}\left(\mathbf{R}\mathbf{X}^{i}+\mathbf{t}\right)\right\|^{2}_{\Sigma}\right) \tag{2}$$

where $\rho$ is the robust Huber cost function and $\Sigma$ the covariance matrix associated to the scale of the keypoint. The projection functions $\pi_{(\cdot)}$, monocular $\pi_{\mathrm{m}}$ and rectified stereo $\pi_{\mathrm{s}}$, are defined as follows:

其中 $\rho$ 是稳健的 Huber 代价函数，$\Sigma$ 是与关键点尺度相关的协方差矩阵。投影函数 $\pi_{(\cdot)}$，即单目 $\pi_{\mathrm{m}}$ 和已校正双目 $\pi_{\mathrm{s}}$，定义如下：

$$\pi_{\mathrm{m}}\left(\begin{bmatrix}X\
Y\
Z\end{bmatrix}\right)=\begin{bmatrix}f_{x}\frac{X}{Z}+c_{x}\[1.99997pt]
f_{y}\frac{Y}{Z}+c_{y}\end{bmatrix},\pi_{\mathrm{s}}\left(\begin{bmatrix}X\
Y\
Z\end{bmatrix}\right)=\begin{bmatrix}f_{x}\frac{X}{Z}+c_{x}\[1.99997pt]
f_{y}\frac{Y}{Z}+c_{y}\[1.99997pt]
f_{x}\frac{X-b}{Z}+c_{x}\end{bmatrix} \tag{3}$$

where $(f_{x},f_{y})$ is the focal length, $(c_{x},c_{y})$ is the principal point and $b$ the baseline, all known from calibration.

其中 $(f_{x},f_{y})$ 是焦距，$(c_{x},c_{y})$ 是主点，$b$ 是基线，均由标定已知。

Local BA optimizes a set of covisible keyframes $\mathcal{K}_{L}$ and all points seen in those keyframes $\mathcal{P}_{L}$. All other keyframes $\mathcal{K}_{F}$, not in $\mathcal{K}_{L}$, observing points in $\mathcal{P}_{L}$ contribute to the cost function but remain fixed in the optimization. Defining $\mathcal{X}_{k}$ as the set of matches between points in $\mathcal{P}_{L}$ and keypoints in a keyframe $k$, the optimization problem is the following:

局部 BA 优化一组共视关键帧 $\mathcal{K}_{L}$ 以及在这些关键帧中看到的所有点 $\mathcal{P}_{L}$。所有不在 $\mathcal{K}_{L}$ 中、但观测到 $\mathcal{P}_{L}$ 中的点的其他关键帧 $\mathcal{K}_{F}$，对代价函数有贡献，但在优化中保持固定。记 $\mathcal{X}_{k}$ 为 $\mathcal{P}_{L}$ 中的点与关键帧 $k$ 中关键点之间的匹配集合，优化问题如下：

$$\begin{gathered}{\mathbf{X}^{i},\mathbf{R}{l},\mathbf{t}{l}|i\in\mathcal{P}{L},l\in\mathcal{K}{L}}=\argmin_{\mathbf{X}^{i},\mathbf{R}{l},\mathbf{t}{l}}\sum_{k\in\mathcal{K}{L}\cup\mathcal{K}{F}}\sum_{j\in\mathcal{X}{k}}\rho\left(E{kj}\right)\
E_{kj}=\left|\mathbf{x}^{j}{\mathrm{(\cdot)}}-\pi{\mathrm{(\cdot)}}\left(\mathbf{R}{k}\mathbf{X}^{j}+\mathbf{t}{k}\right)\right|^{2}_{\Sigma}\end{gathered} \tag{4}$$

Full BA is the specific case of local BA, where all keyframes and points in the map are optimized, except the origin keyframe that is fixed to eliminate the gauge freedom.

完整 BA 是局部 BA 的特例，此时地图中的所有关键帧和点都被优化，只有原点关键帧保持固定，以消除规范自由度。

III-D Loop Closing and Full BA

III-D 闭环与完整 BA

Loop closing is performed in two steps, firstly a loop has to be detected and validated, and secondly the loop is corrected optimizing a pose-graph. In contrast to monocular ORB-SLAM, where scale drift may occur [20], the stereo/depth information makes scale observable and the geometric validation and pose-graph optimization no longer require dealing with scale drift and are based on rigid body transformations instead of similarities.

闭环分两步进行：首先必须检测并验证回环，然后通过优化位姿图来纠正回环。与可能出现尺度漂移的单目 ORB-SLAM [20] 不同，双目／深度信息使尺度可观测，几何验证和位姿图优化不再需要处理尺度漂移，而是基于刚体变换而不是相似变换。

In ORB-SLAM2 we have incorporated a full BA optimization after the pose-graph to achieve the optimal solution. This optimization might be very costly and therefore we perform it in a separate thread, allowing the system to continue creating map and detecting loops. However this brings the challenge of merging the bundle adjustment output with the current state of the map. If a new loop is detected while the optimization is running, we abort the optimization and proceed to close the loop, which will launch the full BA optimization again. When the full BA finishes, we need to merge the updated subset of keyframes and points optimized by the full BA, with the non-updated keyframes and points that where inserted while the optimization was running. This is done by propagating the correction of updated keyframes (i.e. the transformation from the non-optimized to the optimized pose) to non-updated keyframes through the spanning tree. Non-updated points are transformed according to the correction applied to their reference keyframe.

在 ORB-SLAM2 中，我们在位姿图优化之后加入了完整 BA，以得到最优解。这一优化可能代价很高，因此放在单独线程中执行，使系统可以继续建图和检测回环。不过这也带来了如何把光束法平差输出与地图当前状态合并的挑战。若在优化运行期间检测到新的回环，我们就中止优化并继续闭环，后者会再次启动完整 BA。完整 BA 结束后，需要把完整 BA 优化过的那部分已更新关键帧和点，与优化运行期间新插入的、尚未更新的关键帧和点合并。做法是把已更新关键帧的纠正（即从未优化位姿到已优化位姿的变换）沿生成树传播到未更新的关键帧。未更新的点则按其参考关键帧所施加的纠正来变换。

III-E Keyframe Insertion

III-E 关键帧插入

ORB-SLAM2 follows the policy introduced in monocular ORB-SLAM of inserting keyframes very often and culling redundant ones afterwards. The distinction between close and far stereo points allows us to introduce a new condition for keyframe insertion, which can be critical in challenging environments where a big part of the scene is far from the stereo sensor, as shown in Fig. 3. In such environment we need to have a sufficient amount of close points to accurately estimate translation, therefore if the number of tracked close points drops below $\tau_{t}$ and the frame could create at least $\tau_{c}$ new close stereo points, the system will insert a new keyframe. We empirically found that $\tau_{t}=100$ and $\tau_{c}=70$ works well in all our experiments.

ORB-SLAM2 沿用单目 ORB-SLAM 中的策略：非常频繁地插入关键帧，再事后剔除冗余的关键帧。近处与远处双目点的区分让我们可以引入一条新的关键帧插入条件，这在场景很大一部分远离双目传感器的困难环境中可能至关重要，如图 3 所示。在这类环境中，需要有足够数量的近处点才能准确估计平移，因此若被跟踪的近处点数量降到 $\tau_{t}$ 以下，且该帧至少能够创建 $\tau_{c}$ 个新的近处双目点，系统就会插入一个新关键帧。我们经验性地发现，$\tau_{t}=100$ 和 $\tau_{c}=70$ 在全部实验中都工作良好。

Fig. 3: Tracked points in KITTI 01 [2]. Green points have a depth less than 40 times the stereo baseline, while blue points are further away. In this kind of sequences it is important to insert keyframes often enough so that the amount of close points allows for accurate translation estimation. Far points contribute to estimate orientation but provide weak information for translation and scale.

图 3：KITTI 01 [2] 中被跟踪的点。绿色点的深度小于双目基线的 40 倍，蓝色点更远。在这类序列中，需要足够频繁地插入关键帧，使近处点的数量足以准确估计平移。远处点有助于估计方位，但对平移和尺度只提供较弱的信息。

III-F Localization Mode

III-F 定位模式

We incorporate a Localization Mode which can be useful for lightweight long-term localization in well mapped areas, as long as there are not significant changes in the environment. In this mode the local mapping and loop closing threads are deactivated and the camera is continuously localized by the tracking using relocalization if needed. In this mode the tracking leverages visual odometry matches and matches to map points. Visual odometry matches are matches between ORB in the current frame and 3D points created in the previous frame from the stereo/depth information. These matches make the localization robust to unmapped regions, but drift can be accumulated. Map point matches ensure drift-free localization to the existing map. This mode is demonstrated in the accompanying video.

我们加入了一种定位模式，在已经建好图且环境没有显著变化的区域中，可用于轻量的长期定位。该模式下局部建图和闭环线程被关闭，相机由跟踪持续定位，必要时进行重定位。此时跟踪同时利用视觉里程计匹配以及与地图点的匹配。视觉里程计匹配是当前帧中的 ORB 与上一帧根据双目／深度信息创建的三维点之间的匹配。这些匹配使定位对未建图区域保持稳健，但可能累积漂移。与地图点的匹配则保证相对于已有地图的无漂移定位。该模式在配套视频中做了演示。

IV Evaluation

IV 评估

We have evaluated ORB-SLAM2 in three popular datasets and compared to other state-of-the-art SLAM systems, using always the results published by the original authors and standard evaluation metrics in the literature. We have run ORB-SLAM2 in an Intel Core i7-4790 desktop computer with 16Gb RAM. In order to account for the non-deterministic nature of the multi-threading system, we run each sequence 5 times and show median results for the accuracy of the estimated trajectory. Our open-source implementation includes the calibration and instructions to run the system in all these datasets.

我们在三个常用数据集上评估了 ORB-SLAM2，并与其他当时最先进的 SLAM 系统比较，始终采用原作者公布的结果以及文献中的标准评估指标。ORB-SLAM2 运行在一台配备 16GB 内存的 Intel Core i7-4790 台式计算机上。为计入多线程系统的非确定性，每条序列运行 5 次，并给出估计轨迹精度的中位数结果。我们的开源实现包含在这些数据集上运行系统所需的标定和说明。

IV-A KITTI Dataset

IV-A KITTI 数据集

The KITTI dataset [2] contains stereo sequences recorded from a car in urban and highway environments. The stereo sensor has a ${\sim}54\textrm{cm}$ baseline and works at 10Hz with a resolution after rectification of $1240\times 376$ pixels. Sequences 00, 02, 05, 06, 07 and 09 contain loops. Our ORB-SLAM2 detects all loops and is able to reuse its map afterwards, except for sequence 09 where the loop happens in very few frames at the end of the sequence. Table I shows results in the 11 training sequences, which have public ground-truth, compared to the state-of-the-art Stereo LSD-SLAM [11], to our knowledge the only stereo SLAM showing detailed results for all sequences. We use two different metrics, the absolute translation RMSE $t_{abs}$ proposed in [3], and the average relative translation $t_{rel}$ and rotation $r_{rel}$ errors proposed in [2]. Our system outperforms Stereo LSD-SLAM in most sequences, and achieves in general a relative error lower than 1%. The sequence 01, see Fig. 3, is the only highway sequence in the training set and the translation error is slightly worse. Translation is harder to estimate in this sequence because very few close points can be tracked, due to high speed and low frame-rate. However orientation can be accurately estimated, achieving an error of 0.21 degrees per 100 meters, as there are many far point that can be long tracked. Fig. 4 shows some examples of estimated trajectories.

KITTI 数据集 [2] 包含在城市和高速公路环境中从汽车上录制的双目序列。双目传感器基线约 ${\sim}54\textrm{cm}$，以 10Hz 工作，校正后分辨率为 $1240\times 376$ 像素。序列 00、02、05、06、07 和 09 包含回环。我们的 ORB-SLAM2 检测到了全部回环，并能够随后复用地图，只有序列 09 例外：该回环发生在序列末尾、只覆盖极少几帧。表 I 给出在 11 条有公开真值的训练序列上的结果，并与据我们所知唯一对全部序列给出详细结果的当时最先进双目 SLAM，即 Stereo LSD-SLAM [11] 进行比较。我们使用两种不同指标：[3] 提出的绝对平移 RMSE $t_{abs}$，以及 [2] 提出的平均相对平移 $t_{rel}$ 和旋转 $r_{rel}$ 误差。我们的系统在多数序列上优于 Stereo LSD-SLAM，相对误差总体上低于 1%。序列 01（见图 3）是训练集中唯一的高速公路序列，平移误差略差。这条序列上平移更难估计，因为车速高、帧率低，能够跟踪到的近处点很少。不过方位可以准确估计，达到每 100 米 0.21 度的误差，因为有许多远处点可以被长时间跟踪。图 4 给出了一些估计轨迹的例子。

Compared to the monocular results presented in [1], the proposed stereo version is able to process the sequence 01 where the monocular system failed. In this highway sequence, see Fig. 3, close points are in view only for a few frames. The ability of the stereo version to create points from just one stereo keyframe instead of the delayed initialization of the monocular, consisting on finding matches between two keyframes, is critical in this sequence not to lose tracking. Moreover the stereo system estimates the map and trajectory with metric scale and does not suffer from scale drift, as seen in Fig. 5.

与 [1] 中给出的单目结果相比，所提出的双目版本能够处理单目系统失败的序列 01。在这条高速公路序列中（见图 3），近处点只在少数几帧内可见。双目版本能够仅从一帧双目关键帧创建点，而不像单目那样需要在两帧关键帧之间寻找匹配、因而初始化被推迟，这一能力在该序列中对避免跟丢至关重要。此外，双目系统以米制尺度估计地图和轨迹，并且不受尺度漂移影响，如图 5 所示。

Sequence	ORB-SLAM2 (stereo) $t_{rel}$ (%)	ORB-SLAM2 (stereo) $r_{rel}$ (deg/100m)	ORB-SLAM2 (stereo) $t_{abs}$ (m)	Stereo LSD-SLAM $t_{rel}$ (%)	Stereo LSD-SLAM $r_{rel}$ (deg/100m)	Stereo LSD-SLAM $t_{abs}$ (m)
00	0.70	0.25	1.3	0.63	0.26	1.0
01	1.39	0.21	10.4	2.36	0.36	9.0
02	0.76	0.23	5.7	0.79	0.23	2.6
03	0.71	0.18	0.6	1.01	0.28	1.2
04	0.48	0.13	0.2	0.38	0.31	0.2
05	0.40	0.16	0.8	0.64	0.18	1.5
06	0.51	0.15	0.8	0.71	0.18	1.3
07	0.50	0.28	0.5	0.56	0.29	0.5
08	1.05	0.32	3.6	1.11	0.31	3.9
09	0.87	0.27	3.2	1.14	0.25	5.6
10	0.60	0.27	1.0	0.72	0.33	1.5

TABLE I: Comparison of accuracy in the KITTI Dataset.

序列	ORB-SLAM2（双目）$t_{rel}$（%）	ORB-SLAM2（双目）$r_{rel}$（度/100m）	ORB-SLAM2（双目）$t_{abs}$（m）	Stereo LSD-SLAM $t_{rel}$（%）	Stereo LSD-SLAM $r_{rel}$（度/100m）	Stereo LSD-SLAM $t_{abs}$（m）
00	0.70	0.25	1.3	0.63	0.26	1.0
01	1.39	0.21	10.4	2.36	0.36	9.0
02	0.76	0.23	5.7	0.79	0.23	2.6
03	0.71	0.18	0.6	1.01	0.28	1.2
04	0.48	0.13	0.2	0.38	0.31	0.2
05	0.40	0.16	0.8	0.64	0.18	1.5
06	0.51	0.15	0.8	0.71	0.18	1.3
07	0.50	0.28	0.5	0.56	0.29	0.5
08	1.05	0.32	3.6	1.11	0.31	3.9
09	0.87	0.27	3.2	1.14	0.25	5.6
10	0.60	0.27	1.0	0.72	0.33	1.5

表 I：KITTI 数据集上的精度比较。

Fig. 4: Estimated trajectory (black) and ground-truth (red) in KITTI 00, 01, 05 and 07.

图 4：KITTI 00、01、05 和 07 中的估计轨迹（黑）与真值（红）。

Fig. 5: Estimated trajectory (black) and ground-truth (red) in KITTI 08. Left: monocular ORB-SLAM [1], right: ORB-SLAM2 (stereo). Monocular ORB-SLAM suffers from severe scale drift in this sequence, especially at the turns. In contrast the proposed stereo version is able to estimate the true scale of the trajectory and map without scale drift.

图 5：KITTI 08 中的估计轨迹（黑）与真值（红）。左图为单目 ORB-SLAM [1]，右图为 ORB-SLAM2（双目）。单目 ORB-SLAM 在该序列中出现严重的尺度漂移，尤其是在转弯处。相比之下，所提出的双目版本能够估计轨迹和地图的真实尺度，并且没有尺度漂移。

IV-B EuRoC Dataset

IV-B EuRoC 数据集

The recent EuRoC dataset [21] contains 11 stereo sequences recorded from a micro aerial vehicle (MAV) flying around two different rooms and a large industrial environment. The stereo sensor has a ${{\sim}11}\textrm{cm}$ baseline and provides WVGA images at 20Hz. The sequences are classified as easy , medium and difficult depending on MAV’s speed, illumination and scene texture. In all sequences the MAV revisits the environment and ORB-SLAM2 is able to reuse its map, closing loops when necessary. Table II shows absolute translation RMSE of ORB-SLAM2 for all sequences, comparing to Stereo LSD-SLAM, for the results provided in [11]. ORB-SLAM2 achieves a localization precision of a few centimeters and is more accurate than Stereo LSD-SLAM. Our tracking get lost in some parts of V2_03_difficult due to severe motion blur. As shown in [22], this sequence can be processed using IMU information. Fig. 6 shows examples of computed trajectories compared to the ground-truth.

近期的 EuRoC 数据集 [21] 包含 11 条双目序列，由微型飞行器（MAV）在两间不同的房间以及一处大型工业环境中飞行录制。双目传感器基线约 ${{\sim}11}\textrm{cm}$，以 20Hz 提供 WVGA 图像。序列按 MAV 速度、光照和场景纹理分为 easy、medium 和 difficult。在所有序列中，MAV 都会再次访问环境，ORB-SLAM2 能够复用地图，并在必要时闭环。表 II 给出 ORB-SLAM2 在全部序列上的绝对平移 RMSE，并与 [11] 中给出的 Stereo LSD-SLAM 结果比较。ORB-SLAM2 达到数厘米的定位精度，并且比 Stereo LSD-SLAM 更准确。由于严重的运动模糊，我们的跟踪在 V2_03_difficult 的某些部分会跟丢。如 [22] 所示，该序列可以利用 IMU 信息来处理。图 6 给出了若干计算轨迹与真值比较的例子。

IV-C TUM RGB-D Dataset

IV-C TUM RGB-D 数据集

The TUM RGB-D dataset [3] contains indoors sequences from RGB-D sensors grouped in several categories to evaluate object reconstruction and SLAM/odometry methods under different texture, illumination and structure conditions. We show results in a subset of sequences where most RGB-D methods are usually evaluated. In Table III we compare our accuracy to the following state-of-the-art methods: ElasticFusion [15], Kintinuous [12], DVO-SLAM [14] and RGB-D SLAM [13]. Our method is the only one based on bundle adjustment and outperforms the other approaches in most sequences. As we already noticed for RGB-D SLAM results in [1], depthmaps for freiburg2 sequences have a 4$\%$ scale bias, probably coming from miscalibration, that we have compensated in our runs and could partly explain our significantly better results. Fig. 7 shows the point clouds that result from backprojecting the sensor depth maps from the computed keyframe poses in four sequences. The good definition and the straight contours of desks and posters prove the high accuracy localization of our approach.

TUM RGB-D 数据集 [3] 包含来自 RGB-D 传感器的室内序列，按若干类别分组，用于在不同纹理、光照和结构条件下评估物体重建以及 SLAM／里程计方法。我们在多数 RGB-D 方法通常评估的一个子集上给出结果。表 III 中，我们把精度与下列当时最先进的方法比较：ElasticFusion [15]、Kintinuous [12]、DVO-SLAM [14] 和 RGB-D SLAM [13]。我们的方法是其中唯一基于光束法平差的，并在多数序列上优于其他方法。正如我们在 [1] 的 RGB-D SLAM 结果中已经注意到的，freiburg2 序列的深度图有 4% 的尺度偏差，很可能来自标定不准；我们在运行中对此做了补偿，这也可以部分解释为何我们的结果明显更好。图 7 展示了在四条序列中，由计算得到的关键帧位姿反投影传感器深度图所得到的点云。办公桌和海报的轮廓清晰、边缘平直，证明了我们方法的高定位精度。

Sequence	ORB-SLAM2 (stereo)	Stereo LSD-SLAM
V1_01_easy	0.035	0.066
V1_02_medium	0.020	0.074
V1_03_difficult	0.048	0.089
V2_01_easy	0.037	-
V2_02_medium	0.035	-
V2_03_difficult	X	-
MH_01_easy	0.035	-
MH_02_easy	0.018	-
MH_03_medium	0.028	-
MH_04_difficult	0.119	-
MH_05_difficult	0.060	-

TABLE II: EuRoC Dataset. Comparison of Translation RMSE (${m}$).

序列	ORB-SLAM2（双目）	Stereo LSD-SLAM
V1_01_easy	0.035	0.066
V1_02_medium	0.020	0.074
V1_03_difficult	0.048	0.089
V2_01_easy	0.037	-
V2_02_medium	0.035	-
V2_03_difficult	X	-
MH_01_easy	0.035	-
MH_02_easy	0.018	-
MH_03_medium	0.028	-
MH_04_difficult	0.119	-
MH_05_difficult	0.060	-

表 II：EuRoC 数据集。平移 RMSE（${m}$）比较。

Fig. 6: Estimated trajectory (black) and groundtruth (red) in EuRoC V1_02_medium, V2_02_medium, MH_03_medium and MH_05_difficutlt.

图 6：EuRoC 中 V1_02_medium、V2_02_medium、MH_03_medium 和 MH_05_difficutlt 的估计轨迹（黑）与真值（红）。

IV-D Timing Results

IV-D 计时结果

In order to complete the evaluation of the proposed system, we present in Table IV timing results in three sequences with different image resolutions and sensors. The mean and two standard deviation ranges are shown for each thread task. As these sequences contain one single loop, the full BA and some tasks of the loop closing thread are executed just once and only a single time measurement is reported.The average tracking time per frame is below the inverse of the camera frame-rate for each sequence, meaning that our system is able to work in real-time. As ORB extraction in stereo images is parallelized, it can be seen that extracting 1000 ORB features in the stereo WVGA images of V2_02 is similar to extracting the same amount of features in the single VGA image channel of fr3_office.

为完整评估所提出的系统，我们在表 IV 中给出三条具有不同图像分辨率和传感器的序列上的计时结果。对每个线程任务给出均值以及正负两个标准差的范围。由于这些序列只包含一次回环，完整 BA 以及闭环线程中的部分任务只执行一次，因而只报告单次时间测量。每帧的平均跟踪时间低于各序列相机帧率的倒数，意味着我们的系统能够实时工作。由于双目图像上的 ORB 提取是并行的，可以看到：在 V2_02 的双目 WVGA 图像上提取 1000 个 ORB 特征，耗时与在 fr3_office 的单路 VGA 图像上提取同样数量的特征相近。

The number of keyframes in the loop is shown as reference for the times related to loop closing. While the loop in KITTI 07 contains more keyframes, the covisibility graph built for the indoor fr3_office is denser and therefore the loop fusion, pose-graph optimization and full BA tasks are more expensive. The higher density of the covisibility graph makes the local map contain more keyframes and points and therefore local map tracking and local BA are also more expensive.

回环中的关键帧数量作为与闭环相关时间的参照给出。尽管 KITTI 07 的回环包含更多关键帧，室内 fr3_office 所建立的共视图更密，因此回环融合、位姿图优化和完整 BA 这些任务代价更高。共视图密度更高，也使局部地图包含更多关键帧和点，因而局部地图跟踪和局部 BA 同样更昂贵。

Sequence	ORB-SLAM2 (RGB-D)	ElasticFusion	Kintinuous	DVO-SLAM	RGBD SLAM
fr1/desk	0.016	0.020	0.037	0.021	0.026
fr1/desk2	0.022	0.048	0.071	0.046	-
fr1/room	0.047	0.068	0.075	0.043	0.087
fr2/desk	0.009	0.071	0.034	0.017	0.057
fr2/xyz	0.004	0.011	0.029	0.018	-
fr3/office	0.010	0.017	0.030	0.035	-
fr3/nst	0.019	0.016	0.031	0.018	-

TABLE III: TUM RGB-D Dataset. Comparison of Translation RMSE (${m}$).

序列	ORB-SLAM2（RGB-D）	ElasticFusion	Kintinuous	DVO-SLAM	RGBD SLAM
fr1/desk	0.016	0.020	0.037	0.021	0.026
fr1/desk2	0.022	0.048	0.071	0.046	-
fr1/room	0.047	0.068	0.075	0.043	0.087
fr2/desk	0.009	0.071	0.034	0.017	0.057
fr2/xyz	0.004	0.011	0.029	0.018	-
fr3/office	0.010	0.017	0.030	0.035	-
fr3/nst	0.019	0.016	0.031	0.018	-

表 III：TUM RGB-D 数据集。平移 RMSE（${m}$）比较。

Fig. 7: Dense pointcloud reconstructions from estimated keyframe poses and sensor depth maps in TUM RGB-D fr3_office, fr1_room, fr2_desk and fr3_nst.

图 7：由估计的关键帧位姿和传感器深度图得到的稠密点云重建，对应 TUM RGB-D 的 fr3_office、fr1_room、fr2_desk 和 fr3_nst。

		V2_02	07	fr3_office
Settings	Dataset	EuRoC	KITTI	TUM
Settings	Sensor	Stereo	Stereo	RGB-D
Settings	Resolution	$752\times 480$	$1226\times 370$	$640\times 480$
Settings	Camera FPS	20Hz	10Hz	30Hz
Settings	ORB Features	1000	2000	1000
Tracking	Stereo Rectification	$3.43\pm 1.10$	-	-
Tracking	ORB Extraction	$13.54\pm 4.60$	$24.83\pm 8.28$	$11.48\pm 1.84$
Tracking	Stereo Matching	$11.26\pm 6.64$	$15.51\pm 4.12$	$0.02\pm 0.00$
Tracking	Pose Prediction	$2.07\pm 1.58$	$2.36\pm 1.84$	$2.65\pm 1.28$
Tracking	Local Map Tracking	$10.13\pm 11.40$	$5.38\pm 3.52$	$9.78\pm 6.42$
Tracking	New Keyframe Decision	$1.40\pm 1.14$	$1.91\pm 1.06$	$1.58\pm 0.92$
Tracking	Total	$41.66\pm 18.90$	$49.47\pm 12.10$	$25.58\pm 9.76$
Mapping	Keyframe Insertion	$10.30\pm 7.50$	$11.61\pm 3.28$	$11.36\pm 5.04$
Mapping	Map Point Culling	$0.28\pm 0.20$	$0.45\pm 0.38$	$0.25\pm 0.10$
Mapping	Map Point Creation	$40.43\pm 36.10$	$47.69\pm 29.52$	$53.99\pm 23.62$
Mapping	Local BA	$137.99\pm 248.18$	$69.29\pm 61.88$	$196.67\pm 213.42$
Mapping	Keyframe Culling	$3.80\pm 8.20$	$0.99\pm 0.92$	$6.69\pm 8.24$
Mapping	Total	$174.10\pm 278.80$	$129.52\pm 88.52$	$267.33\pm 245.10$
Loop	Database Query	$3.57\pm 5.86$	$4.13\pm 3.54$	$2.63\pm 2.26$
Loop	SE3 Estimation	$0.69\pm 1.82$	$1.02\pm 3.68$	$0.66\pm 1.68$
Loop	Loop Fusion	$21.84$	$82.70$	$298.45$
Loop	Essential Graph Opt.	$73.15$	$178.31$	$281.99$
Loop	Total	$108.59$	$284.88$	$598.70$
BA	Full BA	$349.25$	$1144.06$	$1640.96$
BA	Map Update	$3.13$	$11.82$	$5.62$
BA	Total	$396.02$	$1205.78$	$1793.02$
	Loop size (#keyframes)	82	248	225

TABLE IV: Timing Results of Each Thread in Miliseconds (mean $\pm$ 2 std. deviations).

		V2_02	07	fr3_office
设置	数据集	EuRoC	KITTI	TUM
设置	传感器	双目	双目	RGB-D
设置	分辨率	$752\times 480$	$1226\times 370$	$640\times 480$
设置	相机帧率	20Hz	10Hz	30Hz
设置	ORB 特征数	1000	2000	1000
跟踪	立体校正	$3.43\pm 1.10$	-	-
跟踪	ORB 提取	$13.54\pm 4.60$	$24.83\pm 8.28$	$11.48\pm 1.84$
跟踪	双目匹配	$11.26\pm 6.64$	$15.51\pm 4.12$	$0.02\pm 0.00$
跟踪	位姿预测	$2.07\pm 1.58$	$2.36\pm 1.84$	$2.65\pm 1.28$
跟踪	局部地图跟踪	$10.13\pm 11.40$	$5.38\pm 3.52$	$9.78\pm 6.42$
跟踪	新关键帧判定	$1.40\pm 1.14$	$1.91\pm 1.06$	$1.58\pm 0.92$
跟踪	合计	$41.66\pm 18.90$	$49.47\pm 12.10$	$25.58\pm 9.76$
建图	关键帧插入	$10.30\pm 7.50$	$11.61\pm 3.28$	$11.36\pm 5.04$
建图	地图点剔除	$0.28\pm 0.20$	$0.45\pm 0.38$	$0.25\pm 0.10$
建图	地图点创建	$40.43\pm 36.10$	$47.69\pm 29.52$	$53.99\pm 23.62$
建图	局部 BA	$137.99\pm 248.18$	$69.29\pm 61.88$	$196.67\pm 213.42$
建图	关键帧剔除	$3.80\pm 8.20$	$0.99\pm 0.92$	$6.69\pm 8.24$
建图	合计	$174.10\pm 278.80$	$129.52\pm 88.52$	$267.33\pm 245.10$
闭环	数据库查询	$3.57\pm 5.86$	$4.13\pm 3.54$	$2.63\pm 2.26$
闭环	SE3 估计	$0.69\pm 1.82$	$1.02\pm 3.68$	$0.66\pm 1.68$
闭环	回环融合	$21.84$	$82.70$	$298.45$
闭环	本质图优化	$73.15$	$178.31$	$281.99$
闭环	合计	$108.59$	$284.88$	$598.70$
BA	完整 BA	$349.25$	$1144.06$	$1640.96$
BA	地图更新	$3.13$	$11.82$	$5.62$
BA	合计	$396.02$	$1205.78$	$1793.02$
	回环规模（关键帧数）	82	248	225

表 IV：各线程计时结果，单位为毫秒（均值 $\pm$ 2 倍标准差）。

V Conclusion

V 结论

We have presented a full SLAM system for monocular, stereo and RGB-D sensors, able to perform relocalization, loop closing and reuse its map in real-time on standard CPUs. We focus on building globally consistent maps for reliable and long-term localization in a wide range of environments as demonstrated in the experiments. The proposed localization mode with the relocalization capability of the system yields a very robust, zero-drift, and ligthweight localization method for known environments. This mode can be useful for certain applications, such as tracking the user viewpoint in virtual reality in a well-mapped space.

我们提出了一套面向单目、双目和 RGB-D 传感器的完整 SLAM 系统，能够在普通 CPU 上实时地重定位、闭环并复用地图。我们的重点是建立全局一致的地图，以在广泛环境中实现可靠的长期定位，实验对此做了验证。所提出的定位模式，加上系统的重定位能力，为已知环境提供了一种非常稳健、零漂移且轻量的定位方法。该模式对某些应用可能有用，例如在已建好地图的空间中跟踪用户视点以用于虚拟现实。

The comparison to the state-of-the-art shows that ORB-SLAM2 achieves in most cases the highest accuracy. In the KITTI visual odometry benchmark ORB-SLAM2 is currently the best stereo SLAM solution. Crucially, compared with the stereo visual odometry methods that have flourished in recent years, ORB-SLAM2 achieves zero-drift localization in already mapped areas.

与当时最先进方法的比较表明，ORB-SLAM2 在多数情况下达到最高精度。在 KITTI 视觉里程计基准上，ORB-SLAM2 当时是最好的双目 SLAM 方案。尤为关键的是，与近年来蓬勃发展的双目视觉里程计方法相比，ORB-SLAM2 能在已建图区域实现零漂移定位。

Surprisingly our RGB-D results demonstrate that if the most accurate camera localization is desired, bundle adjustment performs better than direct methods or ICP, with the additional advantage of being less computationally expensive, not requiring GPU processing to operate in real-time.

出乎意料的是，我们的 RGB-D 结果表明：如果目标是尽可能准确的相机定位，光束法平差优于直接法或 ICP，而且还更节省计算，不必依靠 GPU 就能实时运行。

We have released the source code of our system, with examples and instructions so that it can be easily used by other researchers. ORB-SLAM2 is to the best of our knowledge the first open-source visual SLAM system that can work either with monocular, stereo and RGB-D inputs. Moreover our source code contains an example of an augmented reality application^2 using a monocular camera to show the potential of our solution.

我们已经发布了系统的源代码，并附有示例和说明，以便其他研究者能够方便地使用。据我们所知，ORB-SLAM2 是首个既能用于单目、也能用于双目和 RGB-D 输入的开源视觉 SLAM 系统。此外，源代码中还包含一个用单目相机实现的增强现实应用示例^2，以展示该方案的潜力。

Future extensions might include, to name some examples, non-overlapping multi-camera, fisheye or omnidirectional cameras support, large scale dense fusion, cooperative mapping or increased motion blur robustness.

未来的扩展可能包括（仅举数例）：非重叠多相机、鱼眼或全向相机支持、大规模稠密融合、协同建图，或更强的运动模糊鲁棒性。

References

参考文献

[1] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós, “ORB-SLAM: a versatile and accurate monocular SLAM system,” IEEE Trans. Robot. , vol. 31, no. 5, pp. 1147–1163, 2015.

[2] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The KITTI dataset,” Int. J. Robot. Res. , vol. 32, no. 11, pp. 1231–1237, 2013.

[3] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in IEEE/RSJ Int. Conf. Intell. Robots and Syst. (IROS) , 2012, pp. 573–580.

[4] R. A. Newcombe, A. J. Davison, S. Izadi, P. Kohli, O. Hilliges, J. Shotton, D. Molyneaux, S. Hodges, D. Kim, and A. Fitzgibbon, “KinectFusion: Real-time dense surface mapping and tracking,” in IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR) , 2011.

[5] L. M. Paz, P. Piniés, J. D. Tardós, and J. Neira, “Large-scale 6-DOF SLAM with stereo-in-hand,” IEEE Trans. Robot. , vol. 24, no. 5, pp. 946–957, 2008.

[6] J. Civera, A. J. Davison, and J. M. M. Montiel, “Inverse depth parametrization for monocular SLAM,” IEEE Trans. Robot. , vol. 24, no. 5, pp. 932–945, 2008.

[7] H. Strasdat, J. M. M. Montiel, and A. J. Davison, “Visual SLAM: Why filter?” Image and Vision Computing , vol. 30, no. 2, pp. 65–77, 2012.

[8] H. Strasdat, A. J. Davison, J. M. M. Montiel, and K. Konolige, “Double window optimisation for constant time visual SLAM,” in IEEE Int. Conf. Comput. Vision (ICCV) , 2011, pp. 2352–2359.

[9] C. Mei, G. Sibley, M. Cummins, P. Newman, and I. Reid, “RSLAM: A system for large-scale mapping in constant-time using stereo,” Int. J. Comput. Vision , vol. 94, no. 2, pp. 198–214, 2011.

[10] T. Pire, T. Fischer, J. Civera, P. De Cristóforis, and J. J. Berlles, “Stereo parallel tracking and mapping for robot localization,” in IEEE/RSJ Int. Conf. Intell. Robots and Syst. (IROS) , 2015, pp. 1373–1378.

[11] J. Engel, J. Stueckler, and D. Cremers, “Large-scale direct SLAM with stereo cameras,” in IEEE/RSJ Int. Conf. Intell. Robots and Syst. (IROS) , 2015.

[12] T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense RGB-D SLAM with volumetric fusion,” Int. J. Robot. Res. , vol. 34, no. 4-5, pp. 598–626, 2015.

[13] F. Endres, J. Hess, J. Sturm, D. Cremers, and W. Burgard, “3-D mapping with an RGB-D camera,” IEEE Trans. Robot. , vol. 30, no. 1, pp. 177–187, 2014.

[14] C. Kerl, J. Sturm, and D. Cremers, “Dense visual SLAM for RGB-D cameras,” in IEEE/RSJ Int. Conf. Intell. Robots and Syst. (IROS) , 2013.

[15] T. Whelan, R. F. Salas-Moreno, B. Glocker, A. J. Davison, and S. Leutenegger, “ElasticFusion: Real-time dense SLAM and light source estimation,” Int. J. Robot. Res. , vol. 35, no. 14, pp. 1697–1716, 2016.

[16] D. Gálvez-López and J. D. Tardós, “Bags of binary words for fast place recognition in image sequences,” IEEE Trans. Robot. , vol. 28, no. 5, pp. 1188–1197, 2012.

[17] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “ORB: an efficient alternative to SIFT or SURF,” in IEEE Int. Conf. Comput. Vision (ICCV) , 2011, pp. 2564–2571.

[18] R. Mur-Artal and J. D. Tardós, “Fast relocalisation and loop closing in keyframe-based SLAM,” in IEEE Int. Conf. on Robot. and Autom. (ICRA) , 2014, pp. 846–853.

[19] R. Kuemmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “g2o: A general framework for graph optimization,” in IEEE Int. Conf. on Robot. and Autom. (ICRA) , 2011, pp. 3607–3613.

[20] H. Strasdat, J. M. M. Montiel, and A. J. Davison, “Scale drift-aware large scale monocular SLAM.” in Robot.: Sci. and Syst. (RSS) , 2010.

[21] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Siegwart, “The EuRoC micro aerial vehicle datasets,” Int. J. Robot. Res. , vol. 35, no. 10, pp. 1157–1163, 2016.

[22] R. Mur-Artal and J. D. Tardós, “Visual-inertial monocular SLAM with map reuse,” IEEE Robot. and Autom. Lett. , vol. 2, no. 2, pp. 796 – 803, 2017.