Deep High-Resolution Representation Learning for Human Pose Estimation | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表
摘要1 引言2 相关工作3 方法4 实验4.1 COCO 关键点检测4.2 MPII 人体姿态估计4.3 应用于姿态跟踪4.4 消融实验5 结论与未来工作附录MPII 验证集上的结果PoseTrack 数据集上的更多结果ImageNet 验证集上的结果参考文献
返回论文列表
目录

Deep High-Resolution Representation Learning for Human Pose Estimation

收录于 CVPR 2019 会议论文；arXiv:1902.09212被引 5,551 次
打开论文来源

Deep High-Resolution Representation Learning for Human Pose Estimation

用于人体姿态估计的深度高分辨率表示学习

Ke Sun     Bin Xiao 1 1 footnotemark: 1     Dong Liu     Jingdong Wang † † thanks: Equal contribution. † † thanks: This work is done when Ke Sun was an intern at Microsoft Research, Beijing, P.R. China Affiliation:  University of Science and Technology of China    Affiliation:  Microsoft Research Asia {sunk,dongeliu}@ustc.edu.cn, {Bin.Xiao,jingdw}@microsoft.com

Abstract

摘要

In this paper, we are interested in the human pose estimation problem with a focus on learning reliable high-resolution representations. Most existing methods recover high-resolution representations from low-resolution representations produced by a high-to-low resolution network. Instead, our proposed network maintains high-resolution representations through the whole process.

本文关注人体姿态估计，重点是学习可靠的高分辨率表示。现有方法大多从高低分辨率串联网络得到的低分辨率表示中恢复高分辨率表示。相反，我们提出的网络在整个过程中始终保持高分辨率表示。

We start from a high-resolution subnetwork as the first stage, gradually add high-to-low resolution subnetworks one by one to form more stages, and connect the mutli-resolution subnetworks in parallel. We conduct repeated multi-scale fusions such that each of the high-to-low resolution representations receives information from other parallel representations over and over, leading to rich high-resolution representations. As a result, the predicted keypoint heatmap is potentially more accurate and spatially more precise. We empirically demonstrate the effectiveness of our network through the superior pose estimation results over two benchmark datasets: the COCO keypoint detection dataset and the MPII Human Pose dataset. In addition, we show the superiority of our network in pose tracking on the PoseTrack dataset. The code and models have been publicly available at https://github.com/leoxiaobin/deep-high-resolution-net.pytorch.

我们以一个高分辨率子网络作为第一阶段，再逐步加入由高到低的分辨率子网络以构成更多阶段，并把这些多分辨率子网络并行连接。我们进行反复的多尺度融合，使每一路由高到低的表示都能不断从其他并行表示中接收信息，从而得到内容更丰富的高分辨率表示。由此，预测出的关键点热图有望更准确、空间定位也更精细。我们在两个基准数据集上用更优的姿态估计结果验证了该网络：COCO 关键点检测数据集与 MPII Human Pose 数据集。此外，我们还在 PoseTrack 数据集上展示了该网络在姿态跟踪中的优势。代码与模型已公开：https://github.com/leoxiaobin/deep-high-resolution-net.pytorch。

1 Introduction

1 引言

222D human pose estimation has been a fundamental yet challenging problem in computer vision. The goal is to localize human anatomical keypoints (e.g., elbow, wrist, etc.) or parts. It has many applications, including human action recognition, human-computer interaction, animation, etc. This paper is interested in single-person pose estimation, which is the basis of other related problems, such as multi-person pose estimation [6, 27, 33, 39, 47, 57, 41, 46, 17, 71], video pose estimation and tracking [49, 72], etc.

二维人体姿态估计是计算机视觉中一个基础而困难的问题，目标是定位人体解剖关键点（如肘、腕等）或部位。它有许多应用，包括人体动作识别、人机交互、动画等。本文关注单人姿态估计，它也是多人姿态估计 [6, 27, 33, 39, 47, 57, 41, 46, 17, 71]、视频姿态估计与跟踪 [49, 72] 等相关问题的基础。

The recent developments show that deep convolutional neural networks have achieved the state-of-the-art performance. Most existing methods pass the input through a network, typically consisting of high-to-low resolution subnetworks that are connected in series, and then raise the resolution. For instance, Hourglass [40] recovers the high resolution through a symmetric low-to-high process. SimpleBaseline [72] adopts a few transposed convolution layers for generating high-resolution representations. In addition, dilated convolutions are also used to blow up the later layers of a high-to-low resolution network (e.g., VGGNet or ResNet) [27, 77].

近期进展表明，深度卷积神经网络已取得当时最优水平。现有方法大多让输入经过一个通常由高低分辨率子网络串联而成的网络，然后再提升分辨率。例如，Hourglass [40] 通过对称的由低到高过程恢复高分辨率；SimpleBaseline [72] 用若干转置卷积层生成高分辨率表示。此外，也有工作用空洞卷积放大高低分辨率网络（如 VGGNet 或 ResNet）的后几层 [27, 77]。

Figure 1: Illustrating the architecture of the proposed HRNet. It consists of parallel high-to-low resolution subnetworks with repeated information exchange across multi-resolution subnetworks (multi-scale fusion). The horizontal and vertical directions correspond to the depth of the network and the scale of the feature maps, respectively.

图 1：所提出 HRNet 的结构示意。它由并行的高低分辨率子网络组成，并在多分辨率子网络之间反复交换信息（多尺度融合）。横轴对应网络深度，纵轴对应特征图尺度。

We present a novel architecture, namely High-Resolution Net (HRNet), which is able to maintain high-resolution representations through the whole process. We start from a high-resolution subnetwork as the first stage, gradually add high-to-low resolution subnetworks one by one to form more stages, and connect the multi-resolution subnetworks in parallel. We conduct repeated multi-scale fusions by exchanging the information across the parallel multi-resolution subnetworks over and over through the whole process. We estimate the keypoints over the high-resolution representations output by our network. The resulting network is illustrated in Figure 1.

我们提出一种新结构，称为高分辨率网络（High-Resolution Net，HRNet），能够在整个过程中保持高分辨率表示。我们以高分辨率子网络作为第一阶段，再逐步加入由高到低的分辨率子网络以形成更多阶段，并把多分辨率子网络并行连接。我们通过在并行的多分辨率子网络之间反复交换信息，进行反复的多尺度融合。关键点在网络输出的高分辨率表示上估计。所得网络见图 1。

Figure 2: Illustration of representative pose estimation networks that rely on the high-to-low and low-to-high framework. (a) Hourglass [40]. (b) Cascaded pyramid networks [11]. (c) SimpleBaseline [72]: transposed convolutions for low-to-high processing. (d) Combination with dilated convolutions [27]. Bottom-right legend: reg. = regular convolution, dilated = dilated convolution, trans. = transposed convolution, strided = strided convolution, concat. = concatenation. In (a), the high-to-low and low-to-high processes are symmetric. In (b), (c) and (d), the high-to-low process, a part of a classification network (ResNet or VGGNet), is heavy, and the low-to-high process is light. In (a) and (b), the skip-connections (dashed lines) between the same-resolution layers of the high-to-low and low-to-high processes mainly aim to fuse low-level and high-level features. In (b), the right part, refinenet, combines the low-level and high-level features that are processed through convolutions.

图 2：依赖“由高到低、再由低到高”框架的代表性姿态估计网络。(a) Hourglass [40]。(b) 级联金字塔网络 [11]。(c) SimpleBaseline [72]：用转置卷积完成由低到高的处理。(d) 与空洞卷积的结合 [27]。右下图例：reg. = 常规卷积，dilated = 空洞卷积，trans. = 转置卷积，strided = 步长卷积，concat. = 拼接。在 (a) 中，由高到低与由低到高过程是对称的。在 (b)、(c) 和 (d) 中，由高到低过程作为分类网络（ResNet 或 VGGNet）的一部分较重，由低到高过程较轻。在 (a) 和 (b) 中，高低过程同分辨率层之间的跳跃连接（虚线）主要用于融合低层与高层特征。在 (b) 中，右侧的 refinenet 把经过卷积处理的低层与高层特征结合起来。

Our network has two benefits in comparison to existing widely-used networks [40, 27, 77, 72] for pose estimation. (i) Our approach connects high-to-low resolution subnetworks in parallel rather than in series as done in most existing solutions. Thus, our approach is able to maintain the high resolution instead of recovering the resolution through a low-to-high process, and accordingly the predicted heatmap is potentially spatially more precise. (ii) Most existing fusion schemes aggregate low-level and high-level representations. Instead, we perform repeated multi-scale fusions to boost the high-resolution representations with the help of the low-resolution representations of the same depth and similar level, and vice versa, resulting in that high-resolution representations are also rich for pose estimation. Consequently, our predicted heatmap is potentially more accurate.

与现有广泛使用的姿态估计网络 [40, 27, 77, 72] 相比，我们的网络有两点好处。(i) 我们把高低分辨率子网络并行连接，而不是像大多数方案那样串联。因此，我们能够保持高分辨率，而不必再通过由低到高的过程去恢复分辨率，预测热图在空间上有望更精细。(ii) 现有融合方案大多聚合低层与高层表示。相反，我们进行反复的多尺度融合：用同一深度、相近层级的低分辨率表示来增强高分辨率表示，反之亦然，从而使高分辨率表示对姿态估计也足够丰富。因此，预测热图有望更准确。

We empirically demonstrate the superior keypoint detection performance over two benchmark datasets: the COCO keypoint detection dataset [36] and the MPII Human Pose dataset [2]. In addition, we show the superiority of our network in video pose tracking on the PoseTrack dataset [1].

我们在两个基准数据集上实证展示了更优的关键点检测表现：COCO 关键点检测数据集 [36] 与 MPII Human Pose 数据集 [2]。此外，我们还在 PoseTrack 数据集 [1] 上展示了该网络在视频姿态跟踪中的优势。

2 Related Work

2 相关工作

Most traditional solutions to single-person pose estimation adopt the probabilistic graphical model or the pictorial structure model [79, 50], which is recently improved by exploiting deep learning for better modeling the unary and pair-wise energies [9, 65, 45] or imitating the iterative inference process [13]. Nowadays, deep convolutional neural network provides dominant solutions [20, 35, 62, 42, 43, 48, 58, 16]. There are two mainstream methods: regressing the position of keypoints [66, 7], and estimating keypoint heatmaps [13, 14, 78] followed by choosing the locations with the highest heat values as the keypoints.

单人姿态估计的传统方法大多采用概率图模型或图结构模型 [79, 50]；近期则通过深度学习更好地建模一元与成对能量 [9, 65, 45]，或模仿迭代推断过程 [13]。如今，深度卷积神经网络提供了主流方案 [20, 35, 62, 42, 43, 48, 58, 16]。有两条主线：直接回归关键点位置 [66, 7]，以及估计关键点热图 [13, 14, 78]，再取热值最高的位置作为关键点。

Most convolutional neural networks for keypoint heatmap estimation consist of a stem subnetwork similar to the classification network, which decreases the resolution, a main body producing the representations with the same resolution as its input, followed by a regressor estimating the heatmaps where the keypoint positions are estimated and then transformed in the full resolution. The main body mainly adopts the high-to-low and low-to-high framework, possibly augmented with multi-scale fusion and intermediate (deep) supervision.

大多数用于关键点热图估计的卷积神经网络包含一个类似分类网络的茎干部，用于降低分辨率；一个主体，产生与其输入分辨率相同的表示；再接一个回归器估计热图，从中得到关键点位置并变换回全分辨率。主体主要采用由高到低、再由低到高的框架，并可能辅以多尺度融合与中间（深度）监督。

High-to-low and low-to-high. The high-to-low process aims to generate low-resolution and high-level representations, and the low-to-high process aims to produce high-resolution representations [4, 11, 23, 72, 40, 62]. Both the two processes are possibly repeated several times for boosting the performance [77, 40, 14].

由高到低与由低到高。 由高到低过程旨在生成低分辨率、高层表示；由低到高过程旨在产生高分辨率表示 [4, 11, 23, 72, 40, 62]。这两个过程都可能重复若干次以提升表现 [77, 40, 14]。

Representative network design patterns include: (i) Symmetric high-to-low and low-to-high processes. Hourglass and its follow-ups [40, 14, 77, 31] design the low-to-high process as a mirror of the high-to-low process. (ii) Heavy high-to-low and light low-to-high. The high-to-low process is based on the ImageNet classification network, e.g., ResNet adopted in [11, 72], and the low-to-high process is simply a few bilinear-upsampling [11] or transpose convolution [72] layers. (iii) Combination with dilated convolutions. In [27, 51, 35], dilated convolutions are adopted in the last two stages in the ResNet or VGGNet to eliminate the spatial resolution loss, which is followed by a light low-to-high process to further increase the resolution, avoiding expensive computation cost for only using dilated convolutions [11, 27, 51]. Figure 2 depicts four representative pose estimation networks.

代表性的网络设计模式包括：(i) 对称的由高到低与由低到高过程。Hourglass 及其后续工作 [40, 14, 77, 31] 把由低到高过程设计成由高到低过程的镜像。(ii) 重的由高到低、轻的由低到高。由高到低过程基于 ImageNet 分类网络，例如 [11, 72] 采用的 ResNet；由低到高过程只是若干双线性上采样 [11] 或转置卷积 [72] 层。(iii) 与空洞卷积结合。在 [27, 51, 35] 中，ResNet 或 VGGNet 的最后两个阶段采用空洞卷积以减少空间分辨率损失，再接一个较轻的由低到高过程进一步提高分辨率，避免只使用空洞卷积带来的昂贵计算 [11, 27, 51]。图 2 画出了四种代表性姿态估计网络。

Multi-scale fusion. The straightforward way is to feed multi-resolution images separately into multiple networks and aggregate the output response maps [64]. Hourglass [40] and its extensions [77, 31] combine low-level features in the high-to-low process into the same-resolution high-level features in the low-to-high process progressively through skip connections. In cascaded pyramid network [11], a globalnet combines low-to-high level features in the high-to-low process progressively into the low-to-high process, and then a refinenet combines the low-to-high level features that are processed through convolutions. Our approach repeats multi-scale fusion, which is partially inspired by deep fusion and its extensions [67, 73, 59, 80, 82].

多尺度融合。 最直接的做法是把多分辨率图像分别送入多个网络，再聚合输出响应图 [64]。Hourglass [40] 及其扩展 [77, 31] 通过跳跃连接，把由高到低过程中的低层特征逐步并入由低到高过程中的同分辨率高层特征。在级联金字塔网络 [11] 中，globalnet 把由高到低过程里的低到高层特征逐步并入由低到高过程，随后 refinenet 再结合经过卷积处理的这些特征。我们的方法反复进行多尺度融合，部分受到深度融合及其扩展的启发 [67, 73, 59, 80, 82]。

Intermediate supervision. Intermediate supervision or deep supervision, early developed for image classification [34, 61], is also adopted for helping deep networks training and improving the heatmap estimation quality, e.g., [69, 40, 64, 3, 11]. The hourglass approach [40] and the convolutional pose machine approach [69] process the intermediate heatmaps as the input or a part of the input of the remaining subnetwork.

中间监督。 中间监督或深度监督最早用于图像分类 [34, 61]，后来也被用来帮助训练更深的网络并改善热图估计质量，例如 [69, 40, 64, 3, 11]。Hourglass 方法 [40] 与卷积姿态机 [69] 把中间热图作为后续子网络的输入或输入的一部分。

Our approach. Our network connects high-to-low subnetworks in parallel. It maintains high-resolution representations through the whole process for spatially precise heatmap estimation. It generates reliable high-resolution representations through repeatedly fusing the representations produced by the high-to-low subnetworks. Our approach is different from most existing works, which need a separate low-to-high upsampling process and aggregate low-level and high-level representations. Our approach, without using intermediate heatmap supervision, is superior in keypoint detection accuracy and efficient in computation complexity and parameters.

我们的方法。 我们的网络把高低分辨率子网络并行连接，在整个过程中保持高分辨率表示，以便空间上更精细地估计热图。它通过反复融合高低分辨率子网络产生的表示，生成可靠的高分辨率表示。这与大多数现有工作不同：后者需要单独的由低到高上采样过程，并聚合低层与高层表示。我们的方法不使用中间热图监督，在关键点检测精度上更优，计算复杂度与参数量也更高效。

There are related multi-scale networks for classification and segmentation [5, 8, 74, 81, 30, 76, 55, 56, 24, 83, 55, 52, 18]. Our work is partially inspired by some of them [56, 24, 83, 55], and there are clear differences making them not applicable to our problem. Convolutional neural fabrics [56] and interlinked CNN [83] fail to produce high-quality segmentation results because of a lack of proper design on each subnetwork (depth, batch normalization) and multi-scale fusion. The grid network [18], a combination of many weight-shared U-Nets, consists of two separate fusion processes across multi-resolution representations: on the first stage, information is only sent from high resolution to low resolution; on the second stage, information is only sent from low resolution to high resolution, and thus less competitive. Multi-scale densenets [24] does not target and cannot generate reliable high-resolution representations.

分类与分割领域也有相关的多尺度网络 [5, 8, 74, 81, 30, 76, 55, 56, 24, 83, 55, 52, 18]。本工作部分受到其中一些工作的启发 [56, 24, 83, 55]，但存在明显差异，使它们并不适用于我们的问题。Convolutional neural fabrics [56] 与 interlinked CNN [83] 由于对各子网络（深度、批归一化）及多尺度融合缺少恰当设计，无法给出高质量分割结果。网格网络 [18] 是许多权值共享 U-Net 的组合，它在多分辨率表示之间有两个分开的融合过程：第一阶段信息只从高分辨率流向低分辨率，第二阶段只从低分辨率流向高分辨率，因而竞争力较弱。Multi-scale densenets [24] 不以生成可靠的高分辨率表示为目标，也无法做到这一点。

3 Approach

3 方法

Human pose estimation, a.k.a. keypoint detection, aims to detect the locations of KKK keypoints or parts (e.g., elbow, wrist, etc) from an image I\mathbf{I}I of size W×H×3W\times H\times 3W×H×3. The state-of-the-art methods transform this problem to estimating KKK heatmaps of size W′×H′W^{{}^{\prime}}\times H^{{}^{\prime}}W′×H′, {H1,H2,…,HK}\{\mathbf{H}_{1},\mathbf{H}_{2},\dots,\mathbf{H}_{K}\}{H1​,H2​,…,HK​}, where each heatmap Hk\mathbf{H}_{k}Hk​ indicates the location confidence of the kkkth keypoint.

人体姿态估计，亦即关键点检测，旨在从大小为 W×H×3W\times H\times 3W×H×3 的图像 I\mathbf{I}I 中检测 KKK 个关键点或部位（如肘、腕等）的位置。当时最优方法把该问题转化为估计 KKK 张大小为 W′×H′W^{{}^{\prime}}\times H^{{}^{\prime}}W′×H′ 的热图 {H1,H2,…,HK}\{\mathbf{H}_{1},\mathbf{H}_{2},\dots,\mathbf{H}_{K}\}{H1​,H2​,…,HK​}，其中每张热图 Hk\mathbf{H}_{k}Hk​ 表示第 kkk 个关键点的位置置信度。

We follow the widely-adopted pipeline [40, 72, 11] to predict human keypoints using a convolutional network, which is composed of a stem consisting of two strided convolutions decreasing the resolution, a main body outputting the feature maps with the same resolution as its input feature maps, and a regressor estimating the heatmaps where the keypoint positions are chosen and transformed to the full resolution. We focus on the design of the main body and introduce our High-Resolution Net (HRNet) that is depicted in Figure 1.

我们沿用广泛采用的流程 [40, 72, 11]，用卷积网络预测人体关键点：茎干部由两次步长卷积组成以降低分辨率；主体输出与其输入特征图分辨率相同的特征图；回归器估计热图，从中选取关键点位置并变换回全分辨率。我们关注主体的设计，并介绍图 1 所示的高分辨率网络（HRNet）。

Sequential multi-resolution subnetworks. Existing networks for pose estimation are built by connecting high-to-low resolution subnetworks in series, where each subnetwork, forming a stage, is composed of a sequence of convolutions and there is a down-sample layer across adjacent subnetworks to halve the resolution.

串行多分辨率子网络。 现有姿态估计网络把高低分辨率子网络串联起来，每个子网络构成一个阶段，由一串卷积组成，相邻子网络之间有一层下采样，把分辨率减半。

Let Nsr\mathcal{N}_{sr}Nsr​ be the subnetwork in the sssth stage and rrr be the resolution index (Its resolution is 12r−1\frac{1}{2^{r-1}}2r−11​ of the resolution of the first subnetwork). The high-to-low network with SSS (e.g., 444) stages can be denoted as:

令 Nsr\mathcal{N}_{sr}Nsr​ 表示第 sss 阶段、分辨率索引为 rrr 的子网络（其分辨率是第一子网络分辨率的 12r−1\frac{1}{2^{r-1}}2r−11​）。具有 SSS（例如 444）个阶段的由高到低网络可以记为：

\begin{array}[]{llll}\mathcal{N}_{11}&\rightarrow~~\mathcal{N}_{22}&\rightarrow~~\mathcal{N}_{33}&\rightarrow~~\mathcal{N}_{44}.\end{array} \tag{1}

Parallel multi-resolution subnetworks. We start from a high-resolution subnetwork as the first stage, gradually add high-to-low resolution subnetworks one by one, forming new stages, and connect the multi-resolution subnetworks in parallel. As a result, the resolutions for the parallel subnetworks of a later stage consists of the resolutions from the previous stage, and an extra lower one.

并行多分辨率子网络。 我们以高分辨率子网络作为第一阶段，再逐步加入由高到低的分辨率子网络，形成新的阶段，并把多分辨率子网络并行连接。因此，较后阶段的并行子网络所含分辨率，包括前一阶段的全部分辨率，再加上一个更低的分辨率。

An example network structure, containing 444 parallel subnetworks, is given as follows,

一个包含 444 路并行子网络的示例结构如下，

$$\begin{array}[]{llll}\mathcal{N}{11}&\rightarrow~~\mathcal{N}{21}&\rightarrow~~\mathcal{N}{31}&\rightarrow~~\mathcal{N}{41}\
&\searrow~~\mathcal{N}{22}&\rightarrow~~\mathcal{N}{32}&\rightarrow~~\mathcal{N}{42}\
&&\searrow~~\mathcal{N}{33}&\rightarrow~~\mathcal{N}{43}\
&&&\searrow~~\mathcal{N}{44}.\end{array} \tag{2}$$

Figure 3: Illustrating how the exchange unit aggregates the information for high, medium and low resolutions from the left to the right, respectively. Right legend: strided 3×33\times 33×3 = strided 3×33\times 33×3 convolution, up samp. 1×11\times 11×1 = nearest neighbor up-sampling following a 1×11\times 11×1 convolution.

图 3：交换单元如何分别从左到右聚合高、中、低分辨率的信息。右侧图例：strided 3×33\times 33×3 = 步长为 2 的 3×33\times 33×3 卷积，up samp. 1×11\times 11×1 = 最近邻上采样后再接 1×11\times 11×1 卷积。

Repeated multi-scale fusion. We introduce exchange units across parallel subnetworks such that each subnetwork repeatedly receives the information from other parallel subnetworks. Here is an example showing the scheme of exchanging information. We divided the third stage into several (e.g., 333) exchange blocks, and each block is composed of 333 parallel convolution units with an exchange unit across the parallel units, which is given as follows,

反复多尺度融合。 我们在并行子网络之间引入交换单元，使每个子网络反复从其他并行子网络接收信息。下面给出信息交换方案的一个例子。我们把第三阶段分成若干（例如 333）个交换块，每个块由 333 个并行卷积单元以及跨这些并行单元的一个交换单元组成，记为：

$$\begin{array}[]{llllll}\mathcal{C}^{1}{31}&\searrow&\nearrow~~\mathcal{C}^{2}{31}&\searrow&\nearrow~~\mathcal{C}^{3}{31}&\searrow\
\mathcal{C}^{1}{32}&\rightarrow~~\mathcal{E}^{1}{3}&\rightarrow~~\mathcal{C}^{2}{32}&\rightarrow~~\mathcal{E}^{2}{3}&\rightarrow~~\mathcal{C}^{3}{32}&\rightarrow~~\mathcal{E}^{3}{3},\
\mathcal{C}^{1}{33}&\nearrow&\searrow~~\mathcal{C}^{2}{33}&\nearrow&\searrow~~\mathcal{C}^{3}{33}&\nearrow\
\end{array} \tag{3}$$

$$\begin{array}[]{llllll}\mathcal{C}^{1}{31}&\searrow&\nearrow~~\mathcal{C}^{2}{31}&\searrow&\nearrow~~\mathcal{C}^{3}{31}&\searrow\\mathcal{C}^{1}{32}&\rightarrow~~\mathcal{E}^{1}{3}&\rightarrow~~\mathcal{C}^{2}{32}&\rightarrow~~\mathcal{E}^{2}{3}&\rightarrow~~\mathcal{C}^{3}{32}&\rightarrow~~\mathcal{E}^{3}{3},\\mathcal{C}^{1}{33}&\nearrow&\searrow~~\mathcal{C}^{2}{33}&\nearrow&\searrow~~\mathcal{C}^{3}{33}&\nearrow\
\end{array} \tag{3}$$

where Csrb\mathcal{C}^{b}_{sr}Csrb​ represents the convolution unit in the rrrth resolution of the bbbth block in the sssth stage, and Esb\mathcal{E}^{b}_{s}Esb​ is the corresponding exchange unit.

其中 Csrb\mathcal{C}^{b}_{sr}Csrb​ 表示第 sss 阶段第 bbb 块中第 rrr 个分辨率上的卷积单元，Esb\mathcal{E}^{b}_{s}Esb​ 是相应的交换单元。

We illustrate the exchange unit in Figure 3 and present the formulation in the following. We drop the subscript sss and the superscript bbb for discussion convenience. The inputs are sss response maps: {X1,X2,…,Xs}\{\mathbf{X}_{1},\mathbf{X}_{2},\dots,\mathbf{X}_{s}\}{X1​,X2​,…,Xs​}. The outputs are sss response maps: {Y1,Y2,…,Ys}\{\mathbf{Y}_{1},\mathbf{Y}_{2},\dots,\mathbf{Y}_{s}\}{Y1​,Y2​,…,Ys​}, whose resolutions and widths are the same to the input. Each output is an aggregation of the input maps, Yk=∑i=1sa(Xi,k)\mathbf{Y}_{k}=\sum_{i=1}^{s}a(\mathbf{X}_{i},k)Yk​=∑i=1s​a(Xi​,k). The exchange unit across stages has an extra output map Ys+1\mathbf{Y}_{s+1}Ys+1​: Ys+1=a(Ys,s+1)\mathbf{Y}_{s+1}=a(\mathbf{Y}_{s},s+1)Ys+1​=a(Ys​,s+1).

我们在图 3 中示意交换单元，并在下文给出形式化描述。为讨论方便，我们略去下标 sss 与上标 bbb。输入是 sss 张响应图：{X1,X2,…,Xs}\{\mathbf{X}_{1},\mathbf{X}_{2},\dots,\mathbf{X}_{s}\}{X1​,X2​,…,Xs​}。输出也是 sss 张响应图：{Y1,Y2,…,Ys}\{\mathbf{Y}_{1},\mathbf{Y}_{2},\dots,\mathbf{Y}_{s}\}{Y1​,Y2​,…,Ys​}，其分辨率与宽度与输入相同。每个输出是输入图的聚合，Yk=∑i=1sa(Xi,k)\mathbf{Y}_{k}=\sum_{i=1}^{s}a(\mathbf{X}_{i},k)Yk​=∑i=1s​a(Xi​,k)。跨阶段的交换单元还有一个额外输出图 Ys+1\mathbf{Y}_{s+1}Ys+1​：Ys+1=a(Ys,s+1)\mathbf{Y}_{s+1}=a(\mathbf{Y}_{s},s+1)Ys+1​=a(Ys​,s+1)。

The function a(Xi,k)a(\mathbf{X}_{i},k)a(Xi​,k) consists of upsampling or downsampling Xi\mathbf{X}_{i}Xi​ from resolution iii to resolution kkk. We adopt strided 3×33\times 33×3 convolutions for downsampling. For instance, one strided 3×33\times 33×3 convolution with the stride 222 for 2×2\times2× downsampling, and two consecutive strided 3×33\times 33×3 convolutions with the stride 222 for 4×4\times4× downsampling. For upsampling, we adopt the simple nearest neighbor sampling following a 1×11\times 11×1 convolution for aligning the number of channels. If i=ki=ki=k, a(⋅,⋅)a(\cdot,\cdot)a(⋅,⋅) is just an identify connection: a(Xi,k)=Xia(\mathbf{X}_{i},k)=\mathbf{X}_{i}a(Xi​,k)=Xi​.

函数 a(Xi,k)a(\mathbf{X}_{i},k)a(Xi​,k) 把 Xi\mathbf{X}_{i}Xi​ 从分辨率 iii 上采样或下采样到分辨率 kkk。下采样采用步长 3×33\times 33×3 卷积：例如，一次步长为 222 的 3×33\times 33×3 卷积完成 2×2\times2× 下采样，两次连续的步长为 222 的 3×33\times 33×3 卷积完成 4×4\times4× 下采样。上采样采用简单的最近邻采样，再接一个 1×11\times 11×1 卷积以对齐通道数。若 i=ki=ki=k，a(⋅,⋅)a(\cdot,\cdot)a(⋅,⋅) 就是恒等连接：a(Xi,k)=Xia(\mathbf{X}_{i},k)=\mathbf{X}_{i}a(Xi​,k)=Xi​。

Heatmap estimation. We regress the heatmaps simply from the high-resolution representations output by the last exchange unit, which empirically works well. The loss function, defined as the mean squared error, is applied for comparing the predicted heatmaps and the groundtruth heatmaps. The groundtruth heatmpas are generated by applying 222D Gaussian with standard deviation of 111 pixel centered on the grouptruth location of each keypoint.

热图估计。 我们直接从最后一个交换单元输出的高分辨率表示回归热图，经验上效果很好。损失函数取均方误差，用于比较预测热图与真值热图。真值热图由以每个关键点真值位置为中心、标准差为 111 个像素的二维高斯生成。

Network instantiation. We instantiate the network for keypoint heatmap estimation by following the design rule of ResNet to distribute the depth to each stage and the number of channels to each resolution.

网络实例化。 我们按 ResNet 的设计规则，把深度分配到各个阶段、把通道数分配到各个分辨率，以此来实例化用于关键点热图估计的网络。

The main body, i.e., our HRNet, contains four stages with four parallel subnetworks, whose the resolution is gradually decreased to a half and accordingly the width (the number of channels) is increased to the double. The first stage contains 444 residual units where each unit, the same to the ResNet-505050, is formed by a bottleneck with the width 646464, and is followed by one 3×33\times 33×3 convolution reducing the width of feature maps to CCC. The 222nd, 333rd, 444th stages contain 111, 444, 333 exchange blocks, respectively. One exchange block contains 444 residual units where each unit contains two 3×33\times 33×3 convolutions in each resolution and an exchange unit across resolutions. In summary, there are totally 888 exchange units, i.e., 888 multi-scale fusions are conducted.

主体即我们的 HRNet，包含四个阶段、四路并行子网络：分辨率逐步减半，宽度（通道数）相应地加倍。第一阶段包含 444 个残差单元，每个单元与 ResNet-505050 相同，由宽度为 646464 的瓶颈块构成，随后用一个 3×33\times 33×3 卷积把特征图宽度降到 CCC。第 222、333、444 阶段分别包含 111、444、333 个交换块。一个交换块在每个分辨率上包含 444 个残差单元（每个单元含两个 3×33\times 33×3 卷积），外加一个跨分辨率的交换单元。总计有 888 个交换单元，也就是进行了 888 次多尺度融合。

In our experiments, we study one small net and one big net: HRNet-W323232 and HRNet-W484848, where 323232 and 484848 represent the widths (CCC) of the high-resolution subnetworks in last three stages, respectively. The widths of other three parallel subnetworks are 64,128,25664,128,25664,128,256 for HRNet-W323232, and 96,192,38496,192,38496,192,384 for HRNet-W484848.

实验中我们研究一个小网络和一个大网络：HRNet-W323232 与 HRNet-W484848，其中 323232 与 484848 分别表示后三个阶段里高分辨率子网络的宽度（CCC）。其余三路并行子网络的宽度，对 HRNet-W323232 为 64,128,25664,128,25664,128,256，对 HRNet-W484848 为 96,192,38496,192,38496,192,384。

4 Experiments

4 实验

Method Backbone Pretrain Input size #Params GFLOPs AP⁡\operatorname{AP}AP AP⁡50\operatorname{AP}^{50}AP50 AP⁡75\operatorname{AP}^{75}AP75 AP⁡M\operatorname{AP}^{M}APM AP⁡L\operatorname{AP}^{L}APL AR⁡\operatorname{AR}AR
888-stage Hourglass [40] 888-stage Hourglass N 256×192256\times 192256×192 25.125.125.1M 14.314.314.3 66.966.966.9 −-− −-− −-− −-− −-−
CPN [11] ResNet-50 Y 256×192256\times 192256×192 27.027.027.0M 6.206.206.20 68.668.668.6 −-− −-− −-− −-− −-−
CPN + OHKM [11] ResNet-50 Y 256×192256\times 192256×192 27.027.027.0M 6.206.206.20 69.469.469.4 −-− −-− −-− −-− −-−
SimpleBaseline [72] ResNet-50 Y 256×192256\times 192256×192 34.034.034.0M 8.908.908.90 70.4{70.4}70.4 88.6{88.6}88.6 78.3{78.3}78.3 67.1{67.1}67.1 77.2{77.2}77.2 76.3{76.3}76.3
SimpleBaseline [72] ResNet-101 Y 256×192256\times 192256×192 53.053.053.0M 12.412.412.4 71.4{71.4}71.4 89.3{89.3}89.3 79.3{79.3}79.3 68.1{68.1}68.1 78.1{78.1}78.1 77.1{77.1}77.1
SimpleBaseline [72] ResNet-152 Y 256×192256\times 192256×192 68.668.668.6M 15.715.715.7 72.0{72.0}72.0 89.3{89.3}89.3 79.8{79.8}79.8 68.7{68.7}68.7 78.9{78.9}78.9 77.8{77.8}77.8
HRNet-W323232 HRNet-W323232 N 256×192256\times 192256×192 28.528.528.5M 7.107.107.10 73.473.473.4 89.589.589.5 80.780.780.7 70.270.270.2 80.180.180.1 78.978.978.9
HRNet-W323232 HRNet-W323232 Y 256×192256\times 192256×192 28.528.528.5M 7.107.107.10 74.474.474.4 90.590.590.5 81.981.981.9 70.870.870.8 81.081.081.0 79.879.879.8
HRNet-W484848 HRNet-W484848 Y 256×192256\times 192256×192 63.663.663.6M 14.614.614.6 75.175.175.1 90.690.690.6 82.282.282.2 71.571.571.5 81.881.881.8 80.480.480.4
SimpleBaseline [72] ResNet-152 Y 384×288384\times 288384×288 68.668.668.6M 35.635.635.6 74.3{74.3}74.3 89.6{89.6}89.6 81.1{81.1}81.1 70.5{70.5}70.5 79.7{79.7}79.7 79.7{79.7}79.7
HRNet-W323232 HRNet-W323232 Y 384×288384\times 288384×288 28.528.528.5M 16.016.016.0 75.875.875.8 90.690.690.6 82.7{82.7}82.7 71.971.971.9 82.882.882.8 81.081.081.0
HRNet-W484848 HRNet-W484848 Y 384×288384\times 288384×288 63.663.663.6M 32.932.932.9 76.3 90.8 82.9 72.3 83.4 81.2

Table 1: Comparisons on the COCO validation set. Pretrain = pretrain the backbone on the ImageNet classification task. OHKM = online hard keypoints mining [11].

方法 骨干网络 预训练 输入尺寸 参数量 GFLOPs AP⁡\operatorname{AP}AP AP⁡50\operatorname{AP}^{50}AP50 AP⁡75\operatorname{AP}^{75}AP75 AP⁡M\operatorname{AP}^{M}APM AP⁡L\operatorname{AP}^{L}APL AR⁡\operatorname{AR}AR
888-stage Hourglass [40] 888-stage Hourglass N 256×192256\times 192256×192 25.125.125.1M 14.314.314.3 66.966.966.9 −-− −-− −-− −-− −-−
CPN [11] ResNet-50 Y 256×192256\times 192256×192 27.027.027.0M 6.206.206.20 68.668.668.6 −-− −-− −-− −-− −-−
CPN + OHKM [11] ResNet-50 Y 256×192256\times 192256×192 27.027.027.0M 6.206.206.20 69.469.469.4 −-− −-− −-− −-− −-−
SimpleBaseline [72] ResNet-50 Y 256×192256\times 192256×192 34.034.034.0M 8.908.908.90 70.4{70.4}70.4 88.6{88.6}88.6 78.3{78.3}78.3 67.1{67.1}67.1 77.2{77.2}77.2 76.3{76.3}76.3
SimpleBaseline [72] ResNet-101 Y 256×192256\times 192256×192 53.053.053.0M 12.412.412.4 71.4{71.4}71.4 89.3{89.3}89.3 79.3{79.3}79.3 68.1{68.1}68.1 78.1{78.1}78.1 77.1{77.1}77.1
SimpleBaseline [72] ResNet-152 Y 256×192256\times 192256×192 68.668.668.6M 15.715.715.7 72.0{72.0}72.0 89.3{89.3}89.3 79.8{79.8}79.8 68.7{68.7}68.7 78.9{78.9}78.9 77.8{77.8}77.8
HRNet-W323232 HRNet-W323232 N 256×192256\times 192256×192 28.528.528.5M 7.107.107.10 73.473.473.4 89.589.589.5 80.780.780.7 70.270.270.2 80.180.180.1 78.978.978.9
HRNet-W323232 HRNet-W323232 Y 256×192256\times 192256×192 28.528.528.5M 7.107.107.10 74.474.474.4 90.590.590.5 81.981.981.9 70.870.870.8 81.081.081.0 79.879.879.8
HRNet-W484848 HRNet-W484848 Y 256×192256\times 192256×192 63.663.663.6M 14.614.614.6 75.175.175.1 90.690.690.6 82.282.282.2 71.571.571.5 81.881.881.8 80.480.480.4
SimpleBaseline [72] ResNet-152 Y 384×288384\times 288384×288 68.668.668.6M 35.635.635.6 74.3{74.3}74.3 89.6{89.6}89.6 81.1{81.1}81.1 70.5{70.5}70.5 79.7{79.7}79.7 79.7{79.7}79.7
HRNet-W323232 HRNet-W323232 Y 384×288384\times 288384×288 28.528.528.5M 16.016.016.0 75.875.875.8 90.690.690.6 82.7{82.7}82.7 71.971.971.9 82.882.882.8 81.081.081.0
HRNet-W484848 HRNet-W484848 Y 384×288384\times 288384×288 63.663.663.6M 32.932.932.9 76.3 90.8 82.9 72.3 83.4 81.2

表 1：COCO 验证集上的比较。预训练 = 在 ImageNet 分类任务上预训练骨干网络。OHKM = 在线困难关键点挖掘 [11]。

Method Backbone Input size #Params GFLOPs AP⁡\operatorname{AP}AP AP⁡50\operatorname{AP}^{50}AP50 AP⁡75\operatorname{AP}^{75}AP75 AP⁡M\operatorname{AP}^{M}APM AP⁡L\operatorname{AP}^{L}APL AR⁡\operatorname{AR}AR
Bottom-up: keypoint detection and grouping
OpenPose [6] \hfil−\hfil-\hfil− −-− −-− −-− 61.861.861.8 84.984.984.9 67.567.567.5 57.157.157.1 68.268.268.2 66.566.566.5
Associative Embedding [39] \hfil−\hfil-\hfil− −-− −-− −-− 65.565.565.5 86.886.886.8 72.372.372.3 60.660.660.6 72.672.672.6 70.270.270.2
PersonLab [46] \hfil−\hfil-\hfil− −-− −-− −-− 68.768.768.7 89.089.089.0 75.475.475.4 64.164.164.1 75.575.575.5 75.475.475.4
MultiPoseNet [33] \hfil−\hfil-\hfil− −-− −-− −-− 69.669.669.6 86.386.386.3 76.676.676.6 65.065.065.0 76.376.376.3 73.573.573.5
Top-down: human detection and single-person keypoint detection
Mask-RCNN [21] ResNet-50-FPN −-− −-− −-− 63.163.163.1 87.387.387.3 68.768.768.7 57.857.857.8 71.471.471.4 −-−
G-RMI [47] ResNet-101 353×257353\times 257353×257 42.642.642.6M 57.057.057.0 64.964.964.9 85.585.585.5 71.371.371.3 62.362.362.3 70.070.070.0 69.769.769.7
Integral Pose Regression [60] ResNet-101 256×256256\times 256256×256 45.045.045.0M 11.011.011.0 67.867.867.8 88.288.288.2 74.874.874.8 63.963.963.9 74.074.074.0 −-−
G-RMI + extra data [47] ResNet-101 353×257353\times 257353×257 42.642.642.6M 57.057.057.0 68.568.568.5 87.187.187.1 75.575.575.5 65.865.865.8 73.373.373.3 73.373.373.3
CPN [11] ResNet-Inception 384×288384\times 288384×288 −-− −-− 72.172.172.1 91.491.491.4 80.080.080.0 68.768.768.7 77.277.277.2 78.578.578.5
RMPE [17] PyraNet [77] 320×256320\times 256320×256 28.128.128.1M 26.726.726.7 72.372.372.3 89.289.289.2 79.179.179.1 68.068.068.0 78.678.678.6 −-−
CFN [25] \hfil−\hfil-\hfil− −-− −-− −-− 72.672.672.6 86.186.186.1 69.769.769.7 78.378.378.3 64.164.164.1 −-−
CPN (ensemble) [11] ResNet-Inception 384×288384\times 288384×288 −-− −-− 73.073.073.0 91.791.791.7 80.980.980.9 69.569.569.5 78.178.178.1 79.079.079.0
SimpleBaseline [72] ResNet-152 384×288384\times 288384×288 68.668.668.6M 35.635.635.6 73.7{73.7}73.7 91.9{91.9}91.9 81.1{81.1}81.1 70.3{70.3}70.3 80.0{80.0}80.0 79.0{79.0}79.0
HRNet-W323232 HRNet-W323232 384×288384\times 288384×288 28.528.528.5M 16.016.016.0 74.974.974.9 92.592.592.5 82.882.882.8 71.371.371.3 80.980.980.9 80.180.180.1
HRNet-W484848 HRNet-W484848 384×288384\times 288384×288 63.663.663.6M 32.932.932.9 75.5 92.5 83.3 71.9 81.5 80.5
HRNet-W484848 + extra data HRNet-W484848 384×288384\times 288384×288 63.663.663.6M 32.932.932.9 77.0 92.7 84.5 73.4 83.1 82.0

Table 2: Comparisons on the COCO test-dev set. #Params and FLOPs are calculated for the pose estimation network, and those for human detection and keypoint grouping are not included.

方法 骨干网络 输入尺寸 参数量 GFLOPs AP⁡\operatorname{AP}AP AP⁡50\operatorname{AP}^{50}AP50 AP⁡75\operatorname{AP}^{75}AP75 AP⁡M\operatorname{AP}^{M}APM AP⁡L\operatorname{AP}^{L}APL AR⁡\operatorname{AR}AR
自下而上：关键点检测与分组
OpenPose [6] \hfil−\hfil-\hfil− −-− −-− −-− 61.861.861.8 84.984.984.9 67.567.567.5 57.157.157.1 68.268.268.2 66.566.566.5
Associative Embedding [39] \hfil−\hfil-\hfil− −-− −-− −-− 65.565.565.5 86.886.886.8 72.372.372.3 60.660.660.6 72.672.672.6 70.270.270.2
PersonLab [46] \hfil−\hfil-\hfil− −-− −-− −-− 68.768.768.7 89.089.089.0 75.475.475.4 64.164.164.1 75.575.575.5 75.475.475.4
MultiPoseNet [33] \hfil−\hfil-\hfil− −-− −-− −-− 69.669.669.6 86.386.386.3 76.676.676.6 65.065.065.0 76.376.376.3 73.573.573.5
自上而下：人体检测与单人关键点检测
Mask-RCNN [21] ResNet-50-FPN −-− −-− −-− 63.163.163.1 87.387.387.3 68.768.768.7 57.857.857.8 71.471.471.4 −-−
G-RMI [47] ResNet-101 353×257353\times 257353×257 42.642.642.6M 57.057.057.0 64.964.964.9 85.585.585.5 71.371.371.3 62.362.362.3 70.070.070.0 69.769.769.7
Integral Pose Regression [60] ResNet-101 256×256256\times 256256×256 45.045.045.0M 11.011.011.0 67.867.867.8 88.288.288.2 74.874.874.8 63.963.963.9 74.074.074.0 −-−
G-RMI + extra data [47] ResNet-101 353×257353\times 257353×257 42.642.642.6M 57.057.057.0 68.568.568.5 87.187.187.1 75.575.575.5 65.865.865.8 73.373.373.3 73.373.373.3
CPN [11] ResNet-Inception 384×288384\times 288384×288 −-− −-− 72.172.172.1 91.491.491.4 80.080.080.0 68.768.768.7 77.277.277.2 78.578.578.5
RMPE [17] PyraNet [77] 320×256320\times 256320×256 28.128.128.1M 26.726.726.7 72.372.372.3 89.289.289.2 79.179.179.1 68.068.068.0 78.678.678.6 −-−
CFN [25] \hfil−\hfil-\hfil− −-− −-− −-− 72.672.672.6 86.186.186.1 69.769.769.7 78.378.378.3 64.164.164.1 −-−
CPN (ensemble) [11] ResNet-Inception 384×288384\times 288384×288 −-− −-− 73.073.073.0 91.791.791.7 80.980.980.9 69.569.569.5 78.178.178.1 79.079.079.0
SimpleBaseline [72] ResNet-152 384×288384\times 288384×288 68.668.668.6M 35.635.635.6 73.7{73.7}73.7 91.9{91.9}91.9 81.1{81.1}81.1 70.3{70.3}70.3 80.0{80.0}80.0 79.0{79.0}79.0
HRNet-W323232 HRNet-W323232 384×288384\times 288384×288 28.528.528.5M 16.016.016.0 74.974.974.9 92.592.592.5 82.882.882.8 71.371.371.3 80.980.980.9 80.180.180.1
HRNet-W484848 HRNet-W484848 384×288384\times 288384×288 63.663.663.6M 32.932.932.9 75.5 92.5 83.3 71.9 81.5 80.5
HRNet-W484848 + extra data HRNet-W484848 384×288384\times 288384×288 63.663.663.6M 32.932.932.9 77.0 92.7 84.5 73.4 83.1 82.0

表 2：COCO test-dev 集上的比较。参数量与 FLOPs 按姿态估计网络计算，不包括人体检测与关键点分组。

4.1 COCO Keypoint Detection

4.1 COCO 关键点检测

Dataset. The COCO dataset [36] contains over 200,000200,000200,000 images and 250,000250,000250,000 person instances labeled with 171717 keypoints. We train our model on COCO train201720172017 dataset, including 57K57K57K images and 150K150K150K person instances. We evaluate our approach on the val201720172017 set and test-dev201720172017 set, containing 500050005000 images and 20K20K20K images, respectively.

数据集。 COCO 数据集 [36] 包含超过 200,000200,000200,000 张图像与 250,000250,000250,000 个标有 171717 个关键点的人体实例。我们在 COCO train201720172017 上训练，含 57K57K57K 张图像与 150K150K150K 个人体实例。我们在 val201720172017 与 test-dev201720172017 上评估，分别含 500050005000 张与 20K20K20K 张图像。

Evaluation metric. The standard evaluation metric is based on Object Keypoint Similarity (OKS): OKS⁡=∑iexp⁡(−di2/2s2ki2)δ(vi>0)∑iδ(vi>0).\operatorname{OKS}=\frac{\sum_{i}\exp(-d_{i}^{2}/2s^{2}k_{i}^{2})\delta(v_{i}>0)}{\sum_{i}\delta(v_{i}>0)}.OKS=∑i​δ(vi​>0)∑i​exp(−di2​/2s2ki2​)δ(vi​>0)​. Here did_{i}di​ is the Euclidean distance between the detected keypoint and the corresponding ground truth, viv_{i}vi​ is the visibility flag of the ground truth, sss is the object scale, and kik_{i}ki​ is a per-keypoint constant that controls falloff. We report standard average precision and recall scores[^1]: AP⁡50\operatorname{AP}^{50}AP50 (AP⁡\operatorname{AP}AP at OKS⁡=0.50\operatorname{OKS}=0.50OKS=0.50) AP⁡75\operatorname{AP}^{75}AP75, AP⁡\operatorname{AP}AP (the mean of AP⁡\operatorname{AP}AP scores at 101010 positions, OKS⁡=0.50,0.55,…,0.90,0.95\operatorname{OKS}=0.50,0.55,\dots,0.90,0.95OKS=0.50,0.55,…,0.90,0.95; AP⁡M\operatorname{AP}^{M}APM for medium objects, AP⁡L\operatorname{AP}^{L}APL for large objects, and AR⁡\operatorname{AR}AR at OKS⁡=0.50,0.55,…,0.90,0.955\operatorname{OKS}=0.50,0.55,\dots,0.90,0.955OKS=0.50,0.55,…,0.90,0.955.

评价指标。 标准指标基于目标关键点相似度（OKS）：OKS⁡=∑iexp⁡(−di2/2s2ki2)δ(vi>0)∑iδ(vi>0)\operatorname{OKS}=\frac{\sum_{i}\exp(-d_{i}^{2}/2s^{2}k_{i}^{2})\delta(v_{i}>0)}{\sum_{i}\delta(v_{i}>0)}OKS=∑i​δ(vi​>0)∑i​exp(−di2​/2s2ki2​)δ(vi​>0)​。其中 did_{i}di​ 是检测关键点与对应真值的欧氏距离，viv_{i}vi​ 是真值的可见性标志，sss 是目标尺度，kik_{i}ki​ 是控制衰减的逐关键点常数。我们报告标准平均精度与召回[^1]：AP⁡50\operatorname{AP}^{50}AP50（OKS⁡=0.50\operatorname{OKS}=0.50OKS=0.50 时的 AP⁡\operatorname{AP}AP）、AP⁡75\operatorname{AP}^{75}AP75、AP⁡\operatorname{AP}AP（在 OKS⁡=0.50,0.55,…,0.90,0.95\operatorname{OKS}=0.50,0.55,\dots,0.90,0.95OKS=0.50,0.55,…,0.90,0.95 这 101010 个位置上 AP⁡\operatorname{AP}AP 的均值）、中等目标的 AP⁡M\operatorname{AP}^{M}APM、大目标的 AP⁡L\operatorname{AP}^{L}APL，以及在 OKS⁡=0.50,0.55,…,0.90,0.95\operatorname{OKS}=0.50,0.55,\dots,0.90,0.95OKS=0.50,0.55,…,0.90,0.95 上的 AR⁡\operatorname{AR}AR。

Training. We extend the human detection box in height or width to a fixed aspect ratio: height⁡:width⁡=4:3\operatorname{height}:\operatorname{width}=4:3height:width=4:3, and then crop the box from the image, which is resized to a fixed size, 256×192256\times 192256×192 or 384×288384\times 288384×288. The data augmentation includes random rotation ([[[,,,]]]), random scale ([0.65,1.35][0.65,1.35][0.65,1.35]), and flipping. Following  [68], half body data augmentation is also involved.

训练。 我们把人体检测框在高度或宽度上扩展到固定宽高比：height⁡:width⁡=4:3\operatorname{height}:\operatorname{width}=4:3height:width=4:3，再从图像中裁出该框，并缩放到固定尺寸 256×192256\times 192256×192 或 384×288384\times 288384×288。数据增强包括随机旋转、随机尺度（[0.65,1.35][0.65,1.35][0.65,1.35]）以及翻转。沿用 [68]，也加入半身数据增强。

We use the Adam optimizer [32]. The learning schedule follows the setting [72]. The base learning rate is set as 1e−31\mathrm{e}{-3}1e−3, and is dropped to 1e−41\mathrm{e}{-4}1e−4 and 1e−51\mathrm{e}{-5}1e−5 at the 170170170th and 200200200th epochs, respectively. The training process is terminated within 210210210 epochs.

我们使用 Adam 优化器 [32]。学习率日程沿用 [72] 的设置。基础学习率设为 1e−31\mathrm{e}{-3}1e−3，并在第 170170170 与第 200200200 个周期分别降到 1e−41\mathrm{e}{-4}1e−4 与 1e−51\mathrm{e}{-5}1e−5。训练在 210210210 个周期内结束。

Testing. The two-stage top-down paradigm similar as [47, 11, 72] is used: detect the person instance using a person detector, and then predict detection keypoints.

测试。 采用与 [47, 11, 72] 类似的两阶段自上而下范式：先用人体检测器检测人体实例，再预测关键点。

We use the same person detectors provided by SimpleBaseline[^2] [72] for both validation set and test-dev set. Following the common practice [72, 40, 11], we compute the heatmap by averaging the headmaps of the original and flipped images. Each keypoint location is predicted by adjusting the highest heatvalue location with a quarter offset in the direction from the highest response to the second highest response.

验证集与 test-dev 集都使用 SimpleBaseline[^2] [72] 提供的同一套人体检测器。按常见做法 [72, 40, 11]，我们把原图与翻转图像的热图取平均。每个关键点位置由热值最高处出发，沿最高响应指向次高响应的方向做四分之一偏移来调整。

Results on the validation set. We report the results of our method and other state-of–the-art methods in Table 1. Our small network - HRNet-W323232, trained from scratch with the input size 256×192256\times 192256×192, achieves an 73.473.473.4 AP score, outperforming other methods with the same input size. (i) Compared to Hourglass [40], our small network improves AP by 6.56.56.5 points, and the GFLOPs of our network is much lower and less than half, while the number of parameters are similar and ours is slightly larger. (ii) Compared to CPN [11] w/o and w/ OHKM, our network, with slightly larger model size and slightly higher complexity, achieves 4.84.84.8 and 4.04.04.0 points gain, respectively. (iii) Compared to the previous best-performed SimpleBaseline [72], our small net HRNet-W323232 obtains significant improvements: 3.03.03.0 points gain for the backbone ResNet-505050 with a similar model size and GFLOPs, and 1.41.41.4 points gain for the backbone ResNet-152152152 whose model size (#Params) and GLOPs are twice as many as ours.

验证集结果。 表 1 报告了我们的方法与其他当时最优方法的结果。我们的小网络 HRNet-W323232 从零训练、输入尺寸为 256×192256\times 192256×192 时得到 73.473.473.4 AP，超过了相同输入尺寸下的其他方法。(i) 相对 Hourglass [40]，我们的小网络把 AP 提高了 6.56.56.5 个点，GFLOPs 远低且不到其一半，参数量相近、我们的略多。(ii) 相对不带与带 OHKM 的 CPN [11]，我们的网络在模型规模与复杂度略高的情况下分别获得 4.84.84.8 与 4.04.04.0 个点的提升。(iii) 相对此前表现最好的 SimpleBaseline [72]，小网络 HRNet-W323232 取得显著改进：相对骨干为 ResNet-505050、模型规模与 GFLOPs 相近的设置提升 3.03.03.0 个点；相对骨干为 ResNet-152152152、参数量与 GFLOPs 约为我们两倍的设置提升 1.41.41.4 个点。

Our nets can benefit from (i) training from the model pretrained for the ImageNet classification problem: The gain is 1.01.01.0 points for HRNet-W323232; (ii) increasing the capacity by increasing the width: Our big net HRNet-W484848 gets 0.70.70.7 and 0.50.50.5 improvements for the input sizes 256×192256\times 192256×192 and 384×288384\times 288384×288, respectively.

我们的网络还能从以下两方面受益：(i) 用 ImageNet 分类预训练模型初始化：HRNet-W323232 提升 1.01.01.0 个点；(ii) 通过增加宽度来增大容量：大网络 HRNet-W484848 在输入尺寸 256×192256\times 192256×192 与 384×288384\times 288384×288 上分别再提升 0.70.70.7 与 0.50.50.5。

Considering the input size 384×288384\times 288384×288, our HRNet-W323232 and HRNet-W484848, get the 75.875.875.8 and 76.376.376.3 AP, which have 1.41.41.4 and 1.21.21.2 improvements compared to the input size 256×192256\times 192256×192. In comparison to the SimpleBaseline [72] that uses ResNet-152152152 as the backbone, our HRNet-W323232 and HRNet-W484848 attain 1.51.51.5 and 2.02.02.0 points gain in terms of AP at 45%45\%45% and 92.4%92.4\%92.4% computational cost, respectively.

当输入尺寸为 384×288384\times 288384×288 时，HRNet-W323232 与 HRNet-W484848 分别得到 75.875.875.8 与 76.376.376.3 AP，相对输入尺寸 256×192256\times 192256×192 提升 1.41.41.4 与 1.21.21.2。与以 ResNet-152152152 为骨干的 SimpleBaseline [72] 相比，HRNet-W323232 与 HRNet-W484848 分别以 45%45\%45% 与 92.4%92.4\%92.4% 的计算代价获得 1.51.51.5 与 2.02.02.0 个 AP 点的提升。

Results on the test-dev set. Table 2 reports the pose estimation performances of our approach and the existing state-of-the-art approaches. Our approach is significantly better than bottom-up approaches. On the other hand, our small network, HRNet-W323232, achieves an AP of 74.974.974.9. It outperforms all the other top-down approaches, and is more efficient in terms of model size (#Params) and computation complexity (GFLOPs). Our big model, HRNet-W484848, achieves the highest 75.575.575.5 AP. Compared to the SimpleBaseline [72] with the same input size, our small and big networks receive 1.21.21.2 and 1.81.81.8 improvements, respectively. With additional data from AI Challenger [70] for training, our single big network can obtain an AP of 77.077.077.0.

test-dev 集结果。 表 2 报告了我们的方法与现有当时最优方法的姿态估计表现。我们的方法明显优于自下而上的方法。另一方面，小网络 HRNet-W323232 达到 74.974.974.9 AP，超过了所有其他自上而下方法，并在模型规模（参数量）与计算复杂度（GFLOPs）上更高效。大模型 HRNet-W484848 取得最高的 75.575.575.5 AP。在相同输入尺寸下，相对 SimpleBaseline [72]，小网络与大网络分别提升 1.21.21.2 与 1.81.81.8。若再加入 AI Challenger [70] 的额外数据训练，单个大网络可达到 77.077.077.0 AP。

4.2 MPII Human Pose Estimation

4.2 MPII 人体姿态估计

Dataset. The MPII Human Pose dataset [2] consists of images taken from a wide-range of real-world activities with full-body pose annotations. There are around 25K25K25K images with 40K40K40K subjects, where there are 12K12K12K subjects for testing and the remaining subjects for the training set. The data augmentation and the training strategy are the same to MS COCO, except that the input size is cropped to 256×256256\times 256256×256 for fair comparison with other methods.

数据集。 MPII Human Pose 数据集 [2] 由覆盖广泛真实活动的图像组成，带全身姿态标注。大约有 25K25K25K 张图像、40K40K40K 个对象，其中约 12K12K12K 个用于测试，其余用于训练集。数据增强与训练策略与 MS COCO 相同，只是为了与其他方法公平比较，输入裁成 256×256256\times 256256×256。

Testing. The testing procedure is almost the same to that in COCO except that we adopt the standard testing strategy to use the provided person boxes instead of detected person boxes. Following [14, 77, 62], a six-scale pyramid testing procedure is performed.

测试。 测试流程与 COCO 几乎相同，不同的是我们采用标准测试策略，使用提供的人体框而不是检测框。沿用 [14, 77, 62]，进行六尺度金字塔测试。

Evaluation metric. The standard metric [2], the PCKh (head-normalized probability of correct keypoint) score, is used. A joint is correct if it falls within αl\alpha lαl pixels of the groundtruth position, where α\alphaα is a constant and lll is the head size that corresponds to 60%60\%60% of the diagonal length of the ground-truth head bounding box. The PCKh@0.5@0.5@0.5 (α=0.5\alpha=0.5α=0.5) score is reported.

评价指标。 采用标准指标 [2]，即 PCKh（以头部归一化的正确关键点概率）分数。若关节落在真值位置的 αl\alpha lαl 个像素以内则判定正确，其中 α\alphaα 为常数，lll 为头部尺寸，对应真值头部包围盒对角线长度的 60%60\%60%。报告 PCKh@0.5@0.5@0.5（α=0.5\alpha=0.5α=0.5）分数。

Results on the test set. Tables 3 and 4 show the PCKh@0.5@0.5@0.5 results, the model size and the GFLOPs of the top-performed methods. We reimplement the SimpleBaseline [72] by using ResNet-152152152 as the backbone with the input size 256×256256\times 256256×256. Our HRNet-W323232 achieves a 92.392.392.3 PKCh@0.50.50.5 score, and outperforms the stacked hourglass approach [40] and its extensions [58, 14, 77, 31, 62]. Our result is the same as the best one [62] among the previously-published results on the leaderboard of Nov. 161616th, 201820182018[^3]. We would like to point out that the approach [62], complementary to our approach, exploits the compositional model to learn the configuration of human bodies and adopts multi-level intermediate supervision, from which our approach can also benefit. We also tested our big network - HRNet-W484848 and obtained the same result 92.392.392.3. The reason might be that the performance in this datatset tends to be saturate.

测试集结果。 表 3 与表 4 给出了表现最好的方法的 PCKh@0.5@0.5@0.5 结果、模型规模与 GFLOPs。我们用 ResNet-152152152 为骨干、输入尺寸 256×256256\times 256256×256 复现了 SimpleBaseline [72]。我们的 HRNet-W323232 达到 92.392.392.3 的 PCKh@0.5@0.5@0.5，超过堆叠 Hourglass [40] 及其扩展 [58, 14, 77, 31, 62]。该结果与 2018 年 11 月 16 日排行榜上此前已发表结果中最好的一项 [62][^3] 持平。我们想指出，方法 [62] 与我们的方法互补：它利用组合模型学习人体配置，并采用多层次中间监督，我们的方法也可以从中受益。我们还测试了大网络 HRNet-W484848，得到同样的 92.392.392.3。原因可能是该数据集上的表现已趋于饱和。

Method Hea Sho Elb Wri Hip Kne Ank Total
Insafutdinov et al. [27] 96.896.896.8 95.295.295.2 89.389.389.3 84.484.484.4 88.488.488.4 83.483.483.4 78.078.078.0 88.588.588.5
Wei et al. [69] 97.897.897.8 95.095.095.0 88.788.788.7 84.084.084.0 88.488.488.4 82.882.882.8 79.479.479.4 88.588.588.5
Bulat et al. [4] 97.997.997.9 95.195.195.1 89.989.989.9 85.385.385.3 89.489.489.4 85.785.785.7 81.781.781.7 89.789.789.7
Newell et al. [40] 98.298.298.2 96.396.396.3 91.291.291.2 87.187.187.1 90.190.190.1 87.487.487.4 83.683.683.6 90.990.990.9
Sun et al. [58] 98.198.198.1 96.296.296.2 91.291.291.2 87.287.287.2 89.889.889.8 87.487.487.4 84.184.184.1 91.091.091.0
Tang et al. [63] 97.497.497.4 96.496.496.4 92.192.192.1 87.787.787.7 90.290.290.2 87.787.787.7 84.384.384.3 91.291.291.2
Ning et al. [44] 98.198.198.1 96.396.396.3 92.292.292.2 87.887.887.8 90.690.690.6 87.687.687.6 82.782.782.7 91.291.291.2
Luvizon et al. [37] 98.198.198.1 96.696.696.6 92.092.092.0 87.587.587.5 90.690.690.6 88.088.088.0 82.782.782.7 91.291.291.2
Chu et al. [14] 98.598.598.5 96.396.396.3 91.991.991.9 88.188.188.1 90.690.690.6 88.088.088.0 85.085.085.0 91.591.591.5
Chou et al. [12] 98.298.298.2 96.896.896.8 92.292.292.2 88.088.088.0 91.391.391.3 89.189.189.1 84.984.984.9 91.891.891.8
Chen et al. [10] 98.198.198.1 96.596.596.5 92.592.592.5 88.588.588.5 90.290.290.2 89.6 86.086.086.0 91.991.991.9
Yang et al. [77] 98.598.598.5 96.796.796.7 92.592.592.5 88.788.788.7 91.191.191.1 88.688.688.6 86.086.086.0 92.092.092.0
Ke et al. [31] 98.598.598.5 96.896.896.8 92.792.792.7 88.488.488.4 90.690.690.6 89.389.389.3 86.3 92.192.192.1
Tang et al. [62] 98.498.498.4 96.9 92.692.692.6 88.788.788.7 91.8 89.489.489.4 86.286.286.2 92.3
SimpleBaseline [72] 98.598.598.5 96.696.696.6 91.991.991.9 87.687.687.6 91.191.191.1 88.188.188.1 84.184.184.1 91.591.591.5
HRNet-W323232 98.6 96.9 92.8 89.0 91.591.591.5 89.089.089.0 85.785.785.7 92.3

Table 3: Performance comparisons on the MPII test set (PCKh@0.5@0.5@0.5).

方法 头 肩 肘 腕 髋 膝 踝 总计
Insafutdinov et al. [27] 96.896.896.8 95.295.295.2 89.389.389.3 84.484.484.4 88.488.488.4 83.483.483.4 78.078.078.0 88.588.588.5
Wei et al. [69] 97.897.897.8 95.095.095.0 88.788.788.7 84.084.084.0 88.488.488.4 82.882.882.8 79.479.479.4 88.588.588.5
Bulat et al. [4] 97.997.997.9 95.195.195.1 89.989.989.9 85.385.385.3 89.489.489.4 85.785.785.7 81.781.781.7 89.789.789.7
Newell et al. [40] 98.298.298.2 96.396.396.3 91.291.291.2 87.187.187.1 90.190.190.1 87.487.487.4 83.683.683.6 90.990.990.9
Sun et al. [58] 98.198.198.1 96.296.296.2 91.291.291.2 87.287.287.2 89.889.889.8 87.487.487.4 84.184.184.1 91.091.091.0
Tang et al. [63] 97.497.497.4 96.496.496.4 92.192.192.1 87.787.787.7 90.290.290.2 87.787.787.7 84.384.384.3 91.291.291.2
Ning et al. [44] 98.198.198.1 96.396.396.3 92.292.292.2 87.887.887.8 90.690.690.6 87.687.687.6 82.782.782.7 91.291.291.2
Luvizon et al. [37] 98.198.198.1 96.696.696.6 92.092.092.0 87.587.587.5 90.690.690.6 88.088.088.0 82.782.782.7 91.291.291.2
Chu et al. [14] 98.598.598.5 96.396.396.3 91.991.991.9 88.188.188.1 90.690.690.6 88.088.088.0 85.085.085.0 91.591.591.5
Chou et al. [12] 98.298.298.2 96.896.896.8 92.292.292.2 88.088.088.0 91.391.391.3 89.189.189.1 84.984.984.9 91.891.891.8
Chen et al. [10] 98.198.198.1 96.596.596.5 92.592.592.5 88.588.588.5 90.290.290.2 89.6 86.086.086.0 91.991.991.9
Yang et al. [77] 98.598.598.5 96.796.796.7 92.592.592.5 88.788.788.7 91.191.191.1 88.688.688.6 86.086.086.0 92.092.092.0
Ke et al. [31] 98.598.598.5 96.896.896.8 92.792.792.7 88.488.488.4 90.690.690.6 89.389.389.3 86.3 92.192.192.1
Tang et al. [62] 98.498.498.4 96.9 92.692.692.6 88.788.788.7 91.8 89.489.489.4 86.286.286.2 92.3
SimpleBaseline [72] 98.598.598.5 96.696.696.6 91.991.991.9 87.687.687.6 91.191.191.1 88.188.188.1 84.184.184.1 91.591.591.5
HRNet-W323232 98.6 96.9 92.8 89.0 91.591.591.5 89.089.089.0 85.785.785.7 92.3

表 3：MPII 测试集上的表现比较（PCKh@0.5@0.5@0.5）。

Method #Params GFLOPs PCKh@0.50.50.5
Insafutdinov et al. [27] 42.642.642.6M 41.241.241.2 88.588.588.5
Newell et al. [40] 25.125.125.1M 19.119.119.1 90.990.990.9
Yang et al. [77] 28.128.128.1M 21.321.321.3 92.092.092.0
Tang et al. [62] 15.515.515.5M 15.615.615.6 92.392.392.3
SimpleBaseline [72] 68.668.668.6M 20.920.920.9 91.591.591.5
HRNet-W323232 28.528.528.5M 9.59.59.5 92.392.392.3

Table 4: #Params and GFLOPs of some top-performed methods reported in Table 3. The GFLOPs is computed with the input size 256×256256\times 256256×256.

方法 参数量 GFLOPs PCKh@0.50.50.5
Insafutdinov et al. [27] 42.642.642.6M 41.241.241.2 88.588.588.5
Newell et al. [40] 25.125.125.1M 19.119.119.1 90.990.990.9
Yang et al. [77] 28.128.128.1M 21.321.321.3 92.092.092.0
Tang et al. [62] 15.515.515.5M 15.615.615.6 92.392.392.3
SimpleBaseline [72] 68.668.668.6M 20.920.920.9 91.591.591.5
HRNet-W323232 28.528.528.5M 9.59.59.5 92.392.392.3

表 4：表 3 中部分表现最好方法的参数量与 GFLOPs。GFLOPs 按输入尺寸 256×256256\times 256256×256 计算。

4.3 Application to Pose Tracking

4.3 应用于姿态跟踪

Dataset. PoseTrack [28] is a large-scale benchmark for human pose estimation and articulated tracking in video. The dataset, based on the raw videos provided by the popular MPII Human Pose dataset, contains 550550550 video sequences with 66,37466,37466,374 frames. The video sequences are split into 292292292, 505050, 208208208 videos for training, validation, and testing, respectively. The length of the training videos ranges between 41−15141-15141−151 frames, and 303030 frames from the center of the video are densely annotated. The number of frames in the validation/testing videos ranges between 65−29865-29865−298 frames. The 303030 frames around the keyframe from the MPII Pose dataset are densely annotated, and afterwards every fourth frame is annotated. In total, this constitutes roughly 23,00023,00023,000 labeled frames and 153,615153,615153,615 pose annotations.

数据集。 PoseTrack [28] 是视频中人体姿态估计与关节跟踪的大规模基准。该数据集基于流行的 MPII Human Pose 数据集提供的原始视频，包含 550550550 段视频、共 66,37466,37466,374 帧。视频被划分为训练、验证与测试，分别有 292292292、505050、208208208 段。训练视频长度为 414141–151151151 帧，并从视频中部稠密标注 303030 帧。验证/测试视频长度为 656565–298298298 帧：围绕 MPII Pose 关键帧的 303030 帧被稠密标注，之后每隔四帧标注一次。总计约 23,00023,00023,000 个标注帧与 153,615153,615153,615 个姿态标注。

Evaluation metric. We evaluate the results from two aspects: frame-wise multi-person pose estimation, and multi-person pose tracking. Pose estimation is evaluated by the mean Average Precision (mAP) as done in [51, 28]. Multi-person pose tracking is evaluated by the multi-object tracking accuracy (MOTA) [38, 28]. Details are given in [28].

评价指标。 我们从两方面评估：逐帧多人姿态估计，以及多人姿态跟踪。姿态估计采用与 [51, 28] 相同的平均精度均值（mAP）。多人姿态跟踪采用多目标跟踪准确率（MOTA） [38, 28]。细节见 [28]。

Training. We train our HRNet-W484848 for single person pose estimation on the PoseTrack201720172017 training set, where the network is initialized by the model pre-trained on COCO dataset. We extract the person box, as the input of our network, from the annotated keypoints in the training frames by extending the bounding box of all the keypoints (for one single person) by 15%15\%15% in length. The training setup, including data augmentation, is almost the same as that for COCO except that the learning schedule is different (as now it is for fine-tuning): the learning rate starts from 1e−41\mathrm{e}{-4}1e−4, drops to 1e−51\mathrm{e}{-5}1e−5 at the 101010th epoch, and to 1e−61\mathrm{e}{-6}1e−6 at the 151515th epoch; the iteration ends within 202020 epochs.

训练。 我们在 PoseTrack201720172017 训练集上训练用于单人姿态估计的 HRNet-W484848，网络用在 COCO 上预训练的模型初始化。作为网络输入的人体框，由训练帧中标注关键点的包围盒在各边延长 15%15\%15% 得到。训练设置（含数据增强）与 COCO 几乎相同，只是学习率日程不同（此时是微调）：学习率从 1e−41\mathrm{e}{-4}1e−4 开始，第 101010 个周期降到 1e−51\mathrm{e}{-5}1e−5，第 151515 个周期降到 1e−61\mathrm{e}{-6}1e−6；在 202020 个周期内结束。

Testing. We follow [72] to track poses across frames. It consists of three steps: person box detection and propagation, human pose estimation, and pose association cross nearby frames. We use the same person box detector as used in SimpleBaseline [72], and propagate the detected box into nearby frames by propagating the predicted keypoints according to the optical flows computed by FlowNet 2.0 [26][^4], followed by non-maximum suppression for box removing. The pose association scheme is based on the object keypoint similarity between the keypoints in one frame and the keypoints propagated from the nearby frame according to the optical flows. The greedy matching algorithm is then used to compute the correspondence between keypoints in nearby frames. More details are given in [72].

测试。 我们沿用 [72] 在帧间跟踪姿态。它包含三步：人体框检测与传播、人体姿态估计，以及邻近帧之间的姿态关联。我们使用与 SimpleBaseline [72] 相同的人体框检测器，并根据 FlowNet 2.0 [26][^4] 计算的光流，把预测关键点传播到邻近帧，从而传播检测框，再以非极大值抑制去除多余框。姿态关联基于一帧中的关键点与按光流从邻近帧传播过来的关键点之间的目标关键点相似度。随后用贪心匹配算法计算邻近帧关键点之间的对应。更多细节见 [72]。

Results on the PoseTrack201720172017 test set. Table 5 reports the results. Our big network - HRNet-W484848 achieves the superior result, a 74.974.974.9 mAP score and a 57.957.957.9 MOTA score. Compared with the second best approach, the FlowTrack in SimpleBaseline [72], that uses ResNet-152152152 as the backbone, our approach gets 0.30.30.3 and 0.10.10.1 points gain in terms of mAP and MOTA, respectively. The superiority over the FlowTrack [72] is consistent to that on the COCO keypoint detection and MPII human pose estimation datasets. This further implies the effectiveness of our pose estimation network.

PoseTrack201720172017 测试集结果。 表 5 报告了结果。我们的大网络 HRNet-W484848 取得更优结果：74.974.974.9 mAP 与 57.957.957.9 MOTA。相对第二好的方法、以 ResNet-152152152 为骨干的 SimpleBaseline 中的 FlowTrack [72]，我们的方法在 mAP 与 MOTA 上分别提升 0.30.30.3 与 0.10.10.1 个点。相对 FlowTrack [72] 的优势与 COCO 关键点检测、MPII 人体姿态估计上的结论一致。这进一步说明了我们的姿态估计网络是有效的。

Entry Additional training Data mAP MOTA
ML-LAB [84] COCO+MPII-Pose 70.370.370.3 41.841.841.8
SOPT-PT [53] COCO+MPII-Pose 58.258.258.2 42.042.042.0
BUTD2 [29] COCO 59.259.259.2 50.650.650.6
MVIG [53] COCO+MPII-Pose 63.263.263.2 50.750.750.7
PoseFlow [53] COCO+MPII-Pose 63.063.063.0 51.051.051.0
ProTracker [19] COCO 59.659.659.6 51.851.851.8
HMPT [53] COCO+MPII-Pose 63.763.763.7 51.951.951.9
JointFlow [15] COCO 63.663.663.6 53.153.153.1
STAF [53] COCO+MPII-Pose 70.370.370.3 53.853.853.8
MIPAL [53] COCO 68.868.868.8 54.554.554.5
FlowTrack [72] COCO 74.674.674.6 57.857.857.8
HRNet-W484848 COCO 74.9 57.9

Table 5: Results of pose tracking on the PoseTrack201720172017 test set.

条目 额外训练数据 mAP MOTA
ML-LAB [84] COCO+MPII-Pose 70.370.370.3 41.841.841.8
SOPT-PT [53] COCO+MPII-Pose 58.258.258.2 42.042.042.0
BUTD2 [29] COCO 59.259.259.2 50.650.650.6
MVIG [53] COCO+MPII-Pose 63.263.263.2 50.750.750.7
PoseFlow [53] COCO+MPII-Pose 63.063.063.0 51.051.051.0
ProTracker [19] COCO 59.659.659.6 51.851.851.8
HMPT [53] COCO+MPII-Pose 63.763.763.7 51.951.951.9
JointFlow [15] COCO 63.663.663.6 53.153.153.1
STAF [53] COCO+MPII-Pose 70.370.370.3 53.853.853.8
MIPAL [53] COCO 68.868.868.8 54.554.554.5
FlowTrack [72] COCO 74.674.674.6 57.857.857.8
HRNet-W484848 COCO 74.9 57.9

表 5：PoseTrack201720172017 测试集上的姿态跟踪结果。

4.4 Ablation Study

4.4 消融实验

We study the effect of each component in our approach on the COCO keypoint detection dataset. All results are obtained over the input size of 256×192256\times 192256×192 except the study about the effect of the input size.

我们在 COCO 关键点检测数据集上研究本方法各组成部分的作用。除关于输入尺寸的研究外，所有结果都在输入尺寸 256×192256\times 192256×192 上获得。

Method Final exchange Int. exchange across Int. exchange within AP⁡\operatorname{AP}AP
(a) ✓ 70.870.870.8
(b) ✓ ✓ 71.971.971.9
(c) ✓ ✓ ✓ 73.473.473.4

Table 6: Ablation study of exchange units that are used in repeated multi-scale fusion. Int. exchange across = intermediate exchange across stages, Int. exchange within = intermediate exchange within stages.

方法 最终交换 跨阶段中间交换 阶段内中间交换 AP⁡\operatorname{AP}AP
(a) ✓ 70.870.870.8
(b) ✓ ✓ 71.971.971.9
(c) ✓ ✓ ✓ 73.473.473.4

表 6：用于反复多尺度融合的交换单元消融。跨阶段中间交换 = 阶段之间的中间交换，阶段内中间交换 = 同一阶段内部的中间交换。

Figure 4: Qualitative results of some example images in the MPII (top) and COCO (bottom) datasets: containing viewpoint and appearance change, occlusion, multiple persons, and common imaging artifacts.

图 4：MPII（上）与 COCO（下）数据集中一些示例图像的定性结果：包含视角与外观变化、遮挡、多人以及常见成像伪影。

Figure 5: Ablation study of high and low representations. 1×1\times1×, 2×2\times2×, 4×4\times4× correspond to the representations of the high, medium, low resolutions, respectively.

图 5：高、低分辨率表示的消融。1×1\times1×、2×2\times2×、4×4\times4× 分别对应高、中、低分辨率的表示。

Repeated multi-scale fusion. We empirically analyze the effect of the repeated multi-scale fusion. We study three variants of our network. (a) W/o intermediate exchange units (111 fusion): There is no exchange between multi-resolution subnetworks except the last exchange unit. (b) W/ across-stage exchange units only (333 fusions): There is no exchange between parallel subnetworks within each stage. (c) W/ both across-stage and within-stage exchange units (totally 888 fusion): This is our proposed method. All the networks are trained from scratch. The results on the COCO validation set given in Table 6 show that the multi-scale fusion is helpful and more fusions lead to better performance.

反复多尺度融合。 我们实证分析反复多尺度融合的作用。我们研究网络的三个变体。(a) 无中间交换单元（111 次融合）：除最后一个交换单元外，多分辨率子网络之间没有交换。(b) 仅有跨阶段交换单元（333 次融合）：每个阶段内部的并行子网络之间没有交换。(c) 同时具有跨阶段与阶段内交换单元（共 888 次融合）：这就是我们提出的方法。所有网络都从零训练。表 6 中 COCO 验证集上的结果表明，多尺度融合是有帮助的，融合次数越多表现越好。

Resolution maintenance. We study the performance of a variant of the HRNet: all the four high-to-low resolution subnetworks are added at the beginning and the depth are the same; the fusion schemes are the same to ours. Both our HRNet-W323232 and the variant (with similar #Params and GFLOPs) are trained from scratch and tested on the COCO validation set. The variant achieves an AP of 72.572.572.5, which is lower than the 73.473.473.4 AP of our small net, HRNet-W323232. We believe that the reason is that the low-level features extracted from the early stages over the low-resolution subnetworks are less helpful. In addition, the simple high-resolution network of similar parameter and computation complexities without low-resolution parallel subnetworks shows much lower performance .

分辨率保持。 我们研究 HRNet 的一个变体：四路高低分辨率子网络在一开始就全部加入，深度相同；融合方案与我们的相同。HRNet-W323232 与该变体（参数量与 GFLOPs 相近）都从零训练，并在 COCO 验证集上测试。该变体达到 72.572.572.5 AP，低于我们的小网络 HRNet-W323232 的 73.473.473.4 AP。我们认为原因是：早期阶段在低分辨率子网络上提取的低层特征帮助较小。此外，在参数量与计算复杂度相近、但没有并行低分辨率子网络的简单高分辨率网络上，表现要差得多。

Representation resolution. We study how the representation resolution affects the pose estimation performance from two aspects: check the quality of the heatmap estimated from the feature maps of each resolution from high to low, and study how the input size affects the quality.

表示分辨率。 我们从两方面研究表示分辨率如何影响姿态估计表现：检查由高到低各分辨率特征图估计出的热图质量，并研究输入尺寸如何影响质量。

We train our small and big networks initialized by the model pretrained for the ImageNet classification. Our network outputs four response maps from high-to-low solutions. The quality of heatmap prediction over the lowest-resolution response map is too low and the AP score is below 101010 points. The AP scores over the other three maps are reported in Figure 5. The comparison implies that the resolution does impact the keypoint prediction quality.

我们用 ImageNet 分类预训练模型初始化，训练小网络与大网络。网络由高到低输出四张响应图。在最低分辨率响应图上预测热图的质量过低，AP 低于 101010 个点。其余三张图上的 AP 见图 5。比较表明，分辨率确实会影响关键点预测质量。

Figure 6: Illustrating how the performances of our HRNet and SimpleBaseline [72] are affected by the input size.

图 6：展示输入尺寸如何影响我们的 HRNet 与 SimpleBaseline [72] 的表现。

Figure 6 shows how the input image size affects the performance in comparison with SimpleBaseline (ResNet-50) [72]. We can find that the improvement for the smaller input size is more significant than the larger input size, e.g., the improvement is 4.04.04.0 points for 256×192256\times 192256×192 and 6.36.36.3 points for 128×96128\times 96128×96. The reason is that we maintain the high resolution through the whole process. This implies that our approach is more advantageous in the real applications where the computation cost is also an important factor. On the other hand, our approach with the input size 256×192256\times 192256×192 outperforms the SimpleBaseline [72] with the large input size of 384×288384\times 288384×288.

图 6 展示了与 SimpleBaseline（ResNet-50） [72] 相比，输入图像尺寸如何影响表现。可以看到，较小输入尺寸上的提升比更大输入尺寸更显著，例如 256×192256\times 192256×192 上提升 4.04.04.0 个点，128×96128\times 96128×96 上提升 6.36.36.3 个点。原因是我们在整个过程中保持高分辨率。这意味着在计算代价同样重要的实际应用中，我们的方法更有优势。另一方面，输入尺寸为 256×192256\times 192256×192 的本方法超过了输入尺寸为 384×288384\times 288384×288 的 SimpleBaseline [72]。

5 Conclusion and Future Works

5 结论与未来工作

In this paper, we present a high-resolution network for human pose estimation, yielding accurate and spatially-precise keypoint heatmaps. The success stems from two aspects: (i) maintain the high resolution through the whole process without the need of recovering the high resolution; and (ii) fuse multi-resolution representations repeatedly, rendering reliable high-resolution representations.

本文提出一种用于人体姿态估计的高分辨率网络，能够得到准确且空间定位精细的关键点热图。成功来自两方面：(i) 在整个过程中保持高分辨率，无需再去恢复高分辨率；(ii) 反复融合多分辨率表示，从而得到可靠的高分辨率表示。

The future works include the applications to other dense prediction tasks, e.g., semantic segmentation, object detection, face alignment, image translation, as well as the investigation on aggregating multi-resolution representations in a less light way. All them are available at https://jingdongwang2017.github.io/Projects/HRNet/index.html.

未来工作包括应用于其他稠密预测任务，例如语义分割、目标检测、人脸对齐、图像翻译，以及以不那么轻量的方式聚合多分辨率表示。相关内容见 https://jingdongwang2017.github.io/Projects/HRNet/index.html。

Appendix

附录

Results on the MPII Validation Set

MPII 验证集上的结果

We provide the results on the MPII validation set [2]. Our models are trained on a subset of MPII training set and evaluate on a heldout validation set of 2975 images. The training procedure is the same to that for training on the whole MPII training set. The heatmap is computed as the average of the heatmaps of the original and flipped images for testing. Following [77, 62], we also perform six-scale pyramid testing procedure (multi-scale testing). The results are shown in Table 7.

我们给出 MPII 验证集 [2] 上的结果。模型在 MPII 训练集的一个子集上训练，并在留出的 297529752975 张图像验证集上评估。训练流程与在整个 MPII 训练集上训练时相同。测试时热图取原图与翻转图像热图的平均。沿用 [77, 62]，我们也进行六尺度金字塔测试（多尺度测试）。结果见表 7。

Method Hea Sho Elb Wri Hip Kne Ank Total
Single-scale testing
Newell et al. [40] 96.596.596.5 96.096.096.0 90.390.390.3 85.485.485.4 88.888.888.8 85.085.085.0 81.981.981.9 89.289.289.2
Yang et al. [77] 96.8{96.8}96.8 96.0{96.0}96.0 90.490.490.4 86.086.086.0 89.589.589.5 85.285.285.2 82.382.382.3 89.689.689.6
Tang et al. [62] 95.695.695.6 95.995.995.9 90.7{90.7}90.7 86.5{86.5}86.5 89.9{89.9}89.9 86.6{86.6}86.6 82.5{82.5}82.5 89.889.889.8
SimpleBaseline [72] 97.097.097.0 95.995.995.9 90.390.390.3 85.085.085.0 89.289.289.2 85.385.385.3 81.381.381.3 89.689.689.6
HRNet-W323232 97.197.197.1 95.995.995.9 90.390.390.3 86.4{86.4}86.4 89.189.189.1 87.187.187.1 83.3{83.3}83.3 90.3
Multi-scale testing
Newell et al. [40] 97.197.197.1 96.196.196.1 90.890.890.8 86.286.286.2 89.989.989.9 85.985.985.9 83.583.583.5 90.090.090.0
Yang et al. [77] 97.497.497.4 96.2{96.2}96.2 91.1{91.1}91.1 86.986.986.9 90.190.190.1 86.086.086.0 83.983.983.9 90.390.390.3
Tang et al. [62] 97.497.497.4 96.2{96.2}96.2 91.091.091.0 86.986.986.9 90.6{90.6}90.6 86.886.886.8 84.5{84.5}84.5 90.590.590.5
SimpleBaseline [72] 97.597.597.5 96.196.196.1 90.590.590.5 85.485.485.4 90.190.190.1 85.785.785.7 82.382.382.3 90.190.190.1
HRNet-W323232 97.7{97.7}97.7 96.396.396.3 90.990.990.9 86.7{86.7}86.7 89.789.789.7 87.4{87.4}87.4 84.184.184.1 90.8

Table 7: Performance comparisons on the MPII validation set (PCKh@0.5@0.5@0.5).

方法 头 肩 肘 腕 髋 膝 踝 总计
单尺度测试
Newell et al. [40] 96.596.596.5 96.096.096.0 90.390.390.3 85.485.485.4 88.888.888.8 85.085.085.0 81.981.981.9 89.289.289.2
Yang et al. [77] 96.8{96.8}96.8 96.0{96.0}96.0 90.490.490.4 86.086.086.0 89.589.589.5 85.285.285.2 82.382.382.3 89.689.689.6
Tang et al. [62] 95.695.695.6 95.995.995.9 90.7{90.7}90.7 86.5{86.5}86.5 89.9{89.9}89.9 86.6{86.6}86.6 82.5{82.5}82.5 89.889.889.8
SimpleBaseline [72] 97.097.097.0 95.995.995.9 90.390.390.3 85.085.085.0 89.289.289.2 85.385.385.3 81.381.381.3 89.689.689.6
HRNet-W323232 97.197.197.1 95.995.995.9 90.390.390.3 86.4{86.4}86.4 89.189.189.1 87.187.187.1 83.3{83.3}83.3 90.3
多尺度测试
Newell et al. [40] 97.197.197.1 96.196.196.1 90.890.890.8 86.286.286.2 89.989.989.9 85.985.985.9 83.583.583.5 90.090.090.0
Yang et al. [77] 97.497.497.4 96.2{96.2}96.2 91.1{91.1}91.1 86.986.986.9 90.190.190.1 86.086.086.0 83.983.983.9 90.390.390.3
Tang et al. [62] 97.497.497.4 96.2{96.2}96.2 91.091.091.0 86.986.986.9 90.6{90.6}90.6 86.886.886.8 84.5{84.5}84.5 90.590.590.5
SimpleBaseline [72] 97.597.597.5 96.196.196.1 90.590.590.5 85.485.485.4 90.190.190.1 85.785.785.7 82.382.382.3 90.190.190.1
HRNet-W323232 97.7{97.7}97.7 96.396.396.3 90.990.990.9 86.7{86.7}86.7 89.789.789.7 87.4{87.4}87.4 84.184.184.1 90.8

表 7：MPII 验证集上的表现比较（PCKh@0.5@0.5@0.5）。

More Results on the PoseTrack Dataset

PoseTrack 数据集上的更多结果

We provide the results for all the keypoints on the PoseTrack dataset [1]. Table 8 shows the multi-person pose estimation performance on the PoseTrack201720172017 dataset. Our HRNet-W484848 achieves 77.3 and 74.9 points mAP on the validation and test setss, and outperforms previous state-of-the-art method [72] by 0.6 points and 0.3 points respectively. We provide more detailed results of multi-person pose tracking performance on the PoseTrack2017 test set as a supplement of the results reported in the paper, shown in Table 9.

我们给出 PoseTrack 数据集 [1] 上全部关键点的结果。表 8 展示了 PoseTrack201720172017 上的多人姿态估计表现。HRNet-W484848 在验证集与测试集上分别达到 77.377.377.3 与 74.974.974.9 mAP，相对此前最优方法 [72] 分别高出 0.60.60.6 与 0.30.30.3 个点。作为文中结果的补充，表 9 给出了 PoseTrack2017 测试集上更详细的多人姿态跟踪结果。

Method Head Sho. Elb. Wri. Hip Knee Ank. Total
PoseTrack validation set
Girdhar et al. [19] 67.567.567.5 70.270.270.2 62.062.062.0 51.751.751.7 60.760.760.7 58.758.758.7 49.849.849.8 60.660.660.6
Xiu et al. [75] 66.766.766.7 73.373.373.3 68.368.368.3 61.161.161.1 67.567.567.5 67.067.067.0 61.361.361.3 66.566.566.5
Bin et al. [72] 81.781.781.7 83.483.483.4 80.080.080.0 72.472.472.4 75.375.375.3 74.874.874.8 67.167.167.1 76.7{76.7}76.7
HRNet-W484848 82.1{82.1}82.1 83.6{83.6}83.6 80.4{80.4}80.4 73.3{73.3}73.3 75.5{75.5}75.5 75.3{75.3}75.3 68.5{68.5}68.5 77.3
PoseTrack test set
Girdhar et al.* [19] −-− −-− −-− −-− −-− −-− −-− 59.659.659.6
Xiu et al. [75] 64.964.964.9 67.567.567.5 65.065.065.0 59.059.059.0 62.562.562.5 62.862.862.8 57.957.957.9 63.063.063.0
Bin et al.* [72] 80.1{80.1}80.1 80.2{80.2}80.2 76.9{76.9}76.9 71.571.571.5 72.572.572.5 72.472.472.4 65.765.765.7 74.6{74.6}74.6
HRNet-W484848* 80.1{80.1}80.1 80.2{80.2}80.2 76.9{76.9}76.9 72.0{72.0}72.0 73.4{73.4}73.4 72.5{72.5}72.5 67.0{67.0}67.0 74.9

Table 8: Multi-person pose estimation performance (MAP) on the PoseTrack2017 dataset. “*” means models trained on thr train+valid set.

方法 头 肩 肘 腕 髋 膝 踝 总计
PoseTrack 验证集
Girdhar et al. [19] 67.567.567.5 70.270.270.2 62.062.062.0 51.751.751.7 60.760.760.7 58.758.758.7 49.849.849.8 60.660.660.6
Xiu et al. [75] 66.766.766.7 73.373.373.3 68.368.368.3 61.161.161.1 67.567.567.5 67.067.067.0 61.361.361.3 66.566.566.5
Bin et al. [72] 81.781.781.7 83.483.483.4 80.080.080.0 72.472.472.4 75.375.375.3 74.874.874.8 67.167.167.1 76.7{76.7}76.7
HRNet-W484848 82.1{82.1}82.1 83.6{83.6}83.6 80.4{80.4}80.4 73.3{73.3}73.3 75.5{75.5}75.5 75.3{75.3}75.3 68.5{68.5}68.5 77.3
PoseTrack 测试集
Girdhar et al.* [19] −-− −-− −-− −-− −-− −-− −-− 59.659.659.6
Xiu et al. [75] 64.964.964.9 67.567.567.5 65.065.065.0 59.059.059.0 62.562.562.5 62.862.862.8 57.957.957.9 63.063.063.0
Bin et al.* [72] 80.1{80.1}80.1 80.2{80.2}80.2 76.9{76.9}76.9 71.571.571.5 72.572.572.5 72.472.472.4 65.765.765.7 74.6{74.6}74.6
HRNet-W484848* 80.1{80.1}80.1 80.2{80.2}80.2 76.9{76.9}76.9 72.0{72.0}72.0 73.4{73.4}73.4 72.5{72.5}72.5 67.0{67.0}67.0 74.9

表 8：PoseTrack2017 数据集上的多人姿态估计表现（MAP）。“*”表示在训练集+验证集上训练的模型。

Method Head Sho. Elb. Wri Hip Knee Ank. Total
Girdhar et al.* [19] −-− −-− −-− −-− −-− −-− −-− 51.851.851.8
Xiu et al. [75] 52.052.052.0 57.457.457.4 52.852.852.8 46.646.646.6 51.051.051.0 51.251.251.2 45.345.345.3 51.051.051.0
Xiao et al.* [72] 67.367.367.3 68.568.568.5 52.352.352.3 49.349.349.3 56.856.856.8 57.257.257.2 48.648.648.6 57.8{57.8}57.8
HRNet-W484848* 67.167.167.1 68.968.968.9 52.252.252.2 49.649.649.6 57.757.757.7 57.057.057.0 48.548.548.5 57.9

Table 9: Multi-person pose tracking performance (MOTA) on the PoseTrack2017 test set.“*” means models trained on the train+validation set.

方法 头 肩 肘 腕 髋 膝 踝 总计
Girdhar et al.* [19] −-− −-− −-− −-− −-− −-− −-− 51.851.851.8
Xiu et al. [75] 52.052.052.0 57.457.457.4 52.852.852.8 46.646.646.6 51.051.051.0 51.251.251.2 45.345.345.3 51.051.051.0
Xiao et al.* [72] 67.367.367.3 68.568.568.5 52.352.352.3 49.349.349.3 56.856.856.8 57.257.257.2 48.648.648.6 57.8{57.8}57.8
HRNet-W484848* 67.167.167.1 68.968.968.9 52.252.252.2 49.649.649.6 57.757.757.7 57.057.057.0 48.548.548.5 57.9

表 9：PoseTrack2017 测试集上的多人姿态跟踪表现（MOTA）。“*”表示在训练集+验证集上训练的模型。

Results on the ImageNet Validation Set

ImageNet 验证集上的结果

We apply our networks to image classification task. The models are trained and evaluated on the ImageNet 2013 classification dataset [54]. We train our models for 100 epochs with a batch size of 256. The initial learning rate is set to 0.1 and is reduced by 10 times at epoch 30, 60 and 90. Our models can achieve comparable performance as those networks specifically designed for image classification, such as ResNet [22]. Our HRNet-W323232 has a single-model top-5 validation error of 6.5% and has a single-model top-1 validation error of 22.7% with the single-crop testing. Our HRNet-W484848 gets better performance: 6.1% top-5 errors and 22.1% top-1 error. We use the models trained on the ImageNet dataset to initialize the parameters of our pose estimation networks.

我们把网络应用到图像分类任务。模型在 ImageNet 2013 分类数据集 [54] 上训练与评估。我们训练 100100100 个周期，批次大小为 256256256。初始学习率为 0.10.10.1，并在第 303030、606060 与 909090 个周期各降为原来的十分之一。我们的模型可以达到与专为图像分类设计的网络（如 ResNet [22]）相当的表现。HRNet-W323232 在单模型、单裁剪测试下的 top-5 验证错误率为 6.5%6.5\%6.5%，top-1 验证错误率为 22.7%22.7\%22.7%。HRNet-W484848 更好：top-5 错误率 6.1%6.1\%6.1%，top-1 错误率 22.1%22.1\%22.1%。我们用在 ImageNet 上训练的模型来初始化姿态估计网络的参数。

Acknowledgements. The authors thank Dianqi Li and Lei Zhang for helpful discussions.

致谢。 作者感谢 Dianqi Li 与 Lei Zhang 的有益讨论。

References

参考文献

[1] M. Andriluka, U. Iqbal, A. Milan, E. Insafutdinov, L. Pishchulin, J. Gall, and B. Schiele. Posetrack: A benchmark for human pose estimation and tracking. In CVPR , pages 5167–5176, 2018.

[2] M. Andriluka, L. Pishchulin, P. V. Gehler, and B. Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR , pages 3686–3693, 2014.

[3] V. Belagiannis and A. Zisserman. Recurrent human pose estimgation. In FG , pages 468–475, 2017.

[4] A. Bulat and G. Tzimiropoulos. Human pose estimation via convolutional part heatmap regression. In ECCV , volume 9911 of Lecture Notes in Computer Science , pages 717–732. Springer, 2016.

[5] Z. Cai, Q. Fan, R. S. Feris, and N. Vasconcelos. A unified multi-scale deep convolutional neural network for fast object detection. In ECCV , pages 354–370, 2016.

[6] Z. Cao, T. Simon, S. Wei, and Y. Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR , pages 1302–1310, 2017.

[7] J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik. Human pose estimation with iterative error feedback. In CVPR , pages 4733–4742, 2016.

[8] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. , 40(4):834–848, 2018.

[9] X. Chen and A. L. Yuille. Articulated pose estimation by a graphical model with image dependent pairwise relations. In NIPS , pages 1736–1744, 2014.

[10] Y. Chen, C. Shen, X. Wei, L. Liu, and J. Yang. Adversarial posenet: A structure-aware convolutional network for human pose estimation. In ICCV , pages 1221–1230, 2017.

[11] Y. Chen, Z. Wang, Y. Peng, Z. Zhang, G. Yu, and J. Sun. Cascaded pyramid network for multi-person pose estimation. CoRR , abs/1711.07319, 2017.

[12] C. Chou, J. Chien, and H. Chen. Self adversarial training for human pose estimation. CoRR , abs/1707.02439, 2017.

[13] X. Chu, W. Ouyang, H. Li, and X. Wang. Structured feature learning for pose estimation. In CVPR , pages 4715–4723, 2016.

[14] X. Chu, W. Yang, W. Ouyang, C. Ma, A. L. Yuille, and X. Wang. Multi-context attention for human pose estimation. In CVPR , pages 5669–5678, 2017.

[15] A. Doering, U. Iqbal, and J. Gall. Joint flow: Temporal flow fields for multi person tracking, 2018.

[16] X. Fan, K. Zheng, Y. Lin, and S. Wang. Combining local appearance and holistic view: Dual-source deep neural networks for human pose estimation. In CVPR , pages 1347–1355, 2015.

[17] H. Fang, S. Xie, Y. Tai, and C. Lu. RMPE: regional multi-person pose estimation. In ICCV , pages 2353–2362, 2017.

[18] D. Fourure, R. Emonet, É. Fromont, D. Muselet, A. Trémeau, and C. Wolf. Residual conv-deconv grid network for semantic segmentation. In British Machine Vision Conference 2017, BMVC 2017, London, UK, September 4-7, 2017 , 2017.

[19] R. Girdhar, G. Gkioxari, L. Torresani, M. Paluri, and D. Tran. Detect-and-track: Efficient pose estimation in videos. In CVPR , pages 350–359, 2018.

[20] G. Gkioxari, A. Toshev, and N. Jaitly. Chained predictions using convolutional neural networks. In ECCV , pages 728–743, 2016.

[21] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick. Mask R-CNN. In ICCV , pages 2980–2988, 2017.

[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , pages 770–778, 2016.

[23] P. Hu and D. Ramanan. Bottom-up and top-down reasoning with hierarchical rectified gaussians. In CVPR , pages 5600–5609, 2016.

[24] G. Huang, D. Chen, T. Li, F. Wu, L. van der Maaten, and K. Q. Weinberger. Multi-scale dense convolutional networks for efficient prediction. CoRR , abs/1703.09844, 2017.

[25] S. Huang, M. Gong, and D. Tao. A coarse-fine network for keypoint localization. In ICCV , pages 3047–3056. IEEE Computer Society, 2017.

[26] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR , pages 1647–1655, 2017.

[27] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, and B. Schiele. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In ECCV , pages 34–50, 2016.

[28] U. Iqbal, A. Milan, and J. Gall. Posetrack: Joint multi-person pose estimation and tracking. In CVPR , pages 4654–4663, 2017.

[29] S. Jin, X. Ma, Z. Han, Y. Wu, W. Yang, W. Liu, C. Qian, and W. Ouyang. Towards multi-person pose tracking: Bottom-up and top-down methods. In ICCV PoseTrack Workshop , 2017.

[30] A. Kanazawa, A. Sharma, and D. W. Jacobs. Locally scale-invariant convolutional neural networks. CoRR , abs/1412.5104, 2014.

[31] L. Ke, M. Chang, H. Qi, and S. Lyu. Multi-scale structure-aware network for human pose estimation. CoRR , abs/1803.09894, 2018.

[32] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014.

[33] M. Kocabas, S. Karagoz, and E. Akbas. Multiposenet: Fast multi-person pose estimation using pose residual network. In ECCV , volume 11215 of Lecture Notes in Computer Science , pages 437–453. Springer, 2018.

[34] C. Lee, S. Xie, P. W. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. In AISTATS , 2015.

[35] I. Lifshitz, E. Fetaya, and S. Ullman. Human pose estimation using deep consensus voting. In ECCV , pages 246–260, 2016.

[36] T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: common objects in context. In ECCV , pages 740–755, 2014.

[37] D. C. Luvizon, H. Tabia, and D. Picard. Human pose regression by combining indirect part detection and contextual information. CoRR , abs/1710.02322, 2017.

[38] A. Milan, L. Leal-Taixé, I. D. Reid, S. Roth, and K. Schindler. MOT16: A benchmark for multi-object tracking. CoRR , abs/1603.00831, 2016.

[39] A. Newell, Z. Huang, and J. Deng. Associative embedding: End-to-end learning for joint detection and grouping. In NIPS , pages 2274–2284, 2017.

[40] A. Newell, K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In ECCV , pages 483–499, 2016.

[41] X. Nie, J. Feng, J. Xing, and S. Yan. Pose partition networks for multi-person pose estimation. In ECCV , September 2018.

[42] X. Nie, J. Feng, and S. Yan. Mutual learning to adapt for joint human parsing and pose estimation. In ECCV , September.

[43] X. Nie, J. Feng, Y. Zuo, and S. Yan. Human pose estimation with parsing induced learner. In CVPR , June 2018.

[44] G. Ning, Z. Zhang, and Z. He. Knowledge-guided deep fractal neural networks for human pose estimation. IEEE Trans. Multimedia , 20(5):1246–1259, 2018.

[45] W. Ouyang, X. Chu, and X. Wang. Multi-source deep learning for human pose estimation. In CVPR , pages 2337–2344, 2014.

[46] G. Papandreou, T. Zhu, L.-C. Chen, S. Gidaris, J. Tompson, and K. Murphy. Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In ECCV , September 2018.

[47] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy. Towards accurate multi-person pose estimation in the wild. In CVPR , pages 3711–3719, 2017.

[48] X. Peng, Z. Tang, F. Yang, R. S. Feris, and D. Metaxas. Jointly optimize data augmentation and network training: Adversarial data augmentation in human pose estimation. In CVPR , June 2018.

[49] T. Pfister, J. Charles, and A. Zisserman. Flowing convnets for human pose estimation in videos. In ICCV , pages 1913–1921, 2015.

[50] L. Pishchulin, M. Andriluka, P. V. Gehler, and B. Schiele. Poselet conditioned pictorial structures. In CVPR , pages 588–595, 2013.

[51] L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. V. Gehler, and B. Schiele. Deepcut: Joint subset partition and labeling for multi person pose estimation. In CVPR , pages 4929–4937, 2016.

[52] T. Pohlen, A. Hermans, M. Mathias, and B. Leibe. Full-resolution residual networks for semantic segmentation in street scenes. In CVPR , 2017.

[53] PoseTrack. PoseTrack Leader Board. https://posetrack.net/leaderboard.php .

[54] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 115(3):211–252, 2015.

[55] M. Samy, K. Amer, K. Eissa, M. Shaker, and M. ElHelw. Nu-net: Deep residual wide field of view convolutional neural network for semantic segmentation. In CVPRW , June 2018.

[56] S. Saxena and J. Verbeek. Convolutional neural fabrics. In NIPS , pages 4053–4061, 2016.

[57] T. Sekii. Pose proposal networks. In ECCV , September 2018.

[58] K. Sun, C. Lan, J. Xing, W. Zeng, D. Liu, and J. Wang. Human pose estimation using global and local normalization. In ICCV , pages 5600–5608, 2017.

[59] K. Sun, M. Li, D. Liu, and J. Wang. IGCV3: interleaved low-rank group convolutions for efficient deep neural networks. In BMVC , page 101. BMVA Press, 2018.

[60] X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei. Integral human pose regression. In ECCV , pages 536–553, 2018.

[61] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR , pages 1–9, 2015.

[62] W. Tang, P. Yu, and Y. Wu. Deeply learned compositional models for human pose estimation. In ECCV , September 2018.

[63] Z. Tang, X. Peng, S. Geng, L. Wu, S. Zhang, and D. N. Metaxas. Quantized densely connected u-nets for efficient landmark localization. In ECCV , pages 348–364, 2018.

[64] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler. Efficient object localization using convolutional networks. In CVPR , pages 648–656, 2015.

[65] J. J. Tompson, A. Jain, Y. LeCun, and C. Bregler. Joint training of a convolutional network and a graphical model for human pose estimation. In NIPS , pages 1799–1807, 2014.

[66] A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. In CVPR , pages 1653–1660, 2014.

[67] J. Wang, Z. Wei, T. Zhang, and W. Zeng. Deeply-fused nets. CoRR , abs/1605.07716, 2016.

[68] Z. Wang, W. Li, B. Yin, Q. Peng, T. Xiao, Y. Du, Z. Li, X. Zhang, G. Yu, and J. Sun. Mscoco keypoints challenge 2018. In Joint Recognition Challenge Workshop at ECCV 2018 , 2018.

[69] S. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh. Convolutional pose machines. In CVPR , pages 4724–4732, 2016.

[70] J. Wu, H. Zheng, B. Zhao, Y. Li, B. Yan, R. Liang, W. Wang, S. Zhou, G. Lin, Y. Fu, et al. Ai challenger: A large-scale dataset for going deeper in image understanding. arXiv preprint arXiv:1711.06475 , 2017.

[71] F. Xia, P. Wang, X. Chen, and A. L. Yuille. Joint multi-person pose estimation and semantic part segmentation. In CVPR , pages 6080–6089, 2017.

[72] B. Xiao, H. Wu, and Y. Wei. Simple baselines for human pose estimation and tracking. In ECCV , pages 472–487, 2018.

[73] G. Xie, J. Wang, T. Zhang, J. Lai, R. Hong, and G. Qi. Interleaved structured sparse convolutional neural networks. In CVPR , pages 8847–8856. IEEE Computer Society, 2018.

[74] S. Xie and Z. Tu. Holistically-nested edge detection. In ICCV , pages 1395–1403, 2015.

[75] Y. Xiu, J. Li, H. Wang, Y. Fang, and C. Lu. Pose flow: Efficient online pose tracking. In BMVC , page 53, 2018.

[76] Y. Xu, T. Xiao, J. Zhang, K. Yang, and Z. Zhang. Scale-invariant convolutional neural networks. CoRR , abs/1411.6369, 2014.

[77] W. Yang, S. Li, W. Ouyang, H. Li, and X. Wang. Learning feature pyramids for human pose estimation. In ICCV , pages 1290–1299, 2017.

[78] W. Yang, W. Ouyang, H. Li, and X. Wang. End-to-end learning of deformable mixture of parts and deep convolutional neural networks for human pose estimation. In CVPR , pages 3073–3082, 2016.

[79] Y. Yang and D. Ramanan. Articulated pose estimation with flexible mixtures-of-parts. In CVPR , pages 1385–1392, 2011.

[80] T. Zhang, G. Qi, B. Xiao, and J. Wang. Interleaved group convolutions. In ICCV , pages 4383–4392, 2017.

[81] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In CVPR , pages 6230–6239, 2017.

[82] L. Zhao, M. Li, D. Meng, X. Li, Z. Zhang, Y. Zhuang, Z. Tu, and J. Wang. Deep convolutional neural networks with merge-and-run mappings. In IJCAI , pages 3170–3176, 2018.

[83] Y. Zhou, X. Hu, and B. Zhang. Interlinked convolutional neural networks for face parsing. In ISNN , pages 222–231, 2015.

[84] X. Zhu, Y. Jiang, and Z. Luo. Multi-person pose estimation for posetrack with enhanced part affinity fields. In ICCV PoseTrack Workshop , 2017.