【作者】 张哲；

【导师】 秦文虎；

【作者基本信息】 东南大学 ， 仪器科学与技术， 2021， 博士

【摘要】 “以人为中心”的计算机视觉研究愈发成为人工智能领域的重要研究方向,其所面临的挑战首先在于要让计算机能够理解人。因此,准确地识别人体的姿态和动作是必要的条件。但由于视觉遮挡问题难以解决,人体姿态估计仍是非常具有挑战性的问题。本文以多视角为基本场景,重点从设备、数据和方法三个角度,开展遮挡环境下多视角三维人体姿态估计的研究。本文主要研究工作包括:(1)针对多相机系统构建与应用过程中面临的多相机时间不同步问题,提出一种基于二维人体关键点的多相机时间同步方法。为了衡量两个相机之间的时间不同步程度,本文提出使用跨视角特征点之间极线距离进行表征,并通过最小化该距离进行同步。使用图结构表示多相机系统,用最小生成树算法得到多相机系统的全局最优时间偏移配置,降低相机间的同步误差。在四种涵盖广泛应用场景的数据集上评估了多相机同步的精度,平均同步误差均小于一帧。(2)针对三维人体姿态数据集稀缺,且目前相关数据集均不提供人体关键点是否被遮挡的标注的问题,提出了一种基于三维人体姿态的数据合成方法,构建了具有丰富人体姿态、且包含遮挡情况的多视角合成数据集。为了丰富合成数据集中人体姿态的多样性,本文从网络视频中估计三维人体姿态,作为动作捕捉数据库的补充,以驱动虚拟人运动。在虚拟世界中使用物品对人体产生更多样的遮挡情况,通过对比关键点在三维场景和深度图中的深度差异,自动生成关键点是否被遮挡的标注。该数据集目前包含约六十万帧图像,被遮挡关键点占比超过20%。此外,通过改变场景中物品、人体和姿态等属性,可方便地生成更多数据。(3)针对稀疏多相机系统中三维人体姿态估计精度受到遮挡影响的问题,提出一种基于几何关系的多视角特征融合框架,使用可见视角中的特征增强被遮挡关键点的特征。具有相同语义的位置处的特征才能进行融合,特征融合方法需要解决的一个核心问题是判定不同视角上位置的相关关系,本文提出利用人体姿态估计热图的稀疏特性以有效解决该问题。为了避免优质特征在融合过程中退化,提出从外观和几何一致性两个角度,基于学习的方式为每个视角的检测结果赋予置信度权重,用于特征融合过程。在多个基准数据集取得最优性能。此外,特征融合模型在新场景下无需重新训练,能够有效地迁移使用。(4)为进一步扩展特征融合框架,提出了融合可穿戴惯性传感器IMU和多视角图像的特征融合框架。该方法的核心是通过相邻关键点的特征,提高对被遮挡关键点的检测性能。首先利用IMU对相邻关键点的特征进行相互增强,提高二维人体姿态检测率。然后使用图结构模型从多视角二维人体姿态估计三维人体姿态,提出一种基于IMU的人体结构先验,优化估计三维人体姿态。该特征融合方法在公开数据集上显著降低了三维人体姿态估计的误差,取得最优的结果。更多还原

【Abstract】 Research on “Human-Centric” computer vision has become an important aspect of artificial intelligence.To achieve this,the first challenge is to make computer understand human.It is necessary to accurately recognize human poses and actions.However,occlusion problems impose great challenges on human pose estimation.We research several key problems related to multiview images for practical and accurate 3D human pose estimation,including multiple camera systems,synthetic data and feature fusion algorithms.Our work are summarized as follows:(1)We solve the problem of temporal synchronization when developing multiple camera systems.We propose an out-of-the-box framework to temporally synchronize multiple cameras using semantic human pose estimation from the videos.Our method firstly calibrates each pair of cameras by minimizing an energy function related to epipolar distances.Secondly,we integrate the synchronized camera pairs into a graph and derive the optimal temporal displacement configuration for the multiple-camera system.We evaluate our method on four public benchmark datasets and demonstrate robust sub-frame synchronization accuracy on all of them.(2)Three-dimensional human pose annotations are hard to obtain,and none of the previous benchmarks provide occlusion label for joints in image.To address the limitations,we create a large scale multiview synthetic dataset which provides occlusion labels for every joint.We retarget human motions from videos in addition to large scale motion capture library to increase pose diversity.Objects are purposely placed to occlude some body joints,and then the joint locations from depth maps and 3D position annotations are compared to generate occlusion labels.The synthetic dataset contains more than 600 k images with more than 20% joints occluded.It can be easily expanded by modifying object attributes to generate more data.(3)Human pose estimation results are prone to be affected by occlusion in sparse camera settings.We present an adaptive feature fusion framework based on multiview geometry,which enhances the features in occluded views by leveraging those in visible views.The core of this method is to determine the point-point correspondence between two views which we solve effectively by exploring the sparsity of the heatmap representation.We also learn an adaptive fusion weight from both appearance and geometric consistency aspects for each camera view to reflect its feature quality in order to reduce the chance that good features are undesirably corrupted by “bad”views.We extensively evaluate the approach on the public datasets including Human3.6M and CMU Panoptic.It outperforms the state-of-the-arts on all of them.We also show that feature fusion can effectively handle occlusion by evaluating on the synthetic dataset.Besides,feature fusion model trained on synthetic dataset can generalize to real world applications with effect.The fusion model can be directly applied to new camera configurations without additional adaptation.(4)To extend the multiview feature fusion framework to other sensor modality,we propose to combine multiview images and a few IMUs attached at person’s limbs for improved 3D human pose estimation.It detects 2D poses for each camera view and then IMUs are used to reinforce the features of each pair of joints.We use IMU as human body structure prior in the pictorial structure model and lift the multiview 2D poses to the 3D space by jointly minimizing the projection error between the 3D and 2D poses,along with the discrepancy between the 3D pose and IMU orientations.This approach reduces the error of the state-of-the-art by a large margin on public datasets.更多还原