【作者】 王彬；

【导师】 常发亮； 刘春生；

【作者基本信息】 山东大学 ， 电子信息（专业学位）， 2024， 博士

【摘要】 视频人体行为识别技术是计算机视觉领域中的关键研究方向之一。近年来,随着深度学习技术的飞速发展,视频人体行为识别技术取得了显著进步。人体行为识别技术在智能视频监控、智能驾驶、人机智能交互等领域展现出极其广泛的应用前景和研究价值。目前的主流行为识别方法根据其时序建模关注点的不同可以划分为运动表征学习、时空特征聚合以及运动对象交互模型三类主要方法,尽管针对人体行为识别技术已经开展了大量的研究工作并取得了一定进展,但仍存在以下不足:运动表征学习方法中局部密集采样策略的方向单一和帧数冗余导致识别性能受限;时空特征聚合方法的准确率与网络模块效率之间平衡性差、网络模块之间契合度低;运动对象交互模型缺乏场景上下文与时序模型之间的逻辑关联能力。本文针对以上问题,从时空特征及运动特征角度出发,研究基于时序建模的视频人体行为识别方法,主要研究工作和内容归纳如下:1.基于自适应动作节奏学习的视频人体行为识别方法。针对运动表征学习方法局部密集采样策略中采样帧数冗余问题,提出了一种基于自适应运动节奏学习的人体行为识别方法。该方法首先设计了一个用于捕获和建模视频浅层运动特征的自适应运动节奏学习模块(Adaptive Motion Tempo Learning Module,AMTLM),通过双向帧差分互补的方式使其能够自动适应不同时序的视频输入特征并进行时序建模。其次,提出了一个动态相关性计算模块(Dynamic Correlation Computation Module,DC2M),该模块可以有效地建模不同时间尺度下的复杂视频信息,通过抑制无效运动干扰信息和背景干扰信息的方式提升行为识别性能。最后,从多个标准人体行为识别基准数据集上进行的实验表明,该方法能够显著提升视频人体行为识别性能。2.基于联合增强网络的视频人体行为识别方法。针对运动表征学习方法中局部密集采样帧方向单一以及局部时空特征聚合方法与网络模块之间契合程度较差的问题,提出了一个用于时序建模的二阶段联合增强网络(Adjoint Enhancement Network,AE-Net)学习方法。首先,该方法针对浅层特征设计了一种初级联合增强模块(Initial Adjoint Enhancement Module,IAEM),该模块负责处理输入视频的浅层特征从而解决运动表征学习方法中采样帧方向单一问题。其次,针对网络的深层特征提出一种全局联合增强模块(Global Adjoint Enhancement Module,GAEM),该模块负责处理网络深层结构的时序特征以解决局部时空特征聚合方法与模块之间契合程度较差的问题。最后,通过在公开基准行为识别数据集中的实验表明,所提出的方法能够实现良好的视频人体行为识别性能。3.基于多维自适应融合网络的视频人体行为识别方法。针对时空特征聚合方法不能有效平衡识别准确率与网络模块效率的问题,提出了一个性能高效的多维自适应融合网络(Multi-dimensional Adaptive Fusion Network,MDAF-Net),该方法可以充分激活深度网络中视频行为特征在长时空跨帧结构中的传递和表达,并且尽可能的减少计算量。首先,设计了一种用于时序维度特征建模的自适应时间捕获模块(Adaptive Temporal Capture Module,ATCM),该模块可以有效地抑制相邻帧中相似特征在时空特征聚合过程中的过度表达,从而激活运动流信息在深度网络中的有效传递。其次,提出了一种可扩展的空间及通道模块(Extended Spatial and Channel Module,ESCM),该模块通过扩展空间特征感知场范围以及增强通道注意力的方式,进一步提高时序建模效率。最后,将所提出的方法在公开行为识别数据集上进行了实验,结果表明所提出的方法能够有效提升视频人体行为识别的性能。4.面向人机交互场景的视频人体行为识别方法与任务系统实现。针对运动对象交互模型方法存在缺乏场景上下文与时序建模模型的逻辑关联能力问题,提出了一个面向人机交互任务的视频人体行为识别方法。结合上述方法在时序建模中的优点,首先提出了一个自适应时间敏感性和运动捕捉网络(Adaptive Temporal Sensitivity and Motion Capture Network,ATSMC-Net),该网络能够自适应地调整时间特征的像素级敏感区域从而有效捕捉运动特征信息。然后,针对场景上下文信息如何关联的问题,设计了一个知识功能图模块(Knowledge Function Graph Module,KFG-Module),该模块可以基于对象和动作之间的逻辑关系进行推理,从而帮助机器人完成人机交互任务。最后,在公开的行为识别数据集中评估了所提方法的表现。此外,本章的最后实现了一个用于验证本文人体行为识别研究工作在实际场景表现的人机交互任务系统,采集并创建了一个用于人机交互应用场景的数据集,并在该数据集中训练后通过搭建的任务系统来验证方法的有效性。实验结果表明,所提出的方法可以显著提高机器人面向人机交互任务的视频理解能力。更多还原

【Abstract】 Video human action recognition technology is one of the key research directions in the field of computer vision.In recent years,with the rapid development of deep learning technology,video human action recognition technology has made significant progress.Action recognition technology shows an extremely wide range of application prospects and research value in the fields of intelligent video surveillance,intelligent driving,and human-computer intelligent interaction.Currently,the mainstream action recognition methods can be divided into three main categories:motion representation learning,spatio-temporal feature aggregation,and motion object interaction model,according to their different temporal modeling concerns.Although a large amount of research has been carried out and some progress has been made in human action recognition technology,the following shortcomings still exist:a single direction of sampling strategy or redundancy in the method of motion representation learning;the accuracy and computational efficiency of the method of spatio-temporal feature aggregation are different from those of the method of human action recognition,and the accuracy and computational efficiency of the method of human action recognition are different.However,there are still the following shortcomings:single direction or redundant sampling strategy in the motion representation learning method;poor balance between accuracy and computational efficiency of the spatio-temporal feature aggregation method and its low fit with the model architecture;and the lack of logical correlation between the scene context and the temporal model in the interaction model of the motion object.Aiming at the above difficulties,this paper studies the video action recognition method based on temporal modelling from the perspective of spatio-temporal features and motion features,and the main research work and contents are summarized as follows:1.Video Action Recognition Method Based on Adaptive Action Tempo Learning.An efficient action recognition method based on adaptive motion tempo learning is proposed to address the problem of computational enhancement due to the redundancy of sampled frames caused by the use of local dense sampling strategy in motion representation learning methods.The method firstly designs an Adaptive Motion Tempo Learning Module(AMTLM)for capturing and modelling video shallow motion information,which is enabled to automatically adapt to video input features of different timings and perform temporal modelling by means of bi-directional frame differential complementation.Secondly,a Dynamic Correlation Computation Module(DC2M)is proposed,which can effectively model complex video transport information in different time scales,and improve the performance of action recognition by suppressing invalid motion interference information and background interference information.Finally,experiments conducted on several standard human action recognition benchmark datasets show that the method can significantly improve video human action recognition performance.2.Video Action Recognition Method Based on Adjoint Enhancement Network.A two-stage Adjoint Enhancement Network(AE-Net)learning method for temporal modelling is proposed to address the problems of single direction of sampled frames in the motion representation learning method and the poor fit between the local spatio-temporal feature aggregation method and the model architecture.Firstly,the method designs an Initial Adjoint Enhancement Module(IAEM)for the shallow features,which is responsible for processing the shallow features of the input video to solve the problem of a single direction of the sampled frames in the motion representation learning method.Secondly,a Global Adjoint Enhancement Module(GAEM)is proposed for the deep features of the learning model,which is responsible for processing the global features of the deep structure in order to solve the problem of poor fit between the local spatio-temporal feature aggregation method and the model architecture.Finally,experiments on several publicly available benchmark action recognition datasets are conducted to demonstrate that the proposed method can achieve good video human action recognition tasks.3.Video Action Recognition Method Based on Multidimensional Adaptive Fusion Network.Aiming at the problem that some existing spatio-temporal feature aggregation methods cannot effectively balance recognition accuracy and computational efficiency,a performance-efficient Multi-dimensional Adaptive Fusion Network(MDAF-Net)is proposed,which can be adapted to most of the mainstream action recognition backbone models,fully activate the transmission and representation of video action features in deep networks in a long spatio-temporal cross-frame structure,and reduce the amount of computation as much as possible.Firstly,an Adaptive Temporal Capture Module(ATCM)for temporal dimension feature modelling is designed,which can effectively inhibit the overexpression of similar features in adjacent frames during spatio-temporal feature aggregation,and thus activate the effective transmission of motion flow information in deep networks.Secondly,an Extended Spatial and Channel Module(ESCM)is proposed,which further improves the efficiency of temporal modelling by extending the range of spatial feature perception field and enhancing the channel attention.Finally,the proposed method is experimented on a mainstream large-scale action recognition benchmark dataset,and the results show that the proposed method can effectively improve the performance of video human action recognition.4.Video Human Action Recognition Method and Task System Implementation for Human-robot Interaction Scenarios.A video human action recognition method for human-computer interaction tasks is proposed to address the problem of lack of logical correlation capability between scene context and temporal modelling models in the moving object interaction modelling approach.Firstly,this chapter summarises the advantages of the approaches in the previous chapters in temporal modelling,and proposes an Adaptive Temporal Sensitivity and Motion Capture Network(ATSMC-Net)for mounting on a mobile robot platform,which is able to adaptively adjust the the pixel-level sensitivity area of temporal features to effectively capture motion feature information.Then,to address the problem of how the scene context information is related,a Knowledge Function Graph Module(KFG-Module)is designed,which can reason based on the logical relationship between objects and actions,and thus help the robot to complete human-robot interaction tasks.Finally,the performance of the proposed method is evaluated on a publicly available behaviour recognition dataset.In addition,a human-robot interaction task system for verifying the performance of the human behaviour recognition research work in this paper in real scenarios is implemented at the end of this chapter.At the same time,a dataset for human-computer interaction application scenarios is collected and created,and the effectiveness of the method is verified by the constructed task system after training in this dataset.The experimental results show that the proposed method can significantly improve the robot’s video comprehension ability for human-robot interaction oriented tasks.更多还原