【摘要】 针对手势识别在单目视觉下面临遮挡干扰与精度不足的问题，提出一种融合二维骨骼关键点、手关节角度与手指曲率多模态数据的手势识别方法。为实现数据融合，构建了基于弯曲传感器数据手套与单目相机的采集系统，收集了7名受试者手势动作(包含遮挡场景)的数据集。通过提取手部关键点并计算关节角度，将骨骼、关节角度与曲率信息拼接为多模态输入，进而利用卷积神经网络—双向长短期记忆(CNN-BiLSTM)混合网络分别学习空间与时间特征。实验结果表明，所提出的多模态融合方法相比仅使用骨骼信息，识别准确率从68.34%显著提升至84.13%,证明融合手指曲率与关节角度能有效克服遮挡问题，提高手势识别的鲁棒性与准确性。更多还原

【Abstract】 To address the issues of occlusion interference and low accuracy in gesture recognition under monocular vision, a gesture recognition method that integrates multimodal data from 2D skeletal keypoints, hand joint angles, and finger curvature is proposed. To achieve data fusion, a data acquisition system based on a bending sensor data glove and a monocular camera is constructed, and a dataset of gesture actions(including occlusion scene)is collected from seven subjects. By extracting hand keypoints and calculating joint angles, the skeletal, joint angle, and curvature information are concatenated into a multimodal input. Then, a convolutional neural network-bidirectional long short-term memory(CNN-BiLSTM) hybrid network is used to learn spatial and temporal features, respectively. Experimental results show that the proposed multimodal fusion method significantly improves the recognition accuracy from 68.34 % to 84.13 % compared to using only skeletal information, demonstrating that fusing finger curvature and joint angles effectively overcomes the occlusion problems and improves the robustness and accuracy of gesture recognition.更多还原