【摘要】 针对单模态生理信号情感识别率不高，稳定性不足等问题，提出一种基于眼动和光电容积脉搏(Photoplethysmogram, PPG)多模态特征融合的情感识别方法。从眼动和PPG的浅层特征中使用卷积神经网络FECNN提取深层特征，采用特征层融合的方法将深浅层特征进行融合。使用长短期记忆网络(LSTM)作为分类器，将融合后的多模态特征作为LSTM的输入，实现高兴，感兴趣，困惑和无聊四种情感识别。采用在线视频学习场景下采集的数据对上述模型进行训练和评估。使用眼动单模态浅层特征的最高识别率为71.25%,PPG单模态浅层特征的最高识别率为73.40%,基于FECNN-LSTM的眼动和PPG多模态融合情感识别方法取得平均识别准确率达84.68%,实验结果表明，上述模型能充分利用眼动和PPG中的情感特征信息，提高了情感分类准确率。更多还原

【Abstract】 Aiming at the problems of low emotion recognition rate and insufficient stability of single-modal physiological signals, an emotion recognition method based on multi-modal feature fusion of eye movement and photoplethysmogram(PPG) is proposed. From the shallow features of eye movement and PPG,the convolution neural network FECNN is used to extract the deep features and the feature layer fusion method is used to fuse the deep and shallow features. The long-term and short-term memory network(LSTM) is used as the classifier, and the fused multimodal features are used as the input of LSTM to realize four emotion recognition: happiness, interest, confusion and boredom. The model is trained and evaluated by using the data collected in the online video learning scene. The highest recognition rate of eye movement single-mode shallow features is 71.25%,and the highest recognition rate of PPG single-mode shallow features is 73.40%. The average recognition accuracy of eye movement and PPG multimodal fusion emotion recognition method based on FECNN-LSTM is 84.68%. The experimental results show that the model can make full use of the emotional feature information in eye movement and PPG and improve the accuracy of emotion classification.更多还原