【作者】 张雪；

【导师】 王明江；

【作者基本信息】 哈尔滨工业大学 ， 微电子学与固体电子学， 2020， 硕士

【摘要】 人工智能的飞速发展导致人们从人际互动转向更多的人机互动,并对情感交互技术提出了更高的要求,情感是人类交流的具体体现,而情感识别的研究热度也逐渐增大,单模态情感识别往往存在信息不完整、干扰性强、识别率低等缺点,近期以来,广大研究者对于多模态情感识别给予高度的重视,并在语音、视频、文本和生理信号情感识别领域开展了大量的研究工作,多模态情感识别通过融合不同模态之间的信息,来进行互相补充,从而提高最终识别率。而深度学习神经网络发展至今已经在情感识别方向上有一定的应用,我们可以通过改变神经网络来优化情感特征提取的准确性。本课题主要研究内容是对语音、视频和文本三种模态信息基于深度学习神经网络进行多模态情感特征提取并搭建情感识别模型。本文研究了分别适用于语音、视频和文本三种模态的有效特征提取方法,对于语音信息输入,本文使用长短时记忆神经网络(LSTM)进行语音特征提取,由于语音信号的每一时刻的输出与前后时刻均有联系,该网络能够使语音信号更好地利用前后时刻的信息;对于视频信息的输入,本文使用一种密集连接的卷积神经网络(Dense Net)来提取图像特征,脱离了加深网络层数(Res Net)和加宽网络结构(Inception)来提升网络性能的定式思维,从特征的视角综合性分析,采用特征重用与旁路(Bypass)设置的方式,能够有效降低网络中的参数量,同时将梯度消失问题有效解决;对于文本信号的情感特征提取,采用的是LSTM神经网络,可以有效地提取富含情感的前后语义和语序信息。为了使三种模态的信息有效融合,本课题深层剖析了多模态情感识别的融合方式,其中,基于特征层的融合方式可以有效地利用各个模态之间的信息,但直接级联的特征层所采用的融合方式,仅仅是将每种模态的输出情感特征向量进行了拼接,本文针对研究需求将注意力机制有效引入到特征层融合的过程中,该机制通过学习来获得一个符合数据集分布的合理权重,并在最后情感识别时在特征融合层进行权重加和,进而确保了多模态情感识别结果的精确度及合理性。本文设计了单模态、双模态和多模态的对比试验,并针对IEMOCAP数据集中的十种情感分类进行了五分类输出、四分类输出、三分类输出、二分类输出。并分析讨论实验结果,我们在五分类输出的情况下,双模态情感识别比单模态情感识别的准确率提高了6.2%,相对于双模态情感识别而言,语音视频和文本的三种模态情感识别的准确率提高了8.98%。在其他三种分类输出模式下也符合该规律。通过设计实验从而验证了本课题的基于深度学习的多模态情感识别模型的准确性和有效性。更多还原

【Abstract】 The research of emotion recognition is an important branch in the field of emotion computing.With the continuous development of artificial intelligence technology,human-computer interaction experience is constantly pursuing more humanization and intelligence.Emotion recognition has become a research hotspot.Single mode emotion recognition often has the disadvantages of incomplete information,strong interference and low recognition rate.Recently,multi-mode emotion recognition has been widely used in many fields It has been widely concerned by researchers,and a lot of research work has been carried out in the field of speech,video,text and physiological signal emotion recognition.Multimodal emotion recognition can complement each other by fusing the information between different modes,so as to improve the final recognition rate.In this paper,we build a multi-modal emotion recognition model based on voice,video and text.This paper studies the effective feature extraction methods for speech,video and text.For speech information input,this paper uses long-term and short-term memory neural network(LSTM)for speech feature extraction.Because the output of each time of speech signal is related to the front and back time,this network can make better use of the information of the front and back time of speech signal;for visual For the input of frequency information,this paper uses a dense connected convolutional neural network(DenseNet)to extract image features,which breaks away from the fixed thinking of deepening network layers(ResNet)and widening network structure(perception)to improve network performance.From the perspective of features,through feature reuse and bypass setting,the network parameters are greatly reduced,To a certain extent,it alleviates the problem of gradient disappearance;for text feature extraction,this paper uses LSTM neural network,which can effectively extract emotional semantic and word order information.In order to fuse the information of three modes effectively,this paper studies the fusion method of multi-modal emotion recognition.Among them,the fusion method based on feature layer can effectively use the information of each mode,but the direct cascade feature layer fusion method only splices the output emotion feature vectors of each mode.In this paper,attention mechanism is introduced into feature layer fusion The mechanism obtains a reasonable weight according to the data set distribution through learning,and adds the weight in the final feature fusion,making the multi-modal emotion recognition results more accurate.In this paper,the single-mode,dual-mode and multi-mode comparative tests are designed,and five kinds of output,four kinds of output,three kinds of output and two kinds of output are carried out for ten kinds of emotion classification in the data set of IEMOCAP.The experimental results show that the accuracy of bimodal emotion recognition is 6.2%higher than that of single-mode emotion recognition,and the accuracy of multimodal emotion recognition is 8.98%higher than that of bimodal emotion recognition.The design verifies the effectiveness of multimodal emotion recognition.更多还原