【作者】 白宇；

【导师】 张引；

【作者基本信息】 浙江大学 ， 计算机技术， 2020， 硕士

【摘要】 对话系统是自然语言处理的研究热点之一,其研究任务包括对话生成、对话匹配、对话状态跟踪和对话动作识别等。目前有关研究集中在各个任务的模型性能提升上,然而不同的任务使用的基础编码器仍然以循环神经网络(Recurrent Neural Network,RNN)或卷积神经网络(Convolutional Neural Networks,CNN)为主。近期出现的Transformer模型能够捕获句子内部词汇之间的关系,被证明在自然语义上具有比RNN、CNN更强的编码能力。但是Transformer模型不适合编码长文本信息,而对话数据的一个样本通常具有多个对话轮次,文本信息过长,因此该模型无法直接应用到对话任务中。此外,该模型的计算量与显存占用量过大,难以在实际中广泛应用。针对上述问题,本文对如何将Transformer模型应用于对话任务进行展开研究,本文的主要工作包括:1)设计了适合编码对话数据的Mem-Transformer模型。该模型通过记忆网络在不同对话轮次之间传递信息,实现多轮对话的编码,并通过信息压缩方式减小模型计算量,最终在对话匹配、对话状态跟踪和对话动作识别任务中超过其他比较模型。2)提出了Transformer模型的压缩方法。该模型使用滑动窗口进行信息压缩从而减少信息损失,并通过语言模型协同训练方法保持原有句子信息,最终通过实验证明压缩后的模型能够在保持语义编码能力的前提下减少模型的显存占用量与计算量。更多还原

【Abstract】 The intelligent dialog systems,which reserach tasks include dialogue generation,dialogue matching,dialogue state tracking,and dialogue action recognition,is one of the research hotspots in natural language processing.At present,the research on intelligent dialogue systems mainly focuses on the improvement of model performance in various tasks.However,the basic encoders used in different tasks are still based on Recurrent Neural Network(RNN)or Convolutional Neural Networks(CNN).The Transformer model is able to capture the relationship between vocabularies within a sentence and has been proven to have a stronger encoding ability than RNN or CNN in natural semantics.Transformer model is not suitable for encoding long text,while a sample of dialog usually has Multiple dialogue rounds,so the Transformer model cannot be directly applied to dialogue tasks.In addition,the amount of calculation and memory usage of the Transformer model is too large to be widely applied in practice.To solve the above problems,we conduct research on the application of the Transformer model to dialogue tasks.The specific contributions of this article are as follows:1)We propose a model named Mem-Transformer which suitable for encoding dialogue text.The model transfers information between different conversation rounds through the memory network and reduces the amount of calculation through information compression.Finally,it surpasses other models in multiple conversation tasks including dialogue matching,dialogue state tracking,and dialogue action recognition.2)We propose a compression method of the Transformer model.The model uses sliding windows to compress information to reduce information losses and it uses collaborative training of language models to maintain the original sentence information.Finally,experiments show that the compressed model can reduce memory consumption and Calculation amount at the premise of maintaining semantic encoding capabilities.更多还原