【作者】 李博；

【导师】 王硕苹； 金苍宏；

【作者基本信息】 浙江大学 ， 工程硕士（专业学位）， 2022， 硕士

【摘要】 在“信息过载”的时代,推荐系统是帮助用户过滤无用信息,获取感兴趣信息的重要工具,它已经在众多领域被广泛使用。序列推荐系统是推荐系统的一种,它在特定的场景能取得比传统推荐系统更好的效果。序列推荐系统建模用户动态偏好和序列行为模式的能力很强,因此已经成为了如今学术界的研究热点。为了提高模型的表达能力,越来越多研究者将深度学习与序列推荐系统结合,并取得了成功。目前,基于深度学习的序列推荐系统的模型架构正变得越来越大,然而,这会降低其推理速度,限制了其在生产环境的广泛应用。为了解决这一问题,本文探究如何对基于深度学习的序列推荐系统进行压缩。首先,我们探究如何使用知识蒸馏来压缩序列推荐系统。目前,较前沿的序列推荐系统均由嵌入层、中间层和预测层构成。根据序列推荐系统的结构特点,我们分别在这三个部分进行知识蒸馏。在嵌入层部分,我们使用均方误差来计算距离。在中间层部分,我们提出了“基于动态权重的推土机距离”来衡量老师模型和学生模型中间层表示的距离。该方法不会遗漏老师模型中间层的信息,还通过动态计算“推土机距离”权重的方式提高了老师模型向学生模型传递“暗知识”的效率,能增强学生模型的蒸馏效果。在预测层部分,我们使用Hinton提出的经典知识蒸馏方法进行处理。最后,我们通过先使用老师模型初始化学生模型的参数,再蒸馏学生模型的方式,提高学生模型最终的预测准确度。其次,我们使用低秩分解和参数共享技术来压缩序列推荐系统。在第一部分,我们在不更改模型结构的前提下压缩模型,在这一部分,我们探究如何通过改变待压缩模型的网络结构完成压缩。对于模型的嵌入层矩阵和预测层矩阵,我们使用低秩分解压缩模型的嵌入层,提出了“基于向量分数与k均值聚类的分块低秩分解”技术。该方法充分利用了数据集和待压缩模型提供的信息,能尽可能地减少模型的精度损失。在中间层部分,我们提出了“自适应块间参数共享”机制,相比于传统的层间参数共享,该方法是更适合序列推荐系统的选择。不仅如此,该方法还能够根据待压缩模型的参数特点自适应地确定参数共享的策略。除此之外,我们还将第一部分的知识蒸馏办法与这一部分的压缩方法结合,使用已预训练好的待压缩模型来指导使用低秩分解和参数共享技术压缩的模型,尽可能减少模型预测精度的损失。最后,我们在两个真实的序列推荐领域数据集ML-50和Weishi-10上,对两个基于深度学习的序列推荐系统,即SASRec和NextItNet,进行对比实验,充分证明了我们方法的有效性。更多还原

【Abstract】 In the era of big data,recommendation systems are important technology to help users filter useless information and obtain valuable information,and they have been widely used in many fields.The sequential recommendation systems are a special kind of recommendation systems,they perform better in some particular scenes than the traditional ones.Sequential recommendation systems are good at modeling users’ dynamic preference and sequences’ pattern,so they have become hot spot in academia.Nowadays,more and more researchers begin to combine deep learning with sequential recommendation systems,and have achieved great success.In general,the models base on deep learning require lots of parameters to obtain optimal performance,which may hinder them from being deployed in production.To resolve this issues,we plan to compress the sequential recommendation systems which base on deep learning in the paper.First of all,we use knowledge distillation to compress sequential recommendation systems.At present,the state-of-art sequential recommendation systems are typically based on a sandwich-structured neural network,which consists of an embedding layer,intermediate layers and a prediction layer.Therefore,we apply knowledge distillation in three aspects.we use mean squared error to deal with the distance of embedding layer between the student model and the teacher model.For the intermediate layers,we propose a method named earth mover’s distances based on dynamic weights.This method does not omit any information of the teacher model.At the same time,it increases the delivery efficiency of the dark knowledge by dynamically calculating the weight in earth mover’s distance,which can enhance the student model’s performance.Besides,we deal with the prediction layer by the vanilla knowledge distillation technology proposed by Geoffrey Hinton.Finally,we use a kind of two-stage knowledge distillation approach to initialize student model,which can improve the accuracy of the student model greatly.Moreover,we also use low-rank decomposition and parameter sharing techniques to compress the sequential recommendation systems.In the first part,we compress the model without changing the structure.However,in this part,we explore how to compress the model by changing its structure.To compress the matrix in embedding layer and prediction layer,we propose the block-wise low-rank decomposition based on vectors’ score and k-means clustering algorithm.This method makes full use of the information that provided by datasets and the over-parameterized network.For the middle layers,we propose a technique,namely adaptive block-wise parameter sharing,which is much more suitable for the compression of sequential recommendation system.Not only that,we also propose an algorithm to help the model determine which blocks’ parameters should be shared,Besides,we also combine these methods with knowledge distillation.Finally,extensive experiments on two recommendation datasets show the effectiveness of our methods.更多还原