【作者】 陈宇；

【导师】 高礼忠；

【作者基本信息】 东南大学 ， 信号与信息处理（专业学位）， 2020， 硕士

【摘要】 随着深度学习技术的成熟和普及,以及在海量数据和丰富应用场景的催生下,以卷积神经网络为代表的深度卷积网络开始逐渐替代机器学习时代基于人工提取特征的传统算法。而不断逼近精度极限的代价就是网络深度、尺寸的增长,网络模型越来越趋于臃肿,这对于深度学习的产品落地化是一个严峻的考验。为了更好地在计算资源有限的设备端部署模型且不影响使用,模型压缩的相关研究应运而生。本文主要基于基础算法和具体应用场景,对模型压缩的算法和方案实现展开了系统的研究,具体工作如下:1.针对量化的模型压缩算法,研究了传统二值网络的二值策略以及训练流程,并在卷积神经网络上对多值网络完成策略和权重更新的优化。针对二值网络精度损失较大的缺陷,提出了一种基于集成学习的二值组合模型,并改进网络结构,在CIFAR-10数据集上达到和原始网络相当的精度水平。2.针对知识蒸馏的模型压缩算法,研究了基础的教师学生模型以及蒸馏损失函数,设计蒸馏训练实验测试算法性能。针对蒸馏效果和教师网络的低关联性,提出了一种自学习的知识蒸馏优化方式,在CIFAR-100数据集上的性能提升与传统蒸馏相近,但有效节省了教师网络的模型资源。3.从具体应用场景的层面,论文选用语义分割任务作为目标,并使用U-Net网络搭建模型,在经处理的二分类人体解析数据集上进行训练,实现基本人体语义分割的预测效果。在此基础上提出了一种基于剪枝微调-稀疏张量解析-查找表量化存储的系统压缩方案,并应用在U-Net人体语义分割网络中。通过超参数筛选获得最佳剪枝率,在处理后的Human Parsing数据集上准确率基本没有损失,同时实现了接近20的实际模型压缩比。更多还原

【Abstract】 With the maturity and popularization of deep learning technology,as well as the birth of massive data and rich application scenarios,deep convolutional network,represented by convolutional neural network,began to gradually replace the traditional algorithms based on artificial feature extraction in the machine learning era.The cost of constantly approaching the precision limit is the growth of network depth and size,and the network model is becoming more and more bloated,which is a severe test for the implementation of deep learning products.In order to better deploy the model on the device side with limited computing resources without affecting its usage,the research on model compression came into being.Based on the basic algorithm and specific application scenarios,this paper systematically studied the algorithm and scheme implementation of model compression.The specific work is as follows:1.Aiming at the quantified model compression algorithm,the binary strategy and training flow of traditional binary network are studied,and the strategy and weight update are optimized on the convolutional neural network.Aiming at the defect of large loss of binary network precision,a binary combination model based on integrated learning was proposed,and the structure of the network was improved to achieve the same precision level as the original network on the cifar-10 data set.2.Based on the model compression algorithm of knowledge distillation,the basic teacherstudent model and the distillation loss function are studied,and the performance of the distillation training experiment is designed to test the algorithm.Aiming at the low correlation between distillation effect and teacher network,a self-learning knowledge distillation optimization method was proposed.The performance improvement of cifar-100 data set was similar to that of traditional distillation,but it effectively saved the model resources of teacher network.3.From the perspective of specific application scenarios,this paper chooses semantic segmentation task as the target,and uses U-Net network to build a model,and conducts training on processed binary human body analytic data set,so as to achieve the prediction effect of basic human body semantic segmentation.On this basis,a system compression scheme based on pruning and fine-tuning,sparse tensor resolution,quantization and storage of lookup table is proposed and applied in U-Net human semantic segmentation network.The optimal pruning rate is obtained through super parameter screening,and the accuracy rate is basically no loss in the Human Parsing data set after processing,and the actual model compression ratio of nearly20 is realized at the same time.更多还原