【作者】 王萌；

【导师】 方维维；

【作者基本信息】 北京交通大学 ， 计算机科学与技术， 2024， 硕士

【摘要】 深度神经网络(Deep Neural Network,DNN)在许多领域都取得了广泛的应用,由于其庞大的算力和内存需求,往往被部署于云端。然而,当今处在万物互联的时代,大量终端设备和分布式设备快速涌现,用户对计算的实时性和数据的私密性提出了更高的要求,将DNN推理任务部署在边缘侧至关重要。同云端设备相比,边缘设备在体积、功耗和延迟等方面具备优势的同时,也受限于内存和算力方面的严格要求。因此,如何将DNN高效部署在边缘设备是一项重大挑战。本文将边缘设备上的处理单元分为中央处理单元(Central Processing Unit,CPU)、图形处理单元(Graphics Processing Unit,GPU)和神经网络处理单元(Neural Processing Unit,NPU)三大类。三者在通用性、功耗、精度和计算速度等方面各有优劣。由于需求和应用场景的复杂化,许多边缘设备拥有多个处理单元,称其为异构设备。因此,如何运用模型早退、量化等技术优化模型,并根据各处理单元的特点优化DNN模型调度,对于提升DNN模型推理性能至关重要。针对以上问题和挑战,本文基于不同的异构处理单元组合,对边缘设备上的DNN模型推理优化技术进行系统研究,研究的主要工作和创新点包括:(1)GPU是边缘设备中用于处理DNN推理任务的最常见的处理单元,但是作为一种通用设备,可能会被其他任务占用,造成DNN推理任务时延的增加。针对处理单元为CPU和GPU的边缘设备,本文提出了基于早退和Neural-UCB的DNN推理调度方案。一方面,本文使用了结合知识蒸馏的早退方案,早退模型的多分支结构可以通过提前退出实现模型加速。通过知识蒸馏,可以大幅提升早期早退点的推理精度,使得推理过程中可以选择更靠近输入层的早退点,进一步缩短推理时间。另一方面,本文使用多臂赌博机Neural UCB算法来进行推理调度,为当前的推理任务选择合适的处理单元与退出点。实验结果表明,在相同的精度要求下,基于Neural UCB的DNN调度方案的推理时延比探索后固定、贪心方案的推理时延分别减少37-51%,49%-61%。(2)针对处理单元包含NPU的边缘设备,本文提出了基于图像复杂度和动态规划的调度方案。由于NPU资源限制,必须在其上部署量化模型,以牺牲精度为代价提高推理速度。为解决低精度问题,本文采用如下方法:首先,提出了一种图像复杂度估计方案。而后,根据图像的复杂度,使用动态规划算法调度,确定是否将该帧卸载到GPU上使用早退模型进行推理,并确定早退点。调度目标是在固定时间限制下,最大化精度。通过在搭载高通骁龙855平台的手机上实验,证明该方案在一定时间约束下的精度显著高于直接在NPU上进行推理,当每帧的推理时间约束为30-200ms,帧率为10-30FPS时,本文方案比直接在NPU上推理精度提高10%左右,有效减轻了精度损失。更多还原

【Abstract】 Deep Neural Networks(DNNs)have achieved widespread applications in many fields.Due to their massive computing and memory requirements,DNN models are often deployed in the cloud servers.However,we are living in an era of ubiquitous connectivity.A large number of terminal devices and distributed devices are emerging rapidly.Users have higher requirements for the timeliness of computing and privacy of data,making it crucial to deploy DNN inference tasks on the edge side.Compared with cloud devices,edge devices have advantages in terms of size,power consumption and latency,while facing tremendous limitations in memory and computing capabilities.Therefore,how to efficiently deploy DNNs on edge devices poses a major challenge.This paper divides the processing units on edge devices into three major categories:Central Processing Unit(CPU),Graphics Processing Unit(GPU),and Neural Processing Unit(NPU).Each has its own advantages and disadvantages in terms of generality,power consumption,inference accuracy and computing speed.Due to the complexity of demand,many edge devices have multiple processing units,called heterogeneous devices.Therefore,it is crucial to optimize DNN models by appropriately apply technologies such as model pruning,quantization,etc.,and optimize DNN model scheduling according to the characteristics of different processing units.To address the aforementioned issues and challenges,this paper conducts an in-depth study of DNN model inference optimization techniques for edge devices based on different combinations of heterogeneous processing units.The main works and innovations include:(1)The GPU is one of the most prevalent processing units for undertaking DNN inference tasks on edge devices.However,as a general-purpose processing unit,the GPU risks being occupied by other workloads as well,which could lead to increased latency for DNN inference jobs.For edge devices with CPU and GPU,this paper proposes a DNN inference scheduling scheme based on Early Exit and Neural UCB.On the one hand,this paper uses an Early Exit scheme combined with knowledge distillation.The multi-branch structure of Early Exit models can accelerate models by exiting early.Through knowledge distillation,the inference accuracy of exit points can be greatly improved in the early stage,so that even earlier exit points closer to the input layer can be selected during inference,further shortening the inference time.On the other hand,this paper uses the Neural UCB multi-armed bandit algorithm for inference scheduling to select the suitable processing unit and exit point for the current inference task.Experimental results show that under the same accuracy requirements,the inference latency of the DNN scheduling scheme based on Neural UCB is 37-51% and 49%-61% shorter than that of explorationthen-exploitation and-greedy scheduling schemes,respectively.(2)For edge devices with NPU,this paper proposes a DNN scheduling scheme based on image complexity and dynamic programming.Due to resource constraints of NPUs,quantized models must be deployed for inference to improve speed at the cost of reduced precision.To address the low precision problem,this paper adopts the following methods:First,it proposes an image complexity estimation approach.Afterwards,a dynamic programming scheduling algorithm is used to determine whether to offload the frame to the GPU for inference using an early-exit model according to the complexity of the image,and to determine the early-exit point.The scheduling target is to maximize precision within a fixed time constrain.The experiment conducted on a smartphone equipped with the Qualcomm Snapdragon 855 platform demonstrates that under certain time constraints,the accuracy of the proposed approach is significantly higher than direct inference on the NPU.When the inference time constraint per frame is set between 30-200 ms and the frame rate was 10-30 FPS,the accuracy of the proposed approach is approximately 10%higher than direct inference on the NPU,effectively reducing the accuracy loss.更多还原