【作者】 刘博；

【导师】 金海；

【作者基本信息】 华中科技大学 ， 计算机系统结构， 2020， 博士

【摘要】 深度学习作为目前人工智能领域的前沿技术,在处理现实世界复杂问题时优势显著。与传统的人工神经网络相比,深度神经网络拥有更多的隐藏层及神经元,同时产生了大量不同种类的中间数据。这些数据对训练与推理任务的执行效果能够起到至关重要的作用,但与此同时,也在系统层面上带来庞大的计算体量、存储开销以及通信负载。目前深度学习系统优化领域存在不少问题,训练运行时效率较为低下,主要问题集中在两方面,其一是高性能加速硬件的内存容量有限,深度学习系统对内存资源的利用率较低,难以完成大规模迭代训练任务;其二是分布式加速硬件集群的互联带宽有限,集群节点间频繁的梯度通信成为性能瓶颈。解决上述多重资源受限的问题主要涉及到神经网络中特征映射、参数等中间数据的内存管理与稀疏通信优化技术等方面的研究。具体来说,针对深度学习系统优化的研究工作分别从以下三个角度开展。针对加速硬件存储资源不足以满足训练运行时内存需求的问题,考虑以神经网络模型层结构为中心的内存空间重用思想,提出一种基于空间复用的特征映射数据内存管理机制Layrub。首先对反向计算的执行序列进行细粒度的重排,然后针对卷积神经网络特征映射数据规模较大的特性,提出神经网络层敏感的内存复用策略,实现高内存利用率的特征映射数据组织方式。通过对数据在加速硬件与主存之间进行合理迁移与放置,使得训练运行时的内存消耗明显减少,能够对深度神经网络的形态设计和进一步研究提供系统级的支撑。实验表明,与原生未优化的深度学习框架Caffe相比,Layrub可以使系统内存使用率平均降低58.2%以上,最高可达98.9%,同时达到适中的时间代价,训练执行时间平均增加24.1%。实验结果还表明,Layrub明显优于一些具备内存优化能力的主流深度学习系统,并且可以执行极限规模的深度学习任务。针对分布式训练中高性能计算节点间梯度交换的通信瓶颈问题,提出了一种基于陈旧补偿的参数梯度稀疏通信优化机制Grad SA。首先提出基于神经网络层结构的梯度稀疏化思想,分析并结合参数梯度数据的特点,减少梯度传输规模,显著降低跨节点通信开销;其次,利用近似梯度进行历史梯度的积累,以此来加快收敛速度。随后,针对历史梯度累积后的稀疏梯度进行高效编码。实验结果表明,即使在通信网络带宽较低的极端环境下,所提出的梯度优化算法也能够获得良好的吞吐量和加速性能。Grad SA能够在保证迭代计算性能不下降的前提下,将参数梯度通信量减少514倍。通过与8-bit量化、Tern Grad和DGC等主流方法进行比较,进一步证明了Grad SA在各种神经网络模型和数据集上的优势。针对分布式训练中计算、内存、通信带宽等资源消耗巨大、协同优化困难的问题,考虑系统内存与通信资源协同优化的思想,提出一种内存高效的分布式稀疏通信机制Lay SA。首先对内存管理策略进行数据对象的增广与重新定义,解决由稀疏通信引起的内存膨胀问题,其次提出镜像参数更新机制,解决参数梯度数据的内存优化与稀疏通信优化难以共存的矛盾,填补分布式深度学习系统在多重资源优化方面存在的空缺,最终实现内存与通信两类优化思路的深度融合、协同执行。实验结果表明,经过系统资源协同优化后,计算节点的内存压力得到有效缓解,在资源利用率及分布式训练效率方面取得了良好效果。与仅采用Layrub和Grad SA的训练系统相比,Lay SA最高能够节约80.5%的系统内存使用量,神经网络模型在单GPU上整体训练时间缩短了12.25%左右,除此之外,Lay SA能够使用更大批量数据进行分布式训练,系统吞吐量整体提升150%以上,明显优于单独使用内存或通信优化机制的训练系统。综上所述,针对深度学习系统优化这个研究命题,从体系结构层面上考虑使用更合理的中间数据存储管理机制,从分析数据结构、系统调优等思路进行,对深度神经网络的训练执行效率进行提升。与此同时,从系统运行时层面上优化增强中间数据(其中包括特征映射数据及参数梯度),从分析中间数据特性、数据体量的思路出发,对深度神经网络的训练质量进行提升。通过上述研究,能够在硬件环境的计算、存储、带宽等资源有限的情况下,尽可能的挖掘大规模深度学习的效率和性能。更多还原

【Abstract】 Deep learning is one of the popular fields in artificial intelligence,which outperforms the traditional machine learning when dealing with complex problems of real world.Compared with artificial shallow neural networks,deep neural networks have much more hidden layers and neurons,as well as generate enormous amounts of intermediate data with different types.These data play important roles for the efficiency on the training or inference tasks.However,they also bring mammoth amounts of computation load,storage cost,and communication overhead for the GPU-based deep learning systems.There still exist many issues on deep learning systems being addressed.First,due to the limited memory capacity of accelerated hardware(i.e.,GPU,FPGA),as well as the low utilization of memory resources,it is difficult to accomplish the large-scale iterative training tasks.Second,since the interconnection bandwidth of distributed accelerated cluster is limited,the frequent gradient communication between cluster nodes becomes the performance bottleneck.When solving these above problems,it mainly involves core techniques on the memory management and sparse communication optimization for the intermediate data(i.e.,feature maps,weights,and their gradients)of the training.Specifically,this research focuses on the runtime optimization for the deep learning systems,and develops the corresponding strategies as follows.To address the memory issue on the training program,we first identify the memory usage characteristics for deep and wide convolutional neural networks and demonstrate the opportunities for memory reuse at both the intra-layer and inter-layer levels.We then present Layrub,a runtime GPU memory management strategy that is specific for the feature map data of convolutional neural networks.First,it orchestrates the execution of the backward phase on the training process.Then,it proposes an idea of layer-centric memory reuse to reduce memory consumption based on the memory characteristics of network models and achieves a proper data organization for the feature map with high memory utilization.By reasonably transferring and placing the data between the acceleration hardware and the main memory,the memory consumption during the training process will be significantly reduced,which can provide system-level support for the morphological design and further research of deep neural networks.Experiments show that,compared to the original Caffe,Layrub can cut down the memory usage rate by an average of 58.2% and by up to 98.9%,at the moderate cost of 24.1% higher training execution time on average.Results also show that Layrub outperforms some popular deep learning systems with memory opitimzation strategy and can perform extreme-scale training tasks.To tackle the communication bottleneck on the distributed training,we propose a staleness-compensated sparse gradient optimization mechanism,Grad SA,to improve communication efficiency.First,layer-level gradients are sparsified to reduce the communication overhead,which conforms to characteristics of the network structure.Second,the historical accumulations of the approximated gradients are utilized to speed up convergence.Then,an efficient encoding mechanism is proposed to compress the sparse accumulated historical gradients.Experimental results show that the proposed gradient optimization algorithm can obtain ideal throughput and acceleration performance even in extreme environments with low communication network bandwidth.Furthermore,extensive experiments indicate that Grad SA can reduce gradient size by 514× without performance degradation.By comparing with competitors such as 8-bit quantization,Tern Grad,and DGC,it further proves the advantages of Grad SA in various neural network models and data sets.To reduce the huge consumption of the multiple resources(i.e.,computing,memory,and communication bandwidth)and tackle the conflict between different optimizations,we consider the idea of collaborative optimization on both system memory and bandwidth resources and propose a memory-efficient distributed sparse communication mechanism,called Lay SA.First,to tackle the memory ballooning issue caused by sparse communication,the memory reuse strategy of Layrub is refined,and the data object of the memory optimization is augmented and redefined.Second,a mirror parameter update mechanism is proposed to solve the contradiction between the memory management and the sparse communication optimization for weight gradients.The deep integration and collaborative execution of these two types of strategies can fill the gap in the distributed GPU-based training system in terms of multiple resource optimization.Experimental results show that the proposed collaborative optimization can significantly alleviate the memory pressure of computing nodes and improve both the resource utilization and the system throughput of the distributed training systems.Compared with the baseline system using either Layrub or Grad SA,Lay SA can help to save the system memory usage by up to 80.5%,and the overall training time of the neural network models on a single GPU is reduced by about 12.25%.Furthermore,Lay SA can extremely scale up the batch size of datasets during the distributed training,and the overall throughput has increased by more than 150%,which outperforms the current training systems that use memory optimization or communication optimization mechanisms alone.In summary,this research studies the memory management and sparse communication optimization policies for the intermediate data of deep neural networks,aiming to guarantee the efficiency of the runtime training program.At the same time,it also proposes to enhance and refine the intermediate data by considering both the data characteristics and quantity,aiming to keep the convergence quality of the training process.Through the above research,it can excavate the efficiency and performance of large-scale deep learning as much as possible with the restricted hardware resources such as computing,memory,and bandwidth.更多还原