【作者】 孙婷；

【导师】 李洪亮；

【作者基本信息】 吉林大学 ， 计算机技术（专业学位）， 2021， 硕士

【摘要】 随着深度学习技术在各行各业中的快速发展,样本数据迅速增大,深度神经网络日益复杂。深度学习应用所需的计算和存储资源快速增长。如何高效利用有限的计算资源来满足快速增长的深度学习应用需求成为亟待解决的问题。分布式深度学习技术成为大规模深度学习应用的重要支撑技术。本文重点研究如何高效地利用分布式深度学习集群的资源,从而提高深度学习作业的吞吐率,缩短深度学习作业的训练时间。目前,分布式深度学习集群的资源分配算法的研究多基于传统的高性能计算或云计算平台资源分配方法。这些方法往往未充分考虑深度学习作业所特有的资源需求,例如深度学习训练周期性导致的作业需求波动和作业共享GPU等资源时带来的相互干扰等问题。本文深入研究分布式深度学习集群中作业共享场景下的运行特征,基于机会主义共享特征开展深度学习作业资源分配算法研究。本文的主要贡献包括:1.本文研究了分布式深度学习集群中共享资源的多种深度学习模型作业在运行时性能的变化。根据深度学习作业具有迭代性质并利用作业在集群上运行过程中的历史数据进行观测分析,得知运行过程中作业的资源需求具有波动性,对此建立机会主义资源共享模型。针对共享主机的作业内存使用量频繁波动的现象,我们探究作业之间资源争抢对性能产生干扰的问题,并基于机会主义资源共享模型建立分布式深度学习资源分配问题模型。2.本文对提升分布式集群中深度学习模型的整体性能,进行合理地资源调度分析建模。本文基于这些分析和建立的分布式深度学习资源分配问题模型,考虑作业负载波动和作业性能干扰约束,提出分布式深度学习集群静态资源分配算法,包括计算复杂度分别为O(7)nl ogn(8)和O(7)n ~2l ogn(8)的贪婪算法和启发式算法。3.我们在静态资源分配基础上,新作业申请或作业完成等集群中资源的改变带来负载均衡问题,进一步研究相应的动态负载下启发式资源调整算法。算法中引入迁移机制来有效地解决作业与主机映射关系改变的问题。4.基于GPU集群中Tensor Flow深度学习框架,本文分别使用静态资源分配算法和动态负载调整算法进行了广泛的实验,验证本文方法的正确性、有效性和可调控性。结果显示了本文所提出的方法仅使用不到30%额外资源即可达到平均作业完成时间80%以上的性能。更多还原

【Abstract】 With the booming application and development of deep learning technology in various industries,sample data is rapidly increasing and deep neural networks are becoming increasingly complex.The computing and storage resources required for deep learning applications are rapidly increasing.How to efficiently use limited computing resources to adapt to the rapidly growing demand for deep learning applications has become an urgent problem to be solved.Distributed deep learning technology has become an important supporting technology for large-scale deep learning applications.This article focuses on how to efficiently use the resources of distributed deep learning clusters to improve the throughput of deep learning tasks and shorten the training time of deep learning tasks.At present,the research on resource allocation algorithms of distributed deep learning clusters is mostly based on traditional high-performance computing or cloud computing platform resource allocation methods.These methods often fail to fully consider the unique resource requirements of deep learning jobs,such as the fluctuation of job size demand caused by the periodicity of deep learning training and the mutual interference caused by job sharing resources such as GPU.This paper deeply researches the operating characteristics of the distributed deep learning cluster in the job sharing scenario,and carries out the research on the resource allocation algorithm of deep learning job based on opportunistic sharing.The main contributions of this article include:1.This paper studies the changes in the run-time performance of multiple model jobs that sharing resources in a distributed deep learning cluster.The observation and analysis of the iterative nature of the deep learning job and the use of historical data during the operation of the job on the cluster,and it is known that the resource demand of the job during the operation is volatile,then an opportunistic resource sharing model is established for this.In response to the frequent fluctuations in the memory usage of shared hosts,we explored the problem of resource contention between jobs that interfere with performance,and established a distributed deep learning resource allocation model based on the opportunistic resource sharing model.2.This paper aims to improve the overall performance of deep learning models in distributed clusters,and conduct reasonable resource scheduling analysis and modeling.Based on these analyses we establish the distributed deep learning resource allocation problem model.This paper considers job load fluctuations and job performance interference constraints,and proposes a distributed deep learning cluster static resource allocation algorithm,including greedy algorithms and heuristics algorithm with computational complexity of O(7)nl ogn(8)and O(7)n ~2l ogn(8).3.On the basis of static resource allocation,changes in resources in the cluster,such as new job application or job completion,bring load balancing problems,and we further study the corresponding heuristic resource adjustment algorithm under dynamic load.The migration mechanism is introduced in the algorithm to effectively solve the problem that the mapping relationship between the job and the host changes due to adjustment.4.Based on the Tensor Flow deep learning framework,extensive experiments were carried out on the static resource allocation algorithm and dynamic load adjustment algorithm in GPU clusters to verify the correctness,effectiveness and controllability of the method in this paper.The results show that the method proposed in this paper uses less than 30%of additional resources to achieve the performance of more than 80%of the average job completion time.更多还原