【作者】 陈剑；

【导师】 刘金硕；

【作者基本信息】 武汉大学 ， 计算机软件与理论， 2017， 硕士

【摘要】 卷积神经网络是一种多阶段全局可训练的神经网络模型,最近这几年广泛使用在图像识别、自然语言处理等相关领域,具有模型简单、识别效率高等优点。卷积神经网络采用串行的方法来实现的时候,存在训练时间长,伸缩性差的缺点。随着解决问题的复杂化,需要规模更大,复杂度更高的训练数据,采用串行方法实现的卷积神经网络在处理海量数据的时候存在劣势,效率低下。本文的研究工作主要为卷积神经网络的多粒度并行优化策略。其中,本文介绍了基于分布式集群和GPU架构进行并行加速的方法。然后通过并行优化的结构来进行多粒度并行的卷积网络的训练。实验验证该优化结构具有良好的并行效果,相比较于单机串行训练的卷积神经网络和普通的集群并行架构,取得了良好的加速比结果,同时提升了并行效率。(1)采用分布式集群进行粗粒度的并行计算。利用MapReduce将整个训练集分成若干小的数据块,分别分发保存到每个节点中。每个节点保存一样的卷积神经网络模型,采用数据并行的方式每个节点使用预先划分的数据块进行训练,map任务进行前向传播计算和反向传播计算,计算结果是各个权值和偏置的局部变化值,reduce任务进行每个权值和偏置的局部变化值的汇总,最终结果得到全局变化值。若干次迭代之后,最终完成卷积神经网络的训练。(2)采用基于集群和GPU的多粒度并行优化框架。当进行粗粒度的网络密集型计算的时候,分布式集群的方式并行计算,可以分解大规模任务,分散数据集合;当进行细粒度的计算密集型并行计算时候,采用GPU通用计算方式进行并行计算。(3)对任务调度、负载均衡和数据传输通信开销方面进行过程分析以及提出相应的优化的策略。在任务调度过程中,采用任务调度线程池来对不同的操作任务进行管理,降低线程的创建开销和销毁频率,提升并行效率。负载均衡器监控系统当中的任务执行情况,对节点中剩余计算能力和提交请求的任务队列进行监控,然后进行决策。然后负载均衡器会指挥任务调度线程池来执行调度任务,达到负载均衡的效果。同时针对数据传输并行过程中传输通信时间的开销问题,采用计算时间来掩盖数据传输时间的方法,来减少数据传输时间对训练时间所产生的影响。更多还原

【Abstract】 Convolution neural network is a multi-stage global training neural network model.In recent years,it has been widely used in image recognition,natural language processing and other related fields.It has the advantages of simple model and high recognition efficiency.When it uses serial method to train,there is a long training time,poor flexibility of the shortcomings.With the complexity of solving the problem,it needs a larger and more complex training data and the convolution neural network implemented by the serial method is less efficient when dealing with massive data.The research work of this paper is mainly a multi-granularity parallel optimization strategy for convolution neural networks.First of all,this paper introduces a method of parallel acceleration based on distributed clustering and GPU architecture.Then,the parallel-optimized convolution network is trained by the parallel optimization structure.Finally,the experimental results show that the structure has good parallel effect.Compared with the convolution neural network of stand-alone serial training and the common cluster parallel architecture,this paper achieves a good acceleration ratio effect,while improving the parallel efficiency.(1)Distributed clustering is used to perform coarse-grained parallel computing.Using Mapreduce to divide the entire training set into several small data blocks,which are saved to each node.Each node holds the same convolution neural network model,which is trained in data-parallel method using pre-divided data blocks.Map task is responsible for forward propagation calculation and back propagation calculation and the result of the calculation is the local change of each weight and offset.Reduce task summarizes the localized values of each weight and offset,resulting in a global change.After several iterations,the training of the convolution neural network is completed.(2)In this paper,multi-granularity parallel optimization framework based on cluster and GPU is adopted.When performing coarse-grained network-intensive computing,it takes a distributed clustering approach to parallel computing which can break down large-scale tasks and disperse the data set.When performing fine-grained computationally intensive parallel computing,this paper uses GPU general-purpose computing in parallel.(3)It analyzes the communication overhead of task scheduling,load balancing and data transmission,and puts forward the corresponding optimization strategy.In the process of task scheduling,this paper manages the different operational tasks through the task scheduling thread pool which reduces the creation cost and the destruction frequency of the thread,and improves the parallel efficiency.The load balancer monitors the task execution in the system,monitors the remaining computing power in the node,and the task queue that submits the request,and then makes the decision.Then the load balancer will command the task scheduling thread pool to perform scheduling tasks,to achieve the effect of load balancing.Aiming at the overhead of transmitting communication time in the process of data transmission,this paper adopts the method of computing time to cover the data transmission time to reduce the influence of data transmission time on training time.更多还原