【作者】 安涛；

【导师】 王彬；

【作者基本信息】 西安理工大学 ， 计算机应用技术， 2018， 硕士

【摘要】 随着深度学习网络规模的增大,导致网络训练参数数量快速增长及网络训练时间随之增加,卷积神经网络作为深度学习的一部分,虽然利用权值共享降低了部分参数的数量,但是参数数量依然庞大,网络训练耗时过长问题依然存在。针对这个问题,本文实现了分布式环境下的卷积神经网络,并提出基于二叉树的异步参数更新算法(B-APU)和基于时间因素的FIFO调度策略(FIFO-T)对分布式卷积神经网络进行优化。主要工作如下:(1)实现多机环境下基于Spark网络集群的分布式卷积神经网络。通过与Spark平台的结合探索在多机环境下卷积神经网络的并行方案。利用Spark平台搭建分布式环境,并在其基础上实现分布式卷积神经网络。(2)针对分布式卷积神经网络在训练中同步数据并行方式存在集群节点等待的问题,设计了B-APU(Asynchronous Parameters Update Based Binary-tree)算法。B-APU算法将二叉树与同步和异步并行方式相结合,首先把网络节点按照计算机的处理速度构造出这样的二叉树:同一个子树的左右节点的值相差较小并且在二叉树中左子树节点是处理速度最快的节点,二叉树的上层是计算速度相对较差的节点,下层是速度较快的节点。其次,对于参数更新,同一层的节点采用参数平均法进行参数计算,保证准确率;不同层之间的使用异步随机梯度下降算法进行参数传递,提高训练速度;每一层参数的传递按照二叉树的深度遍历进行,确保速度快的节点优先进行参数传递,实现计算速度较快的节点的通信与较慢节点的计算并行,从而减少网络训练中网络通信时间。(3)Spark平台下,对于分布式卷积神经网络缺省的FIFO调度策略单纯考虑作业提交顺序带来的节点等待问题,设计了 FIFO-T(First In First Out Based Time)的作业调度策略。针对作业执行时间的差异而引起节点互相等待的问题,FIFO-T作业调度策略将作业执行时间作为作业调度判断优先级的因素之一,从而缩短由于作业执行时间的不一致而导致集群节点之间互相等待的时间。实验结果表明将B-APU算法和FIFO-T调度策略应用在分布式卷积神经网络的训练中取得了良好的效果。更多还原

【Abstract】 With the increase of depth learning network scale,the number of network training parameters increase rapidly and the network training time increases.The convolution neural network is a part of deep learning.Although the number of parameters is reduced by the use of weight sharing,the number of parameters is still large and the time-consuming problem of network training is still too long.Existence.In order to solve this problem,the convolution neural network in distributed environment is realized,and the asynchronous parameter updating algorithm(B-APU)based on binary tree and the time factor based FIFO scheduling strategy(FIFO-T)are proposed to optimize the distributed convolution neural network.The main work is as follows:(1)We realized distributed convolution neural network based on Spark network cluster under multi machine environment.Through the integration with Spark platform,we explore a parallel scheme of convolution neural network in multi machine environment.We realized distributed convolution neural network based on Spark platform.(2)We designed B-APU(Asynchronous Parameters Update Based Binary-tree)algorithm to solve the problem that the distributed convolution neural network has a cluster node waiting for synchronous data parallel in training.The B-APU algorithm combined the binary tree with synchronous and asynchronous parallel methods.First,the network nodes were constructed according to the processing speed of the computer.The difference between the values of the left and right nodes of the same subtree is small and the left subtree node is the fastest node in the binary tree,and the upper layer of the two tree is the calculation speed phase,and the lower layer is the faster node.Secondly,for the parameter updating,the nodes of the same layer used the parameter averaging method to ensure the accuracy.The asynchronous random gradient descent algorithm is used between different layers to improve the training speed,and the transmission of each layer is carried out according to the depth of the binary tree to ensure that the fast speed nodes have priority to transfer the parameters.The node communication with faster nodes is is in parallel with the calculation of slower nodes,thus reducing network communication time in network training.(3)We designed the FIFO-T(First In First Out Based Time)operation scheduling strategy for the FIFO scheduling strategy of the distributed convolution neural network,under the Spark platform,which simply considers the node waiting problem brought by the job submission order.In view of the difference in job execution time that causes nodes to wait for each other,the FIFO-T job scheduling strategy uses the job execution time as one of the factors to judge the priority of the job scheduling,thus shortening the time that the cluster nodes wait for each other because of the inconsistency of the execution time.The experimental results show that the B-APU algorithm and FIFO-T scheduling strategy are applying to the training of distributed convolution neural network achieves better results.更多还原