【作者】 谭涛；

【导师】 尹世群；

【作者基本信息】 西南大学 ， 计算机应用技术， 2020， 硕士

【摘要】 人工智能技术经过多年的积累变得日益成熟,其应用领域也不断扩大。其中基于神经网络的深度学习技术由于其突出的效果而成为了这一领域中的研究热点。在深度学习中,卷积神经网络的性能往往取决于其模型结构和学习算法。在卷积神经网络模型结构确定的前提下,各神经元之间连接的网络参数会直接决定模型的最终性能。作为调节模型参数的最基本学习算法,随机梯度下降（Stochastic Gradient Descent,SGD）已经成为深度学习实际工程应用中必不可少的一部分。根据深度学习的底层运行方式,SGD对卷积神经网络的模型参数具有串行和并行两种计算方式。通过对SGD的分析,发现SGD主要存在以下两个问题:第一,在SDG的串行计算中,其学习率是固定的,选择一个合适的学习率对SGD来说是很困难的。当选择的学习率太小时,算法的收敛速度会很慢;当选择的学习率太大时,算法容易导致模型参数在迭代过程中发生大幅度的振动,甚至导致模型不收敛。第二,在SGD的并行计算中,有同步运行和异步运行两种方式。相比于同步运行,异步并行具有更加快的运行速度。但是,在异步随机梯度下降（Asynchronous Stochastic Gradient Descent,ASGD）中存在梯度延迟的问题,梯度延迟会影响模型的收敛速度和准确率,严重时会让模型在某个特定点的更新上出现大幅度抖动的现象,甚至让模型不收敛,导致整个训练过程出错。针对SGD串行计算中存在的学习率选择困难问题,本文给出一种基于卷积神经网络的自适应学习率优化算法;针对SGD并行计算中存在的梯度延迟问题,本文给出一种基于卷积神经网络的梯度延迟优化算法;最后对这两种优化算法的有效性进行实验验证。本文的主要研究内容如下:1.给出一种基于卷积神经网络的自适应学习率优化算法。本文通过对SGD串行计算中模型参数更新公式的分析,发现SGD串行计算中存在学习率选择困难的问题。针对这一问题,本文给出ACADG算法,其是一种加速收敛的自适应学习率优化算法。ACADG算法的基本思想就是:在模型迭代过程中按照g_t-1g_t（其中g表示梯度,t表示迭代步数）的正负分为两个情况进行讨论,并对应使用不同的算法对模型参数进行更新。最后,通过对比经典的优化算法Adam和Amsgrad,发现本文给出的ACADG算法在模型的收敛性、收敛速度以及准确率方面均较优,在模型参数的串行更新中能够达到自适应调整学习率的效果。2.给出一种基于卷积神经网络的梯度延迟优化算法。本文通过对典型的SGD并行计算算法ASGD中模型参数更新公式的分析,发现ASGD算法中存在梯度延迟的问题。针对这一问题,本文给出DASGD算法,其是一种动态调整梯度延迟的异步随机梯度下降优化算法。DASGD算法的基本思想就是:根据每个worker中参数梯度的延迟度来动态计算梯度延迟项和动量项的权重,以达到动态调节梯度延迟的效果。最后,通过对比典型的异步算法ASGD和MDCASGD,发现本文给出的DASGD算法对梯度延迟具有更强的处理能力,在模型的准确率、损失值以及高延迟情况下的收敛性方面均较优,在模型参数的异步更新中能够有效解决梯度延迟的问题。3.给出验证自适应学习率优化算法和梯度延迟优化算法有效性的多组对比实验。针对自适应学习率优化算法,给出ACADG算法在合成损失函数、Mnist数据集、Cifar10数据集上的三组实验,并选取Adam算法和Amsgrad算法作为优化算法的对比对象。针对梯度延迟优化算法,给出DASGD算法在Cifar10数据集、Tiny-ImageNet数据集上的两组实验,并选取ASGD算法和MDCASGD算法作为优化算法的对比对象。经过多组对比实验以及实验结果的分析,具体结果如下:相比于Adam算法和Amsgrad算法,本文给出的自适应学习率优化算法ACADG在收敛性、收敛速度以及准确率这三个方面都是较优的。并且在Mnist测试数据上,ACADG算法使用CNN模型的准确率比Amsgrad算法和Adam算法的分别高3.12%和2.81%;在Cifar10测试数据集上,ACADG算法使用CNN模型的准确率比Amsgrad算法和Adam算法的分别高15.59%和1.99%。相比于ASGD算法和MDCASGD算法,本文给出的梯度延迟优化算法DASGD在准确率、损失值以及高延迟情况下的收敛性这三方面都是较优的。并且在Cifar10测试数据集上,DASGD算法使用Le Net5模型在worker数量为12时的准确率比MDCASGD算法的高3.736%;在Tiny-Image Net测试数据集上,DASGD算法使用Vgg16模型在worker数量为6时的top＿1准确率比MDCASGD算法的高3.525%。因此,本文给出的两个随机梯度下降优化算法对卷积神经网络模型参数的调整都是有效的,其不仅可以优化卷积神经网络模型的收敛速度、准确率、梯度延迟等相关性能,还可以有效帮助深度学习在人工智能中的进一步应用。更多还原

【Abstract】 After years of accumulation,artificial intelligence technology has become increasingly mature,and its application fields have also been continuously expanded.Among them,deep learning technology based on neural network has become a research focus in this field due to outstanding effects.In deep learning,the performance of a convolutional neural network often depends on its model structure and learning algorithm.Under the premise of determining the structure of the convolutional neural network model,the network parameters connected between each neuron will directly determine the final performance of the model.As the most basic learning algorithm for adjusting model parameters,Stochastic Gradient Descent（SGD）has become an indispensable part of practical engineering applications for deep learning.According to the underlying operation of deep learning,SGD has serial and parallel calculation methods for model parameters of convolutional neural network.Through the analysis of SGD,it is found that SGD mainly has the following two problems:Firstly,in the serial calculation of SDG,its learning rate is fixed,and it is very difficult for SGD to choose an appropriate learning rate.When the learning rate is too small,the algorithm’s convergence speed will be very slow.When the learning rate is too large,SGD will easily cause large-scale vibration of the model parameters during the iteration process,and even cause the model to not converge.Secondly,in the parallel calculation of SGD,there are two ways of running synchronously and asynchronously.Compared to synchronous parallelism,asynchronous parallelism has a faster running speed.However,there is a problem with gradient delay in Asynchronous Stochastic Gradient Descent（ASGD）.The gradient delay will affect the model’s convergence speed and accuracy,and in severe cases,it will cause the model to update significantly at a specific point,and even makes the model not converge,leads to the error in the entire training.In view of the difficulty of selecting learning rate in SGD serial computing,this paper gives an adaptive learning rate optimization algorithm based on convolutional neural network.In view of the problem of gradient delay in SGD parallel computing,this paper gives a gradient delay optimization algorithm based on convolutional neural network.Moreover,this paper performs experimental verification to the effectiveness of these two optimization algorithms.The main research contents of this paper are as follows:1.Give an adaptive learning rate optimization algorithm based on convolutional neural network.This paper analyzes the model parameter update formula in serial SGD and finds that the problem of selecting the learning rate is difficult in serial SGD.In view of this problem,this paper presents the ACADG algorithm,which is an adaptive learning rate algorithm to accelerate convergence.In the process of model iteration,ACADG algorithm are divided into two situations for discussion according to the positive and negative ofg_t-1g_t（where g is the gradient,t is the step）,and corresponding to the use of different algorithms to update the model parameters.Moreover,by comparing the Adam algorithm and the Amsgrad algorithm,it is found that the ACADG algorithm is the best in terms of model convergence,convergence speed,and accuracy.Therefore,ACADG can achieve the effect of adaptive adjusting the learning rate in the serial updating of model parameters.2.Give a gradient delay optimization algorithm based on convolutional neural network.This paper analyzes the model parameter update formula in the ASGD algorithm and finds that there is a gradient delay problem in the ASGD algorithm.In view of this problem,this paper presents the DASGD algorithm,which is a asynchronous stochastic gradient descent algorithm for dynamically adjusting the stale gradient.The basic idea of the DASGD algorithm is to dynamically calculate the weight of the gradient delay term and the momentum term according to the delay degree of the parameter gradient in each worker,so as to achieve the effect of dynamically adjusting the gradient delay.Moreover,by comparing the ASGD algorithm and the MDCASGD algorithm,it is found that the DASGD algorithm has a stronger processing capacity for gradient delay,and is the best optimization algorithm in terms of model accuracy,loss value,and convergence under high delay conditions.Therefore,DASGD can solve the problem of gradient delay in the asynchronous updating of model parameters.3.Give some comparative experiments to verify the effectiveness of the adaptive learning rate optimization algorithm ACADG and the gradient delay optimization algorithm DASGD.Aiming at the adaptive learning rate optimization algorithm ACADG,this paper gives three experiments of the ACADG algorithm on the synthetic loss function,the Mnist data set,and the Cifar10 data set,and the Adam algorithm and the Amsgrad algorithm are selected as comparison objects of the optimization algorithm.For the gradient delay optimization algorithm DASGD,this paper gives two experiments of the DASGD algorithm on the Cifar10 dataset and the Tiny-ImageNet dataset,and the ASGD algorithm and the MDCASGD algorithm are selected as the comparison objects of the optimization algorithm.According to the comparative experiments and the analysis of experimental results,the specific results are as follows:Compared with Adam algorithm and Amsgrad algorithm,the adaptive learning rate optimization algorithm ACADG given in this paper is the best in three aspects of convergence,convergence speed,and accuracy.And on the Mnist test data,the accuracy of the ACADG algorithm using the CNN model is 3.12%and 2.81%higher than that of the Amsgrad algorithm and the Adam algorithm,respectively;on the Cifar10 test data set,the accuracy of the ACADG algorithm using the CNN model is 15.59%and 1.99%higher than that of the Amsgrad algorithm and Adam algorithms,respectively.Compared with the ASGD algorithm and the MDCASGD algorithm,the gradient delay optimization algorithm DASGD given in this paper is the best in three aspects of accuracy,loss,and convergence under high latency.And on the Cifar10 test data set,the accuracy of the DASGD algorithm with the LeNet5 model is 3.736%higher than that of the MDCASGD algorithm when the number of workers is 12;on the Tiny-ImageNet test data set,the top＿1 accuracy of the DASGD algorithm with the Vgg16 model is3.525%higher than that of the MDCASGD algorithm when the number of workers is 6.Therefore,the two stochastic gradient descent optimization algorithms given in this paper are effective in adjusting the parameters of the convolutional neural network model,which can not only optimize the convergent speed,accuracy,gradient delay and other related performance of the convolutional neural network model,but also effectively help the further application of deep learning in artificial intelligence.更多还原