【作者】 张良；

【导师】 贾广岩； 张波；

【作者基本信息】 山东大学 ， 金融数学与金融工程， 2022， 硕士

【摘要】 参数优化是深度学习中十分重要的一环,它通过模型损失函数来训练网络中的参数,使损失函数达到最小从而获得网络最优参数。深度学习网络的参数对模型的成功至关重要,决定了网络拟合的准确性,而优化算法是参数训练的核心,一个好的优化算法能够帮助网络更快更准确地找到最优参数,从而提升模型的准确性和效率。一阶优化算法可以分为两类:随机梯度下降法和自适应梯度算法。随机梯度下降法是较早提出的一类方法,在许多领域得到了广泛的应用,但随机梯度下降法对学习率保持固定,在参数训练的早期阶段容易陷入局部最优,效率低下。以Adam算法为代表的自适应梯度算法,其通过梯度的二阶矩动量缩放学习率,能够实现深度神经网络的快速训练,在机器学习中取得了重大成功。虽然Adam算法前期效率高,但是其训练的参数在测试集上表现不佳,相较于随机梯度下降法泛化能力较差。为此,我们在本文中提出一种新的优化算法,称为AdamX算法。该算法在实现快速训练的同时,泛化能力大大提高。AdamX优化器成功的关键因素在于添加一个超参数,它利用梯度和梯度的一阶动量协调控制学习步长,在保证自适应算法高效的同时提高优化器的泛化性。此外,本文在优化器迭代后期采用三种方法进一步控制学习率,使得参数训练后期更加稳定。在在线学习框架的基础上,本文对损失函数是凸函数的情况下关于AdamX算法对参数迭代的收敛性给出了证明。根据分析可以发现,AdamX算法获得的参数迭代点与最优参数迭代点之间的后悔值上界为O((?)),可以认为在损失函数是凸函数情况下AdamX算法对参数的训练误差是有上界的。本文将AdamX优化器应用于字符图像识别、语言处理和金融领域,采用残差神经网络和LSTM两种经典的深度神经网络,来测试AdamX优化器的性能,并与近期提出的几种自适应优化器进行对比。从实验数据和实验结果上看,AdamX优化器在保证前期参数快速训练的同时,提高模型在测试集上的准确性。对于不同的数据集和深度网络,AdamX优化器始终具有很好的性能和稳定性。相比较于Adabelief和AdaMomentum等其他自适应改进算法,AdamX优化器泛化性提升更多,在测试集上准确率更高。可以认为,作为一种改进的自适应优化算法,AdamX是一个有效的优化器,可以应用于多种任务和深度学习模型中。更多还原

【Abstract】 Parameter optimization is a very important part of deep learning.It trains the network parameters through the loss function of the model and minimizes the loss function so as to obtain the best parameters of the network.The parameters of deep learning network are the key to the model,which determines the accuracy of network fitting.Optimization algorithm is the core of parameter training,and a good optimization algorithm can help the network find the optimal parameters faster and more accurately,so as to improve the accuracy and efficiency of the model.First-order optimization algorithms can be divided into two categories:stochastic gradient descent and adaptive gradient optimization algorithm.Stochastic gradient descent is one of the earliest methods,which is widely used and still used in some fields.However,stochastic gradient descent methods keeps the learning rate fixed,so it is easy to fall into local optimum in the early stage of parameter training,which is inefficient.The adaptive gradient methods represented by Adam algorithm can achieve fast training of deep neural networks by scaling the learning rate of the second moment momentum of the gradient,and has achieved great success in machine learning.Although Adam algorithm has high efficiency in the early stage,its trained parameters perform poorly in the test set,and its generalization ability is poor compared with stochastic gradient descent method.In this paper,we propose a new optimization algorithm called AdamX algorithm.The algorithm can achieve fast training and improve the generalization ability greatly.The key to the success of the AdamX optimizer is the addition of a hyperparameter,which uses the gradient and the first-order momentum coordination of the gradient to control the learning step size and improve the generalization of the optimizer while ensuring the efficiency of the adaptive algorithm.In addition,this paper adopts three methods to further control the learning rate in the late iteration of the optimizer,so as to make the late parameter training more stable.On the basis of online learning framework,this paper proves the convergence of AdamX algorithm for parameter iteration when the loss function is convex.According to the analysis,it can be found that the regret value between a series of parameter iteration points obtained by AdamX algorithm and the optimal parameter iteration points is O((?)).It can be considered that the training error of AdamX algorithm on parameters is upper bounded when the loss function is convex.In this paper,AdamX optimizer is applied to the fields of character image recognition,language processing and finance.Two classical deep neural networks,residual neural network and LSTM,are used to test the performance of AdamX optimizer and compare with several recently proposed adaptive optimizers.According to the experimental data and results,AdamX optimizer can improve the accuracy of the model on the test set while ensuring the fast training of the early parameters.In the face of different data sets and deep networks,AdamX optimizer always shows advanced performance and excellent stability.Compared with other adaptive improved algorithms such as Adabelief and AdaMomentum,AdamX optimizer has more improvement in generalization and higher accuracy on the test set.It can be considered that AdamX,as an improvement of the adaptive algorithm,is an effective and excellent optimizer that can be applied to various tasks and deep learning models.更多还原