【作者】 张浩；

【导师】 王泉；

【作者基本信息】 西安电子科技大学 ， 计算机科学与技术， 2024， 硕士

【摘要】 随着人工智能的发展,神经网络以其在处理非结构化数据时能够自动提取和表现特征的强大能力,在计算机视觉、自然语言处理、机器人技术等领域取得了显著的成绩。面对各种实际应用场景,人们设计了多种高效的神经网络结构,但这一过程往往需要大量的专业知识和人力成本。为了解决这一问题,自动化的神经网络结构设计吸引了学术界和工业界的广泛关注,神经网络结构搜索(NAS)成为了这一领域的主流技术。可微分神经网络结构搜索(DARTS)将离散的体系结构搜索空间松弛为连续的,从而利用梯度来优化体系结构参数来搜索网络结构,具有搜索时间短、搜索效率高的优点成为现今NAS的主流方法。然而,搜索空间的连续松弛和可微分的搜索策略在利用梯度下降完成网络结构搜索的同时也致使DARTS算法存在一系列的固有缺陷。如显存需求高、搜索后期跳跃连接过多以及搜索结果与评估结果排名相关度低等问题。因此,本论文旨在对可微分神经网络结构搜索算法进行优化,弥补其自身缺陷,提升可微分神经网络结构搜索算法性能和稳定性。首先,对于可微分神经网络结构搜索中对显存需求太高的问题,本论文提出了基于最优特征图估计的网络搜索算法与相匹配的二次结构选择策略。该方法通过减少可微分神经网络结构搜索中每两个节点间可选操作数量来降低对显存的需求,从而增加了体系构建单元的中间节点数量,因此能够搜索到更复杂、性能更优秀的神经网络结构。其次,对于搜索后期跳跃连接过多导致的搜索性能急剧下降问题,本论文设计了辅助跳跃连接和相应的权重映射函数。辅助跳跃连接分离出跳跃连接既可以单独作为可选操作的作用,又可以与其他操作形成残差模块的作用。权重映射函数在训练中动态调整两种作用的权重大小,简单高效地解决了可微分神经网络结构搜索中极易出现的“搜索性能塌缩”问题。以上两点改进构成的基于最优特征图估计的搜索策略在CIFAR10数据集上以极少的显存需求(5个中间节点仅需8G的显存)以及搜索用时(5个中间节点仅需1.0 GPU/天)实现了97.42%的预测精度,比基础DARTS算法提高了0.52%,在Image Net以及Tiny Image Net数据集上分别实现了76.4%和62.06%的预测精度,证明该算法搜索到的体系结构拥有更强的泛化能力。最后,对于搜索过程中网络结构权重相互影响导致的搜索阶段网络性能排名与真实网络性能排名不一致问题,本论文提出了基于梯度匹配的超网分割算法。梯度匹配分数是以共享边缘的结构权重在选取不同操作时计算梯度的余弦距离与欧式距离,并通过自适应权重映射函数给予其合理的权重并相加得到。以梯度匹配分数来衡量不同子网络在训练过程中的动态匹配度,将动态匹配度高的子网划分到同一子超网中。对子网络分割成数个子超网,子超网内部网络结构权重在训练过程梯度动态变化更相似,从而降低了网络结构权重共享的危害。基于梯度匹配的超网分割算法在CIFAR10数据集上以97.63%的分类准确度实现了比超网随机分割算法更好性能。同时,基于梯度匹配的超网分割算法与基于最优特征图估计的搜索策略相结合,以略微增加搜索用时为代价(5节点下仅需1.6GPU/天),在基准数据集上分类准确度达到了98.58%,实现了最先进的性能。综上,本文对可微分神经网络结构搜索算法进行优化,实现了以更低的硬件资源开销搜索到了更优秀的神经网络结构,使得神经网络结构搜索能够在日渐复杂的任务以及计算资源有限的嵌入式设备中得到更广泛的应用。更多还原

【Abstract】 With the development of artificial intelligence,neural networks have made remarkable achievements in computer vision,natural language processing,robotics,and other fields with their powerful ability to automatically extract and represent features when dealing with un-structured data.Facing various practical application scenarios,a variety of efficient neural network structures have been designed,but this process often requires a great deal of spe-cialized knowledge and human cost.To solve this problem,automated neural network struc-ture design has attracted extensive attention from both academia and industry,and neural network structure search(NAS)has become a mainstream technology in this field.Differ-entiable neural network structure search(DARTS)relaxes the discrete architecture search space into a continuous one,so as to use the gradient to optimize the architecture parameters to search the network structure,which has the advantages of short search time and high search efficiency to become the mainstream method of NAS nowadays.However,the continuous relaxation of the search space and the differentiable search strategy,while utilizing gradient descent to complete the network structure search,also lead to a series of inherent defects in the DARTS algorithm.Problems such as high video memory requirements,too many jump connections late in the search,and low correlation between search results and evalua-tion result rankings.Therefore,this thesis aims to optimize the differentiable neural network structure search algorithm,make up for its own defects,and improve the performance and stability of the differentiable neural network structure search algorithm.First,for the problem of too high demand for video memory in the structural search of differentiable neural networks,this thesis proposes a network search algorithm based on the estimation of the optimal feature graph with a matching secondary structure selection strategy.The method reduces the demand for video memory by reducing the number of optional operations between every two nodes in the search for differentiable neural network structures,thus increasing the number of intermediate nodes in the system building unit,and therefore enabling the search for more complex neural network structures with better performance.Secondly,for the problem of sharp decrease in search performance caused by too many jump connections at the late stage of search,this thesis designs auxiliary jump connections and corresponding weight mapping functions.The auxiliary jump connection separates the roles of jump connections that can either act as optional operations alone or form a residual module with other operations.The weight mapping function dynamically adjusts the weights of the two roles during training,which solves the problem of”search performance collapse”that occurs easily in the structural search of differentiable neural networks in a simple and efficient way.The search strategy based on optimal feature graph estimation with the above two improvements achieves 97.42%prediction accuracy on the CIFAR10 dataset with a very small memory requirement(only 8G of memory for 5 intermediate nodes)and search time(only 1.0 GPU/day for 5 intermediate nodes),which is 0.52%higher than the basic DARTS algorithm,and has stronger generalization ability.Prediction accuracies of 76.4%and 62.06%are achieved on Image Net and Tiny Image Net datasets,respectively,demonstrating that the algorithm searches for architectures with stronger generalization capabilities.Finally,for the problem of inconsistency between the network performance ranking in the search phase and the real network performance ranking caused by the interaction of net-work structure weights during the search process,this thesis proposes a super-network seg-mentation algorithm based on gradient matching.The gradient matching score is obtained by calculating the cosine distance and Euclidean distance of the gradient with the struc-tural weights of the shared edges when different operations are selected,and giving them reasonable weights and summing them up through the adaptive weight mapping function.The gradient matching score is used to measure the dynamic matching degree of different sub-networks during the training process,and the sub-networks with high dynamic matching degree are divided into the same sub-supernet.By partitioning the subnetwork into several sub-supernet,the network structure weights within the sub-supernet are more similar to the gradient dynamic changes during the training process,which reduces the harm of sharing the network structure weights.The gradient matching-based supernet segmentation algo-rithm achieves better performance than the supernet random segmentation algorithm with97.63%classification accuracy on the CIFAR10 dataset.Meanwhile,the gradient matching-based hypergrid segmentation algorithm combined with the search strategy based on optimal feature graph estimation achieves state-of-the-art performance with 98.58%classification ac-curacy on the benchmark dataset at the cost of a slight increase in the search time(only 1.6GPUs/day for 5 nodes).In summary,the optimization of the differentiable neural network structure search al-gorithm in this paper achieves a better neural network structure search with lower hardware resource overhead,enabling a wider application of neural network structure search for in-creasingly complex tasks as well as embedded devices with limited computational resources.更多还原