【作者】 李卫；

【导师】 李珂；

【作者基本信息】 电子科技大学 ， 计算机科学与技术， 2023， 硕士

【摘要】 近年来,视觉Transformer(ViT)凭借其全局信息捕捉能力在各个图像识别任务性能上取得了出色的表现,使其成为了计算机视觉领域学术界研究和工业界应用热点。然而目前几乎所有的视觉Transformer模型通常都依赖于大量的计算成本来实现高性能,使其难以部署到资源或算力有限的硬件设备上。因此,本文以研究视觉Transformer模型结构设计及其轻量化策略为目标,针对视觉Transformer模型参数量大、计算成本高、推理效率低和部署困难等问题,进行了视觉Transformer轻量化模型设计研究以及视觉Transformer和CNN混合模型设计研究。首先,为了解决视觉Transformer模型参数量、计算复杂度较高问题,本文研究了视觉Transformer轻量化模型设计,提出了深度可分离视觉Transformer模型(SepViT)。受启发于深度可分离卷积轻量化的思想,SepViT从Transformer自注意力的计算方式出发,首次提出了深度可分离自注意力,实现了在一个Transformer中窗口局部信息交流和窗口之间的全局信息交换。同时,为了高效地建模局部窗口之间的自注意力关系,SepViT设计了Window Token嵌入方案来学习每个局部窗口的全局特征表示,从而节省计算成本。此外,SepViT也借鉴了分组卷积思想将深度可分离自注意力进行扩展并设计了分组自注意力,实现了对多个局部窗口的信息捕捉并进一步提升了模型性能。最后,在分类、分割和检测等视觉任务上的大量实验表明,本文所提出的SepViT实现了性能和推理速度的最佳平衡。另一方面,纯ViT模型由于其低效的大矩阵运算较多而无法像CNN一样在各种硬件设备或Tensor RT、CoreML等推理框架下高效部署和应用,因此本文进行了视觉Transformer和CNN混合模型设计研究,从模型实际部署时的推理速度出发,提出了既具有ViT一样性能又具有CNN一样推理速度的下一代视觉Transformer模型(Next-ViT)。Next-ViT首先设计了下一代CNN模块和Transformer模块来分别实现图像特征的局部和全局信息建模。与此同时,为了更加高效地堆叠CNN和Transformer模块,Next-ViT提出了下一代Transformer和CNN混合策略,有效地提升了模型推理速度和下游任务的识别性能。在视觉的各种基准任务上的相关实验表明,本文所提出的Next-ViT既超越了前沿ViT模型性能又具有经典CNN模型一样的推理速度,非常有利于在不同的硬件或推理引擎上部署和应用。更多还原

【Abstract】 Recently,vision Transformer(ViT)has witnessed prevailing success in a series of vision tasks with its capability of capturing the global information,making it a hot topic in academic research and industrial application of computer vision.However,the existing ViTs often rely on extensive computational costs to achieve high performance,which is burdensome to deploy on resource-constrained or computing-power-constrained devices.Therefore,the thesis mainly researches on the architecture design and lightweight strategy of vision Transformer.Aiming at the issues of large number of parameters,high computing cost,low inference speed,difficult deployment,and etc,this thesis will study the lightweight architecture design of ViT and the hybrid design of ViT and CNN.Firstly,in order to relieve the problem of large parameters and high computational complexity of vision Transformer,this thesis firstly studies the lightweight architecture design of vision Transformer and propose a depthwise separable vision Transformer,abbreviated as SepViT.Inspired by the lightweight ideology of depthwise separable convolution,SepViT firstly proposes depthwise separable self-attention based on the computing of self-attention components in Transformer,which achieves the local-global information interaction within and among windows in sequential order in a single Transformer block.Meanwhile,in order to efficiently model the attention relationship among windows,SepViT also designs the Window Token embedding scheme to learn the global feature representation of each window with negligible cost.In addition,SepViT also draws lesson from grouped convolution and extends depthwise separable self-attention to grouped selfattention,which can establish long-range visual interactions across multiple windows and further improve the performance.Finally,extensive experiments on classification,segmentation,object detection tasks show that SepViT achieves the best trade-off between performance and latency.On the other hand,since the inefficient large matrix operation in Transformer,most existing ViTs can not perform as efficiently as CNNs on various hardware devices or inference frameworks,e.g.Tensor RT and CoreML.Therefore,the thesis studies the hybrid design of vision Transformer and CNN.From the perspective of inference speed during deployment,the thesis proposes next generation vision Transformer(Next-ViT),which infers as fast as CNN and performs as powerful as ViT.Next-ViT firstly designs the next convolution block(NCB)and next Transformer block(NTB)to achieve the local and global information interaction,respectively.At the same time,Next-ViT proposes the next hybrid strategy(NHS)to efficiently stack NCB and NTB,which also improves the inference speed and the performance of downstream tasks.Experiments on various benchmark visual tasks show that Next-ViT not only outperforms the recent ViTs,but also achieves the same inference speed as the famous CNNs,and it’s also deployment-friendly on different hardware devices or inference frameworks.更多还原