【作者】 李宁；

【导师】 钟必能；

【作者基本信息】 广西师范大学 ， 软件工程， 2025， 博士

【摘要】 视觉目标跟踪对动态环境感知及目标行为分析有着重要的作用,是人工智能视觉领域的核心任务之一,被广泛应用于智能交通、机器人导航、自动驾驶和智能安防监控等多个领域。近年来,基于深度学习的视觉目标跟踪算法取得了令人曙目的成绩,但在开放世界的复杂场景中,受到遮挡与干扰、外观变化、以及弱光或恶劣天气等挑战,现有基于深度学习的视觉目标跟踪算法在建模目标语义信息时依然面临着诸多问题,其中包括:（1）现有主流的视觉目标跟踪方法的骨干网使用的注意力机制是自下而上的,缺乏内在先验语义信息引导,导致对输入图像的全部物体的关注,而不是聚焦任务相关的物体,对于多物体复杂场景的跟踪无疑带来一定的性能限制。（2）现有单流结构跟踪器因其骨干网是非层次的,无法有效捕捉多尺度语义表示,而层次结构的骨干网又由于相对复杂的窗口操作而导致跟踪性能相对较低。此外,基于单流结构ViT的跟踪器往往只利用模型的最后一层输出进行结果预测,忽略了来自浅层的低级别语义的利用,缺乏多级别语义建模能力。（3）由于浅层高频语义建模的不足、历史时序语义信息利用不足等限制,当前视觉-语言跟踪器获取目标语义信息的能力尚未得到充分发挥。一方面,骨干网浅层模块的建模输出往往不直接参与跟踪结果的预测,造成一定程度的模型能力浪费;此外,历史跟踪结果的语义信息在跟踪过程中也未得到充分利用,导致语义辅助能力有所欠缺。（4）现有可见光-热红外跟踪方法存在多尺度语义信息捕捉不足的问题,且大多数方法依赖单个初始帧模板与当前帧的匹配,属于“图像级”跟踪,导致对时序上下文语义信息建模的欠缺。此外,现有方法以可见光模态为主导,红外模态为辅助,这种固定的“主-辅”分类,可能导致跨模态语义信息交互不足的问题。基于以上现有目标跟踪算法面临的四个问题,本文从语义信息增强的角度考虑,开展以下四个方面的研究工作:（1）提出基于自上而下与自下而上注意力相结合的目标跟踪算法,在原有注意力机制的基础上,加上生成的自适应先验信号,形成自下而上和自上而下注意力相结合的注意力机制,从而使得跟踪器充分利用目标先验语义信息,并更加关注任务相关的目标物体,进一步排除相似物与背景的干扰。性能上,所提出的跟踪器在GOT-10k测试数据集上的达到73.8%的成功率,在TNL2K测试数据集上达到56.7%的成功率,在大规模LaSOT_ext测试数据集上的成功率达到50.7%,分别比同期最先进的跟踪器OSTrack高出2.8%、2.41%和3.3%。同时,推理速度仍然能够保持在50 FPS左右。（2）提出基于多尺度和多级别表征学习的鲁棒多阶段目标跟踪算法,在有效结合层次结构和单流结构的ViT作为跟踪骨干的优势基础上,充分挖掘目标多尺度语义和多层级语义信息,提高模型多尺度和多层次的表示学习能力。在前两个阶段,利用双分支的层次结构使模型能够分别获得模板和搜索区域的多尺度特征。同时,设计了基于简单MLP层而不是复杂窗口操作的局部规模感知模块,获得轻量级跟踪阶段,避免了传统分层ViT等复杂操作导致的性能下降。在模型的第三阶段,通过利用单流结构的ViT来实现高效简洁的特征提取和交互,提高跟踪器的效率。最后,在主阶段设计了一个多级特征集成模块,显式利用从浅层学习到的表示信息,并将其与来自最终层的特征融合,从而获得多级表示信息。得益于上述设计,所提出的跟踪器可以有效地捕捉多尺度和多层次的表示。特别地,该算法在LaSOT测试数据集上成功率达到73.8%,在UAV123测试数据集上成功率为72%,取得了同期而言更为领先的成绩。同时保持了约47 FPS的实时速度。（3）提出基于语义信息增强的视觉-语言目标跟踪算法,在设计多尺度多阶段视觉-语言框架基础上,充分建模多层级深浅层语义信息和历史时序语义信息,进一步挖掘视觉可见光与语言文本之间的跨模态语义信息。首先,设计了一个多阶段视觉-语言跟踪框架,用于在视觉语言跟踪流程中对多尺度语义信息进行建模。其次,提出了一个选择性深浅语义信息融合模块,将浅层输出特征显式地融合到深层输出特征中,从而减少建模能力的浪费,并获取更多与目标相关的高频语义信息。最后,设计了一个基于语言分类和多帧历史信息的时间线索建模模块,旨在更全面地利用历史时间语义信息。通过这些设计提升了模型捕捉语义的能力。特别地,该算法在OTB99-Lang测试数据集上达到了72.2%的成功率,在WebUAV-3M测试数据集上达到了59.6%的成功率,击败了当时大多数主流的VL跟踪方法。这验证了所设计的跟踪算法及其模块的有效性和优越性。（4）提出融合多尺度感知与上下文感知的可见光-热红外目标跟踪算法,通过多模板帧时序建模,进一步有效利用全局上下文和历史语义信息;同时,通过设计多尺度分层分支跟踪框架实现对目标多尺度语义信息的利用,并通过引入跨模态双向交互机制与动态融合机制,进一步提升可见光与热红外两种模态特征交互的动态性与灵活性,高效捕捉跨模态交互语义信息,提升模型在在复杂红外场景中的识别精度。在LasHeR、RGBT210和RGBT234等三个主流视觉-红外目标测试数据集上的实验结果（例如,所提出的跟踪方法在LasHeR数据集上取得73%的跟踪精度,在RGBT210测试集上取得87.7%的跟踪精度等）表明,所提出的算法取得了较为鲁棒的和具有竞争力的跟踪效果。更多还原

【Abstract】 Visual object tracking plays a crucial role in dynamic environmental perception and target behavior analysis.It is one of the core tasks in the field of artificial intelligence vision and is widely applied in various domains such as intelligent transportation,robot navigation,autonomous driving,and intelligent security surveillance.In recent years,visual object tracking algorithms based on deep learning have achieved remarkable results.However,in complex scenarios of the open world,due to challenges such as occlusion and interference,appearance changes,as well as low light or adverse weather conditions,existing visual object tracking algorithms based on deep learning still face numerous issues when modeling target semantic information.These issues include:(1)The attention mechanism used in the backbone network of existing mainstream visual object tracking methods is bottom-up,lacking intrinsic prior semantic information guidance.This leads to a focus on all objects in the input image rather than task-related objects,undoubtedly imposing certain performance limitations on tracking in complex multi-object scenarios.(2)Existing one-stream structure trackers,due to their non-hierarchical backbone networks,cannot effectively capture multi-scale semantic representations.Meanwhile,hierarchical backbone networks suffer from relatively low tracking performance due to the relatively complex window operations.Furthermore,trackers based on one-stream structure ViT often only utilize the output of the last layer of the model for result prediction,neglecting the utilization of low-level semantics from the shallow layers and lacking multi-level semantic modeling capabilities.(3)Due to limitations such as insufficient modeling of shallow high-frequency semantics and under-utilization of historical temporal semantic information,the current visual-language tracker’s ability to obtain target semantic information has not been fully utilized.On the one hand,the modeling output of the shallow modules in the backbone network often does not directly participate in the prediction of tracking results,resulting in a certain degree of waste of model capabilities.Furthermore,the semantic information from historical tracking results is also not fully utilized during the tracking process,leading to a lack of semantic assistance capabilities.(4)The existing visible light-thermal infrared tracking methods suffer from the problem of insufficient capture of multi-scale semantic information.Moreover,most methods rely on matching a single initial frame template with the current frame,which belongs to"image-level"tracking,leading to a lack of modeling for temporal contextual semantic information.In addition,existing methods prioritize the visible light modality as the primary one,with the infrared modality serving as an auxiliary one.This fixed"primary-auxiliary"classification may lead to insufficient cross-modality semantic interaction.To address the aforementioned four issues faced by object tracking algorithms,we explore the following four aspects of research from the perspective of semantic information enhancement:(1)A object tracking algorithm combining top-down and bottom-up attention is proposed.Based on the original attention mechanism,an adaptive prior signal is generated to form an attention mechanism that integrates both bottom-up and top-down attention.This enables the tracker to fully utilize the prior semantic information of the target and pay more attention to task-related target objects,further eliminating interference from similar objects and background.In terms of performance,the proposed tracker achieves a success rate of 73.8%on the GOT-10k test dataset,56.7%on the TNL2K test dataset,and 50.7%on the large-scale LaSOText test dataset,which are 2.8%,2.41%,and 3.3%higher than the state-of-the-art tracker OSTrack,respectively.Meanwhile,the inference speed remains at around 50 FPS.(2)A robust multi-stage object tracking algorithm based on multi-scale and multi-level representation learning is proposed.By effectively combining the advantages of hierarchical and one-stream structures in ViT as the backbone of tracking,it fully exploits the multi-scale and multi-level semantic information of the target,enhancing the model’s ability to learn multi-scale and multi-level representations.In the first two stages,a dual-branch hierarchical structure is utilized to enable the model to obtain multi-scale features of templates and search regions,respectively.Simultaneously,a local scale-aware module based on a simple MLP layer rather than complex window operations is designed to achieve a lightweight tracking stage,avoiding performance degradation caused by complex operations such as traditional hierarchical ViT.In the third stage of the model,one-stream structure ViT is employed to achieve efficient and concise feature extraction and interaction,thereby enhancing the tracker’s efficiency.Finally,a multi-level feature integration module is designed in the main stage to explicitly utilize the representation information learned from shallow layers and fuse it with features from the final layer,thus obtaining multi-level representation information.Thanks to the aforementioned design,our tracker can effectively capture multi-scale and multi-level representations.Specifically,the algorithm achieves a success rate of 73.8%on the LaSOT test dataset and 72%on the UAV123 test dataset,achieving more advanced results compared to contemporaneous methods.Meanwhile,the proposed tracker maintains a real-time speed of approximately47 FPS.(3)A visual-language object tracking algorithm based on semantic information enhancement is proposed.Based on the design of a multi-scale and multi-stage visual-language framework,it fully models multi-level deep and shallow semantic information as well as historical temporal semantic information,further exploring cross-modal semantic information between visual visible light and language text.Firstly,a multi-stage visual-language tracking framework is designed to model multi-scale semantic information in the visual-language tracking process.Secondly,a selective deep-shallow semantic information fusion module is proposed,which explicitly integrates shallow output features into deep output features,thereby reducing the waste of modeling capabilities and obtaining more high-frequency semantic information related to the target.Finally,a temporal cue modeling module based on language classification and multi-frame historical information is designed,aiming to utilize historical temporal semantic information more comprehensively.These designs enhance the model’s ability to capture semantics.Specifically,the algorithm achieved a success rate of 72.2%on the OTB99-Lang test dataset and 59.6%on the Web UAV-3M test dataset,beating most mainstream VL tracking methods at that time.This verifies the effectiveness and superiority of the designed tracking algorithm and its modules.(4)A visible light-thermal infrared target tracking algorithm that integrates multi-scale perception and context awareness is proposed.Through multi-template frame temporal modeling,global context and historical semantic information are further effectively utilized.Simultaneously,a multi-scale hierarchical branch tracking framework is designed to leverage multi-scale semantic information of the target.By introducing a cross-modal bidirectional interaction mechanism and a dynamic fusion mechanism,the dynamic and flexibility of feature interaction between visible light and thermal infrared modalities are further enhanced,efficiently capturing cross-modal interaction semantic information and improving the model’s recognition accuracy in complex infrared scenes.Experimental results on three mainstream visual-infrared object tracking test datasets,namely LasHeR,RGBT210,and RGBT234(For example,the proposed tracking method achieves a tracking accuracy of 73%on the LasHeR dataset and earns a tracking precision of 87.7%on the RGBT210 testing set),demonstrate that the algorithm we proposed achieves relatively robust and competitive tracking performance.更多还原