【作者】 王凯；

【导师】 范建平； 韩婷婷；

【作者基本信息】 杭州电子科技大学 ， 计算机技术， 2023， 硕士

【摘要】 视频时序定位旨在根据用户提供的文本描述,在一个完整的视频中定位出与用户描述语义最相关的特定视频片段。视频时序定位任务不仅连接了计算机视觉和自然语言处理两大领域,为高层的视觉任务提供理论基础,在现实场景中也有许多重要的应用需求,因此具有十分重要的研究和应用价值。目前,视频时序定位任务的难点主要体现在两个方面:1)现有的视频时序定位方法大多采用简单的线性模型进行多模态交互,由于不能获取深层次的语义相关性而影响模型最终的性能。2)现有的弱监督视频时序定位方法所采用的基于重构的模型优化方式,由于缺少更加真实、准确的监督信息从而导致模型性能仍有待进一步提高。针对这两个问题,本文开展了以下研究:针对问题1,本文提出了基于语义重构的弱监督视频时序定位方法。本方法旨在建立视频模态和文本模态之间的细粒度交互,生成更具表现性的多模态特征表示,提高视频时序定位的性能。本方法包含候选片段生成和语义重构两个模块。候选片段生成模块使用基于Transformer的编码器分别增强文本模态内部和视频模态内部的细粒度交互,使用基于Transformer的解码器对文本和视频之间跨模态的交互进行建模,生成更具表现性的候选片段特征表示。语义重构模块根据候选片段对文本查询进行重构,为语义重构效果好的视频片段赋予更高的分数,促使候选片段生成模块生成更加精准的视频片段,从而提升时序定位的精准度。针对问题2,本文提出了基于外部知识的对抗弱监督视频时序定位方法。本方法旨在从外部真实数据中学习文本-视频的跨模态对齐,继而为模型的训练提供更加真实、更具说服力的监督信号,提高视频时序定位的性能。本方法包含生成器和判别器两个模块。首先使用互联网上随处可见的真实片段-文本描述对判别器模块进行预训练,从而赋予其对文本模态和视频模态语义匹配程度的初步鉴别能力。然后,将判别器模块与生成器模块进行交互对抗地训练,进一步优化判别器模块对跨模态语义相似度的判别能力及生成器模块生成与文本查询具有更高语义相似度的候选片段的能力。本文提出的方法在Activity Net-Captions和Charades-STA基准数据集上进行了充分的实验。结果表明,本文所提出的方法能够充分利用视频和文本中所包含的信息,有效的提升弱监督视频时序定位的性能。更多还原

【Abstract】 The purpose of video temporal localization is to locate the specific video segment that is most relevant to the semantics of the user description in a complete video based on the text description provided by the user.The video temporal localization task not only connects the two fields of computer vision and natural language processing,providing theoretical basis for high-level visual tasks,but also has many important application requirements in real scenes,so it of great importance both in the research and application fields.At present,the difficulties of video temporal localization tasks are mainly reflected in two aspects: 1)Most of the existing video temporal localization methods adopt simple linear models for multi-modal interaction,and the final performance of the models will be affected due to the inability to obtain deep semantic correlation.2)Due to the lack of more real and accurate supervision information,the model performance of the model optimization based on reconstruction adopted by the existing weak supervision video temporal localization still needs to be further improved.To solve these two problems,this paper carried out the following research:For the first problem,this paper proposes a weak supervised video temporal localization method based on semantic reconstruction.This method aims to establish fine-grained interactions between video modes and text modes,generate more expressive multimodal feature representations,and improve the performance of video temporal localization.This method consists of two modules: candidate fragment generation and semantic reconstruction.The candidate fragment generation module uses a Transformer based encoder to enhance the fine-grained interaction within text modes and video modes respectively,and uses a Transformer based decoder to model the cross-modal interaction between text and video to generate a more expressive representation of candidate fragment characteristics.The semantic reconstruction module reconstructs the text query according to the candidate fragments,assigns higher scores to the video fragments with good semantic reconstruction effect,and prompts the candidate fragment generation module to generate more accurate video fragments,thus improving the accuracy of video temporal localization.For the second problem,this paper proposes an external knowledge based weakly supervised video temporal localization method.This method aims to learn cross-modal alignment of text-video from external data,providing real and more convincing supervision guidance for model training,and improve the performance of video temporal localization.This method consists of two modules: the generator and the discriminator.Firstly,for the discriminator module,this method firstly uses the real and matched clip-text description pair which can be seen everywhere on the Internet for pre-training,so as to give the discriminator module the initial ability of discriminating the semantic matching degree for the text modality and the video modality.Then,the discriminator module and the generator module are trained in an adversarial way to further improve its ability of differentiating the cross-modal semantic similarity of the discriminator and at the same time,promote the generator module to generate temporal localization proposals with higher semantic similarity.The proposed method has been fully tested on Activity Net-Captions and Charades-STA datasets.The experimental results show that the proposed method can make full use of the information contained in videos and texts,and effectively improve the performance of weak supervised video temporal localization.更多还原