【作者】 王健；

【导师】 蔡海滨；

【作者基本信息】 华东师范大学 ， 软件工程， 2023， 硕士

【摘要】 作为连接视觉和语言的桥梁,关系检测是一个能对视觉内容有更全面理解的方式。关系检测需要很好地概括和融合不同物体之间的视觉特征、语意信息、运动信息等,但是传统的神经网络存在一些不足之处,例如在处理序列数据时往往需要固定长度的输入,无法处理可变长度的序列,对于视频中长期依赖关系的建模也存在一定的困难。传统神经网络中信息的传递是通过固定的连接方式进行的,无法考虑节点之间的复杂关系,同时在处理长序列数据时,容易出现梯度消失或爆炸等问题。此外,视频数据增加了时间维度,如何利用好时域信息,进行正确高效的视频关系检测成了近年来研究的热点。以前的工作大多采用卷积或循环神经网络进行视频视觉关系检测,这些方法不能很好地捕捉长时间序列的时空信息,也存在效率低下等问题。经过上述问题,本文的主要研究工作可分为以下三点:1.提出一种基于Transformer的视频关系检测算法。由空间编码器和时间解码器两个核心模块组成的Transformer网络Vrd Tran,可以更好捕捉物体多帧间的时空依赖性并推理动态关系。实验证明了Vrd Tran方法对于强化时域信息来进行视频关系检测的有效性。2.提出一种基于图注意力的视频关系检测算法。由关系生成网络和注意力图卷积构成了ST-a GCN模型。ST-a GCN使用关系图生成网络将关系三元组构造成具有空间信息的关系图,并使用基于注意力的图卷积网络来融合多帧关系图的时空上下文信息。实验证明了图结构对于关系的推理帮助,提出的ST-a GCN模型在视频关系检测任务中的表现要优于基准的视频关系检测算法。3.提出一种基于图Transformer的视频关系检测算法。综合利用了图结构对于局部关系推理和Transformer对于捕获视频全局信息的优势,设计了图Transformer网络VGTran,其中包括空间编码器,关系图生成、位置编码和图时间解码器模块。并在实验中证明了Transformer网络处理图结构数据的可行性,并验证了VGTran优于其他基准视频关系检测算法的性能。基于注意力机制的模型可以更好地处理长序列数据。本文通过引入注意力机制,让模型能自适应地选择感兴趣的图像区域或特征,更准确地完成任务。基于注意力机制的Transformer和图卷积,可以动态地计算输入数据的权重分布,从而处理可变长度的序列数据。此外,利用好关系数据图的结构可以有效地计算节点之间的权重,从而更好地考虑节点之间的关系。实验证明综合利用图结构优势和注意力机制可以更加高效地进行视频关系的推理检测。更多还原

【Abstract】 As a bridge between vision and language,relational detection is a way to gain a more comprehensive understanding of visual content.Relational detection requires a good generalisation and integration of visual features,semantic information and motion information between different objects,but traditional neural networks have some shortcomings,for example,they often require fixed-length inputs when processing sequential data and are unable to handle variable-length sequences,and have difficulties in modelling long-term dependency relationships in video.The transfer of information in traditional neural networks is carried out through fixed connections that cannot take into account the complex relationships between nodes,while problems such as gradient disappearance or explosion can easily occur when dealing with long sequential data.In addition,video data has a temporal dimension,and how to make good use of timedomain information for correct and efficient video relationship detection has become a hot topic of research in recent years.Most previous works have used convolutional or recurrent neural networks for video visual relationship detection,and these methods cannot capture the spatio-temporal information of long time sequences well,and also have problems in inefficiency.After the above problems,the main research work in this paper can be divided into three points as follows:· A Transformer-based video relationship detection algorithm is proposed.The model named Vrd Tran consisting of two core modules,a spatial encoder and a temporal decoder,can better capture the temporal dependencies between multiple frames of an object and reason about the dynamic relationships.Experiments demonstrate the effectiveness of the Vrd Tran model for enhancing time-domain information for video relationship detection.· A graph attention-based video relationship detection algorithm is proposed.The model named ST-a GCN is formed by a relation generation network and an attentional graph convolution.ST-a GCN uses a relation graph generation network to construct relation triples into relation graphs with spatial information,and an attention-based graph convolution network to fuse spatio-temporal contextual information of multi-frame relation graphs.Experiments demonstrate the help of graph structure for reasoning about relationships,and the proposed STa GCN model outperforms benchmark video relationship detection algorithms in the video relationship detection task.· A video relationship detection algorithm based on the graph Transformer is proposed.Combining the advantages of graph structure for local relational inference and Transformer for capturing global information of video,the graph Transformer network VGTran is designed,which includes spatial encoder,relational graph generation,position encoding and graph time decoder modules.The feasibility of the Transformer network for processing graph structure data is also demonstrated in experiments,and the performance of VGTran over other benchmark video relationship detection algorithms is verified.Models based on the attention mechanism can better handle long sequence data like video.In this paper the model with attention mechanism can adaptively select the image regions or features of interest and perform the task more accurately.The Transformer and graph convolution based on the attention mechanism can dynamically calculate the weight distribution of the input data.This weight distribution can be used to processing variable length sequence data.In addition,the structure of graph can be used to efficiently calculate the weights between relational nodes and thus better consider the relationships between nodes.Experiments demonstrate that the combined use of graph structure advantages and attention mechanisms can be more efficient for inference detection of video relationships.更多还原