【作者】 张琦；

【导师】 张铭泉； 黎颂暄；

【作者基本信息】 华北电力大学 ， 电子信息硕士（专业学位）， 2024， 硕士

【摘要】 图像描述生成旨在通过简明的语句对给定的图像自动生成自然语言描述,是一项涉及计算机视觉和自然语言处理的跨领域研究任务。该研究被广泛应用于图像检索、医疗影像报告生成、跨媒体数据管理等场景。目前主流的图像描述模型聚焦于深度神经网络,但大都忽略多个对象之间的相互关系和位置关系等语义信息,无法得到图像中对象之间的层次化交互,导致信息提取不全面;并且对来自文本和图像语义线索的研究并不够深入。为解决上述问题,本课题对基于深度学习的图像描述技术进行了研究,主要工作概括如下:(1)提出了基于多层次卷积神经网络图像描述模型。为了得到图像中对象之间的关系及其层次化交互,该模型首先使用结合了通道和空间注意力的Faster R-CNN提取图像特征,随后利用图卷积神经网络处理不同抽象层次的视觉信息及其边界框之间的层次关联。将语义和空间视觉关系全面集成到图像编码器中,细化图像中的区域特征表示和图像之间的连接。最后利用学习到的多级特性,使用带有注意力机制的双层LSTM解码器生成图像描述语句。实验结果表明,与主流模型最高评分相比,所提模型在MS COCO数据集上的BLEU-4、METEOR、ROUGE-L和CIDEr-D评分均有显著提升,可以有效提高生成的图像描述语句质量。(2)设计了基于弱监督和多任务学习的图像描述模型。为了更好的探索和利用图像标题中的语义信息,在标题和图像之间建立结构性连接,在编码端使用弱监督多实例学习引入有益的归纳偏差,并构造图像标题引导的视觉关系图。调整图卷积操作获得对象节点和谓词节点的表示,相邻节点和上下文节点的文本和视觉特征进一步增强了表示能力。在解码生成过程中,将视觉关系图中的图节点作为输入,结合多任务学习联合预测单词和标签序列,以规范生成单词时应该考虑的信息类型,使描述语句更加准确。实验结果表明,所提模型在MS COCO数据集上的BLEU-4、METEOR、ROUGE-L和CIDEr-D评分分别达到0.385、0.283、0.581和1.192,与主流模型最高评分相比均有所提升。(3)设计实现了图像描述生成原型系统。为了更好的验证本课题提出的图像描述生成方法,基于以上提出的两种模型设计实现了图像描述生成原型系统,经测试,所设计系统可较为准确的生成图像描述。更多还原

【Abstract】 Image description generation aims to automatically generate natural language descriptions for a given image through concise statements,and is a cross disciplinary research task involving computer vision and natural language processing.This study is widely applied in scenarios such as image retrieval,medical image report generation,and cross media data management.At present,mainstream image description models focus on deep neural networks,but most of them ignore semantic information such as the interrelationships and positional relationships between multiple objects,and cannot obtain hierarchical interactions between objects in the image,resulting in incomplete information extraction;And the research on semantic clues from text and images is not deep enough.To address the aforementioned issues,this project has conducted research on image description techniques based on deep learning.The main work is summarized as follows:(1)A multi-level convolutional neural network based image description model was proposed.In order to obtain the relationships and hierarchical interactions between objects in the image,the model first uses Faster R-CNN,which combines channel and spatial attention,to extract image features.Then,a graph convolutional neural network is used to process visual information at different levels of abstraction and the hierarchical correlations between their bounding boxes.Integrate semantic and spatial visual relationships comprehensively into the image encoder,refine the representation of regional features in the image and the connections between images.Finally,utilizing the learned multi-level characteristics,a dual layer LSTM decoder with attention mechanism is used to generate image description statements.The experimental results show that compared with mainstream models with the highest scores,the proposed model has significantly improved BLEU-4,METEOR,ROUGE-L,and CIDEr-D scores on the MS COCO dataset,which can effectively improve the quality of generated image description statements.(2)Designed an image description model based on weakly supervised and multi task learning.In order to better explore and utilize the semantic information in image titles,structural connections are established between titles and images.Weakly supervised multi instance learning is used at the encoding end to introduce beneficial inductive biases,and a visual relationship graph guided by image titles is constructed.Adjusting the graph convolution operation to obtain representations of object nodes and predicate nodes further enhances the representation ability of adjacent nodes and contextual nodes with text and visual features.In the decoding and generation process,the graph nodes in the visual relationship graph are used as inputs,combined with multi task learning to jointly predict word and label sequences,in order to standardize the information types that should be considered when generating words and make the description statement more accurate.The experimental results show that the proposed model achieves BLEU-4,METEOR,ROUGE-L,and CIDEr-D scores of 0.385,0.283,0.581,and 1.192 on the MS COCO dataset,respectively,which are improved compared to the highest scores of mainstream models.(3)Designed and implemented a prototype system for image description generation.In order to better validate the image description generation method proposed in this project,a prototype system for image description generation was designed and implemented based on the two models proposed above.After testing,the designed system can accurately generate image descriptions.更多还原