【作者】 刘恒；

【导师】 田春娜；

【作者基本信息】 西安电子科技大学 ， 信号与信息处理， 2019， 硕士

【摘要】 随着互联网带宽的大幅增加和各种移动设备的激增,Web2.0技术下图像数据已经出现爆炸性的生成,发布和传播,成为当今大数据不可或缺的一部分。然而互联网上的部分图像是无标记的,为了更有效地存储、管理、检索和利用这些数据,近年来,研究人员致力于用完整的自然语句自动描述图像的内容,即为图像字幕生成。然而,图像字幕生成是一个非常具有挑战性的任务,它不仅需要利用模型去捕获图像中呈现的对象或者场景,表达图像中对象和场景的相关性,而且还要用恰当的自然语言来描述它们。针对上述问题,本文利用深度学习方法对图像字幕生成进行了较为系统深入的研究,主要的研究成果如下:(1)提出了一种基于长短时记忆网络(Long Short Term Memory,LSTM)的从多特征序列到语句序列的图像字幕生成框架。为了更全面的描述图像特征,此方法提出分别使用基于ImageNet ILSVRC图像分类数据集训练的ResNet152提取图像中目标特征,使用基于Places365场景数据库训练的ResNet152提取图像中的场景上下文特征,利用二者的互补性全面描述目标和场景上下文特征。为了充分利用图像字幕描述时的常规语义信息,本文使用基于MSCOCO数据库训练的多实例属性分类器提取图像的视觉语义特征,作为图像视觉特征在语义先验上的补充。本文将图像的目标特征、场景上下文特征和图像的视觉语义特征序列输入到LSTM编码器,完成图像的特征表达。最后,通过LSTM解码器将该特征翻译成语言描述,基于交叉熵损失函数实现从多特征序列到自然语言序列的图像字幕生成框架的训练。本文在MSCOCO数据库上对模型进行了实验评估,对比实验结果证明相比经典方法在多个指标上均有优势。(2)提出了一种协同常规-特定语义的多特征图像字幕生成方法。为更好的表示图像的语义特征,本文提出在MSCOCO数据库上训练多实例属性分类器提取图像的常规语义属性,同时对测试图像在视觉语义嵌入空间检索其相近语义作为图像的特定语义属性,协同图像的常规和特定语义属性,作为图像的语义先验,并将其与图像的目标特征、场景上下文特征序列一起输入到LSTM编码器中作为图像的特征表示。除此之外,本文还提出在图像解码中,将特定语义作为“特定语义监督器”用于LSTM解码中候选短句之间的BLEU 4相似性监督约束,实现特定语义协同监督下的图像字幕生成。本文在MSCOCO数据库上的实验证明了协同常规和特定语义的语义属性和“特定语义监督器”的优势,同时较其它经典方法取得了更好的实验效果。更多还原

【Abstract】 With the dramatic increase in Internet bandwidth and various mobile devices,image data has been generated,released and spread in fast speed under the Web 2.0 technology,which has become an indispensable part of today’s big data.However,some images on the Internet are untagged,in order to store,manage,retrieve and utilize these data more efficiently,researchers have been worked on automatically describing image contents with complete sentences,viz.image captioning,in recent years.However,image captioning is very challenging.It not only needs to capture the visual representation of objects and scene presented in images and express the relationship between them,but also needs to describe them with appropriate natural language.In address those problems,we systematically study the deep learning based image captioning.The main research contributions are as follows:(1)We propose a LSTM-based image captioning framework to generate sentence sequence from multi-feature sequence.In order to describe the image features more comprehensively,we trained one Res Net152 on Image Net dataset to extract object features and one Res Net152 on Places365 dataset to extract scene context features,respectively.Then,we use these two complementary features to fully represent the object features and scene context features in images.Besides,we use the multi-instance attribute classifiers trained on MSCOCO dataset to extract the semantic information in the image as a supplement of general semantic priors for image captioning.We feed the object features,scene context features and visual semantics sequentially to the encoder of LSTM to complete the feature representation in the image.Finally,the feature is translated into a language description by a LSTM decoder,which is achieved by training the framework of translating multi-feature sequences to the natural language sequences based on the cross entropy loss function.In this paper,we evaluate our model on MSCOCO dataset.The comparison results show the superiority of our algorithm over state-of-the-art approaches on standard evaluation metrics.(2)We propose a multi-feature based image captioning framework that collaborates general and specific semantics.In order to better represent the semantic features of images,we propose to extract the general semantic attributes of image through the multi-instance attribute classifier trained on MSCOCO dataset,and then retrieve similar semantics for the test image in an improved visual semantic embedding(VSE++)space as the specific semantic attributes for the image.Then,we collaborate general and specific semantic attributes as semantic priors,and sequentially feed the collaborated semantic attributes,object features and scene context features to the encoder of LSTM as the feature representation of the image.In addition,we also employ the specific semantics as the “specific semantic supervisor” for BLEU 4 similarity supervision between the candidate phrases in the decoding of LSTM,which results in the captioning method collaborating specific semantics supervision and general semantics.The evaluation on MSCOCO dataset shows the superiority of our model,which achieves better experiment results over the state-of-the-art approaches.更多还原