【作者】 苏静；

【导师】 戴青云；

【作者基本信息】 广东工业大学 ， 信息与通信工程， 2022， 博士

【摘要】 图像作为人类视觉感知的重要途径,是人工智能时代海量数据的主要来源,因此,如何对海量的图像数据进行智能化理解是当前图像理解领域需要解决的主要问题。近年来,深度学习在图像分类和图像描述等图像理解研究与应用中取得了显著的研究成果,但是,具有复杂语义的图像理解,例如层级图像分类和视觉叙事等任务,仍然存在很大的探索空间。层次图像分类和视觉叙事任务需要对图像本身高层复杂语义进行理解,它们相对于当前流行的图像分类与图像描述任务显得更加复杂、也更具挑战性,有着很强的理论研究与应用价值。层级图像分类问题中,由于层级图像中的类别之间具有严格的层级关系,且精细级类别之间相似度较高,准确识别这些层级类别对计算机仍然是个难点问题。视觉叙事旨在为连续的图像生成连贯的和富有表现力的故事性描述,它不仅需要对复杂场景以及图像之间的关联关系进行识别,更需要对抽象语义进行理解,这给当前计算机图像理解技术提出了更高的挑战。基于以上问题,本文在对深度学习相关理论研究基础上,采用关注机制分别从双关注、局部特征关注-全局语义以及多层级关注这三个方面对于图像理解中的层级图像分类和视觉叙事问题展开了深入研究。论文的主要研究工作如下:(1)提出了一种基于双关注机制的层级图像分类模型。针对现有层级图像分类方法多用于固定层级的识别,本文构建了一种基于CNN-LSTM的通用识别模型DACL(dual-attention CNN-LSTM),引入空间特征维度和空间语义维度的双关注模块,用于同时解决固定和可变层级分类问题。该模型通过空间特征关注机制学习不同类别对应的更具判别性的细粒度特征,并且通过空间语义关注机制对各类别间的相关性进行建模,从而增强模型关键信息的判别能力,有效提升模型的泛化性。本文使用CIFAR10、CIFAR100和外观专利图像数据集对所提方法进行了验证,实验结果表明了所提出的DACL方法在精准度和准确率方面相对其他现有的层级图像分类方法的优越性。(2)提出了一种融合局部特征关注机制和全局上下文语义的视觉叙事方法。本方法采用一种端到端的长短时记忆网络模块并行方法实现视觉叙事,解决了传统的视觉叙事方法采用串行长短时序记忆网络模块,网络参数过多,计算量大,过度耗费网络资源的缺陷。本文在考虑全局上下文语义的条件下结合局部特征关注机制,将序列图像信息作为全局图像特征,通过多层感知器学习序列图像的故事主题信息,同时将单张图像信息作为局部特征并引入关注机制,得到文本对应的特征关注图,分别实现对图像与图像间、图像与文本间依赖关系的构建。本方法有效解决了传统长短时记忆网络模块方法中因序列图像分开输入,只关注单图像与文本之间的关系,忽略了序列图像间关联关系存在的不足。本文的方法在两个公用图像数据集(DII和SIS)上进行了实验,实验结果显示本文的模型取得了良好的效果。(3)提出了一种基于层级关注机制的视觉叙事生成算法。本文利用BERT模型丰富的语义提取能力,构建了句子级和词语级两层长短时记忆网络模型,并引入句级与词级关注机制实现序列图像的故事性描述。该模型在底层首先对句级语义进行建模,关注每个图像与对应句子语义间的映射关系同时也关注图像与图像、句子与句子间的关联关系,负责提取每个图像的高层主题信息,再在第二层基于该主题对词级语义进行建模,重点关注每个图像与该句文本中的每个单词的映射关系,负责学习每个单词对应的图像特征信息。本方法能够有效改进传统视觉叙事方法生成的句子语法问题多,表达方式过于简单的缺点。实验结果表明,在自动评估指标BLEU和CIDEr下,本文的模型优于大多数方法,同时,本文的方法在人类评估中的各项指标中表现良好。综上所述,本文基于关注机制的图像理解中的若干关键问题,结合最新深度学习理论方法展开研究,其内容是计算机视觉与自然语言处理交叉学科的创新研究,对解决现实中的应用问题有着十分重要的研究意义。更多还原

【Abstract】 As an important way of human visual perception,image is the main source of massive data in the era of artificial intelligence.Therefore,how to intelligently understand massive image data is the main problem to be solved in the field of image understanding.In recent years,deep learning has made remarkable achievements in image understanding tasks such as image classification and image description,which further promotes the development of image semantic understanding tasks.However,image understanding with complex semantics,such as hierarchical image classification and visual storytelling,still has a lot of room for exploration.How to accurately and effectively understand images with complex semantics has far-reaching theoretical significance and broad application prospects.In hierarchical image classification problems,due to strict hierarchical relationships between the categories in the hierarchical image and high similarity between fine categories,it remains a difficult problem for computers to accurately identify these hierarchical categories.Visual storytelling,which aims to generate coherent and expressive storytelling for sequential images,requires not only to recognize the complex scenes and the dependencies among images,but also to understand the abstract semantics,which poses a higher challenge to the current image understanding technology for computers.Based on the above problems,this thesis investigates hierarchical image classification and visual storytelling in the image understanding field through attention mechanisms based on deep learning theories and methods from three aspects,namely,dual attention,local-feature attention with global semantic,and hierarchical attention,respectively.The main contributions of this thesis are as follows.(1)A hierarchical image classification model is proposed based on a dual-attention mechanism.Since most existing hierarchical image classification methods are used for fixedlevel recognition,this paper constructs a general recognition model DACL(dual-attention CNN-LSTM)based on CNN-LSTM,which introduces dual-attention modules of spatial feature dimension and spatial semantic dimension for solving both fixed and variable hierarchical classification problems.The model learns more discriminative fine-grained features corresponding to different categories by the spatial feature attention mechanism,and models the correlation between categories by the spatial semantic attention mechanism,so as to enhance the discriminative ability of key information of the model and effectively improve the generalization of the model.In this study,the algorithm is applied to CIFAR10,CIFAR100 and the design patent image datasets,and its performance is evaluated compared with the existing methods.The experimental results demonstrate that the proposed DACL method outperforms other existing hierarchical image classification methods in terms of both precision and accuracy.(2)A visual storytelling method is proposed by incorporating the local feature attention mechanism and global context semantics.This method adopts an end-to-end parallel Long Short-Term Memory module to implement visual storytelling.By contrast,traditional visual storytelling methods mainly employed serial Long Short-Term Memory modules,bringing problems such as too many network parameters,large computation,and excessive consumption of network resources.The proposed method overcomes the mentioned drawbacks.The thesis uses sequence image information as global image features while combining global context semantics with a local feature attention mechanism.Single image information is employed as a local feature,and an attention mechanism is introduced to obtain the image feature attention graph corresponding to the text and realize the association relationship construction among images and between image and text,respectively.Due to the separate input of sequence images,the traditional Long Short-Term Memory method only focuses on the relationship between a single image and text and ignores the association between sequence images.The proposed method effectively solves the above deficiency.The method was implemented on two public image datasets(DII and SIS)and achieved superior experimental results.(3)A visual storytelling algorithm is proposed based on a hierarchical attention mechanism.This thesis utilizes the rich semantic extraction capability of the BERT model to establish a two-layer Long Short-Term Memory model and introduces sentence-level and word-level attention mechanisms to realize the storytelling description of sequence images.The model first models the sentence-level semantic at the bottom layer,focusing on the mapping relationship between each image and the corresponding sentence semantic and also on the association relationship between image and image,sentence and sentence,and extracts the high-level topic information of each image,and then models the word-level semantic subject to the high-level topic at the second layer,focusing on the mapping relationship between each image and each word in the sentence text,and learns the corresponding image feature information of each word.Traditional visual storytelling methods generate sentences with many syntactic problems and over-simplified expressions.This method can effectively overcome these shortcomings.The experimental results demonstrate that the proposed model outperforms most of the methods under the automatic BLEU and CIDEr metrics as well as various metrics in human assessment.In summary,this thesis is aimed at solving several key issues in image understanding with attention mechanisms,combined with the latest deep learning theoretical methods.This thesis includes innovative research in the intersection disciplines of CV and NLP,which has important research significance for solving real-world application problems.更多还原