【作者】 王旭；

【导师】 彭德中；

【作者基本信息】 四川大学 ， 计算机科学与技术， 2021， 博士

【摘要】 由于互联网多媒体技术的快速发展、智能手机的广泛使用以及社交网络的不断普及,人们能够随时随地在互联网上分享有趣的内容,使得互联网上不同模态的多媒体数据(如文本,图像和视频等)呈现爆炸性增长、海量性集聚的特点。如此大规模的数据标志着多媒体大数据时代的到来,同时给基于多模态学习的研究和应用带来了新的机遇和挑战。随着以深度神经网络为代表的人工智能技术的迅猛发展,如何基于深度神经网络模拟人类大脑的认知和理解过程以实现多模态数据在语义上的联系与内容上的理解,正成为人工智能和多媒体领域的研究热点,也是多模态学习中亟待解决的一个关键研究问题。该研究问题也被称作“跨模态内容理解”。跨模态内容理解在国家、社会和个人生活方面都有着至关重要的研究意义和研究价值。“跨模态检索和分类”作为“跨模态内容理解”中的一个具有实际应用价值的具体研究任务,其目标是在用户给定任意模态的查询数据的情况下,从数据库中检索和分类出所有模态中的语义相关联数据。但由于不同模态的数据组织形式和结构上有着巨大的差异(该差异也被称作“异构鸿沟”),导致难以直接度量不同模态数据间的语义相似性,使得跨模态检索和分类面临着巨大挑战。受不同监督模式的影响,跨模态内容理解也呈现出不同的特点及难点。本文面向异构鸿沟的挑战,围绕不同监督模式下跨模态内容理解的研究问题,针对跨模态检索和分类任务中的以下四个科学问题开展研究:·问题1:在多模态数据有标注的情况下,如何避免由于度量标准和损失函数不匹配而引入的误差,同时尽可能平衡地利用来自不同视图的信息?·问题2:在多模态数据有标注的情况下,如何突破多视图哈希学习过程中由于耦合训练而导致的时间局限性和空间局限性?·问题3:在有少量标注数据和大量无标注数据的情况下,如何有效地以半监督学习方式,更好地利用无标注数据帮助提高跨模态内容理解的准确性和鲁棒性?·问题4:在多模态数据没有标注的情况下,如何有效地以无监督学习方式进行跨模态内容理解,缩小不同模态数据间的异构鸿沟?针对这些问题,本文分别设计和提出了一系列不同监督模式下的多视图神经网络学习方法。本文的研究成果总结如下:1.提出了一种深度关系相似性学习方法。与现有的跨模态学习方法不同,所提出的方法直接学习成对的关系相似性矩阵,而不显式地学习公共空间。因此,它可以避免不同模态之间信息不均衡的问题。同时,将成对的关系相似性作为跨模态检索的度量,不会给损失函数和检索度量的匹配带来额外的误差。在四个公共数据集上的大量实验结果表明,提出的模型在跨模态检索任务中具有良好的性能。2.设计了一种可分离变分哈希网络学习方法,可以将任意数量的模态数据分别映射到一个公共的汉明空间。该方法的网络结构由一个标签网络和多个特定模态的网络组成。标签网络利用所有可用的标签,通过将语义标签投影到公共的二值哈希编码中来学习潜在的公共汉明空间。然后,特定模态的变分网络可以将多个模态分别投影到由标签网络学习的公共语义二值表征中。这是通过将标签网络的哈希编码向量的聚合后验与任意先验分布进行匹配的变分推理来实现的。在四个基准数据集上的实验结果和综合分析表明了该方法的有效性,与现有的最新方法相比,该方法达到了更好的跨模态检索性能,同时获得了更高效灵活的训练方式。3.提出了一种深度半监督类别与相关性坍缩的跨视图学习方法。该方法通过将同一个类的数据实例坍缩至同一个点,将不同类的数据实例同时坍缩至不同的点来学习判别性公共空间。同时,为了充分利用未标记数据,该方法提出通过将相关的数据样本坍缩到同一个点,而将不相关样本坍缩到其他点来建模未标记数据的相关性。这两个目标是通过最小化两个Kullback-Leibler散度来联合优化的。此外,该方法可以应用于两个以上的视图。在五个公共数据集上的实验表明,该方法在跨模态检索和分类任务中取得了良好效果。4.设计了一种对抗式相关自动编码器,用以学习多视图数据的公共表征。该方法通过将潜在变量的聚合后验分布与特定先验分布进行匹配来进行变分推理,使不同模态的表征可以遵循相同分布。在五个基准数据集上的综合分析实验表明,该方法在跨视图检索和分类任务中具有良好的性能。总而言之,本文针对不同监督模式下的跨模态检索与分类问题,提出了一系列多视图神经网络学习方法。这些方法在广泛使用的公共数据集上的表现证明了其有效性,其能够处理不同监督模式下的跨模态检索与分类任务,取得了良好的效果。更多还原

【Abstract】 Due to the rapid development of Internet multimedia technology,the widespread use of smartphones,and the growing popularity of social networks,people can share interesting content on the Internet anytime and anywhere,making all kinds of multimedia data(text,images,and videos,etc.)on the Internet showing the characteristics of explosive growth and massive agglomeration.Such a large scale of data marks the arrival of the era of multimedia big data,and at the same time brings new opportunities as well as challenges to the research and application based on multimodal learning.With the rapid development of artificial intelligence technology represented by the deep neural network,how to simulate the cognitive and understanding process of the human brain based on the deep neural network to realize the semantic connection and content understanding of multimodal data,is becoming a research hotspot in the field of artificial intelligence and multimedia.It is also a key research problem to be solved in multimodal learning.This research problem is also known as Cross-modal Content Understanding(CCU).CCU is of great significance and value in national,social,and personal life.Cross-modal Retrieval and Classification,as a practical research task in CCU,aims to retrieve and classify semantically related data in all modalities from the database when the user is given query data of any modality.However,due to the huge differences in the data organization form and structure of different modalities(also known as the heterogeneous gap),it is hard to measure the semantic similarities across different modalities directly.The heterogeneous gap makes cross-modal retrieval and classification face great challenges.Under the influence of different supervision modes,CCU also presents different characteristics and difficulties.To solve the challenge of the heterogeneous gap,this thesis focuses on the research problem of CCU under different supervision modes,and focuses on the following four scientific issues in cross-modal retrieval and classification tasks:·Issue 1: In the case where multimodal data is labeled,how to avoid the errors caused by the mismatch between metric and loss function,and at the same time exploit the information from different views as balanced as possible?·Issue 2: In the case where multimodal data is labeled,how to break through the time and space limitations caused by coupling training in the process of multi-view hash learning?·Issue 3: In the case of a small amount of labeled data and a large amount of unlabeled data,how to effectively exploit semi-supervised learning to make better use of unlabeled data to help improve the accuracy and robustness of cross-modal content understanding?·Issue 4: In the case where multimodal data is unlabeled,how to effectively exploit unsupervised learning to understand cross-modal content and narrow the heterogeneous gap between different modalities?To solve these issues,this thesis designs and proposes a series of multi-view neural network learning approaches under different supervision modes.The research results of the thesis are summarized as follows:1.A new approach called Deep Relational Similarity Learning(DRSL)is proposed in the thesis.Different from existing approaches,the proposed DRSL is supposed to directly learn a pairwise relational similarity matrix instead of explicitly learning a shared space.Hence it can refrain from the issue of unbalanced information across distinct modalities.Meanwhile,the pairwise relational similarity serves as the cross-modal retrieval metric,thereby introducing no extra error to the matching of the loss function and retrieval metric.Extensive experiments on four public datasets show the promising performance of the proposed approach in cross-modal retrieval task.2.A novel approach named Separated Variational Hashing Networks(SVHNs)is designed to separately transform any number of modalities into a common Hamming space.SVHNs approach consists of a label network(Lab Net)and multiple modality-specific networks.Lab Net is used to exploit all available label annotations to learn a latent common Hamming space by projecting the semantic labels into the common binary codes.Then,the modality-specific variational networks can separately project multiple modalities into their common semantic binary representations learned by Lab Net.It is realized by variational inferring that matches the aggregated posteriori of Lab Net’s hashing code vector with any prior distribution.Extensive experimental results on four widely-used benchmark datasets and the comprehensive analysis have demonstrated the effectiveness of the designed Lab Net and the variational inference,leading to superior cross-modal retrieval performance as well as more efficient and flexible training patterns compared to current state-of-the-art methods.3.A new approach dubbed Deep Semisupervised Class-and Correlation-Collapsed CrossView Learning(DSC3L)is proposed in the thesis.DSC3 L learns the discriminative space through collapsing instances of the same class into the same point,and instances of different classes into different points simultaneously.Meanwhile,to fully exploit the unlabeled data,DSC3 L proposes modeling the correlation of unlabeled data by collapsing the correlated samples into the same point while collapsing the uncorrelated samples into other points.These two objectives are jointly optimized by minimizing two KullbackLeibler divergences.Furthermore,DSC3 L can be applied to more than two views.A large number of experiments on five benchmark datasets show that DSC3 L has promising performance on cross-view retrieval and classification tasks.4.A novel unsupervised multi-view representation learning approach called Adversarial Correlated Autoencoder(Adv CAE)is designed to learn common representations for multiview data.The approach conducts variational inference by matching the aggregated posteriors of the latent variable with a specific prior distribution.Benefiting from the model,the representations of different views could follow the same distribution.Comprehensive experiments on five benchmark datasets show the promising performance of the proposed method on cross-view classification and cross-view retrieval tasks.更多还原