【作者】 何儒汉；

【导师】 金海；

【作者基本信息】 华中科技大学 ， 计算机系统结构， 2007， 博士

【摘要】 基于文本的图像检索(Text-Based Image Retrieval,TBIR)是当前商业图像搜索引擎采用的主要方法,它面临的问题是只利用了Web图像的间接文本来间接地检索图像,没有利用图像本身的内容信息;基于内容的图像检索(Content-Based Image Retrieval,CBIR)则是当前图像检索学术研究领域的主流方法,它面临的主要困难是“语义鸿沟”问题,即图像的底层视觉特征(如颜色、纹理、形状等)不能有效描述其高层语义概念。Web图像所处的Web环境使得它具有明显的多模特性,针对Web图像的多模特性,设计了一个层次化、细粒度的Web图像模型及其检索模型,以便于综合利用TBIR和CBIR中的技术来实现Web图像的多模融合检索。该模型的核心思想是在检索过程中同时利用Web图像的文本关键字特征、视觉内容特征、链接信息等多种信息实现相互作用或关联以缩减图像的“语义鸿沟”问题,建立文本关键字和视觉特征的联系,从而达到提高Web图像检索性能的目的。在此模型的基础上,从不同角度和不同应用场景出发,提出了三种新的多模融合检索方法。首先提出了一种基于多种联系的交互式相关反馈模式,以充分利用Web图像的多模特性。该方法遵循以用户为中心,主要用于多样例反馈的检索模式下。它在传统的交互式相关反馈的基础上,扩展并无缝地结合了流形评级算法(Manifold Ranking Algorithm,MRA)和相似性传播算法(Similarity Propagation Algorithm,SPA),对Web图像在文本特征空间和视觉特征空间中的全局结构进行流形学习,以及两个特征空间之间的链接关系进行相互强化学习,实现文本特征和视觉特征的非线性融合检索。该方法在交互式相关反馈的迭代过程中,充分利用了Web图像的多模特性及其内在的模内联系和模间联系。为避免Web环境下的“懒惰用户”问题,提出了一个基于跨模关联规则的自动融合方法。该方法以适应多数普通Web用户的要求为目的,主要应用在自动反馈检索模式下。该方法基于倒排序索引文件,利用频集挖掘(Frequent Itemset Mining,FIM)和关联规则(Association Rule,AR)来寻找文本关键字和多种类型的视觉特征聚类之间的跨模关联,实现自动的多模顺序性融合检索。该方法不需要用户的额外反馈,有效避免了Web环境下的“懒惰用户”问题,并继承了数据挖掘技术良好的扩展性。为平衡用户相关反馈和“懒惰用户”问题,提出了一种结合语义网络和长期反馈学习的方法。该方法把自动反馈和长期学习结合起来,其思想在于既要避免“懒惰用户”问题对系统的困扰,又要充分利用以往用户的反馈信息,使得该方法充分利用但不完全依赖用户的相关反馈。该方法通过倒排序索引文件建立关键字和多种视觉特征聚类之间的初始语义网络,并利用用户的长期反馈学习进一步更新该语义网络,实现自动的多模融合检索。该方法具有良好的动态更新特性,并同时考虑到了多数的普通Web用户(他/她们通常具有“懒惰用户”问题)和少量的高级Web用户(他/她们可能愿意使用相关反馈技术)。在VAST(VsiuAl & SemanTic image search)图像搜索系统中验证了上述方法,并和不同的方法进行了实验对比;结果表明所提出的三种方法可适应不同类型的Web用户的需要并从不同角度实现了Web图像的多模融合检索,这对于突破TBIR和CBIR的局限性,缩小“语义鸿沟”,提高Web图像的检索性能有着显著作用。更多还原

【Abstract】 Text-Based Image Retrieval (TBIR) is the main technology in the current commercial image search engine, which depends on the text only to indirectly retrieve Web images. In contrast, Content-Based Image Retrieval (CBIR) has recently received a great deal of interest in the research community, the major challenge of which is the“semantic gap”problem, i.e. the gap between the low-level visual features (such as color, texture, shape etc.) and the high-level semantic concepts (such as mountain, monkey, flower etc.).The Web images in the Web context have multi-modal characteristics obviously. In this paper, a hierarchy, fine-grained Web image model and corresponding retrieval model are built based on the multi-modal attributes of Web images, which integrate the techniques of TBIR and CBIR to implement the multi-modal retrieval of Web images. The key idea is to simultaneously leverage all types of data which are related to Web image (mainly including the textual features, visual features and hyperlink information), explore their mutual reinforcement, and construct the associations between keywords and visual features to bridge the“semantic gap”and increase the retrieval performance. Based on the retrieval model, three approaches are proposed from different prospectives and scenarios to realize the multi-modal retrieval.First, a Multi-Relationship Based Relevance Feedback (MRBRF) scheme is proposed to fully utilize the multi-modal attribute of Web image. This approach takes the user to the center and mainly is applied in the scenario of multi-example feedback. It is based on the traditional relevance feedback scheme, and extends and seamlessly integrates the Manifold Ranking Algorithm (MRA) and Similarity Propagation Algorithm (SPA). The approach simultaneously utilizes the intrinsic global manifold structure of Web images in both textual feature space and visual feature space, and further implements the mutual reinforcement between the two feature spaces by the hyperlinks. Therefore, the approach realizes a non-linear combination of textual feature and visual feature. This approach makes full use of the multi-modal characteristics and multiple relationships of Web images in the iterations of interactive relevance feedback. Second, a new automatic combination method based on the cross-modal association rule is proposed to avoid the“lazy user”problem in Web context. This approach is suitable for most of the common Web users, which is used in the scenario of automatic feedback. It starts from the inverted file, and explores the cross-modal association between the keyword and several visual feature clusters by Frequent Itemset Mining (FIM) and Association Rule (AR) techniques. Therefore, the approach realizes the sequential multi-modal retrieval automatically, without user’s feedback. This approach avoids the“lazy user”problem in the Web context, and inherits the good extensibility of data mining technique.Finally, an approach that integrates the semantic network and long-term relevance feedback learning is proposed to balance the user’s relevance feedback and“lazy user”problem. This approach integrates the automatic feedback and long-term relevance feedback learning together. The main idea is to fully utilize but not depend on the user’s feedback. Therefore, it not only avoids the“lazy user”problem, but also fully uses the feedback information of the previous users. The approach builds a semantic network between keyword and visual feature clusters based on the inverted file, and updates the semantic network by the long-term relevance feedback learning. This approach has good dynamiticity, and considers the characteristics of both most of the common Web users (which have the“Lazy User”problem) and a few advanced users (which use the Relevance Feedback).The above three approaches are tested in our VAST (VisuAl & SemanTic image search) system, and are compared with other methods. The experimental results show that the proposed approaches implement the multi-modal retrieval from different prospectives and adapt to different Web users. These approaches really benefit to break through the limitation of TBIR and CBIR, alleviate the“semantic gap”and increase the retrieval performance.更多还原