基于概念的信息检索模型研究

【作者】 杨宾伟；

【导师】 杨国纬；

【作者基本信息】 电子科技大学 ， 计算机软件与理论， 2006， 硕士

【摘要】 一直以来,在自然语言处理领域存在着两种截然不同的理论,即理性主义和经验主义。理性主义强调的是基于规则的语义分析,它试图寻找一种百分之百解决问题的途径;而经验主义则依靠已经存在的语言事实,通过统计的方法来追求利益的最大化。本论文从理性主义自然语言处理出发,根据概念依存理论和复杂特征集提出了概念的动态特征集表示方法,简单的分析了利用合一运算构建动态特征征集的过程,提出了一种将概念特征集应用于信息检索的匹配理论,初步探讨了基于该理论的信息检索的实现,通过匹配过程的深入分析得出文档与查询语句相匹配的充要条件是文档必须蕴含查询语句所包含所有概念基并且必须与查询语句中的概念基之间关联关系相匹配的结论。由于概念的动态特征集匹配理论与其他理性主义方法一样,面临着维护庞大的规则集等众多问题。所以本文转向通过统计自然语言处理寻找一种实现上述结论的最佳方法,最终提出了基于段语言模型的信息检索实现模型。段语言模型对传统的统计语言模型在两个方面进行改进:首先针对概念基在语言中可以对应多个词汇,而查询语句中的词汇仅仅是其特例的情况,本文引入了相关词表的概念,在相关词表中维护了每个概念基对应的所有可能的词汇表示,在构建语言模型时不只是根据查询语句的词汇,而是通过查询语句中概念基的所有相关词汇,这就有效的提高了检索结果的召回率。其次针对查询语句中概念基之间的联系无法确定的问题,本文提出根据一个窗口来统计相关词汇的出现频率,而不是整篇文档。因为如果一个小窗口中相关词汇大量出现,那么这些相关词汇构成与查询语句中概念基之间相同联系的概率将会增加。本文中采用段作为窗口大小。以此为基础本文提出了两种具体的段语言模型,即一元段语言模型和二元段语言模型,对模型中的相关词集合、模型推导、模型平滑、参数估计、模型实现等问题作了深入的研究,并通过实验确定了各个参数的估值范围。论文最后将段语言模型与传统语言模型的实验结果作了对比分析,结果显示:段语言模型较传统的语言模型在准确率和召回率上都有显著提高。更多还原

【Abstract】 Since always, there are two kinds of entirely different theories in the Natural Language Processing field, namely rationalism and empiricism. The emphasis of rationalism is rule-based semantic analysis, and it attempts to find an absolutely effective solution to problems. But the empiricism depends upon the language facts that already exist and achieves the maximum benefit through statistical methods.This paper embarks on from rationalism Natural Language Processing and proposes the method of expressing concept using the Dynamic Attribute Set according to the theory of Conceptual Dependency and Complex Attribute Set. Then the paper briefly analyzes the process of constructing the Dynamic Attribute Set via unification, proposes the matching theory that Conceptual Attribute Set can be applied to information retrieval, and basically discusses the realization of information retrieval based on this theory. Finally the paper concludes that the necessary and sufficient condition that a document matches a query is that the document must contain all the Conceptual Bases that appear in the query and be consistent with the relationship among Conceptual Bases in query.But it is found that the matching theory of Conceptual Dynamic Attribute Set has to face many problems such as the maintenance of a huge rule set, just as many other rationalism methods. Therefore this article switches to the Statistical Natural Language Processing and seeks the best way to implement the above conclusion. Finally a new language model named Section Language Model in the information retrieval field is proposed.The Section Language Model makes the improvement on the traditional statistical language model in two aspects:Firstly, aimed at the situation that the Conceptual Base can possibly correspond to many words in language, but the words in query are merely its particular cases, this paper has introduced the Correlation Vocabulary Table. It contains all the possible words that may correspond to each Conceptual Base. When constructing the language model, not only the query’s words are considered, but also all the words corresponding更多还原