【作者】 孙越恒；

【导师】 何丕廉；

【作者基本信息】 天津大学 ， 计算机应用技术， 2005， 博士

【摘要】 中文信息检索是信息检索领域的一个重要分支。尽管中文信息检索的技术和研究已经取得了相当多的成果,但目前中文信息检索的效果却并不乐观。本文就中文信息检索的几个主要问题,在NLP技术的基础上,利用统计学和代数学方法,从词语层和文档层对信息检索中文档和文档集的处理方法进行了深入研究。本文首先从理论上对中文信息检索索引单位的选择做了详细分析,改进了传统的最大匹配分词算法,一定程度上解决了切分歧义的问题,同时在改进算法中引入了一种基于统计的窗口移动扩展方法,简单而有效地改善了未登录词的识别问题。信息抽取已成为制约信息检索性能的一个瓶颈,而关键词提取则是信息抽取的一个重要环节。本文针对中文,实现了基于χ2统计的单文档关键词提取算法,其主要基础是词与词之间的共现次数的统计,并使用χ2统计量来衡量两个词语之间的关联。同时改进了传统的KEA算法,扩展了标示关键词的特征,实现了基于朴素贝叶斯理论的中文多文档关键词提取模型。文本分类是对信息检索中的文档集进行组织的一项关键技术。本文首先对文本分类算法进行了研究,探讨了文本特征抽取方法,其中综合考虑了频度、分散度和集中度三项指标,设计并实现了一种新的特征抽取算法,使得选出的特征项整体优化。另外,提出了一种基于向量空间模型的词共现模型,并将通过该模型统计出的共现词信息应用于文本分类研究中。上述技术都在一定程度上提高了文本分类系统的性能。最后,本文把分类技术应用到了信息检索中的用户查询歧义消除方面,实现了一个分类检索系统,使用户可以快速获取自己真正需要的信息。针对高维词-文档矩阵所造成的高存储空间和运算时间开销,本文将线性(LSI)和非线性(Isomap、SIE)维数约减算法引入到高维文档数据的降维处理中,并在文档聚类方面对三种算法处理后的数据进行了性能比较。实验结果表明,采用了局部嵌入技术的SIE算法取得了与LSI相当的性能,优于全局优化的Isomap算法,并且降维处理时的运算复杂度也大大降低。最后,本文实现了基于N层向量空间模型的Windows上的信息检索系统。该系统对Web文档采用了一种分层处理机制,重点改进了Web文档中的关键信息的权重计算。更多还原

【Abstract】 Chinese Information Retrieval (CIR) is an important branch of InformationRetrieval, and has achieved rapid development in recent years. However, there arestill some issues need to be studied further for improving the effectiveness andefficiency of today’s CIR system. This paper uses the NLP technologies based onstatistics and algebra, studies the processing methods for document(s) at the wordlevel and document level, and presents solutions for several key problems in CIR.This paper first provides a detail theoretical analysis on the choice of indexingunit in CIR, and improves the traditional Chinese segmentation algorithm based onmaximum matching, which solves the segmentation ambiguity problem to a certaindegree. In addition, a window moving and expanding method based on statistics isintroduced to this segmentation algorithm, which simply and effectively improves theissue of unknown words identification.Information Extraction (IE) has been a bottleneck restricting the performance ofIR system, in which keyword extraction is one of important factors. This paperpresents a single-document keyword extraction algorithm based on χ 2 statistic. Thisalgorithm uses the co-occurring information between words to get a χ 2 statistic tomeasure their relation. Also, this paper improves the traditional KEA algorithm,extends the features used in identifying keywords, and implements a multi-documentskeyword extraction model based on Naive Bayes theory.Text classification is a key technique for organizing document set in IR. Thispaper first studies the text classification algorithms, discusses how to extract featureterms, and implements a new feature extraction algorithm. In addition, this paperprovides a word co-occurrence model based on Vector Space Model (VSM), andapplies the word co-occurrence resources obtained by this model to text classification,and improves the performance of text classification system Finally, this paper appliesthe idea of classification into reducing users’ query ambiguity in IR, and implements aclassification search system, which enables users quickly and accurately get theirrequired information.For reducing the high memory and time cost for processing high-dimensionalterm-document matrix, this paper introduces linear (LSI) and nonlinear (Isomap, SIE)dimension reduction algorithms into the processing of high-dimensional documentdata, and compares theirs performance in document clustering. Experimental resultsshow that SIE algorithm adopting local embedding technology achieves acomparative performance with LSI, and is better than Isomap algorithm using globaloptimization technology.Finally, this paper implements an IR system based on N-level VSM on Windowplatform. This system uses a hierarchical scheme in processing Web documents, andprimarily improves the weight computation for key information in Web documents.更多还原