中文词性标注的研究与实现

【作者】 张卫；

【导师】 杨晓江；

【作者基本信息】 南京师范大学 ， 教育技术学， 2007， 硕士

【摘要】 随着信息检索和自然语言处理的发展，在信息检索中引入自然语言处理技术是信息检索发展的重要趋势之一，而词性标注作为自然语言处理领域的基础，可以提高信息检索的效果和效率，因此在信息检索领域有着非常重要的作用。因此，本文围绕词性标注对其中的关键问题进行了系统研究，并着重实现一个词性标注系统。本文首先对词性标注的基础技术——中文分词作了系统的研究。中文分词模块采用了逆向最大匹配作为基本分词方法，结合规则和统计的方法解决分词歧义问题，并对中文姓名和高频未登录词分别采用相应策略进行识别，所设计和实现的中文分词系统具有较高的分词准确率和较快的分词速度，可以为词性标注提供良好的基础。本文主要对基于统计的词性标注技术进行了研究，所实现词性标注系统主要通过隐马尔可夫模型对训练语料库进行数据统计，获取所需词性和词汇概率信息，并采用Viterbi算法进行标注。针对训练语料库规模较小导致的数据稀疏的问题，运用了简单而高效的Katz算法进行数据平滑处理，有效地避免了因数据稀疏导致词性标注准确率下降的问题。同时，针对Web文档中未登录词较多的特点，增加了对未登录词的词性进行猜测的范闱，提高了对未登录词词性识别的力度。实验证明，本文所设计和实现的中文词性标注系统能达到较高的正确率和消歧率，满足基础教育搜索引擎系统上层模块的需求。最后，本文对所设计与实现的词性标注系统的模块结构、逻辑单元设计等方面进行了详细阐述，并对系统总体性能进行测试。更多还原

【Abstract】 With the development of information retrieval and Natural Language Processing(NLP), the introduction of Natural Language Processing technology in the field of information retrieval is one of the most important trends in the development of information retrieval. As the base of Natural Language Processing, part-of-speech (POS) tagging can improve the effectiveness and efficiency of information retrieval, and therefore has a very important role in the field of information retrieval.This paper first research Chinese Word Segmentation, the basis of Chinese part-of-speech tagging technology. The Chinese segmentation module adopts maximum matching as a basic segmentation method, rules and statistical methods to solve problems of segmentation ambiguities, corresponding strategies are used to identify Chinese names and high frequent unknown words. Design and realization of the Chinese word segmentation system with higher accuracy and faster breakdown of the sub-term rate provides a good basis for the POS tagging.In this paper, the study of POS tagging is based on statistical techniques, and the POS tagging system is mainly achieved through Hidden Markov Model for the mature corpus statistical data, obtains POS and terms information that are requisite, and Viterbi algorithm is used for tagging. In view of the statistical data sparse, a simple and efficient algorithm for data smoothing, Katz algorithm is used to avoid the lack of statistical data resulting decline in the accuracy of POS tagging. Meanwhile, the appropriate POS is chosen for unknown word. Experimental results show that the technology can be used to achieve higher accuracy and disambiguation rate in the POS tagging.Finally, the design and implementation of POS tagging system, such as module structure, logic module design are illuminated. At last, the overall performance of the system is tested.更多还原