【作者】 李明；

【导师】 刘晋；

【作者基本信息】 南京航空航天大学 ， 管理科学与工程， 2012， 硕士

【摘要】 随着信息技术的发展，各个领域的电子文献变得日益丰富，在各个专业领域，文献数据库中的文献数量呈现出指数增长的趋势，对这些文献信息的处理难度也变得越来越大。中文有着特殊的组织结构，其信息处理对分词技术有着较高的要求，因而中文文献相对于英文文献有着较高的处理难度。如何针对中文的语法特征，找出其中存在的新的专业词汇，在信息检索中扮演着极为重要的角色。本文的研究对象是单一特定的领域，而非多个领域。此领域具有可扩充性，可以为金融领域，也可以为IT领域，根据具体需求而定。在确定一个领域并给出这个领域的文献数据集合的前提下。本文的信息处理主要分为两个步骤进行，首先对特定领域的文献集合进行分词处理，在进行分词处理方面使用了基于统计的N-Gram方法，较为有效地找出了词典中所不存在地新词汇。第二个步骤为新的专业词汇的抽取，这是一个根据已有专业词汇来发现未知专业词汇的过程，目的从第一步中所产生的新的词汇中抽取出新的属于目标领域的专业词汇。在这个步骤中，本文使用了Apriori方法，首先发现词汇组合中的频繁项目集合，再生成相应的关联规则以达到新的专业词汇提取的目的。这个步骤涉及噪声词过滤的问题，考虑到低频噪声词可以在使用关联规则处理的过程中过滤掉，本文所指的噪声词过滤主要是对高频噪声词的过滤，对于这些词汇，本文的解决方式是使用文献分类的方式，将一个大的领域分为多个细分领域，这样，那些同属于多个细分领域的词汇就可以被看做是非专业词汇，也就是噪声词，可以将这些词汇过滤掉。在研究中文分词算法以及专业词汇提取算的基础上，本文设计了一个原型系统来测试算法的有效性，该系统包括文献预处理、中文分词、文档切割、高频噪声词过滤、发现频繁项目集、提取新词等模块，涵盖了本文所研究的各个环节。测试表明，本文的中文分词算法可以有效地发现新的词汇，而专业词汇提取算法也取得了良好的效果。本文的创新之处有：（1）将N-Gram方法和Apriori方法结合在一起，并对两种方法加以改进。利用N-Gram方法结合词典方法进行中文分词，利用Apriori方法抽取出特定领域的专业词汇。两种方法的结合形成了一个完整的文本信息处理的过程，有利于实际应用。（2）设计一个智能化的新词发现系统。将本课题所研究的算法应用于实际中去，使之不仅仅局限于在样本数据中进行新词发现，还能够对新出现的文本随时发现新词。这个功能可以应用于搜索引擎当中，随时捕获搜索结果中所出现的新的专业词汇。本课题所研究的系统只需要更换一个预设专业词典，便可以适用于不同的领域，具有良好的扩展性。更多还原

【Abstract】 Along with development of information technology,electric documents in all areas become moreand more rich, it’s difficult to deal with the information of these documents.Chinese has its ownorganizational structure,and its information processing has a high demand for word segmentationtechnology,so the processing of Chinese is more difficult than English.It’s very important to find thenew professional terms against the grammatical features.The subject of this thesis is not multi areas,but single specific area. The area could expand, it maybe the area of financial, or IT, depend on yourdemanding.If you determined your subject area and have the documents of this area, the processing of thisthesis could devide into two steps. Firstly, word segmentation. In this step, the thesis uses the methodof N-Gram, through what we could find new words in the documents.The second step is extracting the new professional terms. In this step, we find the new professionalterms according to the professional terms in the dictionary. The thesis uses the method of Apriori inthe step, firstly, find the frequent itemsets, and then generate association rules to extract professionalterms. There is a problem in the step, that’s the filtering of noise words, considering that thelow-frequency noise words could be filtered in the processing of Apriori, the main problem ishigh-frequency noise words. For these words, the thesis use the way of classification. Cut a big area toseveral subdivision areas, and the words which belong to several areas are high-frequency noisewords.Based on algorithm studing, designed a prototype system to test the effectiveness of the algorithm.The system includes pretreatment, word segmentation, documents cutting, high-frequency noisewords filtering, frequent itemset finding, new professional terms extracting and so on. The result oftesting proves that the algorithm could find the new professional terms effectively.The creative works of this thesis are：(1) Combine methods N-Gram and Apriori, and improve the algorithm.Use N-Gram method anddictionary to segment Chinese documents，use Apriori method to extract professional terms of thearea. The two algorithms combined and formed a complete processing of documents information,which is helpful for practical application.(2) Design a intelligent professional terms-finding system.Used the algorithm of the thesis.Thissystem will not only adjust to sample data,it could find new words depending on your needs.Thisfunction could used in search engine,so the engine could capture the new words from search results.If your change a default professional terms dictionary，it will adjust to another area，so it has goodscalability.更多还原