【作者】 徐中一；

【导师】 刘磊；

【作者基本信息】 吉林大学 ， 计算机软件与理论， 2007， 硕士

【摘要】 本文对中文信息抽取的三个方面进行了研究。本文提出了一种效率较高的中文分词机制,在基于词典形成有向无环词图后加入基于碎片合并的未登录词的推测算法,使词图的节点减少,从而提高基于N-最短路径算法的分词速度,同时提出了基于零孤立点原则的消歧算法,实验表明,该算法能在保证准确率的条件下提高中文分词速度。本文使用最大熵模型实现中文命名实体识别,通过特征选取、参数估计、解码三步实现了该系统,并对简单特征和复杂特征进行了比较,实验表明,基于最大熵进行中文命名实体识别是可行的,并且利用复杂特征模板达到最好效果。本文选择条件随机域作为中文组块分析的模型,采用北大公开的中文树库语料,选用了单一特征、复合特征、状态转移特征,分别利用1阶、1阶+、2阶模型对中文组块进行标注,实验表明,利用2阶条件随机域模型进行中文组块分析达到最好效果。更多还原

【Abstract】 Along with the popularization of computer and the high speed of thedevelopment of Internet, a wealth of information in the form of electronicfiles comes in front of people. In response to the serious challenges posed bythe information explosion, automatic tools are needed urgently to help peoplefind really useful information quickly in amounts of information sources. Asthis background, the study of Information Extraction came into being.Information Extraction is a process which extracts specified events, factsand other information from a section of text, forms the structured data andinserts into a database for users to query. It extracts facts, entities andrelationships which users are interested in from the text, and then entersdatabases, analyzes trends, gives digest or goes along online services. Thispaper describes the history and actuality of Information Extraction, and givesthe distinction and association between Information Extraction andInformation Retrieval. Because of particularity of Chinese, the research onChinese Information Extraction started relatively later, the research mainlyfocused on Chinese Named Entity Recognition, designing while realizing aintegrated Chinese Information Extraction System is still in the exploratorystage.The first chapter of this paper outlines Information Extraction generallyand gives the paper’s work. The second chapter describes four statisticallanguage models. Statistical language model is distribution function of alanguage’s basic unit (words, phrases, sentences. We generally regard asentence as a language’s basic unit).The model describes generated rules ofthe language based on statistics. This paper introduces the Hidden MarkovModel (HMM), Maximum Entropy Model (ME), Maximum Entropy MarkovModel (MEMM) and Conditions Random Field (CRF),and givescomparisons of the above models, lays a good foundation for the followingchapters which deal with Chinese based on machine learning.Because Chinese sentences have no separation among words, so ChineseInformation Extraction System needs to produce a dictionary which hasproper scale, reasonable classification and flexible collocation, to establish aChinese automatic segmentation algorithm devoted to Information Extraction. The third chapter of this paper gives Chinese automatic word segmentationalgorithm applied in Information Extraction some improvements as follows.Due to the time efficiency, this paper uses Trie index tree which inquiriesmore rapidly to establish a dictionary. Word segmentation algorithm, first,form a word diagram which has orientation but no ring based on thedictionary, second, conjecture the unknown words based on fragmentmerging algorithm, so that nodes of word diagram reduces, and improveeffectively the speed of word rough segmentation based on N-shortest pathnext, and finally, carry through disambiguation base on zero isolating pointprinciple. Experiments show the word segmentation mechanism is effectivein Chinese Information Extraction, it can improve the speed, but the accuracyand recall rate have to be raised further.In an article, entity names are basic information elements, often theyinstructed the main content of an article. Named Entity Recognition isjudging whether or not a text string represents a named entity anddetermining its type. Named Entity Recognition is a prerequisite forunderstanding the text, and will directly affect the quality of following work.The fourth chapter of this paper tries to use the Maximum Entropy Model torealize the recognition of four Chinese named entities (persons, places,organizations, and proper nouns). Maximum Entropy Model has no strictindependence hypothesis, can select features free and integrate moreinformation. This paper realized Chinese Named Entity Recognition based onMaximum Entropy Model through feature selection, parameter estimationand decoding, and compared with the simple feature and complex feature,achieved an overall result F=92.7%. Experimental datasets indicate thatChinese Named Entity Recognition based on Maximum Entropy Model isfeasible. and it can achieve best result using the complex feature。But therewas still a certain distance from the practical application, we can also see thatthe recognition of person and place names can get high accuracy and recallrate, organizations and some proper nouns come to be a little lower.Organizations often have a greater span scope, therefore, using differentfeature template for different kinds of Named Entity is also a possiblesolution.Chunking is analyzing sentences into smaller units, in order to provide abasis for further revealing the syntactic relations among these modules. In the fifth chapter of this paper, Chinese Chunk were divided into 11 typesaccording to Chinese’s characteristic, then the process of Chinese textChunking was transformed into a process that labeling the words with chunktags. This paper selected Conditions Random Field(CRF) as the model forChinese Chunking, because the condition model can utilize words ,pos ,multi-layered resources and also can avoid Strict Independence Hypothesisand Label Bias problem. Using Chinese Treebank of Beijing University asthe dataset, this paper chooses single feature, composite features and statetransferring feature, uses 1 bands, 1 band +, 2 band model separately to tagChinese Chunk. Experiments show that Chunking using CRF model isfeasible. and using 2 Band CRF model can achieve best results. However, theestablishment of CRF model in the process of training data is more slowly.So finding a rapid modeling algorithm is a problem to be resolved.The above three aspects have a common problem, the scale of the corpusis small. So using large-scale and high-quality corpus to train modelparameters is essential for improving the performance of the system. Trainingcorpus as resources which need to spend huge manpower and financial, hasfew free resources, so the next step can consider paying corpus to increasethe size of the corpus.This paper only studied Chinese word segmentation, Chinese NamedEntity Recognition and syntax parsing, the three areas of Chinese InformationExtraction. And following affairs might be chapter analysis and reasoning,knowledge acquisition and other tasks. Therefore, The next step we shouldconsider is increasing the accuracy, recall rate and time efficiency of theabove-mentioned three aspects as high as possible, and applying thetechnology of natural language processing for text analysis and knowledgeacquisition, thus extracting the information we need. Other problems couldbe further researched such as how to transform the text information weextracted into XML (Extensible Markup Language) documents and how tointegrate all algorithms and techniques, for instance extracting informationfrom Word documents, various file format conversion. Realizing a ChineseInformation Extraction system which can meet real world applying is adirection that we make effort.更多还原