【作者】 王威；

【导师】 李丹程；

【作者基本信息】 东北大学 ， 软件工程， 2015， 硕士

【摘要】 中文分词作为中文自然语言处理的基础正受到越来越多的关注,准确快速地识别出中文语句中的词语对于有效地理解汉语语句内容以及后续处理任务意义重大。基于统计学习的中文分词方法通过对已标注语料库的分析来训练分词模型,并采用分词模型来预测未知语句中汉字的词位标记从而达到识别句中词语边界的目的。由于汉语本身特点,现有中文分词方法难以有效识别出语句中存在的命名实体,而相关改进方法在模型训练效率和预测效率上难以让人满意。如何快速准确地完成中文分词一直是中文自然语言处理所需解决的主要问题之一。本文采用条件随机场作为分词方法所用模型,通过对中文分词预处理方法和中文分词算法的分析提出一种改进的基于统计学习的中文分词方法。首先,本文对中文词语特征进行分析,提出一种复合词位标记集合来使模型在引入少量参数的前提下能够在分词的同时更好地识别出语句中的命名实体,并提出了采用复合词位标记集合的分词模型的计算方法。然后,对于现有模型特征提取算法获取到的特征无法表达其对标记影响程度的不足,本文提出一种改进的特征提取算法,该算法通过综合考虑特征共现频率以及特征对标记结果的间接影响来计算实值特征函数,通过为模型训练设置合理迭代初始点来提高训练效率。其次,针对现有基于L-BFGS模型训练算法效率较低的不足,本文提出一种改进模型训练算法,通过合理设置每次迭代的学习步长提高训练速度并减弱噪声数据的负面影响,另外,本文提出一种基于Viterbi的改进模型预测算法以用于采用复合词位标记集合分词模型的预测标注,并通过引入遍历剪枝策略来提高算法预测效率。最后,提出一种基于错误转换的改进分词后处理算法来进一步提高分词方法的准确率。本文采用实际的标注语料库作为训练数据集和测试数据集来对提出方法的合理性和有效性进行验证。实验结果证明,本文提出的改进方法能够有效地识别中文语句中的词语边界,而且相比于其他方法具有更优的分词准确率以及相对较好的分词效率。更多还原

【Abstract】 As a foundation of Chinese natural language processing,Chinese word segmentation(CWS)attracts more and more attention,and accurate and quick CWS is significant for understanding Chinese statement content and subsequent processing task.CWS method based on statistical learning trains segmentation model by analyzing annotated corpus and predicts the lexeme label of each character in unknown statement to recognize word boundary.Due to Chinese characteristics,existing CWS methods are difficult to recognize named entities and improved methods are dissatisfactory on the training and prediction efficiency.How to achieve CWS correctly and accurately is one of the main issues that Chinese natural language processing needs to solve.In this paper,conditional random field is adopted as CWS model,and an improved CWS method based on statistical learning is presented by analyzing CWS preprocess method and CWS algorithm.Firstly,Chinese word characteristics is studied and a compound lexeme label set(CLLS)is presented to enable CWS model recognize named entities better on the premise of introducting few parameters,and calculation method of CWS model adopting CLLS is proposed.Then,for the drawback that features acquired by existing model feature extraction(MFE)algorithm cannot express their influence on labeling result,an improved MFE algorithm is presented,which analyzes co-occurrence frequency and inter-influence of features to calculate the real-valued feature function and set a reasonable initial iteration point for model training to improve training efficiency.Furthermore,for the drawback that existing model training algorithms based on L-BFGS have a low efficiency,an improved model training algorithms is proposed to accelerate training speed and weaken negative effect of noise data by setting reasonable learning step.Moreover,an improved model prediction algorithm based on Viterbi for the CWS model adopting CLLS is proposed and then a traversal pruning strategy is introduced to improve the prediction efficiency.Finally,an improved CWS postprocessing algorithm based on error-driven transformation is presented to to further improve the accuracy of CWS method.In this paper,the actual annotated corpuses are adopted as training data set and test data sets to validate the rationality and validity of the proposed CWS method.Experimental results show that the proposed method can effectively identify the word boundaries in a given Chinese statement,and has a better accuracy and a relatively good CWS efficiency compared with other related methods.更多还原