【摘要】 随着全球化趋势和国际交流的日益频繁,语言之间的渗透与融合日渐增加,不同语种夹杂的表达方式在生活中也相当普遍,特别是在使用双语或多语的地区。作为一个多民族融合的国家,我国各地的方言与普通话夹杂的表达方式也不在少数。这种语言混用的现象造成了分词识别上的困难。本文收集了各种多语夹杂的相关语料,分析了多语夹杂的语言特征,在此基础上提出了以互信息(mutual information)和熵(entropy)过滤为基础的新词识别算法,有效提高了从混合语料中识别新词的精度。更多还原

【Abstract】 With the trend of globalization and the increasing frequency of international communication, the penetration and integration of languages are increasing day by day. The expression of mixed languages is quite common in our life, especially in areas where bilingual or multilingual languages are used. As a multi-ethnic country, there are also a few expressions mixed with Mandarin in local dialects, which is a phenomenon of mixed language.This paper collects relevant corpus of various multilingual inclusions and analyses the linguistic characteristics of multilingual inclusions. Based on this, a new word recognition algorithm based on mutual information and Entropy filtering is proposed, which effectively improves the accuracy of recognizing new words from mixed corpus.更多还原