【摘要】 日语分词和词性标注是以日语为源语言的机器翻译等自然语言处理工作的第一步。该文提出了一种基于规则和统计的日语分词和词性标注方法,使用基于单一感知器的联合分词和词性标注算法作为基本框架,在其中加入了基于规则的词语的邻接属性作为特征。在小规模测试集上的实验结果表明,这种方法分词的F值达到了98.2%,分词加词性标注的F值达到了94.8%。该文所采用的方法已经成功应用到日汉机器翻译系统中。更多还原

【Abstract】 Word segmentation and part-of-speech tagging is the first step of Japanese natural language processing tasks,such as machine translation in which Japanese is the source language.In this paper,a Japanese word segmentation and POS tagging approach based on rules and statistics is proposed.Adopting a single perceptron based joint word segmentation and POS tagging algorithm as the basic framework,this method is combined with the features of adjacency attributes which are derived by heuristic rules.The experiment on a small test dataset shows that the new approach achieves an F-score of 98.2% on word segmentation,and 94.8% on both word segmentation and POS tagging.This work has already been applied into the Japanese-Chinese machine translation system successfully.更多还原