【摘要】 为降低内容无关的特征词对文本分类系统的影响,在对与文本内容无关的特征词进行分析后发现:不相关特征词的词频普遍较低,利用最小词频阈值滤除低频特征可以明显降低无关特征的数量。为此,提出基于最小词频阈值的文档频评估函数。利用该函数选择特征可以有效减少与内容无关的噪声特征,改善分类质量。实验结果显示,几种基于最小词频阈值的文档频评估函数比基于普通文档频的评估函数的分类准确性有不同程度的改进,其中对互信息的改进最为显著,宏平均F₁值比词频方法提高40%,比普通文档频方法提高15%～30%。更多还原

【Abstract】 In this paper, a novel method of feature evaluation function based on document frequency with the minimum term frequency threshold (DF_n) is presented. To decrease the influence of the unrelated features on the system of text categorization, the attribute of the unrelated features is analyzed and the term frequency of the unrelated feature is commonly low. By applying minimum term frequency to filter the low frequency features, the unrelated features are obviously decreased. The experimental results validate the proposed method greatly reduces the number of the unrelated features and effectively improves the accuracy of the text categorization. The improvement to Mutual Information(MI) is very obvious, the Macro-average F1 value based on DF_n is 40% higher than that of Term Frequency, and 15～30% higher than that of Document Frequency(DF).更多还原