【作者】 张爱华； 靖红芳； 王斌； 徐燕；

【Author】 Zhang Aihua~1,Jing Hongfang~1,Wang Bin~1,Xu Yan~2 1 Institute of Computing Technology,Chinese Academy of Sciences,Beijing,100190 2 Beijing Language and Culture University,Beijing,100083

【机构】 中国科学院计算技术研究所； 北京语言大学；

【摘要】 在文本分类任务中,通常首先以TFIDF模型表示特征权重,进而通过特征选择函数给出的得分将特征排序,排名靠前的特征以其TFIDF值输入分类器。特征选择函数所计算出来的得分能一定程度代表特征的重要性,然而该得分却没有被纳入权重表示中,造成特征表示不精确,影响了分类性能。许多改进方法使用特征选择函数等反映特征在不同类别间分布的指标来替换IDF因子或将其加入TFIDF模型中,使分类性能得到了一定的提高,但没有探究各个因子是如何影响分类性能的。本文以词频(TF)、逆文档频率(IDF)以及特征选择函数(IG及CHI)作为衡量特征的文档代表性、文档区分性以及类别区分性的因子,通过实验方法测试了这些因子对于分类性能的影响,得到如下结论:文档代表性因子能够使得分类效果的峰值最高但抗噪音能力差,文档区分性因子具有抗噪音能力但性能不稳定,类别区分性因子的抗噪音能力最强,性能也最稳定。进而说明了如何组合这些因子以得到分类性能的优化,给出了四点构造原则,并通过实验得到了验证。更多还原

【Abstract】 In text categorization,terms were reprensented by TFIDF model,then ranked by scores calculated according to feature selection functions,finally top ranked features were put into classifiers with their TFIDF forms.Scores calculated by feature selection functions can express the importance of features,but they were not elements of the TFIDF weighting schema.As a result,when treating the term weights,the classifier did not take the ranking of features’ importance into accout.Many improving methods used feature selection functions to express the distribution of features and used it to replace the IDF factor or added it into the TFIDF model. However,they did not explain how these factors influence the performance.This paper used term frequency(TF), inverse document frequency(IDF) and feature selection funcetion to express the ability of the features to represent a document,distinguish different documents and distinguish different categories.According to the experiments, TF can raise the peak of the performance but its antinoise performance is poor,IDF resists noise but unstable, feature selection functions have strong ability to resist noise.Furthermore,how to combine these features to get optimal weighting methods was suggested,experiments verify that.更多还原