【摘要】 TF-IDF方法是文本向量化过程中一种常用的特征项权重计算方法,衡量的是特征项在整个文档集中的重要性.针对文本分类过程中TF-IDF方法未能体现特征项对类别的区分能力和对类别的代表性问题,基于文档类别,结合特征项的类间区分度和类内贡献度,提出一种改进的TF-IDF权重计算方法,并采用KNN和SVM模型对改进后算法的分类性能进行了验证.实验结果表明,与传统的TF-IDF方法相比,改进后的权重计算方法不仅在整个测试数据集上能够取得较高的宏平均精确率、宏平均召回率和宏平均F1,而且使测试数据集绝大部分类别的分类性能得到了较大提升.因此,改进后的TF-IDF权重计算方法是有效且可行的.更多还原

【Abstract】 TF-IDF is a commonly used method to weigh features in the process of text vectorization,which measures the importance of features in the whole data set.Aiming at the problem that TF-IDF method fails to reflect the ability of features to distinguish and represent classes in text classification,this paper proposed an improved TF-IDF method based on document category,and the inter-class discrimination and intra-class contribution of features.Besides,this paper used KNN and SVM models to validate the performance of the improved method in text classification.Result shows that compared with the traditional TF-IDF method,the improved one not only achieved higher macro-average precision,macro-average recall and macro-average F1 on the whole test dataset,but also greatly improved the classification performance on most categories of the test dataset.Therefore,the improved method is effective and feasible.更多还原