【摘要】 针对Spark平台下文本分类中文本表示方法不够完善，导致分类准确率低的问题，结合Spark ML下的TF-IDF算法和Word2vec模型，提出一种基于Spark ML的加权词向量文本表示方法。首先对文本进行分词，去停用词等预处理，基于Spark ML计算出每个词语的词频和逆文档频率，同时计算词语的词向量。使用词语的TF-IDF值作为词向量的权重，将文本表示为加权词向量，再使用SVM分类器进行分类。在THUNews数据集上进行实验。实验结果表明，该方法相比于传统的TF-IDF算法、平均Word2Vec词向量文本表示，可以提升分类的精度。更多还原

【Abstract】 Aiming at the problem of low classification accuracy due to imperfect text representation method in Spark platform,a text representation method of weighted word vector based on Spark ML was proposed by combining TF-IDF algorithm and Word2vec model. Firstly, the text is preprocessed by word segmentation and word stopping, and Spark ML is based to calculate the word frequency and reverse document frequency of each word, as well as the word vector of the word. The TF-IDF value of the words is used as the weight of the word vector, and the text is represented as the weighted word vector, and then the SVM classifier is used for classification. Experiments were performed on the THUNews dataset. Experimental results show that this method can improve the classification accuracy compared with traditional TF-IDF algorithm and average Word2Vec word vector text representation.更多还原