【作者】 肖伟；

【导师】 唐卓；

【作者基本信息】 湖南大学 ， 计算机科学与技术， 2019， 硕士

【摘要】 随着信息社会的迅速发展,互联网得到了广泛的应用,目前已然成为最重要的信息来源。特别是随着云计算和大数据的出现,互联网产生的数据量呈指数等级增长。这些数据具有以下特点:数量大,维度高,结构复杂,噪声多,但具有广泛的应用前景。此外大部分的数据都是以文本的形式存储在互联网上。如何去有效的组织并使用这些数据,在当前计算能力不足的情况下将会成为一个巨大的难题,特别是在互联网应用程序中,当用户希望能准确、高效、快速地从大量的数据查找到自己需要的数据时,会对系统的计算能力和自然语言处理的算法提出更高的要求。朴素贝叶斯算法(Naive Bayesian)是十大经典的数据挖掘算法之一,它作为在文本分类中的基本理论而广为流传。随着互联网和信息系统的高速发展,海量的数据在随时随地被产生出来。当传统的Bayesian分类算法应用到海量的大数据中时一些问题就会暴露出来。文本提出了一种在Spark计算环境中进行文本分类的Bayesian改进算法:INBPCS。因为多元伯努利模型并没有考虑属性维度对文本分类的影响,因此本文改进了基于多项式模型的Bayesian算法。另外对于文本分类的数据处理过程,本文首先提出了一种并行的噪声消除算法,然后在计算各个属性的信息增益值算法的基础之上提出另一种并行的维度缩减的算法。基于前面的数据预处理的过程,一种用于计算条件概率的并行算法就被提了出来,该条件概率主要考虑到了属性维度在不同的文本、类别以及训练集中的影响,同时本文提出了一种基于多种机器学习算法的混合预测算法提升Spark在Shuffle阶段内存预测的准确度。最终,在Spark平台上对多种不同的大型语料库数据集进行测试,结果显示改进后的INBPCS算法要比其他流行的文本分类算法具有更高的性能和准确率。更多还原

【Abstract】 With the rapid development of information society,the Internet has been widely used and currently has become the most important source of information.In particular,with the emergence of cloud computing and the big data era,the data generated from the Internet are rapidly growing with the index grade.These data have the following characteristics: large in amount,high in dimension,complex in structure and containing much noise,but widespread application prospects.Furthermore,most of the information and data stored on the Internet are text.How to organize,manage,and utilize these text data is a great challenge for the currently limited computing power,especially when confronts with a large amount of information that needs to be searched effectively,quickly and accurately by users for the Internet applications.The Naive Bayesian algorithm is one of the ten classical algorithms in data mining,which is widely used as the basic theory for text classification.With the high-speed development of the Internet and information systems,huge amount of data are being produced all the time.Some problems are certain to arise when the traditional Bayesian classification algorithm addresses massive amount of data,especially without the parallel computing framework.This paper proposes an improved Bayesian algorithm INBPCS,for text classification in the Spark computing environment and improves the Naive Bayesian algorithm based on a polynomial model.For the data preprocessing,this paper first proposes a parallel noise elimination algorithm,and then proposes another parallel dimension reduction algorithm based on Information Gain computation in the Spark environment.Based on these preprocessed data,an improved parallel method is proposed for calculating the conditional probability that comprehensively considers the effects of the feature items in each document,class and training set,Meanwhile,this paper proposes a hybrid prediction algorithm based on multiple machine learning algorithms to improve the accuracy of Spark’s memory prediction in the Shuffle phase.Finally,through experiments on different widely used corpuses on the Spark computation platform,the results illustrate that INBPCS can obtain higher accuracy and efficiency than some current popular text classification algorithms.更多还原