【作者】 万中英；

【导师】 王明文；

【作者基本信息】 江西师范大学 ， 计算机软件与理论， 2004， 硕士

【摘要】 如何对网上的文本信息进行分类，使Internet这个巨大分布式信息空间的无序状态有序化，已成为文本信息处理和计算机研究领域亟待解决的一个问题。 一般来说，由于文本集中的特征项可能多达数万个，那么把文本表示为向量形式时，特征空间的维数也就高达数万维，如此高维的特征向量的处理具有极高的计算复杂度，用常用的分类算法进行处理，高维带来的噪音会淹没真正的对分类有用的信息，尤其是会产生所谓的“维数灾难问题”，所以人们为提高分类速度，降低噪音的影响，应用降维技术处理维数灾难。现有的文本自动分类中的降维大多采用特征选择的方法，选择一些主要特征，即通过评价函数进行降维，但通过这种方法选择的特征项中可能还包含一些彼此相关的因素，也就是说有些特征是冗余的。降维的另外一种方式是利用映射(或变换)的方法(也称特征提取)把原始项集映射到较低维的空间中，通过降维映射，构造总数量较少的新特征集，其中每个特征都是原有特征的函数，并通过新特征进行识别。经典的算法有：主成分分析和Fisher线性判别分析。但这些方法都是建立在数据总体服从正态分布这个假定基础之上的，而文本特征数据并不满足正态分布假定，需要用稳健的或非参数的方法来解决这个问题。 基于上述原因，我们提出了基于投影寻踪的中文网页分类算法。本文的主要思想是：把高维数据投影到低维子空间上，寻找出最能反映原高维数据的结构和特征的投影方向，然后将文本投影到这一方向，使高维数据降低维数，通过研究降维后低维数据的散布情况来揭示高维数据的结构特征。 本文的主要创新点如下： (1) 对数据不做正态分布等任何假定，利用遗传算法寻找最优投影，将高维的文本向量投影到一维空间。 (2) 首次将投影寻踪的方法应用于文本分类，将文本向量投影到一维空间后，再利用KNN方法进行文本的自动分类，是投影寻踪和KNN算法的结合。实验证明，这种算法对中文网页的自动分类具有较好的召回率和准确率，而且大大地缩短了分类时间。 并通过与相似度计算和朴素贝叶斯方法的对比实验，表明此方法不仅具有较好的召回率和准确率，而且有着较好的稳定性。更多还原

【Abstract】 How to classify the document information of Internet? This important problem needs to be solved in document information processing and computer research domain.In general, text is represented by vector model, this representation raised the high dimensionality of the feature space. If we use this high dimensional vector for text classification, the curse of dimensionality will be raised, so we should use dimensional reduction to avoid this problem for classification. Most text classifications reduce dimensionality by using feature selection that can choose a subset from the original feature set according to some criterions, this method may neglect some relevant factor. Another method is feature extraction. Feature extraction is a process that extracts a set of new feature from the original feature through some functional mapping, such as Principal Component Analysis and Fisher Linear Discriminant Analysis. Those techniques have an assumption of normal distribution. Text data does not satisfy the assumption of normality that those methods are based on ,so we need a Robust or nonparametric method to resolve this problem.Based on above reasons, we propose a Chinese Webpages classification algorithm based on Projection Pursuit. The procedure is to project the data from a high dimensional space to a lower dimensional subspace, and find the projection direction that can reflect the construction and feature of the high dimensional data, and then the text is projected to this direction. The distribution of data in a lower dimensional subspace that is a result of projection from the original high dimensional space will discover the construction of the high dimensional space.The main creative points of this paper are:(1) We use genetic algorithm to search the best projection direction without the assumption of normality, and text vector is projected to a one dimensional space.(2) Projection Pursuit is used firstly text classification. After text vector is projected to one dimensional space, we classify the test-set using KNN algorithm. The result of experiment shows that the recall and precision are better than other method.We also do some experiment by using Similarity method and navie Bayes method. The experiment shows that Projection Pursuit not only has better recall and precision, but also has better stability.更多还原