【作者】 张莉；

【导师】 焦李成；

【作者基本信息】 西安电子科技大学 ， 电路与系统， 2002， 博士

【摘要】 统计学习理论研究基于样本的机器学习问题，由于最初的发展主要停留在对学习模型和学习算法的理论分析上，并没有得到一些实际有效的结果。但是到九十年代早期，在Vapnik等人的不断努力下，统计学习理论得到了进一步完善和发展，已基本形成一套比较完整的理论体系，并且由该理论直接导出了一种非常有效的全新的学习算法：支撑矢量机。支撑矢量机是一种普适的算法，已经广泛地用于模式识别、回归估计、函数逼近、密度估计等方面。本论文研究的主要内容包括四个方面：支撑矢量机算法的改进、核函数的构造、核方法的应用以及基于统计学习理论构造新的学习机。主要的工作有： 1．在算法改进方面，提出了支撑矢量预选取的中心距离比值法。已有的支撑矢量机分类学习算法的优化过程中不仅包含了对支撑矢量的优化，也包括了对非支撑矢量的优化，这大大增加了不必要的计算量。而在通常情况下支撑矢量在训练样本集中只占很小的比重。我们严格证明了支撑矢量机的学习过程、学习结果以及决策函数与非支撑矢量无关。我们提出的方法是在给定的样本集中提取一个包含支撑矢量的边界矢量集合作为新的训练样本集合。如果中心距离比值的阈值选择合适，我们的算法能够在给定样本中提取出一个包含了所有支撑矢量的边界矢量集合作为新的训练样本集。在保证支撑矢量机的分类能力情况下，该方法大大地减少了训练样本的个数，同时提高了支撑矢量机的训练速度。对于多分类问题，我们提出了一种基于决策树的支撑矢量机算法。该算法把一个多类问题分解成多个两类问题，整体采用二叉决策树结构，使得训练样本或类别数逐级减少。在每一个非终止节点上首先采用核聚类算法生成一个两类问题，由支撑矢量机完成两类识别。这样逐级往下实现基于二叉决策树的支撑矢量机多分类算法。该支撑矢量机多分类算法和其它的基于二分问题的多类算法比较起来，在训练次数、每次训练的样本数和支撑矢量的个数上都相对要少，因此该方法在训练速度和检验速度上有一定的优势。 2．在核函数的构造方面，我们提出了几种可容许的支撑矢量核，包括三种坐标变换核、子波核和尺度核函数。坐标变换核函数是一种能够提供变换角度观测和处理的工具，特别是对于某些具有一定先验信息的学习问题，该方法是非常有效的。子波基函数包括子波函数和尺度函数，都具有良好的时间尺度(时频)多分辨特性和逼近性能。为此我们构造并证明了子波核函数和尺度核函数，为其在支撑矢量机中的应用架设了桥梁。理论和实验证明基于子波核函数和基于尺度核函数的支撑矢量机不仅继承了支n大娜翻月州卜匀翻匕少.闷傲犯................撑矢量机良好的推广能力而且也拥有了子波基函数优越的通近性能。这些核函数为支撑矢量机或其他的采用核方法的算法提供了更多的选择. 3.在核方法的应用方面，我们把核方法引人到聚类分析和神经网络中，提出了一种核聚类算法和用子波核构造的一种神经网络一一子波核函数网络.通过核方法，把模式从输人空间映射到高维的特征空间，在此空间中对样本进行线性的逼近、识别或聚类。子波核函数网络最大的优点是网络的规模不会随着输人模式维数的增长而呈指数增长。网络隐层的单元数只与训练样本的个数有关，而与输人模式的维数无关。子波核函数网络的训练算法采用的核感知器算法，避免了求解有约束的二次凸规划问题，并且该网络不存在局部极小点。另一方面我们提出了一种核聚类方法，在性能上比经典的聚类算法有较大的改进，它通过非线性映射能够较好地分辨、提取并放大有用的特征，从而实现更为准确的聚类，算法收敛速度也较快。在经典聚类算法失效的情况下，核聚类算法也能够得到理想的聚类。 4.基于统计学习理论中函数集容量的概念，我们提出了一种基于微分容量控制的学习机。在支撑矢量机中，对函数集容量的控制是通过函数集的VC维数进行的，而函数集的VC维数是一个不依赖于样本分布的量，由它导出的推广能力界应该说也是一个宽松的界。我们发现函数的微分范数可以控制函数集的多样性，它依赖于样本的分布。我们类似于确p献引人结构风险实现经验风险最小化和对函数集多样性的控制，构造了基于微分容量控制的学习机，其推广能力的界依赖于样本的分布。在另一方面基于微分容量的学习机与正则理论也是一致的，微分容量控制项相当于一个高通滤波器，满足正则泛函中的光滑项的要求。基于微分容量控制学习机可以采用任意一阶可微函数构成假设函数集(包括M~核函数)，若假设函数集取线性函数集，此时的学习机就是线性支撑矢量机。更多还原

【Abstract】 The statistical learning theory (SLT) is a tool for studying the model of learning from examples. At the early days of the research on SLT, it did not come into front for its purely theoretical analysis. But in the middle of the 1990’s, some new types of the learning algorithms (or support vector machines, SVMs) based on the developed theory were proposed by Vapnik et al. SVM is a kind of general learning algorithms, which has been widely used in pattern recognition, regression estimation, function approximation, density estimation, etc. This dissertation studies the improvement of support vector algorithm, the construction of admissible support vector kernels, the application of kernel method and the construction of new learning machine based on SLT.For the improvement of SVMs, center distance ratio methods are proposed for pre-extracting support vectors. The learning procedure of SVM includes all examples, in which,, however, only a few examples (namely the support vectors) play an important role on the decision function of SVM. The method provided in this dissertation is able to extract a verge vector set containing all support vectors from a given training set if the threshold is chosen reasonable. This method greatly reduces the number of training examples and so improves the training speed of SVM, without the loss in generalization performance. For the multi-class problem, a method based on decision tree is presented that is able to reduce a multi-class classification problem to multiple binary classification problems each of which is solved by SVM. Besides, a binary decision tree structure is adopted, which makes the number of the training examples and the classes on the children nodes are smaller than that on the parent nodes. Compared with other multi-class methods for reducing a multi-class problem to multiple binary problems, our method has the less workload system training, as well as the smaller number of the training examples for every time and the smaller number of the support vectors. Therefore the method has faster testing speeding than the previous.On the other hand, some admissible support vector kernels are proposed, including three coordinates-transform kernels, a wavelet kernel and a scaling kernel. The coordinates-transform kernels are a kind of kernels that incorporates the prior knowledge of some learning problems for which these kernels are efficient. Both the wavelet function and the scaling function have good approximation performance. Besides, both the wavelet kerneland the scaling kernel can be regarded as high-dimensional wavelet and scaling function respectively. Therefore, the wavelet kernel and the scaling kernel also have good performance in approximation ability. All these kernel functions provide more choices for SVMs and the other algorithms in which the kernel method is used.By introducing the kernel method in neural networks and the clustering analysis, a wavelet kernel function network (WKFN) and a kernel clustering algorithm are constructed, respectively. By the kernel mapping, the examples in the input space can be mapped into a high-dimensional feature space in which linear approximation, recognition and clustering are performed. The merit of WKFN is that the curse of dimension can be avoided. The number of hidden nodes is decided not by the dimension of the examples but the number of training examples. WKFN uses the kernel perceptron algorithm so as to avoid solving the constrained quadratic programming. Moreover the solution obtained by WKFN is global optimal, and the kernel clustering algorithm can obtain better performance than classical clustering algorithms by using kernel mapping.In the end, a novel learning machine is presented based on differential capacity control in which the bounds on the generalization ability are the distribution-dependent bounds. The capacity control is implemented by controlling on the differential of the set of functions. The learning machines can use any one-order differentiable functions including Mercer kernel functio更多还原