【作者】 董元方；

【导师】 李雄飞；

【作者基本信息】 吉林大学 ， 计算机软件与理论， 2011， 博士

【摘要】 本文首先阐述机器学习中模型选择问题的研究背景、现状,以及相关的研究工作。然后,围绕模型选择的度量、比较度量的方法和模型平均方法等方面的问题,展开论述和深入研究。本文研究内容和主要贡献包括以下几个方面:(1)提出能够描述经验ROC曲线不确定性的度量——gROC和gAUC,从理论上分析了gROC的若干性质,并利用双正态模型检验了gROC的合理性。理论和实验表明,gROC不仅考虑了得分向量的排序,而且考虑了得分及其不确定性,这使得它与其他ROC置信带方法相比,能够更好地估计真实ROC曲线。基于gROC和gAUC定义两种分类器性能评估度量——λAUC和ρAUC,这两种度量都同时反映AUC值和ROC曲线的内在不确定性。(2)利用等高线分析和秩相关分析方法,直观而清晰地探讨和比较了若干分类器性能度量。特别地,利用等高线分析,证实了分类器性能度量——wAUC在应对不平衡数据情况下的有效性。wAUC考虑代价的偏置,更加关注分类器在正类上的性能。在线性函数加权和指数函数加权情况下,绘制wAUC的等高线,探查了这两种情况下wAUC的性质。同时,通过Spearman秩相关分析,考察了wAUC与其他常见度量的相关性,结果表明wAUC与TPrate相关程度最大,适合用于不平衡数据分类器性能评估。(3)利用集成学习方法,提出一种XML文档分类算法——IL-AdaBoost。基于XML数据的动态特性,应用H-Dom模型挖掘频繁变化子结构,提出一种利用频繁变化子结构构建特征空间来表示样本空间的方法,并在特征空间上使用IL-AdaBoost算法,建立能够处理XML数据分类的集成增量学习算法。算法以XML频繁变化子结构为特征构建决策树桩,作为Boosting算法的弱分类器,改进了AdaBoost算法;利用泊松过程模拟新增XML文档的生成,反映XML文档的时变性,并更新样本分布,实现了增量学习;利用采样改善基本分类器的差异性,提高集成学习效果。本文针对分类器性能评价度量及模型平均等方面展开研究工作,分别取得了相应的研究成果,公开发表了相关论文。更多还原

【Abstract】 Machine learning classification algorithms all have inductive bias. And model selection, as an important part of machine learning, aims to choosing an optimal model after building multiple models based on given training data, i.e. choosing the proper bias. The optimal model generally has less complexity, and less prone to overfitting or underfitting. The common model selection methods include data reuse techniques, analytically methods, heuristic methods, the performance metrics and model average. Model average methods in machine learning, i.e. ensemble learning, choose and combine some models to build a new model which owns better performance.In this paper, the following aspects of work are researched based on the in-depth exploration of many model selection strategies for machine learning:(1) gROC curve method based on discernible granularity is proposed to compare the performance of classifiers. ROC curve is an important visual model selection method. However, in practice, we can only obtain the experience ROC curve than real ROC curve because the complete and whole data can not be obtained. The uncertainty of experience ROC curve affects the correctness of model selection. So, the concepts of gROC and gAUC are put forward based on analysis of discernible granularity of the scoring sequence, and some relative properties are discussed theoretically. The concepts use rank and score, not like ROC which only use rank, and take full consideration of the score’s uncertainty. Then the rationality of gROC is tested by binormal model after the calculation method is given. Using gROC to estimate ROC curve can avoid a large number sampling and improve the efficiency than other confidence band methods.On this basis, two model selection metrics,λAUC andρAUC, are proposed. The similar ratioλreflects the inherent uncertainty of experience ROC curve.λAUC integrates AUC value and similar ratioλ, and is a model selection metric under given discernible granularity.ρAUC takes care of AUC value and the uncertainty of ROC, butρis the average measure under all discernible granularities. Experimental results show thatρAUC is more effective thanλAUC on large-scale samples. All in all, the gROC can effectively reflect the uncertainty of ROC curve, and the model selection methods based onλAUC andρAUC are better than those based on AUC or sAUC. In some cases, gROC has stronger capability on comparison of classifiers’performance.(2) Isometrics analysis and rank correlation analysis are used to compare the performance measures of classifiers intuitively and clearly, and prove the effectivity of wAUC to dealing with imbalanced data.The shortcomings of traditional AUC is that it doesn’t consider the cost bias, and adapts the same weights (i.e. which is assumed to have the same cost with value 1) for each region during the calculation of AUC. For two-class imbalanced data, wAUC make the weights vary with the values of the true positive rate (TPrate) to pay attention to minority class that is more important in common. wAUC is compared with other common performance evaluation metrics by isometric analysis. The natures of wAUC with linear weighted function and exponential weighted function are explored by the simulation experiments, and the two isometrics are non-linear parallel. In addition, the case that the isometric of wAUC and the isometric of AUC under exponential weighted function intersect is analyzed. The isometrics analysis experimental results show that wAUC can distinguish the classifiers with the same AUC values, and can better evaluate the classifiers for imbalanced data. The rank correlation analysis confirms that wAUC has greater correlation with TPrate than other measures, and fits for imbalanced data learning.(3) An improved AdaBoost algorithm, IL-AdaBoost, for XML documents classification is proposed by making use of ensemble learning.XML document integrates the text content and structure information. And XML document classification is a research branch in XML data mining. Most those research work remain on static XML data. But in practical applications, XML data is often dynamic. Therefore, the data mining algorithms that can reflect the variability of XML data are urgently needed. IL-AdaBoost is proposed based on discussing whether the ensemble learning has possibilities to be used for XML classification.Based on the dynamic characteristic of XML documents, a method that builds feature space is proposed by applying the H-Dom model to mine frequently changing substructure. This method uses the feature space to represent the sample space, uses the IL-AdaBoost algorithm on feature space, and builds the ensemble incremental learning algorithm that can deal with XML data classification. It uses XML frequently changing substructure as the feature to build the decision stumps as weak classifiers of boosting algorithm to improve AdaBoost algorithms. It simulates new generation of XML documents through the Poisson process to reflect the time-varying characteristics of XML documents, updates the distribution of the sample to achieve incremental learning, and improves the differences of basic classifier by sampling to enhance performance of ensemble learning.To be conclusion, in this paper, the following tasks of the model selection problems for machine learning are accomplished in the light of classifier performance metrics, ensemble learning and other aspects. At first, gROC and gAUC are proposed, and two performance evaluation metrics based on gROC,λAUC andρAUC, are designed. Secondly, the experimental results on UCI data set show the advantages of these two metrics. The properties of the performance evaluation metric, wAUC, are analyzed by isometrics analysis and rank correlation analysis. Finally, the IL-AdaBoost algorithm for XML document classification is proposed.Although some research results have been obtained, there are also some areas for further study, for example, building the classifiers by optimizing wAUC, refining the general approach of model selection, and so on.更多还原