【作者】 陆声链；

【导师】 林士敏；

【作者基本信息】 广西师范大学 ， 计算机软件与理论， 2005， 硕士

【摘要】 孤立点挖掘又称孤立点分析、异常检测、例外挖掘、小事件检测、挖掘极小类、偏差检测。孤立点可能是“脏数据”,也可能是与实际对应的有意义的事件。从知识发现的角度看,在某些应用里,那些很少发生的事件往往比经常发生的事件更有趣、也更有研究价值。因此,异常数据的检测和分析是一项重要且有意义的研究工作。本文的主要工作可以概括为以下五个方面: (1)从孤立点挖掘的现实意义、算法、应用领域、挖掘工具、算法的评价等各个方面对孤立点挖掘问题进行了综述。(2)针对基于距离的方法需要设置阈值的不足,本文提出了一个新的孤立点定义,通过对象与数据集中所有其它对象的距离之和来判别异常,从而有效解决了算法需要设置最近邻参数p 或者k 的问题。为提高算法的效率,设计了基于抽样的近似检测算法。真实数据集上的实验表明该定义与DB(p,d)孤立点定义有相同的结果,同时给出了数据对象在数据集中的孤立程度,降低了阈值设置对用户的要求。(3)对分类、多维数据中的局部异常检测问题进行了研究,基于属性值的频数差异性理论给出了一个新的异常定义,并提出了判断异常显著性的异常存在判别准则,实验结果表明,该判别准则可有效去除大量异常程度并不显著的对象。文章从检测结果的有趣性、与同类算法的比较、对提高分类准确率的贡献和检测小类的能力四个方面对算法的有效性进行了评价。(4)实现了一个异常检测实验平台SOD(Smart Outlier Detection),其中集成了所提出或改进的算法,为异常检测提供了一个分析工具。所提供的数据接口能够从多个外部数据源获取数据,增强了平台的实用性,现已集成到某高校的教务管理系统中进行测试。(5)结合教务管理系统的特点,探讨了在教务管理系统中使用异常检测的必要性,并就实际需求给出了应用实例。 本文的研究目的是构建一个从真实数据中检测异常数据及发现其内涵知识的实验平台,主要包括有机结合的五个方面:消除阈值设置的基于距离和的近似算法;针对分类数据、高维数据的异常检测算法;为有效发现异常中的内涵知识的例外规则挖掘算法;针对子序列长度相等的静态时间序列数据中的异常挖掘算法;集成了以上几种算法的实验平台。异常检测的最终目的在于应用,本文对异常挖掘在教务管理系统中的应用进行了有益的探索。更多还原

【Abstract】 The problem of outlier mining has been variously called outlier analysis, anomaly detection, exception mining, detecting rare events, mining rare classes, deviation detection, etc. Outlier may be “dirty data”, but it also can means meaningful event corresponding to the reality. From the point of knowledge discovery, rare events are often more interesting and valuable than others in many domains, where the rare events’importance is quite high compared to other events, making their detection and analysis extremely important. The main work in the thesis are listed as follows: (1)Summarizing the problem of outlier mining from the realistic meaning, algorithms, application ranges, detection tools, algorithm’s evaluation, etc. (2)To overcome the limitation of requiring threshold in existing distance-based algorithms, this paper proposes a new definition of outlier. This definition use one object’s distance with all others objects in the dataset to judge the object if an outlier or not, so the problem that the algorithm needs setting the near neighbour parameter p or k is solved effectively. To improve the efficiency, a sampling-based approximate detection algorithm has been developed. Experiments have been carried out with real data, the results indicates that not only the newly definition get the same results as DB(p,d)’s, but also points out the outlier’s outlying degree in the dataset, and simplifies the requirement for detecting outliers. (3)This paper researches the problem of local outlier in mutil-dimensional and categorical attribute datasets, and give a new definition of outlier base on the theory of difference in frequently of the attribute value, and propose the anomaly existing criterion for estimating the significance of the detected outlier. The experimental result indicates that the criterion can get rid of a large amount of objects which degree of deviation is not remarkable really. We eveluate the validity of our algorithm from four aspects: interesting of the result, comparison with similar algorithms, contributions to the improving of the accuracy rate of classification and ability of detection rare class. (4)An experimental platform named SOD(Smart Outlier Detection) has been constructed, which integrates the algorithms proposed or improved in this paper, and provides a tool for the analysis of outlier detection. It can obtain data from several outside data sources by the data access interface, this strengthens its practicability. SOD has been integrated with a teaching management system. (5)Combining the characteristic of the teaching management system, this page discusses the necessity for using outlier detection in the administrative system, and provide some instances basing on the actual demand. Our purpose is to construct an experiment platform for Mining Outlier and Finding Intentional Knowledge from real data. Five organized aspects are included in this page: distance-based approximate algorithm which simplify the requirement for setting the threshold, algorithm deal with categorical and high-dimentinal data, effective algorithm for mining exceptional rule and the intensional knowledge, algorithm for mining anomalous patters in static time-series data in which the subsequences’length are equal, and the software platform integrated the above four algorithms. The final purpose of outlier detection lies in its application. this theise make a beneficial discussion and attempt to the application of outlier mining in the teaching management system.更多还原