【作者】 楼巍；

【导师】 曹家麟；

【作者基本信息】 上海大学 ， 控制理论与控制工程， 2013， 博士

【摘要】 面对大数据时代的到来，传统的数据处理方式面临着新的严峻挑战，大数据时代的大量化（Volume）、多样化（Variety）、快速化（Velocity）和价值密度低（Value）等特点让传统的搜索方法和工具有时只能望“数据”兴叹。只有面向大数据的技术不断发展，才能将大数据时代带来的挑战变为机遇，更好地运用这个重大战略资源，并有效构建相适应的数学模型和工具，真正将海量数据变化为有效信息。本文以面向大数据的高维数据挖掘技术为研究对象，结合项目的需求，为了实现“从数据的个体研究变为数据系统研究”、“从数据的被动验证变为主动发现”，做了以下几个主要方面的研究和探索：(1)在大数据时代，由于数据的来源较多，使得数据融合尤为重要。本文采用了数据预处理技术，如数据清理、数据集成、数据选择等方法，最大范围地将数据整合在同一个标准下，解决了数据的融合问题。大大提高了数据挖掘的质量，降低实际处理所需要的时间。(2)建立了基于三维矩阵的数学模型。将数据的每个属性定义为空间中的一个维度，构成一个N维空间。将其属性值用向量来表达，再转化以矩阵形式表示，每条记录的信息用一个M×N矩阵表示。这样一系列的矩阵可以将所有的记录表示成一个三维矩阵，此矩阵是后续算法的基础。(3)将仿生学优化算法应用于高维数据的关联规则分析中。对传统遗传算法应用于高维数据关联规则挖掘时出现的早熟收敛和后期收敛速度慢等问题，采用了一种协同进化算法，并引入了一种信息交互机制，使两个种群协同进化，弥补遗传算法的不足。实验证明在可接受的时间复杂度前提下，该协同进化算法是一种能有效避免早熟收敛等现象的全局优化算法，应用于高维数据集时，提取出的关联规则更为有效。(4)引入了超图和系统的概念，探索在三维矩阵模型上建立超图，并针对大数据的特点，结合系统的概念采用了新的超边定义方法，提高了处理问题的能力；在基于无向超图的聚类分析中，论文应用超图分割算法HMETIS进行聚类，实现了高质量的聚类分析；在基于有向超图的关联规则冗余及环路检测中，论文将关联规则转化为有向超图，并重新定义了邻接矩阵，对冗余和环路的检测转化为对于超图中连通块和圈的处理，为关联规则的冗余处理提供了新的思路和方法。(5)将以上研究产生的新方法应用于实际项目的数据处理中，实验结果验证了本论文采用的三维矩阵数学模型和相关的数据挖掘算法能够从高维数据中发现新的高质量的知识。更多还原

【Abstract】 With the upcoming era of Big Data，the traditional way of data processing facesstem challenges. Big Date processing has many new characteristics such as highvolume, variety, velocity and low density of value, which outperforms the traditionalways of searching. The only way to survive is to follow the development of the BigData technologies and turn challenges into opportunities. By better using of Big Data,the important strategic resources, mathematic models and tools can be effectivelyconstructed to change the massive data into real useful information.This paper focuses on the study of high dimensional data mining technologies ofBig Data to meet the requirements of the project. In order to “turn the individual datastudy into data system study” and “turn the passive data verification to active datadiscovery”, the researches and explorations were made in the following respects:(1) In the era of Big Data, the merging of data has becomes extremely importantbecause of too many sources of data. Our researches use the technologies of datapre-processing like data cleaning, integrating, selecting, etc., to maximallyintegrate the data according to one criterion which solves the problems of datamerging andgreatly improve the quality of data mining and minimize the timethat data processing needs.(2) Building up a mathematic model based on the three-dimensional matrix, wecreate an N-dimensional space by defining each attribute of the data as adimension of the space. The attributes are expressed in terms of vectors and thentransformed to matrix. Each information recorded is expressed by a M×N matrix.A series of matrix can express all the records by a three-dimensional matrixwhich is the basis of the following algorithm.(3) Applying the bionic optimization algorithm to the association rules analysis ofhigh dimensional data. The problems like premature convergence and low convergence rate in later period often arise when the traditional genetic algorithmis applied to high dimensional data. We use a coevolution algorithm andintroduce the information exchanging mechanism to make two species evolved ina coordinating way so as to make up the deficiency of the traditional geneticalgorithm. The experiments have proved that the coevolution algorithm caneffectively avoid some problems like premature convergence under theprecondition of the acceptable time complexity. The algorithm is an all-roundoptimization and the relevance rule extracted is more effective when it is appliedto analyzing the collection of high dimensional data.(4) Introducing the concepts of hypergraph and system and exploring to construct thehypergraph on the model of three-dimensional matrix. According to thecharacteristics of Big Data, the new hyperedge definition method is adoptedcombining the concept of system, thus improving the processing capacity. In theassembled analysis based on undirected hypergraph, we use HMETIS,hypergraph partitioning algorithm, to cluster and practice the clustering analysisin high quality. In the detection of the relevance rule redundancy and loop circuitbased on directed hypergraph, we turn the relevance rule into directed hypergraphand redefine the adjoining matrix, which means turning the detection ofredundancy and loop circuit into the handling of connection block and ring in thehypergraph. It provides a new mentality and method for the redundancy disposalof relevance rule.(5) Applying the above new method to the practical data processing. The results ofthe experiments testify that high quality knowledge can be discovered by themathematic model of three-dimensional matrix and relevant data miningalgorithm we adopt.更多还原