【作者】 刘学军；

【导师】 徐宏炳； 董逸生；

【作者基本信息】 东南大学 ， 计算机应用技术， 2006， 博士

【摘要】 数据流模型的出现对传统的数据管理技术提出了巨大的挑战,由于数据的流动性和无限性等特点,已有的数据库技术无法对数据流数据进行有效的管理,因此,必须进行数据流管理新技术的研究。数据流管理技术已经引起了数据库界的广泛关注,成为当前的一个研究热点。研究数据流相关技术不仅有重要的学术价值,而且在传感器网络、气象监测与分析、移动物体位置跟踪、股票分析、邮件过滤、网络监控与安全等领域有着巨大的应用前景。本文对数据流管理系统和数据流挖掘中的若干关键问题进行了深入探索,主要有以下内容:(1)数据流管理系统的体系结构:面向高速数据流,提出了一个基于硬件预处理的数据流管理系统体系结构。目前已有的原型系统都是从查询优化、系统调度等方面来提高数据的处理速度,在高速数据流环境下都存在明显的不足,因此,本文从一个全新的角度构建新一代数据流管理系统,在体系结构上采用软硬件协同的思想和前端硬件预处理技术,实现数据的高速处理。(2)高速数据流聚集查询:目前已有的聚集算法绝大多数是采用近似技术,以牺牲精度来换取速度的提高。随着硬件技术的快速发展和硬件成本的迅速下降,软硬件协同技术逐渐引起了人们的关注。本文提出了一种软硬件协同的高速数据流聚集查询方法,发挥了硬件在处理速度上的优势和软件在灵活性方面的长处,也研究了提高查询资源共享度的方法。(3)分布式数据流增量聚集查询:分布式处理是数据流管理系统发展的必然趋势。而在分布式系统中,传输量往往是系统的主要瓶颈,因此,本文研究并提出了一种分布式数据流增量聚集方法,可以显著地降低系统的通信量。(4)数据流频繁闭合模式:频繁闭合模式能够唯一地决定所有的频繁模式及其准确的支持度,并且往往数量比频繁模式小几个数量级,在实际中更容易理解和应用。本文研究了动态数据流环境下的频繁闭合模式挖掘,目前还很少有这方面的研究报道。滑动窗口和界标窗口是数据流环境下两种最重要的窗口类型,本文分别研究并提出了基于滑动窗口和基于界标窗口的数据流频繁闭合模式挖掘新算法,算法具有较好的适应性和可扩展性,用户可以根据需要,通过调整允许误差在执行效率和结果精度方面取得平衡。(5)数据流变化检测:在数据流环境下,模式的改变往往比正常模式提供更多有价值的信息,因此,数据流变化检测是数据流挖掘的核心问题之一。本文运用信息熵理论,从频繁项集角度出发,提出了一种基于最大频繁项集信息熵的数据流变化检测方法,不仅可以反映关联规则挖掘中频繁模式的变化,而且也可以有效地反映数据集的改变。更多还原

【Abstract】 The appearance of data stream models is bringing great challenges to traditional data management technology. Because of features of fluidity and infinity, the existing database management techniques can hardly be applied to process data streams effectively. Therefore, it’s necessary to study the new data stream management techniques which have arrested a lot of database researchers’attention. Data stream processing is becoming a hotspot research issue. To study data streams has not only important scientific value but also huge application outlooks, such as sensor networks, weather monitoring and analysis, mobile object tracking, stock analysis, mail filtering, network monitoring and security, and so on. In this paper, we explore a few principal problems over data streams management system and data mining, in depth the main contents are as following:(1) System Architecture of Data Stream Management: facing to the data streams with high rate, a new data stream management system architecture based on hardware preprocessing is proposed. In existing data stream prototypes system, query optimization and system scheduling are applied to improve data processing rate, but the insufficiency is existed with those methods while in the environment of high rate data streams. So, in this paper, the data stream management system of a new generation is constructed with a completely new angle, and some novel ideas of hardware/software co-design and hardware preprocessing techniques are adopted, thus, the data processing with high rate can be realized.(2) Aggregation query over high rate data streams: basically, most existing aggregation algorithms adopt the approximation technology to obtain the advancement of rate by sacrificing the accuracy. With the rapid development and depreciation of hardware, technology of hardware-software co-design has recently gained the increasing attention. In this paper, we propose a kind of novel aggregate query algorithm based on hardware-software co-design, which incorporate with the advantage of hardware in processing rate and advantage of software in agility.(3) Incremental aggregation query over distributed data streams: Distributed processing will be an inevitable trend of the development of data stream management system, . But in distributed system, traffic is usually the bottleneck resource, therefore, we study and propose a new method of incremental aggregation for distributed data traffic, by this method, the network traffic can be reduced prominently.(4) Frequent closed patterns over data streams: The set of frequent closed patterns determines exactly the complete set of all frequent patterns and is usually a few levels smaller than the frequent patterns. Moreover, frequent closed patterns are much easier to be understood and to be applied. But how to mine frequent closed patterns over data streams is a very big challenge and few study reports on it can be found currently. Sliding window and landmark window are two most important windows in data streams. In this paper, two novel algorithms which for mining of data stream frequent pattern and which are based on sliding and landmark windows are suggested and studied respectively. Such algorithms have good adaptability and extensibility. furthermore, according the need of users, the balance between precision and efficiency can be obtained through adjusting permissible error.(5) Detection of change over data streams: The changes of data models often provide more information than normal data models, consequently, the detection over data changes is one of the most important core issues of data streams mining. In this paper, we present a novel method for the detection and estimation of change based on maximum frequent itemsets information entropy, this method can reflect effectively not only the changes of the data stream models of mining associate rules but also the changes of data sets.更多还原