【作者】 高伟；

【导师】 鲍玉斌；

【作者基本信息】 东北大学 ， 计算机应用技术， 2010， 硕士

【摘要】 信息化技术的不断深入应用,特别是网络技术的迅速发展,导致了数据的规模急剧增长。因此,如何保证海量数据的有效存储和管理,提高针对海量数据的处理效率成为人们日益关注的焦点。聚集运算是数据预处理中最典型操作之一,对提高查询效率有着非常重要的意义,但是聚集运算非常耗费计算资源,特别是针对海量级别数据的聚集运算,更是需要巨大的计算能力和存储能力,而普通的PC机难以提供这样的计算资源,所以研究适合海量数据的聚集运算具有重要意义。本文通过详细研究了Google的分布式文件系统(GFS)以及MapReduce并行计算框架,充分利用该分布式文件系统的高扩展型、高容错性等特性,结合MapReduce在处理大规模数据集时的并行性处理特性,提出了一组面向海量数据的数据聚集运算算法,该组算法主要包括基于MapReduce的关系型数据的选择、投影以及等值连接等算法,并在此基础之上,实现了基于MapReduce的计数(Count)、求和(Sum)、均值(Average)、最大值(Max)和最小值(Min)等聚集运算,形成了比较完整的面向海量数据的聚集运算算法。该套算法充分利用了集群系统的计算能力和存储能力,以及集群系统的网络带宽,极大的提高了海量数据的聚集运算效率,有效地减少了的运算时间,提高了基于聚集运算结果上的数据查询效率本文还在以上聚集运算算法的基础上,实现了基于MapReduce的全局封闭数据立方体生成算法,以及在全局封闭数据立方体上的查询算法。实验表明该算法充分发挥了集群系统的并行处理能力,可以高效的生成全局封闭数据立方体以及快速的完成在其上的查询任务。更多还原

【Abstract】 The development of information technology, especially the rapid development of network technology, led to the rapid growth of data amount. Therefore, modern rearch has centered on how to ensure the effective storage and management for massive data to improve the computing efficiency of massive data. Aggregation computing is one of the most typical data pre-processing methods, which can be great significance improving query efficiency on massive data. However, aggregation of massive data requires enormous computing power and storage capacity. The general PC machine can not provide such huge computing resources. Therefore, research on aggregation computing of massive data is very important.On the base of the distributed Google File System (GFS), parallel computing framework (MapReduce), we study the scalability and fault-tolerance ability of GFS and thes parallelity and highly scalable computing power of MapReduce in large data sets, this paper proposes creative MapReduce-based aggregation algorithms for massive data, which includes the selection, projection and the equivalent joint on relational data. Based on this we realize MapReduce-based Counting, Summing, Averaging, Maxing. and Mining and so on. The algorithms, which make use of cluster computing power, storage capacity, and network bandwidth, improves the aggregation efficiency of massive data and reduce the processing time, and improve query efficiency greatly.This thesis also proposes global closed data cube generation algorithm based on MapReduce. Experiment results show that with the exploring cluster computing resources, the algorithms can generate the closed data cubeand reduce the query time on global closed data cube quickly and efficiently.更多还原