【作者】 李雷；

【导师】 高宏；

【作者基本信息】 哈尔滨工业大学 ， 计算机技术， 2014， 硕士

【摘要】 在这个大数据时代，由于数据量的增大使许多原本很简单的数据操作都变得很难。如何去分析处理海量的数据成为了一个很有挑战性的难题。为了解决这个难题，人们从数据存储、数据传输、数据分析上提出了很多新的方法和模型。当下最热门的大数据存储与处理模型是Hadoop和MapReduce，他们被很多产业界的公司和学术界的学者所研究和认同。作为大数据处理的一种解决方案，MapReduce确实能够解决一些问题，但在很多情况下它依旧是不适用的，所以新的解决方案的研究还需要继续。我们主要是基于CMD存储模型来做大数据存储与查询的探索。传统CMD存储模型是基于单机多磁盘的，它已无法适应当前数据库领域所面临的挑战。我们第一次将CMD扩展到了分布式环境下，在集群上建立CMD存储方案，并通过它解决了大数据量的多路不等值连接操作问题，设计了全新的图数据存储模型，并对CMD在高维数据与大规模分布式集群上遇到的难题做了一定的优化。对于普通的关系数据，我们并设计了基于CMD的多路不等值连接操作这一新的分布式连接操作算法,并与传统关系型数据库和Hadoop分布式并行计算环境进行了效率对比。因为该算法充分利用了CMD存储方法对各属性的索引，所以在效率上要比关系型数据库和Hadoop分布式环境高很多，是解决大数据量的多路不等值连接操作的有效方案。对于图数据，我们也将其转化成CMD能够存储的数据格式，并对其可行性与效率做了一定探索，给出了基于CMD存储方案的图数据存储方法与基本操作。这是一个全新的图数据存储模型，相比于之前的图数据存储模型，它减弱了点的重要性而更关注于边的属性，使基于边的查询的速度得以大大提升。对于CMD存储方法在高维数据与大规模集群上会遇到的一些难题，我们给出了一些改进方案。我们提出的属性分组方案，使属性有层次的进行划分，解决了原始CMD在面对高维数据会产生大量碎片的问题。我们提出的节点分组方案，也解决了当CMD部署在大规模集群上会使数据划分过于零散而网络通讯代价激增的问题。更多还原

【Abstract】 In the era of the Big Data, many simple database operations become impracti-cal due to the growing size of the data. How to handle and analyze this massive ofdata becomes a challenging problem. To solve this big headache, researchers areproposing many novel methods and models on data storage, data transportation anddata analysis. Among all these new tools, Hadoop and MapReduce are the mostpopular big data storage and analyzing tools, which receive the approval from manyindustrial companies and academic researchers. Although MapReduce can solvesome of the problems, it still unfit to many scenario, so new methods are still inneed.We explore the storage method and the query method on big data mainly de-pends on CMD storage method. The traditional CMD storage method, which isbased on stand-alone multi-disk environment, is no longer fitful to the challenge thedatabase society is facing to. For the first time, we extend the CMD to a distributed,parallel environment, propose a CMD storage method on cluster and use it to solvethe multi-way theta-join on large data volume problem, design a brand new graphdata storage model and adapt the CMD to be able to store the high dimension dataand deployed on large cluster.For ordinary relation data, we propose a new multi-way theta-join algorithmbased on CMD storage method and compare the efficiency of it with traditional re-lation database and Hadoop distributed computation environment. Since this algo-rithm makes the best use of the index which is born with CMD, it is much faster thatthose algorithms on traditional relation database and Hadoop, and it could serve asan efficient solution of multi-way theta-join query on big data.For graph data, we adjust it to fit in the CMD storage method, explore the ef-fectiveness and efficiency and finally propose an graph model on CMD and presentsome basic operations. This is a brand new graph data model, which focuses moreon the edges than the vertices when compared to the previous graph data models. Itcan boost the efficiency of queries which mainly deal with edges. For the headaches that CMD used to be facing when confront with high dimen-sional data and large cluster, we propose some improvementof the classical CMDstorage. The attribute group notion we proposed can divide the attributes in groups,which can solve the large quantity of fragments when CMD is trying to store a highdimensional data. The cluster group notion we proposed can solve the scattered datafragments and the crowded network communication when CMD is deployed onlarge cluster.更多还原