【作者】 张光辉；

【导师】 丁祥武；

【作者基本信息】 东华大学 ， 计算机软件与理论， 2014， 硕士

【摘要】 随着网络的普及,信息时代的到来,人们日常生活所面临的数据已经非常巨大,如何围绕这些数据建立数据仓库、进行数据挖掘和数据分析正逐步成为数据处理的主题。如何快速准确分析这些海量数据,对数据仓库系统有了更高的要求。传统的基于行存储的关系型数据库系统已经不能很好地适应海量数据分析的要求,而基于列的存储模型能够较好地支持海量数据分析。近年来,微处理器的发展非常迅速。由于处理器功耗与设计的限制,处理器的发展趋势正逐步从单核心高主频到多核心处理器发展。现在单核心处理器几乎从处理器市场中消失,而片上多核处理器(chip multipro-cessor,CMP)已经成为该市场上的主流。多核处理器能够为列存储并行查询执行提供硬件环境。本文主要研究多核处理器环境下列存储系统的并行查询设计与实现技术。论文以我们实验室研制的列存储系统DWMS为依托,研究现有的查询技术,设计并实现了一套基于多核处理器的并行查询模块。1、分析一条查询语句在处理过程中的各个阶段的基础上,并为多个查询阶段的查询优化做出并行化分析。2、设计了基于OpenMP技术动态分配的多线程快速排序算法。3、设计了多个多线程hash连接算法,可根据具体的情况选择合适的连接算法。4、分析多查询基础上设计了一个简单的多查询并行。5、对基于传递块的查询执行模型进行分析后,建立了传递块缓冲区。以传递块缓冲区为数据流转中心设计了基于流水线的查询并行化方案。这种设计主要将父子节点间传送数据的方式由传递块改为传递块缓冲区,这样操作节点只对缓冲区读写数据,父子操作节点之间实现了分离。通过对缓冲区的有效管理,能够很好地提高DWMS的查询性能。6、论文最后对整个查询的并行设计做出分析,通过对相关参数、缓冲区数量和并行模块的进行设置,进一步提高查询性能。在多核环境下,论文对DWMS数据仓库原型系统进行了多线程设计,主要是对操作节点的并行化设计和流水线并行化设计。通过理论分析和相关实验的验证,表明我们的查询并行化设计可以有效地提高DWMS的查询效率。更多还原

【Abstract】 With the popularity of the network and the arrival of the information age, people’s daily life has been facing enormous data. How to establish a data warehouse system around these data, and then to do data mining and date analysis is becoming a hotspot of data processing. These data has a higher requirement for the speed of query. So the traditional row-store system cannot adapt to the requirements of modern mass data any more. However, the column-store system can provide the underlying storage model for massive data processing.In recent years, the microprocessor has developed rapidly. Due to the limitation of processor power consumption and design, the development trend of the CPU is gradually moving from high-frequency single-core to multi-core processors. Single-core processor almost disappeared in modern processor market, while CMP (on-chip multi-core processors) has become the mainstream in the market. Multi-core processors can provide hardware environment available for the parallel query processing.The main contents of this paper are the design and implementation of parallel query column-store system. In column-stores D WMS developed by our laboratory, we analyzed the existing query technology, then designed and achieved a set of parallel query module. First, we choose the various stages of a query processing to make analysis for parallel query optimization at each stage. For example, in the hash-join stage, multiple joins can hash operations simultaneously. After we analyzed query execution mode based on pass block, established the pass block buffer for Pipelined Parallel Processing. The way of transmitted data changes from pass block to pass block buffer, so that each node only requests data from the buffer. This achieves a separation between father and son nodes. Through the effective management of the buffer, we can improve DWMS query performance. Finally, we make an analysis of the parallel design of the entire query. In order to Improve query efficiency further, we will set the relevant parameters, the number of buffers and parallel modules.In our parallel multi-core environment, we have a multithreading design for the DWMS data warehouse system, and this design mainly includes parallelization and Pipelined of nodes. Through the theoretical analysis and relative experiments, the design of query parallelization can improve the query effectiveness of DWMS.更多还原