【作者】 杜明；

【导师】 乐嘉锦；

【作者基本信息】 东华大学 ， 管理科学与工程， 2013， 博士

【摘要】 当前电子商务的发展态势如火如荼,交易规模越来越大,导致了海量的电子商务数据,使得电子商务数据库的读写、更新和查询等性能提升变得越来越困难。当前主要的解决方案都已接近了数据库系统性能优化的极限。而Flash存储技术的发展为解决上述问题带来了新机遇,其超过磁盘100倍的I/O性能可以作为突破数据库性能优化瓶颈的方法。因此本文将从底层存储技术研究入手,兼顾上层软件算法来实现提升电子商务数据库在海量数据环境下的运行性能。研究发现,Flash芯片的顺序读写和随机读取速度都远快于磁盘,但是随机写入和更新性能不足,且芯片价格较高。因此本文提出采用Flash和磁盘混合存储系统,既可以综合利用两者各自的优势,又可以兼顾设备价格成本。但是基于混合存储系统的数据库部署方案目前尚无系统的研究。本文探索了基于混合存储系统的电子商务数据库性能优化,研究成果丰富了现有电子商务数据库的研究内容,对电子商务企业的发展具有重要的现实指导意义。本论文的研究内容主要有：(1)基于混合存储系统的差异存储模型的构建。从适合电子商务企业应用的角度出发,在数据库表和属性两个层面上构建了差异存储模型,并设计了判断表和属性数据读写敏感性和冷热性的阈值算法,实现了表及属性数据的读写敏感性和冷热性分析以及迁移存储方案。(2)基于混合存储系统的差异存储模型的查询性能优化研究。消费者使用电子商务平台时需要进行大量的数据库查询操作,因而确保并提高电子商务数据库的查询性能对于保证电子商务企业运营质量至关重要。本文首先设计研究了HS-Join算法用于提高存储于不同存储介质的、以表为单位的多表连接查询性能；其次设计研究了PHS-Join算法,在Flash设备上提取主键列和连接列建立连接子表并基于子表重构整体连接表。实验表明了两种算法能够明显提高电子商务数据库的查询性能。(3)基于混合存储系统的差异存储模型的事务处理性能优化研究。电子商务数据库的事务处理性能是电子商务企业提高响应能力,维护更多消费者人数,并在同一时间内满足每位消费者个性化需求的重要技术基础。本文以Oracle数据库为例,首先分析了事务处理过程中大量的临时数据和日志数据的读写敏感性及冷热性,确定了将这两类文件迁移存储到Flash设备中。其次根据日志文件增长迅速的特点,提出了Dynamic Logging方法,当Flash空间不足时,日志记录可以实现分散存储。实验表明上述管理方法明显提高了电子商务数据库的事务并发性能。(4)基于混合存储系统的差异存储模型的数据仓库性能优化研究。优化电子商务数据仓库的查询分析性能是电子商务企业更快速、更准确地作出商业决策的技术基础。本文首先提出了采用列存储方案,并提出了列迁移存储算法,提高了决策所需属性数据的查询效率。此外针对数据仓库中多属性关联分析时连接代价大的问题,提出了行列混合存储方法,通过提前固化减少了连接代价。最后提出了数据差异化压缩存储方案,既保障了数据的查询效率,又充分利用了磁盘的存储空间。本论文的创新点和主要贡献有：(1)本文从电子商务企业角度考虑其经济可行和技术可行两个方面,提出了在电子商务数据库中采用混合存储系统,并从表和属性两个层面构建了差异存储模型。既有利于电子商务企业应用实现,又能满足用户对响应速度可感知的要求。(2)本文提出了适合混合存储系统查询的HS-Join算法和PHS-Join算法,充分利用了Flash的随机读取优势,提高了电子商务数据库的查询性能；提出了数据库相关文件在混合存储系统中的部署方案,极大提高了数据库的事务处理性能,提升了电子商务企业的业务处理能力。(3)本文为了实现电子商务数据仓库性能优化采用了行列混合存储技术,提出将高频率的连接列提前固化,提高了分析决策的速度和效率。此外在混合存储系统中采用数据差异化压缩存储方案,充分发挥了设备的存储效率,又确保了数据仓库的查询速度。更多还原

【Abstract】 The current development trend of e-commerce is in full swing. The increasing scale of e-commerce transactions result in a massive e-commerce data, witch makes the performance improvement of e-commerce database to read and write, update and query more and more difficult. The main solutions have been close to the limit of the performance optimization of database systems. However, flash storage technology development to bring new opportunities to solve the above problem. The I/O performance of flash is100times over disks, which is a way to break through the above bottleneck. This thesis intends to start from the underlying storage technology research, taking into account the upper software algorithms to enhance the performance of e-commerce database running in the massive data environment.The study found that the speed of sequential read and write, random read of flash chip is much faster than the disk, but the random write and update performance of flash are lower than disks, and the price of flash chip is much higher. So, this thesis proposes to use a hybrid storage system of flash and disk. Witch comprehensively utilizes of the advantages of both, and also takes into account the cost of equipment prices. But there is no system study on database deployment scheme based on hybrid storage systems. This thesis explores the performance optimization of e-commerce database based on hybrid storage system. The research results enrich the content of existing e-commerce database, and based on the development of e-commerce enterprises run by the e-commerce database. And the research has important practical significance for the development of e-commerce businesses based on the e-commerce database.The main contents of the thesis consist of following four aspects:(1) To build differentiated storage model based on hybrid storage system.From the point of view of e-commerce enterprise applications, the thesis creates differentiated storage model from two levels of data tables and data attributes, designes the threshold algorithm to determine the reading and writing sensitivity and hot or cold characteristic for table and attribute, realizes the analysis the reading and writing sensitivity and hot or cold characteristic for table and attribute and proposes migration storage solutions.(2) The study of query performance optimization of the differentiated storage model on the hybrid storage systems.Using e-commerce platform, consumers need to execute a large number of database query operations. To ensure and improve the e-commerce database query performance is essential to ensure the operational quality of the e-commerce businesses. Firstly, the thesis designed to study the HS-Join algorithm, used to improve the join query performance of the multi-table stored in different storage media which storage unit is table. Secondly, the thesis design to study the PHS-Join algorithm to establish a child connection table on flash and then reconstruct to generate the overall connection table based on the child table. The experiments show that the HS-Join and PHS-Join algorithm can significantly improve database query performance.(3) The study of transaction processing performance optimization of the differentiated storage model on the hybrid storage systems.The performance of e-commerce database transaction processing is an important technical foundation for e-commerce enterprises to improve their ability to respond, to maintain more number of consumers, and to meet the individual needs of each consumer at the same time. Taking Oracle database as an example, the thesis firstly analyzes the reading and writing sensitivity and hot or cold characteristic for the data of temporary table and log file, and migrate these two types of file to the flash device. Secondly, based on the rapid growth of the log file, when the flash lack for space, the thesis proposes Dynamic Logging method, logging dispersed stored in the flash space. The experiments show that the management method can significantly improve the performance of transaction concurrency of the e-commerce database.(4) The study of performance optimization of the data warehouse on the differentiated storage model on the hybrid storage systems.Optimize the query analysis performance of e-commerce data warehouse is the important technical foundation for e-commerce business to make businesses decisions more quickly and accurately. The thesis proposes the column migration storage algorithm on using column storage solution, which improved query efficiency of the decision-making attribute data. In addition, this thesis presents the hybrid storage of row and column method to reduce connection cost by solidifying associated attributes in advance. Finally, this thesis proposes differentiated data compression scheme, witch not only protects the data query efficiency, but also takes effectively use of the disk space.The innovative points and the main contributions of this paper are:(1) Considering from the economic feasibility and technical feasibility of the e-commerce enterprises, the thesis proposed to use hybrid storage system of flash and disk for the e-commerce database and build differentiated storage model from the data table and attribute granularity level, which is not only conducive to achieve the application of e-commerce businesses, but also to meet the response speed requirements by the user.(2) The thesis proposed the HS-Join circulation algorithm and PHS-Join algorithm suitable for query join of large-scale e-commerce database, which takes full advantage of the random read performance of flash, improves the query performance of the system。 Through reasonable deployment of the database file in a hybrid storage system, the study greatly improves the performance of concurrent transaction processing of the database and enhanced the business processing capabilities of the e-commerce business, and makes it possible that e-commerce enterprises quickly and easily applying the flash device and directly using of existed disk-based database management systems.(3) The thesis proposes to use hybrid storage technology of row and column for performance optimization of the e-commerce data warehouse, to solidify associated attributes of high connection frequency in advance, which improves the speed and efficiency of the decision-making analysis. Additionally, utilization differentiated data compression scheme according on hybrid storage model. The compression scheme not only improves the storage efficiency of two types of storage devices, but also to ensure the query speed of data warehouse.更多还原