【作者】 张程；

【导师】 王全民；

【作者基本信息】 北京工业大学 ， 计算机科学与技术， 2016， 硕士

【摘要】 在网络数据量急速增长的现代社会,大数据存储处理技术正在蓬勃发展,其中HDFS(Hadoop Distributed File System)系统是大数据存储处理技术中应用最广泛的。由于HDFS的存储机制,在面对海量的小文件时会出现名字节点的内存瓶颈问题,所以对HDFS的文件存储与读取的优化策略的研究与应用,对解决海量小文件存储和大数据处理有着重要的探索价值和实际意义。海量小文件存储平台是随着信息化建设不断深入而被提出的课题,而现代网络资源存在着数量大、文件体积小等特点,本文的研究成果能够为海量小文件存储平台的建设添砖加瓦。论文从海量小文件的特点和HDFS的存储机制入手,分析了HDFS存储与读取文件的策略,提出了一种基于文件关联关系和基于数据块平衡的PS文件合并算法,利用核心的PS文件合并算法构建了HMM(Hadoop Merging Middleware)中间层,所有用户文件的上传下载都要经过HMM中间层,通过实验测试验证能够提高HDFS处理小文件的性能。论文主要工作有:(1)对当前国内外对于海量小文件处理和海量小文件存储平台的现状进行了调研,研究了HDFS工作机制,对建立海量小文件存储平台所需要的技术进行了深入的学习。(2)提出了一种适合海量小文件存储的PS文件合并算法,能够通过文件关联关系和数据块平衡,将小文件组成大文件存储在HDFS,将文件的合并信息存储到Redis,通过该算法能够用尽量少的数据块来存储数据。依照算法构建了HMM中间层来处理海量小文件,当从HDFS获取数据时,利用缓存来提高读取效率。(3)对海量小文件存储平台的用户进行了研究,总结用户的功能需求,使用开源的Hadoop来部署平台开发环境,针对数据资源文件体积小、数量大、非结构化程度高的特点,融合了My SQL关系型数据库与内存型数据库Redis,构建了web海量小文件存储平台。更多还原

【Abstract】 In the modern society,the quantity of network data grow rapidly, large data storage technology is booming, the HDFS(Hadoop Distributed File System) System is the most widely used in large data storage processing technology. Because of HDFS storage mechanism, in the face of the massive small file namenode memory bottleneck problems would appear, so the optimization of HDFS file storage and reading strategies of research and application, to solve the mass small file storage and the exploration of the large data processing has important value and practical significance. Massive small file storage platform is with the continuous deepening of information construction and subject is proposed, and modern network resources exist characteristics of large quantity and small file size. The results of this study can contribute to the construction of massive small file storage platform.From the characteristics of the mass of small files and HDFS storage mechanism, i analyzes the strategy HDFS store and read the file, a new file-based and block-based relationship balanced PS merging algorithm is proposed, using the PS core document merging algorithm Construction of the HMM(Hadoop Merging Middleware) intermediate layer, all users who upload files must go through the HMM intermediate layer, can be verified by experimental tests to improve the processing performance of small files. The main work are:(1) On the current domestic and foreign to the status quo of the massive small document processing and massive small file storage platform research of HDFS working mechanism, i study the technology required for massive small file storage platform.(2) Proposed a PS file merge algorithm suitable for mass storage of small files, the algorithm can through the file association and data balance block, small file composition a large file storage in HDFS, merging files information storage to redis, able to use less data block to store the data by the algorithm. In accordance with the algorithm to build a HMM intermediate layer to deal with massive small files, when get data from the HDFS, then use of cache to improve the reading efficiency.(3) I summarize the functional requirements of the users of the massive small file storage platform, using open source Hadoop to deploy platform development environment, according to the characteristics of data resource file has the advantages of small volume, large quantity, non structure. I combined the MySQL relational database and the memory database Redis, built the web mass small file storage platform.更多还原