【摘要】 Hadoop底层基础存储框架Hadoop分布式文件系统HDFS(Hadoop Distributed File System)是采用基于Master/Slave主从构架设计的分布式文件系统。Name Node负责管理所有文件的元数据和处理客户端对文件的访问。随着应用数据的不断增加,Data Node的不断加入,Name Node维护文件元数据、处理Hadoop应用的读取和删除等文件操作的工作量就会随之增加。Name Node的性能成为整个HDFS的瓶颈。提出一种思想,即将Name Node部分任务交与HDFS客户端完成,使每个HDFS客户端在某些功能上成为Name Node。用LRFU(Least Recently Used and Least Frequently Used)缓存替换策略提出一种在HDFS客户端建立本地缓存的解决方案,从而降低Name Node负载。通过实验证明Hadoop应用可以从缓存中读取文件块信息,而不必请求Name Node,提高了数据访问效率,降低了Name Node负载。更多还原

【Abstract】 Hadoop distributed file system( HDFS) is a distributed file system based on Master/Slave master-slave architecture. Name Node manages the metadata of all files and handles client-side access to files. As application data continues to increase,as Data Nodes continue to be added,the workload of Name Node maintaining file metadata and handling file operations such as read and delete of Hadoop applications increases. The performance of Name Node becomes the bottleneck of the entire HDFS. In this paper,we proposed an idea that some of the Name Node tasks should be submitted to the HDFS client so that each HDFS client became a Name Node on some functions. Based on the least recently used and least frequently used( LRFU) cache replacement strategy,a solution to establish a local cache on the HDFS client was proposed to reduce the Name Node load. Experiments show that Hadoop applications read file block information from the cache without having to request Name Node,which improves data access efficiency and reduces Name Node load.更多还原