【作者】 朱锦涛；

【导师】 唐卓； 吴炜；

【作者基本信息】 湖南大学 ， 计算机技术（专业学位）， 2020， 硕士

【摘要】 近年来,由于生活水平的提高和互联网的发展,人们使用互联网的时间越来越多,频率越来越高,再加上物联网的发展,政府管理的需要等原因,产生了大量的数据。大量数据的出现对大规模数据的存储技术需求日益增大,技术要求日益提高。由此,出现了许多的分布式存储系统,而其中,Ceph文件系统由于其可扩展性和可靠性赢得不少青睐。Ceph文件系统从本地文件系统出发,它保留了本地文件系统中元数据的角色,并以目录树的形式对用户的文件进行管理,让用户很自然地从本地文件系统过渡到分布式文件系统。Ceph文件系统分为两大部分,一个是存储元数据的服务器端,为元数据服务器,另一个是供应用程序调用的客户端。现有的客户端将部分使用频繁的元数据缓存在客户端的缓存空间中,并使用最近最久未使用算法进行替换。Ceph文件系统客户端根据元数据的使用时间进行管理,并未考虑元数据内部的信息以及元数据在文件系统中的空间位置。根据上述提出的问题,本文在深入研究客户端对元数据缓存机制的基础上,做了如下工作:(1)对现有客户端中缓存的元数据进行聚类分析,以找出两个聚类中心。这两个聚类中心所包含的是冷和热元数据的时间特征。具体来说,本文通过设置元数据统计装置,并利用文件系统测试工具来统计某段时间内Dentry元数据的使用次数,同时获取的还有Dentry和Inode中包含的信息,将这些信息组成五元组,该五元组记录的是文件在当前时间段内各种属性。在收集到这些五元组后,利用聚类算法对这些五元组进行聚类,而聚类的结果为两个聚类中心,其中,一个聚类中心的使用次数比另外一个聚类中心高。而使用次数较高的聚类中心中的其他信息将会在后面的预取工作中指导Ceph文件系统客户端预取更合适的“热”数据。(2)在获得文件元数据的聚类中心的基础上,本文将根据两种情况进行缓存预取,这两种情况会激发预取模块工作。第一种情况是:在某次使用文件时,其文件的元数据离某一分类的聚类中心特别接近,本文认为该元数据应当被视为使用频繁的数据,利用该元数据指导客户端请求元数据服务器将其他相关元数据提前存入客户端缓存中。第二种情况是,如果元数据使用次数已经超过一定界限,同样,客户端将请求元数据服务器将其他元数据提前存入客户端缓存中。通过这两种措施来达到提高客户端缓存命中率的目的。本文在研究了Ceph客户端缓存机制的基础上,提出了缓存预取的方法,该方法能帮助客户端缓存系统对使用频繁的元数据做出更全面的预测,从而提高了缓存命中率,并通过实验证明了预取机制的有效性。更多还原

【Abstract】 In recent years,due to the improvement of living standards and the development of the Internet,people have been spending more and more time on the Internet and using it more frequently.In addition to,the development of the Internet of Things,the need for government management,and so on,which results in a large amount of data.With the emergence of large amounts of data,the demand for large-scale data storage technology is increasing,and the technical requirements are increasing.As a result,many distributed storage systems have emerged,and among them,the Ceph distributed file system has won many favors because of its scalability and reliability.The Ceph distributed file system inherits from the local file system.It retains the role of metadata in the local file system and manages users’ files in the form of a directory tree,It also allows users to adapt to distributed files system from local file system naturally.The Ceph file system is divided into two parts,one is the metadata server that stores the metadata,and the other is the client that is used by the application.Existing clients cache some frequently used metadata in the client’s cache space and use the most recently unused algorithm for replacement.The cache space in client of Ceph file system is managed according to the usage time of the metadata,without considering the internal information of the metadata and the spatial location of the metadata in the file system.Based on the abovementioned questions,this article has done the following work based on the in-depth study of the client’s metadata caching mechanism:(1)Performing cluster analysis on the metadata cached in the client helps to find two cluster centers.The two clustering centers contain the temporal characteristics of cold and hot metadata.Specifically,this article sets up a metadata statistics model and uses the file system testing tools in order to count the number of times Dentry metadata is used in a certain period of time,which also obtains other information contained in Dentry and Inode at the same time,which is composed of a quintuple,the quintuple records various attributes of the file in the current time period.After collecting these quintuples,they are clustered using a clustering algorithm to find two cluster centers,that is,the number of uses in one cluster center is higher than that in another cluster center.The other information in the cluster center with higher usage will guide the Ceph file system client to prefetch more suitable "hot" data in prefetching.(2)On the basis of the cluster center of file metadata,This article will perform cache prefetching according to two situations,which will stimulate the prefetching module to work.The first case is that when a file is used at a certain time,the metadata of the file is particularly close to the cluster center with higher usage.Because this metadata should be regarded as frequently used data,and should be used to instruct the client to request the metadata server to store other relevant metadata in the client cache in advance.The second case is that if the number of uses of one metadata has exceeded a certain limit,then client also need request the metadata server to fetch some other metadata and store it in the client cache similarly.Through these two measures to achieve the purpose of improving the client cache hit rate.Based on the study of the Ceph client caching mechanism,this paper proposes a method of cache prefetching,which can help the client caching system to make more comprehensive prediction of frequently used metadata,thereby improving cache hits.The experiments has proved the effectiveness of the prefetching mechanism.更多还原