【作者】 刘榴；

【导师】 李小勇；

【作者基本信息】 上海交通大学 ， 计算机科学与技术， 2018， 硕士

【摘要】 随着人类社会数据爆炸性增长,分布式存储系统的规模也越来越大,节点数少则几千多则上万,磁盘或者节点出现故障的概率大大提高。因此,容错技术是分布式存储系统中不可或缺的重要研究内容。本文在实验室自主研发的分布式存储系统BOSS基础上,对分布式存储系统的容错关键技术进行研究。本文的主要工作如下:(1)针对数据布局算法进行研究,提出了一种基于层次结构的数据布局算法。该算法通过两次哈希选取数据布局位置,能够较好的兼顾数据可靠性与负载均衡性,并且以较小的代价支持集群的动态拓扑变化。(2)在BOSS系统上,实现了RS纠删码,比多副本技术提供更高的容错能力,并提升存储空间利用率。针对数据修复消耗网络带宽过大的问题,提出了一种基于Prime最小生成树的数据修复策略,可有效降低纠删码数据修复时的网络占用带宽。(3)提出了一种磁盘健康状态检测方法,该方法将磁盘空间均等分割为多个采样区,然后在每个采样区内随机取点,综合考虑IOPS性能和延迟,在保证准确率的同时,能够在较短时间内检测出故障磁盘,为及时识别和替换故障磁盘,保证数据可靠性发挥重要作用。更多还原

【Abstract】 With the explosive growth of human society data,the size of distributed storage system is growing,the number of nodes range from several thousand to hundreds of thousands,the probability of single disk or node failure is greatly increased.Therefore,fault-tolerant technology is an indispensable and important research content in distributed storageIn order to ensure the reliability of distributed storage system,it is necessary to study the high reliability technology in distributed storage system.On the basis of Blue Ocean Storage System,a distributed storage system independently developed by our laboratory,this paper studies the key technologies of the fault tolerance in distributed storage system.The main contents of this paper are as follows:(1)Aiming at the data layout algorithm,we propose a hierarchical layout algorithm.The algorithm selects the data laout loation by two hashes.It provides high data reliability and loading balancing,and support the dynamic changes of cluster scale at a low cost.(2)On the Blue Ocean Storage system,RS erasure code is implemented,it can provide higher fault tolerance than multi-copy technology and improve storage space utilization.And for the problem that data bandwidth is too large in the process of data recovery,we propose a data recovery strategy based on Prime minimum spanning tree,it can reduce erasure code data recovery network bandwidth effectively.(3)This paper proposes a disk health detection method.This method divides the disk space equally into multiple sample areas,and then randomly takes points in each sample area.Considering the performace and delay of IOPS,it can detect failure disks in a short time while ensuring the accuracy.It play an important role in timely identification and replacement of failure disks and ensuring data reliability.更多还原