【作者】 刘钢；

【导师】 周敬利；

【作者基本信息】 华中科技大学 ， 计算机系统结构， 2007， 博士

【摘要】 随着分布式系统与网络技术的不断发展,分布式存储系统在存储容量、性能、可扩展性等方面已取得了飞速进步,但同时在容错性方面也面临着巨大的挑战:现代的分布式存储系统往往由成百上千个存储节点组成,多个节点出现故障的可能性大大增加,因节点故障导致数据丢失会造成灾难性的后果。因此,如何设计高效、可靠的容错机制,已经是分布式存储系统中迫切需要解决的问题。纠错码是通信领域中广泛采用的检测和纠正通信误差、提高信息传输可靠性的技术。而组通信服务是分布式系统中构建自治区域的重要方法,可保证系统在局部故障的情况下仍能稳定正常地运行。目前,组通信服务是分布式系统中实现容错的一项关键技术。基于纠错码的容错机制将纠错码技术与组通信服务相结合,提出了基于纠错码的容错组,解决分布式存储系统中的容错问题。容错组是实现纠错码的存储节点集合,数据经过纠错码编码后保存在容错组相应的存储节点中。容错组中的存储节点利用组通信服务组织在一起,并通过协调合作提供高可用与高可靠的数据服务。出现节点故障时,可以利用容错组中其它节点保存的数据恢复出原始数据。容错组利用组视图记录当前组中成员列表,组中存储节点加入、退出、故障都将引起组视图的更新。为了方便数据的读/写与管理,容错组中引入了基于对象存储技术。系统运行时,容错组中的存储节点与元数据服务器、管理服务器进行消息通信,更新元数据服务器与管理服务器保存的容错组信息。采取纠错码作为冗余策略的容错组需要减少计算校验数据带来的计算开销,并保证容错组中各存储节点数据一致。设计了容错组中基于组视图的I/O操作,包括读操作、写操作与更新操作。容错组中每一个存储节点都可以完成存储控制器的功能,客户端将I/O请求发送给任意存储节点,存储节点作为协调者再访问组中节点,完成请求。容错组的纠错码都采用系统码形式,编码后的数据块分为信息数据与校验数据。读操作中协调者可以直接访问保存信息数据的节点,只是节点故障时才需要译码恢复数据。数据第一次写入容错组需要编码,然后将编码后的数据存储到相应的节点中。根据纠错码理论,更新操作并不需要再次对数据进行编码,而是先读取已保存的信息数据,与更新的数据计算异或,再将信息数据与校验数据分别更新。为了保证更新数据的一致性,更新操作还需应用时间戳排序、两阶段提交、日志等技术。为了保证容错组的稳定性与容错组中I/O操作的正确性,容错组需要及时检测出故障节点。设计了容错组中基于服务质量的QB-AFD故障检测器。为了探测出组中的故障节点,容错组的每个成员都要向其它成员发送心跳消息,同时监听来自其它成员的心跳消息。如果存储节点在超时时刻还没有接收到某节点的心跳消息,即开始怀疑该节点。故障检测需要满足速度与精度的要求,因此,QB-AFD故障检测器动态调整预期心跳消息到达时间与安全时间余量,以满足服务质量的评价指标。利用组成员关系服务完成容错组的重构任务,设计了基于两阶段提交的视图更新算法2pc-VUA保证组中存储节点维护一致的全局组视图,并设计了节点加入算法负责在新加入容错组的存储节点中生成与其它成员一致的组视图。更多还原

【Abstract】 With the development of distributed systems and network technology, the storage capacity, performance, and scalability of distributed storage systems have increased rapidly. However, distributed storage systems are confronted with great challenges in fault tolerance. As storage systems scale up, they consist of thousands of storage nodes. Multiple failures will occur frequently, which would induce a disaster because of data loss. How to design a efficient and reliable fault tolerance mechanism has become a issue that needs to be resolved urgently in distributed storage systems.Fault tolerance mechanism based on erasure code integrates erasure code and group communication services, which aims to resolve the fault tolerance in distributed storage systems. By using group communication services, storage nodes are organized in a fault tolerance group. Through negotiating and cooperating, storage nodes in a group can provide high available and reliable data services. After encoding, data are stored in corresponding storage nodes. When a storage node fails, original data can be recovered by decoding the data stored in other nodes. To keep fault tolerance group stabilizing, fault nodes should be detected in time and the fault tolerance group need to reconfigure itself.Information of fault tolerance groups are kept in metadata server, and user can fetch relative fault tolerance group when sending request to metadata server. While distributed storage system is running, storage nodes in a fault tolerance group communicates with metadata server and manage server, and update the information of fault tolerance group in them. By updating these information, metadata server can ensure that I/O requests are delivered to correct store nodes, and manage server can ensure the validity of history record in a fault tolerating group.The major contributions of fault tolerance mechanism based on erasure code are as follows:Present fault tolerance groups in distributed storage systems that merge erasure code and group communication services. A fault tolerance group is a set of storage nodes that implement erasure code, and the Identifier of a node correspond to a serial number of the data after encoding. Data are chosen storage nodes to store according to the serial number. A fault tolerance group uses a group view to record members of the group, and the events that a storage node joins, leaves and fails will cause update of the group view. To make read/write and management more convenience, object-based storage and storage virtualization are introduced in fault tolerance groups. Because of the effect of cache, the performance of a storage node is better than that of a disk.Propose I/O operation based on erasure code in a fault tolerance group, include read, write, and update. Unlike the traditional method where a client accesses a storage controller, a storage node picked randomly acts as a coordinator. A client sends its read or write request to the coordinator, and the coordinator forwards the request to members in the group. Responses are returned through the same coordinator to the client. The I/O operation based on erasure code needs to avoid computation of encoding and decoding, and keep data in every storage node consistent. So the erasure codes in fault tolerance groups are systematic code, original data will be contained in data after encoding. In a read operation, coordinator directly accesses data, and decoding is only needed when node fails and data loss. If data are stored into fault tolerance group in the first time, encoding process are absolutely necessary. However, when updating data, delta that is exclusive-or of the last data and newest data can be calculated and then sends to the storage nodes storing related checksum data. To keep data consistent, timestamp, two-phase protocol and log are applied in update operation.Propose a QB-AFD failure detector in fault tolerance based on QoS. To detect failure nodes in a group, each node sends heartbeat messages to each other member and listens others heartbeat messages at the same time. A failure detector may suspect a storage node when it can’t receive heartbeat messages from that node at a timeout point. A failure detector should also satisfy the requirement of speed and accuracy, so the QB-AFD failure detector dynamically adapts the heartbeat period and timeout delay to fulfill the metrics of QoS.Present reconfiguration of fault tolerance group through group membership services. When nodes join, leave and fail, a view update algorithm based on two-phase commit maintains a consistent global view in every storage node. And an adding algorithm generates a view when a node joins a group.更多还原