【摘要】 大数据应用对信息系统的底层存储提出了极大挑战,其首选方案为元数据服务器(metadata server,MDS)集群架构.MDS集群系统为了实现负载均衡,采用的基本机制为元数据服务的负载迁移.当前主流方案存在迁移时间长,迁移中写相关元数据访问需要阻塞的问题.提出细粒度的元数据服务负载迁移方案,由迁出端恢复迁移机制和细粒度控制迁移机制2部分组成.在迁移过程中,以文件为粒度在迁入端重建迁出端的元数据相关状态结构,并根据被访问状态的类型选择在迁出端或迁入端进行立即响应,而非阻塞操作直到迁移完成.在蓝鲸MDS集群系统上实现了此方案,测试和分析表明,在10 000规模的迁移场景中,迁移时间控制在800ms以下,元数据或状态写入延迟最高为215ms.与系统原方案中90s的固有延时相比,此方案有效解决了MDS集群中负载迁移时间过长的问题,并且极大地降低了迁移过程对正常元数据访问的影响.更多还原

【Abstract】 Big data applications pose a big challenge to the storage infrastructure of information systems.The best solution is metadata server(MDS)cluster infrastructure.To implement load balance,MDS cluster systems adopt load migration of metadata service as their fundamental mechanism.A fine-grained solution for load migration of metadata service is proposed which consists of two parts.One is recovery-based migration mechanism utilizing emigrated end information and the other is fine-grained controlled migration mechanism.In main solutions,there are two problems that migration time is too long and write related metadata access should be blocked.In the process of migration,metadata related states of emigration end can be fine-grained reconstructed in immigration end.The accessed state can be decide to response immediately in emigration end or immigration end,instead of blocking metadata operations till the end of migration.The solution is deployed in blue whale MDS cluster system.Results of performance test show that migration time is less than 800 ms with 10 000 states,and the write time of metadata or state is less than 215 ms.Contrast with the original solution which has 90 sinherent delay,the proposed solution can effectively resolve the problem that migration time is too long,and minimize the impact for blocking common metadata access in migration.更多还原