【作者】 张子龙；

【导师】 林伟伟； 邓健爽；

【作者基本信息】 华南理工大学 ， 工程硕士（专业学位）， 2017， 硕士

【摘要】 数据平台在大数据海量计算能力的背后起着重要的技术支撑作用,在过去几年中,数据平台的演变经历了组件化、服务化两个阶段。一方面,组件化使得数据平台从紧耦合的复杂架构中解脱出来,开始向模块化和扁平化方向扩展,大数据组件越来越丰富,也就让数据平台的运维管理和版本控制更加复杂。在过去,平台的运维多半都是重复的体力劳动,随着集群规模的扩大,这种被动管理模式逐渐开始不堪重负。在另一方面,大数据技术的发展也不断推动了各个产业的变革,新的商业模式以及业务需求层出不穷,在日益成熟的云计算领域也开始出现大数据分析服务,但由于数据平台的架构设计与多租户服务模式契合度不高,导致大数据服务价格昂贵且造成资源浪费。本文立足大数据技术发展的背景,针对大数据服务的多租户使用需求,设计并实现了面向大数据的多租户管理平台的主要技术解决方案。文中主要提出了三个技术实现:第一,针对规模不断膨胀的大数据计算工具,设计一种容器化的组件仓库服务,通过将大数据组件封装成Docker容器,并基于容器管理平台和编排工具进行拆分和封装,进而实现数据平台的容器化管理。第二,针对大数据服务的多样化、非结构化等特征,设计了一种基于Raft分布式一致性协议的大数据服务发现技术,为数据平台的弹性伸缩提供主要技术支撑。第三,针对不同负载类型的作业资源需求,设计一种面向多租户使用场景的,基于数据感知的Weighted-DRF资源调度算法,为容器化大数据平台的资源管理和作业调度提供一定的参考意义。最后我们对所实现的关键技术展开一系列的性能测试:首先对容器化的数据平台进行I/O和网络负载测试,测试结果表明容器的性能介于虚拟机和主机之间,能够作为支撑大数据计算平台的基础设施。然后对基于Raft协议的大数据服务发现技术进行功能测试,测试结果表明在系统资源繁忙时,通过服务发现技术可以及时为大数据计算集群补充更多的计算资源,提高计算效率。最后对基于数据感知的Weighted-DRF算法进行性能测试,测试结果表明数据感知能够有效的提升I/O繁忙型作业的计算效率,而该算法也是数据平台面向多租户使用场景的一个有力补充。更多还原

【Abstract】 The data platform plays an important role behind the large-scaled data computing capacity of big data,the evolution of the data platform has gone through two stages during the last few years: Componentization and Big data as a Service.The architecture of big data platform changed from the tightly coupled complex to the modular and flatten,which made the operation and maintenance management of data platform become more difficult.Additionally the maintenance management of the platform are mostly repeated manual labor,with the expansion of the cluster size,this passive management model gradually overwhelmed.On the other hand,the development of big data technology also has contributed to the transformation of various industries.Big data analysis has already been released in the cloud computing services,since the design of data platform architecture failed to fit the multi-tenany application,big data analysis services were expensive and waste of resources.For the needs of multi-tenancy services,this article proposed three technical solutions of multi-tenancy management platform based on the big data technology.In the first part we designed a components containerization service,through which we packaged the components into Docker containers to achieve the containerized management of big data platform.In the second part we implement a service discovery system based on the distributed consistency protocol of Raft,to support the auto-scale architecture of big data platform.In the third part we implement a data-aware based weighted-DRF resource scheduler algorithm for the resource management of Kubernetes platform.Finally we carried out a series of performance tests on the proposed key technologies,the experiment result shows that,the containerization of data platform can bring higher I/O and network load performance than virtualization,and the service discovery technology based on Raft protocol can adjust system resource dynamically for the big data computing cluster without the human intervention,and the data-aware based weighted-DRF algorithm can effectively improve the computing performance of I/O busy job,which can be a powerful complement of kubernetes for multi-tenancy usages.更多还原