【作者】 张超；

【导师】 万继光；

【作者基本信息】 华中科技大学 ， 计算机技术， 2020， 硕士

【摘要】 键值数据库由于其优秀的性能被广泛用于各种分布式存储系统中,而实现键值数据库的负载均衡性,有利于提高系统的稳定和效率。在现有的商业数据库中,数据的访问负载呈现周期性变化的规律,而传统的负载均衡策略在这种场景下具有调度的滞后性。随着深度学习技术的发展,数据预测技术越来越成熟,如果能准确地对未来一段时间内的负载做出预测,并结合负载均衡技术,将热点数据提前做出均衡处理,将有效地提高集群的整体性能。在此思想的基础上,设计并实现一种基于智能预测的负载均衡技术。该技术分为两个模块:预测模块和调度模块。预测模块以集群历史数据为基础训练模型,从而预测集群未来一段时间的负载曲线;调度模块每次在做均衡调度时,从预测模块中获取集群未来一段时间内的负载预测曲线,设定高负载阈值,通过阈值计算下个周期内的低负载时刻区间,再设计一种具有热度衰减特征的热度统计算法,以计算出集群热度最高的Top-K个Region(调度的最小单位)作为调度的热点Region。在计算结果的基础上生成调度计划,调度计划以低负载时刻区间作为调度时机,热点Region作为调度内容。改进后的负载均衡技术能够对集群中的热点数据提前作出均衡处理,且每次调度都在低负载时刻进行,最大程度地降低调度本身对集群性能的影响。基于开源的TiKV代码,采用LSTM(Long Short-Term Memory)模型,实现了系统原型,并在负载周期性变化的场景下做了充分的测试。测试表明,在预测方面,负载曲线的真实值和预测值的拟合度在93%～97%之间;在调度方面,基于智能预测负载均衡调度下的集群平均QPS峰值是改进前的1.24倍,平均查询延迟降低158～288毫秒。更多还原

【Abstract】 The key-value database is widely used in various distributed storage systems due to its excellent performance,and the load balancing of the database is conducive to improving the system stability and efficiency.In existing commercial databases,data access loads show periodic changes,and traditional load balancing strategies have scheduling lag in this scenario.With the development of deep learning,data prediction technology is becoming more and more mature.If the future load could be accurately predicted for a certain period of time,the hot data would be balanced in advance,which will effectively improve the performance of the database cluster.Based on this idea,this paper designs and implements a load balancing algorithm with the intelligent prediction.The algorithm is divided into two modules: the prediction module and the scheduling module.The prediction module trains the deep learning model by the historical data of the cluster to predict the future load curve of the cluster.Once the scheduling module is triggered to balance the hot data,the newly predicted load information of the cluster will be obtained to join the scheduling from the prediction module.With the given high-load threshold,the future low-load time will be calculated.A heat statistics algorithm is also designed with the heat attenuation characteristics to calculate the Top-K hot-data region.Given these information,a scheduling plan is generated.The scheduling plan is mainly executed during the low-load time interval to minimize the impact on performance,and the hot-data regions will be moved to balance the load in advance.The system prototype is implemented with the open source TiKV code and the LSTM model,and there are plenty of tests.The results show that the degree of fitting of the readworld load curve and the predicted load curve is between 93% and 97%;the average QPS peak value of the cluster using intelligent predictive load balancing scheduling is 1.24 times of the na(?)ve method,and the average query delay is reduced by 158 to 288 ms.更多还原