【作者】 刘远；

【导师】 章勤；

【作者基本信息】 华中科技大学 ， 计算机应用技术， 2015， 硕士

【摘要】 近年来,越来越多的数据被实时流处理。流处理应用在处理数据的过程中需要保持尽可能低的处理延迟,才能充分利用数据的价值。然而,流处理应用在处理数据的过程中存在不稳定的计算资源供给、不确定的数据输入率等问题,导致流处理应用的任务处理步调不一致,需要合理的调度以加速处理。但是,传统的并行与分布式系统负载均衡、调度技术并不适用于流处理系统,而且,当前的流处理系统均衡方法也不能有效地调度任务以解决数据处理过程存在的问题,亟需寻求新的调度方法。针对数据处理过程中存在的问题,结合流处理系统的特性,提出并实现了运行时感知调度方案,以达到动态调整任务部署保证低延迟的目的。首先,提出节点性价比评测方法,感知运行环境变化对节点处理数据效率的影响,为调度提供依据;其次,在对常用调度算法研究的基础上,提出基于节点性价比的调度算法,依据节点处理效率在集群上重新调度任务,使节点上的任务计算量与节点计算能力相匹配,同时尽可能减少节点间的任务迁移;再次,改进指数平滑预测方法,使其适应流处理特点,预测任务调度方案是否可行;最后,重新部署任务,重部署结果是任务被原节点运行,或被性能更好节点运行。调度方案与流处理系统Storm高度集成,并重用Storm模块,减少额外开销。相同的实验条件下,将运行时感知调度方案与Storm默认调度方案进行了对比。实验表明,相比于默认调度方案,运行时感知调度方案使算子平均处理延迟降低29.6%,使节点间最大任务处理延迟差降低47.4%。此外,它处理相同的数据量时消耗更少的计算资源。更多还原

【Abstract】 In recent years, more and more data are real-time processed. Only stream processing systems keep very low latency, the value of data in stream applications is completely used. However, since stream input rates or resource availability fluctuates as the stream applications run, the stream operator tasks have different processing steps and should be redistributed dynamically for speed up processing. Load balancing and scheduling in traditional paralle l and distributed systems do not work in stream processing systems. The current load balancing and scheduling technique in stream processing systems cannot dynamically redistribute the tasks to solving the input rates or resource availability fluctuation problem. Therefore, it is very critical to find a new scheduling technique.For solving the problem described above, we present and implement a runtime-aware scheduling mechanism that aims at decreasing processing latency by redistributing tasks among nodes and taking the features of stream processing into account. First, a performance cost ratio(PCR) method is proposed for detecting the node processing efficiency affected by running environment fluctuation and providing information for scheduling decision at runtime. Second, the scheduling mechanism proposes a based PCR scheduling algorithm, which assigns the amount of computation according to the node current processing capacity and considers how to reduce task migration. Third, the exponent smoothing method is improved for stream processing systems, which is used for predicting the scheduling feasibility. Finally, the scheduling is deployed. The result is that one task runs on its initial node or a faster node. We have implemented a scheduler as an extension to Storm and reused some modules for reducing overhead.We evaluate the scheduler and compare it with the initial scheduling with the same experimental conditions. The result shows that the scheduler can decrease processing latency outstandingly by 29.6% and the latency difference between different nodes by 47.4%. What’s more, the same amount of data uses less computational resources.更多还原