【作者】 杨鹏；

【导师】 陈彩；

【作者基本信息】 北京工业大学 ， 计算机技术（专业学位）， 2016， 硕士

【摘要】 大数据时代,数据的流式特征愈加显著,很多应用场景部署在流式数据处理平台上,然而,随着应用场景的复杂化及数据量的快速增长,分布式计算平台中节点间的负载失衡已经成为制约流式应用性能提升的瓶颈,因此对流式数据处理平台上负载均衡问题的研究已经凸显重要的研究价值。本文以流式数据处理平台JStorm作为研究对象,致力于解决作业资源分配不当导致的JStorm平台负载不均衡问题。论文对JStorm平台下的作业资源分配策略进行了深入分析,指出了在节点间可用资源异构场景下,该作业资源分配策略将导致的集群负载不均衡问题,并针对这个问题提出了改进的作业资源分配策略,通过考虑节点工作负载的非对称性,合理分配计算任务,提升了流式应用的运行性能。本文完成的主要工作有:1)详细介绍了流式数据处理、JStorm流式数据处理平台、分布式平台资源分配等技术。重点分析JStorm中计算任务从提交到分解为具体作业任务的整个过程,研究了JStorm为作业进行资源分配的过程,指出了在资源分配过程中因为缺乏对计算资源的整体评估,而导致集群负载不均衡的问题。2)提出了JStorm工作节点负载评估模型。通过对现有节点负载评估模型的研究,结合JStorm作业处理数据的特性,从JStorm工作节点的计算资源利用率及节点本身的数据处理能力出发,设计了JStorm工作节点负载的多权值评估模型,并对模型中的多个概念进行了详细说明。3)改进了JStorm作业资源分配策略。针对JStorm作业资源分配策略中忽视工作节点负载非对称的缺点,结合提出的工作节点负载评估模型,在作业资源分配过程时考虑工作节点的负载情况,对计算资源进行合理分配,保证集群的负载均衡,使得流式应用能够充分利用每个节点的计算资源。4)设计并实现了作业资源分配改进策略。将源码实现嵌入到JStorm的调度器中,搭建JStorm集群,通过实验验证了改进策略的可行性。实验结果表明,本文提出的改进作业资源分配策略解决了JStorm因为作业计算资源分配不当导致的集群负载不均衡问题,同时提高了JStorm处理数据的性能,将数据的吞吐量提高了10%。更多还原

【Abstract】 In the era of big data, the streaming characteristic of data is becoming more and more remarkable. A number of application scenarios are deployed on streaming data processing platform. However, with application scenarios becoming more complex and vigorous growth in data volume, imbalanced load among nodes in distributed computing platform has already become a bottleneck to restrict performance improvement of streaming application. Therefore, the value of research about load balancing issues on streaming data processing platform has become significant. This dissertation takes JStorm streaming data processing platform as the object of study and it devotes itself to resolve the imbalanced load of JStorm platform which is caused by inappropriate allocation of operation resource.This dissertation makes deep analysis on the strategy of resource allocation under JStorm platform and points out the clustering imbalanced load problem which may be caused by this distribution strategy of resource allocation under heterogeneous circumstance of available resource among pitch points. In addition, aiming at this problem, improved distribution strategy of resource allocation is put forward. Through considering the asymmetry of node workload and reasonable allocation of computational tasks, the operating performance of streaming application is enhanced. Main work accomplished by this dissertation include:1) This dissertation makes detailed introduction to such technologies as streaming data processing, JStorm streaming data processing platform and distributed platform resource allocation etc. It focused on analyzing the whole process of computational task in JStorm from its submission to decomposition into specific operation tasks. It also researches on the process of resource allocation by JStorm for operation and points out the clustering load imbalance resulted from lack of overall evaluation of computing resource in the process of resource allocation.2) This dissertation puts forward the load evaluation model for JStorm worknode. By researching present load evaluation model for node and combining the character of JStorm operation data processing, starting from computational resource utilization rate of JStorm worknode and Data processing capability of node itself, multi-weight load evaluation model for JStorm worknode is designed and plentiful concepts in the model are described in detail.3) The resource allocation strategy of JStorm operation has been improved. Aiming at the situation where the characteristics of unsymmetrical load of worknode is overlooked in resource allocation strategy of JStorm operation, in combination with the proposed worknode load evaluation model, the loading condition of worknode shall be taken into consideration in process of operation resource allocation and the computing resource must be distributed reasonably so that balanced cluster load is guaranteed and the streaming application is able to make full use of computational resource at each node.4) This dissertation designs and implements the improved strategy of resource allocation. It embeds the source code into dispatcher of JStorm, puts up a JStorm cluster to verifies the feasibility of improved strategy through experiment. The experimental result indicates that the improved resource allocation strategy which is proposed in this dissertation has solved the clustering load imbalance resulted from inappropriate allocation of computing resource in JStorm, at the same time, the data processing performance of JStorm is also improved, with data throughput being increased by 10%.更多还原