【摘要】 MapReduce作为一种分布式计算框架,在大规模数据处理中已经被广泛应用.负载均衡是影响分布式计算性能的关键因素.当前的MapReduce实现在给集群分发任务时,多采用Hash的随机划分方式,无法根据数据的分布情况来调整集群的负载,容易出现负载不均衡的问题.为解决这一问题,对当前的MapReduce框架进行拓展,根据对Map端中间结果的采样来动态确定Reduce任务数目以及划分函数,保证Reduce任务的负载均衡.具体地,对Map的结果进行采样,并将其发送给Job Tracker.Job Tracker根据采样数据的分布情况动态确定划分函数,以保证每个Reduce任务处理的记录数目均衡.更重要的是,在Hadoop框架内实现了负载均衡方法,测试结果证明方法具有良好的有效性、兼容性和可用性.更多还原

【Abstract】 As a distributed computing framework,MapReduce gains popularity in handing massive data.The load balance is the key to the performance of distributed computing.The current MapReduce implementation cannot adjust the load balance over clusters according to the data distribution adaptively with default random hash partition function.In this paper,we make extensions to MapReduce framework.Based on the sample results in the map side,we dynamically determine the number of reducers and build a partition function to enable load balance in the reduce side. Specifically,we sample the output of map and then report the results to job tracker.Job tracker will construct a dynamic partition function according to the data distribution,which guarantees the number of records processed in each reduce task is nearly even.More importantly,we implement load balance inside Hadoop framework,and the experimental results demonstrate that our approach achieves high efficiency,compatibility and applicability.更多还原