【摘要】 详细介绍了MapReduce编程框架,具体分析了MapReduce中shuffle阶段流程。分别从Map端数据压缩、重构远程数据拷贝传输协议、Reduce端内存分配优化三方面来优化和重构Shuffle。最后通过搭建Hadoop集群,运用MapReduce分布式算法测试实验数据。实验结果证明优化重构后的shuffle能显著提高MapReduce计算性能。更多还原

【Abstract】 We describe the MapReduce programming framework in detail,and analyze the shuffle-stage process.Shuffle in MapReduce is optimized and reconstructed through the following three measures:compressing the output of the Map end,reconstructing the protocol used to copy the data form the Map end to the Reduce end,and optimizing memory allocation on the Reduce end.Finally,through building a Hadoop cluster,the experimental data are tested using the MapReduce distributed algorithm.Experimental results show that the MapReduce computing performance improves significantly after optimizing the reconstructed shuffle.更多还原