【作者】 刘婷；

【导师】 闵华清； 李慧琪；

【作者基本信息】 华南理工大学 ， 计算机技术， 2012， 硕士

【摘要】 在云计算环境中被广泛使用的并行编程模型是Google提出的Map/Reduce模型。当前，如何在现有云平台上有效的使用Map/Reduce编程模型，避免资源浪费成为研究热点。其中有两点值得关注：一、如何利用用户给出的作业信息来预测不同系统配置下的作业运行时间；二、如何利用预测结果信息来实现作业调优，从而提高系统性能和作业运行效率。目前没有较可靠且实用的作业运行时间预测机制和作业调优机制，用户只能通过以往的作业执行经验来进行粗略的估计。在本文中，首先我们调研了主流的商用云计算平台，介绍了云计算发展的现状和核心技术，在此基础上自主设计了AppleCloud云计算平台架构，并实现了AppleCloud云计算平台的“基础云”和“平台云”功能。接着我对作业监控部分进行了深入的研究，给出了作业运行时间预测模型，实现了AppleCloud云计算平台上作业运行时间的预测与作业调优，该预测模型解决了不同复杂度Map/Reduce作业的预测问题。最后使用了大量数据进行了实验，实验结果显示能够比较准确的对不同大小的作业以不同分片大小运行的时间进行预测，并且证明了根据预测模型给出的作业最优配置是稳定可靠的。该预测模型基于Google的Map/Reduce逻辑模型，适合不同的系统和网络环境，具有较强的通用性；对于不同时间复杂度和空间复杂度的类型的Map/Reduce作业，除了待定参数不同以外，其公式形式是统一的；对于不同云计算环境，除了待定参数不同以外，其公式形式是统一的；待定系数的计算仅需5条作业执行数据即可完成，作业运行时间预测简便迅速；预测准确度较高。实验结果显示线性复杂度作业的平均误差为3%，排序复杂度作业的平均误差为9%。基于该预测模型的作业调优实用可靠，大量实验结果证明，通过论文中提出的调优机制进行作业调优后的运行时间，在众多实验结果中排名前列，与实验中最短运行时间误差：线性复杂度在45秒内，排序复杂度在150秒内。并得出这样的结论：当云计算环境和作业的复杂度一定，待定系数就一定，其作业最优分片大小是固定的，并用大量的实验对该结论进行了验证。更多还原

【Abstract】 The parallel programming model is Google’s Map/Reduce model which is widely usedin cloud computing environment. At present, how to effectively use Map/Reduceprogramming model on the existing cloud platforms and how to avoid the waste of resourceshave become focus of current research. There are two issues of concern: one is how to use jobinformation given by the users to predict the time-consuming of the job in differentconfigurations; another is how to use the prediction results to achieve the job tuning, thusimproving the efficiencyof job execution and saving resources.Currently, there are no methods for the prediction and optimization of the job. So theusers can only give rough estimates based on previous experience in the job execution.In the paper, at first we research mainstream commercial cloud computing platforms,introduce the development and core technologies of cloud computing, independently designthe AppleCloud cloud computing platform architecture, and implement the functions of basiccloud and platform cloud. Then I depth research the part of the job monitoring, give a modelfor the job running time prediction, implement the job running time prediction and job tuning.Finally, we experiment on a large number of test data, the results show that the model isapplied to the different complexity of the Map/Reduce jobs, can accurately predict the job ofrunning time for different input data size and the slice size, and prove the optimal allocationof jobs given according to the prediction model is reliable.The time prediction model bases on Google’s Map/Reduce logic model, is suitable fordifferent system and network environment, so it has strong versatility; For the job withdifferent complexity of time and space, the form of the formula is unity, except for theundetermined parameters; For the job which is running in different cloud computingenvironment, the form of the formula is unity,except for the undetermined parameters; Thecalculation of the undetermined parameters just needs five job execution data, and theprediction of the job running time is easy and quick; Higher forecast accuracy,the averageerror of the linear complexity of the job is3%, and the average error of the sort of complexityof the job is9%.The job tuning method based on time prediction model is practical and reliable. A largenumber of experiments show that the running time of the job which is tuned with the methodproposed in the paper ranks in the previous five, away from the sampling shortest runningtime: linear complexity is within45seconds, the sorting complexity is150seconds. Come tothe conclusion: if the cloud computing environment and the complexity of the job are bothcertain, the undetermined parameters are fixed, and the optimal fragment size is fixed. The conclusion is validated with a large number of experiments.更多还原