【摘要】 由于Hadoop能在同一时间处理多个用户提交的不同作业的多个任务,这使得用传统的方法对其进行建模和性能分析变得十分困难。为了解决这个问题,基于马尔可夫排队模型M/MMDP/C/K建立了一个随机Petri网(SPN)模型和一个确定随机Petri网(DSPN)模型来分别描述Hadoop调度中的数据状态和作业公平调度。通过设置DSPN中的使动谓词和随机开关来建模Hadoop公平调度和YARN公平调度。使用嵌入的马尔可夫链模型来分析单用户情景,而在分析多用户情景时则引入分解和迭代技术来减小模型的状态空间,从而避免产生状态爆炸问题。研究侧重于Hadoop中作业调度的平均性能,仅通过求解提出的分析模型,就可以对比和分析服务质量(Qo S)的一些关键指标,如平均吞吐量、平均队列长度和平均时延。采用Matlab进行仿真:当每秒到达任务数大于等于20时,YARN算法的数据积压和平均时延明显少于公平算法;当每秒到达任务数大于等于30时,YARN算法的平均吞吐量明显高于公平算法。实验结果表明,YARN公平算法能够减少平均处理和排队等待时间,在平均吞吐量、平均队列长度和平均时延上明显优于公平算法。更多还原

【Abstract】 Since Hadoop can process multiple tasks from different jobs of multiple users at the same time, it becomes very difficult to do modeling and performance analysis by traditional methods. To solve this problem, based on a Markov queuing model M / MMDP / C / K, a Stochastic Petri Net( SPN) model and a Deterministic and Stochastic Petri Net( DSPN)model were firstly established to describe the data states and job fair scheduling in Hadoop, respectively. Both fair scheduling and YARN fair scheduling were expressed by setting enabling predicates and random switches in the DSPN model. The embedded Markov chain model was used to analyze single user scenario, while decomposition and iteration techniques were applied to analyze multiple users’scenario, which can reduce state space of the proposed model and avoid state explosion.The study emphasized on average performance of job scheduling in Hadoop. Some key indicators of Quality of Service( Qo S),such as average throughput, average queue length and average delay, can be compared and analyzed only by solving the analysis model. Simulation experiments were conducted on Matlab. When the number of arrival tasks was equal to or greater than 20, both data backlog and average delay of YARN fair algorithm were significantly less than those of fair algorithm. When the number of arrival tasks was equal to or greater than 30, average throughput of YARN fair algorithm was significantly higher than those of fair algorithm. The simulation results show that YARN fair algorithm has better performance in average throughput, average queue length and average delay.更多还原