【摘要】 针对大数据处理框架MapReduce中的任务调度问题,提出一种基于Markov决策过程(Markov Decision Process,MDP)的任务调度算法,通过状态集来描述集群中节点的负载和作业的数据本地性需求,使用状态转移函数表示调度策略对状态的影响,采用值迭代求解算法求取最优策略,实现集群中节点的最优调度.实验结果表明,该算法能够保证数据本地性的同时,减少作业响应时间,提高系统综合性能.更多还原

【Abstract】 A task scheduling algorithm based on Markov decision process is proposed to address the problem of task scheduling in MapReduce framework.The algorithm describes the load of node in cluster and data localization using state space.The state transfer function represents the influence scheduling strategy of the state.The optimal scheduling policy is obtained by solving the MDP using value iteration.The experimental results show that this algorithm can guarantee the data locality,reduce job response time and improve the overall performance of the system.更多还原