【作者】 王志刚；

【导师】 申彦明；

【作者基本信息】 大连理工大学 ， 计算机应用技术， 2016， 硕士

【摘要】 近年来,数据呈爆炸式增长,数据中心已经成为处理大数据的核心基础设施。为了能够快速有效地分析和提取数据中有价值的信息,研究人员提出了分布式计算框架,如Hadoop、Dryad等。这些框架将大数据划分成多个数据分片,利用集群中数以千计的机器并行的处理每个数据分片,最后通过网络传输分片的结果并汇总成最终的结果。提高集群利用率,减少运行应用的完成时间是每个数据中心的共同目标。在数据中心中,大数据应用框架如Hadoop在不同的运算阶段会产生大量的网络传输,已经成为影响应用性能的重要因素。通过对网络流的调度可以提高应用的性能。传统的方法大多数从网络流的角度进行调度,并没有考虑网络流之间的相关性。本文以Hadoop作为具体的实例,从应用层获得shuffle阶段网络流的信息,并根据shuffle阶段网络流的特点采用基于作业的优先级调度方法对数据流进行调度。首先,我们观察到关于shuffle阶段网络流的信息存储在Hadoop应用框架的临时文件和日志文件中。因此我们可以从从应用层来获得网络流信息。通过对这些文件的实时监控和分析,我们可以获得网络流信息和相互依赖关系。其次,我们利用全局视图,采用基于作业的优先级调度方法优化shuffle阶段网络传输。关键在于为所属相同作业的网络流分配相同的优先级。采用优先级调度策略,高优先级的网络流优先获得网络资源。通过这种优先级分配方式,使shuffle阶段一个作业的网络流能够尽快地一起完成,避免因为某一个网络流的延迟导致作业长时间的等待。我们从路径管理和交换机端的缓冲队列管理两个角度出发,为作业分配网络资源。在Fat-Tree拓扑下,我们提出flow-based 和 spray两种利用等价多路径的方法,实现网络上的负载均衡。在交换机端,我们设置多个优先级队列,提出队列管理方法。数据包在到达交换机时根据分配的优先级,加入到相应的优先级队列中。最后,在NS2网络模拟平台上模拟我们的调度方法进行大量对比实验。实验结果表明,基于作业的优先级调度方法能够减少shuffle阶段作业的平均完成时间,而且对于最高优先级的作业能够明显减少其网络传输的完成时间。同时,为了使模拟实验结果更加接近真实网络情况,我们设置背景流量情境、调度延迟情景和链路失效情景,在不同情景下运行对比实验。实验结果表明基于作业的优先级调度依然具有优化shuffle阶段网络流的性能。更多还原

【Abstract】 Recently with the explosion of data growth, data center has become a core infrastructure for big data processing. To analyze the data quickly and efficiently and extract value information, many distributed frameworks are proposed, such as Hadoop and Dryad. The frameworks split big data across the clusters of hundreds or thousands of computers, analyze each piece in parallel, then transfer the output of each piece and merge to the final result. To improve cluster utilization and to keep the duration of data processing jobs low are common goals for data centers.In data center, big data framework like Hadoop transfer a mount of data between different computation stages, which has become a key bottleneck of application performance. Optimizing the scheduling of flows can improve big data job performance. Traditional techniques are mostly flow-based scheduling, without considering the flow correlations. In this paper, we use Hadoop as a concrete example to obtain flow information during Hadoop shuffle from application layer and propose job-aware priority scheduling for data stream based on its feature.First, we observe that rich traffic demand information exists in the intermediate file and log files. This observation motivate us to obtain traffic forecasting from application layer. And such information and co-dependency can be extracted through run-time file system monitoring and file analyzing.Then we propose job-aware priority scheduling to optimize shuffle transfer with global view. The key is to allocate identical priority to flows for the same job. With priority policy, flows with high priority give priority to allocation of network resources, which makes flows of a job complete together as soon as possible to avoid job delay for a flow’s long duration. We allocate network resources for jobs from two perspectives, path management and buffer queue management at switch. In Fat-Tree topology, we introduce flow-based and spray to leverage equal cost multipath for load balance. We set priority queues and propose queue management. When a packet is received at switch, it will be enqueued according to its priority argument.Finally we implement our proposed scheme using NS-2 simulator and run contrast experiments. The results show that our scheduling can reduce the average completion time of shuffle, increase the network utilization and remarkably reduce the network transfer time for the job with the highest priority. Moreover, we simulate different settings like background traffic, delay of scheduling command and link failure to make the experiments more authentic. According to the results, we also show that job-aware priority scheduling can optimize the network transfer during shuffle phase.更多还原