【摘要】 海洋地理空间数据具有分布时间长、范围广、类型复杂、时效性强等特点。如何使用分布式并行计算系统(云平台)高效地计算和处理大规模海洋地理空间数据,是一个具有挑战性的问题。本文提出了一种支持Spark、MPI、GPU三种异构任务异构应用的Spark框架(Heterogeneous Application Spark Framework,HASF),适用包含检索、计算和可视化的海洋地理空间数据计算和处理的全过程。HASF包含异构应用Spark调度器(Heterogeneous Application Spark Scheduler,HASS)和异构应用Spark运行时(Heterogeneous Application Spark Runtime,HASR)两部分。HASS扩展了Spark的默认调度机制,使得应用调度不仅依赖于Spark核心数还依赖于MPI进程数和GPU设备数。HASS中使用了一个简单的贪心策略来分配Spark核心数和GPU设备数,而MPI进程绑定到Spark核心。HASR用来支持异构应用在Spark平台上的运行。实验结果表明,HASS对实验中采用的异构海洋地理空间数据处理应用队列进行调度,相较于默认的Spark应用调度,应用任务队列总执行时间缩短了10%～15%。HASS明显提高了异构海洋地理空间数据计算和处理任务的批处理效率。更多还原

【Abstract】 Oceanographic Geospatial Data(OGD) has many features, such as long distribution time, wide range, complex type, strong information timeliness, etc. How to get a desired performance of large-scale Oceanographic Geospatial Data efficienty computing and processing with parallel and distributed system—cloud platform is a challenging probllem. This paper proposes a Heterogeneous Application Spark Framework(HASF) to support three kinds of heterogeneous applications. These applications are Spark, MPI and GPU applications, which cover the procedure of OGD computing and processing which includes retrieval, calculation and visualization. HASF includes two parts, Heterogeneous Application Spark Scheduler(HASS) and Heterogeneous Application Spark Runtime(HASR). HASS expands the default Spark scheduler to take into account not only Spark cores but also MPI processes and GPU devices. HASS uses the simple greedy strategy to allocate Spark cores and GPU devices. MPI processes are attached to Spark cores.The scheduling algorithm aims at improving the excution performance of the batch heterogenous applications which are independent. The application queue is divided into several stages and the applications of a stage will use up all Spark cores of the platform. Then reduce the execution resource of Spark applications and MPI applications in this stage and insert the GPU applications into the stage in order to improve the parallelism of heterogenous execution resource utilization. HASR supports heterogeneous applications running on Spark. It provides a runtime layer in order to solve the problem of how to make the MPI programs and GPU programs run on Spark, and a paradigm which is used to deal with the question of how to bind a spark core with a MPI process or create a mapping between spark cores and GPU devices. Experiments show that HASS can run the designed heterogeneous OGD applications 10% to 15% faster than the default Spark Scheduler. It significantly improves system computing and processing for concurrent heterogeneous OGD applications.更多还原