【作者】 施军；

【导师】 黄宜华；

【作者基本信息】 南京大学 ， 计算机技术（专业学位）， 2019， 硕士

【摘要】 在大数据分析应用中,基于表模型的结构化大数据分析处理,仍然是目前众多行业最基本的需求。DataFrame是一种在编程语言环境中易于使用的表数据编程模型,它对数据分析统计过程有良好的抽象,因而得到了广泛的关注和使用。目前出现了 Pandas和Spark DataFrame等编程框架,但在编程处理能力上还存在诸多不足。为此,本文从编程模型和框架、算子并行化、性能优化、跨平台调度与计算四个层面,研究了 DataFrame大数据编程模型与框架,并设计实现了一个跨平台统一 DataFrame大数据编程系统。本文的主要工作与贡献点包括:(1)研究提出了跨平台统一DataFrame大数据编程模型和框架,建立了一种平台无关的DateFrame高层抽象,为上层用户提供良好的易用性和跨平台特性。(2)研究设计了基于Spark平台的DataFrame框架,在此基础上研究提出了轻量级全局索引及其构建方法,并进一步研究提出了 DataFrame核心算子并行化构建方法。(3)基于Spark平台研究实现了大规模DataFrame性能优化方法,包括轻量级全局索引构建的优化方法、基于辅助索引的标签切片查询和更新优化方法、以及基于局部索引构建实现了算子执行性能的优化。(4)针对跨平台DataFrame环境下不同的DataFrame操作与数据规模,研究提出了一种执行时间评估模型,综合考虑执行时间和平台切换数据传输时间开销,进而实现跨平台DataFrame自动优化调度与执行。(5)在上述关键技术的研究基础上,设计实现了跨平台统一 DataFrame原型系统Octopus-DataFrame,该系统提供了完整的DataFrame编程计算接口,底层集成了多个现有的主流平台,并可以根据不同的数据规模和操作,实现自动的平台选择和优化调度。实验结果表明,所研究提出的技术方法和系统具有显著的性能优化效果。更多还原

【Abstract】 In applications of big data analysis,structured big data analysis and processing based on the table model is still the most basic requirement in many industries.As a table programming model,DataFrame has been widely used since it has a good abstraction of the data analysis and statistics process,and it is easy to use in programming.Although some DataFrame programming frameworks such as Pandas and Spark DataFrame have appeared,they still have many shortcomings.This paper makes a research focusing on the DataFrame big data programming model and framework from four aspects:programming model and framework,parallelism of operators,execution performance optimization,and cross-platform scheduling and computation.In this paper,a cross-platform unified DataFrame big data programming system has been designed and implemented.The primary work and contributions of this paper include:(1)This paper proposes a cross-platform unified DataFrame big data programming model and framework,and establishes a platform-independent DateFrame high-level abstraction,which provides good usability and cross-platform features for upper-level users.(2)Based on the Spark platform,this paper designs a DataFrame framework and proposes a lightweight global index and its construction method.Furthermore,the parallel construction methods of core DataFrame operators are proposed.(3)The large-scale DataFrame performance optimization methods are proposed based on the Spark platform,including an optimization method for lightweight global index construction,a optimization method for label slice query and update based on secondary index,and an operator execution performance optimization based on local index.(4)For different DataFrame operators and different data size in the cross-platform computing environment,an execution time evaluation model is proposed,which comprehensively considers the overhead of execution time and data transmission time between platform switching.Based on the model,a cross-platform DataFrame automatic optimized scheduling and execution method is implemented.(5)Based on the research of the above key technologies,a cross-platform unified DataFrame prototype system called Octopus-DataFrame is designed and implemented.The system provides complete DataFrame programming interface,and integrates several existing mainstream DataFrame platforms.Also,the system supports automatic platform selection and optimized scheduling across different platforms under different operators and data size.The results of experiment show that the proposed technologies and system methods have significant optimization effects for execution performance.更多还原