【摘要】 近年来各领域应用的数据量和计算量需求都大幅增加,传统单个计算设备往往无法胜任如此规模的计算量,因此越来越多的领域开始尝试使用并行计算技术,分布式并行计算是进行并行计算的一种主要方式,常见的框架为基于Map Reduce的Hadoop。提出一种基于集群的通用并行计算框架,参考"管道过滤器"模式,对三个模块"任务划分"、"控制器节点"和"计算节点"都进行详细设计描述,相对于Hadoop,对有向无环图型任务由更好支持,并且支持迭代型任务,另外增加缓存机制,减少系统耗时,一定程度支持实时性应用。更多还原

【Abstract】 In recent years, the amount of data and computation requirements in various fields has increased significantly. Traditional single computing devices are often incapable of performing such computational tasks. More and more fields are beginning to use parallel computing.Distributed computation is one of the main parallel computing methods, Hadoop is a most common framework based on Map-Reduce in distributed computation. Proposes a general parallel computing framework based on cluster. Describes the "task splitter", "master node" and "computing node" in details with reference to "pipeline filter" mode. Compared with Hadoop, directed acyclic graph task is better supported, iteration task is supported. In addition, caching mechanism is added to reduce system time-consuming and support real-time application to a certain extent.更多还原