【作者】 马进；

【导师】 黄忠东；

【作者基本信息】 浙江大学 ， 计算机应用技术， 2013， 硕士

【摘要】 大数据时代的来临,为数据的实时处理技术带来了巨大的变革和挑战,在这个背景下,D-Stream作为D-Ocean非结构化数据管理系统的流式处理子系统,为基于海量数据实时处理的应用提供给了一套通用的、可靠的、可扩展的分布式计算框架。D-Stream系统的实现基于一套流式处理通用框架设计,借鉴了S4, Storm等众多开源流式处理平台的先进思想。它的功能结构主要包含三大部分：首先是一套简洁开放的任务模型,通过D-Stream任务模型,应用能够根据需求动态地定制任务拓扑。其次是一套可靠稳定的流式处理引擎,保障数据在计算任务间快速透明地传输,使不同的任务可以高效地协调工作。最后是一个高可用的调度框架,通过有效调度计算资源,充分发挥集群的计算能力,并在消息积压时提供高效的负载分流机制。围绕这三大问题,本文描述了用D-Stream任务模型为现实应用建模的方法。就D-Stream实现部分,介绍了D-Stream组件架构和对称式调度框架,重点描述了流式处理引擎实现中用到的相关算法和设计模式,并针对D-Stream系统的各方面特性给予了全面评估。最后本文通过D-Ocean CBIR勺应用案例,验证了在海量数据的实时应用中D-Stream系统的优越性。更多还原

【Abstract】 The approach of big data era has brought significant challenge for real-time processing. Under this background, the D-Stream stream processing system provided a general, reliable, efficient and scalable distributed computing framework for the applications which based on real-time processing of massive data, as the task engine of D-Ocean which is an unstructured data management system.The implementation of D-Stream system comes from a design of common stream processing framework, taking in several advanced ideas of open source stream processing platform, such as S4and Storm. Its functional structure mainly includes three parts:the first is a simple and open task model, which could be used to build flexible task topology according to requirements. The second is a reliable and stable stream processing engine, which guarantees rapid and transparent data transmission among processing elements. The last part is a high available and scalable scheduling framework, making full use of the whole cluster by scheduling computing resources efficiently.Around the three parts, we represent the D-Stream modeling method for real application. As for the implementation of the D-Stream system, this article introduces the component architecture and symmetric scheduling framework, focusing on the relevant algorithms and design patterns used in the implementation of the stream processing engine. At last, we make a comprehensive assessment for the D-Stream stream processing system and use the case of D-Ocean CBIR application to verify the high utility of D-Stream system in real-time processing applications.更多还原