【作者】 曹杰；

【导师】 谢夏；

【作者基本信息】 华中科技大学 ， 计算机软件与理论， 2012， 硕士

【摘要】 随着数据规模在互联网、生物、天文学等领域的爆炸性增长，如何有效地从这些大规模的数据集中获取有价值信息的能力变得越发重要。R语言作为流行度最高的统计分析语言，它提供了丰富的数据统计功能，但却无法有效的处理海量数据。针对这一问题，设计并实现基于JVM的R语言海量数据统计分析集成框架JRBridge，并以Hadoop为例在该框架上实现R语言与Hadoop的结合，从而一定程度上实现了R语言对海量数据的分布式并行统计能力。为了利用已有海量数据计算框架和编程模型的研究成果，以R语言和海量数据计算框架的集成作为研究思路，分析了各计算框架与R语言进行集成的模型，总结并提出了基于SFAPI和UDFAPI的集成模型。在此模型的理论基础上，针对模型实现中必须满足的系统要求和公共组件，以及当前流行的开源海量数据计算框架大多基于Java虚拟机实现的特征，为了减小R语言与各框架之间的语言鸿沟，设计并实现了基于JRBridge集成框架的R语言海量数据统计分析系统。其中基于JVM的R语言解释执行机制解决了Java环境下执行嵌入式R语言代码的问题；R语言环境下Java类加载机制通过jload、import、$运算符实现了在R环境下对Java API类库调用并执行；R2J和J2R类型转换机制则为存在R与Java环境切换的方法调用提供参数与返回值的类型转换支持。上述机制的相互协作，配合R语言和Hadoop集成插件的实现，最终使得R语言具有了对海量数据统计分析的能力。从功能测试结果看出，通过HDFSAPI插件，它提供了一种使用R语言访问HDFS中海量数据集的方法；通过MapReduce API插件，它提供了一种使用R语言编写MapReduce统计分析代码的方法。性能测试中，在有5个工作节点并行统计的Hadoop集群环境下，词频统计程序在JRBridge集成框架下的执行时间可以达到原始R语言分析时间的1/7，并且随着测试数据集规模的不断提高，JRBridge集成框架呈现出线性可扩展的性能。更多还原

【Abstract】 Data explosion is experiencing in many domains such as astronomy, informationretrieval, and social network. The knowledge buries in these enormous datasets is soinvaluable that the ability to apply sophisticated statistical analysis methods to this data isbecoming more and more essential. As the most popular statistical language, R providesrich functionality for data analysis, but simply fails when the data becomes too large. Weproposed a generic framework JRBridge which can integrate R and JVM-based opensource computational infrastructures. Integrating R with Hadoop on this frameworkimproves the ability for large-scale statistic computing in R.In order to benefit from the lastest research achievements on computing frameworksand programming models for large scale data process, integrating R language withcomoutational infrastructures is our research thinking. With detailed analysis on how tointegrate R with popular open source infrastructures, we proposed SF and UDF integrationmodels. As most of open source computational infrastructures are JVM-based and thedefault APIs are written in Java, in order to make it easy for integrating, we designedJVM-based framework JRBridge which can integrate R and JVM-based open sourcecomputational infrastructures. JVM-based R language interpreter can interpretive executethe embedded R code in Java; Java Class Loader and Executor in R with jload,import and$operator make it possible to call method in Java library; R2J and J2R type convertersautomatically perform type conversion in the presence of context switching. With abovecooperating mechanisms and the plugin for integrating R and Hadoop, it is easy to handlelarge-scale data statistical computing in R.With the HDFS plugin, it brings a way to store and access datasets with millions ofobjects in HDFS. And with MapReduce plugin, it brings a natural environment to codeMapReduce algorithms in R. In the Hadoop clusters with5worknodes, the consumed timeof wordcount in JRBridge is nearly reduced to1/7of original R. The experiment resultshows that JRBridge scales linearly with the size of the dataset and thus provide a scalablesolution for large-scale statistical computing in R.更多还原