【摘要】 传统的日志分析技术在处理海量数据时存在计算瓶颈。针对该问题,研究了基于大数据技术的日志分析方案,即由多台计算机完成日志文件的存储、分析和挖掘工作;建立了一个分层的网络日志分析系统:Syslog完成日志采集,Hadoop负责分布式存储和并行计算,在MapReduce框架下重新实现了IP统计算法。实验结果表明,在数据密集型计算中使用大数据技术可以明显提高算法的执行效率和系统的可扩展性。更多还原

【Abstract】 There is a calculation bottleneck when traditional log analysis technology processes the massive data. To solve this problem,a log analysis solution based on big data technology is proposed in this paper. In this solution,the log file storage,analysis and mining tasks will be decomposed on multiple computers. A layered network log analysis system was established,in which Syslog fulfils the log acquisition,Hadoop is responsible for distributed storage and parallel calculation,and IP statistics algorithm is realized with Map Reduce technology. The experimental results show that the use of big data technology in data-intensive computation can significantly improve the execution efficiency of algorithms and scalability of the system.更多还原