【摘要】 ELK架构是目前主流的日志大数据分析解决方案之一,ELK拥有比Spark更高的实时性,集成和部署比Spark简单方便,而且几乎可以在任何系统中进行集成。但是它的其通用性导致了它只能适用于一些简单的场景,无法像Spark一样精确针对每个应用系统进行复杂业务分析扩展。为了降低日志收集和清洗的开发成本同时兼顾分析的高实时性和可扩展性,结合ELK Stack与Spark,同时引入Kafka消息队列,构建了一套可扩展、高实时性且具有良好稳定性的征信日志挖掘系统。该系统比全栈Hadoop技术的大数据系统的开发工作量减少60%以上,且具有可根据企业需求灵活进行自定义分析组件的优点。更多还原

【Abstract】 ELK architecture is one of the mainstream log big data analysis solutions at present. ELK has higher real-time performance than Spark, and its integration and deployment are simpler and more convenient than Spark, and can be integrated in almost any system. But just because of its universality, it can only be applied to some simple scenarios, and can’t be expanded precisely for each application system like Spark. In order to reduce the development cost of early log collection and cleaning and realize the high real-time and extensibility of analysis, the paper combines ELK stack and Spark,introduces Kafka message queue at the same time,constructs a set of extensible,high real-time and good stability credit log mining system,which effectively reduces the workload of system development,and has the advantages of flexible customized analysis components according to the needs of enterprises.更多还原