蒋鸿城,陈可,耿继朴,戚伟强,沈志豪
JIANG Hongcheng,CHEN Ke,GENG Jipu,QI Weiqiang,SHEN Zhihao

• 1:国网浙江省电力有限公司信息通信分公司

摘要(Abstract)：

随着近几年国内信息化水平不断提高,国网浙江电力主机、网络等设备数量急剧上升,业务系统的访问和操作量也迅速增加,传统的日志审计系统在应对超大规模海量日志的存储和分析时能力明显不足。针对海量日志采集、处理的需求,结合国网浙江省电力公司的实际情况,提出了一种基于开源ElasticSearch(弹性搜索)和Logstash(日志采集)的统一日志分析系统,并整合强隔离装置、toprowMQ消息队列、Kafka消息队列、Spark Streaming流数据处理框架,保证数据的安全性、可靠性和实时性。最后,通过日志分析系统在国网浙江电力的应用情况,验证了该系统的功能。
With the improvement of domestic informatization level in recent years, the number of hosts and network equipment in Zhejiang Electric Power Company increases rapidly, so do the system access and operations. Therefore, the traditional log audit system is difficult to store and analyze large-scale mass log. In order to meet the needs of log collection and dealing, a unified log analysis system based on ElasticSearch and Logstash is proposed in accordance with practical situation of Zhejiang Electric Power Company, which combines strong isolation devices, toprowMQ, Kafka cluster and Spark Streaming to guarantee the data are safe,reliable and real-time. Finally, the system functions are verified by the application of the log analysis system in Zhejiang Electric Power Company.

关键词(KeyWords)： 统一日志分析系统;海量日志;弹性搜索;日志采集
unified log analysis system;mass log;ElasticSearch;Logstash

Abstract:

Keywords:

基金项目(Foundation):

作者(Author): 蒋鸿城,陈可,耿继朴,戚伟强,沈志豪
JIANG Hongcheng,CHEN Ke,GENG Jipu,QI Weiqiang,SHEN Zhihao

DOI: 10.19585/j.zjdl.201712006

参考文献(References)：

• [1]季根生.计算机系统日志自动分析的实现[J].铁路计算机应用,2007,16(3):48-50.
• [2]周琪锋.基于网络日志的安全审计系统的研究与设计[J].计算机技术与发展,2009,19(11):139-142.
• [3]段娟.基于Web应用的安全日志审计系统研究与设计[J].信息网络安全,2014(10):70-76.
• [4]吕荣峰,杨梦宁,余虹.智能日志审计与预警系统功能设计与实现[J].数字技术与应用,2016(2):187-189.
• [5]陈文,邓韵东,钏涛.电网网络系统运行日志分析[J].云南电力技术,2011,39(5):60-61.
• [6]姜文.基于Hadoop平台的数据分析和应用[D].北京:北京邮电大学,2011.
• [7]LEE K H,LEE Y J,CHOI H,et al.Parallel data processing with Map Reduce:a survey[J].ACM SIGMOD Record,2012,40(4):11-20.
• [8]DEAN J,GHEMAWAT S.Map Reduce:Simplified Data Processing on Large Clusters[J].Communications of the ACM,2008,51(1):107-113.
• [9]吴志用.日志审计系统中预处理关键技术研究[D].杭州:浙江工商大学,2011.
• [10]徐刚.集中式日志系统ELK协议栈详解[EB/OL].[2017-10-11].https://www.ibm.com/developerworks/cn/opensource/os-cn-elk/.
• [11]张彩云,牛永红,赵迦琪.ELK日志分析平台在系统运维中的应用[J].电子技术与软件工程,2017(6):181-183.
• [12]白俊,郭贺彬.基于Elastic Search的大日志实时搜索的软件集成方案研究[J].四平:吉林师范大学学报(自然科学版).2014,35(1):85-87.
• [13]Elasticsearch Reference[EB/OL].[2017-10-11].https://www.elastic.co/downloads/elasticsearch.