【作者】 刘剑锋；

【导师】 张志政； 臧辉；

【作者基本信息】 东南大学 ， 软件工程（专业学位）， 2023， 硕士

【摘要】 在云计算主机资源的运维场景中,对海量主机KPI(Key Performance Indicator)指标进行实时监测分析来保障云服务的稳定性无疑是极具挑战性的任务。针对现行异常检测系统或算法的不足,如百度的Opprentice系统和腾讯的Metis系统在应对主机KPI异常检测场景时存在的需进行人工标注的问题,以及Yahoo的EGADS系统和阿里的Donut工具则存在的异常检测准确率低和模型通用性较差等缺点,本文设计和实现了一种基于无监督学习的主机KPI异常检测系统,本文做的主要研究工作如下:(1)本文针对主机KPI指标数据形态各异,使用单一模型无法对所有指标都达到理想异常检测效果的问题,依据分而治之的思路,将指标时间序列按照是否波动平稳、是否具备明显周期性变化的性质,将所有主机KPI指标划分为平稳型、周期型和随机波动型三类。同时,根据不同类别指标序列特点,设计和使用不同算法模型进行预测,提升了系统的整体异常检测准确率。(2)本文为不同类型指标序列提出了一套较完整的异常检测方案。其中,针对平稳型指标序列,本文结合了Kalman滤波器与Bi-LSTM(Kalman Filter Bidirectional Long Short Term Memory)模型进行数据预测,相较于直接使用LSTM模型进行预测,该算法在对时间序列内在时间模式的捕捉上能够更为准确;对于周期型时间序列,本文直接使用了擅长处理周期型时间序列预测任务的开源模型Prophet来进行预测;针对波动性更为随机、序列时间模式变化不清晰的随机波动型指标序列,本文则通过结合SWT(Stationary Wavelet Transform)变换和AE(Auto Encoder)模型重构的方式,来实现对时间序列数据的重构和异常检测。而对于少部分算法模型无法成功训练拟合的指标数据,本文通过结合LOF(Local Outlier Factor)、MCD(Minimum Covariance Determinant)和IForest(Isolation Forest)等多个基分类器来作为辅助的异常检测手段,取得了较单异常检测器更优的异常检测精度和准确率。(3)本文设计实现了一个主机KPI异常检测系统,系统主要分为七个功能模块,包括数据接入、数据预处理、指标分类和模型匹配、模型训练和调参、异常检测、任务调度、异常可视化和告警。并在对系统进行功能性和非功能性测试的实验中,验证了本文系统各功能模块的有效性和实用性。更多还原

【Abstract】 In the domain of cloud computing,real-time monitoring and analysis of key performance indicators(KPIs)of host resources pose a significant challenge yet are crucial for ensuring the stability of cloud services.Existing anomaly detection systems and algorithms,such as Baidu’s Opprentice system and Tencent’s Metis system,have limitations that necessitate manual annotation to address host KPI anomaly detection scenarios.Similarly,Yahoo’s EGADS system and Alibaba’s Donut tool exhibit drawbacks such as low anomaly detection accuracy and limited model versatility.To overcome these challenges,this thesis proposes an unsupervised learning-based system for host KPI anomaly detection.The main contribution of this thesis is threefold.Firstly,considering the distinctive characteristics of various host KPI data forms and the inefficiency of a single model in achieving ideal anomaly detection,this thesis adopts a divide-and-rule approach.It categorizes all host KPI indicators into three groups: stable,periodic,and random fluctuations.Different models are employed to predict the time series of each category,thereby enhancing the overall system accuracy.Secondly,to address the anomaly detection of different types of indicator sequences,a relatively complete anomaly detection scheme is proposed in this thesis.Specifically,an algorithm which combines Kalman Filter and Bidirectional Long Short-Term Memory(Kalman-Bi-LSTM)model is applied for data prediction of stationary time series.This model captures the intrinsic time patterns of time series more effectively than the LSTM model.For periodic time series,the thesis utilizes the open source model Prophet,which excels in handling periodic time series prediction tasks.In the case of time series exhibiting random fluctuation types,characterized by random volatility and ambiguous changes in sequence time patterns,the thesis combines the Stationary Wavelet Transform(SWT)method with the Auto Encoder(AE)model reconstruction method to enable the reconstruction and anomaly detection of time series data.Additionally,for a small amount of indicator data that does not fit general model predictions or model reconstruction tasks,this thesis employs multiple base learners,including Local Outlier Factor(LOF),Minimum Covariance Determinant(MCD),and Isolation Forest(IForest),for anomaly detection.This approach yields higher accuracy compared to single anomaly detectors.Finally,the thesis designs and implements an anomaly detection system for host KPIs,comprising seven functional modules: data access,data preprocessing,indicator classification and model matching,model training and parameter adjustment,anomaly detection,task scheduling,and anomaly visualization and alarm.The effectiveness and practicality of each functional module are verified through experiments conducted on functional and non-functional testing of the system.更多还原