|
|
基于StackLog的日志异常检测
|
Abstract:
日志信息是一种记录了系统运行状态的重要数据，是进行问题诊断、性能监控以及故障排除的主要依据。当系统出现故障时，通过详细系统地分析日志文本信息，研究人员可以快速准确地定位到系统出现问题的地方。通过分析现有的日志异常检测方法，本文发现当前用于日志异常检测的Text-CNN存在日志文本词向量维度急剧降低导致信息损失的问题。为了更充分地利用日志文本转换的词向量中携带的信息，本文提出了一种基于StackLog的日志异常检测方法。该方法采用堆叠卷积层逐步降低词向量维度的策略，尽可能保留词向量所携带的信息，并通过引入自注意力机制提升模型的检测能力。在HDFS和BGL两个公开数据集上进行对比实验验证了该模型在日志异常检测任务中的有效性。
Log information is an important data that records the operating status of a system, and is the main basis for problem diagnosis, performance monitoring, and troubleshooting. When the system malfunctions, researchers can quickly and accurately locate the problem by analyzing the log text information in detail and systematically. By analyzing existing log anomaly detection methods, this paper finds that the current Text-CNN used for log anomaly detection has the problem of information loss caused by a sharp decrease in the dimensionality of log text word vectors. In order to fully utilize the information carried in the word vectors of log text conversion, this paper proposes a log anomaly detection method based on StackLog. This method adopts a strategy of gradually reducing the dimensionality of word vectors by stacking convolutional layers, preserving the information carried by word vectors as much as possible, and improving the detection ability of the model by introducing self attention mechanism. Comparative experiments were conducted on two publicly available datasets, HDFS and BGL, to validate the effectiveness of the model in log anomaly detection tasks.
| [1] | Landauer, M., Onder, S., Skopik, F. and Wurzenberger, M. (2023) Deep Learning for Anomaly Detection in Log Data: A Survey. Machine Learning with Applications, 12, Article 100470. https://doi.org/10.1016/j.mlwa.2023.100470 |
| [2] | 张颖君, 刘尚奇, 杨牧, 等. 基于日志的异常检测技术综述[J]. 网络与信息安全学报, 2020, 6(6): 1-12. |
| [3] | BlueGene/L Message Types. https://www.usenix.org/cfdr-data#hpc4 |
| [4] | Xu, W., Huang, L., Fox, A., Patterson, D. and Jordan, M.I. (2009) Detecting Large-Scale System Problems by Mining Console Logs. Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, Big Sky Montana, 11-14 October 2009, 117-132. https://doi.org/10.1145/1629575.1629587 |
| [5] | Liu, F.T., Ting, K.M. and Zhou, Z. (2008) Isolation Forest. 2008 Eighth IEEE International Conference on Data Mining, Pisa, 15-19 December 2008, 413-422. https://doi.org/10.1109/icdm.2008.17 |
| [6] | Lou, J.G., Fu, Q., Yang, S., et al. (2010) Mining Invariants from Console Logs for System Problem Detection. Proceedings of 2010 USENIX Annual Technical Conference, Boston, 23-25 June 2010, 1-14. |
| [7] | Lu, S., Wei, X., Li, Y. and Wang, L. (2018) Detecting Anomaly in Big Data System Logs Using Convolutional Neural Network. 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress(DASC/PiCom/DataCom/CyberSciTech), Athens, 12-15 August 2018, 151-158. https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.00037 |
| [8] | 曾闽川, 方勇, 许益家. 基于联邦迁移学习的应用系统日志异常检测研究[J]. 四川大学学报(自然科学版), 2023, 60(3): 79-86. |
| [9] | 谢职权. 云平台日志异常检测技术研究与实现[D]: [硕士学位论文]. 镇江: 江苏大学, 2023. |
| [10] | Yin, C.Y. and Kong, X. (2024) Semi-Supervised Log Anomaly Detection Based on Bidirectional Temporal Convolutional Network. Journal of Computer Applications Research, 41, 2110-2117. |
| [11] | Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., et al. (2019) Robust Log-Based Anomaly Detection on Unstable Log Data. Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Tallinn, 26-30 August 2019, 807-817. https://doi.org/10.1145/3338906.3338931 |
| [12] | Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z. and Lyu, M.M. (2018) Tools and Benchmarks for Automated Log Parsing. arXiv:1811.03509. |
| [13] | Fu, Q., Lou, J., Wang, Y. and Li, J. (2009) Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis. 2009 Ninth IEEE International Conference on Data Mining, Miami Beach, 6-9 December 2009, 149-158. https://doi.org/10.1109/icdm.2009.60 |
| [14] | Tang, L., Li, T. and Perng, C.-S. (2011) LogSig: Generating System Events from Raw Textual Logs. Proceedings of the 20th ACM International Conference on Information and Knowledge Management, Glasgow, 24-28 October 2011, 785-794. |
| [15] | He, P., Zhu, J., He, S., Li, J. and Lyu, M.R. (2016) An Evaluation Study on Log Parsing and Its Use in Log Mining. 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), Toulouse, 28 June 2016-1 July 2016, 654-661. https://doi.org/10.1109/dsn.2016.66 |
| [16] | Devlin, J., Chang, M.W., Lee, K. and Toutanova, K. (2018) BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. |
| [17] | Kim, Y. (2014) Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, 25-29 October 2014, 1746-1751. https://doi.org/10.3115/v1/d14-1181 |
| [18] | Liang, Y., Zhang, Y., Xiong, H. and Sahoo, R. (2007) Failure Prediction in IBM Bluegene/l Event Logs. Seventh IEEE International Conference on Data Mining (ICDM 2007), Omaha, 28-31 October 2007, 583-588. https://doi.org/10.1109/icdm.2007.46 |
| [19] | Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., et al. (2019) Loganomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Macao, 10-16 August 2019, 4739-4745. https://doi.org/10.24963/ijcai.2019/658 |
| [20] | He, P., Zhu, J., Zheng, Z. and Lyu, M.R. (2017) Drain: An Online Log Parsing Approach with Fixed Depth Tree. 2017 IEEE International Conference on Web Services (ICWS), Honolulu, 25-30 June 2017, 33-40. https://doi.org/10.1109/icws.2017.13 |
