- 73 views
- 108 downloads
Advancing Log Anomaly Detection by Deep Log Modeling
-
- Author / Creator
- Lin, Yifei
-
Nowadays systems logs are crucial for ensuring the reliability and security of modern computer systems. Effective log anomaly detection is essential for identifying potential threats and maintaining system integrity. Many existing unsupervised methods depend on additional abnormal data for hyperparameter selection or auxiliary datasets for discriminative model optimization, limiting their practical application. Moreover, current log anomaly detection methods are often static, offline, and struggle with the dynamic and evolving nature of real-world environments. They require extensive preprocessing and frequent retraining, which is resource-intensive and inefficient in handling non-stationary data distributions. This thesis addresses the challenges of log anomaly detection with two novel approaches: FastLogAD and LogREAD, which focus on offline and online anomaly detection, respectively.FastLogAD is designed for offline log sequence anomaly detection, emphasizing both speed and accuracy. By employing a Mask-Guided Anomaly Generator (MGAG) to produce pseudo-anomalies and a Discriminative Abnormality Separation (DAS) network to distinguish these from normal logs, FastLogAD effectively mitigates the dependency on additional training data. Experimental results on HDFS, BGL, and Thunderbird datasets demonstrate that FastLogAD not only achieves superior F1-Scores but also significantly enhances detection speed compared to existing methods.Conversely, LogREAD focuses on online continual log instance anomaly detection, adapting to evolving data distributions without extensive preprocessing. This parsing-free method utilizes an adaptively reduced memory bank and a continuously evolved feature extractor to manage dynamic log patterns. Evaluations reveal that LogREAD performs comparably to, or better than current offline methods and outperforms all online methods, showcasing its robustness in both static and dynamic environments.Together, FastLogAD and LogREAD offer comprehensive solutions for diverse log anomaly detection scenarios. FastLogAD excels in high-throughput offline detection, while LogREAD provides adaptive real-time monitoring. This thesis contributes to advancing the field of log anomaly detection by addressing key challenges such as non-stationarity, data imbalance, and the need for rapid anomaly identification. Future developments in creating a generalized model adapting to a broad range of systems following our work would extend its applicability in real-world scenarios.
-
- Subjects / Keywords
-
- Graduation date
- Fall 2024
-
- Type of Item
- Thesis
-
- Degree
- Master of Science
-
- License
- This thesis is made available by the University of Alberta Library with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.