# AIOps-Driven Enhancement of Log Anomaly Detection in Unsupervised Scenarios

arXiv: https://arxiv.org/abs/2311.02621 · HTML: https://mlsanity.com/paper/2311.02621/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 5 Nov 2023.

## Builds on (most cited first, 10 of 10)

- [Deep Learning for Anomaly Detection in Log Data: A Survey](https://mlsanity.com/paper/2207.03820/) (2022)
- [AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges](https://mlsanity.com/paper/2304.04661/) (2023)
- [TransLog: A Unified Transformer-based Framework for Log Anomaly Detection](https://mlsanity.com/paper/2201.00016/) (2021)
- [A Comprehensive Survey of Logging in Software: From Logging Statements Automation to Log Mining and Analysis](https://mlsanity.com/paper/2110.12489/) (2021)
- [AIOps for a Cloud Object Storage Service](https://mlsanity.com/paper/2005.03094/) (2020)
- [Detecting Log Anomalies with Multi-Head Attention (LAMA)](https://mlsanity.com/paper/2101.02392/) (2021)
- [PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning](https://mlsanity.com/paper/2301.10681/) (2023)
- [Quality Monitoring and Assessment of Deployed Deep Learning Models for Network AIOps](https://mlsanity.com/paper/2202.13642/) (2022)
- [FIXME: Enhance Software Reliability with Hybrid Approaches in Cloud](https://mlsanity.com/paper/2102.09336/) (2021)
- [Leveraging Log Instructions in Log-based Anomaly Detection](https://mlsanity.com/paper/2207.03206/) (2022)

## Abstract (arXiv)

Artificial intelligence operations (AIOps) play a pivotal role in identifying, mitigating, and analyzing anomalous system behaviors and alerts. However, the research landscape in this field remains limited, leaving significant gaps unexplored. This study introduces a novel hybrid framework through an innovative algorithm that incorporates an unsupervised strategy. This strategy integrates Principal Component Analysis (PCA) and Artificial Neural Networks (ANNs) and uses a custom loss function to substantially enhance the effectiveness of log anomaly detection. The proposed approach encompasses the utilization of both simulated and real-world datasets, including logs from SockShop and Hadoop Distributed File System (HDFS). The experimental results are highly promising, demonstrating significant reductions in pseudo-positives. Moreover, this strategy offers notable advantages, such as the ability to process logs in their raw, unprocessed form, and the potential for further enhancements. The successful implementation of this approach showcases a remarkable reduction in anomalous logs, thus unequivocally establishing the efficacy of the proposed methodology. Ultimately, this study makes a substantial contribution to the advancement of log anomaly detection within AIOps platforms, addressing the critical need for effective and efficient log analysis in modern and complex systems.