# Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset

arXiv: https://arxiv.org/abs/2411.09047 · HTML: https://mlsanity.com/paper/2411.09047/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 77 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Builds on (most cited first, 13 of 13)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding](https://mlsanity.com/paper/1802.04431/) (2018)
- [Deep Learning for Anomaly Detection: A Survey](https://mlsanity.com/paper/1901.03407/) (2019)
- [Deep Semi-Supervised Anomaly Detection](https://mlsanity.com/paper/1906.02694/) (2019)
- [Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark](https://mlsanity.com/paper/1510.03336/) (2015)
- [Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series](https://mlsanity.com/paper/2010.05073/) (2020)
- [Anomaly Detection and Failure Root Cause Analysis in (Micro)Service-Based Cloud Applications: A Survey](https://mlsanity.com/paper/2105.12378/) (2021)
- [Anomaly Detection in a Large-scale Cloud Platform](https://mlsanity.com/paper/2010.10966/) (2020)
- [On Challenges of Cloud Monitoring](https://mlsanity.com/paper/1806.05914/) (2018)
- [Anomaly Detection in Cloud Components](https://mlsanity.com/paper/2005.08739/) (2020)
- [Online and Offline Analysis of Streaming Data](https://mlsanity.com/paper/1805.01021/) (2018)
- [Architecture for Analysis of Streaming Data](https://mlsanity.com/paper/1805.01025/) (2018)

## Built on by (most cited first, 7 of 7)

- [Log Anomaly Detection with Large Language Models via Knowledge-Enriched Fusion](https://mlsanity.com/paper/2512.11997/) (2025)
- [Benchmarking Anomaly Detection Across Heterogeneous Cloud Telemetry Datasets](https://mlsanity.com/paper/2602.13288/) (2026)
- [SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents](https://mlsanity.com/paper/2604.17699/) (2026)
- [Towards Generalizable Context-aware Anomaly Detection: A Large-scale Benchmark in Cloud Environments](https://mlsanity.com/paper/2508.01844/) (2025)
- [On Sequence-to-Sequence Models for Automated Log Parsing](https://mlsanity.com/paper/2602.07698/) (2026)
- [ClouDens: Operational Context-Aware Anomaly Detection for Large-scale Cloud System Monitoring](https://mlsanity.com/paper/2607.18127/) (2026)
- [Benchmarking Deep Learning Models for Anomaly Detection in Real-World Cloud Telemetry](https://mlsanity.com/paper/2610.15818/) (2026)

## Abstract (arXiv)

As Large-Scale Cloud Systems (LCS) become increasingly complex, effective anomaly detection is critical for ensuring system reliability and performance. However, there is a shortage of large-scale, real-world datasets available for benchmarking anomaly detection methods. To address this gap, we introduce a new high-dimensional dataset from IBM Cloud, collected over 4.5 months from the IBM Cloud Console. This dataset comprises 39,365 rows and 117,448 columns of telemetry data. Additionally, we demonstrate the application of machine learning models for anomaly detection and discuss the key challenges faced in this process. This study and the accompanying dataset provide a resource for researchers and practitioners in cloud system monitoring. It facilitates more efficient testing of anomaly detection methods in real-world data, helping to advance the development of robust solutions to maintain the health and performance of large-scale cloud infrastructures.