# Towards a Rigorous Evaluation of Time-series Anomaly Detection

arXiv: https://arxiv.org/abs/2109.05257 · HTML: https://mlsanity.com/paper/2109.05257/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 11 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding](https://mlsanity.com/paper/1802.04431/) (2018)
- [Deep Learning for Anomaly Detection: A Survey](https://mlsanity.com/paper/1901.03407/) (2019)
- [Graph Neural Network-Based Anomaly Detection in Multivariate Time Series](https://mlsanity.com/paper/2106.06947/) (2021)
- [Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications](https://mlsanity.com/paper/1802.03903/) (2018)
- [LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection](https://mlsanity.com/paper/1607.00148/) (2016)
- [Patch SVDD: Patch-level SVDD for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2006.16067/) (2020)
- [A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder](https://mlsanity.com/paper/1711.00614/) (2017)
- [A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data](https://mlsanity.com/paper/1811.08055/) (2018)
- [MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks](https://mlsanity.com/paper/1901.04997/) (2019)
- [Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress](https://mlsanity.com/paper/2009.13807/) (2020)
- [Learning Graph Structures with Transformer for Multivariate Time Series Anomaly Detection in IoT](https://mlsanity.com/paper/2104.03466/) (2021)

## Built on by (most cited first, 30 of 97)

- [Deep Learning for Time Series Anomaly Detection: A Survey](https://mlsanity.com/paper/2211.05244/) (2022)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [Local Evaluation of Time Series Anomaly Detection Algorithms](https://mlsanity.com/paper/2206.13167/) (2022)
- [TAB: Unified Benchmarking of Time Series Anomaly Detection Methods](https://mlsanity.com/paper/2506.18046/) (2025)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [TFAD: A Decomposition Time Series Anomaly Detection Architecture with Time-Frequency Analysis](https://mlsanity.com/paper/2210.09693/) (2022)
- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection](https://mlsanity.com/paper/2401.15123/) (2024)
- [AnomalyBERT: Self-Supervised Transformer for Time Series Anomaly Detection using Data Degradation Scheme](https://mlsanity.com/paper/2305.04468/) (2023)
- [Navigating the Metric Maze: A Taxonomy of Evaluation Metrics for Anomaly Detection in Time Series](https://mlsanity.com/paper/2303.01272/) (2023)
- [TimeSeriesBench: An Industrial-Grade Benchmark for Time Series Anomaly Detection Models](https://mlsanity.com/paper/2402.10802/) (2024)
- [T-Rep: Representation Learning for Time Series using Time-Embeddings](https://mlsanity.com/paper/2310.04486/) (2023)
- [MultiRC: Joint Learning for Time Series Anomaly Prediction and Detection with Multi-scale Reconstructive Contrast](https://mlsanity.com/paper/2410.15997/) (2024)
- [Graph Anomaly Detection in Time Series: A Survey](https://mlsanity.com/paper/2302.00058/) (2023)
- [Correlation-aware Spatial-Temporal Graph Learning for Multivariate Time-series Anomaly Detection](https://mlsanity.com/paper/2307.08390/) (2023)
- [Nominality Score Conditioned Time Series Anomaly Detection by Point/Sequential Reconstruction](https://mlsanity.com/paper/2310.15416/) (2023)
- [CARLA: Self-supervised Contrastive Representation Learning for Time Series Anomaly Detection](https://mlsanity.com/paper/2308.09296/) (2023)
- [Explainable Time Series Anomaly Detection using Masked Latent Generative Modeling](https://mlsanity.com/paper/2311.12550/) (2023)
- [PATE: Proximity-Aware Time series anomaly Evaluation](https://mlsanity.com/paper/2405.12096/) (2024)
- [Time Series Anomaly Detection using Diffusion-based Models](https://mlsanity.com/paper/2311.01452/) (2023)
- [Unraveling the "Anomaly" in Time Series Anomaly Detection: A Self-supervised Tri-domain Solution](https://mlsanity.com/paper/2311.11235/) (2023)
- [Deep Contrastive One-Class Time Series Anomaly Detection](https://mlsanity.com/paper/2207.01472/) (2022)
- [OneShotSTL: One-Shot Seasonal-Trend Decomposition For Online Time Series Anomaly Detection And Forecasting](https://mlsanity.com/paper/2304.01506/) (2023)
- [DyEdgeGAT: Dynamic Edge via Graph Attention for Early Fault Detection in IIoT Systems](https://mlsanity.com/paper/2307.03761/) (2023)
- [What Do Machine Learning Researchers Mean by "Reproducible"?](https://mlsanity.com/paper/2412.03854/) (2024)
- [DACAD: Domain Adaptation Contrastive Learning for Anomaly Detection in Multivariate Time Series](https://mlsanity.com/paper/2404.11269/) (2024)
- [Robust and Explainable Detector of Time Series Anomaly via Augmenting Multiclass Pseudo-Anomalies](https://mlsanity.com/paper/2505.20765/) (2025)
- [PaAno: Patch-Based Representation Learning for Time-Series Anomaly Detection](https://mlsanity.com/paper/2602.01359/) (2026)
- [Multivariate Time Series Anomaly Detection: Fancy Algorithms and Flawed Evaluation Methodology](https://mlsanity.com/paper/2308.13068/) (2023)
- [Entropy Causal Graphs for Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2312.09478/) (2023)

## Abstract (arXiv)

In recent years, proposed studies on time-series anomaly detection (TAD) report high F1 scores on benchmark TAD datasets, giving the impression of clear improvements in TAD. However, most studies apply a peculiar evaluation protocol called point adjustment (PA) before scoring. In this paper, we theoretically and experimentally reveal that the PA protocol has a great possibility of overestimating the detection performance; that is, even a random anomaly score can easily turn into a state-of-the-art TAD method. Therefore, the comparison of TAD methods after applying the PA protocol can lead to misguided rankings. Furthermore, we question the potential of existing TAD methods by showing that an untrained model obtains comparable detection performance to the existing methods even when PA is forbidden. Based on our findings, we propose a new baseline and an evaluation protocol. We expect that our study will help a rigorous evaluation of TAD and lead to further improvement in future researches.