# We Need to Rethink Benchmarking in Anomaly Detection

arXiv: https://arxiv.org/abs/2507.15584 · HTML: https://mlsanity.com/paper/2507.15584/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 90 references that are in the map.
- First posted on arXiv 21 Jul 2025.

## Builds on (most cited first, 30 of 32)

- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Deep Learning for Anomaly Detection: A Survey](https://mlsanity.com/paper/1901.03407/) (2019)
- [ADBench: Anomaly Detection Benchmark](https://mlsanity.com/paper/2206.09426/) (2022)
- [Classification-Based Anomaly Detection for General Data](https://mlsanity.com/paper/2005.02359/) (2020)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions](https://mlsanity.com/paper/2201.00382/) (2022)
- [COPOD: Copula-Based Outlier Detection](https://mlsanity.com/paper/2009.09463/) (2020)
- [FLAML: A Fast and Lightweight AutoML Library](https://mlsanity.com/paper/1911.04706/) (2019)
- [Anomaly Detection in Autonomous Driving: A Survey](https://mlsanity.com/paper/2204.07974/) (2022)
- [AnomalyDAE: Dual autoencoder for anomaly detection on attributed networks](https://mlsanity.com/paper/2002.03665/) (2020)
- [BMAD: Benchmarks for Medical Anomaly Detection](https://mlsanity.com/paper/2306.11876/) (2023)
- [Extended Isolation Forest](https://mlsanity.com/paper/1811.02141/) (2018)
- [On Diffusion Modeling for Anomaly Detection](https://mlsanity.com/paper/2305.18593/) (2023)
- [A Closer Look at AUROC and AUPRC under Class Imbalance](https://mlsanity.com/paper/2401.06091/) (2024)
- [NetFlow Datasets for Machine Learning-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2011.09144/) (2020)
- [A Survey on GANs for Anomaly Detection](https://mlsanity.com/paper/1906.11632/) (2019)
- [Statistical Analysis of Nearest Neighbor Methods for Anomaly Detection](https://mlsanity.com/paper/1907.03813/) (2019)
- [Hyperparameter Sensitivity in Deep Outlier Detection: Analysis and a Scalable Hyper-Ensemble Solution](https://mlsanity.com/paper/2206.07647/) (2022)
- [Navigating the Metric Maze: A Taxonomy of Evaluation Metrics for Anomaly Detection in Time Series](https://mlsanity.com/paper/2303.01272/) (2023)
- [On the Nature and Types of Anomalies: A Review of Deviations in Data](https://mlsanity.com/paper/2007.15634/) (2020)
- [Unsupervised anomaly detection algorithms on real-world data: how many do we need?](https://mlsanity.com/paper/2305.00735/) (2023)
- [ImageNot: A contrast with ImageNet preserves model rankings](https://mlsanity.com/paper/2404.02112/) (2024)
- [Meta-survey on outlier and anomaly detection](https://mlsanity.com/paper/2312.07101/) (2023)
- [ADGym: Design Choices for Deep Anomaly Detection](https://mlsanity.com/paper/2309.15376/) (2023)
- [Automating Outlier Detection via Meta-Learning](https://mlsanity.com/paper/2009.10606/) (2020)
- [No Free Lunch But A Cheaper Supper: A General Framework for Streaming Anomaly Detection](https://mlsanity.com/paper/1909.06927/) (2019)
- [A Taxonomy of Anomalies in Log Data](https://mlsanity.com/paper/2111.13462/) (2021)
- [Anomaly Detection in Double-entry Bookkeeping Data by Federated Learning System with Non-model Sharing Approach](https://mlsanity.com/paper/2501.12723/) (2025)
- [On Bootstrapping Machine Learning Performance Predictors via Analytical Models](https://mlsanity.com/paper/1410.5102/) (2014)

## Built on by (most cited first, 4 of 4)

- [MacrOData: New Benchmarks of Thousands of Datasets for Tabular Outlier Detection](https://mlsanity.com/paper/2602.09329/) (2026)
- [FoMo X: Modular Explainability Signals for Outlier Detection Foundation Models](https://mlsanity.com/paper/2603.17570/) (2026)
- [RangeAD: Fast On-Model Anomaly Detection](https://mlsanity.com/paper/2603.17795/) (2026)
- [Evaluating Tabular Representation Learning for Network Intrusion Detection](https://mlsanity.com/paper/2605.02519/) (2026)

## Abstract (arXiv)

Despite the continuous proposal of new anomaly detection algorithms and extensive benchmarking efforts, progress seems to stagnate, with only minor performance differences between established baselines and new algorithms. In this position paper, we argue that this stagnation is due to limitations in how we evaluate anomaly detection algorithms. In current benchmarks, a trivial algorithm that only checks for extreme values in individual features performs competitively with state-of-the-art deep learning methods, despite failing on simple cases such as anomalies within an annulus of normal points. Moreover, existing benchmarks do not adequately reflect the diversity of anomaly detection applications, making it difficult for practitioners to reliably select algorithms for their applications. Consequently, we need to rethink benchmarking in anomaly detection. In our opinion, anomaly detection should be studied using scenarios that group applications sharing relevant characteristics, defined through a common taxonomy. Benchmarking within scenarios enables scenario-specific choices for preprocessing, metrics, and model selection, clarifying which advances transfer across similar applications and providing practitioners with reliable guidance for their specific contexts.