# Why Ranking Anomaly Detection Algorithms Isn't as Reliable as You May Think

arXiv: https://arxiv.org/abs/2608.04613 · HTML: https://mlsanity.com/paper/2608.04613/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Builds on (most cited first, 7 of 7)

- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [ADBench: Anomaly Detection Benchmark](https://mlsanity.com/paper/2206.09426/) (2022)
- [On Diffusion Modeling for Anomaly Detection](https://mlsanity.com/paper/2305.18593/) (2023)
- [Meta-survey on outlier and anomaly detection](https://mlsanity.com/paper/2312.07101/) (2023)
- [From Zero to Hero: Advancing Zero-Shot Foundation Models for Tabular Outlier Detection](https://mlsanity.com/paper/2602.03018/) (2026)
- [MacrOData: New Benchmarks of Thousands of Datasets for Tabular Outlier Detection](https://mlsanity.com/paper/2602.09329/) (2026)
- [FoMo X: Modular Explainability Signals for Outlier Detection Foundation Models](https://mlsanity.com/paper/2603.17570/) (2026)

## Abstract (arXiv)

Anomaly detection is a safety-critical machine learning problem with applications ranging from fraud detection to network intrusion prevention and industrial monitoring. Despite the large number of proposed anomaly detection algorithms, many novel methods claim state-of-the-art performance. However, many authors do so under benchmark settings that are not aligned with one another. This lack of comparability raises concerns regarding the reproducibility and reliability of anomaly detection benchmarks. In this work, we study the impact of common benchmarking choices on the stability of algorithm rankings. Using seven representative anomaly detection algorithms and 690 datasets from the OddBench benchmark suite, we analyze how rankings change under varying dataset selections, evaluation metrics, hyperparameter configurations, and random seeds. To quantify this effect, we introduce a rank instability metric measuring the variability of algorithm rankings across benchmark settings. Our results show that algorithm rankings in anomaly detection are highly unstable. In many cases, almost every competitive algorithm can appear as the best-performing method under some benchmark configuration. Among the studied factors, dataset selection and hyperparameter choice contribute most strongly to ranking uncertainty, while random seeds and evaluation metrics have a comparatively limited impact. We also observe that reliable benchmarking requires substantially larger and more diverse dataset collections than the ones commonly used in prior work.