# A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithms

arXiv: https://arxiv.org/abs/2204.09825 · HTML: https://mlsanity.com/paper/2204.09825/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [Deep Learning for Anomaly Detection: A Survey](https://mlsanity.com/paper/1901.03407/) (2019)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [Classification-Based Anomaly Detection for General Data](https://mlsanity.com/paper/2005.02359/) (2020)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [DROCC: Deep Robust One-Class Classification](https://mlsanity.com/paper/2002.12718/) (2020)
- [Adversarially Learned Anomaly Detection](https://mlsanity.com/paper/1812.02288/) (2018)
- [Deep Structured Energy Based Models for Anomaly Detection](https://mlsanity.com/paper/1605.07717/) (2016)
- [Neural Transformation Learning for Deep Anomaly Detection Beyond Images](https://mlsanity.com/paper/2103.16440/) (2021)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers](https://mlsanity.com/paper/2106.15195/) (2021)
- [Knowledge Base Completion: Baselines Strike Back](https://mlsanity.com/paper/1705.10744/) (2017)
- [Anomaly Detection: How to Artificially Increase your F1-Score with a Biased Evaluation Protocol](https://mlsanity.com/paper/2106.16020/) (2021)

## Built on by (most cited first, 11 of 11)

- [A Closer Look at AUROC and AUPRC under Class Imbalance](https://mlsanity.com/paper/2401.06091/) (2024)
- [Deep Anomaly Detection under Labeling Budget Constraints](https://mlsanity.com/paper/2302.07832/) (2023)
- [Deep Learning for Network Anomaly Detection under Data Contamination: Evaluating Robustness and Mitigating Performance Degradation](https://mlsanity.com/paper/2407.08838/) (2024)
- [Unsupervised Anomaly Detection for Auditing Data and Impact of Categorical Encodings](https://mlsanity.com/paper/2210.14056/) (2022)
- [FedAD-Bench: A Unified Benchmark for Federated Unsupervised Anomaly Detection in Tabular Data](https://mlsanity.com/paper/2408.04442/) (2024)
- [Robustness Evaluation of Deep Unsupervised Learning Algorithms for Intrusion Detection Systems](https://mlsanity.com/paper/2207.03576/) (2022)
- [Zero-Shot Anomaly Detection via Batch Normalization](https://mlsanity.com/paper/2302.07849/) (2023)
- [Generalizability of experimental studies](https://mlsanity.com/paper/2406.17374/) (2024)
- [An Investigation into the Performance of Non-Contrastive Self-Supervised Learning Methods for Network Intrusion Detection](https://mlsanity.com/paper/2510.02349/) (2025)
- [Attention, Anomalies! Handling Attention Layers in Unsupervised Federated Outlier Detection](https://mlsanity.com/paper/2608.04753/) (2026)
- [Kernel-based External Validity of Experimental Studies](https://mlsanity.com/paper/2610.13625/) (2026)

## Abstract (arXiv)

Anomaly detection has many applications ranging from bank-fraud detection and cyber-threat detection to equipment maintenance and health monitoring. However, choosing a suitable algorithm for a given application remains a challenging design decision, often informed by the literature on anomaly detection algorithms. We extensively reviewed twelve of the most popular unsupervised anomaly detection methods. We observed that, so far, they have been compared using inconsistent protocols - the choice of the class of interest or the positive class, the split of training and test data, and the choice of hyperparameters - leading to ambiguous evaluations. This observation led us to define a coherent evaluation protocol which we then used to produce an updated and more precise picture of the relative performance of the twelve methods on five widely used tabular datasets. While our evaluation cannot pinpoint a method that outperforms all the others on all datasets, it identifies those that stand out and revise misconceived knowledge about their relative performances.