# A Large-scale Study on Unsupervised Outlier Model Selection: Do Internal Strategies Suffice?

arXiv: https://arxiv.org/abs/2104.01422 · HTML: https://mlsanity.com/paper/2104.01422/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 39 references that are in the map.
- First posted on arXiv 3 Apr 2021.

## Builds on (most cited first, 6 of 6)

- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [PyOD: A Python Toolbox for Scalable Outlier Detection](https://mlsanity.com/paper/1901.01588/) (2019)
- [Unsupervised Model Selection for Variational Disentangled Representation Learning](https://mlsanity.com/paper/1905.12614/) (2019)
- [How to Evaluate the Quality of Unsupervised Anomaly Detection Algorithms?](https://mlsanity.com/paper/1607.01152/) (2016)
- [Robust Contextual Outlier Detection: Where Context Meets Sparsity](https://mlsanity.com/paper/1607.08329/) (2016)
- [Evaluation Measures for Relevance and Credibility in Ranked Lists](https://mlsanity.com/paper/1708.07157/) (2017)

## Built on by (most cited first, 8 of 8)

- [ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions](https://mlsanity.com/paper/2201.00382/) (2022)
- [Unsupervised Model Selection for Time-series Anomaly Detection](https://mlsanity.com/paper/2210.01078/) (2022)
- [Toward Unsupervised Outlier Model Selection](https://mlsanity.com/paper/2211.01834/) (2022)
- [Unsupervised Anomaly Detection with Rejection](https://mlsanity.com/paper/2305.13189/) (2023)
- [mTSBench: Benchmarking Multivariate Time Series Anomaly Detection and Model Selection at Scale](https://mlsanity.com/paper/2506.21550/) (2025)
- [Meta-Learning for Unsupervised Outlier Detection with Optimal Transport](https://mlsanity.com/paper/2211.00372/) (2022)
- [Hyperparameter Optimization for Unsupervised Outlier Detection](https://mlsanity.com/paper/2208.11727/) (2022)
- [Automated Machine Learning for Unsupervised Tabular Tasks](https://mlsanity.com/paper/2510.07569/) (2025)

## Abstract (arXiv)

Given an unsupervised outlier detection task, how should one select a detection algorithm as well as its hyperparameters (jointly called a model)? Unsupervised model selection is notoriously difficult, in the absence of hold-out validation data with ground-truth labels. Therefore, the problem is vastly understudied. In this work, we study the feasibility of employing internal model evaluation strategies for selecting a model for outlier detection. These so-called internal strategies solely rely on the input data (without labels) and the output (outlier scores) of the candidate models. We setup (and open-source) a large testbed with 39 detection tasks and 297 candidate models comprised of 8 detectors and various hyperparameter configurations. We evaluate 7 different strategies on their ability to discriminate between models w.r.t. detection performance, without using any labels. Our study reveals room for progress -- we find that none would be practically useful, as they select models only comparable to a state-of-the-art detector (with random configuration).