# Unsupervised Anomaly Detection with Rejection

arXiv: https://arxiv.org/abs/2305.13189 · HTML: https://mlsanity.com/paper/2305.13189/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 67 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 27 of 27)

- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [CutPaste: Self-Supervised Learning for Anomaly Detection and Localization](https://mlsanity.com/paper/2104.04015/) (2021)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [Deep Semi-Supervised Anomaly Detection](https://mlsanity.com/paper/1906.02694/) (2019)
- [SSD: A Unified Framework for Self-Supervised Outlier Detection](https://mlsanity.com/paper/2103.12051/) (2021)
- [ADBench: Anomaly Detection Benchmark](https://mlsanity.com/paper/2206.09426/) (2022)
- [PyOD: A Python Toolbox for Scalable Outlier Detection](https://mlsanity.com/paper/1901.01588/) (2019)
- [ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions](https://mlsanity.com/paper/2201.00382/) (2022)
- [Anomaly Detection in Video via Self-Supervised and Multi-Task Learning](https://mlsanity.com/paper/2011.07491/) (2020)
- [COPOD: Copula-Based Outlier Detection](https://mlsanity.com/paper/2009.09463/) (2020)
- [Self-Adaptive Training: beyond Empirical Risk Minimization](https://mlsanity.com/paper/2002.10319/) (2020)
- [Machine Learning with a Reject Option: A survey](https://mlsanity.com/paper/2107.11277/) (2021)
- [Neural Transformation Learning for Deep Anomaly Detection Beyond Images](https://mlsanity.com/paper/2103.16440/) (2021)
- [Learning Representations of Ultrahigh-dimensional Data for Random Distance-based Outlier Detection](https://mlsanity.com/paper/1806.04808/) (2018)
- [Classification with Rejection Based on Cost-sensitive Classification](https://mlsanity.com/paper/2010.11748/) (2020)
- [Unsupervised Model Selection for Variational Disentangled Representation Learning](https://mlsanity.com/paper/1905.12614/) (2019)
- [How to Evaluate the Quality of Unsupervised Anomaly Detection Algorithms?](https://mlsanity.com/paper/1607.01152/) (2016)
- [Estimating the Contamination Factor's Distribution in Unsupervised Anomaly Detection](https://mlsanity.com/paper/2210.10487/) (2022)
- [AUC-based Selective Classification](https://mlsanity.com/paper/2210.10703/) (2022)
- [PAC-Wrap: Semi-Supervised PAC Anomaly Detection](https://mlsanity.com/paper/2205.10798/) (2022)
- [A Large-scale Study on Unsupervised Outlier Model Selection: Do Internal Strategies Suffice?](https://mlsanity.com/paper/2104.01422/) (2021)
- [Probabilistic Robust Autoencoders for Outlier Detection](https://mlsanity.com/paper/2110.00494/) (2021)
- [Reliable Multi-label Classification: Prediction with Partial Abstention](https://mlsanity.com/paper/1904.09235/) (2019)
- [SafePredict: A Meta-Algorithm for Machine Learning That Uses Refusals to Guarantee Correctness](https://mlsanity.com/paper/1708.06425/) (2017)
- [Binary Classification with Bounded Abstention Rate](https://mlsanity.com/paper/1905.09561/) (2019)
- [How to Allocate your Label Budget? Choosing between Active Learning and Learning to Reject in Anomaly Detection](https://mlsanity.com/paper/2301.02909/) (2023)

## Built on by (most cited first, 9 of 9)

- [Gradient-Regularized Out-of-Distribution Detection](https://mlsanity.com/paper/2404.12368/) (2024)
- [A Causal Framework for Evaluating Deferring Systems](https://mlsanity.com/paper/2405.18902/) (2024)
- [Interpretable and Fair Mechanisms for Abstaining Classifiers](https://mlsanity.com/paper/2503.18826/) (2025)
- [Knowing What You Cannot Explain: Learning to Reject Low-Quality Explanations](https://mlsanity.com/paper/2507.12900/) (2025)
- [Swift Hydra: Self-Reinforcing Generative Framework for Anomaly Detection with Multiple Mamba Models](https://mlsanity.com/paper/2503.06413/) (2025)
- [Risk-Based Thresholding for Reliable Anomaly Detection in Concentrated Solar Power Plants](https://mlsanity.com/paper/2503.19146/) (2025)
- [Bounded-Abstention Pairwise Learning to Rank](https://mlsanity.com/paper/2505.23437/) (2025)
- [Bounded-Abstention Multi-horizon Time-series Forecasting](https://mlsanity.com/paper/2602.04714/) (2026)
- [Enhancing Tabular Anomaly Detection via Pseudo-Label-Guided Generation](https://mlsanity.com/paper/2604.18266/) (2026)

## Abstract (arXiv)

Anomaly detection aims at detecting unexpected behaviours in the data. Because anomaly detection is usually an unsupervised task, traditional anomaly detectors learn a decision boundary by employing heuristics based on intuitions, which are hard to verify in practice. This introduces some uncertainty, especially close to the decision boundary, that may reduce the user trust in the detector's predictions. A way to combat this is by allowing the detector to reject examples with high uncertainty (Learning to Reject). This requires employing a confidence metric that captures the distance to the decision boundary and setting a rejection threshold to reject low-confidence predictions. However, selecting a proper metric and setting the rejection threshold without labels are challenging tasks. In this paper, we solve these challenges by setting a constant rejection threshold on the stability metric computed by ExCeeD. Our insight relies on a theoretical analysis of such a metric. Moreover, setting a constant threshold results in strong guarantees: we estimate the test rejection rate, and derive a theoretical upper bound for both the rejection rate and the expected prediction cost. Experimentally, we show that our method outperforms some metric-based methods.