# A Less Biased Evaluation of Out-of-distribution Sample Detectors

arXiv: https://arxiv.org/abs/1809.04729 · HTML: https://mlsanity.com/paper/1809.04729/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2018.

## Built on by (most cited first, 14 of 14)

- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [On the Properties of Kullback-Leibler Divergence Between Multivariate Gaussian Distributions](https://mlsanity.com/paper/2102.05485/) (2021)
- [Bayesian OOD detection with aleatoric uncertainty and outlier exposure](https://mlsanity.com/paper/2102.12959/) (2021)
- [Resisting Out-of-Distribution Data Problem in Perturbation of XAI](https://mlsanity.com/paper/2107.14000/) (2021)
- [CheXstray: Real-time Multi-Modal Data Concordance for Drift Detection in Medical Imaging AI](https://mlsanity.com/paper/2202.02833/) (2022)
- [MAGDiff: Covariate Data Set Shift Detection via Activation Graphs of Deep Neural Networks](https://mlsanity.com/paper/2305.13271/) (2023)
- [Likelihood Training of Cascaded Diffusion Models via Hierarchical Volume-preserving Maps](https://mlsanity.com/paper/2501.06999/) (2025)
- [Relaxed Triangle Inequality for Kullback-Leibler Divergence Between Multivariate Gaussian Distributions](https://mlsanity.com/paper/2602.02577/) (2026)
- [Uncertainty Surrogates for Deep Learning](https://mlsanity.com/paper/2104.08147/) (2021)
- [On the Importance of Regularisation & Auxiliary Information in OOD Detection](https://mlsanity.com/paper/2107.07564/) (2021)
- [Learning Run-time Safety Monitors for Machine Learning Components](https://mlsanity.com/paper/2406.16220/) (2024)
- [Revisiting Energy-Based Model for Out-of-Distribution Detection](https://mlsanity.com/paper/2412.03058/) (2024)
- [Benchmarking Out-of-Distribution Detection for Plankton Recognition: A Systematic Evaluation of Advanced Methods in Marine Ecological Monitoring](https://mlsanity.com/paper/2510.17179/) (2025)
- [Optimal Stability of KL Divergence under Gaussian Perturbations](https://mlsanity.com/paper/2604.11026/) (2026)

## Abstract (arXiv)

In the real world, a learning system could receive an input that is unlike anything it has seen during training. Unfortunately, out-of-distribution samples can lead to unpredictable behaviour. We need to know whether any given input belongs to the population distribution of the training/evaluation data to prevent unpredictable behaviour in deployed systems. A recent surge of interest in this problem has led to the development of sophisticated techniques in the deep learning literature. However, due to the absence of a standard problem definition or an exhaustive evaluation, it is not evident if we can rely on these methods. What makes this problem different from a typical supervised learning setting is that the distribution of outliers used in training may not be the same as the distribution of outliers encountered in the application. Classical approaches that learn inliers vs. outliers with only two datasets can yield optimistic results. We introduce OD-test, a three-dataset evaluation scheme as a more reliable strategy to assess progress on this problem. We present an exhaustive evaluation of a broad set of methods from related areas on image classification tasks. Contrary to the existing results, we show that for realistic applications of high-dimensional images the previous techniques have low accuracy and are not reliable in practice.