# Beyond AUROC & co. for evaluating out-of-distribution detection performance

arXiv: https://arxiv.org/abs/2306.14658 · HTML: https://mlsanity.com/paper/2306.14658/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 28 of 28)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [ViM: Out-Of-Distribution with Virtual-logit Matching](https://mlsanity.com/paper/2203.10807/) (2022)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [Exploring the Limits of Out-of-Distribution Detection](https://mlsanity.com/paper/2106.03004/) (2021)
- [OpenOOD: Benchmarking Generalized Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07242/) (2022)
- [MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space](https://mlsanity.com/paper/2105.01879/) (2021)
- [Reducing Network Agnostophobia](https://mlsanity.com/paper/1811.04110/) (2018)
- [A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges](https://mlsanity.com/paper/2110.14051/) (2021)
- [Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancy](https://mlsanity.com/paper/1908.04951/) (2019)
- [Semantically Coherent Out-of-Distribution Detection](https://mlsanity.com/paper/2108.11941/) (2021)
- [On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law](https://mlsanity.com/paper/2005.09241/) (2020)
- [Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions](https://mlsanity.com/paper/2104.03829/) (2021)
- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [Out-of-Distribution Detection for Medical Applications: Guidelines for Practical Evaluation](https://mlsanity.com/paper/2109.14885/) (2021)
- [Out-of-Distribution Detection for Long-tailed and Fine-grained Skin Lesion Images](https://mlsanity.com/paper/2206.15186/) (2022)

## Built on by (most cited first, 7 of 7)

- [Open-World Semantic Segmentation Including Class Similarity](https://mlsanity.com/paper/2403.07532/) (2024)
- [A noisy elephant in the room: Is your out-of-distribution detector robust to label noise?](https://mlsanity.com/paper/2404.01775/) (2024)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [Improving Wildlife Out-of-Distribution Detection: Africas Big Five](https://mlsanity.com/paper/2506.06719/) (2025)
- [Thinking in Groups: Permutation Tests Reveal Near-Out-of-Distribution](https://mlsanity.com/paper/2403.14058/) (2024)
- [Tumor-anchored deep feature random forests for out-of-distribution detection in lung cancer segmentation](https://mlsanity.com/paper/2512.08216/) (2025)
- [Out of Distribution Detection for Efficient Continual Learning in Quality Prediction for Arc Welding](https://mlsanity.com/paper/2508.16832/) (2025)

## Abstract (arXiv)

While there has been a growing research interest in developing out-of-distribution (OOD) detection methods, there has been comparably little discussion around how these methods should be evaluated. Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs. In this work, we take a closer look at the go-to metrics for evaluating OOD detection, and question the approach of exclusively reducing OOD detection to a binary classification task with little consideration for the detection threshold. We illustrate the limitations of current metrics (AUROC & its friends) and propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples. Scripts and data are available at https://github.com/glhr/beyond-auroc