mlsanityData to 7 Oct 2026

Beyond AUROC & co. for evaluating out-of-distribution detection performance

arXiv 2306.14658 ↗First posted 26 Jun 2023cs.CV cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

While there has been a growing research interest in developing out-of-distribution (OOD) detection methods, there has been comparably little discussion around how these methods should be evaluated. Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs. In this work, we take a closer look at the go-to metrics for evaluating OOD detection, and question the approach of exclusively reducing OOD detection to a binary classification task with little consideration for the detection threshold. We illustrate the limitations of current metrics (AUROC & its friends) and propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples. Scripts and data are available at https://github.com/glhr/beyond-auroc

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

28 of 41 references.

References · 28

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  5. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  6. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  7. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  8. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  9. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  10. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  11. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  12. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 404
16 more
  1. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
  2. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  3. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 338
  4. 2020A Metric Learning Reality Checkcited by 317
  5. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  6. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
  7. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  8. 2018Reducing Network Agnostophobiacited by 165
  9. 2021A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challengescited by 154
  10. 2019Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancycited by 112
  11. 2021Semantically Coherent Out-of-Distribution Detectioncited by 107
  12. 2020On the Value of Out-of-Distribution Testing: An Example of Goodhart's Lawcited by 74
  13. 2021Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditionscited by 68
  14. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  15. 2021Out-of-Distribution Detection for Medical Applications: Guidelines for Practical Evaluationcited by 15
  16. 2022Out-of-Distribution Detection for Long-tailed and Fine-grained Skin Lesion Imagescited by 10

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 3.

7 papers.

  1. 2024Open-World Semantic Segmentation Including Class Similaritycited by 22
  2. 2024A noisy elephant in the room: Is your out-of-distribution detector robust to label noise?cited by 13
  3. 2024Open-World Panoptic Segmentationcited by 2
  4. 2025Improving Wildlife Out-of-Distribution Detection: Africas Big Fivecited by 2
  5. 2024Thinking in Groups: Permutation Tests Reveal Near-Out-of-Distributioncited by 1
  6. 2025Tumor-anchored deep feature random forests for out-of-distribution detection in lung cancer segmentationcited by 1
  7. 2025Out of Distribution Detection for Efficient Continual Learning in Quality Prediction for Arc Weldingcited by 0