# Out-of-Distribution Detection Should Use Conformal Prediction (and Vice-versa?)

arXiv: https://arxiv.org/abs/2403.11532 · HTML: https://mlsanity.com/paper/2403.11532/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 27 of 27)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [ViM: Out-Of-Distribution with Virtual-logit Matching](https://mlsanity.com/paper/2203.10807/) (2022)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [Least Ambiguous Set-Valued Classifiers with Bounded Error Levels](https://mlsanity.com/paper/1609.00451/) (2016)
- [OpenOOD: Benchmarking Generalized Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07242/) (2022)
- [ADBench: Anomaly Detection Benchmark](https://mlsanity.com/paper/2206.09426/) (2022)
- [Extremely Simple Activation Shaping for Out-of-Distribution Detection](https://mlsanity.com/paper/2209.09858/) (2022)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Understanding Softmax Confidence and Uncertainty](https://mlsanity.com/paper/2106.04972/) (2021)
- [Prediction and outlier detection in classification problems](https://mlsanity.com/paper/1905.04396/) (2019)
- [A statistical framework for efficient out of distribution detection in deep neural networks](https://mlsanity.com/paper/2102.12967/) (2021)
- [iDECODe: In-distribution Equivariance for Conformal Out-of-distribution Detection](https://mlsanity.com/paper/2201.02331/) (2022)
- [Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers](https://mlsanity.com/paper/2208.11111/) (2022)

## Built on by (most cited first, 7 of 7)

- [Out-of-Distribution Detection for Safety Assurance of AI and Autonomous Systems](https://mlsanity.com/paper/2510.21254/) (2025)
- [Thinking in Groups: Permutation Tests Reveal Near-Out-of-Distribution](https://mlsanity.com/paper/2403.14058/) (2024)
- [Geometrically Constrained Outlier Synthesis](https://mlsanity.com/paper/2603.08413/) (2026)
- [Uncertainty-Aware Concept Bottleneck Models with Enhanced Interpretability](https://mlsanity.com/paper/2510.00773/) (2025)
- [General OOD Detection via Model-aware and Subspace-aware Variable Priority](https://mlsanity.com/paper/2512.13003/) (2025)
- [Softmax is not Enough (for Adaptive Conformal Classification)](https://mlsanity.com/paper/2602.19498/) (2026)
- [Beyond a Single Signal: SPECTREG2, A Unified MultiExpert Anomaly Detector for Unknown Unknowns](https://mlsanity.com/paper/2603.21160/) (2026)

## Abstract (arXiv)

Research on Out-Of-Distribution (OOD) detection focuses mainly on building scores that efficiently distinguish OOD data from In Distribution (ID) data. On the other hand, Conformal Prediction (CP) uses non-conformity scores to construct prediction sets with probabilistic coverage guarantees. In this work, we propose to use CP to better assess the efficiency of OOD scores. Specifically, we emphasize that in standard OOD benchmark settings, evaluation metrics can be overly optimistic due to the finite sample size of the test dataset. Based on the work of (Bates et al., 2022), we define new conformal AUROC and conformal FRP@TPR95 metrics, which are corrections that provide probabilistic conservativeness guarantees on the variability of these metrics. We show the effect of these corrections on two reference OOD and anomaly detection benchmarks, OpenOOD (Yang et al., 2022) and ADBench (Han et al., 2022). We also show that the benefits of using OOD together with CP apply the other way around by using OOD scores as non-conformity scores, which results in improving upon current CP methods. One of the key messages of these contributions is that since OOD is concerned with designing scores and CP with interpreting these scores, the two fields may be inherently intertwined.