# Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers

arXiv: https://arxiv.org/abs/2208.11111 · HTML: https://mlsanity.com/paper/2208.11111/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 88 references that are in the map.
- First posted on arXiv 23 Aug 2022.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Least Ambiguous Set-Valued Classifiers with Bounded Error Levels](https://mlsanity.com/paper/1609.00451/) (2016)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Adversarially Learned One-Class Classifier for Novelty Detection](https://mlsanity.com/paper/1802.09088/) (2018)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging](https://mlsanity.com/paper/2202.05265/) (2022)
- [Distributional conformal prediction](https://mlsanity.com/paper/1909.07889/) (2019)
- [Nested conformal prediction and quantile out-of-bag ensemble methods](https://mlsanity.com/paper/1910.10562/) (2019)
- [Conformal Prediction using Conditional Histograms](https://mlsanity.com/paper/2105.08747/) (2021)
- [One-Class Classification: Taxonomy of Study and Review of Techniques](https://mlsanity.com/paper/1312.0049/) (2013)
- [Flexible distribution-free conditional predictive bands using density estimators](https://mlsanity.com/paper/1910.05575/) (2019)
- [Training Uncertainty-Aware Classifiers with Conformalized Deep Learning](https://mlsanity.com/paper/2205.05878/) (2022)
- [Adaptive, Distribution-Free Prediction Intervals for Deep Networks](https://mlsanity.com/paper/1905.10634/) (2019)
- [Predictive Inference Is Free with the Jackknife+-after-Bootstrap](https://mlsanity.com/paper/2002.09025/) (2020)
- [Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design](https://mlsanity.com/paper/2202.03613/) (2022)
- [Effects of Degradations on Deep Neural Network Architectures](https://mlsanity.com/paper/1807.10108/) (2018)
- [Real-time Out-of-distribution Detection in Learning-Enabled Cyber-Physical Systems](https://mlsanity.com/paper/2001.10494/) (2020)
- [Prediction and outlier detection in classification problems](https://mlsanity.com/paper/1905.04396/) (2019)
- [With Malice Towards None: Assessing Uncertainty via Equalized Coverage](https://mlsanity.com/paper/1908.05428/) (2019)
- [Cautious Deep Learning](https://mlsanity.com/paper/1805.09460/) (2018)
- [A statistical framework for efficient out of distribution detection in deep neural networks](https://mlsanity.com/paper/2102.12967/) (2021)

## Built on by (most cited first, 19 of 19)

- [Adaptive conformal classification with noisy labels](https://mlsanity.com/paper/2309.05092/) (2023)
- [Derandomized Novelty Detection with FDR Control via Conformal E-values](https://mlsanity.com/paper/2302.07294/) (2023)
- [Conformal inference is (almost) free for neural networks trained with early stopping](https://mlsanity.com/paper/2301.11556/) (2023)
- [Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage](https://mlsanity.com/paper/2403.03868/) (2024)
- [Does confidence calibration improve conformal prediction?](https://mlsanity.com/paper/2402.04344/) (2024)
- [Conformal Inference under High-Dimensional Covariate Shifts via Likelihood-Ratio Regularization](https://mlsanity.com/paper/2502.13030/) (2025)
- [Conformal Frequency Estimation using Discrete Sketched Data with Coverage for Distinct Queries](https://mlsanity.com/paper/2211.04612/) (2022)
- [Conformal link prediction for false discovery rate control](https://mlsanity.com/paper/2306.14693/) (2023)
- [Synthetic-Powered Predictive Inference](https://mlsanity.com/paper/2505.13432/) (2025)
- [SymmPI: Predictive Inference for Data with Group Symmetries](https://mlsanity.com/paper/2312.16160/) (2023)
- [Out-of-Distribution Detection Should Use Conformal Prediction (and Vice-versa?)](https://mlsanity.com/paper/2403.11532/) (2024)
- [Membership Inference Attacks with False Discovery Rate Control](https://mlsanity.com/paper/2508.07066/) (2025)
- [A distribution-free valid p-value for finite samples of bounded random variables](https://mlsanity.com/paper/2405.08975/) (2024)
- [On the Adversarial Robustness of Learning-based Conformal Novelty Detection](https://mlsanity.com/paper/2510.00463/) (2025)
- [General OOD Detection via Model-aware and Subspace-aware Variable Priority](https://mlsanity.com/paper/2512.13003/) (2025)
- [Reliable conformal novelty detection at the decision boundary](https://mlsanity.com/paper/2601.02610/) (2026)
- [ConformalHDC: Uncertainty-Aware Hyperdimensional Computing with Application to Neural Decoding](https://mlsanity.com/paper/2602.21446/) (2026)
- [Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference](https://mlsanity.com/paper/2605.20726/) (2026)
- [Conformal calibration and look-elsewhere effect in anomaly detection for new-physics searches](https://mlsanity.com/paper/2606.13780/) (2026)

## Abstract (arXiv)

This paper develops novel conformal methods to test whether a new observation was sampled from the same distribution as a reference set. Blending inductive and transductive conformal inference in an innovative way, the described methods can re-weight standard conformal p-values based on dependent side information from known out-of-distribution data in a principled way, and can automatically take advantage of the most powerful model from any collection of one-class and binary classifiers. The solution can be implemented either through sample splitting or via a novel transductive cross-validation+ scheme which may also be useful in other applications of conformal inference, due to tighter guarantees compared to existing cross-validation approaches. After studying false discovery rate control and power within a multiple testing framework with several possible outliers, the proposed solution is shown to outperform standard conformal p-values through simulations as well as applications to image recognition and tabular data.