# Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference

arXiv: https://arxiv.org/abs/2605.20726 · HTML: https://mlsanity.com/paper/2605.20726/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 20 May 2026.

## Builds on (most cited first, 20 of 20)

- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Selection by Prediction with Conformal p-values](https://mlsanity.com/paper/2210.01408/) (2022)
- [Model-free selective inference under covariate shift via weighted conformal p-values](https://mlsanity.com/paper/2307.09291/) (2023)
- [Transductive conformal inference with adaptive scores](https://mlsanity.com/paper/2310.18108/) (2023)
- [Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers](https://mlsanity.com/paper/2208.11111/) (2022)
- [Adaptive novelty detection with false discovery rate guarantee](https://mlsanity.com/paper/2208.06685/) (2022)
- [Universal distribution of the empirical coverage in split conformal prediction](https://mlsanity.com/paper/2303.02770/) (2023)
- [Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimization](https://mlsanity.com/paper/2411.17983/) (2024)
- [Multivariate Conformal Selection](https://mlsanity.com/paper/2505.00917/) (2025)
- [Diversifying Conformal Selections](https://mlsanity.com/paper/2506.16229/) (2025)
- [Permutation-based simultaneous confidence bounds for the false discovery proportion](https://mlsanity.com/paper/1808.05528/) (2018)
- [Simultaneous false discovery proportion bounds via knockoffs and closed testing](https://mlsanity.com/paper/2212.12822/) (2022)
- [Global envelope tests for spatial processes](https://mlsanity.com/paper/1307.0239/) (2013)
- [Higher criticism: p-values and criticism](https://mlsanity.com/paper/1411.1437/) (2014)
- [Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing](https://mlsanity.com/paper/1611.06739/) (2016)
- [Permutation-Based True Discovery Guarantee by Sum Tests](https://mlsanity.com/paper/2102.11759/) (2021)
- [Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions](https://mlsanity.com/paper/1901.04885/) (2019)
- [A central limit theorem for the Benjamini-Hochberg false discovery proportion under a factor model](https://mlsanity.com/paper/2104.08687/) (2021)
- [Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables](https://mlsanity.com/paper/2604.11305/) (2026)

## Abstract (arXiv)

Modern applications of conformal inference to multiple testing problems, such as outlier detection and candidate selection, often involve selecting test samples whose conformal p-values fall below a threshold. The quality of such methods is often measured by the false discovery proportion (FDP), defined as the fraction of incorrect selections. Existing approaches typically control the expected value of the FDP, using methods such as the Benjamini-Hochberg procedure. This approach fails to provide high-probability bounds on the realized false discovery proportion and invalidates statistical guarantees if the rejection threshold is selected after inspecting the data. This paper establishes finite-sample, distribution-free upper bounds on the FDP that hold simultaneously over all possible rejection thresholds, enabling arbitrary post hoc selection of the threshold. Simultaneous validity is achieved by constructing a high-probability envelope for the empirical distribution function of null conformal p-values by sampling from their joint distribution. Furthermore, our framework allows practitioners to modulate the envelope's shape, thereby producing tight bounds in rejection regions of primary interest. We use this flexible approach to derive simultaneous FDP upper bounds for both outlier detection and conformal selection. We demonstrate through synthetic and real-data experiments that the resulting bounds are both valid and substantially less conservative than those derived from existing approaches.