# Online Selective Conformal Prediction with Asymmetric Rules: A Permutation Test Approach

arXiv: https://arxiv.org/abs/2602.10018 · HTML: https://mlsanity.com/paper/2602.10018/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Theoretical Foundations of Conformal Prediction](https://mlsanity.com/paper/2411.11824/) (2024)
- [Exact post-selection inference, with application to the lasso](https://mlsanity.com/paper/1311.6238/) (2013)
- [Hypothesis testing with e-values](https://mlsanity.com/paper/2410.23614/) (2024)
- [Selection by Prediction with Conformal p-values](https://mlsanity.com/paper/2210.01408/) (2022)
- [Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees](https://mlsanity.com/paper/2405.10301/) (2024)
- [Model-free selective inference under covariate shift via weighted conformal p-values](https://mlsanity.com/paper/2307.09291/) (2023)
- [Post-selection inference for e-value based confidence intervals](https://mlsanity.com/paper/2203.12572/) (2022)
- [Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage](https://mlsanity.com/paper/2403.03868/) (2024)
- [SAFFRON: an adaptive algorithm for online control of the false discovery rate](https://mlsanity.com/paper/1802.09098/) (2018)
- [ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls](https://mlsanity.com/paper/1905.11465/) (2019)
- [Selective conformal inference with false coverage-statement rate control](https://mlsanity.com/paper/2301.00584/) (2023)
- [Unifying approach to selective inference with applications to cross-validation](https://mlsanity.com/paper/1703.06559/) (2017)
- [CAP: A General Algorithm for Online Selective Conformal Prediction with FCR Control](https://mlsanity.com/paper/2403.07728/) (2024)
- [Selecting informative conformal prediction sets with false coverage rate control](https://mlsanity.com/paper/2403.12295/) (2024)
- [On Online Control of False Discovery Rate](https://mlsanity.com/paper/1502.06197/) (2015)
- [Post-selection inference for L1-penalized likelihood models](https://mlsanity.com/paper/1602.07358/) (2016)
- [Permutation tests using arbitrary permutation distributions](https://mlsanity.com/paper/2204.13581/) (2022)
- [Post-selection point and interval estimation of signal sizes in Gaussian samples](https://mlsanity.com/paper/1405.3340/) (2014)
- [Online Selective Conformal Prediction: Errors and Solutions](https://mlsanity.com/paper/2503.16809/) (2025)
- [Online selective conformal inference: adaptive scores, convergence rates and optimality](https://mlsanity.com/paper/2508.10336/) (2025)

## Abstract (arXiv)

Selective conformal prediction aims to construct prediction sets with valid coverage for a test unit conditional on it being selected by a data-driven mechanism. While existing methods in the offline setting handle any selection mechanism that is permutation invariant to the labeled data, their extension to the online setting -- where data arrives sequentially and later decisions depend on earlier ones -- is challenged by the fact that the selection mechanism is naturally asymmetric. As such, existing methods only address a limited collection of selection mechanisms. In this paper, we propose PErmutation-based Mondrian Conformal Inference (PEMI), a general permutation-based framework for selective conformal prediction with arbitrary asymmetric selection rules. Motivated by full and Mondrian conformal prediction, PEMI identifies all permutations of the observed data (or a Monte-Carlo subset thereof) that lead to the same selection event, and calibrates a prediction set using conformity scores over this selection-preserving reference set. Under standard exchangeability conditions, our prediction sets achieve finite-sample exact selection-conditional coverage for any asymmetric selection mechanism and any prediction model. PEMI naturally incorporates additional offline labeled data, extends to selection mechanisms with multiple test samples, and achieves FCR control with fine-grained selection taxonomies. We further work out several efficient instantiations for commonly-used online selection rules, including covariate-based rules, conformal p/e-values-based procedures, and selection based on earlier outcomes. Finally, we demonstrate the efficacy of our methods across various selection rules on a real drug discovery dataset and investigate their performance via simulations.