# Conformal prediction after data-dependent model selection

arXiv: https://arxiv.org/abs/2408.07066 · HTML: https://mlsanity.com/paper/2408.07066/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2024.

## Built on by (most cited first, 14 of 14)

- [Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimization](https://mlsanity.com/paper/2411.17983/) (2024)
- [Improving the statistical efficiency of cross-conformal prediction](https://mlsanity.com/paper/2503.01495/) (2025)
- [Singleton-Optimized Conformal Prediction](https://mlsanity.com/paper/2509.24095/) (2025)
- [Uncertainty-Guided Model Selection for Tabular Foundation Models in Biomolecule Efficacy Prediction](https://mlsanity.com/paper/2510.02476/) (2025)
- [Learning Polyhedral Conformal Sets for Robust Optimization](https://mlsanity.com/paper/2605.08506/) (2026)
- [Data-light Uncertainty Set Merging with Admissibility](https://mlsanity.com/paper/2410.12201/) (2024)
- [Joint Registration and Conformal Prediction for Partially Observed Functional Data](https://mlsanity.com/paper/2502.15000/) (2025)
- [Improving Coverage in Combined Prediction Sets with Weighted p-values](https://mlsanity.com/paper/2505.11785/) (2025)
- [Uncertainty Quantification Via the Posterior Predictive Variance](https://mlsanity.com/paper/2603.19804/) (2026)
- [Conformalized Super Learner](https://mlsanity.com/paper/2604.22391/) (2026)
- [Who Wins Where? Conformal Model Comparison for Local Superiority](https://mlsanity.com/paper/2607.29053/) (2026)
- [Multi-source conformal prediction: leveraging heterogeneity via localization](https://mlsanity.com/paper/2609.14531/) (2026)
- [Calibration Count Reuse: Validity Does Not Determine Efficiency](https://mlsanity.com/paper/2609.25138/) (2026)
- [Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection](https://mlsanity.com/paper/2609.28522/) (2026)

## Abstract (arXiv)

Given a family of pretrained models and a hold-out set, how can we construct a valid conformal prediction set while selecting a model that minimizes the width of the set? If we use the same hold-out data set both to select a model (the model that yields the smallest conformal prediction sets) and then to construct a conformal prediction set based on that selected model, we suffer a loss of coverage due to selection bias. Alternatively, we could further split the data to perform selection and calibration separately, but this comes at a steep cost if the size of the dataset is limited. In this paper, we address the challenge of constructing a valid prediction set after data-dependent model selection -- commonly, selecting the model that minimizes the width of the resulting prediction sets. Our novel methods can be implemented efficiently and admit finite-sample validity guarantees without invoking additional sample-splitting. We show that our methods yield prediction sets with asymptotically optimal width under certain notion of regularity for the model class. The improvement in the width of the prediction sets constructed by our methods are further demonstrated through applications to synthetic datasets in various settings and a real data example.