# Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better

arXiv: https://arxiv.org/abs/2601.21455 · HTML: https://mlsanity.com/paper/2601.21455/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 68 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 30 of 44)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Least Ambiguous Set-Valued Classifiers with Bounded Error Levels](https://mlsanity.com/paper/1609.00451/) (2016)
- [Conformal Prediction With Conditional Guarantees](https://mlsanity.com/paper/2305.12616/) (2023)
- [Conformal Inference of Counterfactuals and Individual Treatment Effects](https://mlsanity.com/paper/2006.06138/) (2020)
- [Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction](https://mlsanity.com/paper/2106.08460/) (2021)
- [Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging](https://mlsanity.com/paper/2202.05265/) (2022)
- [A Conformal Prediction Approach to Explore Functional Data](https://mlsanity.com/paper/1302.6452/) (2013)
- [Flexible distribution-free conditional predictive bands using density estimators](https://mlsanity.com/paper/1910.05575/) (2019)
- [Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach](https://mlsanity.com/paper/2111.12161/) (2021)
- [Regression Conformal Prediction with Nearest Neighbours](https://mlsanity.com/paper/1401.3880/) (2014)
- [Conformalized Survival Analysis](https://mlsanity.com/paper/2103.09763/) (2021)
- [Individual Calibration with Randomized Forecasting](https://mlsanity.com/paper/2006.10288/) (2020)
- [Improving Conditional Coverage via Orthogonal Quantile Regression](https://mlsanity.com/paper/2106.00394/) (2021)
- [Learning Robust Statistics for Simulation-based Inference under Model Misspecification](https://mlsanity.com/paper/2305.15871/) (2023)
- [Conformal prediction with local weights: randomization enables local guarantees](https://mlsanity.com/paper/2310.07850/) (2023)
- [Length Optimization in Conformal Prediction](https://mlsanity.com/paper/2406.18814/) (2024)
- [Criteria of efficiency for conformal prediction](https://mlsanity.com/paper/1603.04416/) (2016)
- [Predictive Inference with Feature Conformal Prediction](https://mlsanity.com/paper/2210.00173/) (2022)
- [Conformal Prediction Sets with Limited False Positives](https://mlsanity.com/paper/2202.07650/) (2022)
- [Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2202.08756/) (2022)
- [Boosted Conformal Prediction Intervals](https://mlsanity.com/paper/2406.07449/) (2024)

## Abstract (arXiv)

Conformal prediction(CP) has become a cornerstone of distribution-free uncertainty quantification, conventionally evaluated by its coverage and interval length. This work critically examines the sufficiency of these standard metrics. We demonstrate that the interval length might be deceptively improved through a counter-intuitive approach termed Prejudicial Trick(PT), while the coverage remains valid. Specifically, for any given test sample, PT probabilistically returns an interval, which is either null or constructed using an adjusted confidence level, thereby preserving marginal coverage. While PT potentially yields a deceptively lower interval length, it introduces practical vulnerabilities: the same input can yield completely different prediction intervals across repeated runs of the algorithm. We formally derive the conditions under which PT achieves these misleading improvements and provide extensive empirical evidence across various regression and classification tasks. Furthermore, we introduce a new metric interval stability which helps detect whether a new CP method implicitly improves the length based on such PT-like techniques. Code is available at https://github.com/benben-cd/PT-Conformal-Prediction.