mlsanityData to 14 Oct 2026

Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better

arXiv 2601.21455 ↗First posted 29 Jan 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Conformal prediction(CP) has become a cornerstone of distribution-free uncertainty quantification, conventionally evaluated by its coverage and interval length. This work critically examines the sufficiency of these standard metrics. We demonstrate that the interval length might be deceptively improved through a counter-intuitive approach termed Prejudicial Trick(PT), while the coverage remains valid. Specifically, for any given test sample, PT probabilistically returns an interval, which is either null or constructed using an adjusted confidence level, thereby preserving marginal coverage. While PT potentially yields a deceptively lower interval length, it introduces practical vulnerabilities: the same input can yield completely different prediction intervals across repeated runs of the algorithm. We formally derive the conditions under which PT achieves these misleading improvements and provide extensive empirical evidence across various regression and classification tasks. Furthermore, we introduce a new metric interval stability which helps detect whether a new CP method implicitly improves the length based on such PT-like techniques. Code is available at https://github.com/benben-cd/PT-Conformal-Prediction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 68 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

44 of 68 references.

References · 44

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  3. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  4. 2019Conformalized Quantile Regressioncited by 629
  5. 2019Conformal Prediction Under Covariate Shiftcited by 519
  6. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  7. 2020Classification with Valid and Adaptive Coveragecited by 396
  8. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  9. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 366
  10. 2022Conformal prediction beyond exchangeabilitycited by 355
  11. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 272
  12. 2023Conformal Prediction With Conditional Guaranteescited by 152
32 more
  1. 2020Conformal Inference of Counterfactuals and Individual Treatment Effectscited by 151
  2. 2021Localized Conformal Prediction: A Generalized Inference Framework for Conformal Predictioncited by 114
  3. 2022Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imagingcited by 107
  4. 2013A Conformal Prediction Approach to Explore Functional Datacited by 86
  5. 2019Flexible distribution-free conditional predictive bands using density estimatorscited by 64
  6. 2021Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approachcited by 62
  7. 2014Regression Conformal Prediction with Nearest Neighbourscited by 59
  8. 2021Conformalized Survival Analysiscited by 53
  9. 2020Individual Calibration with Randomized Forecastingcited by 50
  10. 2021Improving Conditional Coverage via Orthogonal Quantile Regressioncited by 48
  11. 2023Learning Robust Statistics for Simulation-based Inference under Model Misspecificationcited by 48
  12. 2023Conformal prediction with local weights: randomization enables local guaranteescited by 48
  13. 2024Length Optimization in Conformal Predictioncited by 39
  14. 2016Criteria of efficiency for conformal predictioncited by 36
  15. 2022Predictive Inference with Feature Conformal Predictioncited by 34
  16. 2022Conformal Prediction Sets with Limited False Positivescited by 28
  17. 2022Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecastingcited by 27
  18. 2024Boosted Conformal Prediction Intervalscited by 27
  19. 2022Efficient and Differentiable Conformal Prediction with General Function Classescited by 26
  20. 2022Split Localized Conformal Predictioncited by 23
  21. 2019Variational Bayes under Model Misspecificationcited by 22
  22. 2023Improving Adaptive Conformal Prediction Using Self-Supervised Learningcited by 17
  23. 2024On Temperature Scaling and Conformal Prediction of Deep Classifierscited by 13
  24. 2024Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labelingcited by 12
  25. 2024Conformal Inductive Graph Neural Networkscited by 10
  26. 2021T-SCI: A Two-Stage Conformal Inference Algorithm with Guaranteed Coverage for Cox-MLPcited by 8
  27. 2023Conformalized Unconditional Quantile Regressioncited by 8
  28. 2022Conformal Prediction Bands for Two-Dimensional Functional Time Seriescited by 7
  29. 2024Multi-label Classification under Uncertainty: A Tree-based Conformal Prediction Approachcited by 7
  30. 2020Uncertainty Prediction for Deep Sequential Regression Using Meta Modelscited by 6
  31. 2024Two-stage Risk Control with Application to Ranked Retrievalcited by 6
  32. 2024Statistically Optimal Uncertainty Quantification for Expensive Black-Box Modelscited by 1

Built on by

Not cited yet.