mlsanityData to 7 Oct 2026

Empirically Validating Conformal Prediction on Modern Vision Architectures Under Distribution Shift and Long-tailed Data

arXiv 2307.01088 ↗First posted 3 Jul 2023cs.LG cs.CV stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Conformal prediction has emerged as a rigorous means of providing deep learning models with reliable uncertainty estimates and safety guarantees. Yet, its performance is known to degrade under distribution shift and long-tailed class distributions, which are often present in real world applications. Here, we characterize the performance of several post-hoc and training-based conformal prediction methods under these settings, providing the first empirical evaluation on large-scale datasets and models. We show that across numerous conformal methods and neural network families, performance greatly degrades under distribution shifts violating safety guarantees. Similarly, we show that in long-tailed settings the guarantees are frequently violated on many classes. Understanding the limitations of these methods is necessary for deployment in real world and safety-critical applications.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 3 Jul 2023.

Builds on

32 of 43 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2016Concrete Problems in AI Safetycited by 1,582
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  10. 2021ImageNet-21K Pretraining for the Massescited by 607
  11. 2019Conformal Prediction Under Covariate Shiftcited by 511
  12. 2022DeiT III: Revenge of the ViTcited by 417
20 more
  1. 2020Classification with Valid and Adaptive Coveragecited by 390
  2. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  3. 2022Conformal prediction beyond exchangeabilitycited by 348
  4. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 270
  5. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  6. 2020Fast Differentiable Sorting and Rankingcited by 196
  7. 2019Causality matters in medical imagingcited by 193
  8. 2023Conformal Prediction With Conditional Guaranteescited by 146
  9. 2022Conformal Inference for Online Prediction with Arbitrary Distribution Shiftscited by 123
  10. 2021Learning Optimal Conformal Classifierscited by 108
  11. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85
  12. 2023Improved Online Conformal Prediction via Strongly Adaptive Online Learningcited by 76
  13. 2022Batch Multivalid Conformal Predictioncited by 66
  14. 2021Few-shot Conformal Prediction with Auxiliary Taskscited by 53
  15. 2022Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Setscited by 38
  16. 2018Distribution-Free Prediction Sets for Two-Layer Hierarchical Modelscited by 34
  17. 2022Predictive Inference with Feature Conformal Predictioncited by 33
  18. 2023Multi-Agent Reachability Calibration with Conformal Predictioncited by 23
  19. 2023Adaptive Conformal Prediction by Reweighting Nonconformity Scorecited by 16
  20. 2023Group conditional validity via multi-group learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2024TorchCP: A Python Library for Conformal Predictioncited by 9
  2. 2024Adapting Prediction Sets to Distribution Shifts Without Labelscited by 6
  3. 2025WQLCP: Weighted Adaptive Conformal Prediction for Robust Uncertainty Quantification Under Distribution Shiftscited by 5
  4. 2025Conformal Prediction Meets Long-tail Classificationcited by 5
  5. 2025Domain Adaptive Skin Lesion Classification via Conformal Ensemble of Vision Transformerscited by 0
  6. 2026Diagnosing Conformal Prediction Failures Under Distribution Shift: A COVID-19 Case Studycited by 0
  7. 2026Does Marginal Coverage Guarantee Class-Conditional Safety for Zero-Shot VLMs Under Shift?cited by 0