mlsanityData to 7 Oct 2026

Test-time Recalibration of Conformal Predictors Under Distribution Shift Based on Unlabeled Examples

arXiv 2210.04166 ↗First posted 9 Oct 2022cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Modern image classifiers are very accurate, but the predictions come without uncertainty estimates. Conformal predictors provide uncertainty estimates by computing a set of classes containing the correct class with a user-specified probability based on the classifier's probability estimates. To provide such sets, conformal predictors often estimate a cutoff threshold for the probability estimates based on a calibration set. Conformal predictors guarantee reliability only when the calibration set is from the same distribution as the test set. Therefore, conformal predictors need to be recalibrated for new distributions. However, in practice, labeled data from new distributions is rarely available, making calibration infeasible. In this work, we consider the problem of predicting the cutoff threshold for a new distribution based on unlabeled examples. While it is impossible in general to guarantee reliability when calibrating based on unlabeled examples, we propose a method that provides excellent uncertainty estimates under natural distribution shifts, and provably works for a specific model of a distribution shift.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 23 of its 27 references that are in the map. First posted on arXiv 9 Oct 2022.

Builds on

23 of 27 references.

References · 23

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2016Distribution-Free Predictive Inference For Regressioncited by 657
  8. 2019Conformal Prediction Under Covariate Shiftcited by 511
  9. 2020Classification with Valid and Adaptive Coveragecited by 390
  10. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  11. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  12. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 270
11 more
  1. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  2. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  3. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  4. 2021Assessing Generalization of SGD via Disagreementcited by 116
  5. 2021Predicting with Confidence on Unseen Distributionscited by 116
  6. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  7. 2021Distribution-free uncertainty quantification for classification under label shiftcited by 99
  8. 2020Robust Validation: Confident Predictions Even When Distributions Shiftcited by 82
  9. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  10. 2021Mandoline: Model Evaluation under Distribution Shiftcited by 55
  11. 2021PAC Prediction Sets Under Covariate Shiftcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025CPATTA: Conformal Supervision Allocation For Active Test-Time Adaptationcited by 1
  2. 2026The Label Complexity of Useful Class-Conditional Prediction Sets under Distribution Shiftcited by 0
  3. 2026Self-Poisoning in Adaptive Out-of-Distribution Detection: A Sharp-Threshold Theory and Certified Label-Free Calibrationcited by 0