# Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptation

arXiv: https://arxiv.org/abs/2003.00343 · HTML: https://mlsanity.com/paper/2003.00343/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Feb 2020.

## Built on by (most cited first, 30 of 43)

- [PAC Prediction Sets Under Covariate Shift](https://mlsanity.com/paper/2106.09848/) (2021)
- [Robust Calibration with Multi-domain Temperature Scaling](https://mlsanity.com/paper/2206.02757/) (2022)
- [Towards Calibrated Robust Fine-Tuning of Vision-Language Models](https://mlsanity.com/paper/2311.01723/) (2023)
- [Confidence Calibration for Domain Generalization under Covariate Shift](https://mlsanity.com/paper/2104.00742/) (2021)
- [Source-Free Progressive Graph Learning for Open-Set Domain Adaptation](https://mlsanity.com/paper/2202.06174/) (2022)
- [On Calibrating Semantic Segmentation Models: Analyses and An Algorithm](https://mlsanity.com/paper/2212.12053/) (2022)
- [Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2403.11491/) (2024)
- [Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities](https://mlsanity.com/paper/2602.05073/) (2026)
- [Estimating and Explaining Model Performance When Both Covariates and Labels Shift](https://mlsanity.com/paper/2209.08436/) (2022)
- [On the Calibration of Massively Multilingual Language Models](https://mlsanity.com/paper/2210.12265/) (2022)
- [Reliability in Semantic Segmentation: Are We on the Right Track?](https://mlsanity.com/paper/2303.11298/) (2023)
- [Transfer Knowledge from Head to Tail: Uncertainty Calibration under Long-tailed Distribution](https://mlsanity.com/paper/2304.06537/) (2023)
- [Cal-SFDA: Source-Free Domain-adaptive Semantic Segmentation with Differentiable Expected Calibration Error](https://mlsanity.com/paper/2308.03003/) (2023)
- [Training on Test Data with Bayesian Adaptation for Covariate Shift](https://mlsanity.com/paper/2109.12746/) (2021)
- [Learning Unbiased Transferability for Domain Adaptation by Uncertainty Modeling](https://mlsanity.com/paper/2206.01319/) (2022)
- [Beyond Confidence: Adaptive Abstention in Dual-Threshold Conformal Prediction for Autonomous System Perception](https://mlsanity.com/paper/2502.07255/) (2025)
- [Fast and Accurate Importance Weighting for Correcting Sample Bias](https://mlsanity.com/paper/2209.04215/) (2022)
- [Weight Clipping for Robust Conformal Inference under Unbounded Covariate Shifts](https://mlsanity.com/paper/2605.02072/) (2026)
- [Scalable Causal Domain Adaptation](https://mlsanity.com/paper/2103.00139/) (2021)
- [Fair Text Classification via Transferable Representations](https://mlsanity.com/paper/2503.07691/) (2025)
- [On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift](https://mlsanity.com/paper/2505.11380/) (2025)
- [Frustratingly Easy Uncertainty Estimation for Distribution Shift](https://mlsanity.com/paper/2106.03762/) (2021)
- [Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking](https://mlsanity.com/paper/2205.09638/) (2022)
- [IW-GAE: Importance Weighted Group Accuracy Estimation for Improved Calibration and Model Selection in Unsupervised Domain Adaptation](https://mlsanity.com/paper/2310.10611/) (2023)
- [Domain-adaptive and Subgroup-specific Cascaded Temperature Regression for Out-of-distribution Calibration](https://mlsanity.com/paper/2402.09204/) (2024)
- [Harnessing the Power of Vicinity-Informed Analysis for Classification under Covariate Shift](https://mlsanity.com/paper/2405.16906/) (2024)
- [VideoSAM: A Large Vision Foundation Model for High-Speed Video Segmentation](https://mlsanity.com/paper/2410.21304/) (2024)
- [Towards Reliable Test-Time Adaptation: Style Invariance as a Correctness Likelihood](https://mlsanity.com/paper/2512.07390/) (2025)
- [Calibration without Ground Truth](https://mlsanity.com/paper/2601.19862/) (2026)
- [Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift](https://mlsanity.com/paper/2602.14913/) (2026)

## Abstract (arXiv)

Reliable uncertainty estimates are an important tool for helping autonomous agents or human decision makers understand and leverage predictive models. However, existing approaches to estimating uncertainty largely ignore the possibility of covariate shift--i.e., where the real-world data distribution may differ from the training distribution. As a consequence, existing algorithms can overestimate certainty, possibly yielding a false sense of confidence in the predictive model. We propose an algorithm for calibrating predictions that accounts for the possibility of covariate shift, given labeled examples from the training distribution and unlabeled examples from the real-world distribution. Our algorithm uses importance weighting to correct for the shift from the training to the real-world distribution. However, importance weighting relies on the training and real-world distributions to be sufficiently close. Building on ideas from domain adaptation, we additionally learn a feature map that tries to equalize these two distributions. In an empirical evaluation, we show that our proposed approach outperforms existing approaches to calibrated prediction when there is covariate shift.