# Unsupervised Calibration under Covariate Shift

arXiv: https://arxiv.org/abs/2006.16405 · HTML: https://mlsanity.com/paper/2006.16405/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2020.

## Built on by (most cited first, 15 of 15)

- [DocILE Benchmark for Document Information Localization and Extraction](https://mlsanity.com/paper/2302.05658/) (2023)
- [Confidence Calibration for Domain Generalization under Covariate Shift](https://mlsanity.com/paper/2104.00742/) (2021)
- [On the Calibration of Massively Multilingual Language Models](https://mlsanity.com/paper/2210.12265/) (2022)
- [Reliability in Semantic Segmentation: Are We on the Right Track?](https://mlsanity.com/paper/2303.11298/) (2023)
- [Transfer Knowledge from Head to Tail: Uncertainty Calibration under Long-tailed Distribution](https://mlsanity.com/paper/2304.06537/) (2023)
- [Training on Test Data with Bayesian Adaptation for Covariate Shift](https://mlsanity.com/paper/2109.12746/) (2021)
- [Self-Calibrating Neural-Probabilistic Model for Authorship Verification Under Covariate Shift](https://mlsanity.com/paper/2106.11196/) (2021)
- [AUGCAL: Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images](https://mlsanity.com/paper/2312.06106/) (2023)
- [Confidence Calibration for Systems with Cascaded Predictive Modules](https://mlsanity.com/paper/2309.12510/) (2023)
- [Domain-adaptive and Subgroup-specific Cascaded Temperature Regression for Out-of-distribution Calibration](https://mlsanity.com/paper/2402.09204/) (2024)
- [PseudoCal: A Source-Free Approach to Unsupervised Uncertainty Calibration in Domain Adaptation](https://mlsanity.com/paper/2307.07489/) (2023)
- [The Over-Certainty Phenomenon in Modern Test-Time Adaptation Algorithms](https://mlsanity.com/paper/2404.16168/) (2024)
- [Calibration of Network Confidence for Unsupervised Domain Adaptation Using Estimated Accuracy](https://mlsanity.com/paper/2409.04241/) (2024)
- [Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift](https://mlsanity.com/paper/2605.21552/) (2026)
- [Confidence Calibration of Deep Learning Systems](https://mlsanity.com/paper/2608.12100/) (2026)

## Abstract (arXiv)

A probabilistic model is said to be calibrated if its predicted probabilities match the corresponding empirical frequencies. Calibration is important for uncertainty quantification and decision making in safety-critical applications. While calibration of classifiers has been widely studied, we find that calibration is brittle and can be easily lost under minimal covariate shifts. Existing techniques, including domain adaptation ones, primarily focus on prediction accuracy and do not guarantee calibration neither in theory nor in practice. In this work, we formally introduce the problem of calibration under domain shift, and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.