# Confidence Calibration for Domain Generalization under Covariate Shift

arXiv: https://arxiv.org/abs/2104.00742 · HTML: https://mlsanity.com/paper/2104.00742/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Domain Generalization via Model-Agnostic Learning of Semantic Features](https://mlsanity.com/paper/1910.13580/) (2019)
- [A Survey of Unsupervised Deep Domain Adaptation](https://mlsanity.com/paper/1812.02849/) (2018)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Domain Generalization for Object Recognition with Multi-task Autoencoders](https://mlsanity.com/paper/1508.07680/) (2015)
- [Fixing the train-test resolution discrepancy](https://mlsanity.com/paper/1906.06423/) (2019)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Transferrable Prototypical Networks for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1904.11227/) (2019)
- [Domain Adaptation for Visual Applications: A Comprehensive Survey](https://mlsanity.com/paper/1702.05374/) (2017)
- [Unsupervised Domain Adaptation through Self-Supervision](https://mlsanity.com/paper/1909.11825/) (2019)
- [Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptation](https://mlsanity.com/paper/2003.00343/) (2020)
- [Transferable Calibration with Lower Bias and Variance in Domain Adaptation](https://mlsanity.com/paper/2007.08259/) (2020)
- [Reliable and Explainable Machine Learning Methods for Accelerated Material Discovery](https://mlsanity.com/paper/1901.02717/) (2019)
- [Unsupervised Calibration under Covariate Shift](https://mlsanity.com/paper/2006.16405/) (2020)
- [Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context](https://mlsanity.com/paper/2007.08911/) (2020)

## Built on by (most cited first, 27 of 27)

- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [Generalizable Cross-modality Medical Image Segmentation via Style Augmentation and Dual Normalization](https://mlsanity.com/paper/2112.11177/) (2021)
- [Towards Calibrated Robust Fine-Tuning of Vision-Language Models](https://mlsanity.com/paper/2311.01723/) (2023)
- [On Calibrating Semantic Segmentation Models: Analyses and An Algorithm](https://mlsanity.com/paper/2212.12053/) (2022)
- [Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation](https://mlsanity.com/paper/2302.01094/) (2023)
- [Generalizable Medical Image Segmentation via Random Amplitude Mixup and Domain-Specific Image Restoration](https://mlsanity.com/paper/2208.03901/) (2022)
- [Reliability in Semantic Segmentation: Are We on the Right Track?](https://mlsanity.com/paper/2303.11298/) (2023)
- [Beyond In-Domain Scenarios: Robust Density-Aware Calibration](https://mlsanity.com/paper/2302.05118/) (2023)
- [Transfer Knowledge from Head to Tail: Uncertainty Calibration under Long-tailed Distribution](https://mlsanity.com/paper/2304.06537/) (2023)
- [Invariant Shape Representation Learning For Image Classification](https://mlsanity.com/paper/2411.12201/) (2024)
- [Optimizing Calibration by Gaining Aware of Prediction Correctness](https://mlsanity.com/paper/2404.13016/) (2024)
- [Methods for Estimating and Improving Robustness of Language Models](https://mlsanity.com/paper/2206.08446/) (2022)
- [AUGCAL: Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images](https://mlsanity.com/paper/2312.06106/) (2023)
- [Confidence Calibration for Systems with Cascaded Predictive Modules](https://mlsanity.com/paper/2309.12510/) (2023)
- [Consistency-Guided Temperature Scaling Using Style and Content Information for Out-of-Domain Calibration](https://mlsanity.com/paper/2402.15019/) (2024)
- [Quantile Adaptive Temperature Scaling for Confidence Calibration](https://mlsanity.com/paper/2606.21749/) (2026)
- [Hidden Heterogeneity: When to Choose Similarity-Based Calibration](https://mlsanity.com/paper/2202.01840/) (2022)
- [Adaptive Calibrator Ensemble for Model Calibration under Distribution Shift](https://mlsanity.com/paper/2303.05331/) (2023)
- [Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Ranking](https://mlsanity.com/paper/2510.02956/) (2025)
- [DA-Cal: Towards Cross-Domain Calibration in Semantic Segmentation](https://mlsanity.com/paper/2602.20860/) (2026)
- [PseudoCal: A Source-Free Approach to Unsupervised Uncertainty Calibration in Domain Adaptation](https://mlsanity.com/paper/2307.07489/) (2023)
- [Dual-Channel Reliable Breast Ultrasound Image Classification Based on Explainable Attribution and Uncertainty Quantification](https://mlsanity.com/paper/2401.03664/) (2024)
- [The Over-Certainty Phenomenon in Modern Test-Time Adaptation Algorithms](https://mlsanity.com/paper/2404.16168/) (2024)
- [The Bayesian Confidence (BACON) Estimator for Deep Neural Networks](https://mlsanity.com/paper/2410.12604/) (2024)
- [CARD: Calibration via Agreement in Reverse Diffusion for Out-of-Domain MRI Segmentation](https://mlsanity.com/paper/2608.28681/) (2026)
- [Reliability-Aware Checkpoint Selection for Domain Generalization](https://mlsanity.com/paper/2609.39934/) (2026)

## Abstract (arXiv)

Existing calibration algorithms address the problem of covariate shift via unsupervised domain adaptation. However, these methods suffer from the following limitations: 1) they require unlabeled data from the target domain, which may not be available at the stage of calibration in real-world applications and 2) their performance depends heavily on the disparity between the distributions of the source and target domains. To address these two limitations, we present novel calibration solutions via domain generalization. Our core idea is to leverage multiple calibration domains to reduce the effective distribution disparity between the target and calibration domains for improved calibration transfer without needing any data from the target domain. We provide theoretical justification and empirical experimental results to demonstrate the effectiveness of our proposed algorithms. Compared against state-of-the-art calibration methods designed for domain adaptation, we observe a decrease of 8.86 percentage points in expected calibration error or, equivalently, an increase of 35 percentage points in improvement ratio for multi-class classification on the Office-Home dataset.