mlsanityData to 14 Oct 2026

Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift

arXiv 2207.08977 ↗First posted 18 Jul 2022cs.LG stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

We often see undesirable tradeoffs in robust machine learning where out-of-distribution (OOD) accuracy is at odds with in-distribution (ID) accuracy: a robust classifier obtained via specialized techniques such as removing spurious features often has better OOD but worse ID accuracy compared to a standard classifier trained via ERM. In this paper, we find that ID-calibrated ensembles -- where we simply ensemble the standard and robust models after calibrating on only ID data -- outperforms prior state-of-the-art (based on self-training) on both ID and OOD accuracy. On eleven natural distribution shift datasets, ID-calibrated ensembles obtain the best of both worlds: strong ID accuracy and OOD accuracy. We analyze this method in stylized settings, and identify two important conditions for ensembles to perform well both ID and OOD: (1) we need to calibrate the standard and robust models (on ID data, because OOD data is unavailable), (2) OOD has no anticorrelated spurious features.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 47 of its 77 references that are in the map. First posted on arXiv 18 Jul 2022.

Builds on

47 of 77 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2019Invariant Risk Minimizationcited by 1,854
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  12. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
35 more
  1. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  2. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  3. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  4. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  5. 2020In Search of Lost Domain Generalizationcited by 1,054
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  7. 2018Robustness May Be at Odds with Accuracycited by 880
  8. 2021Robust fine-tuning of zero-shot modelscited by 797
  9. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  10. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  11. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  12. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
  13. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  14. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  15. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 364
  16. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  17. 2020Calibration of Pre-trained Transformerscited by 315
  18. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  19. 2019Verified Uncertainty Calibrationcited by 309
  20. 2019Are Labels Required for Improving Adversarial Robustness?cited by 163
  21. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  22. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 149
  23. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  24. 2021On Calibration and Out-of-domain Generalizationcited by 134
  25. 2020The iWildCam 2020 Competition Datasetcited by 86
  26. 2020Self-training Avoids Using Spurious Features Under Domain Shiftcited by 69
  27. 2020Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionatelycited by 52
  28. 2020In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustnesscited by 50
  29. 2021Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classificationcited by 39
  30. 2020Selective Classification Can Magnify Disparities Across Groupscited by 36
  31. 2020Meta-Learning for Few-Shot Land Cover Classificationcited by 34
  32. 2020Diverse Ensembles Improve Calibrationcited by 23
  33. 2014Binary Classifier Calibration: Non-parametric approachcited by 13
  34. 2021Should Ensemble Members Be Calibrated?cited by 11
  35. 2020Privacy Preserving Recalibration under Domain Shiftcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 10; 2024: 14; 2025: 3; 2026: 4.

34 papers.

Sort:

By area:

  1. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
  2. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  3. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  4. 2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
  5. 2023Spurious Feature Diversification Improves Out-of-distribution Generalizationcited by 26
  6. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  7. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  8. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  9. 2023Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothingcited by 18
  10. 2023Invariant Training 2D-3D Joint Hard Samples for Few-Shot Point Cloud Recognitioncited by 13
  11. 2023FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learningcited by 12
  12. 2023Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcementcited by 11
22 more
  1. 2024Classes Are Not Equal: An Empirical Study on Image Recognition Fairnesscited by 10
  2. 2024Controllable Prompt Tuning For Balancing Group Distributional Robustnesscited by 10
  3. 2025Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methodscited by 9
  4. 2024On the Benefits of Over-parameterization for Out-of-Distribution Generalizationcited by 7
  5. 2022Malign Overfitting: Interpolation Can Provably Preclude Invariancecited by 6
  6. 2024Robust Fine-tuning of Zero-shot Models via Variance Reductioncited by 6
  7. 2024LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Viewscited by 4
  8. 2025Where are we with calibration under dataset shift in image classification?cited by 4
  9. 2023Reweighted Mixup for Subpopulation Shiftcited by 2
  10. 2024Bridging Domains with Approximately Shared Featurescited by 2
  11. 2024Beyond Accuracy: On the Effects of Fine-tuning Towards Vision-Language Model's Prediction Rationalitycited by 2
  12. 2026Variance-Gated Ensembles: An Epistemic-Aware Framework for Uncertainty Estimationcited by 2
  13. 2023QuantProb: Generalizing Probabilities along with Predictions for a Pre-trained Classifiercited by 1
  14. 2024Assessing Uncertainty Estimation Methods for 3D Image Segmentation under Distribution Shiftscited by 1
  15. 2024Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancementcited by 1
  16. 2024Integrating uncertainty quantification into randomized smoothing based robustness guaranteescited by 1
  17. 2025TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Modelscited by 1
  18. 2026Missingness Bias Calibration in Feature Attribution Explanationscited by 1
  19. 2023Ethosight: A Reasoning-Guided Iterative Learning System for Nuanced Perception based on Joint-Embedding & Contextual Label Affinitycited by 0
  20. 2024Evaluation of Predictive Reliability to Foster Trust in Artificial Intelligence. A case study in Multiple Sclerosiscited by 0
  21. 2026MultiModal Fine-tuning with Synthetic Captionscited by 0
  22. 2026Uncertainty quantification for trustworthy deep learning: Methods and measurescited by 0