# Combining Ensembles and Data Augmentation can Harm your Calibration

arXiv: https://arxiv.org/abs/2010.09875 · HTML: https://mlsanity.com/paper/2010.09875/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 30 of 44)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [When and How Mixup Improves Calibration](https://mlsanity.com/paper/2102.06289/) (2021)
- [Calibrating Language Models with Adaptive Temperature Scaling](https://mlsanity.com/paper/2409.19817/) (2024)
- [Rethinking Confidence Calibration for Failure Prediction](https://mlsanity.com/paper/2303.02970/) (2023)
- [Correlated Input-Dependent Label Noise in Large-Scale Image Classification](https://mlsanity.com/paper/2105.10305/) (2021)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness](https://mlsanity.com/paper/2206.14502/) (2022)
- [Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning](https://mlsanity.com/paper/2111.03577/) (2021)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [The Calibration Generalization Gap](https://mlsanity.com/paper/2210.01964/) (2022)
- [URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates](https://mlsanity.com/paper/2307.03810/) (2023)
- [Revisiting Confidence Estimation: Towards Reliable Failure Prediction](https://mlsanity.com/paper/2403.02886/) (2024)
- [Posterior Uncertainty Quantification in Neural Networks using Data Augmentation](https://mlsanity.com/paper/2403.12729/) (2024)
- [Curved Representation Space of Vision Transformers](https://mlsanity.com/paper/2210.05742/) (2022)
- [Reassessing How to Compare and Improve the Calibration of Machine Learning Models](https://mlsanity.com/paper/2406.04068/) (2024)
- [Fill In The Gaps: Model Calibration and Generalization with Synthetic Data](https://mlsanity.com/paper/2410.10864/) (2024)
- [On the Limitations of Temperature Scaling for Distributions with Overlaps](https://mlsanity.com/paper/2306.00740/) (2023)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [Ex uno plures: Splitting One Model into an Ensemble of Subnetworks](https://mlsanity.com/paper/2106.04767/) (2021)
- [Self-Aware Trajectory Prediction for Safe Autonomous Driving](https://mlsanity.com/paper/2305.09147/) (2023)
- [Tailoring Mixup to Data for Calibration](https://mlsanity.com/paper/2311.01434/) (2023)
- [RegMixMatch: Optimizing Mixup Utilization in Semi-Supervised Learning](https://mlsanity.com/paper/2412.10741/) (2024)
- [Generalization Gap in Data Augmentation: Insights from Illumination](https://mlsanity.com/paper/2404.07514/) (2024)
- [Are Data Augmentation Methods in Named Entity Recognition Applicable for Uncertainty Estimation?](https://mlsanity.com/paper/2407.02062/) (2024)
- [Contextual Similarity Distillation: Ensemble Uncertainties with a Single Model](https://mlsanity.com/paper/2503.11339/) (2025)

## Abstract (arXiv)

Ensemble methods which average over multiple neural network predictions are a simple approach to improve a model's calibration and robustness. Similarly, data augmentation techniques, which encode prior information in the form of invariant feature transformations, are effective for improving calibration and robustness. In this paper, we show a surprising pathology: combining ensembles and data augmentation can harm model calibration. This leads to a trade-off in practice, whereby improved accuracy by combining the two techniques comes at the expense of calibration. On the other hand, selecting only one of the techniques ensures good uncertainty estimates at the expense of accuracy. We investigate this pathology and identify a compounding under-confidence among methods which marginalize over sets of weights and data augmentation techniques which soften labels. Finally, we propose a simple correction, achieving the best of both worlds with significant accuracy and calibration gains over using only ensembles or data augmentation individually. Applying the correction produces new state-of-the art in uncertainty calibration across CIFAR-10, CIFAR-100, and ImageNet.