# When and How Mixup Improves Calibration

arXiv: https://arxiv.org/abs/2102.06289 · HTML: https://mlsanity.com/paper/2102.06289/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 53 references that are in the map.
- First posted on arXiv 11 Feb 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [Adversarially Robust Generalization Requires More Data](https://mlsanity.com/paper/1804.11285/) (2018)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup](https://mlsanity.com/paper/2009.06962/) (2020)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [How Does Mixup Help With Robustness and Generalization?](https://mlsanity.com/paper/2010.04819/) (2020)
- [MixUp as Locally Linear Out-Of-Manifold Regularization](https://mlsanity.com/paper/1809.02499/) (2018)
- [Individual Calibration with Randomized Forecasting](https://mlsanity.com/paper/2006.10288/) (2020)
- [Combining Ensembles and Data Augmentation can Harm your Calibration](https://mlsanity.com/paper/2010.09875/) (2020)
- [Towards Trustworthy Predictions from Deep Neural Networks with Fast Adversarial Calibration](https://mlsanity.com/paper/2012.10923/) (2020)
- [Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification](https://mlsanity.com/paper/2006.16384/) (2020)
- [Representation via Representations: Domain Generalization via Adversarially Learned Invariant Representations](https://mlsanity.com/paper/2006.11478/) (2020)
- [Improving Adversarial Robustness via Unlabeled Out-of-Domain Data](https://mlsanity.com/paper/2006.08476/) (2020)
- [Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift](https://mlsanity.com/paper/2006.14988/) (2020)
- [Interpreting Robust Optimization via Adversarial Influence Functions](https://mlsanity.com/paper/2010.01247/) (2020)

## Built on by (most cited first, 30 of 60)

- [Analyzing and Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2310.00754/) (2023)
- [Improving Out-of-Distribution Robustness via Selective Augmentation](https://mlsanity.com/paper/2201.00299/) (2022)
- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Discover and Cure: Concept-aware Mitigation of Spurious Correlation](https://mlsanity.com/paper/2305.00650/) (2023)
- [Rethinking Confidence Calibration for Failure Prediction](https://mlsanity.com/paper/2303.02970/) (2023)
- [Improved Probabilistic Image-Text Representations](https://mlsanity.com/paper/2305.18171/) (2023)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [Robust Calibration of Large Vision-Language Adapters](https://mlsanity.com/paper/2407.13588/) (2024)
- [Noisy Feature Mixup](https://mlsanity.com/paper/2110.02180/) (2021)
- [A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective](https://mlsanity.com/paper/2208.09913/) (2022)
- [Beyond Confidence: Reliable Models Should Also Consider Atypicality](https://mlsanity.com/paper/2305.18262/) (2023)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [Towards Understanding the Data Dependency of Mixup-style Training](https://mlsanity.com/paper/2110.07647/) (2021)
- [The Benefits of Mixup for Feature Learning](https://mlsanity.com/paper/2303.08433/) (2023)
- [RankMixup: Ranking-Based Mixup Training for Network Calibration](https://mlsanity.com/paper/2308.11990/) (2023)
- [Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup](https://mlsanity.com/paper/2210.13512/) (2022)
- [FIFA: Making Fairness More Generalizable in Classifiers Trained on Imbalanced Data](https://mlsanity.com/paper/2206.02792/) (2022)
- [Over-training with Mixup May Hurt Generalization](https://mlsanity.com/paper/2303.01475/) (2023)
- [A Benchmark Study on Calibration](https://mlsanity.com/paper/2308.11838/) (2023)
- [SimCalib: Graph Neural Network Calibration based on Similarity between Nodes](https://mlsanity.com/paper/2312.11858/) (2023)
- [Revisiting Confidence Estimation: Towards Reliable Failure Prediction](https://mlsanity.com/paper/2403.02886/) (2024)
- [Reinforcement Learning with Stepwise Fairness Constraints](https://mlsanity.com/paper/2211.03994/) (2022)
- [Scaffolding Sets](https://mlsanity.com/paper/2111.03135/) (2021)
- [Human-in-the-Loop Mixup](https://mlsanity.com/paper/2211.01202/) (2022)
- [Data Augmentation via Subgroup Mixup for Improving Fairness](https://mlsanity.com/paper/2309.07110/) (2023)
- [Feature Clipping for Uncertainty Calibration](https://mlsanity.com/paper/2410.19796/) (2024)
- [On the Limitations of Temperature Scaling for Distributions with Overlaps](https://mlsanity.com/paper/2306.00740/) (2023)
- [LUMix: Improving Mixup by Better Modelling Label Uncertainty](https://mlsanity.com/paper/2211.15846/) (2022)
- [Provable Benefit of Mixup for Finding Optimal Decision Boundaries](https://mlsanity.com/paper/2306.00267/) (2023)

## Abstract (arXiv)

In many machine learning applications, it is important for the model to provide confidence scores that accurately capture its prediction uncertainty. Although modern learning methods have achieved great success in predictive accuracy, generating calibrated confidence scores remains a major challenge. Mixup, a popular yet simple data augmentation technique based on taking convex combinations of pairs of training examples, has been empirically found to significantly improve confidence calibration across diverse applications. However, when and how Mixup helps calibration is still a mystery. In this paper, we theoretically prove that Mixup improves calibration in textithigh-dimensional settings by investigating natural statistical models. Interestingly, the calibration benefit of Mixup increases as the model capacity increases. We support our theories with experiments on common architectures and datasets. In addition, we study how Mixup improves calibration in semi-supervised learning. While incorporating unlabeled data can sometimes make the model less calibrated, adding Mixup training mitigates this issue and provably improves calibration. Our analysis provides new insights and a framework to understand Mixup and calibration.