# RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness

arXiv: https://arxiv.org/abs/2206.14502 · HTML: https://mlsanity.com/paper/2206.14502/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 70 references that are in the map.
- First posted on arXiv 29 Jun 2022.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Certified Robustness to Adversarial Examples with Differential Privacy](https://mlsanity.com/paper/1802.03471/) (2018)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)

## Built on by (most cited first, 27 of 27)

- [A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others](https://mlsanity.com/paper/2212.04825/) (2022)
- [Discover and Cure: Concept-aware Mitigation of Spurious Correlation](https://mlsanity.com/paper/2305.00650/) (2023)
- [Sample-dependent Adaptive Temperature Scaling for Improved Calibration](https://mlsanity.com/paper/2207.06211/) (2022)
- [OpenMix: Exploring Outlier Samples for Misclassification Detection](https://mlsanity.com/paper/2303.17093/) (2023)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [Towards Building Self-Aware Object Detectors via Reliable Uncertainty Quantification and Calibration](https://mlsanity.com/paper/2307.00934/) (2023)
- [A Data-Driven Measure of Relative Uncertainty for Misclassification Detection](https://mlsanity.com/paper/2306.01710/) (2023)
- [Over-training with Mixup May Hurt Generalization](https://mlsanity.com/paper/2303.01475/) (2023)
- [Does Progress On Object Recognition Benchmarks Improve Real-World Generalization?](https://mlsanity.com/paper/2307.13136/) (2023)
- [Multimodality Invariant Learning for Multimedia-Based New Item Recommendation](https://mlsanity.com/paper/2405.15783/) (2024)
- [Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup](https://mlsanity.com/paper/2305.16817/) (2023)
- [Raising the Bar on the Evaluation of Out-of-Distribution Detection](https://mlsanity.com/paper/2209.11960/) (2022)
- [OpenCIL: Benchmarking Out-of-Distribution Detection in Class-Incremental Learning](https://mlsanity.com/paper/2407.06045/) (2024)
- [Tailoring Mixup to Data for Calibration](https://mlsanity.com/paper/2311.01434/) (2023)
- [Class Uncertainty: A Measure to Mitigate Class Imbalance](https://mlsanity.com/paper/2311.14090/) (2023)
- [Improving EEG Classification Through Randomly Reassembling Original and Generated Data with Transformer-based Diffusion Models](https://mlsanity.com/paper/2407.20253/) (2024)
- [Supervision Interpolation via LossMix: Generalizing Mixup for Object Detection and Beyond](https://mlsanity.com/paper/2303.10343/) (2023)
- [Query-based Hard-Image Retrieval for Object Detection at Test Time](https://mlsanity.com/paper/2209.11559/) (2022)
- [Infinite Class Mixup](https://mlsanity.com/paper/2305.10293/) (2023)
- [A Metacognitive Approach to Out-of-Distribution Detection for Segmentation](https://mlsanity.com/paper/2311.07578/) (2023)
- [OpenEarthSensing: Large-Scale Fine-Grained Benchmark for Open-World Remote Sensing](https://mlsanity.com/paper/2502.20668/) (2025)
- [A Comprehensive Assessment Benchmark for Rigorously Evaluating Deep Learning Image Classifiers](https://mlsanity.com/paper/2308.04137/) (2023)
- [SimpleDG: Simple Domain Generalization Baseline without Bells and Whistles](https://mlsanity.com/paper/2210.14507/) (2022)
- [Enhancing Visual Domain Adaptation with Source Preparation](https://mlsanity.com/paper/2306.10142/) (2023)
- [TriAug: Out-of-Distribution Detection for Imbalanced Breast Lesion in Ultrasound](https://mlsanity.com/paper/2402.07452/) (2024)
- [Mix from Failure: Confusion-Pairing Mixup for Long-Tailed Recognition](https://mlsanity.com/paper/2411.07621/) (2024)
- [TrustLoRA: Low-Rank Adaptation for Failure Detection under Out-of-distribution Data](https://mlsanity.com/paper/2504.14545/) (2025)

## Abstract (arXiv)

We show that the effectiveness of the well celebrated Mixup [Zhang et al., 2018] can be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss. This simple change not only provides much improved accuracy but also significantly improves the quality of the predictive uncertainty estimation of Mixup in most cases under various forms of covariate shifts and out-of-distribution detection experiments. In fact, we observe that Mixup yields much degraded performance on detecting out-of-distribution samples possibly, as we show empirically, because of its tendency to learn models that exhibit high-entropy throughout; making it difficult to differentiate in-distribution samples from out-distribution ones. To show the efficacy of our approach (RegMixup), we provide thorough analyses and experiments on vision datasets (ImageNet & CIFAR-10/100) and compare it with a suite of recent approaches for reliable uncertainty estimation.