# On Mixup Regularization

arXiv: https://arxiv.org/abs/2006.06049 · HTML: https://mlsanity.com/paper/2006.06049/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2020.

## Built on by (most cited first, 30 of 83)

- [MedViT: A Robust Vision Transformer for Generalized Medical Image Classification](https://mlsanity.com/paper/2302.09462/) (2023)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts](https://mlsanity.com/paper/2305.19951/) (2023)
- [Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks](https://mlsanity.com/paper/2201.03299/) (2022)
- [BEARS Make Neuro-Symbolic Models Aware of their Reasoning Shortcuts](https://mlsanity.com/paper/2402.12240/) (2024)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [Geodesic Multi-Modal Mixup for Robust Fine-Tuning](https://mlsanity.com/paper/2203.03897/) (2022)
- [Noisy Feature Mixup](https://mlsanity.com/paper/2110.02180/) (2021)
- [A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective](https://mlsanity.com/paper/2208.09913/) (2022)
- [Survey: Image Mixing and Deleting for Data Augmentation](https://mlsanity.com/paper/2106.07085/) (2021)
- [Prompting Language-Informed Distribution for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2305.14428/) (2023)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [Towards Understanding the Data Dependency of Mixup-style Training](https://mlsanity.com/paper/2110.07647/) (2021)
- [The Benefits of Mixup for Feature Learning](https://mlsanity.com/paper/2303.08433/) (2023)
- [Toward Degree Bias in Embedding-Based Knowledge Graph Completion](https://mlsanity.com/paper/2302.05044/) (2023)
- [k-Mixup Regularization for Deep Learning via Optimal Transport](https://mlsanity.com/paper/2106.02933/) (2021)
- [Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup](https://mlsanity.com/paper/2210.13512/) (2022)
- [Semi-Supervised Graph Imbalanced Regression](https://mlsanity.com/paper/2305.12087/) (2023)
- [Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shifts](https://mlsanity.com/paper/2302.06495/) (2023)
- [Preventing Manifold Intrusion with Locality: Local Mixup](https://mlsanity.com/paper/2201.04368/) (2022)
- [GenLabel: Mixup Relabeling using Generative Models](https://mlsanity.com/paper/2201.02354/) (2022)
- [Improving Adversarial Transferability via Model Alignment](https://mlsanity.com/paper/2311.18495/) (2023)
- [A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition](https://mlsanity.com/paper/2211.09146/) (2022)
- [DP-Mix: Mixup-based Data Augmentation for Differentially Private Learning](https://mlsanity.com/paper/2311.01295/) (2023)
- [Adversarially Robust Models may not Transfer Better: Sufficient Conditions for Domain Transferability from the View of Regularization](https://mlsanity.com/paper/2202.01832/) (2022)
- [Sill-Net: Feature Augmentation with Separated Illumination Representation](https://mlsanity.com/paper/2102.03539/) (2021)
- [RegMix: Data Mixing Augmentation for Regression](https://mlsanity.com/paper/2106.03374/) (2021)
- [Contrastive-mixup learning for improved speaker verification](https://mlsanity.com/paper/2202.10672/) (2022)
- [Negative Sampling with Adaptive Denoising Mixup for Knowledge Graph Embedding](https://mlsanity.com/paper/2310.09781/) (2023)

## Abstract (arXiv)

Mixup is a data augmentation technique that creates new examples as convex combinations of training points and labels. This simple technique has empirically shown to improve the accuracy of many state-of-the-art models in different settings and applications, but the reasons behind this empirical success remain poorly understood. In this paper we take a substantial step in explaining the theoretical foundations of Mixup, by clarifying its regularization effects. We show that Mixup can be interpreted as standard empirical risk minimization estimator subject to a combination of data transformation and random perturbation of the transformed data. We gain two core insights from this new interpretation. First, the data transformation suggests that, at test time, a model trained with Mixup should also be applied to transformed data, a one-line change in code that we show empirically to improve both accuracy and calibration of the prediction. Second, we show how the random perturbation of the new interpretation of Mixup induces multiple known regularization schemes, including label smoothing and reduction of the Lipschitz constant of the estimator. These schemes interact synergistically with each other, resulting in a self calibrated and effective regularization effect that prevents overfitting and overconfident predictions. We corroborate our theoretical analysis with experiments that support our conclusions.